Differences

This shows you the differences between two versions of the page.

--- provenance:literature:bibliography [2026/09/17 07:23] – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude
+++ provenance:literature:bibliography [2026/09/17 08:43] (current) – Clarify Unicode key folding in bibliography close-out; Authored by Claude karel.kubicek.claude
@@ Line 3811: / Line 3811: @@
 Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
+===== Close-out, 2026-09-17: duplicate verification, re-key, and save guard =====
+This is the one consolidated close-out for the twelve drain filings that
+described the bibliography defect. No second merge was needed: the five-pair
+consolidation already shipped on 2026-09-04 under
+''dedup-regional-filter-lists-bibkey''. The remaining work below was completed
+in this sitting and is recorded here as one pass.
+^ Status ^ Filed items ^
+| **Already shipped before this sitting; no merge repeated here** | ''merge duplicate bouhoula bibliography keys''; ''bib dedup: 2 pre-existing duplicate keys''; ''dedupe fouad2022 bibliography keys''; ''dedup-bibliography-duplicate-papers''; ''bibliography duplicate keys''; ''bibliography DOI duplicates''; ''dedupe-bibliography''; ''bibliography-doi-title-scan''; ''bib-duplicate-dois''; ''bib-duplicate-urls''; ''dedupe-bibliography-entries'' — all covered by the five-pair save and its 2026-09-04 audit |
+| **Residual work closed in this sitting** | ''bib-dedup-close-out'' — the two malformed keys were re-keyed, the ''bibgen.mjs'' producer was fixed, the marker-regex decision and existing dispositions table were recorded, the three-way save guard was added, and the known missing-DOI disposition was closed |
+==== Verification before the residual work ====
+A fresh live export of ''literature:bibliography'' at revision **1789592914**
+contained **1,057** entries. ''scripts/bib_doi_duplicates.py'' returned zero
+colliding citekeys, zero shared-DOI groups, and zero same-title/no-DOI groups.
+Fresh raw and XHTML exports covered **188** pages. The duplicate-pair part of
+''scripts/bib_dupe_cocitation.py'' found no page citing both keys:
+^ Pair ^ first key ^ second key ^ both on one page ^
+| ''fouad2022my'' / ''fouad2022_cookie'' | 0 | 7 | 0 |
+| ''bottger2025_regional'' / ''boettger2025_regional'' | 5 | 0 | 0 |
+| ''ahmad2026_ipfp'' / ''ahmad2026_more'' | 0 | 2 | 0 |
+| ''bouhoula2024automated'' / ''bouhoula2024_automated'' | 0 | 16 | 0 |
+| ''lerner2016internet'' / ''lerner2016_internet'' | 0 | 5 | 0 |
+The co-citation command's process exit was 1 only because its independent PETS
+smoke check flagged three current cache rows — ''Anonymous'', ''Gareth Tyson'',
+and ''Aniketh Girish'' — not because a duplicate pair survived. The pair check
+itself is clean. The two AndroZoo records remain separate: one is a 2016
+dataset paper and the other a 2024 retrospective, despite their braced short
+titles.
+==== Residuals completed ====
+''khrer2015_going'' is now ''kuhrer2015_going'' (Kührer), and
+''som2017_content'' is now ''some2017_content'' (Somé). A rendered-DOM grep of
+all 188 XHTML exports found the old first key twice on ''design:dns'' and the
+old second key once on ''security:headers'', with no other old-key references;
+those three markers were repointed. The producer now uses NFKD, removes
+combining marks, folds common non-decomposing Latin letters, and only then
+applies the ASCII key filter, so ''bibgen.mjs'' generates ''kuhrer2015_going''
+and ''some2017_content''.
+The decision for ''bib_dedup_apply.py'' is **do not change the marker regex**.
+It deliberately counts the three contributor-instruction examples — ''{[key]}''
+on ''privacy:browser_storage'' and ''{[LePochat2019_tranco]}'' on
+''privacy:requests'' and ''statistics:pvalue_corrections'' — as markers in its
+published mechanical counts. None is one of the five duplicate keys, so
+skipping them would change a disclosure-only number without changing the
+consolidation. ''scripts/bib_dedup_review-notes.txt'' was already the 2026-09-04
+dispositions table; it was retained, with its flag example made markup-safe.
+The write-time guard is ''scripts/bib_collision_guard.mjs''. It parses the full
+proposed bibliography and rejects duplicate **key string**, **normalised DOI**,
+or **NFKD normalised title** values. ''scripts/dw.mjs'' runs it from
+''runPreSaveGuards'' whenever the id is ''literature:bibliography'', before
+calling ''core.savePage''. The live 1,057-entry file passes; three small
+mutation fixtures, one for each identity, were rejected with a non-zero exit.
+<file javascript bib_collision_guard.mjs>
+#!/usr/bin/env node
+// Pre-save guard for literature:bibliography.
+//
+// A citekey check alone misses the same paper under two keys.  A DOI check
+// alone misses PETS and USENIX records, which often have no DOI.  The complete
+// guard checks the three identities that can collide: key string, DOI, and a
+// normalised title.  It reads a proposed full bibliography from a file or
+// stdin and never mutates it.
+import fs from 'node:fs';
+const args = process.argv.slice(2);
+if (args.length !== 1) {
+  console.error('usage: node scripts/bib_collision_guard.mjs <bibliography.txt|->');
+  process.exit(2);
+}
+const text = args[0] === '-' ? fs.readFileSync(0, 'utf8') : fs.readFileSync(args[0], 'utf8');
+function parseEntries(source) {
+  const found = [];
+  const starts = /(^|\n)[ \t]*@[A-Za-z][A-Za-z0-9_-]*\s*\{/gm;
+  for (const match of source.matchAll(starts)) {
+    const at = match.index + match[0].lastIndexOf('@');
+    const open = source.indexOf('{', at);
+    let depth = 0;
+    let close = -1;
+    for (let i = open; i < source.length; i++) {
+      if (source[i] === '{') depth++;
+      else if (source[i] === '}') {
+        depth--;
+        if (depth === 0) {
+          close = i;
+          break;
+        }
+      }
+    }
+    if (close === -1) throw new Error(`unbalanced braces near offset ${at}`);
+    const body = source.slice(open + 1, close);
+    const comma = body.indexOf(',');
+    if (comma === -1) throw new Error(`BibTeX entry has no citekey comma near offset ${at}`);
+    found.push({
+      key: body.slice(0, comma).trim(),
+      text: source.slice(at, close + 1),
+    });
+  }
+  return found;
+}
+function field(entry, name) {
+  const re = new RegExp(`(?:^|[\\n,])[ \\t]*${name}\\s*=\\s*`, 'i');
+  const match = re.exec(entry);
+  if (!match) return null;
+  let i = match.index + match[0].length;
+  if (entry[i] === '{') {
+    const open = i;
+    let depth = 0;
+    for (; i < entry.length; i++) {
+      if (entry[i] === '{') depth++;
+      else if (entry[i] === '}') {
+        depth--;
+        if (depth === 0) return entry.slice(open + 1, i);
+      }
+    }
+    throw new Error(`unbalanced ${name} field in ${entry.slice(0, 60)}`);
+  }
+  if (entry[i] === '"') {
+    const open = i++;
+    for (; i < entry.length; i++) {
+      if (entry[i] === '"' && entry[i - 1] !== '\\') return entry.slice(open + 1, i);
+    }
+    throw new Error(`unbalanced quoted ${name} field in ${entry.slice(0, 60)}`);
+  }
+  const end = entry.slice(i).search(/[,\n}]/);
+  return entry.slice(i, end === -1 ? entry.length : i + end).trim();
+}
+function normaliseDoi(value) {
+  return value.trim().toLowerCase()
+    .replace(/^https?:\/\/(?:dx\.)?doi\.org\//, '')
+    .replace(/^doi:\s*/, '')
+    .replace(/\s+/g, '');
+}
+function normaliseTitle(value) {
+  return value.normalize('NFKD').toLowerCase().replace(/[^a-z0-9]/g, '');
+}
+function collisions(entries, label, valueOf) {
+  const groups = new Map();
+  for (const entry of entries) {
+    const value = valueOf(entry);
+    if (!value) continue;
+    if (!groups.has(value)) groups.set(value, []);
+    groups.get(value).push(entry.key);
+  }
+  return [...groups.entries()]
+    .filter(([, keys]) => keys.length > 1)
+    .map(([value, keys]) => `${label} collision: ${value} (${keys.join(', ')})`);
+}
+if (!/<bibtex\b[^>]*>[\s\S]*<\/bibtex>/i.test(text)) {
+  console.error('REFUSING: bibliography has no complete <bibtex> block');
+  process.exit(1);
+}
+let entries;
+try {
+  entries = parseEntries(text);
+} catch (error) {
+  console.error(`REFUSING: ${error.message}`);
+  process.exit(1);
+}
+const bad = [
+  ...collisions(entries, 'key', (e) => e.key),
+  ...collisions(entries, 'DOI', (e) => {
+    const value = field(e.text, 'doi');
+    return value ? normaliseDoi(value) : null;
+  }),
+  ...collisions(entries, 'title', (e) => {
+    const value = field(e.text, 'title');
+    return value ? normaliseTitle(value) : null;
+  }),
+];
+console.log(`bibliography entries: ${entries.length}`);
+if (bad.length) {
+  console.error('REFUSING TO SAVE: bibliography collisions found');
+  for (const problem of bad) console.error(`  ${problem}`);
+  process.exit(1);
+}
+console.log('key/DOI/normalised-title collisions: 0');
+</file>
+''lukic2026_mv3'' remains the earlier-sitting fix with DOI
+''10.56553/popets-2026-0027''. ''calzavara2026_syntactic'' remains without a DOI:
+its url points to a paper list rather than an article, so there is no article
+number to derive. After the three saves, ''?purge=true'' was requested on the
+bibliography and on ''design:dns'' and ''security:headers'', and fresh raw/XHTML
+exports showed the new keys resolving and no old key in a citation marker.