Differences

This shows you the differences between two versions of the page.

--- provenance:literature:bibliography [2026/09/04 18:43] – Second-sitting audit: apply the four review passes' findings (autoKey STOP-list rule, 20-page union, 28/26 candidate split, 160-vs-161, first-sitting labelling, render-check off-by-one) and publish the full review log. Authored by Claude karel.kubicek.claude
+++ provenance:literature:bibliography [2026/09/17 08:43] (current) – Clarify Unicode key folding in bibliography close-out; Authored by Claude karel.kubicek.claude
@@ Line 139: / Line 139: @@
   * **A → B.** A non-greedy brace match ends at the first inner brace, so ''Adam Doup{\'e}'' came back as ''Adam Doup{\'e''. Brace-counting fixes it.
   * **B → C.** For ''iot-market-dynamics'' USENIX's meta says ''Carlos H. Gañán'' and USENIX's own BibTeX says ''Carlos H. Ganan''. That one is a genuine difference between USENIX's two sources; source 1 is preferred for exactly that reason, and the diacritic fold is what stops it being reported as a defect.
-  * **The third, found in review on 2026-09-04.** USENIX writes Benoît as ''Beno{\^\i}t'', using LaTeX's dotless ''\i''. Nothing in the decoder handled ''\i'', so no accent pattern matched and the final brace strip produced ''Beno^it''. It survived every other check because it is on ''cohn2020_delf'', which is **not** one of the 90 cached lists — only the all-162 comparison could see it. ''usenix_author_audit.py --selftest'' now carries a nine-case ''de_latex'' test including that exact string.
+  * **The third, found in review on 2026-09-04.** USENIX writes Benoît as ''Beno{\^\i}t'', using LaTeX's dotless ''\i''. Nothing in the decoder handled ''\i'', so no accent pattern matched and the final brace strip produced ''Beno^it''. It survived every other check because it is on ''cohn2020_delf'', which is **not** one of the 90 cached lists — only the all-162 comparison could see it. ''usenix_author_audit.py %%--selftest%%'' now carries a nine-case ''de_latex'' test including that exact string.
 So: **three decoder bugs, and four pages where USENIX's own BibTeX drops a
@@ Line 625: / Line 625: @@
 None of the ten keys can have been minted from an author list: PETS and USENIX
 index records carry none, so ''autoKey'' would have fallen back to the surname
-in the landing URL or required an explicit ''--key''. The tie-break is the
+in the landing URL or required an explicit ''%%--key%%''. The tie-break is the
 file's own majority convention, **surname + year + ''_'' + first title word,
 diacritics dropped** — where "first title word" is ''bibgen.mjs'' ''autoKey'':
@@ Line 660: / Line 660: @@
   - Run ''scripts/bib_dedup_scan.py'' — the wider scan this item asked for — on the export: 5 definite pairs, 58 candidates.
   - Run ''scripts/bib_dedup_apply.py'' offline. It rewrites the markers and the bibliography into ''out/dedup_apply/'' and asserts the invariants in its docstring. It saves nothing.
-  - Save the **20 pages first** — the 13 whose markers changed plus the 10 given an amendment, three pages being in both sets — each with ''--if-rev'' against the revision read at export, then the bibliography: 21 saves. In that order nothing ever renders unresolved: the kept keys existed while the deleted ones were still cited, and the deleted ones were gone only once no page cited them.
+  - Save the **20 pages first** — the 13 whose markers changed plus the 10 given an amendment, three pages being in both sets — each with ''%%--if-rev%%'' against the revision read at export, then the bibliography: 21 saves. In that order nothing ever renders unresolved: the kept keys existed while the deleted ones were still cited, and the deleted ones were gone only once no page cited them.
   - Purge the bibtex4dw cache on the bibliography and every repointed page (''?purge=true''; see the first sitting for why a rendered check passes on a stale cache without this).
   - Re-fetch the 13 repointed pages and compare rendered reference-list length and marker count against copies fetched before the edit (''scripts/bib_dedup_render_check.py'').
@@ Line 1050: / Line 1050: @@
 | Carry ''pages'' over to ''bouhoula2024_automated''; drop ''isbn'', ''address'', ''publisher'', ''month'' | Merge every field of the deleted entry | ''pages'' was verified against USENIX's page; the other four appear on no other entry, and adding them to one would make that entry the odd one |
 | Leave the 26 provenance pages' historical prose alone; amend only the 10 whose content page was repointed | Rewrite every mention of a deleted key | A review log records what was true when written. Rewriting it is the mistake the first sitting's own review found — a debunked claim surviving in the log — turned inside out |
-| Pages first, bibliography last, one ''--if-rev'' per save | Bibliography first, or one combined pass | The other order leaves 23 markers unresolved between two saves; ''--if-rev'' is what makes a 21-save run safe against an edit landing in the middle |
+| Pages first, bibliography last, one ''%%--if-rev%%'' per save | Bibliography first, or one combined pass | The other order leaves 23 markers unresolved between two saves; ''%%--if-rev%%'' is what makes a 21-save run safe against an edit landing in the middle |
 | Publish the 58 candidates in full | Publish the count and the verdict | A hand judgement over a list nobody can read is not checkable |
 | Extend this page rather than create ''provenance:literature:bibliography:dedup'' | A page per item | The provenance page mirrors the content page's id; the bibliography has one id. Two sittings, two dated audits, one page |
@@ Line 3733: / Line 3733: @@
 across all 162 could reach it. The live entry was always correct and no
 published figure moved; ''de_latex'' now has a nine-case self-test
-(''usenix_author_audit.py --selftest'') whose first case is that string.
+(''usenix_author_audit.py %%--selftest%%'') whose first case is that string.
 Of the four reviewer findings that led to new code rather than new prose, two
@@ Line 3798: / Line 3798: @@
 | Fable — generic | The headline row describes only the surname rule, though the 58 candidates include a title match and the scan also ran on the 850-entry file | **Accepted.** The row now names all four rules and both files |
 | Fable — generic | 13 changed + 10 amended is stated as 20 pages and 21 files without saying three pages are in both sets | **Accepted.** One clause added |
-| Fable — generic | "all ten were typed by hand" is an inference: ''autoKey'' has a no-authors path that derives a surname from ''landingUrl'', and callers can pass ''--key'' | **Accepted.** Softened to what the evidence supports — none can have been minted from an author list |
+| Fable — generic | "all ten were typed by hand" is an inference: ''autoKey'' has a no-authors path that derives a surname from ''landingUrl'', and callers can pass ''%%--key%%'' | **Accepted.** Softened to what the evidence supports — none can have been minted from an author list |
 | Fable — generic | The 19-of-25 census runs on the 855-entry file, so it counts the Böttger paper twice; after the save it is 19 of 24 | **Accepted.** Labelled, with both figures |
 | Fable — generic | Both reviewer corrections are narrated in the body and again in the review log | **Accepted.** Cut from the body; the log is the right place |
@@ Line 3807: / Line 3807: @@
 <bibtex bibliography></bibtex>
+===== Markup sweep, 2026-09-17 =====
+Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
+===== Close-out, 2026-09-17: duplicate verification, re-key, and save guard =====
+This is the one consolidated close-out for the twelve drain filings that
+described the bibliography defect. No second merge was needed: the five-pair
+consolidation already shipped on 2026-09-04 under
+''dedup-regional-filter-lists-bibkey''. The remaining work below was completed
+in this sitting and is recorded here as one pass.
+^ Status ^ Filed items ^
+| **Already shipped before this sitting; no merge repeated here** | ''merge duplicate bouhoula bibliography keys''; ''bib dedup: 2 pre-existing duplicate keys''; ''dedupe fouad2022 bibliography keys''; ''dedup-bibliography-duplicate-papers''; ''bibliography duplicate keys''; ''bibliography DOI duplicates''; ''dedupe-bibliography''; ''bibliography-doi-title-scan''; ''bib-duplicate-dois''; ''bib-duplicate-urls''; ''dedupe-bibliography-entries'' — all covered by the five-pair save and its 2026-09-04 audit |
+| **Residual work closed in this sitting** | ''bib-dedup-close-out'' — the two malformed keys were re-keyed, the ''bibgen.mjs'' producer was fixed, the marker-regex decision and existing dispositions table were recorded, the three-way save guard was added, and the known missing-DOI disposition was closed |
+==== Verification before the residual work ====
+A fresh live export of ''literature:bibliography'' at revision **1789592914**
+contained **1,057** entries. ''scripts/bib_doi_duplicates.py'' returned zero
+colliding citekeys, zero shared-DOI groups, and zero same-title/no-DOI groups.
+Fresh raw and XHTML exports covered **188** pages. The duplicate-pair part of
+''scripts/bib_dupe_cocitation.py'' found no page citing both keys:
+^ Pair ^ first key ^ second key ^ both on one page ^
+| ''fouad2022my'' / ''fouad2022_cookie'' | 0 | 7 | 0 |
+| ''bottger2025_regional'' / ''boettger2025_regional'' | 5 | 0 | 0 |
+| ''ahmad2026_ipfp'' / ''ahmad2026_more'' | 0 | 2 | 0 |
+| ''bouhoula2024automated'' / ''bouhoula2024_automated'' | 0 | 16 | 0 |
+| ''lerner2016internet'' / ''lerner2016_internet'' | 0 | 5 | 0 |
+The co-citation command's process exit was 1 only because its independent PETS
+smoke check flagged three current cache rows — ''Anonymous'', ''Gareth Tyson'',
+and ''Aniketh Girish'' — not because a duplicate pair survived. The pair check
+itself is clean. The two AndroZoo records remain separate: one is a 2016
+dataset paper and the other a 2024 retrospective, despite their braced short
+titles.
+==== Residuals completed ====
+''khrer2015_going'' is now ''kuhrer2015_going'' (Kührer), and
+''som2017_content'' is now ''some2017_content'' (Somé). A rendered-DOM grep of
+all 188 XHTML exports found the old first key twice on ''design:dns'' and the
+old second key once on ''security:headers'', with no other old-key references;
+those three markers were repointed. The producer now uses NFKD, removes
+combining marks, folds common non-decomposing Latin letters, and only then
+applies the ASCII key filter, so ''bibgen.mjs'' generates ''kuhrer2015_going''
+and ''some2017_content''.
+The decision for ''bib_dedup_apply.py'' is **do not change the marker regex**.
+It deliberately counts the three contributor-instruction examples — ''{[key]}''
+on ''privacy:browser_storage'' and ''{[LePochat2019_tranco]}'' on
+''privacy:requests'' and ''statistics:pvalue_corrections'' — as markers in its
+published mechanical counts. None is one of the five duplicate keys, so
+skipping them would change a disclosure-only number without changing the
+consolidation. ''scripts/bib_dedup_review-notes.txt'' was already the 2026-09-04
+dispositions table; it was retained, with its flag example made markup-safe.
+The write-time guard is ''scripts/bib_collision_guard.mjs''. It parses the full
+proposed bibliography and rejects duplicate **key string**, **normalised DOI**,
+or **NFKD normalised title** values. ''scripts/dw.mjs'' runs it from
+''runPreSaveGuards'' whenever the id is ''literature:bibliography'', before
+calling ''core.savePage''. The live 1,057-entry file passes; three small
+mutation fixtures, one for each identity, were rejected with a non-zero exit.
+<file javascript bib_collision_guard.mjs>
+#!/usr/bin/env node
+// Pre-save guard for literature:bibliography.
+//
+// A citekey check alone misses the same paper under two keys.  A DOI check
+// alone misses PETS and USENIX records, which often have no DOI.  The complete
+// guard checks the three identities that can collide: key string, DOI, and a
+// normalised title.  It reads a proposed full bibliography from a file or
+// stdin and never mutates it.
+import fs from 'node:fs';
+const args = process.argv.slice(2);
+if (args.length !== 1) {
+  console.error('usage: node scripts/bib_collision_guard.mjs <bibliography.txt|->');
+  process.exit(2);
+}
+const text = args[0] === '-' ? fs.readFileSync(0, 'utf8') : fs.readFileSync(args[0], 'utf8');
+function parseEntries(source) {
+  const found = [];
+  const starts = /(^|\n)[ \t]*@[A-Za-z][A-Za-z0-9_-]*\s*\{/gm;
+  for (const match of source.matchAll(starts)) {
+    const at = match.index + match[0].lastIndexOf('@');
+    const open = source.indexOf('{', at);
+    let depth = 0;
+    let close = -1;
+    for (let i = open; i < source.length; i++) {
+      if (source[i] === '{') depth++;
+      else if (source[i] === '}') {
+        depth--;
+        if (depth === 0) {
+          close = i;
+          break;
+        }
+      }
+    }
+    if (close === -1) throw new Error(`unbalanced braces near offset ${at}`);
+    const body = source.slice(open + 1, close);
+    const comma = body.indexOf(',');
+    if (comma === -1) throw new Error(`BibTeX entry has no citekey comma near offset ${at}`);
+    found.push({
+      key: body.slice(0, comma).trim(),
+      text: source.slice(at, close + 1),
+    });
+  }
+  return found;
+}
+function field(entry, name) {
+  const re = new RegExp(`(?:^|[\\n,])[ \\t]*${name}\\s*=\\s*`, 'i');
+  const match = re.exec(entry);
+  if (!match) return null;
+  let i = match.index + match[0].length;
+  if (entry[i] === '{') {
+    const open = i;
+    let depth = 0;
+    for (; i < entry.length; i++) {
+      if (entry[i] === '{') depth++;
+      else if (entry[i] === '}') {
+        depth--;
+        if (depth === 0) return entry.slice(open + 1, i);
+      }
+    }
+    throw new Error(`unbalanced ${name} field in ${entry.slice(0, 60)}`);
+  }
+  if (entry[i] === '"') {
+    const open = i++;
+    for (; i < entry.length; i++) {
+      if (entry[i] === '"' && entry[i - 1] !== '\\') return entry.slice(open + 1, i);
+    }
+    throw new Error(`unbalanced quoted ${name} field in ${entry.slice(0, 60)}`);
+  }
+  const end = entry.slice(i).search(/[,\n}]/);
+  return entry.slice(i, end === -1 ? entry.length : i + end).trim();
+}
+function normaliseDoi(value) {
+  return value.trim().toLowerCase()
+    .replace(/^https?:\/\/(?:dx\.)?doi\.org\//, '')
+    .replace(/^doi:\s*/, '')
+    .replace(/\s+/g, '');
+}
+function normaliseTitle(value) {
+  return value.normalize('NFKD').toLowerCase().replace(/[^a-z0-9]/g, '');
+}
+function collisions(entries, label, valueOf) {
+  const groups = new Map();
+  for (const entry of entries) {
+    const value = valueOf(entry);
+    if (!value) continue;
+    if (!groups.has(value)) groups.set(value, []);
+    groups.get(value).push(entry.key);
+  }
+  return [...groups.entries()]
+    .filter(([, keys]) => keys.length > 1)
+    .map(([value, keys]) => `${label} collision: ${value} (${keys.join(', ')})`);
+}
+if (!/<bibtex\b[^>]*>[\s\S]*<\/bibtex>/i.test(text)) {
+  console.error('REFUSING: bibliography has no complete <bibtex> block');
+  process.exit(1);
+}
+let entries;
+try {
+  entries = parseEntries(text);
+} catch (error) {
+  console.error(`REFUSING: ${error.message}`);
+  process.exit(1);
+}
+const bad = [
+  ...collisions(entries, 'key', (e) => e.key),
+  ...collisions(entries, 'DOI', (e) => {
+    const value = field(e.text, 'doi');
+    return value ? normaliseDoi(value) : null;
+  }),
+  ...collisions(entries, 'title', (e) => {
+    const value = field(e.text, 'title');
+    return value ? normaliseTitle(value) : null;
+  }),
+];
+console.log(`bibliography entries: ${entries.length}`);
+if (bad.length) {
+  console.error('REFUSING TO SAVE: bibliography collisions found');
+  for (const problem of bad) console.error(`  ${problem}`);
+  process.exit(1);
+}
+console.log('key/DOI/normalised-title collisions: 0');
+</file>
+''lukic2026_mv3'' remains the earlier-sitting fix with DOI
+''10.56553/popets-2026-0027''. ''calzavara2026_syntactic'' remains without a DOI:
+its url points to a paper list rather than an article, so there is no article
+number to derive. After the three saves, ''?purge=true'' was requested on the
+bibliography and on ''design:dns'' and ''security:headers'', and fresh raw/XHTML
+exports showed the new keys resolving and no old key in a citation marker.