Differences

This shows you the differences between two versions of the page.

--- provenance:literature:corpus [2026/09/04 08:28] – Review pass: fix 5,848 to 5,853 in Q10 and the denominators, state the geometry detector's missing negative control, name all 57 pages with a stale quote-check figure, correct what was read by hand, and add the generic + re-run review logs. Authored by Cl karel.kubicek.claude
+++ provenance:literature:corpus [2026/09/17 08:14] (current) – Record mirror decision and sitemap caveat; Authored by Claude karel.kubicek.claude
@@ Line 24: / Line 24: @@
 The two full sentences are missing from ''paper.txt'' and ''paper.norm.txt'' for the ordinary reason: those renderings keep the columns side by side, so no sentence spanning a line is contiguous in them. That is the entire reason ''paper.cols.txt'' exists.
-==== Why it was believed: the tool called ''grep'' here is not grep ====
+==== Why it was believed: the tool called grep here is not grep ====
 ''pdftotext'' writes U+0000 wherever a PDF's font encoding maps a glyph to nothing. That paper's ''paper.cols.txt'' has **181** of them, so it is a binary file to any grep. What happens next depends on which grep, and the one you get by typing ''grep'' in this agent's shell is **not the one installed on the machine**:
@@ Line 116: / Line 116: @@
 | Q1 | Do the quotes the retracted section named exist in the stored renderings? | one paper, PoPETs 2025 ''more-and-scammier-ads-…'' | 4 quotes + 10 authors | all present in ''paper.cols.txt''; the retraction is Q0 of this page |
 | Q2 | How many stored renderings does a shell grep report nothing for? | every extraction record | 5,859 papers | 90 (1.5%), holding 2,156 of 135,025 quotes |
-| Q3 | How many quotes does the stored rendering fail to locate? | quotes with a verdict on both sides | 135,004 quotes | 1,708 (1.3%) — the same count ''report_corpus.mjs --quotes'' reports over all 135,025 |
+| Q3 | How many quotes does the stored rendering fail to locate? | quotes with a verdict on both sides | 135,004 quotes | 1,708 (1.3%) — the same count ''report_corpus.mjs %%--quotes%%'' reports over all 135,025 |
 | Q4 | How many of those does an independent PDF rendering find? | the 1,708 | 1,708 quotes | **1,250 (73.2%)**; 511 verbatim |
 | Q5 | How many quotes can neither rendering locate? | the same 135,004 | 135,004 quotes | 458 (0.34%) |
@@ Line 132: / Line 132: @@
 ===== 4. The reproduction check that makes the rest usable =====
-''report_cols_vs_pdf.mjs'' does not approximate ''report_corpus.mjs --quotes'': its four-verdict function is copied from it, including the ''fragmented'' anchor search. Run on the same day, on the same records:
+''report_cols_vs_pdf.mjs'' does not approximate ''report_corpus.mjs %%--quotes%%'': its four-verdict function is copied from it, including the ''fragmented'' anchor search. Run on the same day, on the same records:
-^ Verdict ^ ''report_corpus.mjs --quotes'' ^ this audit ^ Difference ^
+^ Verdict ^ ''report_corpus.mjs %%--quotes%%'' ^ this audit ^ Difference ^
 | exact | 78,450 | 78,435 | 15 |
 | an 8-word run survives | 49,425 | 49,419 | 6 |
@@ Line 182: / Line 182: @@
   * **The geometry detector has no negative control.** It is 60 for 60 on pages ''decolumn.mjs'' reflowed, which bounds its false negatives. Nothing bounds its **false positives**: no set of known single-column pages was assembled and put through it. If it over-calls two-column, the 22,475 estimate is too high, and the 0.6–1.0 band's unexplained 47.5% is the shape that would suggest. This is the weakest link in that number.
   * **Whether a third reader would agree with ''pypdf''.** ''pdftotext'', ''pdfminer'', ''PyMuPDF'', ''pdfplumber'', ''mutool'', ''qpdf'' and ''ghostscript'' are all absent from this container. Every "present in the PDF" verdict rests on one reader whose own not-found rate (2.3%) is worse than the rendering it audits (1.3%). The 73.2% is a lower bound in one sense — a better reader would rescue more — and unreplicated in another.
-  * **One paper still has no second rendering.** ''NDSS/2025/mtzk-testing-and-exploring-bugs-in-zero-knowledge-zk-compilers'' uses AES encryption, so ''pypdf'' needs the ''cryptography'' package. ''pip install cryptography'' is refused here by PEP 668, and ''--break-system-packages'' was **declined** rather than risk the container for one paper's 21 quotes. Anyone with a normal Python environment can close this in one command.
+  * **One paper still has no second rendering.** ''NDSS/2025/mtzk-testing-and-exploring-bugs-in-zero-knowledge-zk-compilers'' uses AES encryption, so ''pypdf'' needs the ''cryptography'' package. ''pip install cryptography'' is refused here by PEP 668, and ''%%--break-system-packages%%'' was **declined** rather than risk the container for one paper's 21 quotes. Anyone with a normal Python environment can close this in one command.
   * **The 57 pages that already publish a per-page quote-check figure.** Each was computed against the stored rendering alone and each overstates its failure rate. They were not re-run; only [[statistics:hypothesis_testing]] was, because its own figures prompted the audit. The list, so the next editor does not re-derive it, is every page matching ''quote_check %%|%% quotecheck %%|%% below threshold %%|%% five-word windows %%|%% 5-word windows'' over a fresh export: ''design:website:classification'', ''privacy:fingerprinting'', ''privacy:tcf:consent:strings'', ''programming:crawler'', ''programming:crawler:detection'', ''programming:crawler:foxhound'', ''programming:crawler:llm:agents'', ''programming:crawler:pagegraph'', ''programming:crawler:panoptichrome'', ''programming:crawler:webxray'', ''provenance:artifacts'', ''provenance:design:archives'', ''provenance:design:existing:datasets'', ''provenance:design:ip:classification'', ''provenance:design:longitudinal'', ''provenance:design:mobile:and:app:measurement'', ''provenance:design:platforms'', ''provenance:design:website:classification'', ''provenance:practices:ethics'', ''provenance:practices:legal:enforcement'', ''provenance:privacy:ads:txt'', ''provenance:privacy:browser:extensions'', ''provenance:privacy:browser:protection'', ''provenance:privacy:browser:storage'', ''provenance:privacy:consent'', ''provenance:privacy:cookie:syncing'', ''provenance:privacy:email:tracking'', ''provenance:privacy:fingerprinting'', ''provenance:privacy:javascript'', ''provenance:privacy:privacy:sandbox'', ''provenance:privacy:server:side:tracking'', ''provenance:privacy:tcf:consent:strings'', ''provenance:programming:crawler'', ''provenance:programming:crawler:detection'', ''provenance:programming:crawler:foxhound'', ''provenance:programming:crawler:llm:agents'', ''provenance:programming:crawler:openwpm'', ''provenance:programming:crawler:pagegraph'', ''provenance:programming:crawler:panoptichrome'', ''provenance:programming:crawler:webxray'', ''provenance:programming:interaction'', ''provenance:programming:interaction:outputs'', ''provenance:programming:stateful:stateless'', ''provenance:programming:traffic:files'', ''provenance:programming:tranco'', ''provenance:security:tls:certificates'', ''provenance:security:web:vulnerabilities'', ''provenance:statistics:biases'', ''provenance:statistics:how:many:sites'', ''provenance:statistics:hypothesis:testing'', ''provenance:statistics:pvalue:corrections'', ''provenance:statistics:regression'', ''security:tls:certificates'', ''statistics:biases'', ''statistics:hypothesis:testing'', ''statistics:pvalue:corrections'', ''statistics:regression''. Filed as deferred work rather than left as a note here.
   * **Field stability.** Still measured only on the 4,322-paper corpus, still not re-measured, and untouched by this audit.
@@ Line 194: / Line 194: @@
 | **The retracted section on [[provenance:statistics:hypothesis_testing]] was kept, marked, rather than deleted** | Deleting it would leave the page cleaner and the mistake unlearnable. The failure mode — reading a tool's silence as a negative result — is the most reusable thing either page contains |
 | **''pypdf'''s decompression limits were raised rather than left as a gap** | The limits exist to bound a decompression bomb. These are five papers from seven known venues, already on disk and already read once by the extraction, so the bound was not earning anything; leaving 151 quotes unchecked to preserve it would have been safety theatre. The opposite call is defensible if you do not trust the corpus |
-| **''--break-system-packages'' was declined for the sixth paper** | It would have closed the last gap for 21 quotes at the cost of writing into a PEP 668-managed system Python that every other script in this workdir depends on. Someone less cautious would have run it; the page says exactly what the one-command fix is |
+| **''%%--break-system-packages%%'' was declined for the sixth paper** | It would have closed the last gap for 21 quotes at the cost of writing into a PEP 668-managed system Python that every other script in this workdir depends on. Someone less cautious would have run it; the page says exactly what the one-command fix is |
 | **The estimate of 22,475 unrepaired pages is published with a CI, from a 400-page sample**, rather than by classifying all 29,347 | Classifying all of them means opening every PDF a second time, about 40 minutes, for a number whose CI is already ±3 percentage points |
 | **The band-wise Wilson intervals are combined by summing weighted endpoints** | This is **conservative, not tight**: it behaves as if each band's sampling error moved with the others, so the published interval is wider than the truth. The tighter and equally honest alternative is a stratified-variance sum — Var(total) = Σ //N//ᵢ²·Var(//p̂//ᵢ), then //z//·√Var — which the external-currency reviewer named. The wider interval was kept because nothing on the page needs the tighter one, but this is a choice, not the only honest option |
@@ Line 232: / Line 232: @@
 Four passes, all given the page text, the scripts, their output and these notes, and all told explicitly that the author's context might not be exhaustive. Three ran in parallel first; their findings were fixed and the figures re-derived before the fourth.
-==== Figures against the scripts (''sonnet'') ====
+==== Figures against the scripts — sonnet ====
 ^ # ^ Finding ^ Disposition ^
@@ Line 240: / Line 240: @@
 | F4 | ''audit_decolumn_pages.mjs'' counts a word-bag-rejected page in both ''rejected'' and ''untouched'' | **NOTED, not fixed.** The corpus has 0 rejected pages, so it affects no published number. Left as a latent defect with this note |
-==== Quotes and attribution (''sonnet'') ====
+==== Quotes and attribution — sonnet ====
 ^ # ^ Finding ^ Disposition ^
@@ Line 250: / Line 250: @@
 | Q6 | Cross-page figures, citekeys and the ''decolumn.mjs''/''normalize_text.mjs'' source claims all check out; the deleted footnote orphaned no citation; the base files are byte-identical to the live pages | No change |
-==== External and tool currency (''sonnet'') ====
+==== External and tool currency — sonnet ====
 ^ # ^ Finding ^ Disposition ^
@@ Line 261: / Line 261: @@
 | C7 | ''visitor_text'''s signature and ''tm[4]'' are current and not deprecated; ''extraction_mode="layout"'' does not support the visitor callbacks, so it is not an alternative. DokuWiki's %%<WRAP>%%/%%<wrap>%% div-vs-span behaviour, ''%%''…''%%'' being monospace rather than nowiki, and %%<file text name.txt>%% are all current. Every internal link resolves, including the ''#quote_groundedness_re-measured_on_this_corpus'' anchor | No change |
-==== Generic (''fable'') ====
+==== Generic — fable ====
 Run after the three above were fixed, with no checklist.
 ^ # ^ Finding ^ Disposition ^
-| G1 | **[[literature:corpus]] published a stale copy of the dose-response table** — three rows and the "5,848 papers" denominator predated the ''pypdf'' limit raise, and contradicted this page's own Appendix A two hundred lines below | **ACCEPTED and FIXED**, and the cause fixed too: the content page's tables were hand-typed literals. ''edit_corpus.py'' now **slices them out of the report script's own ''--wiki'' output**, so they cannot drift again. The re-run figures pass found the same defect independently, which is the strongest signal in this review log |
+| G1 | **[[literature:corpus]] published a stale copy of the dose-response table** — three rows and the "5,848 papers" denominator predated the ''pypdf'' limit raise, and contradicted this page's own Appendix A two hundred lines below | **ACCEPTED and FIXED**, and the cause fixed too: the content page's tables were hand-typed literals. ''edit_corpus.py'' now **slices them out of the report script's own ''%%--wiki%%'' output**, so they cannot drift again. The re-run figures pass found the same defect independently, which is the strongest signal in this review log |
 | G2 | **The two quote-scoring rules were still conflated** on both hypothesis-testing pages: each described this page as "the corpus-wide version of the same measurement" when the thresholds differ | **ACCEPTED and FIXED** on both, plus a 73.1% that should have been 73.2% |
 | G3 | **[[literature:corpus]] now contradicts its own untouched sections**: §11 had a row headed //"Why there is no ''provenance:literature:corpus''"//, §9 said every figure on the page comes from one script, and §11 said "the six ''provenance:'' pages" | **ACCEPTED and FIXED.** This is the //carve-out leaves day-one drift// pattern exactly: adding a section left literal sentences elsewhere on the page false |
@@ Line 278: / Line 278: @@
 | G11 | What not to change: §3's query/denominator table, §4's reproduction check, §8's judgement calls, the two "not fixed / not re-run" bullets on the content page, the RESCUED verdict and ''textSource'' keying, and this log's finding/disposition format | No change. Recorded because a review that lists only defects gives no signal about what to leave alone |
-==== Figures, re-run after the fixes above (''sonnet'') ====
+==== Figures, re-run after the fixes above — sonnet ====
 ^ # ^ Finding ^ Disposition ^
@@ Line 615: / Line 615: @@
 NUL  19 quotes  USENIX/2022/neither-access-nor-control-a-longitudinal-investigation-of-the-efficacy-of-user
 </file>
+===== Markup sweep, 2026-09-17 =====
+Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
+===== Link-hygiene decision and sitemap caveat, 2026-09-17 =====
+**Decision:** keep the pre-rule exemption for ''programming:crux'' and ''programming:similarweb''; both content pages predate the ship-a-provenance-page rule. ''privacy:cookies'' remains covered by its own page-refresh item. Treat ''provenance:programming:interaction:outputs'' as an intentional provenance-only overflow log with no content counterpart, exempt from the mirror-content-id rule. No new mirrors were created.
+**Caveat:** runs between 2026-09-04 and 2026-09-05 that relied on step 1 of the link workflow got a vacuous pass: ''sitemap.mjs'' stopped matching the moved ''www'' sitemap origin, turned every id into the whole URL, and reported zero internal targets and missing pages. Re-check cross-links when relying on those records.
+**Current check:** a fresh 2026-09-17 sitemap/export and rendered-DOM sweep found ''provenance:literature:corpus'' present, with the links from ''literature:corpus'' and ''provenance:statistics:hypothesis_testing'' resolving. No corpus figures or citations changed.