Differences

This shows you the differences between two versions of the page.

--- provenance:literature:corpus [2026/09/04 08:29] – Review-section headings: DokuWiki does not process '' '' inside a heading, so it rendered the quote marks literally. Authored by Claude karel.kubicek.claude
+++ provenance:literature:corpus [2026/09/17 08:14] (current) – Record mirror decision and sitemap caveat; Authored by Claude karel.kubicek.claude
@@ Line 24: / Line 24: @@
 The two full sentences are missing from ''paper.txt'' and ''paper.norm.txt'' for the ordinary reason: those renderings keep the columns side by side, so no sentence spanning a line is contiguous in them. That is the entire reason ''paper.cols.txt'' exists.
-==== Why it was believed: the tool called ''grep'' here is not grep ====
+==== Why it was believed: the tool called grep here is not grep ====
 ''pdftotext'' writes U+0000 wherever a PDF's font encoding maps a glyph to nothing. That paper's ''paper.cols.txt'' has **181** of them, so it is a binary file to any grep. What happens next depends on which grep, and the one you get by typing ''grep'' in this agent's shell is **not the one installed on the machine**:
@@ Line 116: / Line 116: @@
 | Q1 | Do the quotes the retracted section named exist in the stored renderings? | one paper, PoPETs 2025 ''more-and-scammier-ads-…'' | 4 quotes + 10 authors | all present in ''paper.cols.txt''; the retraction is Q0 of this page |
 | Q2 | How many stored renderings does a shell grep report nothing for? | every extraction record | 5,859 papers | 90 (1.5%), holding 2,156 of 135,025 quotes |
-| Q3 | How many quotes does the stored rendering fail to locate? | quotes with a verdict on both sides | 135,004 quotes | 1,708 (1.3%) — the same count ''report_corpus.mjs --quotes'' reports over all 135,025 |
+| Q3 | How many quotes does the stored rendering fail to locate? | quotes with a verdict on both sides | 135,004 quotes | 1,708 (1.3%) — the same count ''report_corpus.mjs %%--quotes%%'' reports over all 135,025 |
 | Q4 | How many of those does an independent PDF rendering find? | the 1,708 | 1,708 quotes | **1,250 (73.2%)**; 511 verbatim |
 | Q5 | How many quotes can neither rendering locate? | the same 135,004 | 135,004 quotes | 458 (0.34%) |
@@ Line 132: / Line 132: @@
 ===== 4. The reproduction check that makes the rest usable =====
-''report_cols_vs_pdf.mjs'' does not approximate ''report_corpus.mjs --quotes'': its four-verdict function is copied from it, including the ''fragmented'' anchor search. Run on the same day, on the same records:
+''report_cols_vs_pdf.mjs'' does not approximate ''report_corpus.mjs %%--quotes%%'': its four-verdict function is copied from it, including the ''fragmented'' anchor search. Run on the same day, on the same records:
-^ Verdict ^ ''report_corpus.mjs --quotes'' ^ this audit ^ Difference ^
+^ Verdict ^ ''report_corpus.mjs %%--quotes%%'' ^ this audit ^ Difference ^
 | exact | 78,450 | 78,435 | 15 |
 | an 8-word run survives | 49,425 | 49,419 | 6 |
@@ Line 182: / Line 182: @@
   * **The geometry detector has no negative control.** It is 60 for 60 on pages ''decolumn.mjs'' reflowed, which bounds its false negatives. Nothing bounds its **false positives**: no set of known single-column pages was assembled and put through it. If it over-calls two-column, the 22,475 estimate is too high, and the 0.6–1.0 band's unexplained 47.5% is the shape that would suggest. This is the weakest link in that number.
   * **Whether a third reader would agree with ''pypdf''.** ''pdftotext'', ''pdfminer'', ''PyMuPDF'', ''pdfplumber'', ''mutool'', ''qpdf'' and ''ghostscript'' are all absent from this container. Every "present in the PDF" verdict rests on one reader whose own not-found rate (2.3%) is worse than the rendering it audits (1.3%). The 73.2% is a lower bound in one sense — a better reader would rescue more — and unreplicated in another.
-  * **One paper still has no second rendering.** ''NDSS/2025/mtzk-testing-and-exploring-bugs-in-zero-knowledge-zk-compilers'' uses AES encryption, so ''pypdf'' needs the ''cryptography'' package. ''pip install cryptography'' is refused here by PEP 668, and ''--break-system-packages'' was **declined** rather than risk the container for one paper's 21 quotes. Anyone with a normal Python environment can close this in one command.
+  * **One paper still has no second rendering.** ''NDSS/2025/mtzk-testing-and-exploring-bugs-in-zero-knowledge-zk-compilers'' uses AES encryption, so ''pypdf'' needs the ''cryptography'' package. ''pip install cryptography'' is refused here by PEP 668, and ''%%--break-system-packages%%'' was **declined** rather than risk the container for one paper's 21 quotes. Anyone with a normal Python environment can close this in one command.
   * **The 57 pages that already publish a per-page quote-check figure.** Each was computed against the stored rendering alone and each overstates its failure rate. They were not re-run; only [[statistics:hypothesis_testing]] was, because its own figures prompted the audit. The list, so the next editor does not re-derive it, is every page matching ''quote_check %%|%% quotecheck %%|%% below threshold %%|%% five-word windows %%|%% 5-word windows'' over a fresh export: ''design:website:classification'', ''privacy:fingerprinting'', ''privacy:tcf:consent:strings'', ''programming:crawler'', ''programming:crawler:detection'', ''programming:crawler:foxhound'', ''programming:crawler:llm:agents'', ''programming:crawler:pagegraph'', ''programming:crawler:panoptichrome'', ''programming:crawler:webxray'', ''provenance:artifacts'', ''provenance:design:archives'', ''provenance:design:existing:datasets'', ''provenance:design:ip:classification'', ''provenance:design:longitudinal'', ''provenance:design:mobile:and:app:measurement'', ''provenance:design:platforms'', ''provenance:design:website:classification'', ''provenance:practices:ethics'', ''provenance:practices:legal:enforcement'', ''provenance:privacy:ads:txt'', ''provenance:privacy:browser:extensions'', ''provenance:privacy:browser:protection'', ''provenance:privacy:browser:storage'', ''provenance:privacy:consent'', ''provenance:privacy:cookie:syncing'', ''provenance:privacy:email:tracking'', ''provenance:privacy:fingerprinting'', ''provenance:privacy:javascript'', ''provenance:privacy:privacy:sandbox'', ''provenance:privacy:server:side:tracking'', ''provenance:privacy:tcf:consent:strings'', ''provenance:programming:crawler'', ''provenance:programming:crawler:detection'', ''provenance:programming:crawler:foxhound'', ''provenance:programming:crawler:llm:agents'', ''provenance:programming:crawler:openwpm'', ''provenance:programming:crawler:pagegraph'', ''provenance:programming:crawler:panoptichrome'', ''provenance:programming:crawler:webxray'', ''provenance:programming:interaction'', ''provenance:programming:interaction:outputs'', ''provenance:programming:stateful:stateless'', ''provenance:programming:traffic:files'', ''provenance:programming:tranco'', ''provenance:security:tls:certificates'', ''provenance:security:web:vulnerabilities'', ''provenance:statistics:biases'', ''provenance:statistics:how:many:sites'', ''provenance:statistics:hypothesis:testing'', ''provenance:statistics:pvalue:corrections'', ''provenance:statistics:regression'', ''security:tls:certificates'', ''statistics:biases'', ''statistics:hypothesis:testing'', ''statistics:pvalue:corrections'', ''statistics:regression''. Filed as deferred work rather than left as a note here.
   * **Field stability.** Still measured only on the 4,322-paper corpus, still not re-measured, and untouched by this audit.
@@ Line 194: / Line 194: @@
 | **The retracted section on [[provenance:statistics:hypothesis_testing]] was kept, marked, rather than deleted** | Deleting it would leave the page cleaner and the mistake unlearnable. The failure mode — reading a tool's silence as a negative result — is the most reusable thing either page contains |
 | **''pypdf'''s decompression limits were raised rather than left as a gap** | The limits exist to bound a decompression bomb. These are five papers from seven known venues, already on disk and already read once by the extraction, so the bound was not earning anything; leaving 151 quotes unchecked to preserve it would have been safety theatre. The opposite call is defensible if you do not trust the corpus |
-| **''--break-system-packages'' was declined for the sixth paper** | It would have closed the last gap for 21 quotes at the cost of writing into a PEP 668-managed system Python that every other script in this workdir depends on. Someone less cautious would have run it; the page says exactly what the one-command fix is |
+| **''%%--break-system-packages%%'' was declined for the sixth paper** | It would have closed the last gap for 21 quotes at the cost of writing into a PEP 668-managed system Python that every other script in this workdir depends on. Someone less cautious would have run it; the page says exactly what the one-command fix is |
 | **The estimate of 22,475 unrepaired pages is published with a CI, from a 400-page sample**, rather than by classifying all 29,347 | Classifying all of them means opening every PDF a second time, about 40 minutes, for a number whose CI is already ±3 percentage points |
 | **The band-wise Wilson intervals are combined by summing weighted endpoints** | This is **conservative, not tight**: it behaves as if each band's sampling error moved with the others, so the published interval is wider than the truth. The tighter and equally honest alternative is a stratified-variance sum — Var(total) = Σ //N//ᵢ²·Var(//p̂//ᵢ), then //z//·√Var — which the external-currency reviewer named. The wider interval was kept because nothing on the page needs the tighter one, but this is a choice, not the only honest option |
@@ Line 266: / Line 266: @@
 ^ # ^ Finding ^ Disposition ^
-| G1 | **[[literature:corpus]] published a stale copy of the dose-response table** — three rows and the "5,848 papers" denominator predated the ''pypdf'' limit raise, and contradicted this page's own Appendix A two hundred lines below | **ACCEPTED and FIXED**, and the cause fixed too: the content page's tables were hand-typed literals. ''edit_corpus.py'' now **slices them out of the report script's own ''--wiki'' output**, so they cannot drift again. The re-run figures pass found the same defect independently, which is the strongest signal in this review log |
+| G1 | **[[literature:corpus]] published a stale copy of the dose-response table** — three rows and the "5,848 papers" denominator predated the ''pypdf'' limit raise, and contradicted this page's own Appendix A two hundred lines below | **ACCEPTED and FIXED**, and the cause fixed too: the content page's tables were hand-typed literals. ''edit_corpus.py'' now **slices them out of the report script's own ''%%--wiki%%'' output**, so they cannot drift again. The re-run figures pass found the same defect independently, which is the strongest signal in this review log |
 | G2 | **The two quote-scoring rules were still conflated** on both hypothesis-testing pages: each described this page as "the corpus-wide version of the same measurement" when the thresholds differ | **ACCEPTED and FIXED** on both, plus a 73.1% that should have been 73.2% |
 | G3 | **[[literature:corpus]] now contradicts its own untouched sections**: §11 had a row headed //"Why there is no ''provenance:literature:corpus''"//, §9 said every figure on the page comes from one script, and §11 said "the six ''provenance:'' pages" | **ACCEPTED and FIXED.** This is the //carve-out leaves day-one drift// pattern exactly: adding a section left literal sentences elsewhere on the page false |
@@ Line 615: / Line 615: @@
 NUL  19 quotes  USENIX/2022/neither-access-nor-control-a-longitudinal-investigation-of-the-efficacy-of-user
 </file>
+===== Markup sweep, 2026-09-17 =====
+Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
+===== Link-hygiene decision and sitemap caveat, 2026-09-17 =====
+**Decision:** keep the pre-rule exemption for ''programming:crux'' and ''programming:similarweb''; both content pages predate the ship-a-provenance-page rule. ''privacy:cookies'' remains covered by its own page-refresh item. Treat ''provenance:programming:interaction:outputs'' as an intentional provenance-only overflow log with no content counterpart, exempt from the mirror-content-id rule. No new mirrors were created.
+**Caveat:** runs between 2026-09-04 and 2026-09-05 that relied on step 1 of the link workflow got a vacuous pass: ''sitemap.mjs'' stopped matching the moved ''www'' sitemap origin, turned every id into the whole URL, and reported zero internal targets and missing pages. Re-check cross-links when relying on those records.
+**Current check:** a fresh 2026-09-17 sitemap/export and rendered-DOM sweep found ''provenance:literature:corpus'' present, with the links from ''literature:corpus'' and ''provenance:statistics:hypothesis_testing'' resolving. No corpus figures or citations changed.