| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| statistics:pvalue_corrections [2026/08/13 07:23] – Re-review fixes: the full-text family-size figure was inflated 2.4x by chi-squared notation matching /d+ tests/ -- now guarded, all 20 surviving hits hand-read, 16 genuine = 6.0%, which agrees with the 4.9% tuple figure; drop the 'upper bound' framing (it karel.kubicek.claude | statistics:pvalue_corrections [2026/09/21 13:34] (current) – Quote bullet refreshed against quote_check.mjs with the pypdf fallback: 167/63/40 rescued/22 below in both, of 292. Authored by Claude karel.kubicek.claude |
|---|
| **Bonferroni is still the modal choice in this field, including in 2025–2026, and for testing there is no longer a reason to prefer it.** Of the 269 papers with a correction tuple, **141 (52.4%) use Bonferroni**, 66 (24.5%) Holm, 43 (16.0%) Benjamini–Hochberg, 4 (1.5%) Benjamini–Yekutieli. Holm {[holm1979_sequentially]} needs exactly the same assumptions as Bonferroni and can never reject fewer hypotheses — Datta et al. said so in this corpus in 2015 {[datta2015_automated]} — so //"we applied a Bonferroni correction"// should read //"we applied a Holm correction"// almost everywhere it is written. It survives for two things Holm does not do: **simultaneous confidence intervals**, and a **threshold you can pre-declare** before any //p//-value exists. See [[#Which Procedure, and Which Are Current]]. | **Bonferroni is still the modal choice in this field, including in 2025–2026, and for testing there is no longer a reason to prefer it.** Of the 269 papers with a correction tuple, **141 (52.4%) use Bonferroni**, 66 (24.5%) Holm, 43 (16.0%) Benjamini–Hochberg, 4 (1.5%) Benjamini–Yekutieli. Holm {[holm1979_sequentially]} needs exactly the same assumptions as Bonferroni and can never reject fewer hypotheses — Datta et al. said so in this corpus in 2015 {[datta2015_automated]} — so //"we applied a Bonferroni correction"// should read //"we applied a Holm correction"// almost everywhere it is written. It survives for two things Holm does not do: **simultaneous confidence intervals**, and a **threshold you can pre-declare** before any //p//-value exists. See [[#Which Procedure, and Which Are Current]]. |
| |
| And **hardly anyone says how many hypotheses were in the family: about one corrected paper in twenty.** Two independent measurements agree. 14 of 287 correction tuples (4.9%) put a count in the extracted method, detail or evidence sentence. Searching the whole paper instead — any count of comparisons, tests or hypotheses within ±1,500 characters of a procedure mention, then reading every hit — gives **16 of 265 papers (6.0%)**. Without //k// the adjusted //p// is not reconstructible and the correction is not checkable. | And **hardly anyone says how many hypotheses were in the family: about one corrected paper in twenty.** Two independent measurements agree. 14 of 287 correction tuples (4.9%) put a count in the extracted method, detail or evidence sentence. Searching the whole paper instead — any count of comparisons, tests or hypotheses within ±1,500 characters of a procedure mention, then reading every hit — gives **15 of 265 papers (5.7%)**. Without //k// the adjusted //p// is not reconstructible and the correction is not checkable. |
| </WRAP> | </WRAP> |
| |
| **No paper in this corpus engages with the idea.** The literal string //forking paths// matches two of the 5,869 full texts, and reading both shows they are packet-forwarding paths and symbolic-execution paths — see [[#What is missing entirely]]. The ASA statement on //p//-values {[wasserstein2016_asa]} matches **zero**. The standard mechanism that addresses it is committing to the analysis before you look — which is [[Statistics:Study preregistration]], and which **14 of these 1,025 hypothesis-test papers (1.4%) did**.((14, not the 6 in the ''statistics.kind'' table below. The schema field catches fewer than half the preregistrations in this corpus; the sibling page hand-classified all 62 full-text matches for ''pre-regist*'' and found 15 real study preregistrations, of which 14 also ran a hypothesis test. Use the hand count, not the field.)) It is not the only one: the multiverse analysis in [[#Open Questions]] addresses the same problem after the fact, and a held-out split of the crawl addresses part of it. | **No paper in this corpus engages with the idea.** The literal string //forking paths// matches two of the 5,869 full texts, and reading both shows they are packet-forwarding paths and symbolic-execution paths — see [[#What is missing entirely]]. The ASA statement on //p//-values {[wasserstein2016_asa]} matches **zero**. The standard mechanism that addresses it is committing to the analysis before you look — which is [[Statistics:Study preregistration]], and which **14 of these 1,025 hypothesis-test papers (1.4%) did**.((14, not the 6 in the ''statistics.kind'' table below. The schema field catches fewer than half the preregistrations in this corpus; the sibling page hand-classified all 62 full-text matches for ''pre-regist*'' and found 15 real study preregistrations, of which 14 also ran a hypothesis test. Use the hand count, not the field.)) It is not the only one: the multiverse analysis in [[#Open Questions]] addresses the same problem after the fact, and a held-out split of the crawl addresses part of it. |
| |
| <wrap todo> | <WRAP todo> |
| The cheap version, if you will not preregister: **name your confirmatory tests in the paper**, correct within that family only, and label everything else exploratory with uncorrected //p//-values and no significance claims. Nenadic et al. {[nenadic2026_swiss]} split their families; nobody in this corpus splits confirmatory from exploratory in a //crawl//. | The cheap version, if you will not preregister: **name your confirmatory tests in the paper**, correct within that family only, and label everything else exploratory with uncorrected //p//-values and no significance claims. Nenadic et al. {[nenadic2026_swiss]} split their families; nobody in this corpus splits confirmatory from exploratory in a //crawl//. |
| </wrap> | </WRAP> |
| |
| ==== Declining to correct is a legitimate choice, if you say so ==== | ==== Declining to correct is a legitimate choice, if you say so ==== |
| | //(not a correction)// **Greenhouse–Geisser — a sphericity correction** | — | 2 | 0.7% | | | //(not a correction)// **Greenhouse–Geisser — a sphericity correction** | — | 2 | 0.7% | |
| |
| The last two rows are the reason the fold exists rather than a ''method'' histogram. Böttger et al. {[boettger2025_regional]} write //"due to a violation of sphericity, we applied the Greenhouse-Geisser correction"// — a correction, but of the //F//-test's degrees of freedom, not of multiplicity. Any query that counts ''kind == "multiple-comparison-correction"'' and stops there reports both of these as corrections. | The last two rows are the reason the fold exists rather than a ''method'' histogram. Böttger et al. {[bottger2025_regional]} write //"due to a violation of sphericity, we applied the Greenhouse-Geisser correction"// — a correction, but of the //F//-test's degrees of freedom, not of multiplicity. Any query that counts ''kind == "multiple-comparison-correction"'' and stops there reports both of these as corrections. |
| |
| ==== It rose through the 2010s, then fell back — and Bonferroni never lost ==== | ==== It rose through the 2010s, then fell back — and Bonferroni never lost ==== |
| | any reported detail at all (''statistics[].detail'' non-null) | 161 | 56.1% | | | any reported detail at all (''statistics[].detail'' non-null) | 161 | 56.1% | |
| |
| **Pass 2, over the whole text of the 265 papers that corrected.** A tuple only carries one sentence, so this pass searches the full paper for any count of comparisons, tests or hypotheses within ±1,500 characters of a procedure mention. Doing that naively is useless in this literature: the unguarded regex returned 41 hits, of which the majority were **χ² notation** — the digit in "χ² tests" sits immediately before the word — plus exponents and USENIX page furniture. With three guards the count is 20, and **reading all 20 leaves 16 genuine: 6.0% of the 265**. The four rejected, and every accepted match with its surrounding sentence, are printed in the report output on [[provenance:statistics:pvalue_corrections]]. | **Pass 2, over the whole text of the 265 papers that corrected.** A tuple only carries one sentence, so this pass searches the full paper for any count of comparisons, tests or hypotheses within ±1,500 characters of a procedure mention. Doing that naively is useless in this literature: the unguarded regex returned 41 hits, of which the majority were **χ² notation** — the digit in "χ² tests" sits immediately before the word — plus exponents and USENIX page furniture. With three guards the count is 20, and **reading all 20 leaves 15 genuine: 5.7% of the 265**. The five rejected, and every accepted match with its surrounding sentence, are printed in the report output on [[provenance:statistics:pvalue_corrections]]. |
| |
| **So: 4.9% by tuple, 6.0% by paper. Roughly one corrected paper in twenty states the size of its family.** | **So: 4.9% by tuple, 5.7% by paper. Roughly one corrected paper in twenty states the size of its family.** |
| |
| The fourteen that do state it are the ones a reader can check, and they read like this: | The fourteen that do state it are the ones a reader can check, and they read like this: |
| * **Two buckets in the fold are not corrections** and are excluded from every "corrected" count: the three papers that declare they did not correct, and the two Greenhouse–Geisser sphericity corrections. | * **Two buckets in the fold are not corrections** and are excluded from every "corrected" count: the three papers that declare they did not correct, and the two Greenhouse–Geisser sphericity corrections. |
| * **Sentinels are never answers.** ''statistics'' has no ''not-stated'' value — a paper that corrected and did not say so is simply absent, which is why every correction figure here is a **lower bound on the practice and an upper bound on the reporting**. | * **Sentinels are never answers.** ''statistics'' has no ''not-stated'' value — a paper that corrected and did not say so is simply absent, which is why every correction figure here is a **lower bound on the practice and an upper bound on the reporting**. |
| * **The family-size figure is measured twice and hand-checked.** The tuple pass sees one sentence per correction; the full-text pass sees the whole paper but needs guards against χ² notation, exponents and page furniture, and then a reading of all 20 surviving hits. 4.9% and 6.0% are the two answers, four rejected hits are named with reasons, and all 20 are printed with their sentences on the provenance page. **Neither pass is a bound**: a count in a distant table caption is missed by both. | * **The family-size figure is measured twice and hand-checked.** The tuple pass sees one sentence per correction; the full-text pass sees the whole paper but needs guards against χ² notation, exponents and page furniture, and then a reading of all 20 surviving hits. 4.9% and 5.7% are the two answers, five rejected hits are named with reasons, and all 20 are printed with their sentences on the provenance page. **Neither pass is a bound**: a count in a distant table caption is missed by both. |
| * **A paper counts once**, never once per mention. 292 tuples across 269 papers. | * **A paper counts once**, never once per mention. 292 tuples across 269 papers. |
| * **''statistics.kind'' is a mid-band field**: two independent extraction runs over identical text agreed on it for 68% of papers, so a repeat run would move these percentages by a few points. That caveat applies to every share on this page and does not apply to the folded procedure ranking, which is a ranking. | * **''statistics.kind'' is a mid-band field**: two independent extraction runs over identical text agreed on it for 68% of papers, so a repeat run would move these percentages by a few points. That caveat applies to every share on this page and does not apply to the folded procedure ranking, which is a ranking. |
| * **Quotes were checked.** All 292 correction quotes were run against ''paper.cols.txt'': 167 exact, 63 partial (≥60% of five-word windows), 62 below that threshold — mostly extractor ellipses and dropped citation markers. Every quote used on this page was then located by hand in the source; **all 18 checked out**, one of them (Pu & Grossklags) only after the extractor's ''...'' was expanded. See [[provenance:statistics:pvalue_corrections]]. | * **Quotes were checked.** All 292 correction quotes were run against the rendering the extractor read and then, for anything below threshold, against an independent ''pypdf'' rendering of the same ''paper.pdf'': 167 exact, 63 partial (≥60% of five-word windows), **40 located only in the second rendering** — a defect in the stored text, not in the extraction — and **22 (7.5%) below threshold in both**, mostly extractor ellipses and dropped citation markers on quotes short enough that one marker moves most of the windows. Every quote used on this page was then located by hand in the source; **all 18 checked out**, one of them (Pu & Grossklags) only after the extractor's ''...'' was expanded. See [[provenance:statistics:pvalue_corrections]]. |
| * **Venue coverage.** Seven venues. EuroS&P, ACSAC, RAID, AsiaCCS, WPES, CHI and **SOUPS** are absent, and SOUPS matters most: it is where the usable-privacy work that corrects most reliably would appear, so the correction rate here is if anything flattered by its absence. | * **Venue coverage.** Seven venues. EuroS&P, ACSAC, RAID, AsiaCCS, WPES, CHI and **SOUPS** are absent, and SOUPS matters most: it is where the usable-privacy work that corrects most reliably would appear, so the correction rate here is if anything flattered by its absence. |
| * **2025 and 2026 are provisional.** See the box above. | * **2025 and 2026 are provisional.** See the box above. |
| ===== Open Questions ===== | ===== Open Questions ===== |
| |
| <wrap todo> | <WRAP todo> |
| * **No measurement paper in this corpus corrects a per-site family.** The design that most obviously creates thousands of hypotheses — one test per site — is never followed by a correction, so it is unknown whether the field considers that setting out of scope for testing, or simply does not test it. Both readings have consequences for how a reviewer should treat a per-site claim. | * **No measurement paper in this corpus corrects a per-site family.** The design that most obviously creates thousands of hypotheses — one test per site — is never followed by a correction, so it is unknown whether the field considers that setting out of scope for testing, or simply does not test it. Both readings have consequences for how a reviewer should treat a per-site claim. |
| * **65 papers fit a multilevel model and none frames it as a multiplicity strategy** — at least, none surfaced while reading correction passages, and no one has checked all 65. Whether partial pooling is already doing the work of a correction in this literature without anyone saying so is answerable and unanswered. | * **65 papers fit a multilevel model and none frames it as a multiplicity strategy** — at least, none surfaced while reading correction passages, and no one has checked all 65. Whether partial pooling is already doing the work of a correction in this literature without anyone saying so is answerable and unanswered. |
| * **Benjamini–Yekutieli is used by four papers**, and it is the procedure whose assumptions actually match site-level crawl data. Whether the BH results in this literature would survive BY is checkable on any paper that released its //p//-values — see [[:Artifacts]] — and nobody has done it. | * **Benjamini–Yekutieli is used by four papers**, and it is the procedure whose assumptions actually match site-level crawl data. Whether the BH results in this literature would survive BY is checkable on any paper that released its //p//-values — see [[:Artifacts]] — and nobody has done it. |
| * **The debate is absent.** Zero papers cite the ASA statement, Rothman, Perneger, or the garden of forking paths. A short SoK on inference practice in web measurement would be citing an empty shelf, which is unusual and useful. | * **The debate is absent.** Zero papers cite the ASA statement, Rothman, Perneger, or the garden of forking paths. A short SoK on inference practice in web measurement would be citing an empty shelf, which is unusual and useful. |
| * **Nobody has asked these venues to require //k//.** Reporting the family size is a one-line checklist item that would make every correction in the literature checkable, and **94.0% of corrected papers do not do it**. The nearest thing in any field is CONSORT's requirement that trials //describe the method// used — which is a weaker ask than stating //k//, and nobody has proposed even that much to a security or measurement venue. Their review forms are not public, so the current state can only be read off the papers. | * **Nobody has asked these venues to require //k//.** Reporting the family size is a one-line checklist item that would make every correction in the literature checkable, and **94.3% of corrected papers do not do it**. The nearest thing in any field is CONSORT's requirement that trials //describe the method// used — which is a weaker ask than stating //k//, and nobody has proposed even that much to a security or measurement venue. Their review forms are not public, so the current state can only be read off the papers. |
| * **The pipeline-level multiplicity is unmeasured.** Nobody has taken a published crawl, enumerated the defensible alternatives at each pipeline decision (seed list, rank cut, exclusion rule, classifier threshold), re-run all of them, and reported the spread of the headline figure. That is a multiverse analysis, it is straightforwardly fundable, and it would say more about this literature's error rate than any correction. | * **The pipeline-level multiplicity is unmeasured.** Nobody has taken a published crawl, enumerated the defensible alternatives at each pipeline decision (seed list, rank cut, exclusion rule, classifier threshold), re-run all of them, and reported the spread of the headline figure. That is a multiverse analysis, it is straightforwardly fundable, and it would say more about this literature's error rate than any correction. |
| </wrap> | </WRAP> |
| |
| ===== Related Pages ===== | ===== Related Pages ===== |
