| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| statistics:pvalue_corrections [2026/08/21 08:33] – [Open Questions] karelkubicek | statistics:pvalue_corrections [2026/09/21 13:34] (current) – Quote bullet refreshed against quote_check.mjs with the pypdf fallback: 167/63/40 rescued/22 below in both, of 292. Authored by Claude karel.kubicek.claude |
|---|
| **No paper in this corpus engages with the idea.** The literal string //forking paths// matches two of the 5,869 full texts, and reading both shows they are packet-forwarding paths and symbolic-execution paths — see [[#What is missing entirely]]. The ASA statement on //p//-values {[wasserstein2016_asa]} matches **zero**. The standard mechanism that addresses it is committing to the analysis before you look — which is [[Statistics:Study preregistration]], and which **14 of these 1,025 hypothesis-test papers (1.4%) did**.((14, not the 6 in the ''statistics.kind'' table below. The schema field catches fewer than half the preregistrations in this corpus; the sibling page hand-classified all 62 full-text matches for ''pre-regist*'' and found 15 real study preregistrations, of which 14 also ran a hypothesis test. Use the hand count, not the field.)) It is not the only one: the multiverse analysis in [[#Open Questions]] addresses the same problem after the fact, and a held-out split of the crawl addresses part of it. | **No paper in this corpus engages with the idea.** The literal string //forking paths// matches two of the 5,869 full texts, and reading both shows they are packet-forwarding paths and symbolic-execution paths — see [[#What is missing entirely]]. The ASA statement on //p//-values {[wasserstein2016_asa]} matches **zero**. The standard mechanism that addresses it is committing to the analysis before you look — which is [[Statistics:Study preregistration]], and which **14 of these 1,025 hypothesis-test papers (1.4%) did**.((14, not the 6 in the ''statistics.kind'' table below. The schema field catches fewer than half the preregistrations in this corpus; the sibling page hand-classified all 62 full-text matches for ''pre-regist*'' and found 15 real study preregistrations, of which 14 also ran a hypothesis test. Use the hand count, not the field.)) It is not the only one: the multiverse analysis in [[#Open Questions]] addresses the same problem after the fact, and a held-out split of the crawl addresses part of it. |
| |
| <wrap todo> | <WRAP todo> |
| The cheap version, if you will not preregister: **name your confirmatory tests in the paper**, correct within that family only, and label everything else exploratory with uncorrected //p//-values and no significance claims. Nenadic et al. {[nenadic2026_swiss]} split their families; nobody in this corpus splits confirmatory from exploratory in a //crawl//. | The cheap version, if you will not preregister: **name your confirmatory tests in the paper**, correct within that family only, and label everything else exploratory with uncorrected //p//-values and no significance claims. Nenadic et al. {[nenadic2026_swiss]} split their families; nobody in this corpus splits confirmatory from exploratory in a //crawl//. |
| </wrap> | </WRAP> |
| |
| ==== Declining to correct is a legitimate choice, if you say so ==== | ==== Declining to correct is a legitimate choice, if you say so ==== |
| | //(not a correction)// **Greenhouse–Geisser — a sphericity correction** | — | 2 | 0.7% | | | //(not a correction)// **Greenhouse–Geisser — a sphericity correction** | — | 2 | 0.7% | |
| |
| The last two rows are the reason the fold exists rather than a ''method'' histogram. Böttger et al. {[boettger2025_regional]} write //"due to a violation of sphericity, we applied the Greenhouse-Geisser correction"// — a correction, but of the //F//-test's degrees of freedom, not of multiplicity. Any query that counts ''kind == "multiple-comparison-correction"'' and stops there reports both of these as corrections. | The last two rows are the reason the fold exists rather than a ''method'' histogram. Böttger et al. {[bottger2025_regional]} write //"due to a violation of sphericity, we applied the Greenhouse-Geisser correction"// — a correction, but of the //F//-test's degrees of freedom, not of multiplicity. Any query that counts ''kind == "multiple-comparison-correction"'' and stops there reports both of these as corrections. |
| |
| ==== It rose through the 2010s, then fell back — and Bonferroni never lost ==== | ==== It rose through the 2010s, then fell back — and Bonferroni never lost ==== |
| * **A paper counts once**, never once per mention. 292 tuples across 269 papers. | * **A paper counts once**, never once per mention. 292 tuples across 269 papers. |
| * **''statistics.kind'' is a mid-band field**: two independent extraction runs over identical text agreed on it for 68% of papers, so a repeat run would move these percentages by a few points. That caveat applies to every share on this page and does not apply to the folded procedure ranking, which is a ranking. | * **''statistics.kind'' is a mid-band field**: two independent extraction runs over identical text agreed on it for 68% of papers, so a repeat run would move these percentages by a few points. That caveat applies to every share on this page and does not apply to the folded procedure ranking, which is a ranking. |
| * **Quotes were checked.** All 292 correction quotes were run against ''paper.cols.txt'': 167 exact, 63 partial (≥60% of five-word windows), 62 below that threshold — mostly extractor ellipses and dropped citation markers. Every quote used on this page was then located by hand in the source; **all 18 checked out**, one of them (Pu & Grossklags) only after the extractor's ''...'' was expanded. See [[provenance:statistics:pvalue_corrections]]. | * **Quotes were checked.** All 292 correction quotes were run against the rendering the extractor read and then, for anything below threshold, against an independent ''pypdf'' rendering of the same ''paper.pdf'': 167 exact, 63 partial (≥60% of five-word windows), **40 located only in the second rendering** — a defect in the stored text, not in the extraction — and **22 (7.5%) below threshold in both**, mostly extractor ellipses and dropped citation markers on quotes short enough that one marker moves most of the windows. Every quote used on this page was then located by hand in the source; **all 18 checked out**, one of them (Pu & Grossklags) only after the extractor's ''...'' was expanded. See [[provenance:statistics:pvalue_corrections]]. |
| * **Venue coverage.** Seven venues. EuroS&P, ACSAC, RAID, AsiaCCS, WPES, CHI and **SOUPS** are absent, and SOUPS matters most: it is where the usable-privacy work that corrects most reliably would appear, so the correction rate here is if anything flattered by its absence. | * **Venue coverage.** Seven venues. EuroS&P, ACSAC, RAID, AsiaCCS, WPES, CHI and **SOUPS** are absent, and SOUPS matters most: it is where the usable-privacy work that corrects most reliably would appear, so the correction rate here is if anything flattered by its absence. |
| * **2025 and 2026 are provisional.** See the box above. | * **2025 and 2026 are provisional.** See the box above. |
