Differences

This shows you the differences between two versions of the page.

--- statistics:pvalue_corrections [2026/08/13 06:56] – Review fixes: replace two false 'appears in zero' claims about forking paths (the string matches twice, both false positives); fix a footnote that said 42 exceeds 45; correct the per-year range to 0.0%-39.1% on 8-128; restore a plural in the Sunlight quot karel.kubicek.claude
+++ statistics:pvalue_corrections [2026/08/21 14:50] (current) – Boxes: <wrap> renders a span, use uppercase <WRAP>; one box per list, not per bullet karel.kubicek.claude
@@ Line 6: / Line 6: @@
 <WRAP important>
-**Roughly a quarter of the papers that run a hypothesis test report a correction, and the papers with the most hypotheses are the least likely to.**
+**Roughly a quarter of the papers that run a hypothesis test report a correction, and whether they do tracks whether they recruited people — not how many hypotheses they had.**
 Of the 5,859 papers extracted from seven security, privacy and measurement venues (2010–2026), **1,025 ran a hypothesis test**. Of those, **247 (24.1%) report a multiple-comparison correction.** Split by the shape of the study:
   * **38.6%** of the 433 that recruited participants and ran no crawl
+  * **34.7%** of the 49 that did both
   * **12.9%** of the 140 that ran a crawl and recruited nobody
   * **11.2%** of the 403 that did neither
-By venue, of each venue's hypothesis-test papers: **PoPETs 45.6%**, USENIX Security 28.5%, IEEE S&P 26.6%, CCS 20.9%, TheWebConf 14.4%, **IMC 14.1%**, **NDSS 10.8%**. Correction is a usable-privacy practice that measurement work has not picked up — and the crawl papers are the ones running 100+ tests: of the 189 papers that ran both a crawl and a hypothesis test, the median largest stated population is **38,090 units**, and **45.0% state a population of 100,000 or more**.
+The two participant rows are the two high ones. **Correcting is a habit that arrived with human-subjects methodology, and a crawl paper that runs a hypothesis test corrects at a third the rate of a user study that does** — in the setting where the number of candidate hypotheses is set by the data rather than by a protocol. By venue: **PoPETs 45.6%**, USENIX Security 28.5%, IEEE S&P 26.6%, CCS 20.9%, TheWebConf 14.4%, **IMC 14.1%**, **NDSS 10.8%**.
-**Bonferroni is still the modal choice in this field, including in 2025–2026, and it is the one procedure with no remaining reason to use.** Of the 269 papers with a correction, **141 (52.4%) use Bonferroni**, 66 (24.5%) Holm, 43 (16.0%) Benjamini–Hochberg, 4 (1.5%) Benjamini–Yekutieli. Holm {[holm1979_sequentially]} needs exactly the same assumptions as Bonferroni and can never reject fewer hypotheses — Datta et al. said so in this corpus in 2015 {[datta2015_automated]} — so //"we applied a Bonferroni correction"// should read //"we applied a Holm correction"// in every paper that writes it.
+**Bonferroni is still the modal choice in this field, including in 2025–2026, and for testing there is no longer a reason to prefer it.** Of the 269 papers with a correction tuple, **141 (52.4%) use Bonferroni**, 66 (24.5%) Holm, 43 (16.0%) Benjamini–Hochberg, 4 (1.5%) Benjamini–Yekutieli. Holm {[holm1979_sequentially]} needs exactly the same assumptions as Bonferroni and can never reject fewer hypotheses — Datta et al. said so in this corpus in 2015 {[datta2015_automated]} — so //"we applied a Bonferroni correction"// should read //"we applied a Holm correction"// almost everywhere it is written. It survives for two things Holm does not do: **simultaneous confidence intervals**, and a **threshold you can pre-declare** before any //p//-value exists. See [[#Which Procedure, and Which Are Current]].
-And **almost nobody says how many hypotheses were in the family: 14 of 287 correction tuples (4.9%)**. Without //k// the adjusted //p// is not reconstructible and the correction is not checkable.
+And **hardly anyone says how many hypotheses were in the family: about one corrected paper in twenty.** Two independent measurements agree. 14 of 287 correction tuples (4.9%) put a count in the extracted method, detail or evidence sentence. Searching the whole paper instead — any count of comparisons, tests or hypotheses within ±1,500 characters of a procedure mention, then reading every hit — gives **15 of 265 papers (5.7%)**. Without //k// the adjusted //p// is not reconstructible and the correction is not checkable.
 </WRAP>
@@ Line 41: / Line 42: @@
 | **Per-treatment survival comparison over time** | arms × time points | Maass et al. {[maass2021_effective]}: //"a single Holm-Bonferroni correction [24] for all 45 significance tests"// |
 | **Per-violation-type compliance test** | one test per legal requirement per subgroup, and again per subgroup comparison | Bouhoula et al. {[bouhoula2024_automated]} apply Holm–Bonferroni twice: to pairwise violation comparisons across popularity ranks, and to Fisher's exact tests comparing each CMP's subset against all websites |
-| **Per-regression-coefficient interaction term** | one test per coefficient, and the family is the model, not the paper | Nenadic et al. {[nenadic2026_overcoming]}: //"Because we estimate separate models for multiple disclosure obligations, conducting parallel hypothesis tests increases the risk of false positives"// — BH //"applied separately to the two families"// |
+| **Per-regression-coefficient interaction term** | one test per coefficient, and the family is the model, not the paper | Nenadic et al. {[nenadic2026_swiss]}: //"Because we estimate separate models for multiple disclosure obligations, conducting parallel hypothesis tests increases the risk of false positives"// — BH //"applied separately to the two families"// |
-| **Per-site test** | one test per site. This is where //k// stops being tractable and the answer is a model, not a correction | no paper in this corpus does it and then corrects |
+| **Per-site test** | one test per site. This is where //k// stops being tractable and the answer is a model, not a correction | none of the 269 papers with a correction tuple does this. Whether papers run per-site tests //without// correcting is not something this corpus was queried for |
 Two consequences a reviewer will pick up on.
@@ Line 49: / Line 50: @@
 **The last row of the table is the honest one.** Once the natural unit of the test is the site, //k// is the size of the crawl, the correction is annihilating, and the procedure is not the answer. See [[#When Correcting Is the Wrong Lever]].
+===== Deciding What Counts as One Family =====
+This is the decision the procedures cannot make for you, it is the one a textbook does not cover, and it is where a crawl-shaped study goes wrong. Three consent arms × 40 countries is 120 cells. Is that one family of 120 tests, three families of 40, 40 families of 3, or — if you report every pairwise country contrast inside each arm — one family of 3 × 780 = 2,340? The answer changes every number in your results table, and **any of those answers is defensible if you fixed it before you looked.** None of them is defensible if you chose after.
+The rule the corpus's better papers follow, stated as a rule:
+  - **One family per confirmatory claim you will make in the abstract.** If the paper's claim is "consent rejection increases tracking", the family is the tests that support that claim — not every test in the paper. Nenadic et al. {[nenadic2026_swiss]} do this at the model level: because they //"estimate separate models for multiple disclosure obligations"//, they apply Benjamini–Hochberg //"separately to the two families of interaction terms (CH vs. EU and CH & EU vs. EU)"// — two contrast definitions of one research question, corrected apart rather than pooled.
+  - **Then, per outcome variable.** Despres et al. {[despres2024_best]} fix α from //"a maximum of 25 hypotheses tested per outcome"//; Zimmeck et al. {[zimmeck2024_website]} correct across 231 comparisons drawn from one comparison set. Both state the grouping, which is what makes them checkable.
+  - **A model is one family, not one test per coefficient across the paper.** If you fit six regressions, that is six families, each the size of its own coefficient set — not one family of all the coefficients you happen to report.
+  - **Everything else is exploratory.** Say so, report unadjusted //p//-values, and drop the word "significant". Correcting a 400-test screen and a 3-test confirmatory claim in one family is not conservative, it is meaningless: it destroys the confirmatory claim to protect a screen you were going to follow up anyway.
+  - **Write the grouping down before the data arrive**, in the paper's methodology and, better, in a registration {[hopewell2025_consort]}. This is the whole point of [[Statistics:Study preregistration]] and it costs one paragraph.
+<WRAP important>
+**One correction across the whole paper is the wrong answer, and it is the one people reach for.** It is not the conservative choice. It inflates //k// with tests nobody was going to interpret, which is exactly the cost the worked example below measures: the same twenty real effects go from eight reportable at //k// = 200 to one at //k// = 2,000, purely because the family got wider. **Splitting into stated families is not //p//-hacking as long as the split is fixed in advance and reported** — and reporting the split is what distinguishes it.
+</WRAP>
 ===== Which Procedure, and Which Are Current =====
@@ Line 55: / Line 72: @@
 ^ Procedure ^ Dated ^ Controls ^ Assumes about dependence ^ Papers ^ Verdict ^
-| **Bonferroni** | 1936 / Dunn 1961 | FWER | nothing — valid under arbitrary dependence | **141 (52.4%)** | **Historical. Superseded by Holm on statistical grounds since 1979, still modal here.** Never wrong, just never optimal |
+| **Bonferroni** | 1936 / Dunn 1961 | FWER | nothing — valid under arbitrary dependence | **141 (52.4%)** | **Historical for testing: dominated by Holm since 1979, still modal here.** Keep it for the two jobs Holm does not do — **simultaneous confidence intervals** at level 1 − α/k, and a **threshold you can state in advance** (α/k needs only //k//, where Holm needs every //p//-value in the family) |
 | **Holm** (step-down) | 1979 {[holm1979_sequentially]} | FWER | nothing | 66 (24.5%) | **Current. The default for a small confirmatory family.** Strictly dominates Bonferroni |
 | **Šidák** | 1967 | FWER | independence | 3 (1.1%) | Marginal. Buys a fraction of a percent over Bonferroni and needs an assumption a crawl cannot support |
@@ Line 70: / Line 87: @@
 **What this field does:** Bonferroni remains the modal procedure in the most recent years available. Of the 45 hypothesis-test papers that corrected in 2025–2026, **19 used Bonferroni, 14 Holm, 9 Benjamini–Hochberg and none Benjamini–Yekutieli**.((Papers, not tuples, and restricted to papers that ran a hypothesis test so that every column shares one denominator. A paper naming two procedures is counted in two columns and a paper using an unlisted procedure in none, so the four figures neither sum to 45 nor are disjoint — here they sum to 42. From ''report_pvalue_corrections.mjs''.)) The FDR share is rising — of the papers whose full text names Bonferroni, Holm or Benjamini–Hochberg, BH's share goes from **15.7% in 2015–2019 (8 of 51) to 28.9% in 2025–2026 (13 of 45)** — but it is still a minority, and 2025–2026 are the provisional years of this corpus.
-**What you should do:** use **Holm** for a small confirmatory family and **Benjamini–Hochberg** for a large exploratory one; use **Benjamini–Yekutieli** instead of BH when you cannot argue the tests are positively dependent, which in a site-level crawl you usually cannot. Do not use plain Bonferroni: Holm makes no additional assumption and can never reject fewer hypotheses, so the choice costs nothing.
+**What you should do**, in this order:
+  - **Decide what the family is, and write it down, before you look at the data.** This matters more than which procedure you pick, and it is the one step no procedure can repair. See [[#Deciding What Counts as One Family]].
+  - **Split confirmatory from exploratory**, correct inside the confirmatory family only, and label the rest as exploratory with unadjusted //p//-values and no significance claims.
+  - **Then pick a procedure:** **Holm** for a small confirmatory family, **Benjamini–Hochberg** for a large exploratory one, **Benjamini–Yekutieli** instead of BH when you cannot argue the tests are positively dependent — which in a site-level crawl you usually cannot. For a test, prefer Holm over plain Bonferroni: same assumptions, never fewer rejections.
+  - **Check the choice against your own numbers.** BY pays a factor of about ln(//k//)+0.58, so it beats Holm only when //k// is large //and// discoveries are plentiful. In the worked example below, at //k// = 200 with sparse signal BY leaves **1** survivor and Holm leaves **2** — the recommendation inverts. Run all four and look before committing.
+  - **Report an effect size with an interval for every corrected comparison.** Past about 10,000 units per group the //p//-value stops being the binding constraint, and no correction fixes that. See [[#When Correcting Is the Wrong Lever]].
 **Nothing here has been overturned recently.** Multiplicity correction is one of the few methodological areas in web measurement where no 2020s development — LLM classification included — changed the answer. The procedures date from 1979, 1995 and 2001, and they are still the right ones. Where the field has moved is in //which// of them it reaches for, and slowly. Research-frontier work does exist — e-values and e-BH since 2022, selective inference — and **none of it has any presence in this corpus or any adopted standing**, so this page does not recommend it.
-What //has// moved is **reporting**, in an adjacent field. **CONSORT 2025** {[hopewell2025_consort]}, the current reporting guideline for randomised trials, added multiplicity to what a paper must say: //"Any methods used to mitigate or account for multiplicity should be described. If no methods have been used to account for multiplicity (eg, not applicable, or not considered), then this should also be reported, particularly when a large number of analyses has been carried out."// That is exactly the standard this page argues for, it is dated 2025, and **none of the seven venues here asks for anything like it.**
+What //has// moved is **reporting**, in an adjacent field, and by a smaller step than it first looks. Clinical trials have had multiplicity in their reporting guideline since CONSORT 2010, whose elaboration already discouraged multiple primary outcomes //"because of the problems of interpretation associated with multiplicity of analyses"// and whose limitations item already named //"multiplicity of analyses"//. What the **CONSORT 2025** elaboration {[hopewell2025_consort]} adds is the //explicit negative// obligation: //"Any methods used to mitigate or account for multiplicity should be described. If no methods have been used to account for multiplicity (eg, not applicable, or not considered), then this should also be reported, particularly when a large number of analyses has been carried out."//((The sentence is in the CONSORT 2025 **explanation and elaboration** document (BMJ 389:e081124, ''10.1136/bmj-2024-081124''), under the elaboration for item 21. It is **not** in the statement/checklist paper (''10.1136/bmj-2024-081123''), which contains no occurrence of "multiplicity" in its checklist text. Both fetched 2026-08-13.)) That is the standard the **Declining to correct** section of this page argues for, and **none of the seven venues here asks for anything like it** — nor does any of them ask for //k//, which CONSORT does not ask for either.
 </WRAP>
@@ Line 83: / Line 106: @@
   * **A compliance or enforcement claim** — "this CMP violates Article 7(3)" — is FWER. One false accusation is a problem in itself, and the audience includes people who will read exactly one row. Correct so that the probability of //any// false claim is bounded: Holm.
   * **A screen** — "which of these 400 third parties set an identifier before consent, so we can go look at them" — is FDR. You expect some of the shortlist to be wrong, you will follow up, and being unable to find anything is the real failure. Correct so that the //expected fraction// of wrong rows is bounded: Benjamini–Hochberg, or Benjamini–Yekutieli if you cannot argue positive dependence.
-  * **A paper usually has both**, and the two families should be adjusted separately with the split stated. Nenadic et al. {[nenadic2026_overcoming]} do exactly this: BH //"applied separately to the two families of interaction terms"//. That is the pattern to copy — one correction across the whole paper is not more conservative, it is less meaningful.
+  * **A paper usually has both**, and the two families should be adjusted separately with the split stated. Nenadic et al. {[nenadic2026_swiss]} do this for two contrast definitions of one question: BH //"applied separately to the two families of interaction terms"//. That is the pattern to copy — one correction across the whole paper is not more conservative, it is less meaningful.
 SciPy's own documentation states the trade-off in one line: FDR procedures //"tend to offer higher power than familywise error rate control procedures (e.g. Bonferroni correction)"//, and the ''by'' method //"is guaranteed to control the FDR even when the p-values are not from independent tests"//.((''scipy.stats.false_discovery_control'', SciPy 1.18.0 documentation, fetched 2026-08-13.))
@@ Line 332: / Line 355: @@
 <WRAP important>
-**At crawl scale, the binding constraint is almost never the //p//-value.** Compare two groups of 100,000 sites each, base rate 10%, and a difference of **1.0 percentage point** gives //p// = 3 × 10⁻¹³ on a two-proportion //z//-test. Bonferroni at //k// = 1,000 leaves it at 3 × 10⁻¹⁰. It survives every procedure on this page, and 11.0% against 10.0% is not a finding.((Two-proportion //z//-test with a pooled standard error, //n// = 100,000 per group: //z// = 7.29. The same 1.0-point difference at //n// = 10,000 per group gives //p// = 0.021 and does not survive //k// = 1,000. Computed with ''scipy.stats.norm'' 1.17.1 on 2026-08-13; the two lines are in ''out/pvalue-external-facts.txt''.)) **45.0% of the papers in this corpus that ran both a crawl and a hypothesis test state a population of 100,000 or more.**
+**At crawl scale, the binding constraint is almost never the //p//-value.** Compare two groups of 100,000 sites each, base rate 10%, and a difference of **1.0 percentage point** gives //p// = 3 × 10⁻¹³ on a two-proportion //z//-test. Bonferroni at //k// = 1,000 leaves it at 3 × 10⁻¹⁰. It survives every procedure on this page. Whether 11.0% against 10.0% //matters// is a question the //p//-value can no longer answer — only the effect size and its interval can, and at the top of a ranking list one percentage point is thousands of sites, which may be the whole story or may be nothing.((Two-proportion //z//-test with a pooled standard error, //n// = 100,000 per group: //z// = 7.29. The same 1.0-point difference at //n// = 10,000 per group gives //p// = 0.021 and does not survive //k// = 1,000. Computed with ''scipy.stats.norm'' 1.17.1 on 2026-08-13; the two lines are in ''out/pvalue-external-facts.txt''.)) **45.0% of the papers in this corpus that ran both a crawl and a hypothesis test state a population of 100,000 or more.**
 Correcting such a //p//-value is arithmetic performed on a quantity that stopped carrying information. What a reader needs is the **effect size and its interval**: the difference in percentage points, the risk ratio, the //r// or Cramér's //V//, with a confidence interval. **Only 159 of the 1,025 hypothesis-test papers (15.5%) report an effect size, and 178 report a correction without one.** That ratio is the wrong way round: a correction with no effect size protects a reader from a false positive and tells them nothing about whether the true positive matters.
@@ Line 351: / Line 374: @@
 Each of those was a choice made with the data in front of you, and each is a test you effectively ran and did not report. Gelman & Loken {[gelman2014_statistical]} call this the garden of forking paths, and their point is precisely that it does not require any dishonesty — a single analysis path, chosen after seeing the data, has an uncontrolled error rate even though no multiple comparison was ever computed.
-**No paper in this corpus engages with the idea.** The literal string //forking paths// matches two of the 5,869 full texts, and reading both shows they are packet-forwarding paths and symbolic-execution paths — see [[#What is missing entirely]]. The ASA statement on //p//-values {[wasserstein2016_asa]} matches **zero**. The only mechanism that addresses it is committing to the analysis before you look — which is [[Statistics:Study preregistration]], and which **6 of these 1,025 hypothesis-test papers (0.6%) did**.
+**No paper in this corpus engages with the idea.** The literal string //forking paths// matches two of the 5,869 full texts, and reading both shows they are packet-forwarding paths and symbolic-execution paths — see [[#What is missing entirely]]. The ASA statement on //p//-values {[wasserstein2016_asa]} matches **zero**. The standard mechanism that addresses it is committing to the analysis before you look — which is [[Statistics:Study preregistration]], and which **14 of these 1,025 hypothesis-test papers (1.4%) did**.((14, not the 6 in the ''statistics.kind'' table below. The schema field catches fewer than half the preregistrations in this corpus; the sibling page hand-classified all 62 full-text matches for ''pre-regist*'' and found 15 real study preregistrations, of which 14 also ran a hypothesis test. Use the hand count, not the field.)) It is not the only one: the multiverse analysis in [[#Open Questions]] addresses the same problem after the fact, and a held-out split of the crawl addresses part of it.
-<wrap todo>
+<WRAP todo>
-The cheap version, if you will not preregister: **name your confirmatory tests in the paper**, correct within that family only, and label everything else exploratory with uncorrected //p//-values and no significance claims. Nenadic et al. {[nenadic2026_overcoming]} split their families; nobody in this corpus splits confirmatory from exploratory in a //crawl//.
+The cheap version, if you will not preregister: **name your confirmatory tests in the paper**, correct within that family only, and label everything else exploratory with uncorrected //p//-values and no significance claims. Nenadic et al. {[nenadic2026_swiss]} split their families; nobody in this corpus splits confirmatory from exploratory in a //crawl//.
-</wrap>
+</WRAP>
 ==== Declining to correct is a legitimate choice, if you say so ====
@@ Line 424: / Line 447: @@
 The last two rows are the reason the fold exists rather than a ''method'' histogram. Böttger et al. {[boettger2025_regional]} write //"due to a violation of sphericity, we applied the Greenhouse-Geisser correction"// — a correction, but of the //F//-test's degrees of freedom, not of multiplicity. Any query that counts ''kind == "multiple-comparison-correction"'' and stops there reports both of these as corrections.
-==== It is rising, and Bonferroni is not losing ====
+==== It rose through the 2010s, then fell back — and Bonferroni never lost ====
 ^ Period ^ Ran a hypothesis test ^ Corrected ^ Share ^ Bonferroni ^ Holm ^ B–H ^ B–Y ^
@@ Line 433: / Line 456: @@
 <WRAP important>
-**Read the asterisk before the trend.** 2025 and 2026 are the provisional years of this corpus: CCS 2026 and IMC 2026 have not been held, and IEEE S&P 2026 and TheWebConf 2026 abstracts are absent from OpenAlex, so those venue-years are under-represented **by construction**. The uncollapsed per-year series moves between **0.0% (2010 and 2011) and 39.1% (2021)** on denominators of **8 to 128** papers, so nothing narrower than the four buckets above is worth quoting. **Treat "about a quarter, up from about a tenth in the early 2010s" as the claim.**
+**The rise is solid; the last bucket's fall is real too, and provisionality does not explain it away.** 9/97 in 2010–2014 against 140/487 in 2020–2024 is Fisher's exact //p// = 2.1 × 10⁻⁵ — the practice genuinely tripled. Then 140/487 against 45/214 in 2025–2026 is //p// = 0.0328 in the other direction, putting the current rate back at its 2015–2019 level (53/227 against 45/214 gives //p// = 0.5689, i.e. indistinguishable).
+The obvious excuse does not work. 2025–2026 are the provisional years — CCS 2026 and IMC 2026 have not been held, and IEEE S&P 2026 and TheWebConf 2026 abstracts are absent from OpenAlex — so one might expect the venue mix to explain the dip. It does not: applying each venue's own correction rate to the 2025–2026 venue mix predicts **25.6%**, and to the 2020–2024 mix **25.4%**. The composition barely moved.((Expected rate = Σ(venue rate × venue's hypothesis-test papers in the period) / period total. NDSS's hypothesis-test count rose from 26 to 40 between the two buckets, which offsets the extra PoPETs weight. Computed by ''report_pvalue_corrections.mjs''.)) So either the dip is sampling noise on 214 papers, or the practice has stopped spreading. **Treat "about a quarter, and no longer climbing" as the claim**, and do not quote the per-year series: it moves between 0.0% (2010, 2011) and 39.1% (2021) on denominators of 8 to 128.
 What is stable across the buckets is that **Bonferroni has never stopped being modal.** Every column above is restricted to hypothesis-test papers, so the four procedure counts and the "corrected" count share one denominator; a paper naming two procedures is counted in two columns, which is why the procedure columns can sum above the corrected count.
@@ Line 442: / Line 467: @@
 ==== Almost nobody states the size of the family ====
-An adjusted //p// cannot be checked, reproduced or compared without //k//. Of the 287 tuples that record a real correction:
+An adjusted //p// cannot be checked, reproduced or compared without //k//. Two independent passes, agreeing to about a point.
-^ What the paper states ^ Tuples ^ Share of 287 ^
+**Pass 1, over the 287 tuples that record a real correction** — what landed in ''statistics[].method'', ''.detail'' or the evidence sentence:
+^ What the tuple states ^ Tuples ^ Share of 287 ^
 | a number of comparisons, tests or hypotheses in the family | **14** | **4.9%** |
 | an adjusted α or significance threshold | 57 | 19.9% |
 | any reported detail at all (''statistics[].detail'' non-null) | 161 | 56.1% |
+**Pass 2, over the whole text of the 265 papers that corrected.** A tuple only carries one sentence, so this pass searches the full paper for any count of comparisons, tests or hypotheses within ±1,500 characters of a procedure mention. Doing that naively is useless in this literature: the unguarded regex returned 41 hits, of which the majority were **χ² notation** — the digit in "χ² tests" sits immediately before the word — plus exponents and USENIX page furniture. With three guards the count is 20, and **reading all 20 leaves 15 genuine: 5.7% of the 265**. The five rejected, and every accepted match with its surrounding sentence, are printed in the report output on [[provenance:statistics:pvalue_corrections]].
+**So: 4.9% by tuple, 5.7% by paper. Roughly one corrected paper in twenty states the size of its family.**
 The fourteen that do state it are the ones a reader can check, and they read like this:
@@ Line 475: / Line 506: @@
 | power-analysis | 72 | 7.0% |
 | bayesian | 12 | 1.2% |
-| preregistration | 6 | 0.6% |
+| preregistration //(schema field; the hand count is 14)// | 6 | 0.6% |
 **69 papers report both an effect size and a correction. 178 report a correction and no effect size.**
@@ Line 501: / Line 532: @@
   * **Two buckets in the fold are not corrections** and are excluded from every "corrected" count: the three papers that declare they did not correct, and the two Greenhouse–Geisser sphericity corrections.
   * **Sentinels are never answers.** ''statistics'' has no ''not-stated'' value — a paper that corrected and did not say so is simply absent, which is why every correction figure here is a **lower bound on the practice and an upper bound on the reporting**.
+  * **The family-size figure is measured twice and hand-checked.** The tuple pass sees one sentence per correction; the full-text pass sees the whole paper but needs guards against χ² notation, exponents and page furniture, and then a reading of all 20 surviving hits. 4.9% and 5.7% are the two answers, five rejected hits are named with reasons, and all 20 are printed with their sentences on the provenance page. **Neither pass is a bound**: a count in a distant table caption is missed by both.
   * **A paper counts once**, never once per mention. 292 tuples across 269 papers.
   * **''statistics.kind'' is a mid-band field**: two independent extraction runs over identical text agreed on it for 68% of papers, so a repeat run would move these percentages by a few points. That caveat applies to every share on this page and does not apply to the folded procedure ranking, which is a ranking.
@@ Line 519: / Line 551: @@
   - **The correction inside your power analysis**, if you are sizing a study rather than analysing one {[ho2025_efficacy]}.
   - **If you decline to correct, say so and say why** — and then drop the word "significant" {[pu2016_model]} {[goetzen2022_ctrl]} {[naji2025_responsibility]}. CONSORT 2025 {[hopewell2025_consort]} makes this an explicit requirement in clinical trials: reporting that //no// method was used is itself a reporting obligation.
-  - **Do not write "Bonferroni".** Write "Holm". It is the same assumption, the same code path, and never fewer findings.
+  - **For a test, write "Holm", not "Bonferroni".** Same assumption, same code path, never fewer findings. Reach for Bonferroni deliberately, not by default: for **simultaneous confidence intervals**, or for a **threshold pre-declared** in a protocol or registration where you do not yet have the //p//-values Holm needs.
 ===== Open Questions =====
-<wrap todo>
+<WRAP todo>
   * **No measurement paper in this corpus corrects a per-site family.** The design that most obviously creates thousands of hypotheses — one test per site — is never followed by a correction, so it is unknown whether the field considers that setting out of scope for testing, or simply does not test it. Both readings have consequences for how a reviewer should treat a per-site claim.
   * **65 papers fit a multilevel model and none frames it as a multiplicity strategy** — at least, none surfaced while reading correction passages, and no one has checked all 65. Whether partial pooling is already doing the work of a correction in this literature without anyone saying so is answerable and unanswered.
   * **Benjamini–Yekutieli is used by four papers**, and it is the procedure whose assumptions actually match site-level crawl data. Whether the BH results in this literature would survive BY is checkable on any paper that released its //p//-values — see [[:Artifacts]] — and nobody has done it.
   * **The debate is absent.** Zero papers cite the ASA statement, Rothman, Perneger, or the garden of forking paths. A short SoK on inference practice in web measurement would be citing an empty shelf, which is unusual and useful.
-  * **No venue asks for //k//.** Reporting the family size is a one-line checklist item that would make every correction in the literature checkable, and no call for papers, artifact-evaluation form or review form in these seven venues requests it.
+  * **Nobody has asked these venues to require //k//.** Reporting the family size is a one-line checklist item that would make every correction in the literature checkable, and **94.3% of corrected papers do not do it**. The nearest thing in any field is CONSORT's requirement that trials //describe the method// used — which is a weaker ask than stating //k//, and nobody has proposed even that much to a security or measurement venue. Their review forms are not public, so the current state can only be read off the papers.
   * **The pipeline-level multiplicity is unmeasured.** Nobody has taken a published crawl, enumerated the defensible alternatives at each pipeline decision (seed list, rank cut, exclusion rule, classifier threshold), re-run all of them, and reported the spread of the headline figure. That is a multiverse analysis, it is straightforwardly fundable, and it would say more about this literature's error rate than any correction.
-</wrap>
+</WRAP>
 ===== Related Pages =====
@@ Line 540: / Line 572: @@
   * [[Design:Website selection]] and [[Design:Sampling|Representative sampling methods]] — the strata that become the family.
   * [[Design:Crawling location]] — per-vantage comparisons, and how many pairs they generate.
-  * [[Design:User studies]] — where correcting is already normal, and where 38.6% of the hypothesis-test papers that do it live.
+  * [[Design:User studies]] — where **two-thirds of the papers that correct live** (167 of 247), and where the rate, 38.6%, is three times the crawl rate. Still a minority practice even there.
   * [[:Artifacts]] — releasing the //p//-values is what makes a correction re-checkable.
   * [[provenance:statistics:pvalue_corrections|Provenance of this page's figures]].