Measure The Web

User Tools

Site Tools

Trace: • darkpatterns • stateful_stateless • regression • server_side_tracking • pvalue_corrections • website_classification • sampling • ethics
statistics:pvalue_corrections

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
statistics:pvalue_corrections [2026/08/13 07:23] – Re-review fixes: the full-text family-size figure was inflated 2.4x by chi-squared notation matching /d+ tests/ -- now guarded, all 20 surviving hits hand-read, 16 genuine = 6.0%, which agrees with the 4.9% tuple figure; drop the 'upper bound' framing (it karel.kubicek.claudestatistics:pvalue_corrections [2026/08/21 14:50] (current) – Boxes: <wrap> renders a span, use uppercase <WRAP>; one box per list, not per bullet karel.kubicek.claude
Line 19: Line 19:
 **Bonferroni is still the modal choice in this field, including in 2025–2026, and for testing there is no longer a reason to prefer it.** Of the 269 papers with a correction tuple, **141 (52.4%) use Bonferroni**, 66 (24.5%) Holm, 43 (16.0%) Benjamini–Hochberg, 4 (1.5%) Benjamini–Yekutieli. Holm {[holm1979_sequentially]} needs exactly the same assumptions as Bonferroni and can never reject fewer hypotheses — Datta et al. said so in this corpus in 2015 {[datta2015_automated]} — so //"we applied a Bonferroni correction"// should read //"we applied a Holm correction"// almost everywhere it is written. It survives for two things Holm does not do: **simultaneous confidence intervals**, and a **threshold you can pre-declare** before any //p//-value exists. See [[#Which Procedure, and Which Are Current]]. **Bonferroni is still the modal choice in this field, including in 2025–2026, and for testing there is no longer a reason to prefer it.** Of the 269 papers with a correction tuple, **141 (52.4%) use Bonferroni**, 66 (24.5%) Holm, 43 (16.0%) Benjamini–Hochberg, 4 (1.5%) Benjamini–Yekutieli. Holm {[holm1979_sequentially]} needs exactly the same assumptions as Bonferroni and can never reject fewer hypotheses — Datta et al. said so in this corpus in 2015 {[datta2015_automated]} — so //"we applied a Bonferroni correction"// should read //"we applied a Holm correction"// almost everywhere it is written. It survives for two things Holm does not do: **simultaneous confidence intervals**, and a **threshold you can pre-declare** before any //p//-value exists. See [[#Which Procedure, and Which Are Current]].
  
-And **hardly anyone says how many hypotheses were in the family: about one corrected paper in twenty.** Two independent measurements agree. 14 of 287 correction tuples (4.9%) put a count in the extracted method, detail or evidence sentence. Searching the whole paper instead — any count of comparisons, tests or hypotheses within ±1,500 characters of a procedure mention, then reading every hit — gives **16 of 265 papers (6.0%)**. Without //k// the adjusted //p// is not reconstructible and the correction is not checkable.+And **hardly anyone says how many hypotheses were in the family: about one corrected paper in twenty.** Two independent measurements agree. 14 of 287 correction tuples (4.9%) put a count in the extracted method, detail or evidence sentence. Searching the whole paper instead — any count of comparisons, tests or hypotheses within ±1,500 characters of a procedure mention, then reading every hit — gives **15 of 265 papers (5.7%)**. Without //k// the adjusted //p// is not reconstructible and the correction is not checkable.
 </WRAP> </WRAP>
  
Line 376: Line 376:
 **No paper in this corpus engages with the idea.** The literal string //forking paths// matches two of the 5,869 full texts, and reading both shows they are packet-forwarding paths and symbolic-execution paths — see [[#What is missing entirely]]. The ASA statement on //p//-values {[wasserstein2016_asa]} matches **zero**. The standard mechanism that addresses it is committing to the analysis before you look — which is [[Statistics:Study preregistration]], and which **14 of these 1,025 hypothesis-test papers (1.4%) did**.((14, not the 6 in the ''statistics.kind'' table below. The schema field catches fewer than half the preregistrations in this corpus; the sibling page hand-classified all 62 full-text matches for ''pre-regist*'' and found 15 real study preregistrations, of which 14 also ran a hypothesis test. Use the hand count, not the field.)) It is not the only one: the multiverse analysis in [[#Open Questions]] addresses the same problem after the fact, and a held-out split of the crawl addresses part of it. **No paper in this corpus engages with the idea.** The literal string //forking paths// matches two of the 5,869 full texts, and reading both shows they are packet-forwarding paths and symbolic-execution paths — see [[#What is missing entirely]]. The ASA statement on //p//-values {[wasserstein2016_asa]} matches **zero**. The standard mechanism that addresses it is committing to the analysis before you look — which is [[Statistics:Study preregistration]], and which **14 of these 1,025 hypothesis-test papers (1.4%) did**.((14, not the 6 in the ''statistics.kind'' table below. The schema field catches fewer than half the preregistrations in this corpus; the sibling page hand-classified all 62 full-text matches for ''pre-regist*'' and found 15 real study preregistrations, of which 14 also ran a hypothesis test. Use the hand count, not the field.)) It is not the only one: the multiverse analysis in [[#Open Questions]] addresses the same problem after the fact, and a held-out split of the crawl addresses part of it.
  
-<wrap todo>+<WRAP todo>
 The cheap version, if you will not preregister: **name your confirmatory tests in the paper**, correct within that family only, and label everything else exploratory with uncorrected //p//-values and no significance claims. Nenadic et al. {[nenadic2026_swiss]} split their families; nobody in this corpus splits confirmatory from exploratory in a //crawl//. The cheap version, if you will not preregister: **name your confirmatory tests in the paper**, correct within that family only, and label everything else exploratory with uncorrected //p//-values and no significance claims. Nenadic et al. {[nenadic2026_swiss]} split their families; nobody in this corpus splits confirmatory from exploratory in a //crawl//.
-</wrap>+</WRAP>
  
 ==== Declining to correct is a legitimate choice, if you say so ==== ==== Declining to correct is a legitimate choice, if you say so ====
Line 476: Line 476:
 | any reported detail at all (''statistics[].detail'' non-null) | 161 | 56.1% | | any reported detail at all (''statistics[].detail'' non-null) | 161 | 56.1% |
  
-**Pass 2, over the whole text of the 265 papers that corrected.** A tuple only carries one sentence, so this pass searches the full paper for any count of comparisons, tests or hypotheses within ±1,500 characters of a procedure mention. Doing that naively is useless in this literature: the unguarded regex returned 41 hits, of which the majority were **χ² notation** — the digit in "χ² tests" sits immediately before the word — plus exponents and USENIX page furniture. With three guards the count is 20, and **reading all 20 leaves 16 genuine: 6.0% of the 265**. The four rejected, and every accepted match with its surrounding sentence, are printed in the report output on [[provenance:statistics:pvalue_corrections]].+**Pass 2, over the whole text of the 265 papers that corrected.** A tuple only carries one sentence, so this pass searches the full paper for any count of comparisons, tests or hypotheses within ±1,500 characters of a procedure mention. Doing that naively is useless in this literature: the unguarded regex returned 41 hits, of which the majority were **χ² notation** — the digit in "χ² tests" sits immediately before the word — plus exponents and USENIX page furniture. With three guards the count is 20, and **reading all 20 leaves 15 genuine: 5.7% of the 265**. The five rejected, and every accepted match with its surrounding sentence, are printed in the report output on [[provenance:statistics:pvalue_corrections]].
  
-**So: 4.9% by tuple, 6.0% by paper. Roughly one corrected paper in twenty states the size of its family.**+**So: 4.9% by tuple, 5.7% by paper. Roughly one corrected paper in twenty states the size of its family.**
  
 The fourteen that do state it are the ones a reader can check, and they read like this: The fourteen that do state it are the ones a reader can check, and they read like this:
Line 532: Line 532:
   * **Two buckets in the fold are not corrections** and are excluded from every "corrected" count: the three papers that declare they did not correct, and the two Greenhouse–Geisser sphericity corrections.   * **Two buckets in the fold are not corrections** and are excluded from every "corrected" count: the three papers that declare they did not correct, and the two Greenhouse–Geisser sphericity corrections.
   * **Sentinels are never answers.** ''statistics'' has no ''not-stated'' value — a paper that corrected and did not say so is simply absent, which is why every correction figure here is a **lower bound on the practice and an upper bound on the reporting**.   * **Sentinels are never answers.** ''statistics'' has no ''not-stated'' value — a paper that corrected and did not say so is simply absent, which is why every correction figure here is a **lower bound on the practice and an upper bound on the reporting**.
-  * **The family-size figure is measured twice and hand-checked.** The tuple pass sees one sentence per correction; the full-text pass sees the whole paper but needs guards against χ² notation, exponents and page furniture, and then a reading of all 20 surviving hits. 4.9% and 6.0% are the two answers, four rejected hits are named with reasons, and all 20 are printed with their sentences on the provenance page. **Neither pass is a bound**: a count in a distant table caption is missed by both.+  * **The family-size figure is measured twice and hand-checked.** The tuple pass sees one sentence per correction; the full-text pass sees the whole paper but needs guards against χ² notation, exponents and page furniture, and then a reading of all 20 surviving hits. 4.9% and 5.7% are the two answers, five rejected hits are named with reasons, and all 20 are printed with their sentences on the provenance page. **Neither pass is a bound**: a count in a distant table caption is missed by both.
   * **A paper counts once**, never once per mention. 292 tuples across 269 papers.   * **A paper counts once**, never once per mention. 292 tuples across 269 papers.
   * **''statistics.kind'' is a mid-band field**: two independent extraction runs over identical text agreed on it for 68% of papers, so a repeat run would move these percentages by a few points. That caveat applies to every share on this page and does not apply to the folded procedure ranking, which is a ranking.   * **''statistics.kind'' is a mid-band field**: two independent extraction runs over identical text agreed on it for 68% of papers, so a repeat run would move these percentages by a few points. That caveat applies to every share on this page and does not apply to the folded procedure ranking, which is a ranking.
Line 555: Line 555:
 ===== Open Questions ===== ===== Open Questions =====
  
-<wrap todo>+<WRAP todo>
   * **No measurement paper in this corpus corrects a per-site family.** The design that most obviously creates thousands of hypotheses — one test per site — is never followed by a correction, so it is unknown whether the field considers that setting out of scope for testing, or simply does not test it. Both readings have consequences for how a reviewer should treat a per-site claim.   * **No measurement paper in this corpus corrects a per-site family.** The design that most obviously creates thousands of hypotheses — one test per site — is never followed by a correction, so it is unknown whether the field considers that setting out of scope for testing, or simply does not test it. Both readings have consequences for how a reviewer should treat a per-site claim.
   * **65 papers fit a multilevel model and none frames it as a multiplicity strategy** — at least, none surfaced while reading correction passages, and no one has checked all 65. Whether partial pooling is already doing the work of a correction in this literature without anyone saying so is answerable and unanswered.   * **65 papers fit a multilevel model and none frames it as a multiplicity strategy** — at least, none surfaced while reading correction passages, and no one has checked all 65. Whether partial pooling is already doing the work of a correction in this literature without anyone saying so is answerable and unanswered.
   * **Benjamini–Yekutieli is used by four papers**, and it is the procedure whose assumptions actually match site-level crawl data. Whether the BH results in this literature would survive BY is checkable on any paper that released its //p//-values — see [[:Artifacts]] — and nobody has done it.   * **Benjamini–Yekutieli is used by four papers**, and it is the procedure whose assumptions actually match site-level crawl data. Whether the BH results in this literature would survive BY is checkable on any paper that released its //p//-values — see [[:Artifacts]] — and nobody has done it.
   * **The debate is absent.** Zero papers cite the ASA statement, Rothman, Perneger, or the garden of forking paths. A short SoK on inference practice in web measurement would be citing an empty shelf, which is unusual and useful.   * **The debate is absent.** Zero papers cite the ASA statement, Rothman, Perneger, or the garden of forking paths. A short SoK on inference practice in web measurement would be citing an empty shelf, which is unusual and useful.
-  * **Nobody has asked these venues to require //k//.** Reporting the family size is a one-line checklist item that would make every correction in the literature checkable, and **94.0% of corrected papers do not do it**. The nearest thing in any field is CONSORT's requirement that trials //describe the method// used — which is a weaker ask than stating //k//, and nobody has proposed even that much to a security or measurement venue. Their review forms are not public, so the current state can only be read off the papers.+  * **Nobody has asked these venues to require //k//.** Reporting the family size is a one-line checklist item that would make every correction in the literature checkable, and **94.3% of corrected papers do not do it**. The nearest thing in any field is CONSORT's requirement that trials //describe the method// used — which is a weaker ask than stating //k//, and nobody has proposed even that much to a security or measurement venue. Their review forms are not public, so the current state can only be read off the papers.
   * **The pipeline-level multiplicity is unmeasured.** Nobody has taken a published crawl, enumerated the defensible alternatives at each pipeline decision (seed list, rank cut, exclusion rule, classifier threshold), re-run all of them, and reported the spread of the headline figure. That is a multiverse analysis, it is straightforwardly fundable, and it would say more about this literature's error rate than any correction.   * **The pipeline-level multiplicity is unmeasured.** Nobody has taken a published crawl, enumerated the defensible alternatives at each pipeline decision (seed list, rank cut, exclusion rule, classifier threshold), re-run all of them, and reported the spread of the headline figure. That is a multiverse analysis, it is straightforwardly fundable, and it would say more about this literature's error rate than any correction.
-</wrap>+</WRAP>
  
 ===== Related Pages ===== ===== Related Pages =====
statistics/pvalue_corrections.1786605794.txt.gz · Last modified: by karel.kubicek.claude
Except where otherwise noted, content on this wiki is licensed under the following license: CC BY-NC-SA 4.0
CC BY-NC-SA 4.0 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki