A web measurement generates hypotheses the way a crawl generates rows: one test per country, per tracker, per site category, per CMP, per browser profile, per consent arm — and then all the pairs of those. The number of tests is a property of the data you collected, not a decision you made, and it is usually in the hundreds before you have noticed. That is what makes multiplicity a measurement problem rather than a statistics-course problem, and it is why this page exists: the arithmetic of Bonferroni is in every textbook, but nothing in a textbook tells you what the family is when your unit of analysis is a site and your strata are whatever the ranking list happened to contain.

This page assumes you know what a p-value and a type-I error are. It is about three things a textbook will not tell you: where the multiplicity comes from in a crawl, which procedure is current and which is merely traditional, and what to report so a reviewer can check you.

• Sunlight [3Lécuyer, Mathias; Spahn, Riley; Spiliopolous, Yannis; Chaintreau, Augustin; Geambasu, Roxana; Hsu, Daniel J. (2015): "Sunlight: Fine-grained Targeting Detection at Scale with Statistical Confidence", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)] (CCS 2015) — the only paper found in this corpus that treats multiple-testing correction as a first-class stage of a web-measurement pipeline rather than a line in the results section — it offers both a FWER and an FDR procedure and measures what the choice costs at scale.¹⁾ Read §4.4 and §6.4–6.5. Everything in The Power Cost Is Not Abstract, and Somebody Measured It comes from it.
• Automated Experiments on Ad Privacy Settings [2Datta, Amit; Tschantz, Michael Carl; Datta, Anupam (2015): "Automated Experiments on Ad Privacy Settings", in: Proceedings on Privacy Enhancing Technologies. (DOI)] (PoPETs 2015) — the AdFisher paper. Its statistical apparatus is the model for an automated web experiment, and it is where the “uniformly more powerful” argument for Holm over Bonferroni appears inside this literature rather than in a statistics journal.
• Holm 1979 [1Holm, Sture (1979): "A Simple Sequentially Rejective Multiple Test Procedure", Scandinavian Journal of Statistics 6:65-70. No DOI: the identifier 10.2307/4615733 reported by OpenAlex is not a registered handle (Link)] — four pages, and the only reason you need to read it is to be able to say in a rebuttal that the procedure makes no extra assumption.
• Benjamini & Hochberg 1995 [4Benjamini, Yoav; Hochberg, Yosef (1995): "Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing", Journal of the Royal Statistical Society: Series B (Statistical Methodology) 57(1):289-300. (DOI)] and Benjamini & Yekutieli 2001 [5Benjamini, Yoav; Yekutieli, Daniel (2001): "The Control of the False Discovery Rate in Multiple Testing under Dependency", The Annals of Statistics 29(4):1165-1188. (DOI)] — FDR, and FDR when the tests are dependent. The second is the one a crawl usually needs and almost nobody in this corpus uses.
• Gelman, Hill & Yajima 2012 [6Gelman, Andrew; Hill, Jennifer; Yajima, Masanao (2012): "Why We (Usually) Don't Have to Worry About Multiple Comparisons", Journal of Research on Educational Effectiveness 5(2):189-211. (DOI)] — the argument for modelling the strata jointly instead of correcting across them. It is the strongest alternative to everything else on this page and it is invisible in these venues.
• Gelman & Loken 2014 [7Gelman, Andrew; Loken, Eric (2014): "The Statistical Crisis in Science", American Scientist 102(6):460. (DOI)] — the garden of forking paths. Correcting the tests you report does nothing about the tests you decided not to run, and in a crawl pipeline that is where most of the multiplicity lives. No paper in this corpus cites the idea: the literal string forking paths matches two of the 5,869 full texts and both are false positives (packet-forwarding paths, symbolic-execution paths).

The reason a measurement paper ends up with a large k without anyone choosing one is that the strata are inherited from the data source. Each row below is a real design from this corpus, with its own arithmetic.

Two consequences a reviewer will pick up on.

Nesting the strata multiplies, it does not add. Three consent arms × four vantage points × eight site categories is 96 cells, and every pairwise contrast you draw from them is another test. The natural thing to write in a paper — “we compared each pair” — is the thing that makes k quadratic.

The last row of the table is the honest one. Once the natural unit of the test is the site, k is the size of the crawl, the correction is annihilating, and the procedure is not the answer. See When Correcting Is the Wrong Lever.

This is the decision the procedures cannot make for you, it is the one a textbook does not cover, and it is where a crawl-shaped study goes wrong. Three consent arms × 40 countries is 120 cells. Is that one family of 120 tests, three families of 40, 40 families of 3, or — if you report every pairwise country contrast inside each arm — one family of 3 × 780 = 2,340? The answer changes every number in your results table, and any of those answers is defensible if you fixed it before you looked. None of them is defensible if you chose after.

The rule the corpus's better papers follow, stated as a rule:

1. One family per confirmatory claim you will make in the abstract. If the paper's claim is “consent rejection increases tracking”, the family is the tests that support that claim — not every test in the paper. Nenadic et al. [14Nenadić, Luka; Rodriguez, David; Calandrino, Joseph A. (2026): "Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law's Impact", Proceedings on Privacy Enhancing Technologies 2026(4):703-723. (DOI)] do this at the model level: because they “estimate separate models for multiple disclosure obligations”, they apply Benjamini–Hochberg “separately to the two families of interaction terms (CH vs. EU and CH & EU vs. EU)” — two contrast definitions of one research question, corrected apart rather than pooled.
2. Then, per outcome variable. Despres et al. [15Despres, Tess; Constantino, Marcelino Ayala; Lizola, Naomi Zacarias; Romero, Gerardo Sánchez; He, Shijing; Zhan, Xiao; Abdi, Noura; Abu-Salma, Ruba; Such, Jose; Bernd, Julia (2024): ""My Best Friend's Husband Sees and Knows Everything": A Cross-Contextual and Cross-Country Approach to Understanding Smart Home Privacy", in: Proceedings on Privacy Enhancing Technologies. (DOI)] fix α from “a maximum of 25 hypotheses tested per outcome”; Zimmeck et al. [16Zimmeck, Sebastian; Goldelman, Daniel; Kaplan, Owen; Brown, Logan; Casler, Justin; Jean-Charles, Judeley; Champeau, Joe; Harkous, Hamza (2024): "Website Data Transparency in the Browser", in: Proceedings on Privacy Enhancing Technologies. (DOI)] correct across 231 comparisons drawn from one comparison set. Both state the grouping, which is what makes them checkable.
3. A model is one family, not one test per coefficient across the paper. If you fit six regressions, that is six families, each the size of its own coefficient set — not one family of all the coefficients you happen to report.
4. Everything else is exploratory. Say so, report unadjusted p-values, and drop the word “significant”. Correcting a 400-test screen and a 3-test confirmatory claim in one family is not conservative, it is meaningless: it destroys the confirmatory claim to protect a screen you were going to follow up anyway.
5. Write the grouping down before the data arrive, in the paper's methodology and, better, in a registration [17Hopewell, Sally; Chan, An-Wen; Collins, Gary S.; Hróbjartsson, Asbjørn; Moher, David; Schulz, Kenneth F.; others, (2025): "CONSORT 2025 explanation and elaboration: updated guideline for reporting randomised trials", BMJ 389:e081124. (DOI)]. This is the whole point of Study preregistration and it costs one paragraph.

Dated, with what it assumes and whether it is current. “Papers” is out of the 269 in this corpus with a multiple-comparison-correction tuple; the fold behind those counts is described in Use in Publications.

The choice is not statistical, it is editorial, and it is decidable in one sentence: does one wrong row in your table discredit the paper, or does it cost you a footnote?

• A compliance or enforcement claim — “this CMP violates Article 7(3)” — is FWER. One false accusation is a problem in itself, and the audience includes people who will read exactly one row. Correct so that the probability of any false claim is bounded: Holm.
• A screen — “which of these 400 third parties set an identifier before consent, so we can go look at them” — is FDR. You expect some of the shortlist to be wrong, you will follow up, and being unable to find anything is the real failure. Correct so that the expected fraction of wrong rows is bounded: Benjamini–Hochberg, or Benjamini–Yekutieli if you cannot argue positive dependence.
• A paper usually has both, and the two families should be adjusted separately with the split stated. Nenadic et al. [14Nenadić, Luka; Rodriguez, David; Calandrino, Joseph A. (2026): "Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law's Impact", Proceedings on Privacy Enhancing Technologies 2026(4):703-723. (DOI)] do this for two contrast definitions of one question: BH “applied separately to the two families of interaction terms”. That is the pattern to copy — one correction across the whole paper is not more conservative, it is less meaningful.

SciPy's own documentation states the trade-off in one line: FDR procedures “tend to offer higher power than familywise error rate control procedures (e.g. Bonferroni correction)”, and the by method “is guaranteed to control the FDR even when the p-values are not from independent tests”.⁴⁾

Sunlight [3Lécuyer, Mathias; Spahn, Riley; Spiliopolous, Yannis; Chaintreau, Augustin; Geambasu, Roxana; Hsu, Daniel J. (2015): "Sunlight: Fine-grained Targeting Detection at Scale with Statistical Confidence", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)] is a targeting-detection system for ad ecosystems whose pipeline has four stages, and the fourth is “Multiple Testing Correction”. It implements both Holm–Bonferroni and Benjamini–Yekutieli and recommends BY as its default. §6.5 reports the same analysis on a small and a large website experiment, and the numbers are the clearest statement of what FWER costs at web-measurement scale that this literature contains:

The paper's own reading:

We conclude that the Holm correction is very hard on experiments with a lot hypotheses. The larger the proverbial haystack, the harder the needles will be to find.

and, on why it ships BY as the default:

Although this guarantee on the expected false discovery rate is weaker than what is provided by the Holm-Bonferroni method, it is widely accepted in applied statistics as an appropriate and preferred correction for exploratory studies.

Two things to take from this table rather than from the procedure names. First, the cost of a FWER correction grows with k, so the same method that is reasonable on a 200-site pilot can wipe out a 100,000-site crawl — and the crawl is not a bigger version of the pilot, it is a different statistical situation. Second, Sunlight also finds an inversion: the Stage-1 algorithm that generates the most hypotheses wins before correction and loses after it, because “the number of hypotheses, and the proportion of high p-value hypotheses play an important role in the correction, both increasing the penalty applied to each p-value”. Generating more candidate hypotheses makes your pipeline look better and your paper weaker.

You can reproduce the shape of this on your own numbers:

adjust_pvalues.py

#!/usr/bin/env python3
"""How many of your findings survive, under each correction, and at what k.
 
Written for a crawl-shaped family of hypotheses: one test per country, per
tracker, per site category, per browser profile. The point is not to pick a
winner but to make the two numbers a reviewer will ask for impossible to omit:
**how large the family is** and **how the answer moves between procedures**.
 
    python3 adjust_pvalues.py pvalues.txt              # one p-value per line
    python3 adjust_pvalues.py pvalues.txt --alpha 0.01
    python3 adjust_pvalues.py --demo                   # worked example, no input
 
Requires scipy >= 1.11 for `false_discovery_control` (Benjamini-Hochberg and
Benjamini-Yekutieli). Bonferroni and Holm are four lines each and are written
out here so the arithmetic is visible; statsmodels' `multipletests` does all
four plus Sidak if you would rather have one call.
 
Crashes on a malformed p-value rather than dropping it: a family whose size is
wrong makes every adjusted p-value wrong.
"""
 
import argparse
import sys
 
import numpy as np
from scipy.stats import false_discovery_control
 
 
def bonferroni(p):
    """Single-step FWER. Valid under arbitrary dependence. Dominated by Holm."""
    return np.minimum(1.0, p * len(p))
 
 
def holm(p):
    """Step-down FWER (Holm 1979). Same assumptions as Bonferroni, never
    rejects fewer hypotheses, so there is no situation in which plain
    Bonferroni is preferable on power grounds."""
    n = len(p)
    order = np.argsort(p)
    adj = np.empty(n)
    running = 0.0
    for rank, i in enumerate(order):
        running = max(running, (n - rank) * p[i])
        adj[i] = min(1.0, running)
    return adj
 
 
def read_pvalues(path):
    """One p-value per line, blank lines and #-comments ignored.
 
    A line that is not a float in [0, 1] raises: silently skipping it would
    shrink the family and inflate every adjusted p-value in it.
    """
    values = []
    with open(path) as fh:
        for lineno, line in enumerate(fh, 1):
            s = line.strip()
            if not s or s.startswith("#"):
                continue
            v = float(s)  # ValueError here is the intended failure
            if not 0.0 <= v <= 1.0:
                raise ValueError(f"{path}:{lineno}: {v} is not a p-value in [0, 1]")
            values.append(v)
    if not values:
        raise ValueError(f"{path}: no p-values found")
    return np.array(values)
 
 
def report(p, alpha):
    k = len(p)
    procedures = [
        ("none (nominal alpha)", p, "FWER not controlled"),
        ("Bonferroni", bonferroni(p), "FWER, any dependence"),
        ("Holm", holm(p), "FWER, any dependence"),
        ("Benjamini-Hochberg", false_discovery_control(p, method="bh"), "FDR, independent or PRDS"),
        ("Benjamini-Yekutieli", false_discovery_control(p, method="by"), "FDR, any dependence"),
    ]
    print(f"family size k = {k}    alpha = {alpha}")
    print(f"raw p-values below alpha: {int((p < alpha).sum())}")
    print()
    print(f"{'procedure':<22}{'controls':<28}{'survive':>8}{'share':>8}   smallest adjusted p")
    print("-" * 88)
    for name, adj, controls in procedures:
        s = int((adj < alpha).sum())
        print(f"{name:<22}{controls:<28}{s:>8}{s / k:>8.1%}   {adj.min():.3g}")
    print()
    print("Report k, the procedure, and the adjusted p-values -- not just the count of")
    print("survivors. A reader cannot reconstruct any of the rows above from the count.")
 
 
def demo():
    """Three cases, in the order the reader needs them.
 
    Deterministic seed, so every number printed here is reproducible.
    """
    print("=" * 88)
    print("Case 1: why Holm rather than Bonferroni. Three tests, alpha = 0.05.")
    print("=" * 88)
    p = np.array([0.001, 0.02, 0.03])
    print(f"p                    = {p}")
    print(f"Bonferroni adjusted  = {bonferroni(p)}  -> 1 survives")
    print(f"Holm adjusted        = {holm(p)}  -> 3 survive")
    print()
    print("Same assumptions, same alpha, three findings instead of one. Holm can never")
    print("reject fewer than Bonferroni. In a real family the two usually agree -- the")
    print("reason to use Holm is that it is never worse and costs one function call.")
 
    print()
    print("=" * 88)
    print("Case 2: a crawl-shaped family. 200 tests, 20 with a real effect.")
    print("=" * 88)
    rng = np.random.default_rng(20260813)
    nulls = rng.uniform(0.0, 1.0, 180)
    signal = rng.beta(1.0, 250.0, 20)
    report(np.concatenate([signal, nulls]), 0.05)
 
    print()
    print("=" * 88)
    print("Case 3: the same 20 real effects, inside a family of 2,000.")
    print("1,980 nulls this time instead of 180 -- the study got wider, not better.")
    print("=" * 88)
    report(np.concatenate([signal, rng.uniform(0.0, 1.0, 1980)]), 0.05)
    print()
    print("Widening the family is not free: every procedure above loses power, while the")
    print("unadjusted count rises and is now almost all noise. Adding strata you did not")
    print("need costs you the findings you did. Decide the family before you look.")
 
 
def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("path", nargs="?", help="file of p-values, one per line")
    ap.add_argument("--alpha", type=float, default=0.05)
    ap.add_argument("--demo", action="store_true")
    args = ap.parse_args()
    if args.demo:
        demo()
        return 0
    if not args.path:
        ap.error("give a file of p-values, or --demo")
    report(read_pvalues(args.path), args.alpha)
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main())

Real output, numpy 2.4.6 / scipy 1.17.1, run 2026-08-13. The holm and bonferroni functions above were checked against statsmodels.stats.multitest.multipletests (0.14.6) on five random 50-value families: maximum absolute difference 0.0. If you would rather have one call than four functions, multipletests takes method in bonferroni, sidak, holm-sidak, holm, simes-hochberg, hommel, fdr_bh, fdr_by, fdr_tsbh and fdr_tsbky as of 0.14.6, which covers every procedure in the table above except Tukey and Games–Howell (statsmodels.stats.multicomp.pairwise_tukeyhsd for the former).⁵⁾

$ python3 adjust_pvalues.py --demo
========================================================================================
Case 1: why Holm rather than Bonferroni. Three tests, alpha = 0.05.
========================================================================================
p                    = [0.001 0.02  0.03 ]
Bonferroni adjusted  = [0.003 0.06  0.09 ]  -> 1 survives
Holm adjusted        = [0.003 0.04  0.04 ]  -> 3 survive

Same assumptions, same alpha, three findings instead of one. Holm can never
reject fewer than Bonferroni. In a real family the two usually agree -- the
reason to use Holm is that it is never worse and costs one function call.

========================================================================================
Case 2: a crawl-shaped family. 200 tests, 20 with a real effect.
========================================================================================
family size k = 200    alpha = 0.05
raw p-values below alpha: 28

procedure             controls                     survive   share   smallest adjusted p
----------------------------------------------------------------------------------------
none (nominal alpha)  FWER not controlled               28   14.0%   2.92e-06
Bonferroni            FWER, any dependence               2    1.0%   0.000584
Holm                  FWER, any dependence               2    1.0%   0.000584
Benjamini-Hochberg    FDR, independent or PRDS           8    4.0%   0.000584
Benjamini-Yekutieli   FDR, any dependence                1    0.5%   0.00343

Report k, the procedure, and the adjusted p-values -- not just the count of
survivors. A reader cannot reconstruct any of the rows above from the count.

========================================================================================
Case 3: the same 20 real effects, inside a family of 2,000.
1,980 nulls this time instead of 180 -- the study got wider, not better.
========================================================================================
family size k = 2000    alpha = 0.05
raw p-values below alpha: 103

procedure             controls                     survive   share   smallest adjusted p
----------------------------------------------------------------------------------------
none (nominal alpha)  FWER not controlled              103    5.1%   2.92e-06
Bonferroni            FWER, any dependence               1    0.1%   0.00584
Holm                  FWER, any dependence               1    0.1%   0.00584
Benjamini-Hochberg    FDR, independent or PRDS           1    0.1%   0.00584
Benjamini-Yekutieli   FDR, any dependence                1    0.1%   0.0478

Report k, the procedure, and the adjusted p-values -- not just the count of
survivors. A reader cannot reconstruct any of the rows above from the count.

Widening the family is not free: every procedure above loses power, while the
unadjusted count rises and is now almost all noise. Adding strata you did not
need costs you the findings you did. Decide the family before you look.

Case 1 is the whole argument for Holm in three numbers. Cases 2 and 3 are the argument for deciding k before you look at the data: the twenty real effects never change, and under Benjamini–Hochberg eight of them are reportable at k = 200 and one at k = 2,000. Under Holm it is 2 and 1 — the widening costs BH the most because BH had the most to lose.

Benjamini–Hochberg controls FDR under independence or positive regression dependence. Site-level web measurement produces tests that are dependent in ways that are neither of those and that you cannot inspect:

• sites share third parties — one Google Tag Manager deployment moves hundreds of sites at once
• sites share a CMS or a CMP — a WordPress-plus-Cookiebot cohort behaves as one unit, and the size of that cohort is not something you chose
• sites share hosting, CDN and IP space, so a network-level failure correlates rows that look independent (see IP classification)
• repeated crawls of the same site are the same site, so a longitudinal panel's tests are correlated by construction (see Archives)

The correlation is positive in some of these and unsigned in others, so the honest position is that you do not know the dependence structure. Two defensible responses, and one indefensible one:

• Benjamini–Yekutieli — valid under arbitrary dependence, at the cost of a factor of about ln(k)+0.58. Four papers in this corpus use it. Zimmeck et al. [16Zimmeck, Sebastian; Goldelman, Daniel; Kaplan, Owen; Brown, Logan; Casler, Justin; Jean-Charles, Judeley; Champeau, Joe; Harkous, Hamza (2024): "Website Data Transparency in the Browser", in: Proceedings on Privacy Enhancing Technologies. (DOI)] report “59 were significant (p⇐0.05 corrected for multiple tests with the Benjamini-Yekutieli procedure)” out of 231 comparisons; Sunlight ships it as its default.
• Resample the dependence — permute or bootstrap at the level of the unit that is actually exchangeable (site, or site-cluster, or user), which estimates the joint distribution instead of assuming it. Bobek & Pröllochs [19Bobek, Michelle; Pröllochs, Nicolas (2026): "Community Fact-Checks Do Not Break Follower Loyalty", in: Proceedings of the ACM Web Conference. (DOI)] cluster standard errors at the user level and report bootstrap simultaneous confidence bands.
• Not defensible: applying BH and saying nothing about dependence. It may well be fine; the point is that the paper gives a reader no way to tell, and the fix is one sentence.

The structural alternative is to stop treating the strata as separate hypotheses. Gelman, Hill & Yajima [6Gelman, Andrew; Hill, Jennifer; Yajima, Masanao (2012): "Why We (Usually) Don't Have to Worry About Multiple Comparisons", Journal of Research on Educational Effectiveness 5(2):189-211. (DOI)] argue that a hierarchical model over the strata — countries, categories, trackers as levels of a random effect — shrinks the extreme estimates toward the mean automatically, which is what a correction is trying to approximate, and gives you an estimate per stratum instead of a survive/die verdict. This is a very good fit for web measurement: the strata are exchangeable-ish, there are many of them, and most have small n. 65 papers in this corpus fit a mixed-effects or multilevel model, and not one frames it as an answer to multiplicity.⁷⁾

A correction is computed over the tests in the paper. The multiplicity that actually threatens a crawl-based finding happened earlier, in the pipeline, and is invisible to any procedure on this page:

• which seed list, and which rank cut (see Website selection, Representative sampling methods)
• which exclusion rule for sites that failed to load, and how many each candidate rule removed
• which denominator each headline percentage sits over
• which classifier, filter list, or confidence threshold, and which version of it (see Website classification, Requests)
• which crawl run counts, after the first one broke

Each of those was a choice made with the data in front of you, and each is a test you effectively ran and did not report. Gelman & Loken [7Gelman, Andrew; Loken, Eric (2014): "The Statistical Crisis in Science", American Scientist 102(6):460. (DOI)] call this the garden of forking paths, and their point is precisely that it does not require any dishonesty — a single analysis path, chosen after seeing the data, has an uncontrolled error rate even though no multiple comparison was ever computed.

No paper in this corpus engages with the idea. The literal string forking paths matches two of the 5,869 full texts, and reading both shows they are packet-forwarding paths and symbolic-execution paths — see What is missing entirely. The ASA statement on p-values [20Wasserstein, Ronald L.; Lazar, Nicole A. (2016): "The ASA Statement on p-Values: Context, Process, and Purpose", The American Statistician 70(2):129-133. (DOI)] matches zero. The standard mechanism that addresses it is committing to the analysis before you look — which is Study preregistration, and which 14 of these 1,025 hypothesis-test papers (1.4%) did.⁸⁾ It is not the only one: the multiverse analysis in Open Questions addresses the same problem after the fact, and a held-out split of the crawl addresses part of it.

The cheap version, if you will not preregister: name your confirmatory tests in the paper, correct within that family only, and label everything else exploratory with uncorrected p-values and no significance claims. Nenadic et al. [14Nenadić, Luka; Rodriguez, David; Calandrino, Joseph A. (2026): "Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law's Impact", Proceedings on Privacy Enhancing Technologies 2026(4):703-723. (DOI)] split their families; nobody in this corpus splits confirmatory from exploratory in a crawl.

Three papers in this corpus have a correction tuple that records the absence of one, and all three give the same reason — the analysis is exploratory — and all three say it in one sentence. That is a defensible position, and it is much better than silence:

• Pu & Grossklags [21Pu, Yu; Grossklags, Jens (2016): "Towards a Model on the Factors Influencing Social App Users' Valuation of Interdependent Privacy", in: Proceedings on Privacy Enhancing Technologies. (DOI)]: “we did not adjust p-values for the multiple testing problem since we consider our preliminary tests of the impact of collection context on privacy valuation as exploratory analysis, where multiplicity adjustments are neither mandatory, nor important”
• Goetzen et al. [22Goetzen, Angelica; Dooley, Samuel; Redmiles, Elissa M. (2022): "Ctrl-Shift: How Privacy Sentiment Changed from 2019 to 2021", in: Proceedings on Privacy Enhancing Technologies. (DOI)]: “because our data are organized from three similar, non-identical populations and the comparisons we make are purely complementary, we do not make any multiple comparison corrections in our regressions”
• Naji et al. [23Naji, Houda; Reichmann, Felix; Bruns, Tobias; Sasse, M. Angela; Naiakshina, Alena (2025): ""It's not my responsibility to write them": An Empirical Study of Software Product Managers and Security Requirements", in: Proceedings of the USENIX Security Symposium. (Link)]: “We did not apply multiple testing corrections, as our analysis is exploratory in nature”

If you declare exploratory, the obligation that comes with it is to stop using the word “significant” and to stop reporting stars. The declaration and the significance claims cannot both stand.

Every figure below comes from the extraction over 5,859 papers from CCS, IMC, NDSS, PoPETs, USENIX Security, TheWebConf and IEEE S&P, 2010–2026, plus a regex sweep over the 5,869 paper.cols.txt full texts. The script is report_pvalue_corrections.mjs, the free-text fold is mcc_fold.mjs, and both live with the full query log, the residue and the sources rejected on pvalue_corrections.

Each row is its own population. “MCC tuple” counts papers with a statistics[].kind == “multiple-comparison-correction” tuple; “corrected” removes the ones whose tuple records the absence of a correction.

hypothesisTest is the denominator that matters: a paper that reports only descriptive statistics has no p-values to correct. The crawled row understates the gap because most crawl papers never run an inferential test at all — cross the two:

A crawl paper that runs a hypothesis test is a third as likely to correct as a user study that does — in the setting where k is set by the data rather than by the protocol. The venue breakdown says the same thing from another angle:

IMC and NDSS are the two lowest. IMC is the most empirical venue in the corpus (98.0% of its papers measure something) and it has the second-lowest correction rate — the same pattern Study preregistration finds for preregistration, where IMC has none at all.

statistics[].method is free text and agrees run-to-run on about 20% of exact strings, so it must be folded before it is counted. 90 distinct strings describe eleven procedures; the fold is mcc_fold.mjs and its unmapped residue is 0 strings and 0 papers. Denominator: the 269 papers with an MCC tuple. Papers are counted once, and a paper naming two procedures appears in two rows, so the shares exceed 100%.

The last two rows are the reason the fold exists rather than a method histogram. Böttger et al. [24Böttger, Christian; Demir, Nurullah; Hörnemann, Jan; Acharya, Bhupendra; Pohlmann, Norbert; Holz, Thorsten; Grosse-Kampmann, Matteo; Urban, Tobias (2025): "Understanding Regional Filter Lists: Efficacy and Impact", in: Proceedings on Privacy Enhancing Technologies. (DOI)] write “due to a violation of sphericity, we applied the Greenhouse-Geisser correction” — a correction, but of the F-test's degrees of freedom, not of multiplicity. Any query that counts kind == “multiple-comparison-correction” and stops there reports both of these as corrections.

An adjusted p cannot be checked, reproduced or compared without k. Two independent passes, agreeing to about a point.

Pass 1, over the 287 tuples that record a real correction — what landed in statistics[].method, .detail or the evidence sentence:

Pass 2, over the whole text of the 265 papers that corrected. A tuple only carries one sentence, so this pass searches the full paper for any count of comparisons, tests or hypotheses within ±1,500 characters of a procedure mention. Doing that naively is useless in this literature: the unguarded regex returned 41 hits, of which the majority were χ² notation — the digit in “χ² tests” sits immediately before the word — plus exponents and USENIX page furniture. With three guards the count is 20, and reading all 20 leaves 16 genuine: 6.0% of the 265. The four rejected, and every accepted match with its surrounding sentence, are printed in the report output on pvalue_corrections.

So: 4.9% by tuple, 6.0% by paper. Roughly one corrected paper in twenty states the size of its family.

The fourteen that do state it are the ones a reader can check, and they read like this:

• Kablo & Cabarcos [10Kablo, Emiram; Cabarcos, Patricia Arias (2023): "Privacy in the Age of Neurotechnology: Investigating Public Attitudes towards Brain Data Collection and Use", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)]: “a total of 106 tests were conducted”, threshold adjusted to 0.00047
• Maass et al. [12Maass, Max; Stöver, Alina; Pridöhl, Henning; Bretthauer, Sebastian; Herrmann, Dominik; Hollick, Matthias; Spiecker, Indra (2021): "Effective Notification Campaigns on the Web: A Matter of Trust, Framing, and Support", in: Proceedings of the USENIX Security Symposium. (Link)]: “a single Holm-Bonferroni correction for all 45 significance tests”
• Weinberg et al. [11Weinberg, Zachary; Sharif, Mahmood; Szurdi, Janos; Christin, Nicolas (2017): "Topics of Controversy: An Empirical Analysis of Web Censorship Lists", in: Proceedings on Privacy Enhancing Technologies. (DOI)]: “a total of 2,904 such combinations”
• Liu et al. [9Liu, Zengrui; Iqbal, Umar; Saxena, Nitesh (2024): "Opted Out, Yet Tracked: Are Regulations Enough to Protect Your Privacy?", in: Proceedings on Privacy Enhancing Technologies. (DOI)]: “compare bid values for all 16 personas”, p “multiplied by 16”
• Despres et al. [15Despres, Tess; Constantino, Marcelino Ayala; Lizola, Naomi Zacarias; Romero, Gerardo Sánchez; He, Shijing; Zhan, Xiao; Abdi, Noura; Abu-Salma, Ruba; Such, Jose; Bernd, Julia (2024): ""My Best Friend's Husband Sees and Knows Everything": A Cross-Contextual and Cross-Country Approach to Understanding Smart Home Privacy", in: Proceedings on Privacy Enhancing Technologies. (DOI)]: “a conservative α of p = 0.002 as the criterion for significance (based on a maximum of 25 hypotheses tested per outcome)”
• Zimmeck et al. [16Zimmeck, Sebastian; Goldelman, Daniel; Kaplan, Owen; Brown, Logan; Casler, Justin; Jean-Charles, Judeley; Champeau, Joe; Harkous, Hamza (2024): "Website Data Transparency in the Browser", in: Proceedings on Privacy Enhancing Technologies. (DOI)]: 59 of 231 comparisons significant after Benjamini–Yekutieli

Note the last two: the family is “per outcome” or “per comparison set”, not “per paper”. That is the right granularity and it is also why k has to be stated — a reader cannot infer which grouping you used.

38 of the 1,025 hypothesis-test papers report both a power analysis and a correction, and the one whose wording makes the sequencing unambiguous is Ho et al. [25Ho, Grant; Mirian, Ariana; Luo, Elisa; Tong, Khang; Lee, Euyhyun; Liu, Lin; Longhurst, Christopher A.; Dameff, Christian; Savage, Stefan; Voelker, Geoffrey M. (2025): "Understanding the Efficacy of Phishing Training in Practice", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)], who fold the correction into the sample-size calculation: “After adjusting for multiple comparisons by taking α* = α/3 to compare three different types of training (interactive, static, and control), we would need N = 2,693 per group, which our study greatly exceeds.” The correction determines how large the study has to be, and computing it afterwards means discovering that the study was too small.¹⁰⁾

Doing it at all is uncommon, because power analysis is uncommon: 72 of the 1,025 hypothesis-test papers (7.0%) report one. The rest of the inference-hygiene stack, over the same denominator:

69 papers report both an effect size and a correction. 178 report a correction and no effect size.

Probes over all 5,869 full texts. Each row is a concept that sits next to a correction in any statistics course, and each is absent or a homograph — worth knowing before you grep this corpus yourself.

The pattern is that this literature applies corrections without ever engaging with the argument about them. The case against routine correction (Rothman [26Rothman, Kenneth J. (1990): "No Adjustments Are Needed for Multiple Comparisons", Epidemiology 1(1):43-46. (DOI)], Perneger [27Perneger, Thomas V. (1998): "What's Wrong with Bonferroni Adjustments", BMJ 316(7139):1236-1238. (DOI)]), the case for modelling instead (Gelman et al. [6Gelman, Andrew; Hill, Jennifer; Yajima, Masanao (2012): "Why We (Usually) Don't Have to Worry About Multiple Comparisons", Journal of Research on Educational Effectiveness 5(2):189-211. (DOI)]), and the case that the whole framing is wrong (Gelman & Loken [7Gelman, Andrew; Loken, Eric (2014): "The Statistical Crisis in Science", American Scientist 102(6):460. (DOI)], the ASA statement [20Wasserstein, Ronald L.; Lazar, Nicole A. (2016): "The ASA Statement on p-Values: Context, Process, and Purpose", The American Statistician 70(2):129-133. (DOI)]) are all missing. A paper that cites any of them is doing something no paper in these seven venues has done.

• Two independent sources, kept apart. The schema field and a regex sweep over full text disagree in both directions and the report prints the disagreement rather than a union. Of the 279 papers whose full text names a procedure, 246 also carry a correction tuple (88.2% schema recall); 33 are named in text and missed by the schema, and 19 carry a tuple whose procedure is not in the named list.
• The free-text fold is a rule list with a printed residue of 0. mcc_fold.mjs has a self-test asserting the rule precedence that matters (Holm-Bonferroni folds to Holm, Dunn-Bonferroni to Bonferroni), and a string matching no rule is printed rather than silently bucketed.
• Two buckets in the fold are not corrections and are excluded from every “corrected” count: the three papers that declare they did not correct, and the two Greenhouse–Geisser sphericity corrections.
• Sentinels are never answers. statistics has no not-stated value — a paper that corrected and did not say so is simply absent, which is why every correction figure here is a lower bound on the practice and an upper bound on the reporting.
• The family-size figure is measured twice and hand-checked. The tuple pass sees one sentence per correction; the full-text pass sees the whole paper but needs guards against χ² notation, exponents and page furniture, and then a reading of all 20 surviving hits. 4.9% and 6.0% are the two answers, four rejected hits are named with reasons, and all 20 are printed with their sentences on the provenance page. Neither pass is a bound: a count in a distant table caption is missed by both.
• A paper counts once, never once per mention. 292 tuples across 269 papers.
• statistics.kind is a mid-band field: two independent extraction runs over identical text agreed on it for 68% of papers, so a repeat run would move these percentages by a few points. That caveat applies to every share on this page and does not apply to the folded procedure ranking, which is a ranking.
• Quotes were checked. All 292 correction quotes were run against paper.cols.txt: 167 exact, 63 partial (≥60% of five-word windows), 62 below that threshold — mostly extractor ellipses and dropped citation markers. Every quote used on this page was then located by hand in the source; all 18 checked out, one of them (Pu & Grossklags) only after the extractor's … was expanded. See pvalue_corrections.
• Venue coverage. Seven venues. EuroS&P, ACSAC, RAID, AsiaCCS, WPES, CHI and SOUPS are absent, and SOUPS matters most: it is where the usable-privacy work that corrects most reliably would appear, so the correction rate here is if anything flattered by its absence.
• 2025 and 2026 are provisional. See the box above.
• Every external claim was fetched on 2026-08-13, not recalled: nine DOIs through doi.org content negotiation, and the SciPy and statsmodels documentation from their own sites. The full log is on pvalue_corrections; corpus-level caveats are on Corpus.

1. k, the size of the family, and what the family is. “Per outcome variable”, “per pair of countries”, “across the 16 personas”. This is the single most-omitted item — 4.9% of correction tuples state it.
2. The procedure by name, with a citation. “Corrected for multiple comparisons” is not a method.
3. Which error rate you chose to control, and why. One sentence: is a single false row fatal (FWER) or costly (FDR)?
4. Adjusted p-values, not just a count of survivors. A reader cannot re-derive the adjustment from “12 comparisons were significant”.
5. Which comparisons are confirmatory and which are exploratory, with the correction applied inside the confirmatory family only, and no significance language on the exploratory part.
6. Your dependence assumption, if you used Benjamini–Hochberg. If you cannot argue positive dependence between site-level tests, say so and use Benjamini–Yekutieli, or resample.
7. An effect size with an interval, for every corrected comparison. At crawl scale this matters more than the p-value, and 178 papers here report a correction without one.
8. The correction inside your power analysis, if you are sizing a study rather than analysing one [25Ho, Grant; Mirian, Ariana; Luo, Elisa; Tong, Khang; Lee, Euyhyun; Liu, Lin; Longhurst, Christopher A.; Dameff, Christian; Savage, Stefan; Voelker, Geoffrey M. (2025): "Understanding the Efficacy of Phishing Training in Practice", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)].
9. If you decline to correct, say so and say why — and then drop the word “significant” [21Pu, Yu; Grossklags, Jens (2016): "Towards a Model on the Factors Influencing Social App Users' Valuation of Interdependent Privacy", in: Proceedings on Privacy Enhancing Technologies. (DOI)] [22Goetzen, Angelica; Dooley, Samuel; Redmiles, Elissa M. (2022): "Ctrl-Shift: How Privacy Sentiment Changed from 2019 to 2021", in: Proceedings on Privacy Enhancing Technologies. (DOI)] [23Naji, Houda; Reichmann, Felix; Bruns, Tobias; Sasse, M. Angela; Naiakshina, Alena (2025): ""It's not my responsibility to write them": An Empirical Study of Software Product Managers and Security Requirements", in: Proceedings of the USENIX Security Symposium. (Link)]. CONSORT 2025 [17Hopewell, Sally; Chan, An-Wen; Collins, Gary S.; Hróbjartsson, Asbjørn; Moher, David; Schulz, Kenneth F.; others, (2025): "CONSORT 2025 explanation and elaboration: updated guideline for reporting randomised trials", BMJ 389:e081124. (DOI)] makes this an explicit requirement in clinical trials: reporting that no method was used is itself a reporting obligation.
10. For a test, write “Holm”, not “Bonferroni”. Same assumption, same code path, never fewer findings. Reach for Bonferroni deliberately, not by default: for simultaneous confidence intervals, or for a threshold pre-declared in a protocol or registration where you do not yet have the p-values Holm needs.

* No measurement paper in this corpus corrects a per-site family. The design that most obviously creates thousands of hypotheses — one test per site — is never followed by a correction, so it is unknown whether the field considers that setting out of scope for testing, or simply does not test it. Both readings have consequences for how a reviewer should treat a per-site claim. * 65 papers fit a multilevel model and none frames it as a multiplicity strategy — at least, none surfaced while reading correction passages, and no one has checked all 65. Whether partial pooling is already doing the work of a correction in this literature without anyone saying so is answerable and unanswered. * Benjamini–Yekutieli is used by four papers, and it is the procedure whose assumptions actually match site-level crawl data. Whether the BH results in this literature would survive BY is checkable on any paper that released its p-values — see Artifacts — and nobody has done it. * The debate is absent. Zero papers cite the ASA statement, Rothman, Perneger, or the garden of forking paths. A short SoK on inference practice in web measurement would be citing an empty shelf, which is unusual and useful. * Nobody has asked these venues to require k. Reporting the family size is a one-line checklist item that would make every correction in the literature checkable, and 94.0% of corrected papers do not do it. The nearest thing in any field is CONSORT's requirement that trials describe the method used — which is a weaker ask than stating k, and nobody has proposed even that much to a security or measurement venue. Their review forms are not public, so the current state can only be read off the papers. * The pipeline-level multiplicity is unmeasured. Nobody has taken a published crawl, enumerated the defensible alternatives at each pipeline decision (seed list, rank cut, exclusion rule, classifier threshold), re-run all of them, and reported the spread of the headline figure. That is a multiverse analysis, it is straightforwardly fundable, and it would say more about this literature's error rate than any correction.

• Hypothesis testing — choosing the test whose p-values you are about to correct.
• Study preregistration — the only mechanism that addresses the multiplicity in the pipeline rather than in the results table.
• Regression — where the family is the set of coefficients, and the correction is applied per model rather than per paper.
• Biases — what a fixed population and a fixed exclusion rule protect you from before any p-value exists.
• Website selection and Representative sampling methods — the strata that become the family.
• Crawling location — per-vantage comparisons, and how many pairs they generate.
• User studies — where two-thirds of the papers that correct live (167 of 247), and where the rate, 38.6%, is three times the crawl rate. Still a minority practice even there.
• Artifacts — releasing the p-values is what makes a correction re-checkable.
• Provenance of this page's figures.

[1]

Holm, Sture (1979): "A Simple Sequentially Rejective Multiple Test Procedure", Scandinavian Journal of Statistics 6:65-70. No DOI: the identifier 10.2307/4615733 reported by OpenAlex is not a registered handle (Link)

[2]

Datta, Amit; Tschantz, Michael Carl; Datta, Anupam (2015): "Automated Experiments on Ad Privacy Settings", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[3]

Lécuyer, Mathias; Spahn, Riley; Spiliopolous, Yannis; Chaintreau, Augustin; Geambasu, Roxana; Hsu, Daniel J. (2015): "Sunlight: Fine-grained Targeting Detection at Scale with Statistical Confidence", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[4]

Benjamini, Yoav; Hochberg, Yosef (1995): "Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing", Journal of the Royal Statistical Society: Series B (Statistical Methodology) 57(1):289-300. (DOI)

[5]

Benjamini, Yoav; Yekutieli, Daniel (2001): "The Control of the False Discovery Rate in Multiple Testing under Dependency", The Annals of Statistics 29(4):1165-1188. (DOI)

[6]

Gelman, Andrew; Hill, Jennifer; Yajima, Masanao (2012): "Why We (Usually) Don't Have to Worry About Multiple Comparisons", Journal of Research on Educational Effectiveness 5(2):189-211. (DOI)

[7]

Gelman, Andrew; Loken, Eric (2014): "The Statistical Crisis in Science", American Scientist 102(6):460. (DOI)

[8]

Valapu, Sulyab Thottungal; Heidemann, John S. (2025): "Towards a Non-Binary View of IPv6 Adoption", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[9]

Liu, Zengrui; Iqbal, Umar; Saxena, Nitesh (2024): "Opted Out, Yet Tracked: Are Regulations Enough to Protect Your Privacy?", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[10]

Kablo, Emiram; Cabarcos, Patricia Arias (2023): "Privacy in the Age of Neurotechnology: Investigating Public Attitudes towards Brain Data Collection and Use", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[11]

Weinberg, Zachary; Sharif, Mahmood; Szurdi, Janos; Christin, Nicolas (2017): "Topics of Controversy: An Empirical Analysis of Web Censorship Lists", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[12]

Maass, Max; Stöver, Alina; Pridöhl, Henning; Bretthauer, Sebastian; Herrmann, Dominik; Hollick, Matthias; Spiecker, Indra (2021): "Effective Notification Campaigns on the Web: A Matter of Trust, Framing, and Support", in: Proceedings of the USENIX Security Symposium. (Link)

[13]

Bouhoula, Ahmed; Kubicek, Karel; Zac, Amit; Cotrini, Carlos; Basin, David (2024): "Automated Large-Scale Analysis of Cookie Notice Compliance", in: Proceedings of the USENIX Security Symposium. (Link)

[14]

Nenadić, Luka; Rodriguez, David; Calandrino, Joseph A. (2026): "Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law's Impact", Proceedings on Privacy Enhancing Technologies 2026(4):703-723. (DOI)

[15]

Despres, Tess; Constantino, Marcelino Ayala; Lizola, Naomi Zacarias; Romero, Gerardo Sánchez; He, Shijing; Zhan, Xiao; Abdi, Noura; Abu-Salma, Ruba; Such, Jose; Bernd, Julia (2024): ""My Best Friend's Husband Sees and Knows Everything": A Cross-Contextual and Cross-Country Approach to Understanding Smart Home Privacy", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[16]

Zimmeck, Sebastian; Goldelman, Daniel; Kaplan, Owen; Brown, Logan; Casler, Justin; Jean-Charles, Judeley; Champeau, Joe; Harkous, Hamza (2024): "Website Data Transparency in the Browser", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[17]

Hopewell, Sally; Chan, An-Wen; Collins, Gary S.; Hróbjartsson, Asbjørn; Moher, David; Schulz, Kenneth F.; others, (2025): "CONSORT 2025 explanation and elaboration: updated guideline for reporting randomised trials", BMJ 389:e081124. (DOI)

[18]

Storey, John D. (2002): "A Direct Approach to False Discovery Rates", Journal of the Royal Statistical Society: Series B (Statistical Methodology) 64(3):479-498. (DOI)

[19]

Bobek, Michelle; Pröllochs, Nicolas (2026): "Community Fact-Checks Do Not Break Follower Loyalty", in: Proceedings of the ACM Web Conference. (DOI)

[20]

Wasserstein, Ronald L.; Lazar, Nicole A. (2016): "The ASA Statement on p-Values: Context, Process, and Purpose", The American Statistician 70(2):129-133. (DOI)

[21]

Pu, Yu; Grossklags, Jens (2016): "Towards a Model on the Factors Influencing Social App Users' Valuation of Interdependent Privacy", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[22]

Goetzen, Angelica; Dooley, Samuel; Redmiles, Elissa M. (2022): "Ctrl-Shift: How Privacy Sentiment Changed from 2019 to 2021", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[23]

Naji, Houda; Reichmann, Felix; Bruns, Tobias; Sasse, M. Angela; Naiakshina, Alena (2025): ""It's not my responsibility to write them": An Empirical Study of Software Product Managers and Security Requirements", in: Proceedings of the USENIX Security Symposium. (Link)

[24]

Böttger, Christian; Demir, Nurullah; Hörnemann, Jan; Acharya, Bhupendra; Pohlmann, Norbert; Holz, Thorsten; Grosse-Kampmann, Matteo; Urban, Tobias (2025): "Understanding Regional Filter Lists: Efficacy and Impact", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[25]

Ho, Grant; Mirian, Ariana; Luo, Elisa; Tong, Khang; Lee, Euyhyun; Liu, Lin; Longhurst, Christopher A.; Dameff, Christian; Savage, Stefan; Voelker, Geoffrey M. (2025): "Understanding the Efficacy of Phishing Training in Practice", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[26]

Rothman, Kenneth J. (1990): "No Adjustments Are Needed for Multiple Comparisons", Epidemiology 1(1):43-46. (DOI)

[27]

Perneger, Thomas V. (1998): "What's Wrong with Bonferroni Adjustments", BMJ 316(7139):1236-1238. (DOI)