| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| statistics:regression [2026/08/19 18:14] – Consistency pass: 33→42 in the limitations bullet, two crawl papers cluster not one (fold vs probe reconciled), zero-inflation and site-random-effect open questions aligned with the corrected findings. Authored by Claude karel.kubicek.claude | statistics:regression [2026/09/21 13:33] (current) – Quote-check bullet refreshed against quote_check.mjs --statistics regression with the pypdf fallback: 267/171/65/27 of 530. Authored by Claude karel.kubicek.claude |
|---|
| ====== Regression ====== | ====== Regression ====== |
| |
| A hypothesis test asks whether two groups of sites differ. A regression asks **by how much, holding other things constant** — and that is almost always the question a web measurement actually has. Do sites with a consent banner set fewer third-party cookies //once you account for site rank, category and country//? Does the number of trackers go up after a policy deadline //once you account for the fact that popular sites had more to begin with//? Every one of those is a regression, and this page is about the four things that make one hard here rather than in a textbook: your outcome is a **count** or a **binary flag per site**, not a nice continuous number; your rows are **not independent** because the same site appears twice or shares a tag manager with two hundred others; your coefficient is **not causal** unless you did something specific to make it so; and half the time the model in the paper is not doing inference at all, it is a **classifier**. | A hypothesis test asks whether two groups of sites differ. A regression asks **by how much, holding other things constant** — and that is almost always the question a web measurement actually has. Do sites with a consent banner set fewer third-party cookies //once you account for site rank, category and country//? Does the number of trackers go up after a policy deadline //once you account for the fact that popular sites had more to begin with//? Every one of those is a regression, and this page is about the four things that make one hard here rather than in a textbook: your outcome is a **count** or a **binary flag per site**, not a nice continuous number; your rows are **not independent** because the same site appears twice or shares a tag manager with two hundred others; your coefficient is **not causal** unless you did something specific to make it so; and often enough to matter the model in the paper is not doing inference at all, it is a **classifier** — 124 papers here use a regression only as a classifier, against 253 that use one as an estimator. |
| |
| This page assumes you know what a regression coefficient and a standard error are. It does not explain the linear model. What it does is date the practice: which model families this literature actually fits, what it does about repeated crawls of the same site (essentially nothing), and what changed in 2025–2026. | This page assumes you know what a regression coefficient and a standard error are. It does not explain the linear model. What it does is date the practice: which model families this literature actually fits, what it does about repeated crawls of the same site (essentially nothing), and what changed in 2025–2026. |
| Two gaps are worth naming on their own. | Two gaps are worth naming on their own. |
| |
| * **Random-effects models are a user-study technique that has not crossed over.** Across all 5,859 papers, **76 fit a random-effects or multilevel model. 58 of the 76 (76.3%) recruited human participants. Four ran a crawl, and not one of the four fits //the site// as the random effect.** Participants get modelled as random effects routinely and correctly — Yuan et al. {[yuan2024_adversarial]} write //"We treat each participant as a random effect because the same user has viewed 15 webpages (i.e., repeated measures)"//. **Exactly one paper in 5,859 fits the //website// as a random effect** — Zeng et al. {[zeng2022_factors]} (IMC 2022), and it is a field study with participants, not a crawl. | * **Random-effects models are a user-study technique that has not crossed over.** Across all 5,859 papers, **76 fit a random-effects or multilevel model. 58 of the 76 (76.3%) recruited human participants. Four ran a crawl, and not one of the four fits //the site// as the random effect.** Participants get modelled as random effects routinely and correctly — Yuan et al. {[yuan2024_adversarial]} write //"We treat each participant as a random effect because the same user has viewed 15 webpages (i.e., repeated measures)"//. **One paper in 5,859 fits the //website// as a random effect** — Zeng et al. {[zeng2022_factors]} (IMC 2022), and it is a field study with participants, not a crawl. That is what two full-text probes and a hand read of every hit found; it is a floor, and [[#What the corpus does about it|an earlier draft of this page got it wrong in the other direction]]. |
| * **Overdispersed counts are usually not fitted with a count model.** A full-text probe across 5,869 stored papers finds **6 that use the word "overdispersion"** and 4 that mention zero-inflation. [[#Counts: third parties per site, requests per page, ads per video|A simulation on this page]] puts the false-positive rate of a Poisson regression on realistically overdispersed tracker counts at **60.4% against a nominal 5%**. | * **Overdispersed counts are usually not fitted with a count model.** A full-text probe across 5,869 stored papers finds **6 that use the word "overdispersion"** and 4 that mention zero-inflation. [[#Counts: third parties per site, requests per page, ads per video|A simulation on this page]] puts the false-positive rate of a Poisson regression on realistically overdispersed tracker counts at **60.4% against a nominal 5%**. |
| |
| And the one thing that surprises people: **whether ignoring the repeated crawls makes your //p//-value too small or too large depends on where your predictor varies.** For a site-level predictor it is too small (7.8% false positives at two waves, **32.4% at twelve**). For a before/after predictor on the same sites it is too **large** — the naive test is conservative and costs you power (61.6% against 75.2% in the same simulation). "Clustering inflates false positives" is the wrong rule to carry around. | And the one thing that surprises people: **whether ignoring the repeated crawls makes your //p//-value too small or too large depends on where your predictor varies.** For a site-level predictor it is too small (8.6% false positives at two waves, **32.4% at twelve**). For a before/after predictor on the same sites it is too **large** — the naive test is conservative and costs you power (61.6% against 75.2% in the same simulation). "Clustering inflates false positives" is the wrong rule to carry around. |
| </WRAP> | </WRAP> |
| |
| |
| * **Gelman, Hill & Vehtari, //Regression and Other Stories//** {[gelman2020_stories]} — **read this one first** if you read only one. It is the 2020 successor to Gelman & Hill's 2007 //Data Analysis Using Regression and Multilevel/Hierarchical Models// {[gelman2007_regression]}, and it is the one to buy now. The catch is that the multilevel material is not in it. //Regression and Other Stories// covers single-level modelling — what a logistic coefficient means, why you should plot the fitted model, how to think about prediction and causal inference — and Cambridge's own listing points readers to the 2007 volume for the multilevel/hierarchical half.((Verified 2026-08-19 against the Cambridge Core listing for //Regression and Other Stories// (2020, 548 pp.), which describes the scope as regression for "comparison, estimation, prediction, and causal inference" and cross-references //Data Analysis Using Regression and Multilevel/Hierarchical Models// as the companion. The 2007 book has not been superseded for the random-effects material.)) **For the random-effects material on this page you still want the 2007 book, chapters 11–13.** | * **Gelman, Hill & Vehtari, //Regression and Other Stories//** {[gelman2020_stories]} — **read this one first** if you read only one. It is the 2020 successor to Gelman & Hill's 2007 //Data Analysis Using Regression and Multilevel/Hierarchical Models// {[gelman2007_regression]}, and it is the one to buy now. The catch is that the multilevel material is not in it. //Regression and Other Stories// covers single-level modelling — what a logistic coefficient means, why you should plot the fitted model, how to think about prediction and causal inference — and Cambridge's own listing points readers to the 2007 volume for the multilevel/hierarchical half.((Verified 2026-08-19 against the Cambridge Core listing for //Regression and Other Stories// (2020, 548 pp.), which describes the scope as regression for "comparison, estimation, prediction, and causal inference" and cross-references //Data Analysis Using Regression and Multilevel/Hierarchical Models// as the companion. The 2007 book has not been superseded for the random-effects material.)) **For the random-effects material on this page you still want the 2007 book, chapters 11–13.** |
| * **Shmueli, "To Explain or to Predict?"** {[shmueli2010_explain]} (Statistical Science 2010) — the four pages to read before you decide whether your logistic regression is a model or a classifier. This is the distinction 124 papers in this corpus are on one side of and 253 on the other, and the two demand entirely different reporting. See [[#Regression as Inference Is Not Regression as a Classifier]]. | * **Shmueli, "To Explain or to Predict?"** {[shmueli2010_explain]} (Statistical Science 2010) — read this before you decide whether your logistic regression is a model or a classifier. This is the distinction 124 papers in this corpus are on one side of and 253 on the other, and the two demand entirely different reporting. See [[#Regression as Inference Is Not Regression as a Classifier]]. |
| * **Cameron & Miller** {[cameron2015_practitioners]} and **MacKinnon, Nielsen & Webb** {[mackinnon2023_clusterrobust]} — cluster-robust inference, written for practitioners rather than econometricians. The direct fix for [[#The Same Site, Crawled Twice]], and the one that fits in a single ''cov_type="cluster"'' argument. Cameron & Miller is the readable introduction; the 2023 guide is the current one and is where the awkward cases live — few clusters, wildly unequal cluster sizes, and the wild bootstrap that fixes both. | * **Cameron & Miller** {[cameron2015_practitioners]} and **MacKinnon, Nielsen & Webb** {[mackinnon2023_clusterrobust]} — cluster-robust inference, written for practitioners rather than econometricians. The direct fix for [[#The Same Site, Crawled Twice]], and the one that fits in a single ''cov_type="cluster"'' argument. Cameron & Miller is the readable introduction; the 2023 guide is the current one and is where the awkward cases live — few clusters, wildly unequal cluster sizes, and the wild bootstrap that fixes both. |
| * **Ver Hoef & Boveng** {[verhoef2007_quasipoisson]} — quasi-Poisson against negative binomial for overdispersed counts, in four pages and one figure. If your outcome is "number of third parties on the site", this decides your model. | * **Ver Hoef & Boveng** {[verhoef2007_quasipoisson]} — quasi-Poisson against negative binomial for overdispersed counts, in seven pages. If your outcome is "number of third parties on the site", this decides your model. |
| * **Mood** {[mood2010_logistic]} (European Sociological Review 2010) — "Logistic Regression: Why We Cannot Do What We Think We Can Do". Odds ratios are **not** comparable across models, across samples, or across groups, even when nothing about the effect changed. If you have ever added a control variable to a logistic model and read the change in the odds ratio as confounding, this is the paper. | * **Mood** {[mood2010_logistic]} (European Sociological Review 2010) — "Logistic Regression: Why We Cannot Do What We Think We Can Do". Odds ratios are **not** comparable across models, across samples, or across groups, even when nothing about the effect changed. If you have ever added a control variable to a logistic model and read the change in the odds ratio as confounding, this is the paper. |
| * **Barr et al.** {[barr2013_maximal]} — "Keep it maximal": which random effects to include, and why leaving one out is an anti-conservative choice rather than a neutral simplification. **Read it with its rebuttal**, Matuschek et al. {[matuschek2017_balancing]}, in the same journal four years later: maximal structures buy type-I error control at a real cost in power, and a data-supported ("parsimonious") structure is often the better trade. The two papers together are the argument; neither alone is. | * **Barr et al.** {[barr2013_maximal]} — "Keep it maximal": which random effects to include, and why leaving one out is an anti-conservative choice rather than a neutral simplification. **Read it with its rebuttal**, Matuschek et al. {[matuschek2017_balancing]}, in the same journal four years later: maximal structures buy type-I error control at a real cost in power, and a data-supported ("parsimonious") structure is often the better trade. The two papers together are the argument; neither alone is. |
| The corpus separates them, and the split is stark. **155 papers name a regression model in ''tools[]'' rather than ''statistics[]'', and 148 of those tuples are categorised ''ml-model-or-algorithm''.** **253 papers name one in ''statistics[]''. The two sets overlap in only 31 papers — so 124 papers use a regression purely as a classifier and never as an estimator.** If you searched the literature for "logistic regression in web measurement" and read the first ten hits, you would probably read ten baselines and learn nothing about inference. | The corpus separates them, and the split is stark. **155 papers name a regression model in ''tools[]'' rather than ''statistics[]'', and 148 of those tuples are categorised ''ml-model-or-algorithm''.** **253 papers name one in ''statistics[]''. The two sets overlap in only 31 papers — so 124 papers use a regression purely as a classifier and never as an estimator.** If you searched the literature for "logistic regression in web measurement" and read the first ten hits, you would probably read ten baselines and learn nothing about inference. |
| |
| <wrap todo> | <WRAP todo> |
| **Say which one you are doing, in the sentence that introduces the model.** "We fit a logistic regression of //banner present// on //rank, category and country// to estimate the association between …" and "we train a logistic-regression classifier as a baseline against our detector" are two different sentences, and a reader cannot tell them apart from a table of coefficients. Shmueli {[shmueli2010_explain]} is the reference if a reviewer pushes back on doing one and reporting the other. | **Say which one you are doing, in the sentence that introduces the model.** "We fit a logistic regression of //banner present// on //rank, category and country// to estimate the association between …" and "we train a logistic-regression classifier as a baseline against our detector" are two different sentences, and a reader cannot tell them apart from a table of coefficients. Shmueli {[shmueli2010_explain]} is the reference if a reviewer pushes back on doing one and reporting the other. |
| </wrap> | </WRAP> |
| |
| The rest of this page is about the first activity. Classifier evaluation belongs with [[Design:Website classification]] and [[Privacy:Javascript]]. | The rest of this page is about the first activity. Classifier evaluation belongs with [[Design:Website classification]] and [[Privacy:Javascript]]. |
| |
| The specific failure is **overdispersion**. A Poisson regression assumes variance equals mean. Real tracker counts have variance many times their mean, and when that assumption breaks the model does not merely fit badly — it reports standard errors that are far too small, and rejects true nulls at a rate nothing like its nominal α. | The specific failure is **overdispersion**. A Poisson regression assumes variance equals mean. Real tracker counts have variance many times their mean, and when that assumption breaks the model does not merely fit badly — it reports standard errors that are far too small, and rejects true nulls at a rate nothing like its nominal α. |
| | |
| | **Compute your own variance-to-mean ratio before you choose the model, and put it in the paper.** It is one line (''y.var() / y.mean()''), it decides the question, and **no paper in this corpus reports one** — which is why the ratio of 13 used below is a plausible guess rather than a measured fact about the web. |
| |
| Part A of the script below measures it. Both groups of sites are drawn from the **same** distribution, so every rejection is a false positive: | Part A of the script below measures it. Both groups of sites are drawn from the **same** distribution, so every rejection is a false positive: |
| * Vu et al. {[vu2025_assessing]} (USENIX Sec 2025) combine the count model with a design: //"we modelled the weekly attack counts … using negative binomial regression - a well-established statistical technique for interrupted time series analysis"//. Weekly counts with a trend, seasonality and an intervention term is the shape of most "did the deadline change anything" questions in this field. | * Vu et al. {[vu2025_assessing]} (USENIX Sec 2025) combine the count model with a design: //"we modelled the weekly attack counts … using negative binomial regression - a well-established statistical technique for interrupted time series analysis"//. Weekly counts with a trend, seasonality and an intervention term is the shape of most "did the deadline change anything" questions in this field. |
| |
| <wrap todo> | <WRAP todo> |
| **If your count is "how many //X// out of an opportunity that varies", you need an offset, not a ratio.** Regressing //trackers per request// or //predatory ads as a share of ads// on covariates throws away the fact that a site with 300 requests and a site with 12 give you very different amounts of information. The count model's answer is a **log offset**: put the exposure on the right-hand side with its coefficient fixed at 1. Mai et al. {[mai2025_more]} do exactly this — //"To normalize the count of predatory ads, we add an offset term that is the log of the total ad count"// — and then report **rate ratios** rather than raw coefficients, which is the form a reader can act on. One paper in 5,859 does this.((Probe: ''offset'' in the regression context, read by hand in the papers the count-model query returned. A paper could use an offset without writing the word; this is a floor.)) | **If your count is "how many //X// out of an opportunity that varies", you need an offset, not a ratio.** Regressing //trackers per request// or //predatory ads as a share of ads// on covariates throws away the fact that a site with 300 requests and a site with 12 give you very different amounts of information. The count model's answer is a **log offset**: put the exposure on the right-hand side with its coefficient fixed at 1. Mai et al. {[mai2025_more]} do exactly this — //"To normalize the count of predatory ads, we add an offset term that is the log of the total ad count"// — and then report **rate ratios** rather than raw coefficients, which is the form a reader can act on. One paper in 5,859 does this.((Probe: ''offset'' in the regression context, read by hand in the papers the count-model query returned. A paper could use an offset without writing the word; this is a floor.)) |
| </wrap> | </WRAP> |
| |
| ==== Binary per-site outcomes ==== | ==== Binary per-site outcomes ==== |
| **Your sample size is the number of //events//, not the number of sites.** A crawl of 100,000 sites where 400 show the behaviour you are modelling gives a logistic regression 400 events, and the usual guidance is on the order of ten events per predictor — relaxable, but knowingly {[vittinghoff2007_relaxing]}. This is the failure mode of the rare-phenomenon paper: an enormous //n// and an unstable model. | **Your sample size is the number of //events//, not the number of sites.** A crawl of 100,000 sites where 400 show the behaviour you are modelling gives a logistic regression 400 events, and the usual guidance is on the order of ten events per predictor — relaxable, but knowingly {[vittinghoff2007_relaxing]}. This is the failure mode of the rare-phenomenon paper: an enormous //n// and an unstable model. |
| |
| <wrap todo> | <WRAP todo> |
| **A pseudo-R² of 0.07 is not a broken model, and saying so is not a weakness.** Kieserman et al. {[kieserman2025_tracker]} fit logistic regressions on 40,150 websites to explain which ones configure Google and Meta trackers to harvest form data, and write: //"It is important to acknowledge that our models are a relatively weak fit; the former has a pseudo R-squared of 0.0721"// … //"This means that their explanatory power is limited. However, we can still draw some useful insights from them"//. They report odds ratios, //p//-values and confidence intervals per feature. That is the model to copy: an inferential regression is not judged on fit, and pretending otherwise is what leads people to report a classifier's AUC in place of a coefficient. | **A pseudo-R² of 0.07 is not a broken model, and saying so is not a weakness.** Kieserman et al. {[kieserman2025_tracker]} fit logistic regressions on 40,150 websites to explain which ones configure Google and Meta trackers to harvest form data, and write: //"It is important to acknowledge that our models are a relatively weak fit; the former has a pseudo R-squared of 0.0721"// … //"This means that their explanatory power is limited. However, we can still draw some useful insights from them"//. They report odds ratios, //p//-values and confidence intervals per feature. That is the model to copy. An inferential regression is not //primarily// judged on fit — a low pseudo-R² says the outcome has other causes, not that the coefficient is wrong — and pretending otherwise is what leads people to report a classifier's AUC in place of a coefficient. (Fit still matters when a reviewer's objection is an omitted variable; that is an argument about which covariates are in the model, not about R².) |
| </wrap> | </WRAP> |
| |
| ==== Proportions, rates and ordinal scales ==== | ==== Proportions, rates and ordinal scales ==== |
| |
| * **A proportion bounded in (0,1)** — share of a site's requests going to trackers, share of a page's cookies that are third-party — is not Gaussian and its variance shrinks at both ends. **Beta regression** {[ferrari2004_beta]} is the model for it; **3 papers in 5,859 use one**, all of them on social-media outcomes rather than crawl outcomes. If the proportion is a count over a known denominator, a binomial or count-with-offset model is usually better than beta regression, because it keeps the denominator. | * **A proportion bounded in (0,1)** — share of a site's requests going to trackers, share of a page's cookies that are third-party — is not Gaussian and its variance shrinks at both ends. **Beta regression** {[ferrari2004_beta]} is the model for it; **3 papers in 5,859 use one**, all of them on social-media outcomes rather than crawl outcomes. If the proportion is a count over a known denominator, a binomial or count-with-offset model is usually better than beta regression, because it keeps the denominator. |
| * **An ordinal outcome** — a Likert response, a severity band, a consent-banner "darkness" grade — should be an ordered logit or a cumulative link model, and **46 papers (11.8%) do this**, almost all of them user studies. Cumulative link mixed models (''ordinal::clmm'' in R) are the standard combination when the same participant answers several items. | * **An ordinal outcome** — a severity band, a consent-banner "darkness" grade, a compliance level — is an ordered logit or a cumulative link model, and **46 papers (11.8%) do this**. Almost all of them are user studies scoring Likert items, so the practice and its tooling live on [[Design:User studies]]; what is worth carrying over is that a crawl's ordinal outcomes get treated as numbers far more casually than a survey's do. |
| * **Time-to-event** — how long until a domain is taken down, until a vulnerability is patched, until a tracker reappears — is a Cox model, used by **7 papers (1.8%)**. If your longitudinal crawl ends before some sites have had the event, that is right-censoring, and dropping those sites or coding them as "never" both bias the answer. | * **Time-to-event** — how long until a domain is taken down, until a vulnerability is patched, until a tracker reappears — is a Cox model, used by **7 papers (1.8%)**. If your longitudinal crawl ends before some sites have had the event, that is right-censoring, and dropping those sites or coding them as "never" both bias the answer. |
| |
| **This is a different failure from a multiplicity problem and from a wrong test.** No correction on [[Statistics:Pvalue corrections|P-value corrections]] repairs a standard error computed from the wrong number of independent units, and no rank test on [[Statistics:Hypothesis testing]] does either. A paper can have all three. | **This is a different failure from a multiplicity problem and from a wrong test.** No correction on [[Statistics:Pvalue corrections|P-value corrections]] repairs a standard error computed from the wrong number of independent units, and no rank test on [[Statistics:Hypothesis testing]] does either. A paper can have all three. |
| </WRAP> | </WRAP> |
| | |
| | ==== What to cluster on when there is only one crawl ==== |
| | |
| | The panel case has an obvious clustering unit: the site. A **single-wave** crawl does not, and it is the commonest design in this corpus — which makes "what do I cluster on?" the question this page is most likely to be asked and the one the literature gives least help with. **No paper in the corpus answers it.** The two crawl papers that cluster at all cluster on the product {[becerrilarreola2023_method]} and on the website across snapshots {[nenadic2026_swiss]} — and the second is a panel, so its unit is the repeated observation rather than a structural group. Nobody has clustered a single-wave crawl on anything. |
| | |
| | What you have to choose between, in rough order of how much dependence each induces and how easily you can observe it: |
| | |
| | ^ Candidate cluster ^ Why sites in it move together ^ How you get it ^ |
| | | **the operator / owning entity** | one company's fifty domains are one decision about trackers | WHOIS, a public entity mapping, or the eTLD+1 of a shared login — see [[Design:Website classification]] | |
| | | **the consent platform** | a CMP's default template decides banner text, categories and pre-ticked boxes for every customer | detectable from the crawl itself; see [[Privacy:Consent]] | |
| | | **the tag manager container** | one GTM container ships the same third parties to everything it is installed on | detectable from the request graph; see [[Privacy:Requests]] | |
| | | **the CMS or site builder** | a Wix/Shopify/WordPress-plus-plugin cohort behaves as one configuration | fingerprintable from headers and paths | |
| | | **hosting / CDN / IP block** | one provider's configuration correlates rows that look unrelated | see [[Design:IP classification]] | |
| | | **the country or language of the site** | a whole legal regime, and often one national CMP vendor | see [[Design:Crawling location]] and [[Programming:Multilingual support]] | |
| | |
| | Three practical points, none of which the corpus can settle for you: |
| | |
| | * **Cluster on the coarsest level you think the dependence lives at**, not on the one that is easiest to compute. Clustering too finely leaves the dependence in; clustering too coarsely costs precision but is not wrong. |
| | * **You cannot cluster on two crossed dimensions with ''cov_type="cluster"''.** If you believe both the CMP and the hosting provider matter, that is two-way clustering, and ''statsmodels'' does not do it — MacKinnon et al. {[mackinnon2023_clusterrobust]} cover the estimator, and in practice people fall back to the coarser of the two. |
| | * **Say which one you chose and why, even if the choice is arguable.** A stated clustering unit a reviewer disagrees with is a conversation; an unstated one is the 79.5%. |
| |
| ==== Four remedies, and which to pick ==== | ==== Four remedies, and which to pick ==== |
| |
| ^ Remedy ^ What it estimates ^ When it is the right one ^ Cost ^ | ^ Remedy ^ What it estimates ^ When it is the right one ^ Cost ^ |
| | **Cluster-robust standard errors**, clustered on the site | the same coefficients; only the standard errors change | the default. You want a population-average effect and you have enough clusters | needs "enough" clusters — the usual rule of thumb is a few dozen at minimum, and web crawls have thousands {[cameron2015_practitioners]}. But **cluster //size// imbalance is its own failure mode**, and a crawl is the worst case: one tag manager on 5,000 sites and another on three. Huang {[huang2025_clusterfail]} shows the CR0 sandwich estimator misbehaves badly under that imbalance and recommends the CR2 small-sample correction; MacKinnon et al. {[mackinnon2023_clusterrobust]} recommend the wild cluster bootstrap | | | **Cluster-robust standard errors**, clustered on the site | the same coefficients; only the standard errors change | the default. You want a population-average effect and you have enough clusters | needs "enough" clusters — the usual rule of thumb is a few dozen at minimum, and web crawls have thousands {[cameron2015_practitioners]}. But **cluster //size// imbalance is its own failure mode**, and a crawl is the worst case: one tag manager on 5,000 sites and another on three. Huang {[huang2026_clusterfail]} shows the CR0 sandwich estimator misbehaves badly under that imbalance and recommends the CR2 small-sample correction; MacKinnon et al. {[mackinnon2023_clusterrobust]} recommend the wild cluster bootstrap | |
| | **Random intercept for the site** (mixed model / GLMM) | a **site-specific** effect: how the outcome changes for a given site | you want to describe between-site variance itself, or you have few observations per site and want to borrow strength | for a logistic model the coefficient is no longer the population-average one, and is not comparable with a cluster-robust fit {[mood2010_logistic]} | | | **Random intercept for the site** (mixed model / GLMM) | a **site-specific** effect: how the outcome changes for a given site | you want to describe between-site variance itself, or you have few observations per site and want to borrow strength | for a logistic model the coefficient is no longer the population-average one, and is not comparable with a cluster-robust fit {[mood2010_logistic]} | |
| | **Fixed effects for the site** (a dummy per site) | a purely **within-site** effect | before/after with a within-site predictor and no interest in between-site comparisons | eats all site-level predictors — you cannot estimate the effect of "is a news site" | | | **Fixed effects for the site** (a dummy per site) | a purely **within-site** effect | before/after with a within-site predictor and no interest in between-site comparisons | eats all site-level predictors — you cannot estimate the effect of "is a news site" | |
| | **GEE with an exchangeable working correlation** | a population-average effect, like cluster-robust | you want the marginal effect and are willing to model the correlation structure | 2 papers in this corpus use one; reviewers in this field will not recognise it | | | **GEE with an exchangeable working correlation** | a population-average effect, like cluster-robust | you want the marginal effect and are willing to model the correlation structure | 2 papers in this corpus use one, so expect to have to explain it | |
| |
| For a crawl the honest default is **cluster-robust standard errors clustered on the site**, because it is one argument, it does not change the estimand, and thousands of clusters is exactly the regime where it works. Say //which// correction you used: ''statsmodels'''s ''cov_type="cluster"'' gives you the CR0 sandwich, which is the one that degrades under unequal cluster sizes {[huang2025_clusterfail]}. Reach for a mixed model when the between-site variance is itself the result — "how much of the variation in tracker count is between sites rather than within them" is a mixed model's question and nothing else's. Barr et al. {[barr2013_maximal]} on which random effects to include; Moineddin et al. {[moineddin2007_simulation]} on why a mixed model with few observations per group is less well-behaved than its output suggests. | For a crawl the honest default is **cluster-robust standard errors clustered on the site**, because it is one argument, it does not change the estimand, and thousands of clusters is exactly the regime where it works. Say //which// correction you used: ''statsmodels'''s ''cov_type="cluster"'' gives you the CR0 sandwich, which is the one that degrades under unequal cluster sizes {[huang2026_clusterfail]}. Reach for a mixed model when the between-site variance is itself the result — "how much of the variation in tracker count is between sites rather than within them" is a mixed model's question and nothing else's. Barr et al. {[barr2013_maximal]} on which random effects to include; Moineddin et al. {[moineddin2007_simulation]} on why a mixed model with few observations per group is less well-behaved than its output suggests. |
| |
| ==== What a good one looks like ==== | ==== What a good one looks like ==== |
| |
| That is five decisions a crawl paper has to make, made explicitly. It is also, as far as this corpus goes, the **only** one that makes all five. | That is five decisions a crawl paper has to make, made explicitly. It is also, as far as this corpus goes, the **only** one that makes all five. |
| | |
| | Note the word **balanced**. Nenadić et al. analyse the 11,800 sites present in both snapshots, which is the clean case and not the usual one: a re-crawl six months later loses sites to expiry, blocking, redirection and rewritten pages. Dropping the lost sites silently turns your panel into a survivorship sample of the sites that stayed reachable, which is precisely the ones least likely to have changed. Report the attrition — how many sites entered, how many survived, and whether the survivors differ on the covariates you care about — before reporting the model. [[Statistics:Hypothesis testing#Attrition: the pairs you lose between the two crawls|The hypothesis-testing page treats attrition for paired tests]] and the argument is the same one; an unbalanced panel additionally means a mixed model and a fixed-effects model no longer answer quite the same question, because the fixed-effects fit discards every site seen once. |
| |
| Two 2026 near-neighbours are worth reading alongside it, both on platform data rather than crawls: Chuai et al. {[chuai2026_consensus]} fit a //"multilevel mixed-effects logistic regression"// with author- and post-level random effects and separately //"cluster robust standard errors at the note level to account for potential within-note correlations over time"//; Bobek et al. {[bobek2026_community]} run a staggered difference-in-differences with Callaway–Sant'Anna group-time treatment effects {[callaway2021_did]}, propensity-score weighting, a pre-trends test and bootstrap simultaneous confidence bands. | Two 2026 near-neighbours are worth reading alongside it, both on platform data rather than crawls: Chuai et al. {[chuai2026_consensus]} fit a //"multilevel mixed-effects logistic regression"// with author- and post-level random effects and separately //"cluster robust standard errors at the note level to account for potential within-note correlations over time"//; Bobek et al. {[bobek2026_community]} run a staggered difference-in-differences with Callaway–Sant'Anna group-time treatment effects {[callaway2021_did]}, propensity-score weighting, a pre-trends test and bootstrap simultaneous confidence bands. |
| ==== Dependence and identification ==== | ==== Dependence and identification ==== |
| |
| ^ Dependence handling ^ Papers ^ Share of 391 ^ | The dependence table is [[#What the corpus does about it|above]] and is not repeated here. Two notes on how to read it. |
| | none stated in the method name | 311 | 79.5% | | |
| | random effects (mixed-effects / multilevel) | 75 | 19.2% | | |
| | robust (heteroskedasticity-consistent) SEs only | 3 | 0.8% | | |
| | fixed effects for the unit | 3 | 0.8% | | |
| | GEE (population-averaged) | 2 | 0.5% | | |
| | "repeated-measures" with no random effect named | 1 | 0.3% | | |
| | cluster-robust standard errors | 1 | 0.3% | | |
| |
| ''cluster-robust'' and ''robust'' are separated on purpose. Heteroskedasticity-consistent (HC / White / Huber) standard errors fix a variance assumption; they do nothing whatever about correlated rows. Merging the two rows would let three papers that hardened against heteroskedasticity read as papers that handled dependence. | ''cluster-robust'' and ''robust'' are separated on purpose. Heteroskedasticity-consistent (HC / White / Huber) standard errors fix a variance assumption; they do nothing whatever about correlated rows. Merging the two rows would let three papers that hardened against heteroskedasticity read as papers that handled dependence. |
| |
| The fold reads the **method string**, so this row is a floor: Chuai et al. {[chuai2026_consensus]} appear here under "robust" because that is what their regression tuples say, while the paper separately states that it clusters at the note level. The full-text probe below is the check on that, and it finds four clustering papers where the fold finds one. | The fold reads the **method string**, so this row is a floor: Chuai et al. {[chuai2026_consensus]} appear here under "robust" because that is what their regression tuples say, while the paper separately states that it clusters at the note level. The full-text probe below is the check on that, and it finds four clustering papers where the fold finds one. |
| | |
| | Rows in both tables below overlap and neither sums to 391: **19 of the 391 have no tuple the fold could place as a model at all**, so they get no dependence or identification value, and a paper fitting two models can appear in two rows. The 372 papers that do have a placeable model produce 396 axis-2 and 377 axis-3 row memberships. |
| |
| ^ Identification strategy ^ Papers ^ Share of 391 ^ | ^ Identification strategy ^ Papers ^ Share of 391 ^ |
| | cluster-robust standard errors | 1 | 1.6% | | | cluster-robust standard errors | 1 | 1.6% | |
| | fixed effects | 1 | 1.6% | | | fixed effects | 1 | 1.6% | |
| | |
| | The dependence rows sum to 61, not 62: two of the 62 have no placeable model and one of those has no dependence value either. |
| |
| Crawl papers are **more** likely to fit a count model than the regression population as a whole (11.3% against 4.3%), which is what you would expect given what they measure, and **less** likely to do anything about dependence (87.1% silent against 79.5%), which is the opposite of what the data structure warrants. **Two crawl papers in 1,120 cluster their standard errors** — Becerril-Arreola {[becerrilarreola2023_method]}, who says so in the method the extraction captured, and Nenadić et al. {[nenadic2026_swiss]}, who say so only in prose, which is why this table counts one and the full-text probe below counts two. | Crawl papers are **more** likely to fit a count model than the regression population as a whole (11.3% against 4.3%), which is what you would expect given what they measure, and **less** likely to do anything about dependence (87.1% silent against 79.5%), which is the opposite of what the data structure warrants. **Two crawl papers in 1,120 cluster their standard errors** — Becerril-Arreola {[becerrilarreola2023_method]}, who says so in the method the extraction captured, and Nenadić et al. {[nenadic2026_swiss]}, who say so only in prose, which is why this table counts one and the full-text probe below counts two. |
| * **A paper counts once**, never once per mention. 530 tuples across 391 papers. | * **A paper counts once**, never once per mention. 530 tuples across 391 papers. |
| * **''statistics.kind'' is a mid-band extraction field.** Two independent runs over identical text agreed on it for about two-thirds of papers, so a repeat run would move these percentages by a few points. That agreement figure was measured on the **previous** corpus run and has not been re-measured on this one; treat it as an order of magnitude. | * **''statistics.kind'' is a mid-band extraction field.** Two independent runs over identical text agreed on it for about two-thirds of papers, so a repeat run would move these percentages by a few points. That agreement figure was measured on the **previous** corpus run and has not been re-measured on this one; treat it as an order of magnitude. |
| * **Quotes were checked in bulk and by hand.** All 530 regression quotes against the text the extractor read: **267 exact (50.4%) after whitespace normalisation, 170 partial (32.1%) at ≥60% of five-word windows, 93 (17.5%) below that**. Below-threshold is **not** "unsupported" — every quote published on this page was located by hand in ''paper.cols.txt'', including several that scored below threshold because the extractor dropped a citation marker or the two-column splice cut the sentence. Details, including which ones, on [[provenance:statistics:regression]]. | * **Quotes were checked in bulk and by hand.** All 530 regression quotes against the text the extractor read, and then — for anything below threshold — against an independent ''pypdf'' rendering of the same ''paper.pdf'': **267 exact (50.4%) after whitespace normalisation, 171 partial (32.3%) at ≥60% of five-word windows, 65 (12.3%) located only in the second rendering, 27 (5.1%) below threshold in both**. The 65 are a defect in the stored text, not in the extraction. Below-threshold is **not** "unsupported" either — every quote published on this page was located by hand in ''paper.cols.txt'', including several that scored below threshold because the extractor dropped a citation marker or the two-column splice cut the sentence. Details, including which ones, on [[provenance:statistics:regression]]. |
| * **''statistics'' has no ''not-stated'' sentinel**, so the reporting-gap figures above are bounds on the silence rather than rates of it. | * **''statistics'' has no ''not-stated'' sentinel**, so the reporting-gap figures above are bounds on the silence rather than rates of it. |
| * **The ''detail'' field is not a measurement of reporting practice.** It is a capped free-text field the extractor filled when a value happened to be in the quoted sentence. Counts of R², CIs and //p//-values inside it establish an ordering and nothing more. | * **The ''detail'' field is not a measurement of reporting practice.** It is a capped free-text field the extractor filled when a value happened to be in the quoted sentence. Counts of R², CIs and //p//-values inside it establish an ordering and nothing more. |
| ===== Open Questions ===== | ===== Open Questions ===== |
| |
| | <WRAP todo> |
| * **Nobody has estimated the intra-class correlation of a web-measurement outcome.** The simulation on this page shows the entire size of the error depends on it, and no paper in this corpus reports one. "What is the ICC of //sets a third-party cookie before consent// by consent platform, by CMS, by hosting provider, and by the same site across monthly crawls?" is a small, self-contained study that would tell every crawl paper how wrong its standard errors are. The same gap is open on [[Statistics:Hypothesis testing]] and it is one study away from being closed for both. | * **Nobody has estimated the intra-class correlation of a web-measurement outcome.** The simulation on this page shows the entire size of the error depends on it, and no paper in this corpus reports one. "What is the ICC of //sets a third-party cookie before consent// by consent platform, by CMS, by hosting provider, and by the same site across monthly crawls?" is a small, self-contained study that would tell every crawl paper how wrong its standard errors are. The same gap is open on [[Statistics:Hypothesis testing]] and it is one study away from being closed for both. |
| * **No crawl paper models the site as a random effect.** One non-crawl field study does {[zeng2022_factors]}, so the technique is not unknown in this community — it has just never been pointed at a repeated crawl. Whether that is because cluster-robust standard errors are the better tool anyway (they usually are), or because nobody has tried, is not answerable from text. Somebody should re-analyse a published longitudinal crawl both ways and report how much moves. | * **No crawl paper models the site as a random effect.** One non-crawl field study does {[zeng2022_factors]}, so the technique is not unknown in this community — it has just never been pointed at a repeated crawl. Whether that is because cluster-robust standard errors are the better tool anyway (they usually are), or because nobody has tried, is not answerable from text. Somebody should re-analyse a published longitudinal crawl both ways and report how much moves. |
| * **The 2022-onward arrival of causal designs is ten papers.** Whether it is a durable turn or a handful of authors with economics training is not decidable yet, and the 2025–2026 evidence for it sits in the provisional years. | * **The 2022-onward arrival of causal designs is ten papers.** Whether it is a durable turn or a handful of authors with economics training is not decidable yet, and the 2025–2026 evidence for it sits in the provisional years. |
| * **None of the seven venues asks for any of this.** As on [[Statistics:Hypothesis testing]], the calls for papers contain no statistical-reporting requirement, so the only thing holding a regression section to a standard is an individual reviewer. | * **None of the seven venues asks for any of this.** As on [[Statistics:Hypothesis testing]], the calls for papers contain no statistical-reporting requirement, so the only thing holding a regression section to a standard is an individual reviewer. |
| | </WRAP> |
| ===== Related Pages ===== | ===== Related Pages ===== |
| |
