Measure The Web

User Tools

Site Tools

Trace: • darkpatterns • stateful_stateless • regression • ethics • server_side_tracking • pvalue_corrections • website_classification
statistics:regression

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
statistics:regression [2026/08/19 18:14] – Consistency pass: 33→42 in the limitations bullet, two crawl papers cluster not one (fold vs probe reconciled), zero-inflation and site-random-effect open questions aligned with the corrected findings. Authored by Claude karel.kubicek.claudestatistics:regression [2026/08/21 14:50] (current) – Boxes: <wrap> renders a span, use uppercase <WRAP>; one box per list, not per bullet karel.kubicek.claude
Line 1: Line 1:
 ====== Regression ====== ====== Regression ======
  
-A hypothesis test asks whether two groups of sites differ. A regression asks **by how much, holding other things constant** — and that is almost always the question a web measurement actually has. Do sites with a consent banner set fewer third-party cookies //once you account for site rank, category and country//? Does the number of trackers go up after a policy deadline //once you account for the fact that popular sites had more to begin with//? Every one of those is a regression, and this page is about the four things that make one hard here rather than in a textbook: your outcome is a **count** or a **binary flag per site**, not a nice continuous number; your rows are **not independent** because the same site appears twice or shares a tag manager with two hundred others; your coefficient is **not causal** unless you did something specific to make it so; and half the time the model in the paper is not doing inference at all, it is a **classifier**.+A hypothesis test asks whether two groups of sites differ. A regression asks **by how much, holding other things constant** — and that is almost always the question a web measurement actually has. Do sites with a consent banner set fewer third-party cookies //once you account for site rank, category and country//? Does the number of trackers go up after a policy deadline //once you account for the fact that popular sites had more to begin with//? Every one of those is a regression, and this page is about the four things that make one hard here rather than in a textbook: your outcome is a **count** or a **binary flag per site**, not a nice continuous number; your rows are **not independent** because the same site appears twice or shares a tag manager with two hundred others; your coefficient is **not causal** unless you did something specific to make it so; and often enough to matter the model in the paper is not doing inference at all, it is a **classifier** — 124 papers here use a regression only as a classifier, against 253 that use one as an estimator.
  
 This page assumes you know what a regression coefficient and a standard error are. It does not explain the linear model. What it does is date the practice: which model families this literature actually fits, what it does about repeated crawls of the same site (essentially nothing), and what changed in 2025–2026. This page assumes you know what a regression coefficient and a standard error are. It does not explain the linear model. What it does is date the practice: which model families this literature actually fits, what it does about repeated crawls of the same site (essentially nothing), and what changed in 2025–2026.
Line 18: Line 18:
 Two gaps are worth naming on their own. Two gaps are worth naming on their own.
  
-  * **Random-effects models are a user-study technique that has not crossed over.** Across all 5,859 papers, **76 fit a random-effects or multilevel model. 58 of the 76 (76.3%) recruited human participants. Four ran a crawl, and not one of the four fits //the site// as the random effect.** Participants get modelled as random effects routinely and correctly — Yuan et al. {[yuan2024_adversarial]} write //"We treat each participant as a random effect because the same user has viewed 15 webpages (i.e., repeated measures)"//. **Exactly one paper in 5,859 fits the //website// as a random effect** — Zeng et al. {[zeng2022_factors]} (IMC 2022), and it is a field study with participants, not a crawl.+  * **Random-effects models are a user-study technique that has not crossed over.** Across all 5,859 papers, **76 fit a random-effects or multilevel model. 58 of the 76 (76.3%) recruited human participants. Four ran a crawl, and not one of the four fits //the site// as the random effect.** Participants get modelled as random effects routinely and correctly — Yuan et al. {[yuan2024_adversarial]} write //"We treat each participant as a random effect because the same user has viewed 15 webpages (i.e., repeated measures)"//. **One paper in 5,859 fits the //website// as a random effect** — Zeng et al. {[zeng2022_factors]} (IMC 2022), and it is a field study with participants, not a crawl. That is what two full-text probes and a hand read of every hit found; it is a floor, and [[#What the corpus does about it|an earlier draft of this page got it wrong in the other direction]].
   * **Overdispersed counts are usually not fitted with a count model.** A full-text probe across 5,869 stored papers finds **6 that use the word "overdispersion"** and 4 that mention zero-inflation. [[#Counts: third parties per site, requests per page, ads per video|A simulation on this page]] puts the false-positive rate of a Poisson regression on realistically overdispersed tracker counts at **60.4% against a nominal 5%**.   * **Overdispersed counts are usually not fitted with a count model.** A full-text probe across 5,869 stored papers finds **6 that use the word "overdispersion"** and 4 that mention zero-inflation. [[#Counts: third parties per site, requests per page, ads per video|A simulation on this page]] puts the false-positive rate of a Poisson regression on realistically overdispersed tracker counts at **60.4% against a nominal 5%**.
  
-And the one thing that surprises people: **whether ignoring the repeated crawls makes your //p//-value too small or too large depends on where your predictor varies.** For a site-level predictor it is too small (7.8% false positives at two waves, **32.4% at twelve**). For a before/after predictor on the same sites it is too **large** — the naive test is conservative and costs you power (61.6% against 75.2% in the same simulation). "Clustering inflates false positives" is the wrong rule to carry around.+And the one thing that surprises people: **whether ignoring the repeated crawls makes your //p//-value too small or too large depends on where your predictor varies.** For a site-level predictor it is too small (8.6% false positives at two waves, **32.4% at twelve**). For a before/after predictor on the same sites it is too **large** — the naive test is conservative and costs you power (61.6% against 75.2% in the same simulation). "Clustering inflates false positives" is the wrong rule to carry around.
 </WRAP> </WRAP>
  
Line 27: Line 27:
  
   * **Gelman, Hill & Vehtari, //Regression and Other Stories//** {[gelman2020_stories]} — **read this one first** if you read only one. It is the 2020 successor to Gelman & Hill's 2007 //Data Analysis Using Regression and Multilevel/Hierarchical Models// {[gelman2007_regression]}, and it is the one to buy now. The catch is that the multilevel material is not in it. //Regression and Other Stories// covers single-level modelling — what a logistic coefficient means, why you should plot the fitted model, how to think about prediction and causal inference — and Cambridge's own listing points readers to the 2007 volume for the multilevel/hierarchical half.((Verified 2026-08-19 against the Cambridge Core listing for //Regression and Other Stories// (2020, 548 pp.), which describes the scope as regression for "comparison, estimation, prediction, and causal inference" and cross-references //Data Analysis Using Regression and Multilevel/Hierarchical Models// as the companion. The 2007 book has not been superseded for the random-effects material.)) **For the random-effects material on this page you still want the 2007 book, chapters 11–13.**   * **Gelman, Hill & Vehtari, //Regression and Other Stories//** {[gelman2020_stories]} — **read this one first** if you read only one. It is the 2020 successor to Gelman & Hill's 2007 //Data Analysis Using Regression and Multilevel/Hierarchical Models// {[gelman2007_regression]}, and it is the one to buy now. The catch is that the multilevel material is not in it. //Regression and Other Stories// covers single-level modelling — what a logistic coefficient means, why you should plot the fitted model, how to think about prediction and causal inference — and Cambridge's own listing points readers to the 2007 volume for the multilevel/hierarchical half.((Verified 2026-08-19 against the Cambridge Core listing for //Regression and Other Stories// (2020, 548 pp.), which describes the scope as regression for "comparison, estimation, prediction, and causal inference" and cross-references //Data Analysis Using Regression and Multilevel/Hierarchical Models// as the companion. The 2007 book has not been superseded for the random-effects material.)) **For the random-effects material on this page you still want the 2007 book, chapters 11–13.**
-  * **Shmueli, "To Explain or to Predict?"** {[shmueli2010_explain]} (Statistical Science 2010) — the four pages to read before you decide whether your logistic regression is a model or a classifier. This is the distinction 124 papers in this corpus are on one side of and 253 on the other, and the two demand entirely different reporting. See [[#Regression as Inference Is Not Regression as a Classifier]].+  * **Shmueli, "To Explain or to Predict?"** {[shmueli2010_explain]} (Statistical Science 2010) — read this before you decide whether your logistic regression is a model or a classifier. This is the distinction 124 papers in this corpus are on one side of and 253 on the other, and the two demand entirely different reporting. See [[#Regression as Inference Is Not Regression as a Classifier]].
   * **Cameron & Miller** {[cameron2015_practitioners]} and **MacKinnon, Nielsen & Webb** {[mackinnon2023_clusterrobust]} — cluster-robust inference, written for practitioners rather than econometricians. The direct fix for [[#The Same Site, Crawled Twice]], and the one that fits in a single ''cov_type="cluster"'' argument. Cameron & Miller is the readable introduction; the 2023 guide is the current one and is where the awkward cases live — few clusters, wildly unequal cluster sizes, and the wild bootstrap that fixes both.   * **Cameron & Miller** {[cameron2015_practitioners]} and **MacKinnon, Nielsen & Webb** {[mackinnon2023_clusterrobust]} — cluster-robust inference, written for practitioners rather than econometricians. The direct fix for [[#The Same Site, Crawled Twice]], and the one that fits in a single ''cov_type="cluster"'' argument. Cameron & Miller is the readable introduction; the 2023 guide is the current one and is where the awkward cases live — few clusters, wildly unequal cluster sizes, and the wild bootstrap that fixes both.
-  * **Ver Hoef & Boveng** {[verhoef2007_quasipoisson]} — quasi-Poisson against negative binomial for overdispersed counts, in four pages and one figure. If your outcome is "number of third parties on the site", this decides your model.+  * **Ver Hoef & Boveng** {[verhoef2007_quasipoisson]} — quasi-Poisson against negative binomial for overdispersed counts, in seven pages. If your outcome is "number of third parties on the site", this decides your model.
   * **Mood** {[mood2010_logistic]} (European Sociological Review 2010) — "Logistic Regression: Why We Cannot Do What We Think We Can Do". Odds ratios are **not** comparable across models, across samples, or across groups, even when nothing about the effect changed. If you have ever added a control variable to a logistic model and read the change in the odds ratio as confounding, this is the paper.   * **Mood** {[mood2010_logistic]} (European Sociological Review 2010) — "Logistic Regression: Why We Cannot Do What We Think We Can Do". Odds ratios are **not** comparable across models, across samples, or across groups, even when nothing about the effect changed. If you have ever added a control variable to a logistic model and read the change in the odds ratio as confounding, this is the paper.
   * **Barr et al.** {[barr2013_maximal]} — "Keep it maximal": which random effects to include, and why leaving one out is an anti-conservative choice rather than a neutral simplification. **Read it with its rebuttal**, Matuschek et al. {[matuschek2017_balancing]}, in the same journal four years later: maximal structures buy type-I error control at a real cost in power, and a data-supported ("parsimonious") structure is often the better trade. The two papers together are the argument; neither alone is.   * **Barr et al.** {[barr2013_maximal]} — "Keep it maximal": which random effects to include, and why leaving one out is an anti-conservative choice rather than a neutral simplification. **Read it with its rebuttal**, Matuschek et al. {[matuschek2017_balancing]}, in the same journal four years later: maximal structures buy type-I error control at a real cost in power, and a data-supported ("parsimonious") structure is often the better trade. The two papers together are the argument; neither alone is.
Line 45: Line 45:
 The corpus separates them, and the split is stark. **155 papers name a regression model in ''tools[]'' rather than ''statistics[]'', and 148 of those tuples are categorised ''ml-model-or-algorithm''.** **253 papers name one in ''statistics[]''. The two sets overlap in only 31 papers — so 124 papers use a regression purely as a classifier and never as an estimator.** If you searched the literature for "logistic regression in web measurement" and read the first ten hits, you would probably read ten baselines and learn nothing about inference. The corpus separates them, and the split is stark. **155 papers name a regression model in ''tools[]'' rather than ''statistics[]'', and 148 of those tuples are categorised ''ml-model-or-algorithm''.** **253 papers name one in ''statistics[]''. The two sets overlap in only 31 papers — so 124 papers use a regression purely as a classifier and never as an estimator.** If you searched the literature for "logistic regression in web measurement" and read the first ten hits, you would probably read ten baselines and learn nothing about inference.
  
-<wrap todo>+<WRAP todo>
 **Say which one you are doing, in the sentence that introduces the model.** "We fit a logistic regression of //banner present// on //rank, category and country// to estimate the association between …" and "we train a logistic-regression classifier as a baseline against our detector" are two different sentences, and a reader cannot tell them apart from a table of coefficients. Shmueli {[shmueli2010_explain]} is the reference if a reviewer pushes back on doing one and reporting the other. **Say which one you are doing, in the sentence that introduces the model.** "We fit a logistic regression of //banner present// on //rank, category and country// to estimate the association between …" and "we train a logistic-regression classifier as a baseline against our detector" are two different sentences, and a reader cannot tell them apart from a table of coefficients. Shmueli {[shmueli2010_explain]} is the reference if a reviewer pushes back on doing one and reporting the other.
-</wrap>+</WRAP>
  
 The rest of this page is about the first activity. Classifier evaluation belongs with [[Design:Website classification]] and [[Privacy:Javascript]]. The rest of this page is about the first activity. Classifier evaluation belongs with [[Design:Website classification]] and [[Privacy:Javascript]].
Line 60: Line 60:
  
 The specific failure is **overdispersion**. A Poisson regression assumes variance equals mean. Real tracker counts have variance many times their mean, and when that assumption breaks the model does not merely fit badly — it reports standard errors that are far too small, and rejects true nulls at a rate nothing like its nominal α. The specific failure is **overdispersion**. A Poisson regression assumes variance equals mean. Real tracker counts have variance many times their mean, and when that assumption breaks the model does not merely fit badly — it reports standard errors that are far too small, and rejects true nulls at a rate nothing like its nominal α.
 +
 +**Compute your own variance-to-mean ratio before you choose the model, and put it in the paper.** It is one line (''y.var() / y.mean()''), it decides the question, and **no paper in this corpus reports one** — which is why the ratio of 13 used below is a plausible guess rather than a measured fact about the web.
  
 Part A of the script below measures it. Both groups of sites are drawn from the **same** distribution, so every rejection is a false positive: Part A of the script below measures it. Both groups of sites are drawn from the **same** distribution, so every rejection is a false positive:
Line 292: Line 294:
   * Vu et al. {[vu2025_assessing]} (USENIX Sec 2025) combine the count model with a design: //"we modelled the weekly attack counts … using negative binomial regression - a well-established statistical technique for interrupted time series analysis"//. Weekly counts with a trend, seasonality and an intervention term is the shape of most "did the deadline change anything" questions in this field.   * Vu et al. {[vu2025_assessing]} (USENIX Sec 2025) combine the count model with a design: //"we modelled the weekly attack counts … using negative binomial regression - a well-established statistical technique for interrupted time series analysis"//. Weekly counts with a trend, seasonality and an intervention term is the shape of most "did the deadline change anything" questions in this field.
  
-<wrap todo>+<WRAP todo>
 **If your count is "how many //X// out of an opportunity that varies", you need an offset, not a ratio.** Regressing //trackers per request// or //predatory ads as a share of ads// on covariates throws away the fact that a site with 300 requests and a site with 12 give you very different amounts of information. The count model's answer is a **log offset**: put the exposure on the right-hand side with its coefficient fixed at 1. Mai et al. {[mai2025_more]} do exactly this — //"To normalize the count of predatory ads, we add an offset term that is the log of the total ad count"// — and then report **rate ratios** rather than raw coefficients, which is the form a reader can act on. One paper in 5,859 does this.((Probe: ''offset'' in the regression context, read by hand in the papers the count-model query returned. A paper could use an offset without writing the word; this is a floor.)) **If your count is "how many //X// out of an opportunity that varies", you need an offset, not a ratio.** Regressing //trackers per request// or //predatory ads as a share of ads// on covariates throws away the fact that a site with 300 requests and a site with 12 give you very different amounts of information. The count model's answer is a **log offset**: put the exposure on the right-hand side with its coefficient fixed at 1. Mai et al. {[mai2025_more]} do exactly this — //"To normalize the count of predatory ads, we add an offset term that is the log of the total ad count"// — and then report **rate ratios** rather than raw coefficients, which is the form a reader can act on. One paper in 5,859 does this.((Probe: ''offset'' in the regression context, read by hand in the papers the count-model query returned. A paper could use an offset without writing the word; this is a floor.))
-</wrap>+</WRAP>
  
 ==== Binary per-site outcomes ==== ==== Binary per-site outcomes ====
Line 308: Line 310:
 **Your sample size is the number of //events//, not the number of sites.** A crawl of 100,000 sites where 400 show the behaviour you are modelling gives a logistic regression 400 events, and the usual guidance is on the order of ten events per predictor — relaxable, but knowingly {[vittinghoff2007_relaxing]}. This is the failure mode of the rare-phenomenon paper: an enormous //n// and an unstable model. **Your sample size is the number of //events//, not the number of sites.** A crawl of 100,000 sites where 400 show the behaviour you are modelling gives a logistic regression 400 events, and the usual guidance is on the order of ten events per predictor — relaxable, but knowingly {[vittinghoff2007_relaxing]}. This is the failure mode of the rare-phenomenon paper: an enormous //n// and an unstable model.
  
-<wrap todo> +<WRAP todo> 
-**A pseudo-R² of 0.07 is not a broken model, and saying so is not a weakness.** Kieserman et al. {[kieserman2025_tracker]} fit logistic regressions on 40,150 websites to explain which ones configure Google and Meta trackers to harvest form data, and write: //"It is important to acknowledge that our models are a relatively weak fit; the former has a pseudo R-squared of 0.0721"// … //"This means that their explanatory power is limited. However, we can still draw some useful insights from them"//. They report odds ratios, //p//-values and confidence intervals per feature. That is the model to copy: an inferential regression is not judged on fit, and pretending otherwise is what leads people to report a classifier's AUC in place of a coefficient. +**A pseudo-R² of 0.07 is not a broken model, and saying so is not a weakness.** Kieserman et al. {[kieserman2025_tracker]} fit logistic regressions on 40,150 websites to explain which ones configure Google and Meta trackers to harvest form data, and write: //"It is important to acknowledge that our models are a relatively weak fit; the former has a pseudo R-squared of 0.0721"// … //"This means that their explanatory power is limited. However, we can still draw some useful insights from them"//. They report odds ratios, //p//-values and confidence intervals per feature. That is the model to copy. An inferential regression is not //primarily// judged on fit — a low pseudo-R² says the outcome has other causes, not that the coefficient is wrong — and pretending otherwise is what leads people to report a classifier's AUC in place of a coefficient. (Fit still matters when a reviewer's objection is an omitted variable; that is an argument about which covariates are in the model, not about R².) 
-</wrap>+</WRAP>
  
 ==== Proportions, rates and ordinal scales ==== ==== Proportions, rates and ordinal scales ====
  
   * **A proportion bounded in (0,1)** — share of a site's requests going to trackers, share of a page's cookies that are third-party — is not Gaussian and its variance shrinks at both ends. **Beta regression** {[ferrari2004_beta]} is the model for it; **3 papers in 5,859 use one**, all of them on social-media outcomes rather than crawl outcomes. If the proportion is a count over a known denominator, a binomial or count-with-offset model is usually better than beta regression, because it keeps the denominator.   * **A proportion bounded in (0,1)** — share of a site's requests going to trackers, share of a page's cookies that are third-party — is not Gaussian and its variance shrinks at both ends. **Beta regression** {[ferrari2004_beta]} is the model for it; **3 papers in 5,859 use one**, all of them on social-media outcomes rather than crawl outcomes. If the proportion is a count over a known denominator, a binomial or count-with-offset model is usually better than beta regression, because it keeps the denominator.
-  * **An ordinal outcome** — a Likert response, a severity band, a consent-banner "darkness" grade — should be an ordered logit or a cumulative link model, and **46 papers (11.8%) do this**, almost all of them user studies. Cumulative link mixed models (''ordinal::clmm'' in R) are the standard combination when the same participant answers several items.+  * **An ordinal outcome** — a severity band, a consent-banner "darkness" grade, a compliance level — is an ordered logit or a cumulative link model, and **46 papers (11.8%) do this**. Almost all of them are user studies scoring Likert items, so the practice and its tooling live on [[Design:User studies]]; what is worth carrying over is that a crawl's ordinal outcomes get treated as numbers far more casually than a survey's do.
   * **Time-to-event** — how long until a domain is taken down, until a vulnerability is patched, until a tracker reappears — is a Cox model, used by **7 papers (1.8%)**. If your longitudinal crawl ends before some sites have had the event, that is right-censoring, and dropping those sites or coding them as "never" both bias the answer.   * **Time-to-event** — how long until a domain is taken down, until a vulnerability is patched, until a tracker reappears — is a Cox model, used by **7 papers (1.8%)**. If your longitudinal crawl ends before some sites have had the event, that is right-censoring, and dropping those sites or coding them as "never" both bias the answer.
  
Line 403: Line 405:
 **This is a different failure from a multiplicity problem and from a wrong test.** No correction on [[Statistics:Pvalue corrections|P-value corrections]] repairs a standard error computed from the wrong number of independent units, and no rank test on [[Statistics:Hypothesis testing]] does either. A paper can have all three. **This is a different failure from a multiplicity problem and from a wrong test.** No correction on [[Statistics:Pvalue corrections|P-value corrections]] repairs a standard error computed from the wrong number of independent units, and no rank test on [[Statistics:Hypothesis testing]] does either. A paper can have all three.
 </WRAP> </WRAP>
 +
 +==== What to cluster on when there is only one crawl ====
 +
 +The panel case has an obvious clustering unit: the site. A **single-wave** crawl does not, and it is the commonest design in this corpus — which makes "what do I cluster on?" the question this page is most likely to be asked and the one the literature gives least help with. **No paper in the corpus answers it.** The two crawl papers that cluster at all cluster on the product {[becerrilarreola2023_method]} and on the website across snapshots {[nenadic2026_swiss]} — and the second is a panel, so its unit is the repeated observation rather than a structural group. Nobody has clustered a single-wave crawl on anything.
 +
 +What you have to choose between, in rough order of how much dependence each induces and how easily you can observe it:
 +
 +^ Candidate cluster ^ Why sites in it move together ^ How you get it ^
 +| **the operator / owning entity** | one company's fifty domains are one decision about trackers | WHOIS, a public entity mapping, or the eTLD+1 of a shared login — see [[Design:Website classification]] |
 +| **the consent platform** | a CMP's default template decides banner text, categories and pre-ticked boxes for every customer | detectable from the crawl itself; see [[Privacy:Consent]] |
 +| **the tag manager container** | one GTM container ships the same third parties to everything it is installed on | detectable from the request graph; see [[Privacy:Requests]] |
 +| **the CMS or site builder** | a Wix/Shopify/WordPress-plus-plugin cohort behaves as one configuration | fingerprintable from headers and paths |
 +| **hosting / CDN / IP block** | one provider's configuration correlates rows that look unrelated | see [[Design:IP classification]] |
 +| **the country or language of the site** | a whole legal regime, and often one national CMP vendor | see [[Design:Crawling location]] and [[Programming:Multilingual support]] |
 +
 +Three practical points, none of which the corpus can settle for you:
 +
 +  * **Cluster on the coarsest level you think the dependence lives at**, not on the one that is easiest to compute. Clustering too finely leaves the dependence in; clustering too coarsely costs precision but is not wrong.
 +  * **You cannot cluster on two crossed dimensions with ''cov_type="cluster"''.** If you believe both the CMP and the hosting provider matter, that is two-way clustering, and ''statsmodels'' does not do it — MacKinnon et al. {[mackinnon2023_clusterrobust]} cover the estimator, and in practice people fall back to the coarser of the two.
 +  * **Say which one you chose and why, even if the choice is arguable.** A stated clustering unit a reviewer disagrees with is a conversation; an unstated one is the 79.5%.
  
 ==== Four remedies, and which to pick ==== ==== Four remedies, and which to pick ====
  
 ^ Remedy ^ What it estimates ^ When it is the right one ^ Cost ^ ^ Remedy ^ What it estimates ^ When it is the right one ^ Cost ^
-| **Cluster-robust standard errors**, clustered on the site | the same coefficients; only the standard errors change | the default. You want a population-average effect and you have enough clusters | needs "enough" clusters — the usual rule of thumb is a few dozen at minimum, and web crawls have thousands {[cameron2015_practitioners]}. But **cluster //size// imbalance is its own failure mode**, and a crawl is the worst case: one tag manager on 5,000 sites and another on three. Huang {[huang2025_clusterfail]} shows the CR0 sandwich estimator misbehaves badly under that imbalance and recommends the CR2 small-sample correction; MacKinnon et al. {[mackinnon2023_clusterrobust]} recommend the wild cluster bootstrap |+| **Cluster-robust standard errors**, clustered on the site | the same coefficients; only the standard errors change | the default. You want a population-average effect and you have enough clusters | needs "enough" clusters — the usual rule of thumb is a few dozen at minimum, and web crawls have thousands {[cameron2015_practitioners]}. But **cluster //size// imbalance is its own failure mode**, and a crawl is the worst case: one tag manager on 5,000 sites and another on three. Huang {[huang2026_clusterfail]} shows the CR0 sandwich estimator misbehaves badly under that imbalance and recommends the CR2 small-sample correction; MacKinnon et al. {[mackinnon2023_clusterrobust]} recommend the wild cluster bootstrap |
 | **Random intercept for the site** (mixed model / GLMM) | a **site-specific** effect: how the outcome changes for a given site | you want to describe between-site variance itself, or you have few observations per site and want to borrow strength | for a logistic model the coefficient is no longer the population-average one, and is not comparable with a cluster-robust fit {[mood2010_logistic]} | | **Random intercept for the site** (mixed model / GLMM) | a **site-specific** effect: how the outcome changes for a given site | you want to describe between-site variance itself, or you have few observations per site and want to borrow strength | for a logistic model the coefficient is no longer the population-average one, and is not comparable with a cluster-robust fit {[mood2010_logistic]} |
 | **Fixed effects for the site** (a dummy per site) | a purely **within-site** effect | before/after with a within-site predictor and no interest in between-site comparisons | eats all site-level predictors — you cannot estimate the effect of "is a news site" | | **Fixed effects for the site** (a dummy per site) | a purely **within-site** effect | before/after with a within-site predictor and no interest in between-site comparisons | eats all site-level predictors — you cannot estimate the effect of "is a news site" |
-| **GEE with an exchangeable working correlation** | a population-average effect, like cluster-robust | you want the marginal effect and are willing to model the correlation structure | 2 papers in this corpus use one; reviewers in this field will not recognise it |+| **GEE with an exchangeable working correlation** | a population-average effect, like cluster-robust | you want the marginal effect and are willing to model the correlation structure | 2 papers in this corpus use one, so expect to have to explain it |
  
-For a crawl the honest default is **cluster-robust standard errors clustered on the site**, because it is one argument, it does not change the estimand, and thousands of clusters is exactly the regime where it works. Say //which// correction you used: ''statsmodels'''s ''cov_type="cluster"'' gives you the CR0 sandwich, which is the one that degrades under unequal cluster sizes {[huang2025_clusterfail]}. Reach for a mixed model when the between-site variance is itself the result — "how much of the variation in tracker count is between sites rather than within them" is a mixed model's question and nothing else's. Barr et al. {[barr2013_maximal]} on which random effects to include; Moineddin et al. {[moineddin2007_simulation]} on why a mixed model with few observations per group is less well-behaved than its output suggests.+For a crawl the honest default is **cluster-robust standard errors clustered on the site**, because it is one argument, it does not change the estimand, and thousands of clusters is exactly the regime where it works. Say //which// correction you used: ''statsmodels'''s ''cov_type="cluster"'' gives you the CR0 sandwich, which is the one that degrades under unequal cluster sizes {[huang2026_clusterfail]}. Reach for a mixed model when the between-site variance is itself the result — "how much of the variation in tracker count is between sites rather than within them" is a mixed model's question and nothing else's. Barr et al. {[barr2013_maximal]} on which random effects to include; Moineddin et al. {[moineddin2007_simulation]} on why a mixed model with few observations per group is less well-behaved than its output suggests.
  
 ==== What a good one looks like ==== ==== What a good one looks like ====
Line 419: Line 441:
  
 That is five decisions a crawl paper has to make, made explicitly. It is also, as far as this corpus goes, the **only** one that makes all five. That is five decisions a crawl paper has to make, made explicitly. It is also, as far as this corpus goes, the **only** one that makes all five.
 +
 +Note the word **balanced**. Nenadić et al. analyse the 11,800 sites present in both snapshots, which is the clean case and not the usual one: a re-crawl six months later loses sites to expiry, blocking, redirection and rewritten pages. Dropping the lost sites silently turns your panel into a survivorship sample of the sites that stayed reachable, which is precisely the ones least likely to have changed. Report the attrition — how many sites entered, how many survived, and whether the survivors differ on the covariates you care about — before reporting the model. [[Statistics:Hypothesis testing#Attrition: the pairs you lose between the two crawls|The hypothesis-testing page treats attrition for paired tests]] and the argument is the same one; an unbalanced panel additionally means a mixed model and a fixed-effects model no longer answer quite the same question, because the fixed-effects fit discards every site seen once.
  
 Two 2026 near-neighbours are worth reading alongside it, both on platform data rather than crawls: Chuai et al. {[chuai2026_consensus]} fit a //"multilevel mixed-effects logistic regression"// with author- and post-level random effects and separately //"cluster robust standard errors at the note level to account for potential within-note correlations over time"//; Bobek et al. {[bobek2026_community]} run a staggered difference-in-differences with Callaway–Sant'Anna group-time treatment effects {[callaway2021_did]}, propensity-score weighting, a pre-trends test and bootstrap simultaneous confidence bands. Two 2026 near-neighbours are worth reading alongside it, both on platform data rather than crawls: Chuai et al. {[chuai2026_consensus]} fit a //"multilevel mixed-effects logistic regression"// with author- and post-level random effects and separately //"cluster robust standard errors at the note level to account for potential within-note correlations over time"//; Bobek et al. {[bobek2026_community]} run a staggered difference-in-differences with Callaway–Sant'Anna group-time treatment effects {[callaway2021_did]}, propensity-score weighting, a pre-trends test and bootstrap simultaneous confidence bands.
Line 491: Line 515:
 ==== Dependence and identification ==== ==== Dependence and identification ====
  
-^ Dependence handling ^ Papers ^ Share of 391 ^ +The dependence table is [[#What the corpus does about it|above]] and is not repeated here. Two notes on how to read it.
-| none stated in the method name | 311 | 79.5% | +
-| random effects (mixed-effects / multilevel) | 75 | 19.2% | +
-| robust (heteroskedasticity-consistent) SEs only | 3 | 0.8% | +
-| fixed effects for the unit | 3 | 0.8% | +
-| GEE (population-averaged) | 2 | 0.5% | +
-| "repeated-measures" with no random effect named | 1 | 0.3% | +
-| cluster-robust standard errors | 1 | 0.3% |+
  
 ''cluster-robust'' and ''robust'' are separated on purpose. Heteroskedasticity-consistent (HC / White / Huber) standard errors fix a variance assumption; they do nothing whatever about correlated rows. Merging the two rows would let three papers that hardened against heteroskedasticity read as papers that handled dependence. ''cluster-robust'' and ''robust'' are separated on purpose. Heteroskedasticity-consistent (HC / White / Huber) standard errors fix a variance assumption; they do nothing whatever about correlated rows. Merging the two rows would let three papers that hardened against heteroskedasticity read as papers that handled dependence.
  
 The fold reads the **method string**, so this row is a floor: Chuai et al. {[chuai2026_consensus]} appear here under "robust" because that is what their regression tuples say, while the paper separately states that it clusters at the note level. The full-text probe below is the check on that, and it finds four clustering papers where the fold finds one. The fold reads the **method string**, so this row is a floor: Chuai et al. {[chuai2026_consensus]} appear here under "robust" because that is what their regression tuples say, while the paper separately states that it clusters at the note level. The full-text probe below is the check on that, and it finds four clustering papers where the fold finds one.
 +
 +Rows in both tables below overlap and neither sums to 391: **19 of the 391 have no tuple the fold could place as a model at all**, so they get no dependence or identification value, and a paper fitting two models can appear in two rows. The 372 papers that do have a placeable model produce 396 axis-2 and 377 axis-3 row memberships.
  
 ^ Identification strategy ^ Papers ^ Share of 391 ^ ^ Identification strategy ^ Papers ^ Share of 391 ^
Line 534: Line 553:
 | cluster-robust standard errors | 1 | 1.6% | | cluster-robust standard errors | 1 | 1.6% |
 | fixed effects | 1 | 1.6% | | fixed effects | 1 | 1.6% |
 +
 +The dependence rows sum to 61, not 62: two of the 62 have no placeable model and one of those has no dependence value either.
  
 Crawl papers are **more** likely to fit a count model than the regression population as a whole (11.3% against 4.3%), which is what you would expect given what they measure, and **less** likely to do anything about dependence (87.1% silent against 79.5%), which is the opposite of what the data structure warrants. **Two crawl papers in 1,120 cluster their standard errors** — Becerril-Arreola {[becerrilarreola2023_method]}, who says so in the method the extraction captured, and Nenadić et al. {[nenadic2026_swiss]}, who say so only in prose, which is why this table counts one and the full-text probe below counts two. Crawl papers are **more** likely to fit a count model than the regression population as a whole (11.3% against 4.3%), which is what you would expect given what they measure, and **less** likely to do anything about dependence (87.1% silent against 79.5%), which is the opposite of what the data structure warrants. **Two crawl papers in 1,120 cluster their standard errors** — Becerril-Arreola {[becerrilarreola2023_method]}, who says so in the method the extraction captured, and Nenadić et al. {[nenadic2026_swiss]}, who say so only in prose, which is why this table counts one and the full-text probe below counts two.
Line 597: Line 618:
 ===== Open Questions ===== ===== Open Questions =====
  
 +<WRAP todo>
   * **Nobody has estimated the intra-class correlation of a web-measurement outcome.** The simulation on this page shows the entire size of the error depends on it, and no paper in this corpus reports one. "What is the ICC of //sets a third-party cookie before consent// by consent platform, by CMS, by hosting provider, and by the same site across monthly crawls?" is a small, self-contained study that would tell every crawl paper how wrong its standard errors are. The same gap is open on [[Statistics:Hypothesis testing]] and it is one study away from being closed for both.   * **Nobody has estimated the intra-class correlation of a web-measurement outcome.** The simulation on this page shows the entire size of the error depends on it, and no paper in this corpus reports one. "What is the ICC of //sets a third-party cookie before consent// by consent platform, by CMS, by hosting provider, and by the same site across monthly crawls?" is a small, self-contained study that would tell every crawl paper how wrong its standard errors are. The same gap is open on [[Statistics:Hypothesis testing]] and it is one study away from being closed for both.
   * **No crawl paper models the site as a random effect.** One non-crawl field study does {[zeng2022_factors]}, so the technique is not unknown in this community — it has just never been pointed at a repeated crawl. Whether that is because cluster-robust standard errors are the better tool anyway (they usually are), or because nobody has tried, is not answerable from text. Somebody should re-analyse a published longitudinal crawl both ways and report how much moves.   * **No crawl paper models the site as a random effect.** One non-crawl field study does {[zeng2022_factors]}, so the technique is not unknown in this community — it has just never been pointed at a repeated crawl. Whether that is because cluster-robust standard errors are the better tool anyway (they usually are), or because nobody has tried, is not answerable from text. Somebody should re-analyse a published longitudinal crawl both ways and report how much moves.
Line 603: Line 625:
   * **The 2022-onward arrival of causal designs is ten papers.** Whether it is a durable turn or a handful of authors with economics training is not decidable yet, and the 2025–2026 evidence for it sits in the provisional years.   * **The 2022-onward arrival of causal designs is ten papers.** Whether it is a durable turn or a handful of authors with economics training is not decidable yet, and the 2025–2026 evidence for it sits in the provisional years.
   * **None of the seven venues asks for any of this.** As on [[Statistics:Hypothesis testing]], the calls for papers contain no statistical-reporting requirement, so the only thing holding a regression section to a standard is an individual reviewer.   * **None of the seven venues asks for any of this.** As on [[Statistics:Hypothesis testing]], the calls for papers contain no statistical-reporting requirement, so the only thing holding a regression section to a standard is an individual reviewer.
 +</WRAP>
 ===== Related Pages ===== ===== Related Pages =====
  
statistics/regression.1787163267.txt.gz · Last modified: by karel.kubicek.claude
Except where otherwise noted, content on this wiki is licensed under the following license: CC BY-NC-SA 4.0
CC BY-NC-SA 4.0 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki