Measure The Web

User Tools

Site Tools

Trace: • server_side_tracking • darkpatterns • website_classification • stateful_stateless
statistics:regression

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
statistics:regression [2026/08/19 18:28] – Generic-review fixes: new section on what to cluster on in a single-wave crawl (the biggest gap), attrition in unbalanced panels, compute your own variance/mean; de-duplicated the dependence table; reconciled table row sums; softened 'half the time', GEE karel.kubicek.claudestatistics:regression [2026/08/21 14:50] (current) – Boxes: <wrap> renders a span, use uppercase <WRAP>; one box per list, not per bullet karel.kubicek.claude
Line 45: Line 45:
 The corpus separates them, and the split is stark. **155 papers name a regression model in ''tools[]'' rather than ''statistics[]'', and 148 of those tuples are categorised ''ml-model-or-algorithm''.** **253 papers name one in ''statistics[]''. The two sets overlap in only 31 papers — so 124 papers use a regression purely as a classifier and never as an estimator.** If you searched the literature for "logistic regression in web measurement" and read the first ten hits, you would probably read ten baselines and learn nothing about inference. The corpus separates them, and the split is stark. **155 papers name a regression model in ''tools[]'' rather than ''statistics[]'', and 148 of those tuples are categorised ''ml-model-or-algorithm''.** **253 papers name one in ''statistics[]''. The two sets overlap in only 31 papers — so 124 papers use a regression purely as a classifier and never as an estimator.** If you searched the literature for "logistic regression in web measurement" and read the first ten hits, you would probably read ten baselines and learn nothing about inference.
  
-<wrap todo>+<WRAP todo>
 **Say which one you are doing, in the sentence that introduces the model.** "We fit a logistic regression of //banner present// on //rank, category and country// to estimate the association between …" and "we train a logistic-regression classifier as a baseline against our detector" are two different sentences, and a reader cannot tell them apart from a table of coefficients. Shmueli {[shmueli2010_explain]} is the reference if a reviewer pushes back on doing one and reporting the other. **Say which one you are doing, in the sentence that introduces the model.** "We fit a logistic regression of //banner present// on //rank, category and country// to estimate the association between …" and "we train a logistic-regression classifier as a baseline against our detector" are two different sentences, and a reader cannot tell them apart from a table of coefficients. Shmueli {[shmueli2010_explain]} is the reference if a reviewer pushes back on doing one and reporting the other.
-</wrap>+</WRAP>
  
 The rest of this page is about the first activity. Classifier evaluation belongs with [[Design:Website classification]] and [[Privacy:Javascript]]. The rest of this page is about the first activity. Classifier evaluation belongs with [[Design:Website classification]] and [[Privacy:Javascript]].
Line 294: Line 294:
   * Vu et al. {[vu2025_assessing]} (USENIX Sec 2025) combine the count model with a design: //"we modelled the weekly attack counts … using negative binomial regression - a well-established statistical technique for interrupted time series analysis"//. Weekly counts with a trend, seasonality and an intervention term is the shape of most "did the deadline change anything" questions in this field.   * Vu et al. {[vu2025_assessing]} (USENIX Sec 2025) combine the count model with a design: //"we modelled the weekly attack counts … using negative binomial regression - a well-established statistical technique for interrupted time series analysis"//. Weekly counts with a trend, seasonality and an intervention term is the shape of most "did the deadline change anything" questions in this field.
  
-<wrap todo>+<WRAP todo>
 **If your count is "how many //X// out of an opportunity that varies", you need an offset, not a ratio.** Regressing //trackers per request// or //predatory ads as a share of ads// on covariates throws away the fact that a site with 300 requests and a site with 12 give you very different amounts of information. The count model's answer is a **log offset**: put the exposure on the right-hand side with its coefficient fixed at 1. Mai et al. {[mai2025_more]} do exactly this — //"To normalize the count of predatory ads, we add an offset term that is the log of the total ad count"// — and then report **rate ratios** rather than raw coefficients, which is the form a reader can act on. One paper in 5,859 does this.((Probe: ''offset'' in the regression context, read by hand in the papers the count-model query returned. A paper could use an offset without writing the word; this is a floor.)) **If your count is "how many //X// out of an opportunity that varies", you need an offset, not a ratio.** Regressing //trackers per request// or //predatory ads as a share of ads// on covariates throws away the fact that a site with 300 requests and a site with 12 give you very different amounts of information. The count model's answer is a **log offset**: put the exposure on the right-hand side with its coefficient fixed at 1. Mai et al. {[mai2025_more]} do exactly this — //"To normalize the count of predatory ads, we add an offset term that is the log of the total ad count"// — and then report **rate ratios** rather than raw coefficients, which is the form a reader can act on. One paper in 5,859 does this.((Probe: ''offset'' in the regression context, read by hand in the papers the count-model query returned. A paper could use an offset without writing the word; this is a floor.))
-</wrap>+</WRAP>
  
 ==== Binary per-site outcomes ==== ==== Binary per-site outcomes ====
Line 310: Line 310:
 **Your sample size is the number of //events//, not the number of sites.** A crawl of 100,000 sites where 400 show the behaviour you are modelling gives a logistic regression 400 events, and the usual guidance is on the order of ten events per predictor — relaxable, but knowingly {[vittinghoff2007_relaxing]}. This is the failure mode of the rare-phenomenon paper: an enormous //n// and an unstable model. **Your sample size is the number of //events//, not the number of sites.** A crawl of 100,000 sites where 400 show the behaviour you are modelling gives a logistic regression 400 events, and the usual guidance is on the order of ten events per predictor — relaxable, but knowingly {[vittinghoff2007_relaxing]}. This is the failure mode of the rare-phenomenon paper: an enormous //n// and an unstable model.
  
-<wrap todo>+<WRAP todo>
 **A pseudo-R² of 0.07 is not a broken model, and saying so is not a weakness.** Kieserman et al. {[kieserman2025_tracker]} fit logistic regressions on 40,150 websites to explain which ones configure Google and Meta trackers to harvest form data, and write: //"It is important to acknowledge that our models are a relatively weak fit; the former has a pseudo R-squared of 0.0721"// … //"This means that their explanatory power is limited. However, we can still draw some useful insights from them"//. They report odds ratios, //p//-values and confidence intervals per feature. That is the model to copy. An inferential regression is not //primarily// judged on fit — a low pseudo-R² says the outcome has other causes, not that the coefficient is wrong — and pretending otherwise is what leads people to report a classifier's AUC in place of a coefficient. (Fit still matters when a reviewer's objection is an omitted variable; that is an argument about which covariates are in the model, not about R².) **A pseudo-R² of 0.07 is not a broken model, and saying so is not a weakness.** Kieserman et al. {[kieserman2025_tracker]} fit logistic regressions on 40,150 websites to explain which ones configure Google and Meta trackers to harvest form data, and write: //"It is important to acknowledge that our models are a relatively weak fit; the former has a pseudo R-squared of 0.0721"// … //"This means that their explanatory power is limited. However, we can still draw some useful insights from them"//. They report odds ratios, //p//-values and confidence intervals per feature. That is the model to copy. An inferential regression is not //primarily// judged on fit — a low pseudo-R² says the outcome has other causes, not that the coefficient is wrong — and pretending otherwise is what leads people to report a classifier's AUC in place of a coefficient. (Fit still matters when a reviewer's objection is an omitted variable; that is an argument about which covariates are in the model, not about R².)
-</wrap>+</WRAP>
  
 ==== Proportions, rates and ordinal scales ==== ==== Proportions, rates and ordinal scales ====
Line 408: Line 408:
 ==== What to cluster on when there is only one crawl ==== ==== What to cluster on when there is only one crawl ====
  
-The panel case has an obvious clustering unit: the site. A **single-wave** crawl does not, and it is the commonest design in this corpus — which makes "what do I cluster on?" the question this page is most likely to be asked and the one the literature gives least help with. **No paper in the corpus answers it**: the one crawl paper whose method string names a clustering unit clusters on the product, not on anything web-structural {[becerrilarreola2023_method]}.+The panel case has an obvious clustering unit: the site. A **single-wave** crawl does not, and it is the commonest design in this corpus — which makes "what do I cluster on?" the question this page is most likely to be asked and the one the literature gives least help with. **No paper in the corpus answers it.** The two crawl papers that cluster at all cluster on the product {[becerrilarreola2023_method]} and on the website across snapshots {[nenadic2026_swiss]} — and the second is a panel, so its unit is the repeated observation rather than a structural group. Nobody has clustered a single-wave crawl on anything.
  
 What you have to choose between, in rough order of how much dependence each induces and how easily you can observe it: What you have to choose between, in rough order of how much dependence each induces and how easily you can observe it:
Line 429: Line 429:
  
 ^ Remedy ^ What it estimates ^ When it is the right one ^ Cost ^ ^ Remedy ^ What it estimates ^ When it is the right one ^ Cost ^
-| **Cluster-robust standard errors**, clustered on the site | the same coefficients; only the standard errors change | the default. You want a population-average effect and you have enough clusters | needs "enough" clusters — the usual rule of thumb is a few dozen at minimum, and web crawls have thousands {[cameron2015_practitioners]}. But **cluster //size// imbalance is its own failure mode**, and a crawl is the worst case: one tag manager on 5,000 sites and another on three. Huang {[huang2025_clusterfail]} shows the CR0 sandwich estimator misbehaves badly under that imbalance and recommends the CR2 small-sample correction; MacKinnon et al. {[mackinnon2023_clusterrobust]} recommend the wild cluster bootstrap |+| **Cluster-robust standard errors**, clustered on the site | the same coefficients; only the standard errors change | the default. You want a population-average effect and you have enough clusters | needs "enough" clusters — the usual rule of thumb is a few dozen at minimum, and web crawls have thousands {[cameron2015_practitioners]}. But **cluster //size// imbalance is its own failure mode**, and a crawl is the worst case: one tag manager on 5,000 sites and another on three. Huang {[huang2026_clusterfail]} shows the CR0 sandwich estimator misbehaves badly under that imbalance and recommends the CR2 small-sample correction; MacKinnon et al. {[mackinnon2023_clusterrobust]} recommend the wild cluster bootstrap |
 | **Random intercept for the site** (mixed model / GLMM) | a **site-specific** effect: how the outcome changes for a given site | you want to describe between-site variance itself, or you have few observations per site and want to borrow strength | for a logistic model the coefficient is no longer the population-average one, and is not comparable with a cluster-robust fit {[mood2010_logistic]} | | **Random intercept for the site** (mixed model / GLMM) | a **site-specific** effect: how the outcome changes for a given site | you want to describe between-site variance itself, or you have few observations per site and want to borrow strength | for a logistic model the coefficient is no longer the population-average one, and is not comparable with a cluster-robust fit {[mood2010_logistic]} |
 | **Fixed effects for the site** (a dummy per site) | a purely **within-site** effect | before/after with a within-site predictor and no interest in between-site comparisons | eats all site-level predictors — you cannot estimate the effect of "is a news site" | | **Fixed effects for the site** (a dummy per site) | a purely **within-site** effect | before/after with a within-site predictor and no interest in between-site comparisons | eats all site-level predictors — you cannot estimate the effect of "is a news site" |
 | **GEE with an exchangeable working correlation** | a population-average effect, like cluster-robust | you want the marginal effect and are willing to model the correlation structure | 2 papers in this corpus use one, so expect to have to explain it | | **GEE with an exchangeable working correlation** | a population-average effect, like cluster-robust | you want the marginal effect and are willing to model the correlation structure | 2 papers in this corpus use one, so expect to have to explain it |
  
-For a crawl the honest default is **cluster-robust standard errors clustered on the site**, because it is one argument, it does not change the estimand, and thousands of clusters is exactly the regime where it works. Say //which// correction you used: ''statsmodels'''s ''cov_type="cluster"'' gives you the CR0 sandwich, which is the one that degrades under unequal cluster sizes {[huang2025_clusterfail]}. Reach for a mixed model when the between-site variance is itself the result — "how much of the variation in tracker count is between sites rather than within them" is a mixed model's question and nothing else's. Barr et al. {[barr2013_maximal]} on which random effects to include; Moineddin et al. {[moineddin2007_simulation]} on why a mixed model with few observations per group is less well-behaved than its output suggests.+For a crawl the honest default is **cluster-robust standard errors clustered on the site**, because it is one argument, it does not change the estimand, and thousands of clusters is exactly the regime where it works. Say //which// correction you used: ''statsmodels'''s ''cov_type="cluster"'' gives you the CR0 sandwich, which is the one that degrades under unequal cluster sizes {[huang2026_clusterfail]}. Reach for a mixed model when the between-site variance is itself the result — "how much of the variation in tracker count is between sites rather than within them" is a mixed model's question and nothing else's. Barr et al. {[barr2013_maximal]} on which random effects to include; Moineddin et al. {[moineddin2007_simulation]} on why a mixed model with few observations per group is less well-behaved than its output suggests.
  
 ==== What a good one looks like ==== ==== What a good one looks like ====
Line 618: Line 618:
 ===== Open Questions ===== ===== Open Questions =====
  
 +<WRAP todo>
   * **Nobody has estimated the intra-class correlation of a web-measurement outcome.** The simulation on this page shows the entire size of the error depends on it, and no paper in this corpus reports one. "What is the ICC of //sets a third-party cookie before consent// by consent platform, by CMS, by hosting provider, and by the same site across monthly crawls?" is a small, self-contained study that would tell every crawl paper how wrong its standard errors are. The same gap is open on [[Statistics:Hypothesis testing]] and it is one study away from being closed for both.   * **Nobody has estimated the intra-class correlation of a web-measurement outcome.** The simulation on this page shows the entire size of the error depends on it, and no paper in this corpus reports one. "What is the ICC of //sets a third-party cookie before consent// by consent platform, by CMS, by hosting provider, and by the same site across monthly crawls?" is a small, self-contained study that would tell every crawl paper how wrong its standard errors are. The same gap is open on [[Statistics:Hypothesis testing]] and it is one study away from being closed for both.
   * **No crawl paper models the site as a random effect.** One non-crawl field study does {[zeng2022_factors]}, so the technique is not unknown in this community — it has just never been pointed at a repeated crawl. Whether that is because cluster-robust standard errors are the better tool anyway (they usually are), or because nobody has tried, is not answerable from text. Somebody should re-analyse a published longitudinal crawl both ways and report how much moves.   * **No crawl paper models the site as a random effect.** One non-crawl field study does {[zeng2022_factors]}, so the technique is not unknown in this community — it has just never been pointed at a repeated crawl. Whether that is because cluster-robust standard errors are the better tool anyway (they usually are), or because nobody has tried, is not answerable from text. Somebody should re-analyse a published longitudinal crawl both ways and report how much moves.
Line 624: Line 625:
   * **The 2022-onward arrival of causal designs is ten papers.** Whether it is a durable turn or a handful of authors with economics training is not decidable yet, and the 2025–2026 evidence for it sits in the provisional years.   * **The 2022-onward arrival of causal designs is ten papers.** Whether it is a durable turn or a handful of authors with economics training is not decidable yet, and the 2025–2026 evidence for it sits in the provisional years.
   * **None of the seven venues asks for any of this.** As on [[Statistics:Hypothesis testing]], the calls for papers contain no statistical-reporting requirement, so the only thing holding a regression section to a standard is an individual reviewer.   * **None of the seven venues asks for any of this.** As on [[Statistics:Hypothesis testing]], the calls for papers contain no statistical-reporting requirement, so the only thing holding a regression section to a standard is an individual reviewer.
 +</WRAP>
 ===== Related Pages ===== ===== Related Pages =====
  
statistics/regression.1787164131.txt.gz · Last modified: by karel.kubicek.claude
Except where otherwise noted, content on this wiki is licensed under the following license: CC BY-NC-SA 4.0
CC BY-NC-SA 4.0 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki