| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| statistics:regression [2026/08/19 18:31] – Rename huang citekey to huang2026_clusterfail (issue year). Authored by Claude karel.kubicek.claude | statistics:regression [2026/09/21 13:33] (current) – Quote-check bullet refreshed against quote_check.mjs --statistics regression with the pypdf fallback: 267/171/65/27 of 530. Authored by Claude karel.kubicek.claude |
|---|
| The corpus separates them, and the split is stark. **155 papers name a regression model in ''tools[]'' rather than ''statistics[]'', and 148 of those tuples are categorised ''ml-model-or-algorithm''.** **253 papers name one in ''statistics[]''. The two sets overlap in only 31 papers — so 124 papers use a regression purely as a classifier and never as an estimator.** If you searched the literature for "logistic regression in web measurement" and read the first ten hits, you would probably read ten baselines and learn nothing about inference. | The corpus separates them, and the split is stark. **155 papers name a regression model in ''tools[]'' rather than ''statistics[]'', and 148 of those tuples are categorised ''ml-model-or-algorithm''.** **253 papers name one in ''statistics[]''. The two sets overlap in only 31 papers — so 124 papers use a regression purely as a classifier and never as an estimator.** If you searched the literature for "logistic regression in web measurement" and read the first ten hits, you would probably read ten baselines and learn nothing about inference. |
| |
| <wrap todo> | <WRAP todo> |
| **Say which one you are doing, in the sentence that introduces the model.** "We fit a logistic regression of //banner present// on //rank, category and country// to estimate the association between …" and "we train a logistic-regression classifier as a baseline against our detector" are two different sentences, and a reader cannot tell them apart from a table of coefficients. Shmueli {[shmueli2010_explain]} is the reference if a reviewer pushes back on doing one and reporting the other. | **Say which one you are doing, in the sentence that introduces the model.** "We fit a logistic regression of //banner present// on //rank, category and country// to estimate the association between …" and "we train a logistic-regression classifier as a baseline against our detector" are two different sentences, and a reader cannot tell them apart from a table of coefficients. Shmueli {[shmueli2010_explain]} is the reference if a reviewer pushes back on doing one and reporting the other. |
| </wrap> | </WRAP> |
| |
| The rest of this page is about the first activity. Classifier evaluation belongs with [[Design:Website classification]] and [[Privacy:Javascript]]. | The rest of this page is about the first activity. Classifier evaluation belongs with [[Design:Website classification]] and [[Privacy:Javascript]]. |
| * Vu et al. {[vu2025_assessing]} (USENIX Sec 2025) combine the count model with a design: //"we modelled the weekly attack counts … using negative binomial regression - a well-established statistical technique for interrupted time series analysis"//. Weekly counts with a trend, seasonality and an intervention term is the shape of most "did the deadline change anything" questions in this field. | * Vu et al. {[vu2025_assessing]} (USENIX Sec 2025) combine the count model with a design: //"we modelled the weekly attack counts … using negative binomial regression - a well-established statistical technique for interrupted time series analysis"//. Weekly counts with a trend, seasonality and an intervention term is the shape of most "did the deadline change anything" questions in this field. |
| |
| <wrap todo> | <WRAP todo> |
| **If your count is "how many //X// out of an opportunity that varies", you need an offset, not a ratio.** Regressing //trackers per request// or //predatory ads as a share of ads// on covariates throws away the fact that a site with 300 requests and a site with 12 give you very different amounts of information. The count model's answer is a **log offset**: put the exposure on the right-hand side with its coefficient fixed at 1. Mai et al. {[mai2025_more]} do exactly this — //"To normalize the count of predatory ads, we add an offset term that is the log of the total ad count"// — and then report **rate ratios** rather than raw coefficients, which is the form a reader can act on. One paper in 5,859 does this.((Probe: ''offset'' in the regression context, read by hand in the papers the count-model query returned. A paper could use an offset without writing the word; this is a floor.)) | **If your count is "how many //X// out of an opportunity that varies", you need an offset, not a ratio.** Regressing //trackers per request// or //predatory ads as a share of ads// on covariates throws away the fact that a site with 300 requests and a site with 12 give you very different amounts of information. The count model's answer is a **log offset**: put the exposure on the right-hand side with its coefficient fixed at 1. Mai et al. {[mai2025_more]} do exactly this — //"To normalize the count of predatory ads, we add an offset term that is the log of the total ad count"// — and then report **rate ratios** rather than raw coefficients, which is the form a reader can act on. One paper in 5,859 does this.((Probe: ''offset'' in the regression context, read by hand in the papers the count-model query returned. A paper could use an offset without writing the word; this is a floor.)) |
| </wrap> | </WRAP> |
| |
| ==== Binary per-site outcomes ==== | ==== Binary per-site outcomes ==== |
| **Your sample size is the number of //events//, not the number of sites.** A crawl of 100,000 sites where 400 show the behaviour you are modelling gives a logistic regression 400 events, and the usual guidance is on the order of ten events per predictor — relaxable, but knowingly {[vittinghoff2007_relaxing]}. This is the failure mode of the rare-phenomenon paper: an enormous //n// and an unstable model. | **Your sample size is the number of //events//, not the number of sites.** A crawl of 100,000 sites where 400 show the behaviour you are modelling gives a logistic regression 400 events, and the usual guidance is on the order of ten events per predictor — relaxable, but knowingly {[vittinghoff2007_relaxing]}. This is the failure mode of the rare-phenomenon paper: an enormous //n// and an unstable model. |
| |
| <wrap todo> | <WRAP todo> |
| **A pseudo-R² of 0.07 is not a broken model, and saying so is not a weakness.** Kieserman et al. {[kieserman2025_tracker]} fit logistic regressions on 40,150 websites to explain which ones configure Google and Meta trackers to harvest form data, and write: //"It is important to acknowledge that our models are a relatively weak fit; the former has a pseudo R-squared of 0.0721"// … //"This means that their explanatory power is limited. However, we can still draw some useful insights from them"//. They report odds ratios, //p//-values and confidence intervals per feature. That is the model to copy. An inferential regression is not //primarily// judged on fit — a low pseudo-R² says the outcome has other causes, not that the coefficient is wrong — and pretending otherwise is what leads people to report a classifier's AUC in place of a coefficient. (Fit still matters when a reviewer's objection is an omitted variable; that is an argument about which covariates are in the model, not about R².) | **A pseudo-R² of 0.07 is not a broken model, and saying so is not a weakness.** Kieserman et al. {[kieserman2025_tracker]} fit logistic regressions on 40,150 websites to explain which ones configure Google and Meta trackers to harvest form data, and write: //"It is important to acknowledge that our models are a relatively weak fit; the former has a pseudo R-squared of 0.0721"// … //"This means that their explanatory power is limited. However, we can still draw some useful insights from them"//. They report odds ratios, //p//-values and confidence intervals per feature. That is the model to copy. An inferential regression is not //primarily// judged on fit — a low pseudo-R² says the outcome has other causes, not that the coefficient is wrong — and pretending otherwise is what leads people to report a classifier's AUC in place of a coefficient. (Fit still matters when a reviewer's objection is an omitted variable; that is an argument about which covariates are in the model, not about R².) |
| </wrap> | </WRAP> |
| |
| ==== Proportions, rates and ordinal scales ==== | ==== Proportions, rates and ordinal scales ==== |
| ==== What to cluster on when there is only one crawl ==== | ==== What to cluster on when there is only one crawl ==== |
| |
| The panel case has an obvious clustering unit: the site. A **single-wave** crawl does not, and it is the commonest design in this corpus — which makes "what do I cluster on?" the question this page is most likely to be asked and the one the literature gives least help with. **No paper in the corpus answers it**: the one crawl paper whose method string names a clustering unit clusters on the product, not on anything web-structural {[becerrilarreola2023_method]}. | The panel case has an obvious clustering unit: the site. A **single-wave** crawl does not, and it is the commonest design in this corpus — which makes "what do I cluster on?" the question this page is most likely to be asked and the one the literature gives least help with. **No paper in the corpus answers it.** The two crawl papers that cluster at all cluster on the product {[becerrilarreola2023_method]} and on the website across snapshots {[nenadic2026_swiss]} — and the second is a panel, so its unit is the repeated observation rather than a structural group. Nobody has clustered a single-wave crawl on anything. |
| |
| What you have to choose between, in rough order of how much dependence each induces and how easily you can observe it: | What you have to choose between, in rough order of how much dependence each induces and how easily you can observe it: |
| * **A paper counts once**, never once per mention. 530 tuples across 391 papers. | * **A paper counts once**, never once per mention. 530 tuples across 391 papers. |
| * **''statistics.kind'' is a mid-band extraction field.** Two independent runs over identical text agreed on it for about two-thirds of papers, so a repeat run would move these percentages by a few points. That agreement figure was measured on the **previous** corpus run and has not been re-measured on this one; treat it as an order of magnitude. | * **''statistics.kind'' is a mid-band extraction field.** Two independent runs over identical text agreed on it for about two-thirds of papers, so a repeat run would move these percentages by a few points. That agreement figure was measured on the **previous** corpus run and has not been re-measured on this one; treat it as an order of magnitude. |
| * **Quotes were checked in bulk and by hand.** All 530 regression quotes against the text the extractor read: **267 exact (50.4%) after whitespace normalisation, 170 partial (32.1%) at ≥60% of five-word windows, 93 (17.5%) below that**. Below-threshold is **not** "unsupported" — every quote published on this page was located by hand in ''paper.cols.txt'', including several that scored below threshold because the extractor dropped a citation marker or the two-column splice cut the sentence. Details, including which ones, on [[provenance:statistics:regression]]. | * **Quotes were checked in bulk and by hand.** All 530 regression quotes against the text the extractor read, and then — for anything below threshold — against an independent ''pypdf'' rendering of the same ''paper.pdf'': **267 exact (50.4%) after whitespace normalisation, 171 partial (32.3%) at ≥60% of five-word windows, 65 (12.3%) located only in the second rendering, 27 (5.1%) below threshold in both**. The 65 are a defect in the stored text, not in the extraction. Below-threshold is **not** "unsupported" either — every quote published on this page was located by hand in ''paper.cols.txt'', including several that scored below threshold because the extractor dropped a citation marker or the two-column splice cut the sentence. Details, including which ones, on [[provenance:statistics:regression]]. |
| * **''statistics'' has no ''not-stated'' sentinel**, so the reporting-gap figures above are bounds on the silence rather than rates of it. | * **''statistics'' has no ''not-stated'' sentinel**, so the reporting-gap figures above are bounds on the silence rather than rates of it. |
| * **The ''detail'' field is not a measurement of reporting practice.** It is a capped free-text field the extractor filled when a value happened to be in the quoted sentence. Counts of R², CIs and //p//-values inside it establish an ordering and nothing more. | * **The ''detail'' field is not a measurement of reporting practice.** It is a capped free-text field the extractor filled when a value happened to be in the quoted sentence. Counts of R², CIs and //p//-values inside it establish an ordering and nothing more. |
| ===== Open Questions ===== | ===== Open Questions ===== |
| |
| | <WRAP todo> |
| * **Nobody has estimated the intra-class correlation of a web-measurement outcome.** The simulation on this page shows the entire size of the error depends on it, and no paper in this corpus reports one. "What is the ICC of //sets a third-party cookie before consent// by consent platform, by CMS, by hosting provider, and by the same site across monthly crawls?" is a small, self-contained study that would tell every crawl paper how wrong its standard errors are. The same gap is open on [[Statistics:Hypothesis testing]] and it is one study away from being closed for both. | * **Nobody has estimated the intra-class correlation of a web-measurement outcome.** The simulation on this page shows the entire size of the error depends on it, and no paper in this corpus reports one. "What is the ICC of //sets a third-party cookie before consent// by consent platform, by CMS, by hosting provider, and by the same site across monthly crawls?" is a small, self-contained study that would tell every crawl paper how wrong its standard errors are. The same gap is open on [[Statistics:Hypothesis testing]] and it is one study away from being closed for both. |
| * **No crawl paper models the site as a random effect.** One non-crawl field study does {[zeng2022_factors]}, so the technique is not unknown in this community — it has just never been pointed at a repeated crawl. Whether that is because cluster-robust standard errors are the better tool anyway (they usually are), or because nobody has tried, is not answerable from text. Somebody should re-analyse a published longitudinal crawl both ways and report how much moves. | * **No crawl paper models the site as a random effect.** One non-crawl field study does {[zeng2022_factors]}, so the technique is not unknown in this community — it has just never been pointed at a repeated crawl. Whether that is because cluster-robust standard errors are the better tool anyway (they usually are), or because nobody has tried, is not answerable from text. Somebody should re-analyse a published longitudinal crawl both ways and report how much moves. |
| * **The 2022-onward arrival of causal designs is ten papers.** Whether it is a durable turn or a handful of authors with economics training is not decidable yet, and the 2025–2026 evidence for it sits in the provisional years. | * **The 2022-onward arrival of causal designs is ten papers.** Whether it is a durable turn or a handful of authors with economics training is not decidable yet, and the 2025–2026 evidence for it sits in the provisional years. |
| * **None of the seven venues asks for any of this.** As on [[Statistics:Hypothesis testing]], the calls for papers contain no statistical-reporting requirement, so the only thing holding a regression section to a standard is an individual reviewer. | * **None of the seven venues asks for any of this.** As on [[Statistics:Hypothesis testing]], the calls for papers contain no statistical-reporting requirement, so the only thing holding a regression section to a standard is an individual reviewer. |
| | </WRAP> |
| ===== Related Pages ===== | ===== Related Pages ===== |
| |
