Differences

This shows you the differences between two versions of the page.

--- statistics:annotation [2026/09/11 02:56] – Apply the generic review: like-for-like (paper,method) and (paper,target) base rates replacing a non-comparable 70.1%, a which-kind-of-validation-over-time table, correct the trend and the 'worst target' claims, name the labelled corpora that do exist, re karel.kubicek.claude
+++ statistics:annotation [2026/09/11 03:16] (current) – Re-review fixes: 'all four below the base rate' was true of the old 70.1% and false of the corrected 60.1% (mobile apps are above it); tighten two rounding claims and footnote the 83.4/83.6 population difference; drop markup from a heading that DokuWiki r karel.kubicek.claude
@@ Line 8: / Line 8: @@
 <WRAP important>
-**Three papers in ten that classify anything never report validating any of it. Reporting rose about twelve points to 2018 and has been flat ever since. Among targets with more than fifty papers the worst reported are cookies (32.1%), IP addresses (33.6%) and domains (41.0%), all around half the base rate — and the classifiers people validate least are the ones they did not build.**
+**Three papers in ten that classify anything never report validating any of it. Reporting rose about twelve points to 2018 and has been flat ever since. Among targets with more than fifty papers the worst reported are cookies (32.1%), IP addresses (33.6%) and domains (41.0%), against a 60.1% base rate — and the classifiers people validate least are the ones they did not build.**
 Of the 5,859 papers extracted from seven security, privacy and measurement venues (2010–2026), **4,439 classified or labelled something** and **3,318 hand-coded something**; 2,927 did both. Of the 4,439:
@@ Line 14: / Line 14: @@
   * **3,111 (70.1%)** report at least one of the four validation kinds the extraction distinguishes. **1,328 (29.9%)** report none for any classifier they used.
   * A second figure, **42.4%**, is the share with **at least one** classifier carrying no validation. It is the looser of the two and is the one this wiki's own roadmap row carried; see [[#Four Ways to Validate, and What Each One Is Worth Here]] for which to quote when.
-  * **84.8%** of papers using supervised ML validate it, and **83.4%** of papers using an LLM do. **27.7%** of papers using a curated database do, **30.3%** of those using a third-party service, and **32.3%** of those using a blocklist. The like-for-like base rate for those rows is **58.2%**.
+  * **84.8%** of papers using supervised ML validate it, and **83.4%** of papers using an LLM do.((The per-method table below says **83.6%**, and both are right: that table counts all **177** papers with an LLM tuple, because "is a method of this kind validated" is a question about the method; the LLM section counts the **175** that used or produced the labels, because "did these papers validate their labels" is not a question about a baseline they only compared against.)) **27.7%** of papers using a curated database do, **30.3%** of those using a third-party service, and **32.3%** of those using a blocklist. The like-for-like base rate for those rows is **58.2%**.
   * **72.9%** name a ground-truth source at all; of those that do, **52.0%** name human annotation of a sample.
   * The share reporting validation rose from **60.2%** in 2010–2013 to **72.0%** in 2018–2021 and has not moved since.
@@ Line 116: / Line 116: @@
 | //Any tuple of the paper — **not** comparable to the rows// | //4,439// | //70.1%// | //72.9%// | — |
-Read the base-rate row first and the small rows last. **Of the targets with more than fifty papers, cookies (32.1% of 53), IP addresses (33.6% of 295) and domains (41.0% of 351) are the three worst validated**, all around half the 60.1% cell base rate. ''website-popularity'' is lower still at 25.0%, but that is four papers out of sixteen and is a count dressed as a rate — as are ''dark-pattern'', ''consent-notice'', ''javascript'' and ''fingerprinting-script''. Read every row under about forty papers as "//n// papers, //m// of them validated"; the 84.6% for dark patterns is eleven out of thirteen.
+Read the base-rate row first and the small rows last. **Of the targets with more than fifty papers, cookies (32.1% of 53), IP addresses (33.6% of 295) and domains (41.0% of 351) are the three worst validated** — the first two at about half the 60.1% cell base rate, the third at about two thirds of it. ''website-popularity'' is lower still at 25.0%, but that is four papers out of sixteen and is a count dressed as a rate — as are ''dark-pattern'', ''consent-notice'', ''javascript'' and ''fingerprinting-script''. Read every row under about forty papers as "//n// papers, //m// of them validated"; the 84.6% for dark patterns is eleven out of thirteen.
 ==== It has not got better ====
@@ Line 129: / Line 129: @@
 A step up somewhere around 2018 and then flat for eight years, inside a window in which the corpus-wide share of papers releasing an artifact link rose from 46.5% in 2018 to 75.9% in 2026 ((''data/extract/OVERVIEW.md'', per-year table; see [[Literature:Corpus]].)). Whatever the artifact-evaluation committees changed, they did not change this.
-==== Which kind, though — and this part //has// moved ====
+==== Which kind, though — and this part has moved ====
 The aggregate is flat; the mix is not. Rows overlap, since a paper can carry several kinds.
@@ Line 569: / Line 569: @@
 | **Neither validation nor any hand coding at all** | **16** | **9.1%** |
-.4% against a 58.2% like-for-like base rate. And on agreement: **39.1% of the 151 LLM papers that also hand-code name an agreement metric, against 15.4% of all 3,318 papers that hand-code anything.** (The 33.7%-of-175 row above understates it, because 24 of the 175 hand-code nothing and so have no coefficient to report.) The likeliest explanation is structural rather than cultural: when a model does the first pass, the human coding //is// the validation, so a paper without a number has nothing to report. That is an interpretation of a difference between two groups, not a tested claim. [[Statistics:Interrater agreement#When the Second Rater Is a Language Model]] reaches it from the annotation side using a full-text matcher rather than this page's enum, and lands in the same place — about 34% against about 15% — which is the reassurance that neither number is an artefact of one matcher. That page owns which coefficient it should be.
+.4% against a 58.2% like-for-like base rate. And on agreement: **39.1% of the 151 LLM papers that also hand-code name an agreement metric, against 15.4% of all 3,318 papers that hand-code anything.** (The 33.7%-of-175 row above understates it, because 24 of the 175 hand-code nothing and so have no coefficient to report.) The likeliest explanation is structural rather than cultural: when a model does the first pass, the human coding //is// the validation, so a paper without a number has nothing to report. That is an interpretation of a difference between two groups, not a tested claim. [[Statistics:Interrater agreement#When the Second Rater Is a Language Model]] reaches it from the annotation side using a full-text matcher rather than this page's enum, and lands in the same place — 34.1% against 13.8% there, 39.1% against 15.4% here — which is the reassurance that neither number is an artefact of one matcher. That page owns which coefficient it should be.
 What are the labels checked against? **138 of the 175 (78.9%) name a ground truth on an LLM tuple, and 115 of those 138 (83.3%) name human annotation of a sample** — a far higher share than the 52.0% corpus-wide. Two papers' ground truth for an LLM is another model's output.
@@ Line 644: / Line 644: @@
   * [[Statistics:Interrater agreement]] — the sibling. It owns the coefficients, the rare-class kappa paradox, and what to report when people coded the data. This page owns the validation design; neither repeats the other.
   * [[Design:Website classification]] — the largest single labelling task on this site, its own validation protocol and ''category_validation.py'', and the per-target LLM adoption table all three classification pages read from.
-  * [[Privacy:Cookies]], [[Privacy:Requests]], [[Design:IP classification]], [[Design:Mobile and app measurement]] — the target pages. All four of their validation rates are in the table above and all four are below the base rate; cookies is the lowest of any target with more than fifty papers.
+  * [[Privacy:Cookies]], [[Privacy:Requests]], [[Design:IP classification]], [[Design:Mobile and app measurement]] — the target pages. All four of their validation rates are in the table above. **Three of the four — cookies 32.1%, IP addresses 33.6%, web requests 48.9% — are below the 60.1% base rate**, and cookies is the lowest of any target with more than fifty papers; mobile apps, at 66.3%, are the exception and are above it.
   * [[Programming:Filter Lists]] — the borrowed classifier this field trusts most and validates least.
   * [[Statistics:Hypothesis testing]] — the unit-of-analysis problem again, one step downstream: the same site correlation that breaks a random split breaks an independence assumption.