Differences

This shows you the differences between two versions of the page.

--- statistics:annotation [2026/09/11 02:24] – New page: labelling data with humans or an LLM and validating the result; hoists the corpus-wide classification.validation figures out of design:website_classification and owns LLM-as-annotator. Authored by Claude karel.kubicek.claude
+++ statistics:annotation [2026/09/11 03:16] (current) – Re-review fixes: 'all four below the base rate' was true of the old 70.1% and false of the corrected 60.1% (mobile apps are above it); tighten two rounding claims and footnote the 83.4/83.6 population difference; drop markup from a heading that DokuWiki r karel.kubicek.claude
@@ Line 3: / Line 3: @@
 Every measurement on this site ends in labels. This third-party domain is advertising; this cookie is analytics; this banner is a dark pattern; this site is News. Something assigned those labels — a filter list, a vendor API, a regex you wrote on a Tuesday, three hundred rows you coded yourself, or since 2024 a language model — and the whole result rests on the claim that the labels mean what their names say.
-This page is about the second half of that: **how you show the labels are right, and what "right" can even mean when the things being labelled are web pages**. It is not about kappa — [[Statistics:Interrater agreement]] owns the coefficients and the reliability of hand coding, and this page does not repeat any of it. It is about the design of the check: what you validate against, whether that source is independent of the thing you are checking, why a held-out set drawn from a crawl is usually not held out, and what an LLM annotator owes you that a human one does not.
+This page is about the second half of that: **how you show the labels are right when the things being labelled are web pages**. It is not about kappa — [[Statistics:Interrater agreement]] owns the coefficients and the reliability of hand coding, and this page does not repeat any of it. It is about the design of the check: what you validate against, whether that source is independent of the thing you are checking, why a held-out set drawn from a crawl is usually not held out, and what an LLM annotator owes you that a human one does not.
 The short version of the corpus evidence is that this is the least-reported part of the pipeline. Papers say which tool they used and what they found; a large minority never say whether the labels were checked, and the targets where they say it least are exactly the web ones.
 <WRAP important>
-**Three papers in ten that classify anything never report validating any of it, and the reporting has not improved in fifteen years. The worst-reported targets are cookies (32.1%), IP addresses (33.6%) and domains (41.0%) — all far below the 70.1% base rate. And the classifiers people validate least are the ones they did not build.**
+**Three papers in ten that classify anything never report validating any of it. Reporting rose about twelve points to 2018 and has been flat ever since. Among targets with more than fifty papers the worst reported are cookies (32.1%), IP addresses (33.6%) and domains (41.0%), against a 60.1% base rate — and the classifiers people validate least are the ones they did not build.**
 Of the 5,859 papers extracted from seven security, privacy and measurement venues (2010–2026), **4,439 classified or labelled something** and **3,318 hand-coded something**; 2,927 did both. Of the 4,439:
   * **3,111 (70.1%)** report at least one of the four validation kinds the extraction distinguishes. **1,328 (29.9%)** report none for any classifier they used.
-  * The often-quoted **42.4%** is a different, looser quantity: the share with **at least one** classifier carrying no validation. Rows overlap, so it is not the share of papers that validated nothing. Both are below.
+  * A second figure, **42.4%**, is the share with **at least one** classifier carrying no validation. It is the looser of the two and is the one this wiki's own roadmap row carried; see [[#Four Ways to Validate, and What Each One Is Worth Here]] for which to quote when.
-  * **84.8%** of papers using supervised ML validate it, and **83.6%** of papers using an LLM do. **27.7%** of papers using a curated database do, **30.3%** of those using a third-party service, and **32.3%** of those using a blocklist.
+  * **84.8%** of papers using supervised ML validate it, and **83.4%** of papers using an LLM do.((The per-method table below says **83.6%**, and both are right: that table counts all **177** papers with an LLM tuple, because "is a method of this kind validated" is a question about the method; the LLM section counts the **175** that used or produced the labels, because "did these papers validate their labels" is not a question about a baseline they only compared against.)) **27.7%** of papers using a curated database do, **30.3%** of those using a third-party service, and **32.3%** of those using a blocklist. The like-for-like base rate for those rows is **58.2%**.
   * **72.9%** name a ground-truth source at all; of those that do, **52.0%** name human annotation of a sample.
-  * The share reporting validation was 63.6% in 2010 and 70.6% in 2026. There is no trend to speak of.
+  * The share reporting validation rose from **60.2%** in 2010–2013 to **72.0%** in 2018–2021 and has not moved since.
 Every figure here is a claim about those seven venues. 2025 and 2026 are provisional venue-years. See [[#Methodology and Limitations of These Figures]].
@@ Line 23: / Line 23: @@
 ===== What to Read First =====
-  * {[arp2022_dosdonts]} (USENIX Security 2022) — //Dos and Don'ts of Machine Learning in Computer Security//. Ten pitfalls, reviewed across 30 papers from the top-four security venues over ten years. Three of the ten (sampling bias, label inaccuracy, data snooping) are this page's subject, and the paper is the reason a reviewer now asks. Read §2 and §3 before you draw a test set.
+  * {[arp2022_dosdonts]} (USENIX Security 2022) — //Dos and Don'ts of Machine Learning in Computer Security//. Ten pitfalls, reviewed across 30 papers from the top-four security venues over ten years. Three of the ten — sampling bias, label inaccuracy, data snooping — are this page's subject. Read §2 and §3 before you draw a test set.
-  * {[pendlebury2019_tesseract]} (USENIX Security 2019) — //TESSERACT//. What happens when the split is drawn without respect to time or to the population's structure: published F1 scores of up to 0.99, and most of it is bias. The mechanism generalises off malware.
+  * {[pendlebury2019_tesseract]} (USENIX Security 2019) — //TESSERACT//. What happens when the split is drawn without respect to time or to the population's structure: //"published F1 scores of up to 0.99"// on Android malware, which the paper argues are inflated by spatial and temporal bias. The mechanism generalises off malware.
   * {[ruth2022_world]} (IMC 2022) — the shortest worked example of a defensible validation in this corpus: stratified sample, ten sites per category, keep only categories above 80% accuracy, say what was dropped. One paragraph, and it is the paragraph to imitate.
   * {[shaoor2023cookiegraph]} (CCS 2023) — the one sentence most web-measurement papers are missing: //"We ensure that there is no overlap in the websites used for training and testing in each fold."// See [[#The Held-Out Set That Is Not Held Out]].
@@ Line 34: / Line 34: @@
 ^ Kind ^ Papers ^ Share of 4,439 ^ What it actually establishes ^
-| ''manual-validation'' | 2,265 | 51.0% | A person re-labelled a sample and you report the agreement. **Usually the only one available**, because for cookies, requests, banners and third-party domains there is no pre-existing labelled corpus. Its weakness is sample size, not principle. |
+| ''manual-validation'' | 2,265 | 51.0% | A person re-labelled a sample and you report the agreement. **Usually the only one available**, for the reason in the box below. Its weakness is sample size, not principle. |
 | ''comparison-to-other-method'' | 791 | 17.8% | Your labels agree with another tool's. This measures **agreement, not correctness** — two filter lists derived from the same community's reports will agree and both be wrong. Useful as a floor, never as a ceiling. |
 | ''held-out-test-set'' | 547 | 12.3% | Performance on items withheld from training. Only tells you about the labels you already had, and only if the split respects the structure of the data — see below. |
@@ Line 40: / Line 40: @@
 | ''none-reported'' | 1,881 | 42.4% | The paper reports no check. |
 | ''not-applicable'' (sentinel) | 1,160 | 26.1% | The extraction could not find a validation step and did not think one was called for. **Read as "no validation reported", not as "validation was unnecessary"**, unless you check the paper. |
+<WRAP important>
+**Labelled corpora for these targets do exist — they are just not independent of the thing you are testing.** Bollinger et al.'s CookieBlock release (USENIX Security 2022) is 2.2M cookies labelled from operators' own CMP declarations and is the standard cookie ground truth; EasyList and EasyPrivacy are a labelled corpus for requests; there are released consent-banner and privacy-policy sets ([[Privacy:Consent]], [[Privacy:Policies]]). **Use them — do not rebuild one.** But a held-out score against CookieBlock measures agreement with CMP declarations, and a held-out score against EasyList measures agreement with EasyList, and in both cases your classifier and your ground truth can be wrong in the same direction. That is why manual validation of a fresh sample is the dominant instrument here and not a fallback: it is the only check most web-measurement targets have that is independent of the artefact under test. [[Privacy:Cookies]] has the cookie sets, [[Programming:Filter Lists]] the request ones.
+</WRAP>
 Rows overlap: the field is multi-valued and a paper with three classifiers appears in three rows. Collapsing to one verdict per paper:
@@ Line 50: / Line 54: @@
 <WRAP important>
-**"42.4% report no validation" and "29.9% validate nothing" are both true and they are different claims.** The first counts papers with at least one unvalidated classifier and is the number this wiki's [[roadmap]] carried; the second counts papers where nothing was validated. Quote whichever you mean, and name which. A page that quotes the 42.4% as "of the papers that classified anything, that share reports no validation" has overstated it by twelve points.
+**Two figures, two claims. Here they are in the form to paste into a related-work section:**
+  * //"29.9% of the 4,439 papers that classify something report no validation of any classifier they used."// **This is the headline figure** — it is the one that answers "how often does this go unchecked".
+  * //"42.4% carry at least one classifier they do not validate."// Use this when your point is about the //unvalidated second classifier// — the blocklist bolted onto the model, the service used to break a tie.
+They differ by twelve points because the second counts a paper once for any unvalidated tuple. Writing the 42.4% and saying "report no validation" — which is what this wiki's own [[:roadmap]] row did until this page was written — states the second and means the first.
 </WRAP>
@@ Line 71: / Line 80: @@
 | ''third-party-service'' | 643 | 30.3% |
 | ''curated-database'' | 649 | 27.7% |
-| **All methods (base rate)** | **4,439** | **70.1%** |
+| **All (paper, method) cells — the like-for-like base rate** | **8,259** | **58.2%** |
+| //Any tuple of the paper validates — **not** comparable to the rows// | //4,439// | //70.1%// |
+**Read the 58.2%, not the 70.1%.** Each row above asks whether the tuple //for that method// carries a validation. "Does any classifier in the paper validate" is a different and easier test, and a paper with three classifiers passes it on one of them while appearing in three rows. The comparable denominator is the (paper, method) cell, and the difference is twelve points — enough to change how big every gap below looks. ((This correction came out of review on 2026-09-11: an earlier draft compared every row to the 70.1% and overstated each gap. The ordering is unaffected.))
-The ordering is the finding, and the base-rate row is why. **You validate what you built and you trust what you downloaded.** The three bottom rows — blocklists, third-party services, curated databases — are precisely EasyList, Disconnect, VirusTotal, Cookiepedia and the vendor categorisation APIs, and they are the instruments a web measurement leans on hardest. A borrowed classifier is not a validated one: it was validated, if at all, for somebody else's population, at some other time, against some other ground truth. [[Programming:Filter Lists]] and [[Design:Website classification]] both measure how far that assumption travels, and it is not far.
+The ordering is the finding. **You validate what you built and you trust what you downloaded.** The three bottom rows — blocklists, third-party services, curated databases — are precisely EasyList, Disconnect, VirusTotal, Cookiepedia and the vendor categorisation APIs, and they are the instruments a web measurement leans on hardest. A borrowed classifier is not a validated one: it was validated, if at all, for somebody else's population, at some other time, against some other ground truth. [[Programming:Filter Lists]] and [[Design:Website classification]] both measure how far that assumption travels, and it is not far.
 ''classification.method'' agrees about 58% run-to-run between independent extraction runs, so read this table as a ranking with a clear top and bottom rather than as thirteen precise percentages. The gap between 84.8% and 27.7% is far too large to be a labelling artefact; the gap between 55.2% and 54.9% is not a gap.
@@ Line 101: / Line 113: @@
 | ''website-popularity'' | 16 | 25.0% | 31.3% | [[Design:Website selection]] |
 | ''dark-pattern'' | 13 | 84.6% | 76.9% | [[Privacy:Darkpatterns]] |
-| **All targets (base rate)** | **4,439** | **70.1%** | **72.9%** | — |
+| **All (paper, target) cells — the like-for-like base rate** | **6,483** | **60.1%** | **65.2%** | — |
+| //Any tuple of the paper — **not** comparable to the rows// | //4,439// | //70.1%// | //72.9%// | — |
-Read the base-rate row first and the small rows last. **Cookie classification, at 32.1% of 53 papers, is the worst-validated target in the corpus that has a page here**, and IP addresses (33.6% of 295) and domains (41.0% of 351) are barely better; all three are far below the 70.1% base rate. The rows with fewer than about 40 papers (''dark-pattern'', ''consent-notice'', ''javascript'', ''fingerprinting-script'', ''website-popularity'') are counts dressed as rates and should be read as "n papers, m of them validated" — the 84.6% for dark patterns is eleven papers out of thirteen.
+Read the base-rate row first and the small rows last. **Of the targets with more than fifty papers, cookies (32.1% of 53), IP addresses (33.6% of 295) and domains (41.0% of 351) are the three worst validated** — the first two at about half the 60.1% cell base rate, the third at about two thirds of it. ''website-popularity'' is lower still at 25.0%, but that is four papers out of sixteen and is a count dressed as a rate — as are ''dark-pattern'', ''consent-notice'', ''javascript'' and ''fingerprinting-script''. Read every row under about forty papers as "//n// papers, //m// of them validated"; the 84.6% for dark patterns is eleven out of thirteen.
 ==== It has not got better ====
@@ Line 114: / Line 127: @@
 | 2025–2026 //(provisional)// | 940 | 69.7% | 76.5% |
-A step up somewhere around 2018 and then flat for eight years, inside a window in which the corpus-wide share of papers releasing an artifact link rose from 46.5% to 75.9% ((''data/extract/OVERVIEW.md'', per-year table; see [[Literature:Corpus]].)). Whatever the artifact-evaluation committees changed, they did not change this.
+A step up somewhere around 2018 and then flat for eight years, inside a window in which the corpus-wide share of papers releasing an artifact link rose from 46.5% in 2018 to 75.9% in 2026 ((''data/extract/OVERVIEW.md'', per-year table; see [[Literature:Corpus]].)). Whatever the artifact-evaluation committees changed, they did not change this.
+==== Which kind, though — and this part has moved ====
+The aggregate is flat; the mix is not. Rows overlap, since a paper can carry several kinds.
+^ Years ^ Papers ^ Manual validation ^ Comparison to another method ^ Held-out test set ^ Cross-validation ^ ''none-reported'' ^
+| 2010–2013 | 364 | 40.4% | 16.8% | 6.9% | 7.7% | 41.5% |
+| 2014–2017 | 595 | 47.4% | 17.6% | 10.9% | 9.9% | 43.5% |
+| 2018–2021 | 1,096 | 53.6% | 16.7% | 12.5% | 9.9% | 39.8% |
+| 2022–2024 | 1,444 | 52.5% | 17.0% | 13.8% | 5.8% | 38.2% |
+| 2025–2026 //(provisional)// | 940 | 52.1% | 21.0% | 12.9% | 3.6% | 51.5% |
+Three readings, in descending order of how much weight they will hold. **Manual validation of a sample is and has always been the dominant instrument**, rising until about 2018 and flat at roughly half of all classifying papers since; it is the current practice, not a legacy one. **Cross-validation is being abandoned** — 9.9% in 2018–2021, 3.6% now — which is what you would expect as the field's classifiers stop being models trained on a fixed table and start being services, lists and prompts, none of which has folds to cross. And **held-out test sets plateaued after 2018** at around one paper in eight.
+<WRAP todo>
+**The ''none-reported'' jump from 38.2% to 51.5% is the largest move in that table and it should not be quoted.** It lands entirely inside the provisional 2025–2026 slice, whose venue composition is not the corpus's, and the same slice is where LLM classification arrives — so "papers stopped validating" and "papers started using instruments the extractor reads differently" are not separable here. Re-check it when CCS 2026 and IMC 2026 exist.
+</WRAP>
 ===== Where Ground Truth Comes From =====
@@ Line 132: / Line 162: @@
 | Residue — no family fits | — | 1,577 | 48.8% |
-The residue is large and is not hidden: 2,434 of the 7,114 tuples (34.2%) fold nowhere, and because a paper carries several tuples that touches 1,577 papers. Reading it, it is dominated by one-off experimental setups — //"known member and non-member samples"//, //"website loaded during each recorded current trace"//, //"experiment phase labels for authentic and replayed samples"// — which means **//known by construction// is the family this fold undercounts most**, and the fold says so rather than absorbing them quietly. The full residue is on [[provenance:statistics:annotation]].
+The residue is large and is not hidden: 2,434 of the 7,114 tuples (34.2%) fold nowhere, and because a paper carries several tuples that touches 1,577 papers. It is a **long tail**: 2,068 of the 2,206 distinct unmapped strings (93.7%) appear exactly once, each naming a resource or procedure specific to one paper that no keyword family can be expected to reach — //"optical lens scanner labels"//, //"touchpal call tags"//, //"indegree median of the population"//, //"market-purchased IoT devices"//. A systematic 30-string sample of it was hand-classified into the family each string //would// have joined, and it spreads across all of them: 6 //existing dataset//, 6 //by construction//, 4 //third-party service//, 3 //human annotation//, 3 //another automated method//, 2 //curated list//, 2 //execution//, and 4 that the string alone cannot place. **So every share in the table above is a floor, and no single family is systematically missing.** The full residue and the sample's verdicts are on [[provenance:statistics:annotation]].
 Two things to take from it. First, **half of all stated ground truth in this literature is a human reading a sample**; there is no shortcut and the field has not found one. Second, the two non-independent families are small but not zero, and they are the ones to watch: a paper whose ground truth is //"majority vote across three LLM judgments"// has measured consistency, not correctness.
@@ Line 152: / Line 182: @@
 ==== And the positive class is rare ====
-The second half of the same problem. "3.4% of pages do X" means a 1,000-item validation sample holds about 34 positives, and a precision computed on 34 items is a range, not a value. At 34/34 correct the 95% Wilson interval is still 10 points wide; at 25/34 it is 29 points wide. A bare point estimate from a sample that size is not a measurement, and the fix is one line of arithmetic rather than more annotation.
+The second half of the same problem. "3.4% of pages do X" means a 1,000-item validation sample holds about 34 positives, and a precision computed on 34 items is a range, not a value. At 34/34 correct the 95% Wilson interval is still 10 points wide; at 25/34 it is 29 points wide. A bare point estimate from a sample that size is not a measurement. Reporting the interval costs one line of arithmetic; //narrowing// it costs annotation, and the cheap way to spend it is to stratify — oversample the predicted positives and weight back — rather than to draw a bigger uniform sample in which 96.6% of the labels tell you nothing about precision.
 The same applies to **coverage**. Items the classifier refused, returned nothing for, or answered outside your taxonomy are a result — they are the closest thing you have to a measurement of how well the instrument understood the task. Folding them into the negatives inflates precision and deflates recall at the same time, and dropping them silently changes the denominator of everything downstream.
@@ Line 501: / Line 531: @@
 almost always is.
 </file>
+**The //direction// of that gap is general; its //size// is a parameter of the toy.** 0.964 → 0.659 follows from how much of the label the site decides (here 1.0 against 0.6) and from a classifier whose only real feature is the site. On your data it could be two points or forty, and the only way to find out is to run both splits — which costs one extra line and is the first open question below.
 ''selftest'' checks the Wilson endpoints against their own defining equation (each endpoint must satisfy |//p̂// − //p//| = //z//·√(//p//(1−//p//)///n//)), rather than against a hard-coded number, so a mistyped term in any position fails. It also checks that an abstention is not scored as a negative and that the grouped split never leaks a group. There is no numpy, scipy or sklearn: it runs where your data is.
@@ Line 511: / Line 543: @@
 **An LLM annotator is a classifier, and it inherits every obligation of one.** A held-out set drawn before you look at the output. A human-agreement figure with its coefficient named. The model version and the date you ran it — a model id is not a version. The prompt, published. The decoding settings. A residue count. None of that is new; what is new is how many papers reach for a model without noticing that they have just added a classifier to their pipeline.
-**And 42.4% of this literature reports no validation of any classifier at all.** An LLM does not get a lighter burden because it is fluent.
+**And three papers in ten that classify anything validate none of it.** An LLM does not get a lighter burden because it is fluent.
 </WRAP>
@@ Line 537: / Line 569: @@
 | **Neither validation nor any hand coding at all** | **16** | **9.1%** |
-.4% against a 70.1% base rate, and 33.7% reporting an agreement metric against **15.4% of the 3,318 papers that hand-code anything**. The likeliest explanation is structural rather than cultural: when a model does the first pass, the human coding //is// the validation, so a paper without a number has nothing to report. [[Statistics:Interrater agreement#When the Second Rater Is a Language Model]] reaches the same conclusion from the annotation side with a different matcher, and owns the question of which coefficient that number should be.
+.4% against a 58.2% like-for-like base rate. And on agreement: **39.1% of the 151 LLM papers that also hand-code name an agreement metric, against 15.4% of all 3,318 papers that hand-code anything.** (The 33.7%-of-175 row above understates it, because 24 of the 175 hand-code nothing and so have no coefficient to report.) The likeliest explanation is structural rather than cultural: when a model does the first pass, the human coding //is// the validation, so a paper without a number has nothing to report. That is an interpretation of a difference between two groups, not a tested claim. [[Statistics:Interrater agreement#When the Second Rater Is a Language Model]] reaches it from the annotation side using a full-text matcher rather than this page's enum, and lands in the same place — 34.1% against 13.8% there, 39.1% against 15.4% here — which is the reassurance that neither number is an artefact of one matcher. That page owns which coefficient it should be.
-What are the labels checked against? **138 of the 175 (78.9%) name a ground truth on an LLM tuple, and 115 of those 138 (83.3%) name human annotation of a sample** — a far higher share than the 52.0% corpus-wide, and the reason the method has not yet gone wrong at scale here. Two papers' ground truth for an LLM is another model's output.
+What are the labels checked against? **138 of the 175 (78.9%) name a ground truth on an LLM tuple, and 115 of those 138 (83.3%) name human annotation of a sample** — a far higher share than the 52.0% corpus-wide. Two papers' ground truth for an LLM is another model's output.
-{[cory2026_wordlevel]} (PoPETs 2026) is the worked example to copy: a hand-built gold set of //"200 privacy policies from Android applications"//, //"equally distributed among the six experts, with two experts randomly assigned to each pol[icy]"//, and the LLM pipeline scored against it. {[xiong2026_tgnn]} (TheWebConf 2026) is the closest thing in the corpus to a web-measurement template — 20,000 requests labelled by human experts as the ground truth, an LLM labelling component scored against a held-out portion, and EasyList compared against the same set — and it is also a fair illustration of the gap in [[#The Held-Out Set That Is Not Held Out]]: the split is //"into training and testing sets with a ratio of 80% and 20%"//, with no statement about whether requests from a site can appear on both sides.
+{[cory2026_wordlevel]} (PoPETs 2026) is the worked example to copy: a hand-built gold set of //"200 privacy policies from Android applications"//, split among six experts with //"two experts randomly assigned to each pol[icy]"//, and the LLM pipeline scored against it. {[xiong2026_tgnn]} (TheWebConf 2026) is the closest thing in the corpus to a web-measurement template: a 20,000-request ground truth, heuristic-labelled and then validated by human experts (//"human experts validate the ground truth dataset"//), an LLM labelling component scored against it, and EasyList compared against the same set. It is also the page's own worked warning, twice over. The LLM component's F1 of 98.17% is computed //"on the ground truth dataset"// — the whole 20,000, with no held-out portion for the labelling step at all — while the extraction records that tuple as ''held-out-test-set''. And the 80/20 split the paper does describe (//"into training and testing sets with a ratio of 80% and 20%"//) belongs to the downstream graph model that consumes those labels; it says nothing about whether requests from one site land on both sides. Reading what a validation enum actually refers to is not optional, and this one is wrong in the source data this page is built on.
 ==== How they ran the model, and whether you could tell ====
@@ Line 552: / Line 584: @@
 | Says in words that a prompt is reproduced somewhere | 26 | **14.9%** | 31 candidates read in context; 4 are a different sense, 1 declines to publish |
 | Names a model resolvable to an actual artefact | 34 | 19.4% | measured on [[Design:Website classification#And almost nobody names a model you could resolve]] |
+| **Supplies //none// of the first three** | **106** | **60.6%** | the joint distribution, not three marginals |
+| Supplies all three | 2 | 1.1% | |
-Three quarters of the papers that let a model assign labels do not say at what temperature, and six in seven do not say the prompt is anywhere. Note what the temperature row is //not// saying: the word "temperature" appears in 55 of the 175, but in eleven of those it is the colour temperature of a smart bulb, a weather API payload, the softmax τ, a simulated-annealing schedule, or a cited paper's title. A probe that had reported 55 would have been 34% too high, which is why the two figures above that carry a percentage were read by hand.
+Three quarters of the papers that let a model assign labels do not say at what temperature, six in seven do not say the prompt is anywhere, and — the figure that matters, because these are not the same papers — **the median paper supplies none of the three**. Two of 175 supply all three. Note what the temperature row is //not// saying: the word "temperature" appears in 55 of the 175, but in eleven of those it is the colour temperature of a smart bulb, a weather API payload, the softmax τ, a simulated-annealing schedule, or a cited paper's title. A probe that had reported 55 would have been 34% too high, which is why the two figures above that carry a percentage were read by hand.
 The prompt figure is a lower bound in both directions and should be quoted that way: a prompt printed in an appendix figure with no sentence pointing at it is invisible. One of the 31 candidates matched because it says it is **deliberately withholding** its prompts — those are attack prompts rather than annotation prompts, so it is not a counter-example to the norm, but it is a reminder that "the prompt is in the appendix" and "the prompt exists" are different claims.
 <WRAP important>
-**A model id is not a version.** ''GPT-4'' names at least six distinct snapshots; ''Qwen3-Plus'' is a hosted alias whose weights you cannot pin. For a hosted model, give the dated snapshot //and// the date you ran it, because the snapshot will be withdrawn. For an open-weight model, give family, size, instruct-variant and ideally the weights digest. [[Design:Website classification#And almost nobody names a model you could resolve]] measures this on the same 175 and finds **19.4%** naming something resolvable.
+**A model id is not a version, and a model id is not always even a model.** ''GPT-4'' names at least six distinct snapshots. {[xiong2026_tgnn]} says //"The specific model of the large language model used in this paper is Qwen3-Plus"// — more than most papers give, and it still does not resolve: Alibaba Cloud's Model Studio catalogue lists a ''qwen3.7-plus'' and no ''qwen3-plus'', and there is no such checkpoint under Qwen's own account on Hugging Face.((Checked 2026-09-11 against ''alibabacloud.com/help/en/model-studio/models'' and the Hugging Face model API. The hosted //Plus// tier's id carries the generation number and churns with it, so a paper naming the tier without the generation and the date has named nothing durable.)) A hosted tier name ages out; a checkpoint does not. [[Design:Website classification#And almost nobody names a model you could resolve]] measures this on the same 175 — **19.4%** name something resolvable — and carries the hosted-versus-open-weight instruction in full. The other half of this box is [[Statistics:Interrater agreement]]'s: do not let your human coders see the model's labels before they code.
-**And do not let your human coders see the model's labels first.** Agreement between a human who has read the model's answer and the model is not a measurement of anything. [[Statistics:Interrater agreement]] has the protocol and the coefficients.
 </WRAP>
 ==== Why the population is an enum and not a keyword search ====
-Worth stating, because it is the mistake this page was nearly built on. The [[roadmap]] row that queued this page came from a title-and-summary probe for LLM annotation, matching 30 papers. All 30 were read:
+Worth one paragraph, because it is the mistake this page was nearly built on. The [[:roadmap]] row that queued it came from a title-and-summary probe for LLM annotation matching 30 papers. All 30 were read: **21 actually point a model at their own data**, seven study LLMs or attacks on them, and two use one as a generator rather than a labeller. So the probe is 70.0% precise — and **13.1% recall**, catching 23 of the 175 papers the ''classification.method'' enum finds.
-^ What the paper actually does with the model ^ Papers ^ Share of 30 ^
-| **annotator** — it assigns labels to items in the paper's own study population | 21 | 70.0% |
-| **object** — the model, or an attack on it, is what is being studied | 7 | 23.3% |
-| **instrument** — it generates artefacts (fuzzing seeds, interface specs), not labels | 2 | 6.7% |
-Precision 70%. **Recall is 13.1%**: the probe catches 23 of the 175 papers the ''classification.method'' enum finds. A title probe is a candidate set — good for deciding whether a page is worth writing, useless as a denominator. Every figure on this page comes from the enum. The per-paper verdicts with their reasons are in ''scripts/annotation_audit.mjs'' and on the provenance page.
+**A title probe is a candidate set: good for deciding whether a page is worth writing, useless as a denominator.** Every figure on this page comes from the enum. The per-paper verdicts, with the deciding reason for each, are on [[provenance:statistics:annotation]].
 ===== What to Report =====
@@ Line 579: / Line 606: @@
   - **What assigned the labels, and what version of it.** A filter list has a commit; a service has a query date; a model has a dated snapshot or a checkpoint. "EasyList" and "GPT-4" are not answers.
-  - **What you validated against, and why it is independent.** One sentence naming the ground truth and one saying what it would take for the ground truth and the classifier to be wrong the same way. **72.9%** of classifying papers name a source; the second sentence is almost never there.
+  - **What you validated against, and why it is independent.** One sentence naming the ground truth and one saying what it would take for the ground truth and the classifier to be wrong the same way. **72.9%** of classifying papers name a source. Nothing in the extraction captures the second sentence and this page did not probe for it, so treat "almost nobody writes it" as an impression rather than a measurement.
   - **How big the validation sample was, how it was drawn, and how it was stratified.** If the positive class is rare, stratify and say so, or your precision interval will be wider than your effect.
   - **Precision //and// recall //per class//, with an interval** — not one global accuracy figure. A macro-average over a class you have 12 examples of is not informative.
@@ Line 586: / Line 613: @@
   - **The categories you dropped and the rule you dropped them by.** A category that failed a precision floor is a finding; a category quietly deleted is not.
   - **If a human coded any of it**, everything on [[Statistics:Interrater agreement#What to Report]] — who, how many, how much was double-coded, which coefficient, resolved how.
-  - **If a model did the first pass**: the model version and date, the **prompt verbatim** as an appendix or artifact, the decoding settings, the run-to-run self-agreement, the residue of unparseable answers, and a human–human coefficient on a subsample coded **blind to the model's output**. Six items; **the corpus median is roughly one of them**.
+  - **If a model did the first pass**: the model version and date, the **prompt verbatim** as an appendix or artifact, the decoding settings, the run-to-run self-agreement, the residue of unparseable answers, and a human–human coefficient on a subsample coded **blind to the model's output**. Three of those six can be measured from full text, and **the median paper supplies none of the three**. See [[Statistics:Interrater agreement#What to Report]] for the coefficient and the blind-coding protocol; this list adds the residue and the joint expectation.
   - **Release the labels.** The annotated sample is small, it is the only part of this a reader can check, and it is the input to somebody else's meta-analysis. See [[:Artifacts]].
@@ Line 593: / Line 620: @@
   * **Two populations, never mixed.** ''CLASSIFIED'' is the **4,439** papers with at least one ''classification[]'' record; ''ANNOTATED'' is the **3,318** with at least one ''humanAnnotation[]'' record; 2,927 papers are in both. Every figure names which. Never "of 5,859 papers".
   * **Papers, never tuples.** A paper with three classifiers using EasyList counts once per row it qualifies for.
-  * **Sentinels are not answers.** ''not-applicable'' is reported as its own row and is never folded into "validated". ''none-reported'' is treated as a real value — it is the paper saying there was no check — and that choice is the difference between the 42.4% and the 29.9% above.
+  * **Sentinels are not answers.** ''not-applicable'' is reported as its own row and is never folded into "validated". ''none-reported'' is treated as a real value — it is the paper saying there was no check. The provenance page writes out why, and what it costs.
   * **The enums are publishable; the free text is not.** ''validation'' and ''target'' are enums. ''method'' is an enum that agrees about 58% run-to-run, so it is a ranking. ''groundTruthSource'' and ''taxonomy'' are free text and are only ever reported as a folded family with the residue printed.
-  * **The ground-truth fold leaves 34.2% of tuples unmapped**, touching 1,577 papers, and the residue is printed in full on the provenance page. It is dominated by one-off experimental setups, so //known by construction// is undercounted. The fold has 21 self-tests including the ordering cases that make the rule order load-bearing.
+  * **The ground-truth fold leaves 34.2% of tuples unmapped**, touching 1,577 papers, and the residue is printed in full on the provenance page. It is a long tail — 93.7% of the distinct unmapped strings appear once — and a hand-classified 30-string sample spreads across every family, so **every share in that table is a floor and none is systematically missing**. The fold has 21 self-tests including the ordering cases that make the rule order load-bearing.
   * **Three of the page's counts are hand audits, not probes** — the 30 roadmap candidates, the 55 temperature candidates and the 31 prompt candidates, each with a per-paper verdict and deciding phrase in ''scripts/annotation_audit.mjs''. The script fails loudly if a probe returns a paper the map does not cover, so the audits cannot go stale silently.
   * **The grouped-split figure is a lower bound from a probe** and is the weakest number here. It supports "rare and almost never stated" and not a count.
-  * **Quotes were checked in bulk.** All **255** evidence quotes on the 177 LLM classification tuples were re-located in the rendering the extractor read: 125 exact, 77 partial, 35 recovered from a second rendering of the PDF, **18 below threshold in both**. The 18 were read by hand; all are elisions, math notation or paraphrase-with-ellipsis, none is a fabrication, and the two behind papers named on this page were confirmed in the source. Full output on the provenance page.
+  * **Quotes were checked in bulk.** All **255** evidence quotes on the LLM classification tuples of those 177 papers (255 tuples, not 255 papers) were re-located in the rendering the extractor read: 125 exact, 77 partial, 35 recovered from a second rendering of the PDF, **18 below threshold in both**. The 18 were read by hand; all are elisions, math notation or paraphrase-with-ellipsis, none is a fabrication, and the two behind papers named on this page were confirmed in the source. Full output on the provenance page.
   * **Venue coverage.** Seven venues — CCS, IMC, NDSS, PoPETs, USENIX Security, TheWebConf, IEEE S&P. **SOUPS, CHI, CSCW, EuroS&P, ACSAC, RAID and AsiaCCS are absent**, and the first three matter here: content analysis is native there and its reporting standards are older. Every figure is a claim about the seven.
   * **2025 and 2026 are provisional.** CCS 2026 and IMC 2026 have not been held; IEEE S&P 2026 and TheWebConf 2026 are under-selected because selection screens on abstracts that are not yet indexed. Any row reaching 2026 is a share of a thin, differently-composed slice.
@@ Line 606: / Line 633: @@
 <WRAP todo>
-  * **Does splitting by site actually change the published numbers, and by how much?** The synthetic demo above shows the mechanism; nobody has re-run a published web classifier under a site-grouped split and reported the delta. Every ingredient is public for at least four corpus papers that released both labels and code. This is a weekend's work and it would recalibrate how every F1 in this literature is read.
+  * **Does splitting by site actually change the published numbers, and by how much?** The synthetic demo above shows the mechanism; nobody has re-run a published web classifier under a site-grouped split and reported the delta. Some corpus papers release both their labels and their code, so the ingredients exist; this page did not enumerate them, and doing so is the first step. It would recalibrate how every F1 in this literature is read.
   * **Is the ''not-applicable'' sentinel really "no validation reported"?** 462 papers are ''not-applicable'' on every tuple, which is 10.4% of the population and the difference between two defensible headline figures. Nobody has read a sample of them to find out how many genuinely did not need a check. A hand audit of 50 would settle it.
   * **Why do borrowed classifiers go unvalidated three times as often as built ones?** The pattern (27.7% vs 84.8%) is clean enough to be a norm rather than an oversight, and the norm may even be defensible for a well-characterised list. Whether it is depends on how far a filter list's precision travels across populations, which [[Programming:Filter Lists]] can measure and has not.
-  * **Does LLM annotation quality hold up on web-measurement targets specifically?** The 175 are overwhelmingly text tasks — posts, policies, reviews, code. On the targets this site is about (a cookie, a request, a script) the counts are 1, 1 and 0. Whether the method transfers to a semi-structured artefact with no natural language in it is untested here and worth testing before the first paper assumes it does.
+  * **Does LLM annotation quality hold up on web-measurement targets specifically?** Judging by ''classification[].target'', the 175 are mostly text tasks — posts, policies, reviews, code. On the targets this site is about the counts are 1 cookie paper, 1 web-request paper and 0 JavaScript papers ([[Design:Website classification#Which task, though]] carries the full per-target table). Whether the method transfers to a semi-structured artefact with no natural language in it is untested here and worth testing before the first paper assumes it does.
   * **Nobody reports the residue.** No field in the extraction captures "how many items the classifier could not label", and a full-text probe for it over the 175 returns mostly discussions of parsing rather than counts. Whether that is because it is rare, because it is embarrassing, or because nobody thinks of it as a number is unresolved — and it is the cheapest missing statistic on this page.
 </WRAP>
@@ Line 617: / Line 644: @@
   * [[Statistics:Interrater agreement]] — the sibling. It owns the coefficients, the rare-class kappa paradox, and what to report when people coded the data. This page owns the validation design; neither repeats the other.
   * [[Design:Website classification]] — the largest single labelling task on this site, its own validation protocol and ''category_validation.py'', and the per-target LLM adoption table all three classification pages read from.
-  * [[Privacy:Cookies]], [[Privacy:Requests]], [[Design:IP classification]], [[Design:Mobile and app measurement]] — the target pages. Their validation rates are in the table above, and three of the four are below the base rate.
+  * [[Privacy:Cookies]], [[Privacy:Requests]], [[Design:IP classification]], [[Design:Mobile and app measurement]] — the target pages. All four of their validation rates are in the table above. **Three of the four — cookies 32.1%, IP addresses 33.6%, web requests 48.9% — are below the 60.1% base rate**, and cookies is the lowest of any target with more than fifty papers; mobile apps, at 66.3%, are the exception and are above it.
   * [[Programming:Filter Lists]] — the borrowed classifier this field trusts most and validates least.
   * [[Statistics:Hypothesis testing]] — the unit-of-analysis problem again, one step downstream: the same site correlation that breaks a random split breaks an independence assumption.