Every measurement on this site ends in labels. This third-party domain is advertising; this cookie is analytics; this banner is a dark pattern; this site is News. Something assigned those labels — a filter list, a vendor API, a regex you wrote on a Tuesday, three hundred rows you coded yourself, or since 2024 a language model — and the whole result rests on the claim that the labels mean what their names say.

This page is about the second half of that: how you show the labels are right when the things being labelled are web pages. It is not about kappa — Interrater agreement owns the coefficients and the reliability of hand coding, and this page does not repeat any of it. It is about the design of the check: what you validate against, whether that source is independent of the thing you are checking, why a held-out set drawn from a crawl is usually not held out, and what an LLM annotator owes you that a human one does not.

The short version of the corpus evidence is that this is the least-reported part of the pipeline. Papers say which tool they used and what they found; a large minority never say whether the labels were checked, and the targets where they say it least are exactly the web ones.

• [1Arp, Daniel; Quiring, Erwin; Pendlebury, Feargus; Warnecke, Alexander; Pierazzi, Fabio; Wressnegger, Christian; Cavallaro, Lorenzo; Rieck, Konrad (2022): "Dos and Don'ts of Machine Learning in Computer Security", in: Proceedings of the USENIX Security Symposium. (Link)] (USENIX Security 2022) — Dos and Don'ts of Machine Learning in Computer Security. Ten pitfalls, reviewed across 30 papers from the top-four security venues over ten years. Three of the ten — sampling bias, label inaccuracy, data snooping — are this page's subject. Read §2 and §3 before you draw a test set.
• [2Pendlebury, Feargus; Pierazzi, Fabio; Jordaney, Roberto; Kinder, Johannes; Cavallaro, Lorenzo (2019): "TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time", in: Proceedings of the USENIX Security Symposium. (Link)] (USENIX Security 2019) — TESSERACT. What happens when the split is drawn without respect to time or to the population's structure: “published F1 scores of up to 0.99” on Android malware, which the paper argues are inflated by spatial and temporal bias. The mechanism generalises off malware.
• [3Ruth, Kimberly; Fass, Aurore; Azose, Jonathan; Pearson, Mark; Thomas, Emma; Sadowski, Caitlin; Durumeric, Zakir (2022): "A World Wide View of Browsing the World Wide Web", in: Proceedings of the ACM Internet Measurement Conference, pp. 317-336. (DOI)] (IMC 2022) — the shortest worked example of a defensible validation in this corpus: stratified sample, ten sites per category, keep only categories above 80% accuracy, say what was dropped. One paragraph, and it is the paragraph to imitate.
• [4Munir, Shaoor; Siby, Sandra; Iqbal, Umar; Englehardt, Steven; Shafiq, Zubair; Troncoso, Carmela (2023): "CookieGraph: Understanding and Detecting First-Party Tracking Cookies", pp. 3490–3504. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] (CCS 2023) — the one sentence most web-measurement papers are missing: “We ensure that there is no overlap in the websites used for training and testing in each fold.” See The Held-Out Set That Is Not Held Out.
• [5Pangakis, Nicholas; Wolken, Samuel; Fasching, Neil (2023): "Automated Annotation with Generative AI Requires Validation". arXiv:2306.00176. (Link)] and [6Törnberg, Petter (2024): "Best Practices for Text Annotation with Large Language Models", Sociologica 18(2):67-85. (DOI)] — the argument, and then the protocol, for validating LLM annotations against humans rather than assuming them.

The extraction distinguishes four. They are not interchangeable, and which is available to you is decided by whether a labelled set of your items already exists — which, for most web-measurement targets, it does not.

Rows overlap: the field is multi-valued and a paper with three classifiers appears in three rows. Collapsing to one verdict per paper:

Split by what did the classifying. The share is of papers using that method whose tuple for that method carries one of the four real validation values — so a paper that validates its own model but not the blocklist it also used counts as validated in one row and not the other.

Read the 58.2%, not the 70.1%. Each row above asks whether the tuple for that method carries a validation. “Does any classifier in the paper validate” is a different and easier test, and a paper with three classifiers passes it on one of them while appearing in three rows. The comparable denominator is the (paper, method) cell, and the difference is twelve points — enough to change how big every gap below looks. ¹⁾

The ordering is the finding. You validate what you built and you trust what you downloaded. The three bottom rows — blocklists, third-party services, curated databases — are precisely EasyList, Disconnect, VirusTotal, Cookiepedia and the vendor categorisation APIs, and they are the instruments a web measurement leans on hardest. A borrowed classifier is not a validated one: it was validated, if at all, for somebody else's population, at some other time, against some other ground truth. Filter Lists and Website classification both measure how far that assumption travels, and it is not far.

classification.method agrees about 58% run-to-run between independent extraction runs, so read this table as a ranking with a clear top and bottom rather than as thirteen precise percentages. The gap between 84.8% and 27.7% is far too large to be a labelling artefact; the gap between 55.2% and 54.9% is not a gap.

The same question by what was classified. The right-hand column is the page here that a reader with that target is probably on.

Read the base-rate row first and the small rows last. Of the targets with more than fifty papers, cookies (32.1% of 53), IP addresses (33.6% of 295) and domains (41.0% of 351) are the three worst validated, all around half the 60.1% cell base rate. website-popularity is lower still at 25.0%, but that is four papers out of sixteen and is a count dressed as a rate — as are dark-pattern, consent-notice, javascript and fingerprinting-script. Read every row under about forty papers as “n papers, m of them validated”; the 84.6% for dark patterns is eleven out of thirteen.

A step up somewhere around 2018 and then flat for eight years, inside a window in which the corpus-wide share of papers releasing an artifact link rose from 46.5% in 2018 to 75.9% in 2026 ²⁾. Whatever the artifact-evaluation committees changed, they did not change this.

The aggregate is flat; the mix is not. Rows overlap, since a paper can carry several kinds.

Three readings, in descending order of how much weight they will hold. Manual validation of a sample is and has always been the dominant instrument, rising until about 2018 and flat at roughly half of all classifying papers since; it is the current practice, not a legacy one. Cross-validation is being abandoned — 9.9% in 2018–2021, 3.6% now — which is what you would expect as the field's classifiers stop being models trained on a fixed table and start being services, lists and prompts, none of which has folds to cross. And held-out test sets plateaued after 2018 at around one paper in eight.

2,854 of the 4,439 (64.3%) name a taxonomy and 3,234 (72.9%) name a ground-truth source. The names themselves are free text — 7,114 tuples carrying 6,223 distinct lower-cased strings, so ranking exact strings is meaningless. What can be ranked is the kind of authority, because that is what decides whether the check is independent of the thing being checked. scripts/gt_fold.mjs does that fold; it is an ordered rule list and its residue is printed in full on the provenance page.

The residue is large and is not hidden: 2,434 of the 7,114 tuples (34.2%) fold nowhere, and because a paper carries several tuples that touches 1,577 papers. It is a long tail: 2,068 of the 2,206 distinct unmapped strings (93.7%) appear exactly once, each naming a resource or procedure specific to one paper that no keyword family can be expected to reach — “optical lens scanner labels”, “touchpal call tags”, “indegree median of the population”, “market-purchased IoT devices”. A systematic 30-string sample of it was hand-classified into the family each string would have joined, and it spreads across all of them: 6 existing dataset, 6 by construction, 4 third-party service, 3 human annotation, 3 another automated method, 2 curated list, 2 execution, and 4 that the string alone cannot place. So every share in the table above is a floor, and no single family is systematically missing. The full residue and the sample's verdicts are on annotation.

Two things to take from it. First, half of all stated ground truth in this literature is a human reading a sample; there is no shortcut and the field has not found one. Second, the two non-independent families are small but not zero, and they are the ones to watch: a paper whose ground truth is “majority vote across three LLM judgments” has measured consistency, not correctness.

This is the failure specific to web measurement and it is almost never mentioned. A held-out test set drawn by shuffling rows is not held out when the rows come in groups. Twenty pages of one site share a CMS, a tag manager, a consent vendor and a URL scheme. Split them at random and pages of the same site land on both sides; a classifier that has memorised the site then scores against itself, and the number you report is the number you get on sites you have already seen. The unit of independence on a crawl is the site, not the page — and arguably not even the site, when five hundred sites in your sample run the same three plugins.

[1Arp, Daniel; Quiring, Erwin; Pendlebury, Feargus; Warnecke, Alexander; Pierazzi, Fabio; Wressnegger, Christian; Cavallaro, Lorenzo; Rieck, Konrad (2022): "Dos and Don'ts of Machine Learning in Computer Security", in: Proceedings of the USENIX Security Symposium. (Link)] name this family data snooping and distinguish test, temporal and selective snooping; [2Pendlebury, Feargus; Pierazzi, Fabio; Jordaney, Roberto; Kinder, Johannes; Cavallaro, Lorenzo (2019): "TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time", in: Proceedings of the USENIX Security Symposium. (Link)] show what the temporal version alone is worth in published F1. Neither is about the web, and that is the point: the mechanism is general and the web makes it worse, because sites are more internally uniform than almost any other population this field samples.

How often does a web-measurement paper say it split by site? Of the 300 web-platform papers whose classification carries a held-out test set or cross-validation, a full-text probe for any grouped-split language returns 10 candidates, of which 7 really split or evaluate across a group — 2.3%. That is a lower bound in both directions: a paper that split by site without writing a sentence about it is invisible to any probe of this shape, and so is one that phrased it differently. What the number supports is “this is rare and almost never stated”, not “only seven papers did it”. The regex and all ten candidates with the deciding phrase are on annotation.

[4Munir, Shaoor; Siby, Sandra; Iqbal, Umar; Englehardt, Steven; Shafiq, Zubair; Troncoso, Carmela (2023): "CookieGraph: Understanding and Detecting First-Party Tracking Cookies", pp. 3490–3504. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] is the model sentence. Its ten-fold cross-validation is site-disjoint and it says so in one clause: “We ensure that there is no overlap in the websites used for training and testing in each fold.” That clause costs nothing and is the difference between a number a reviewer can use and one they cannot.

The second half of the same problem. “3.4% of pages do X” means a 1,000-item validation sample holds about 34 positives, and a precision computed on 34 items is a range, not a value. At 34/34 correct the 95% Wilson interval is still 10 points wide; at 25/34 it is 29 points wide. A bare point estimate from a sample that size is not a measurement. Reporting the interval costs one line of arithmetic; narrowing it costs annotation, and the cheap way to spend it is to stratify — oversample the predicted positives and weight back — rather than to draw a bigger uniform sample in which 96.6% of the labels tell you nothing about precision.

The same applies to coverage. Items the classifier refused, returned nothing for, or answered outside your taxonomy are a result — they are the closest thing you have to a measurement of how well the instrument understood the task. Folding them into the negatives inflates precision and deflates recall at the same time, and dropping them silently changes the denominator of everything downstream.

The script below does the three things this section is about and nothing else: it splits by a grouping column instead of by row, it reports precision and recall with Wilson intervals rather than point estimates, and it refuses to let the residue disappear. It is deliberately narrow — for the stratified-sample-and-blind-double-annotation protocol, use category_validation.py on Validating Your Labels, which is the complementary half and is not repeated here.

holdout_by_site.py

#!/usr/bin/env python3
"""Score a label validation the way a web measurement makes you score it.
 
Two things go wrong when you validate labels over crawled pages, and neither is
a statistics-textbook problem:
 
  1. YOUR HELD-OUT SET IS NOT HELD OUT. Draw it by shuffling rows and pages
     from the same site land on both sides of the split. A classifier that has
     memorised one site's template then scores against itself. The unit of
     independence is the site, not the page -- and usually not even the site,
     if the sites share a CMS, a tag manager or a consent vendor.
 
  2. THE POSITIVE CLASS IS RARE. "3.4% of pages do X" means a 1,000-item sample
     contains about 34 positives, and a precision computed on 34 items is a
     range, not a number. Report the interval or do not report the precision.
 
And one thing that is not an error at all but is usually deleted: items the
classifier refused, returned nothing for, or answered outside your taxonomy.
That residue is a result. This script will not let you drop it silently.
 
    # score your own labels
    python3 holdout_by_site.py score labels.csv --positive tracking --seed 20260911
 
    # show what a page-level split does to a site-structured population
    python3 holdout_by_site.py demo --seed 20260911
 
    # verify the arithmetic (no dependencies to trust)
    python3 holdout_by_site.py selftest
 
`labels.csv` is `item,site,truth,prediction` with one row per labelled item.
Use the literal prediction `none` for items the classifier produced no usable
answer for. `truth` is your hand label; leave it empty for items you did not
hand-label and they are excluded from scoring (and counted, and reported).
 
Standard library only: no numpy, no scipy, no sklearn, no network.
"""
 
import argparse
import csv
import hashlib
import math
import random
import sys
from collections import Counter, defaultdict
 
Z95 = 1.959963984540054  # two-sided normal quantile at 95%
 
 
def wilson(k, n, z=Z95):
    """Wilson score interval for a binomial proportion.
 
    Preferred over the Wald interval here because the counts on a rare class
    are small and Wald gives (0.0, 0.0) at k=0, which reads as certainty.
    """
    if n == 0:
        return (float("nan"), float("nan"))
    p = k / n
    d = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / d
    half = (z / d) * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (max(0.0, centre - half), min(1.0, centre + half))
 
 
def split_by_site(rows, frac, seed):
    """Grouped split: every item of a site goes to the same side.
 
    Hashing the site name rather than shuffling it means the split is stable
    when you add sites later -- a site keeps its side. That matters for a
    longitudinal crawl, where the alternative is a test set that silently
    changes composition between waves.
    """
    train, test = [], []
    for r in rows:
        h = hashlib.sha256(f"{seed}:{r['site']}".encode()).digest()
        u = int.from_bytes(h[:8], "big") / 2**64
        (test if u < frac else train).append(r)
    return train, test
 
 
def split_by_item(rows, frac, seed):
    """The split almost every paper actually does. Here to be compared against."""
    rng = random.Random(seed)
    shuffled = list(rows)
    rng.shuffle(shuffled)
    cut = int(len(shuffled) * frac)
    return shuffled[cut:], shuffled[:cut]
 
 
def score(rows, positive):
    """Precision / recall on `positive`, with the residue kept visible.
 
    An item whose prediction is `none` is NOT a negative prediction. It is an
    abstention, and folding it into the negatives quietly inflates precision
    while deflating recall. It gets its own counts.
    """
    labelled = [r for r in rows if r["truth"]]
    unlabelled = len(rows) - len(labelled)
    residue = [r for r in labelled if r["prediction"] == "none"]
    scored = [r for r in labelled if r["prediction"] != "none"]
 
    tp = sum(1 for r in scored if r["prediction"] == positive and r["truth"] == positive)
    fp = sum(1 for r in scored if r["prediction"] == positive and r["truth"] != positive)
    fn = sum(1 for r in scored if r["prediction"] != positive and r["truth"] == positive)
    fn_residue = sum(1 for r in residue if r["truth"] == positive)
 
    prec_n, rec_n = tp + fp, tp + fn
    return {
        "items": len(rows),
        "unlabelled": unlabelled,
        "labelled": len(labelled),
        "residue": len(residue),
        "sites": len({r["site"] for r in rows}),
        "positives": sum(1 for r in labelled if r["truth"] == positive),
        "tp": tp, "fp": fp, "fn": fn, "fn_residue": fn_residue,
        "precision": (tp / prec_n) if prec_n else float("nan"),
        "precision_ci": wilson(tp, prec_n),
        "precision_n": prec_n,
        # Recall counts the abstentions as misses, because they are: an item
        # you failed to label is an item you failed to find.
        "recall": (tp / (rec_n + fn_residue)) if (rec_n + fn_residue) else float("nan"),
        "recall_ci": wilson(tp, rec_n + fn_residue),
        "recall_n": rec_n + fn_residue,
    }
 
 
def fmt(s, title):
    ci = lambda t: f"[{t[0]:.3f}, {t[1]:.3f}]"
    return "\n".join([
        f"--- {title}",
        f"  items {s['items']}  sites {s['sites']}  hand-labelled {s['labelled']}"
        f"  not hand-labelled {s['unlabelled']}",
        f"  positives in the labelled set {s['positives']}"
        f"  ({100 * s['positives'] / s['labelled']:.1f}%)" if s["labelled"] else "",
        f"  RESIDUE (no usable prediction) {s['residue']}"
        f"  -- of which truly positive {s['fn_residue']}",
        f"  precision {s['precision']:.3f}  95% CI {ci(s['precision_ci'])}  on n={s['precision_n']}",
        f"  recall    {s['recall']:.3f}  95% CI {ci(s['recall_ci'])}  on n={s['recall_n']}",
    ])
 
 
# ---------------------------------------------------------------- demo
def make_population(n_sites=120, pages_per_site=20, seed=1):
    """Sites with a fingerprint, pages that inherit it, and a per-page label.
 
    The label is mostly a property of the SITE (one CMS, one tag manager, one
    consent vendor decided it once) and partly of the page. The features a
    classifier gets are the site fingerprint -- which is memorisable and tells
    it nothing transferable -- and one weak page-level signal that does
    generalise. That is the shape of a real web-measurement label, and it is
    why the two splits disagree.
    """
    rng = random.Random(seed)
    rows = []
    for s in range(n_sites):
        fingerprint = [rng.gauss(0, 1) for _ in range(4)]  # memorisable, useless off-site
        site_offset = rng.gauss(0, 1)  # the part of the label the site decides
        for p in range(pages_per_site):
            signal = rng.gauss(0, 1)  # the part that generalises
            truth = "yes" if site_offset + 0.6 * signal > 0 else "no"
            rows.append({
                "item": f"s{s}p{p}",
                "site": f"site{s}",
                "features": [f + rng.gauss(0, 0.05) for f in fingerprint] + [signal],
                "truth": truth,
            })
    return rows
 
 
def nearest_neighbour_classifier(train, test):
    """1-NN in feature space: the simplest classifier that can memorise.
 
    Not a straw man. A bag-of-words model over page HTML, a URL-pattern
    heuristic, and a few-shot LLM prompt with retrieved examples are all
    memorisers with extra steps, and they all break the same way.
    """
    out = []
    for r in test:
        best, best_d = None, float("inf")
        for t in train:
            d = sum((a - b) ** 2 for a, b in zip(r["features"], t["features"]))
            if d < best_d:
                best, best_d = t, d
        out.append({**r, "prediction": best["truth"] if best else "none"})
    return out
 
 
def demo(seed):
    rows = make_population(seed=seed)
    n_sites = len({r["site"] for r in rows})
    base = sum(1 for r in rows if r["truth"] == "yes") / len(rows)
    print(f"synthetic population: {len(rows)} pages over {n_sites} sites, seed {seed}")
    print(f"positive rate in the whole population: {base:.3f}\n")
    results = {}
    for name, splitter in (("page-level random split (what most papers do)", split_by_item),
                           ("site-grouped split (what the unit of analysis requires)", split_by_site)):
        train, test = splitter(rows, 0.3, seed)
        shared = len({r["site"] for r in train} & {r["site"] for r in test})
        scored = score(nearest_neighbour_classifier(train, test), "yes")
        results[name] = scored
        print(fmt(scored, name))
        print(f"  sites appearing on BOTH sides of the split: {shared}\n")
    print("--- what a rare positive class does to the interval, before anything goes wrong")
    print("    a 1,000-item validation sample at a 3.4% positive rate holds ~34 positives;")
    print("    precision computed on those 34 predicted-positive items:")
    for k in (34, 30, 25):
        lo, hi = wilson(k, 34)
        print(f"      {k}/34 correct -> precision {k / 34:.3f}  95% CI [{lo:.3f}, {hi:.3f}]"
              f"  (width {100 * (hi - lo):.0f} points)")
    print("    A bare point estimate from a sample this size is not a measurement.\n")
 
    a, b = list(results.values())
    print(f"Precision falls from {a['precision']:.3f} to {b['precision']:.3f} and the intervals do not\n"
          f"overlap. That gap is the optimism a page-level split buys. It is not a\n"
          f"property of this toy: it appears whenever the unit you split on is smaller\n"
          f"than the unit your features are constant over -- which, on a crawl, it\n"
          f"almost always is.")
 
 
# ---------------------------------------------------------------- selftest
def selftest():
    """Verify the arithmetic against its own definition, not against a library.
 
    The Wilson endpoints are exactly the two roots of |p_hat - p| = z*sqrt(p(1-p)/n),
    so each endpoint can be checked by substitution. That is a stronger test
    than comparing to a hard-coded number, and it fails if the formula is
    mistyped in any term.
    """
    ok = True
    for k, n in [(8, 10), (41, 175), (1, 100), (26, 175), (3, 3), (0, 50)]:
        lo, hi = wilson(k, n)
        p_hat = k / n
        for end in (lo, hi):
            if end in (0.0, 1.0):
                continue  # clamped; the identity does not apply
            lhs = abs(p_hat - end)
            rhs = Z95 * math.sqrt(end * (1 - end) / n)
            if abs(lhs - rhs) > 1e-9:
                print(f"FAIL wilson({k},{n}) endpoint {end}: {lhs} != {rhs}")
                ok = False
    lo, hi = wilson(0, 50)
    if not (lo == 0.0 and 0.05 < hi < 0.09):
        print(f"FAIL wilson(0,50) = {(lo, hi)}; a zero count must not give a zero-width interval")
        ok = False
 
    # An abstention must not be scored as a negative prediction.
    rows = [
        {"item": "a", "site": "s1", "truth": "yes", "prediction": "yes"},
        {"item": "b", "site": "s1", "truth": "yes", "prediction": "none"},
        {"item": "c", "site": "s2", "truth": "no", "prediction": "no"},
    ]
    s = score(rows, "yes")
    if not (s["residue"] == 1 and s["fn_residue"] == 1 and s["precision"] == 1.0 and abs(s["recall"] - 0.5) < 1e-12):
        print(f"FAIL abstention handling: {s}")
        ok = False
 
    # The grouped split must never put a site on both sides.
    pop = make_population(n_sites=50, pages_per_site=10, seed=7)
    tr, te = split_by_site(pop, 0.3, 7)
    if {r["site"] for r in tr} & {r["site"] for r in te}:
        print("FAIL split_by_site leaked a site across the split")
        ok = False
    if not tr or not te:
        print("FAIL split_by_site produced an empty side")
        ok = False
 
    print("selftest: PASS" if ok else "selftest: FAIL")
    return 0 if ok else 1
 
 
def main():
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    sub = ap.add_subparsers(dest="cmd", required=True)
    sc = sub.add_parser("score")
    sc.add_argument("labels")
    sc.add_argument("--positive", required=True)
    sc.add_argument("--seed", type=int, default=0)
    sc.add_argument("--test-fraction", type=float, default=0.3)
    d = sub.add_parser("demo")
    d.add_argument("--seed", type=int, default=20260911)
    sub.add_parser("selftest")
    a = ap.parse_args()
 
    if a.cmd == "selftest":
        return selftest()
    if a.cmd == "demo":
        demo(a.seed)
        return 0
 
    with open(a.labels, newline="", encoding="utf-8") as fh:
        rows = list(csv.DictReader(fh))
    for need in ("item", "site", "truth", "prediction"):
        if rows and need not in rows[0]:
            sys.exit(f"{a.labels}: missing required column {need!r}")
    print(fmt(score(rows, a.positive), f"all labelled items ({a.labels})"))
    train, test = split_by_site(rows, a.test_fraction, a.seed)
    print()
    print(fmt(score(test, a.positive), f"site-grouped held-out {a.test_fraction:.0%} (train {len(train)})"))
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main())

Its demo subcommand builds a synthetic population with the structure described above — sites with a memorisable fingerprint, a label that is mostly a property of the site — and runs the same 1-NN classifier under both splits. Real, unedited output:

holdout_by_site-demo.txt

synthetic population: 2400 pages over 120 sites, seed 20260911
positive rate in the whole population: 0.510
 
--- page-level random split (what most papers do)
  items 720  sites 120  hand-labelled 720  not hand-labelled 0
  positives in the labelled set 371  (51.5%)
  RESIDUE (no usable prediction) 0  -- of which truly positive 0
  precision 0.964  95% CI [0.940, 0.979]  on n=365
  recall    0.949  95% CI [0.921, 0.967]  on n=371
  sites appearing on BOTH sides of the split: 120
 
--- site-grouped split (what the unit of analysis requires)
  items 840  sites 42  hand-labelled 840  not hand-labelled 0
  positives in the labelled set 497  (59.2%)
  RESIDUE (no usable prediction) 0  -- of which truly positive 0
  precision 0.659  95% CI [0.604, 0.710]  on n=308
  recall    0.408  95% CI [0.366, 0.452]  on n=497
  sites appearing on BOTH sides of the split: 0
 
--- what a rare positive class does to the interval, before anything goes wrong
    a 1,000-item validation sample at a 3.4% positive rate holds ~34 positives;
    precision computed on those 34 predicted-positive items:
      34/34 correct -> precision 1.000  95% CI [0.898, 1.000]  (width 10 points)
      30/34 correct -> precision 0.882  95% CI [0.734, 0.953]  (width 22 points)
      25/34 correct -> precision 0.735  95% CI [0.569, 0.854]  (width 29 points)
    A bare point estimate from a sample this size is not a measurement.
 
Precision falls from 0.964 to 0.659 and the intervals do not
overlap. That gap is the optimism a page-level split buys. It is not a
property of this toy: it appears whenever the unit you split on is smaller
than the unit your features are constant over -- which, on a crawl, it
almost always is.

The direction of that gap is general; its size is a parameter of the toy. 0.964 → 0.659 follows from how much of the label the site decides (here 1.0 against 0.6) and from a classifier whose only real feature is the site. On your data it could be two points or forty, and the only way to find out is to run both splits — which costs one extra line and is the first open question below.

selftest checks the Wilson endpoints against their own defining equation (each endpoint must satisfy |p̂ − p| = z·√(p(1−p)/n)), rather than against a hard-coded number, so a mistyped term in any position fails. It also checks that an abstention is not scored as a negative and that the grouped split never leaks a group. There is no numpy, scipy or sklearn: it runs where your data is.

This is the part that changed after 2023, and the part where the corpus is thinnest. Read the direction as solid and the levels as provisional.

177 papers carry a classification tuple with method == “llm”; 175 of them used or produced the labels rather than only comparing against an LLM baseline. The 175 is the population for everything below.

³⁾ That page also carries the per-target breakdown, which matters more than the curve: adoption is extremely uneven, and for a cookie or a JavaScript file it has barely started. Do not read “one classifying paper in five” as “there is a baseline for my task”.

Against that, the good news, and it is genuinely good:

83.4% against a 58.2% like-for-like base rate. And on agreement: 39.1% of the 151 LLM papers that also hand-code name an agreement metric, against 15.4% of all 3,318 papers that hand-code anything. (The 33.7%-of-175 row above understates it, because 24 of the 175 hand-code nothing and so have no coefficient to report.) The likeliest explanation is structural rather than cultural: when a model does the first pass, the human coding is the validation, so a paper without a number has nothing to report. That is an interpretation of a difference between two groups, not a tested claim. When the Second Rater Is a Language Model reaches it from the annotation side using a full-text matcher rather than this page's enum, and lands in the same place — about 34% against about 15% — which is the reassurance that neither number is an artefact of one matcher. That page owns which coefficient it should be.

What are the labels checked against? 138 of the 175 (78.9%) name a ground truth on an LLM tuple, and 115 of those 138 (83.3%) name human annotation of a sample — a far higher share than the 52.0% corpus-wide. Two papers' ground truth for an LLM is another model's output.

[7Cory, Thomas; Rieder, Wolf; Krämer, Julia; Raschke, Philip; Herbke, Patrick; Küpper, Axel (2026): "Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models", Proceedings on Privacy Enhancing Technologies 2026(1):509-528. (DOI)] (PoPETs 2026) is the worked example to copy: a hand-built gold set of “200 privacy policies from Android applications”, split among six experts with “two experts randomly assigned to each pol[icy]”, and the LLM pipeline scored against it. [8Xiong, Shenping; Wang, Xutong; Jin, Ze; Liu, Xinyu; Wang, Haoqiang; Chen, Zhen; Tan, Ru; Liu, Qixu (2026): "TGNN: Enhancing Pixel Tracking Detection via LLM-driven Annotation and GAT-powered Structural Representation", in: Proceedings of the ACM Web Conference. (DOI)] (TheWebConf 2026) is the closest thing in the corpus to a web-measurement template: a 20,000-request ground truth, heuristic-labelled and then validated by human experts (“human experts validate the ground truth dataset”), an LLM labelling component scored against it, and EasyList compared against the same set. It is also the page's own worked warning, twice over. The LLM component's F1 of 98.17% is computed “on the ground truth dataset” — the whole 20,000, with no held-out portion for the labelling step at all — while the extraction records that tuple as held-out-test-set. And the 80/20 split the paper does describe (“into training and testing sets with a ratio of 80% and 20%”) belongs to the downstream graph model that consumes those labels; it says nothing about whether requests from one site land on both sides. Reading what a validation enum actually refers to is not optional, and this one is wrong in the source data this page is built on.

The reproducibility half is measured over the same 175, by full-text probe. Two rows are hand-audited because the probes over-fire; the rest are lower bounds and are marked.

Three quarters of the papers that let a model assign labels do not say at what temperature, six in seven do not say the prompt is anywhere, and — the figure that matters, because these are not the same papers — the median paper supplies none of the three. Two of 175 supply all three. Note what the temperature row is not saying: the word “temperature” appears in 55 of the 175, but in eleven of those it is the colour temperature of a smart bulb, a weather API payload, the softmax τ, a simulated-annealing schedule, or a cited paper's title. A probe that had reported 55 would have been 34% too high, which is why the two figures above that carry a percentage were read by hand.

The prompt figure is a lower bound in both directions and should be quoted that way: a prompt printed in an appendix figure with no sentence pointing at it is invisible. One of the 31 candidates matched because it says it is deliberately withholding its prompts — those are attack prompts rather than annotation prompts, so it is not a counter-example to the norm, but it is a reminder that “the prompt is in the appendix” and “the prompt exists” are different claims.

Worth one paragraph, because it is the mistake this page was nearly built on. The roadmap row that queued it came from a title-and-summary probe for LLM annotation matching 30 papers. All 30 were read: 21 actually point a model at their own data, seven study LLMs or attacks on them, and two use one as a generator rather than a labeller. So the probe is 70.0% precise — and 13.1% recall, catching 23 of the 175 papers the classification.method enum finds.

A title probe is a candidate set: good for deciding whether a page is worth writing, useless as a denominator. Every figure on this page comes from the enum. The per-paper verdicts, with the deciding reason for each, are on annotation.

A checklist, derived from what the corpus leaves out. Everything here is one or two clauses in a methods section.

1. What assigned the labels, and what version of it. A filter list has a commit; a service has a query date; a model has a dated snapshot or a checkpoint. “EasyList” and “GPT-4” are not answers.
2. What you validated against, and why it is independent. One sentence naming the ground truth and one saying what it would take for the ground truth and the classifier to be wrong the same way. 72.9% of classifying papers name a source. Nothing in the extraction captures the second sentence and this page did not probe for it, so treat “almost nobody writes it” as an impression rather than a measurement.
3. How big the validation sample was, how it was drawn, and how it was stratified. If the positive class is rare, stratify and say so, or your precision interval will be wider than your effect.
4. Precision and recall per class, with an interval — not one global accuracy figure. A macro-average over a class you have 12 examples of is not informative.
5. The grouping unit of your split, explicitly: by page, by site, by publisher, by time. If you split at random, say that you did, and say what the sites' internal correlation might be worth. Around 2% of the web papers that use a held-out set say anything about this at all.
6. The residue. How many items got no label, or a label outside your taxonomy, and what you did with them. As a count and as a share of the population before filtering.
7. The categories you dropped and the rule you dropped them by. A category that failed a precision floor is a finding; a category quietly deleted is not.
8. If a human coded any of it, everything on What to Report — who, how many, how much was double-coded, which coefficient, resolved how.
9. If a model did the first pass: the model version and date, the prompt verbatim as an appendix or artifact, the decoding settings, the run-to-run self-agreement, the residue of unparseable answers, and a human–human coefficient on a subsample coded blind to the model's output. Three of those six can be measured from full text, and the median paper supplies none of the three. See What to Report for the coefficient and the blind-coding protocol; this list adds the residue and the joint expectation.
10. Release the labels. The annotated sample is small, it is the only part of this a reader can check, and it is the input to somebody else's meta-analysis. See Artifacts.

• Two populations, never mixed. CLASSIFIED is the 4,439 papers with at least one classification[] record; ANNOTATED is the 3,318 with at least one humanAnnotation[] record; 2,927 papers are in both. Every figure names which. Never “of 5,859 papers”.
• Papers, never tuples. A paper with three classifiers using EasyList counts once per row it qualifies for.
• Sentinels are not answers. not-applicable is reported as its own row and is never folded into “validated”. none-reported is treated as a real value — it is the paper saying there was no check. The provenance page writes out why, and what it costs.
• The enums are publishable; the free text is not. validation and target are enums. method is an enum that agrees about 58% run-to-run, so it is a ranking. groundTruthSource and taxonomy are free text and are only ever reported as a folded family with the residue printed.
• The ground-truth fold leaves 34.2% of tuples unmapped, touching 1,577 papers, and the residue is printed in full on the provenance page. It is a long tail — 93.7% of the distinct unmapped strings appear once — and a hand-classified 30-string sample spreads across every family, so every share in that table is a floor and none is systematically missing. The fold has 21 self-tests including the ordering cases that make the rule order load-bearing.
• Three of the page's counts are hand audits, not probes — the 30 roadmap candidates, the 55 temperature candidates and the 31 prompt candidates, each with a per-paper verdict and deciding phrase in scripts/annotation_audit.mjs. The script fails loudly if a probe returns a paper the map does not cover, so the audits cannot go stale silently.
• The grouped-split figure is a lower bound from a probe and is the weakest number here. It supports “rare and almost never stated” and not a count.
• Quotes were checked in bulk. All 255 evidence quotes on the LLM classification tuples of those 177 papers (255 tuples, not 255 papers) were re-located in the rendering the extractor read: 125 exact, 77 partial, 35 recovered from a second rendering of the PDF, 18 below threshold in both. The 18 were read by hand; all are elisions, math notation or paraphrase-with-ellipsis, none is a fabrication, and the two behind papers named on this page were confirmed in the source. Full output on the provenance page.
• Venue coverage. Seven venues — CCS, IMC, NDSS, PoPETs, USENIX Security, TheWebConf, IEEE S&P. SOUPS, CHI, CSCW, EuroS&P, ACSAC, RAID and AsiaCCS are absent, and the first three matter here: content analysis is native there and its reporting standards are older. Every figure is a claim about the seven.
• 2025 and 2026 are provisional. CCS 2026 and IMC 2026 have not been held; IEEE S&P 2026 and TheWebConf 2026 are under-selected because selection screens on abstracts that are not yet indexed. Any row reaching 2026 is a share of a thin, differently-composed slice.
• The report script is scripts/report_annotation.mjs; every figure above is in its unedited output. The full query log, the fold's residue, the three hand-audit maps, the quote check and what could not be established are on annotation. Corpus-level caveats are on Corpus.

• Interrater agreement — the sibling. It owns the coefficients, the rare-class kappa paradox, and what to report when people coded the data. This page owns the validation design; neither repeats the other.
• Website classification — the largest single labelling task on this site, its own validation protocol and category_validation.py, and the per-target LLM adoption table all three classification pages read from.
• Cookies, Requests, IP classification, Mobile and app measurement — the target pages. All four of their validation rates are in the table above and all four are below the base rate; cookies is the lowest of any target with more than fifty papers.
• Filter Lists — the borrowed classifier this field trusts most and validates least.
• Hypothesis testing — the unit-of-analysis problem again, one step downstream: the same site correlation that breaks a random split breaks an independence assumption.
• Biases — an unreliable label set is a measurement-error problem before it is anything else.
• Study preregistration — where “we set the precision floor in advance” becomes checkable rather than assertable.
• Artifacts — release the labels, the codebook and the prompt.
• Provenance of this page's figures.

[1]

Arp, Daniel; Quiring, Erwin; Pendlebury, Feargus; Warnecke, Alexander; Pierazzi, Fabio; Wressnegger, Christian; Cavallaro, Lorenzo; Rieck, Konrad (2022): "Dos and Don'ts of Machine Learning in Computer Security", in: Proceedings of the USENIX Security Symposium. (Link)

[2]

Pendlebury, Feargus; Pierazzi, Fabio; Jordaney, Roberto; Kinder, Johannes; Cavallaro, Lorenzo (2019): "TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time", in: Proceedings of the USENIX Security Symposium. (Link)

[3]

Ruth, Kimberly; Fass, Aurore; Azose, Jonathan; Pearson, Mark; Thomas, Emma; Sadowski, Caitlin; Durumeric, Zakir (2022): "A World Wide View of Browsing the World Wide Web", in: Proceedings of the ACM Internet Measurement Conference, pp. 317-336. (DOI)

[4]

Munir, Shaoor; Siby, Sandra; Iqbal, Umar; Englehardt, Steven; Shafiq, Zubair; Troncoso, Carmela (2023): "CookieGraph: Understanding and Detecting First-Party Tracking Cookies", pp. 3490–3504. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)

[5]

Pangakis, Nicholas; Wolken, Samuel; Fasching, Neil (2023): "Automated Annotation with Generative AI Requires Validation". arXiv:2306.00176. (Link)

[6]

Törnberg, Petter (2024): "Best Practices for Text Annotation with Large Language Models", Sociologica 18(2):67-85. (DOI)

[7]

Cory, Thomas; Rieder, Wolf; Krämer, Julia; Raschke, Philip; Herbke, Patrick; Küpper, Axel (2026): "Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models", Proceedings on Privacy Enhancing Technologies 2026(1):509-528. (DOI)

[8]

Xiong, Shenping; Wang, Xutong; Jin, Ze; Liu, Xinyu; Wang, Haoqiang; Chen, Zhen; Tan, Ru; Liu, Qixu (2026): "TGNN: Enhancing Pixel Tracking Detection via LLM-driven Annotation and GAT-powered Structural Representation", in: Proceedings of the ACM Web Conference. (DOI)