| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| design:website_classification [2026/09/03 22:33] – Re-review fixes: the reproducibility buckets counted a paper on the strength of a model it only COMPARED against, so the population is now the 175 papers that used one and the figures are A 10, B 24, A+B 34 (19.4%), C 131, D 10; widen the target=other pro karel.kubicek.claude | design:website_classification [2026/09/21 13:45] (current) – Quote bullet: the 6 unmatched LLM tuples are all located in a pypdf rendering; none fails in both renderings. Authored by Claude karel.kubicek.claude |
|---|
| |
| This is the part most papers skip, and it costs an afternoon. | This is the part most papers skip, and it costs an afternoon. |
| | |
| | <WRAP info> |
| | **[[Statistics:Annotation]] owns this question for every target, not just this one.** It carries the corpus-wide figures the section below cannot state without misstating its own denominator — **29.9%** of the 4,439 papers that classify anything validate none of it, and the rate is worst for cookies, IP addresses and domains — plus what each of the four validation kinds is worth, where ground truth comes from, why a held-out set drawn from one crawl usually is not held out, and what an LLM annotator owes you. The protocol and ''category_validation.py'' below stay here, because a topic-classification reader needs them here; ''holdout_by_site.py'' on that page is the complementary half (grouped splits, rare-class intervals, residue accounting). |
| | </WRAP> |
| |
| The protocol is the one {[ruth2022_world]} used and the one Vallina et al. recommend: draw a stratified random sample, have two people label it independently while **blind to the service's answer**, report agreement, report per-category precision with a confidence interval, and drop the categories that do not clear a floor. | The protocol is the one {[ruth2022_world]} used and the one Vallina et al. recommend: draw a stratified random sample, have two people label it independently while **blind to the service's answer**, report agreement, report per-category precision with a confidence interval, and drop the categories that do not clear a floor. |
| Of those that do validate: 92 papers by manual validation, 21 by cross-validation, 13 by a held-out test set, 10 by comparison to another method. Note the shape — manual spot-checking is the dominant and correct instrument here, because a held-out test set only tells you about the labels you already had. | Of those that do validate: 92 papers by manual validation, 21 by cross-validation, 13 by a held-out test set, 10 by comparison to another method. Note the shape — manual spot-checking is the dominant and correct instrument here, because a held-out test set only tells you about the labels you already had. |
| |
| The 51.2% figure is worth dwelling on: for **website** classification, the share naming a ground-truth source is well below the 72.9% the same extraction measures across all 4,439 papers that classify anything. Website categorisation is reported worse than classification in general, not better. | The 51.2% figure is worth dwelling on: for **website** classification, the share naming a ground-truth source is well below the 72.9% the same extraction measures across all 4,439 papers that classify anything. Website categorisation is reported worse than classification in general, not better. [[Statistics:Annotation#…and the targets reported worst are the web ones]] puts every target side by side against a like-for-like base rate, and website categorisation is mid-table there: cookies, IP addresses and domains are reported worse still. |
| |
| ==== Taxonomies are bespoke, and usually not even sized ==== | ==== Taxonomies are bespoke, and usually not even sized ==== |
| * **The task fold is imperfect and its exclusions are visible.** Eight side-channel and website-fingerprinting papers named their classifier only as "a 1D CNN" or "k-NN", leaving no signal in any extracted field; they were caught by matching the paper's own slug and are excluded by an explicit named list rather than by a general rule. The 94 excluded papers are counted in the table above rather than dropped. | * **The task fold is imperfect and its exclusions are visible.** Eight side-channel and website-fingerprinting papers named their classifier only as "a 1D CNN" or "k-NN", leaving no signal in any extracted field; they were caught by matching the paper's own slug and are excluded by an explicit named list rather than by a general rule. The 94 excluded papers are counted in the table above rather than dropped. |
| * **Enum fields versus free text.** Method and validation are enums, stable enough to publish as rough shares (''classification.method'' agrees 58% run-to-run, so read those as a ranking). Service names and taxonomies are free text and are reported as rankings and folded families only. | * **Enum fields versus free text.** Method and validation are enums, stable enough to publish as rough shares (''classification.method'' agrees 58% run-to-run, so read those as a ranking). Service names and taxonomies are free text and are reported as rankings and folded families only. |
| * **Quotes were checked.** Every figure above traces to tuples carrying a verbatim evidence quote; a sample of these was re-located in the source PDFs. Of six quotes checked by hand, two initially "failed" a literal grep and turned out to be intact but split across a two-column break — normalise whitespace before concluding that a quote is not in the paper. The five LLM website-category tuples were re-checked individually on 2026-08-12 and again on 2026-09-03, this time by a committed script rather than by hand: ''scripts/llm_currency_quotecheck.mjs'' checks all 30 LLM tuples across the targets the three classification pages make claims about, and **6 of the 30 could not be matched even after allowing for elisions, punctuation, the PDF's math-italic glyphs and dropped citation markers**. All six are ''privacy-policy'' or ''consent-notice'' tuples that no page on this site quotes; every tuple behind a figure on //this// page matched. Its unedited output is on [[provenance:design:website_classification]]. | * **Quotes were checked.** Every figure above traces to tuples carrying a verbatim evidence quote; a sample of these was re-located in the source PDFs. Of six quotes checked by hand, two initially "failed" a literal grep and turned out to be intact but split across a two-column break — normalise whitespace before concluding that a quote is not in the paper. The five LLM website-category tuples were re-checked individually on 2026-08-12 and again on 2026-09-03, this time by a committed script rather than by hand: ''scripts/llm_currency_quotecheck.mjs'' checks all 30 LLM tuples across the targets the three classification pages make claims about. **Six of the 30 could not be matched against the rendering the extractor read, even after allowing for elisions, punctuation, the PDF's math-italic glyphs and dropped citation markers — and all six are located in an independent ''pypdf'' rendering of the same PDF, so none fails in both renderings.** The six are ''privacy-policy'' or ''consent-notice'' tuples that no page on this site quotes; every tuple behind a figure on //this// page matched against the stored rendering already. Its unedited output is on [[provenance:design:website_classification]]. |
| * **The 2025–2026 reversal was stress-tested, not just re-derived.** The two composition controls in [[#How they classify]] are computed by the same report script and printed with their bucket sizes. The direction survives both; the magnitude is not established. See the box there for what would settle it. | * **The 2025–2026 reversal was stress-tested, not just re-derived.** The two composition controls in [[#How they classify]] are computed by the same report script and printed with their bucket sizes. The direction survives both; the magnitude is not established. See the box there for what would settle it. |
| * **Coverage.** Seven venues only, with 2025 and 2026 incomplete by construction rather than by relevance, so any row that reaches them is a floor; the scope and the selection funnel are on [[literature:corpus]]. Notably, **{[vallina2020_misshapes]} itself is in the venue index but has no extracted full text** — the reference work for this page is not in the population the page measures. | * **Coverage.** Seven venues only, with 2025 and 2026 incomplete by construction rather than by relevance, so any row that reaches them is a floor; the scope and the selection funnel are on [[literature:corpus]]. Notably, **{[vallina2020_misshapes]} itself is in the venue index but has no extracted full text** — the reference work for this page is not in the population the page measures. |
| * [[Privacy:Requests]] — tracker and advertiser classification, where filter lists beat categorisers. | * [[Privacy:Requests]] — tracker and advertiser classification, where filter lists beat categorisers. |
| * [[Privacy:Cookies]], [[Privacy:Fingerprinting]], [[Privacy:JavaScript]] — classification of the other things a crawl collects. | * [[Privacy:Cookies]], [[Privacy:Fingerprinting]], [[Privacy:JavaScript]] — classification of the other things a crawl collects. |
| | * [[Privacy:Policies|Measuring Privacy Policies and Terms]] — labelling a //document// rather than a site, with the same label-set, segmentation and validation problems, and the field's one widely reused annotated corpus (OPP-115). |
| * [[Programming:SimilarWeb]] — API documentation, code and output example for the one service on this page that has a page of its own. | * [[Programming:SimilarWeb]] — API documentation, code and output example for the one service on this page that has a page of its own. |
| * [[Design:Sampling]], [[Statistics:Biases]] (both not yet written) — drawing a sample you can generalise from, and what a categoriser's uneven coverage does to a downstream estimate. | * [[Design:Sampling]], [[Statistics:Biases]] (both not yet written) — drawing a sample you can generalise from, and what a categoriser's uneven coverage does to a downstream estimate. |
