| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| design:website_classification [2026/09/03 22:13] – Generic-review fixes: retract the curated-database 'open sources returning' reading (p>=0.08 on every cut, and the row is mostly mis-filed vendors and media-bias raters - names now printed by the script); add Fisher exact p and n to every drift cut and re karel.kubicek.claude | design:website_classification [2026/09/21 13:45] (current) – Quote bullet: the 6 unmatched LLM tuples are all located in a pypdf rendering; none fails in both renderings. Authored by Claude karel.kubicek.claude |
|---|
| * **Binary or two-way** ("is this a shopping site?"): use an LLM. Validate on a manually annotated sample of a few hundred and cite {[tsai2025_harmful]} for the precedent — that is now a defensible, refereed choice. | * **Binary or two-way** ("is this a shopping site?"): use an LLM. Validate on a manually annotated sample of a few hundred and cite {[tsai2025_harmful]} for the precedent — that is now a defensible, refereed choice. |
| * **A real multi-class taxonomy** (13 Curlie categories, 400 IAB): Homepage2Vec or a vendor API remains the citable baseline. An LLM may well beat it — the only head-to-head says it does — but that head-to-head is one unrefereed preprint that ran the baseline without reporting its threshold, so if you rely on it, re-run the comparison yourself and report both. | * **A real multi-class taxonomy** (13 Curlie categories, 400 IAB): Homepage2Vec or a vendor API remains the citable baseline. An LLM may well beat it — the only head-to-head says it does — but that head-to-head is one unrefereed preprint that ran the baseline without reporting its threshold, so if you rely on it, re-run the comparison yourself and report both. |
| * **Either way**: name the model to the artefact ([[#And almost nobody names a model you could resolve|only 20% of the field does]]), constrain the output to the taxonomy, and report the rejection count. | * **Either way**: name the model to the artefact ([[#And almost nobody names a model you could resolve|fewer than one paper in five does]]), constrain the output to the taxonomy, and report the rejection count. |
| </WRAP> | </WRAP> |
| |
| |
| This is the part most papers skip, and it costs an afternoon. | This is the part most papers skip, and it costs an afternoon. |
| | |
| | <WRAP info> |
| | **[[Statistics:Annotation]] owns this question for every target, not just this one.** It carries the corpus-wide figures the section below cannot state without misstating its own denominator — **29.9%** of the 4,439 papers that classify anything validate none of it, and the rate is worst for cookies, IP addresses and domains — plus what each of the four validation kinds is worth, where ground truth comes from, why a held-out set drawn from one crawl usually is not held out, and what an LLM annotator owes you. The protocol and ''category_validation.py'' below stay here, because a topic-classification reader needs them here; ''holdout_by_site.py'' on that page is the complementary half (grouped splits, rare-class intervals, residue accounting). |
| | </WRAP> |
| |
| The protocol is the one {[ruth2022_world]} used and the one Vallina et al. recommend: draw a stratified random sample, have two people label it independently while **blind to the service's answer**, report agreement, report per-category precision with a confidence interval, and drop the categories that do not clear a floor. | The protocol is the one {[ruth2022_world]} used and the one Vallina et al. recommend: draw a stratified random sample, have two people label it independently while **blind to the service's answer**, report agreement, report per-category precision with a confidence interval, and drop the categories that do not clear a floor. |
| A checklist, derived from what the 330 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply. | A checklist, derived from what the 330 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply. |
| |
| - **Which service or model, and which version.** "McAfee" identified a different company before and after February 2024. "Homepage2Vec" without a commit hash is not reproducible; "GPT-4" without a dated snapshot is not either — and only **20.3% of the 177 corpus papers that use an LLM name a model resolvable to an artefact at all** (see [[#And almost nobody names a model you could resolve]]). For a hosted model, the dated snapshot //and// the date you ran it; for an open-weight one, family, size, instruct-variant and ideally the weights digest. | - **Which service or model, and which version.** "McAfee" identified a different company before and after February 2024. "Homepage2Vec" without a commit hash is not reproducible; "GPT-4" without a dated snapshot is not either — and only **19.4% of the 175 corpus papers that actually use an LLM name a model resolvable to an artefact at all** (see [[#And almost nobody names a model you could resolve]]). For a hosted model, the dated snapshot //and// the date you ran it; for an open-weight one, family, size, instruct-variant and ideally the weights digest. |
| - **The date you queried.** These are live services whose labels change without notice. | - **The date you queried.** These are live services whose labels change without notice. |
| - **Coverage on //your// population**, as a fraction, before any filtering. How many of your domains got no label — and what you did with them. Dropping them silently changes the denominator of every downstream percentage. | - **Coverage on //your// population**, as a fraction, before any filtering. How many of your domains got no label — and what you did with them. Dropping them silently changes the denominator of every downstream percentage. |
| |
| - **It is not statistically distinguishable from noise.** Fisher's exact test, two-sided, on every cut: p = 0.09 for the full 2025–2026 window, 0.08 for the five-venue control, 0.16 for 2025 alone, 0.23 for 2026 alone. Nothing under 0.05. | - **It is not statistically distinguishable from noise.** Fisher's exact test, two-sided, on every cut: p = 0.09 for the full 2025–2026 window, 0.08 for the five-venue control, 0.16 for 2025 alone, 0.23 for 2026 alone. Nothing under 0.05. |
| - **More importantly, the row does not contain open directories.** ''report_website_classification.mjs'' now prints every name behind it. The twelve papers in the recent window name //Tracker Radar Entity List, Cloudflare Radar, SimilarWeb, Symantec SiteReview, AllSides, MediaBias/FactCheck (×2), Science Feedback, IAB taxonomy, Homepage2Vec, NAICSlite// and "predefined source rules (custom)". **Three of those are commercial vendors, one is a model, one is a taxonomy and four are media-bias raters with closed editorial processes. Not one is DMOZ, Curlie, or anything like them.** ''classification.method'' agrees run-to-run only 58% of the time (see [[#Methodology and limitations of these figures]]), and a vendor filed under ''curated-database'' rather than ''third-party-service'' is exactly the slip that produces. So the openness half of this page's central argument has **no** counter-evidence in the recent window; the enum row that looked like counter-evidence is mostly mis-filed vendors. | - **More importantly, the row does not contain open directories.** ''report_website_classification.mjs'' now prints every name behind it. Sorted, the twelve papers are: **three commercial vendors** (Cloudflare Radar, SimilarWeb, Symantec SiteReview); **four media-bias raters** with closed editorial processes (AllSides, and Media Bias/Fact Check in three papers, one of which also cites Science Feedback); **one model** (Homepage2Vec); **two taxonomies rather than label databases** (IAB, NAICSlite); **one paper's own rule set** ("predefined source rules (custom)"); and **one that genuinely is an open, inspectable database** — DuckDuckGo's Tracker Radar Entity List, which is a public repository((https://github.com/duckduckgo/tracker-radar, checked 2026-09-03: public, not archived, last pushed 2026-09-02.)) — but of //tracker entities//, not website topics. **Not one is DMOZ, Curlie or any comparable open topic directory.** ''classification.method'' agrees run-to-run only 58% of the time (see [[#Methodology and limitations of these figures]]), and a vendor filed under ''curated-database'' rather than ''third-party-service'' is exactly the slip that produces. So the openness half of this page's central argument has **no** counter-evidence in the recent window; the enum row that looked like counter-evidence is mostly mis-filed vendors. |
| |
| **The third-party-service drop, by contrast, holds up under a composition control and a significance test.** The obvious way for it to be spurious is composition: CCS 2026 and IMC 2026 have not been held, so the last bucket is drawn from a different mix of venues than 2022–2024. | **The third-party-service drop, by contrast, holds up under a composition control and a significance test.** The obvious way for it to be spurious is composition: CCS 2026 and IMC 2026 have not been held, so the last bucket is drawn from a different mix of venues than 2022–2024. |
| Of those that do validate: 92 papers by manual validation, 21 by cross-validation, 13 by a held-out test set, 10 by comparison to another method. Note the shape — manual spot-checking is the dominant and correct instrument here, because a held-out test set only tells you about the labels you already had. | Of those that do validate: 92 papers by manual validation, 21 by cross-validation, 13 by a held-out test set, 10 by comparison to another method. Note the shape — manual spot-checking is the dominant and correct instrument here, because a held-out test set only tells you about the labels you already had. |
| |
| The 51.2% figure is worth dwelling on: for **website** classification, the share naming a ground-truth source is well below the 72.9% the same extraction measures across all 4,439 papers that classify anything. Website categorisation is reported worse than classification in general, not better. | The 51.2% figure is worth dwelling on: for **website** classification, the share naming a ground-truth source is well below the 72.9% the same extraction measures across all 4,439 papers that classify anything. Website categorisation is reported worse than classification in general, not better. [[Statistics:Annotation#…and the targets reported worst are the web ones]] puts every target side by side against a like-for-like base rate, and website categorisation is mid-table there: cookies, IP addresses and domains are reported worse still. |
| |
| ==== Taxonomies are bespoke, and usually not even sized ==== | ==== Taxonomies are bespoke, and usually not even sized ==== |
| The zero rows are in the table on purpose. **Nothing in these seven venues points an LLM at a JavaScript file or a fingerprinting script** — 0 of the 44 and 0 of the 31 papers that classify those things — which is why [[privacy:javascript]] can still say the method is absent there while this page says it is arriving here. Neither page is out of date; they are different targets. | The zero rows are in the table on purpose. **Nothing in these seven venues points an LLM at a JavaScript file or a fingerprinting script** — 0 of the 44 and 0 of the 31 papers that classify those things — which is why [[privacy:javascript]] can still say the method is absent there while this page says it is arriving here. Neither page is out of date; they are different targets. |
| |
| Two caveats on the table. **''other'' takes 116 of the 177**, so for two thirds of LLM papers the target enum does not describe what they classified. That is less opaque than it sounds: ''other'' carries a free-text ''targetDetail'', it is stated on **all 157** such tuples, and a keyword probe over it for //website, domain, URL, page, script, tracker, cookie, SDK, categor…// returns 16 tuples, **none of which is a website-topic, JavaScript, tracker or cookie classification** — they are IoT device categories, image content, decompiler clusters, MCP server categories, GDPR data categories and the like. So the zero rows below survive the ''other'' bucket at keyword recall. That is a probe, not a read of 116 papers, and it cannot rule out a paper whose ''targetDetail'' uses none of those words. And a share built on a single paper (''cookie'', ''ip-address'', ''web-request'', ''dark-pattern'') is a count dressed as a rate; read those rows as "one paper exists", not as a percentage. | Two caveats on the table. **''other'' takes 116 of the 177**, so for two thirds of LLM papers the target enum does not describe what they classified. That is less opaque than it sounds: ''other'' carries a free-text ''targetDetail'', it is stated on **all 157** such tuples, and a keyword probe over it for //website, domain, URL, page, web, script, tracker, cookie, SDK, categor…// returns 20 tuples, **none of which is a website-topic, JavaScript, tracker or cookie classification** — they are IoT device categories and control pages, image content, decompiler clusters, MCP server categories, GDPR data categories, phishing-personalisation page text and threat-intelligence page triage. So the zero rows below survive the ''other'' bucket at keyword recall. **The probe's width decided that answer and had to be widened once:** its first version used ''\bpage\b'', which does not match the compound "webpage" and silently dropped two tuples, and it had no bare ''web'' at all. Read the printed list rather than the count — a probe is not a read of 116 papers, and it cannot rule out a ''targetDetail'' phrased in none of these words. And a share built on a single paper (''cookie'', ''ip-address'', ''web-request'', ''dark-pattern'') is a count dressed as a rate; read those rows as "one paper exists", not as a percentage. |
| |
| === And almost nobody names a model you could resolve === | === And almost nobody names a model you could resolve === |
| The reproducibility advice in [[#What to Report]] is checkable against these same 177 papers, and the field is failing it — but //how// it fails depends on whether the model is hosted or open-weight, and an earlier version of this table got that backwards. A paper is counted below in the strongest thing it names: | The reproducibility advice in [[#What to Report]] is checkable against these same 177 papers, and the field is failing it — but //how// it fails depends on whether the model is hosted or open-weight, and an earlier version of this table got that backwards. A paper is counted below in the strongest thing it names: |
| |
| ^ What the paper names ^ Papers ^ Share of 177 ^ | The population here is the **175 papers that actually used or produced LLM labels**, not the 177 that mention one — the two papers whose only LLM tuple is a baseline they compared against are excluded, because "did you name the model you used" is not a question about a model you did not use. |
| | **A** a dated **hosted** snapshot — ''gpt-4-turbo-2024-04-09'' | 11 | 6.2% | | |
| | **B** an **open-weight checkpoint** with a size — ''Llama-3.1-70B-Instruct'', ''Qwen2.5-Coder-32B-Instruct'' | 25 | 14.1% | | ^ What the paper names ^ Papers ^ Share of 175 ^ |
| | **A or B — resolvable to an actual artefact** | **36** | **20.3%** | | | **A** a dated **hosted** snapshot — ''gpt-4-turbo-2024-04-09'' | 10 | 5.7% | |
| | **C** a family with no version — ''GPT-4'', ''ChatGPT'', ''Mistral'', ''DeepSeek'' | 130 | 73.4% | | | **B** an **open-weight checkpoint** with a size — ''Llama-3.1-70B-Instruct'', ''Qwen2.5-Coder-32B-Instruct'' | 24 | 13.7% | |
| | **D** no identifiable model — ''an LLM'', ''foundation LLMs'', ''local LLMs'' | 11 | 6.2% | | | **A or B — resolvable to an actual artefact** | **34** | **19.4%** | |
| | | **C** a family with no version — ''GPT-4'', ''ChatGPT'', ''Mistral'', ''DeepSeek'' | 131 | 74.9% | |
| | | **D** no identifiable model — ''an LLM'', ''foundation LLMs'', ''local LLMs'' | 10 | 5.7% | |
| |
| **One paper in five names something you could resolve; nearly three quarters name a family and stop.** ''GPT-4'' is not one model: OpenAI's own deprecations page lists six distinct ''gpt-4'' snapshots — ''-0314'', ''-0613'', ''-1106-preview'', ''-0125-preview'', ''-turbo-preview'' and ''-turbo-2024-04-09''.((Counted from https://developers.openai.com/api/docs/deprecations on 2026-09-03, matching ''gpt-4[-turbo]-<date-or-preview>'' strings. The count is of snapshots named on that page, not of every snapshot that ever existed.)) So a C-bucket paper's result is not reproducible and, more to the point, is not //comparable// to the next C-bucket paper's. | **One paper in five names something you could resolve; three quarters name a family and stop.** ''GPT-4'' is not one model: OpenAI's own deprecations page lists six distinct ''gpt-4'' snapshots — ''-0314'', ''-0613'', ''-1106-preview'', ''-0125-preview'', ''-turbo-preview'' and ''-turbo-2024-04-09''.((Counted from https://developers.openai.com/api/docs/deprecations on 2026-09-03, matching ''gpt-4[-turbo]-<date-or-preview>'' strings. The count is of snapshots named on that page, not of every snapshot that ever existed.)) So a C-bucket paper's result is not reproducible and, more to the point, is not //comparable// to the next C-bucket paper's. |
| |
| **B is the stronger kind of pin, and the field is already better at it than at A.** An open-weight checkpoint stays resolvable; a hosted snapshot lasts as long as the vendor serves it, and ''gpt-4-turbo-2024-04-09'' — the exact string this page held up as good practice — is on OpenAI's published shutdown list for **23 October 2026**.((https://developers.openai.com/api/docs/deprecations, fetched 2026-09-03: ''gpt-4-turbo'' / ''gpt-4-turbo-2024-04-09'' → ''gpt-5.6-sol'', shutdown 23 October 2026. ''gpt-4o-2024-05-13'' and ''gpt-4-0613'', both also used by corpus papers, share that date.)) So the two buckets are not interchangeable and the table does not add them into a single "pinned" figure without saying so. | **B is the stronger kind of pin, and the field is already better at it than at A.** An open-weight checkpoint stays resolvable; a hosted snapshot lasts as long as the vendor serves it, and ''gpt-4-turbo-2024-04-09'' — the exact string this page held up as good practice — is on OpenAI's published shutdown list for **23 October 2026**.((https://developers.openai.com/api/docs/deprecations, fetched 2026-09-03: ''gpt-4-turbo'' / ''gpt-4-turbo-2024-04-09'' → ''gpt-5.6-sol'', shutdown 23 October 2026. ''gpt-4o-2024-05-13'' and ''gpt-4-0613'', both also used by corpus papers, share that date.)) So the two buckets are not interchangeable and the table does not add them into a single "pinned" figure without saying so. |
| * **The task fold is imperfect and its exclusions are visible.** Eight side-channel and website-fingerprinting papers named their classifier only as "a 1D CNN" or "k-NN", leaving no signal in any extracted field; they were caught by matching the paper's own slug and are excluded by an explicit named list rather than by a general rule. The 94 excluded papers are counted in the table above rather than dropped. | * **The task fold is imperfect and its exclusions are visible.** Eight side-channel and website-fingerprinting papers named their classifier only as "a 1D CNN" or "k-NN", leaving no signal in any extracted field; they were caught by matching the paper's own slug and are excluded by an explicit named list rather than by a general rule. The 94 excluded papers are counted in the table above rather than dropped. |
| * **Enum fields versus free text.** Method and validation are enums, stable enough to publish as rough shares (''classification.method'' agrees 58% run-to-run, so read those as a ranking). Service names and taxonomies are free text and are reported as rankings and folded families only. | * **Enum fields versus free text.** Method and validation are enums, stable enough to publish as rough shares (''classification.method'' agrees 58% run-to-run, so read those as a ranking). Service names and taxonomies are free text and are reported as rankings and folded families only. |
| * **Quotes were checked.** Every figure above traces to tuples carrying a verbatim evidence quote; a sample of these was re-located in the source PDFs. Of six quotes checked by hand, two initially "failed" a literal grep and turned out to be intact but split across a two-column break — normalise whitespace before concluding that a quote is not in the paper. The five LLM website-category tuples were re-checked individually on 2026-08-12 and again on 2026-09-03, this time by a committed script rather than by hand: ''scripts/llm_currency_quotecheck.mjs'' checks all 30 LLM tuples across the targets the three classification pages make claims about, and **6 of the 30 could not be matched even after allowing for elisions, punctuation, the PDF's math-italic glyphs and dropped citation markers**. All six are ''privacy-policy'' or ''consent-notice'' tuples that no page on this site quotes; every tuple behind a figure on //this// page matched. Its unedited output is on [[provenance:design:website_classification]]. | * **Quotes were checked.** Every figure above traces to tuples carrying a verbatim evidence quote; a sample of these was re-located in the source PDFs. Of six quotes checked by hand, two initially "failed" a literal grep and turned out to be intact but split across a two-column break — normalise whitespace before concluding that a quote is not in the paper. The five LLM website-category tuples were re-checked individually on 2026-08-12 and again on 2026-09-03, this time by a committed script rather than by hand: ''scripts/llm_currency_quotecheck.mjs'' checks all 30 LLM tuples across the targets the three classification pages make claims about. **Six of the 30 could not be matched against the rendering the extractor read, even after allowing for elisions, punctuation, the PDF's math-italic glyphs and dropped citation markers — and all six are located in an independent ''pypdf'' rendering of the same PDF, so none fails in both renderings.** The six are ''privacy-policy'' or ''consent-notice'' tuples that no page on this site quotes; every tuple behind a figure on //this// page matched against the stored rendering already. Its unedited output is on [[provenance:design:website_classification]]. |
| * **The 2025–2026 reversal was stress-tested, not just re-derived.** The two composition controls in [[#How they classify]] are computed by the same report script and printed with their bucket sizes. The direction survives both; the magnitude is not established. See the box there for what would settle it. | * **The 2025–2026 reversal was stress-tested, not just re-derived.** The two composition controls in [[#How they classify]] are computed by the same report script and printed with their bucket sizes. The direction survives both; the magnitude is not established. See the box there for what would settle it. |
| * **Coverage.** Seven venues only, with 2025 and 2026 incomplete by construction rather than by relevance, so any row that reaches them is a floor; the scope and the selection funnel are on [[literature:corpus]]. Notably, **{[vallina2020_misshapes]} itself is in the venue index but has no extracted full text** — the reference work for this page is not in the population the page measures. | * **Coverage.** Seven venues only, with 2025 and 2026 incomplete by construction rather than by relevance, so any row that reaches them is a floor; the scope and the selection funnel are on [[literature:corpus]]. Notably, **{[vallina2020_misshapes]} itself is in the venue index but has no extracted full text** — the reference work for this page is not in the population the page measures. |
| * [[Privacy:Requests]] — tracker and advertiser classification, where filter lists beat categorisers. | * [[Privacy:Requests]] — tracker and advertiser classification, where filter lists beat categorisers. |
| * [[Privacy:Cookies]], [[Privacy:Fingerprinting]], [[Privacy:JavaScript]] — classification of the other things a crawl collects. | * [[Privacy:Cookies]], [[Privacy:Fingerprinting]], [[Privacy:JavaScript]] — classification of the other things a crawl collects. |
| | * [[Privacy:Policies|Measuring Privacy Policies and Terms]] — labelling a //document// rather than a site, with the same label-set, segmentation and validation problems, and the field's one widely reused annotated corpus (OPP-115). |
| * [[Programming:SimilarWeb]] — API documentation, code and output example for the one service on this page that has a page of its own. | * [[Programming:SimilarWeb]] — API documentation, code and output example for the one service on this page that has a page of its own. |
| * [[Design:Sampling]], [[Statistics:Biases]] (both not yet written) — drawing a sample you can generalise from, and what a categoriser's uneven coverage does to a downstream estimate. | * [[Design:Sampling]], [[Statistics:Biases]] (both not yet written) — drawing a sample you can generalise from, and what a categoriser's uneven coverage does to a downstream estimate. |
