Differences

This shows you the differences between two versions of the page.

--- design:website_classification [2026/08/07 11:10] – Correct the full-corpus service count: nine label sources were queried across all 4.4M domains, not eight (I had derived it as 11 minus 3, which double-subtracts Trend Micro). Name the nine, and say plainly that 'how many services' is ambiguous in the pap karel.kubicek.claude
+++ design:website_classification [2026/08/27 14:39] (current) – CrUX licence: Tranco homepage CC BY-SA 4.0, methodology silent, Google CC BY 4.0. Authored by Claude karel.kubicek.claude
@@ Line 5: / Line 5: @@
 This page is about turning a domain into a category you can defend. Its companion [[Design:Website Selection]] is about which domains to put on the list in the first place; popularity ranking lives there, category and company data live here.
-Before anything else: **"classify a website" is not one measurement.** It is at least five, they use different data, and they are not equally reliable.
+Before anything else: **"classify a website" is not one measurement.** It is at least six, they use different data, and they are not equally reliable.
 ^ Question ^ Where the answer comes from ^ How good it is ^ Free at scale? ^
 | What //topic// is this site about? | Commercial categorisation vendors, Curlie, an open model, an LLM | The subject of this page. Vendor coverage ranges from 94% to 0.5%; vendors agree with each other poorly | Rarely |
+| Which //country// is it aimed at? | TLD, CrUX country lists (via Tranco), site language, host IP, an LLM | None of these is a target-audience label; they are proxies that disagree. See [[#Target country is not topic]] | Yes |
 | What //industry// is the company behind it in? | Company datasets (PeopleDataLabs, Crunchbase, Orbis), not web categorisers | Decent for large firms, noisy in the long tail, and the domain↔company join is the weak link | Partly |
 | Is it //abusive// — phishing, malware, scam? | Reputation feeds: VirusTotal, Google Safe Browsing, PhishTank | A different question with its own literature. Do not read a security verdict as a topic label | Rate-limited |
@@ Line 14: / Line 15: @@
 | What //software// runs it? | Wappalyzer, WhatWeb, response fingerprints | Unrelated to topic. A crawler-instrumentation question, see [[Programming:Crawler]] | Yes |
-These are different measurements with different failure modes, and they are easy to conflate — the structured extraction behind [[#Use in Publications]] files all five under one label, because papers describe all five as "classifying websites". The tracker row is where the conflation costs most, and unlike the others it has been measured: see [[#The trap that has caught the most papers]].
+These are different measurements with different failure modes, and they are easy to conflate — the structured extraction behind [[#Use in Publications]] files them all under one label, because papers describe them all as "classifying websites". The tracker row is where the conflation costs most, and unlike the others it has been measured: see [[#The trap that has caught the most papers]].
 <WRAP important>
 **A categorisation service is a measurement instrument, and you have to characterise it before you use it.** Not "we categorised domains using X" — that is not a method, it is a citation. What a reviewer needs is: which service and which version, how many of your domains it actually labelled, what its taxonomy is, and what precision it achieved //on your sample//, established by hand.
-Of the 247 papers in our corpus that assign topic categories to websites, **38.5% report any validation of the resulting labels at all.** That is the single largest methodological gap this page exists to close, and [[#Validating Your Labels]] gives you a script that closes it in an afternoon.
+Of the 330 papers in our corpus that assign topic categories to websites, **37.3% report any validation of the resulting labels at all.** That is the single largest methodological gap this page exists to close, and [[#Validating Your Labels]] gives you a script that closes it in an afternoon.
 </WRAP>
@@ Line 70: / Line 71: @@
 The one modern data point comes from a 2025/2026 arXiv preprint on LLM-assisted web measurement {[bozzolan2026_llmweb]}, whose Appendix B explicitly reuses Vallina et al.'s service list. Its findings:
-  * Of the services they surveyed, **three offer a free API at all** — Google Topics, Cloudflare Radar and VirusTotal — and Topics is deprecated and scheduled for removal, leaving two. The rate-limited ones are hopeless at scale: Cloudflare's Domain Threat Intelligence API allows 100 requests //per month//, so 100k websites would take "roughly 30,000 days"; VirusTotal's 500/day cap puts the same job at "around 200 days".
+  * Of the services they surveyed, **three offer a free API at all** — Google Topics, Cloudflare Radar and VirusTotal — and Topics is deprecated and scheduled for removal, leaving two. The rate-limited ones are hopeless at scale: Cloudflare's Domain Threat Intelligence API allows 100 requests //per month//, so 100k websites would take "roughly 30,000 days"; VirusTotal's 500/day cap puts the same job at "around 200 days". **For topic labels specifically that leaves one free API, VirusTotal, at 200 days per 100k domains** — Cloudflare's unmetered endpoint is the other one, and it answers a different question (next bullet).
-  * The Cloudflare **Radar** API has no explicit rate limit, so they could benchmark it. Against a 7,780-site ground truth for target-audience country, Radar **returned a label for 3,925 sites (50.4%)** and, of those, **agreed with the ground truth in 39.7% of cases**. They hand-checked 50 disagreements: their own label was right in 49 (98%), Cloudflare's in 1 (2%).
+  * The Cloudflare **Radar** API has no explicit rate limit, so they could benchmark it — **but on country, not on topic**, and the distinction matters because the number below gets quoted as if it were a categorisation result. Against a 7,780-site ground truth for target-audience country, Radar **returned a label for 3,925 sites (50.4%)** and, of those, **agreed with the ground truth in 39.7% of cases**. They hand-checked 50 disagreements: their own label was right in 49 (98%), Cloudflare's in 1 (2%). The reason is mechanical: Radar assigns a domain the country that issues the most DNS queries for it against Cloudflare's resolver, which is a measure of //where the traffic is//, not of who the site is for. **No comparable 2020s coverage-or-accuracy number exists for anyone's topic labels.**
   * Every open-weight LLM they tested reached 100% coverage and at least 76% accuracy on the same data, without visiting the sites.
@@ Line 194: / Line 195: @@
 === VirusTotal ===
-  * **API**: ''GET /api/v3/domains/{domain}'' returns a ''categories'' attribute, //"a dictionary mapping that relates categorisation services with the category it assigns the domain to"//. Documented at [[https://docs.virustotal.com/reference/domains-object|docs.virustotal.com]].
+  * **API**: ''GET /api/v3/domains/{domain}'' returns a ''categories'' attribute, //"a dictionary mapping that relates categorisation services with the category it assigns the domain to"//. Documented at [[https://docs.virustotal.com/reference/domains-object|docs.virustotal.com]]. **That is a topic label, not a maliciousness verdict.** The oracle use of the same API — engine disagreement, thresholds, URL versus file lookup, snapshot dating — is [[security:virustotal]].
   * **Which vendors**: Dr.Web, Forcepoint ThreatSeeker, BitDefender, Sophos, Trend Micro, Websense and (legacy) Alexa, among others — so several of the vendors above reach you through VirusTotal without you querying them.
-  * **Rate limit**: the free tier is **500 requests per day and 4 requests per minute**.((https://docs.virustotal.com/docs/public-vs-premium-api — fetched 2026-08-07; that page names the "Community" tier but does not mention the October 2025 retiering, so treat the tier names as unconfirmed and the numbers as current.)) The same page adds that the Public API //"must not be used in commercial products or services"//. At 500/day, a 100k-domain study takes 200 days. Vallina et al. had an academic key at 20k/day; we could find no self-serve academic application form in 2026, so budget for a direct conversation with VirusTotal rather than assuming access.
+  * **Rate limit**: the free tier is **500 requests per day and 4 requests per minute**.((https://docs.virustotal.com/docs/public-vs-premium-api — fetched 2026-08-07; that page names the "Community" tier but does not mention the October 2025 retiering, so treat the tier names as unconfirmed and the numbers as current.)) The same page adds that the Public API //"must not be used in commercial products or services"//. At 500/day, a 100k-domain study takes 200 days. Vallina et al. had an academic key at 20k/day. As of 2026-08-27 there is still no standalone quota-application form; the public contact form has a subject "I have an academic research request" ([[https://www.virustotal.com/gui/contact-us/legal|virustotal.com/gui/contact-us/legal]]). Budget for that conversation rather than assuming 20k/day. Maliciousness-oracle use of the same API is [[security:virustotal]].
   * **Advantages**: aggregates many providers in a single call; widely used and easy to cite.
   * **Disadvantages**: **the integration is lossy**. In the 2020 audit, several services returned labels when queried directly but not through VirusTotal, and Trend Micro's VirusTotal labels tracked its 2011 taxonomy rather than its 2019 one. You do not control which product version you are reading, and "we used VirusTotal categories" does not identify the underlying source.
@@ Line 208: / Line 209: @@
 Two different products, easily conflated:
-  * **[[https://developers.cloudflare.com/radar/|Radar API]]** — free, no explicit rate limit, data licensed **CC BY-NC 4.0** (non-commercial only, which matters for industry co-authors). Its measured quality is poor: 50.4% coverage and 39.7% agreement on the one 2026 benchmark that exists {[bozzolan2026_llmweb]}.
+  * **[[https://developers.cloudflare.com/radar/|Radar API]]** — free, no explicit rate limit, data licensed **CC BY-NC 4.0** (non-commercial only, which matters for industry co-authors). **Radar is a popularity-ranking product, not a categoriser**, and this is the single most common misreading of it on this page's topic. It does return a ''categories'' array, but only from ''GET /radar/ranking/domain/{domain}'', only for the ''POPULAR'' ranking, and only for the most recent ranking — so a domain gets categories only if Radar ranks it, and Radar ranks an ordered top 100 (global and per country) plus unordered buckets up to one million.((https://developers.cloudflare.com/api/resources/radar/subresources/ranking/subresources/domain/ and https://developers.cloudflare.com/radar/investigate/domain-ranking-datasets/ — both fetched 2026-08-07. The rate limit is unstated rather than absent; treat it as undocumented, not infinite.)) There is no Radar endpoint that categorises an arbitrary domain. The 50.4% / 39.7% figures quoted above are Radar's **country** output, not its categories {[bozzolan2026_llmweb]}; **nobody has published a coverage or accuracy figure for Radar's topic categories at all.**
   * **Domain Intelligence, now under Security Center Intel** — ''GET /accounts/{account_id}/intel/domain?domain=…'' returns ''content_categories''. This is the one the corpus papers used: {[ruth2022_toppling]} labelled Cloudflare-managed sites with it, and {[ruth2022_world]} built a whole global browsing study on it. It needs an API token with Intel permission scoped to an account; whether it is available on the free plan is not stated in Cloudflare's public docs.((Checked 2026-08-07 against https://developers.cloudflare.com/api/resources/intel/subresources/domains/methods/get/ — the docs describe the permission requirement but not the plan tier.))
@@ Line 225: / Line 226: @@
 === Alexa ===
-Amazon retired **Alexa.com on 1 May 2022**, per its own end-of-service notice ("we will be retiring Alexa.com on May 1, 2022").((Captured on the alexa.com login page, https://web.archive.org/web/20220315000000/https://www.alexa.com/ — retrieved 2026-08-07. [[Design:Website Selection]] currently gives 1 August 2023; we could not find a primary source for that date, and ''alexa.com'' now redirects to the unrelated Amazon Alexa voice assistant.)) Its ranking service and its category service died together.
+Amazon retired **Alexa.com on 1 May 2022**, per its own end-of-service notice ("we will be retiring Alexa.com on May 1, 2022").((Captured on the alexa.com login page, https://web.archive.org/web/20220315000000/https://www.alexa.com/ — retrieved 2026-08-07. An earlier revision of [[Design:Website Selection]] gave 1 August 2023; that date is when Tranco dropped Alexa from the default list, not when Amazon switched the service off. The selection page now carries both dates. ''alexa.com'' redirects to the unrelated Amazon Alexa voice assistant.)) Its ranking service and its category service died together.
-  * **Why it still matters**: it appears in 12 of the 247 corpus papers that categorise websites, under eight different spellings, and papers published as late as 2024 still use it because their data collection predates the shutdown. If you are reading such a paper, the labels are from a dead service with a documented 0.53% coverage rate.
+  * **Why it still matters**: it appears in 12 of the 330 corpus papers that categorise websites, under eight different spellings, and papers published as late as 2024 still use it because their data collection predates the shutdown. If you are reading such a paper, the labels are from a dead service with a documented 0.53% coverage rate.
   * **What it was**: highly granular (7,557 distinct labels observed through VirusTotal, against 279,716 categories in Alexa's own directory, which was derived from DMOZ), and capped at 500 sites per category — hence the tiny coverage. Leaf labels came back in the language of the branch, so the same concept appeared as //Arts//, //Artes// and //Kultur//.
@@ Line 244: / Line 245: @@
 | SimilarWeb | Alive | Paid API only | — | ~24 categories | you also need traffic and industry data |
 | VirusTotal | Alive | 500/day free | Lower than direct | Per underlying vendor | your sample is small, or you want vendor disagreement |
-| Cloudflare Radar | Alive | Free, CC BY-NC | 50.4% (2026, one task) | Yes | you can validate per-category and drop what fails |
+| Cloudflare Radar | Alive | Free, CC BY-NC | **Unmeasured for topics** — ranked domains only | Yes | your domains are inside Radar's top-1M ranking |
+| Cloudflare Domain Intelligence | Alive | Token, ~100/month free | Unmeasured | 26 super / 114 categories | you can validate per-category and drop what fails |
 | Curlie | Alive | **Free bulk dump** | Small but open | Yes, 15 top-level | you need redistributable ground truth |
 | OpenDNS community | **Dead** | — | 15% | — | never |
@@ Line 283: / Line 285: @@
 <WRAP important>
-**This is where our corpus runs out, and you should know exactly where.** Of the 4,322 papers in the 2010–2024 corpus, 29 use an LLM to classify anything at all — 27 of them in 2024 — and they point it at privacy policies, app reviews, vulnerabilities, jailbreak prompts and content moderation. **Zero of them use an LLM to assign a website a topic category.** Any ranking of methods built from this literature will therefore tell you that third-party services and hand-built keyword rules are the state of the art. As of 2026 that is no longer true, and the corpus cannot see why.
+**The corpus now reaches 2026, and the picture changed.** Of the 5,859 papers in the 2010–2026 corpus, **177 use an LLM to classify something** — 2 in 2023, 27 in 2024, 77 in 2025 and 71 in the partial 2026 — and **two of them use one to assign a website a topic category**. The nearer of the two, {[tsai2025_harmful]}, reports **92% accuracy for GPT-4o on shopping-vs-non-shopping over 500 manually annotated Tranco sites, and says explicitly that this is "comparable to commercial website classification services"**. So a ranking of methods built from this literature no longer tells you that third-party services and keyword rules are the only options — but it still gives you a two-paper base for the topic task specifically, in the corpus's two thinnest and provisional years. Treat the peer-reviewed evidence as thin rather than absent, and read the rest of this section for what is outside the corpus.
 </WRAP>
-The evidence outside the corpus is a 2025/2026 preprint, {[bozzolan2026_llmweb]}, which by its authors' account is the first systematic study of LLMs for web-measurement classification. It evaluates five **open-weight, self-hosted** models (gpt-oss:20b, llama3.3:70b, mistral-small3.2:24b, phi4-mini:3.8b, qwen3:32b, all via Ollama, temperature 0) on three tasks with hand-built benchmarks, in two modes: URL string only, and URL plus live browsing through LangChain and Playwright.
+The evidence outside the corpus is a 2025/2026 preprint, {[bozzolan2026_llmweb]}, which by its authors' account is the first systematic study of LLMs for web-measurement classification.
+<WRAP important>
+**It is one unrefereed preprint, and it has moved under its own feet.** Everything in this section rests on a single arXiv posting that has not been through peer review, and whose results are not stable across versions. **v1** (9 October 2025) benchmarked a cloud model — Gemini 2.5 Flash — against Llama 4:109B and Gemma 3:27B, on a **14**-category Curlie task. **v3** (30 April 2026) drops the cloud model entirely, evaluates five open-weight models instead, reduces the topic task to **13** categories, and adds two things that appear nowhere in v1: a Homepage2Vec baseline, and {[vallina2020_misshapes]} — the audit this whole page is built on, which v1 does not cite once.((Version dates from https://arxiv.org/abs/2510.08101 (submission history); v1 text read from https://arxiv.org/html/2510.08101v1, which contains zero occurrences of Vallina et al. 2020 and one passing mention of Homepage2Vec with no comparison. Both fetched 2026-08-07.)) **This page reads v3.** Cite the version you read, and re-derive any number you lean on — this one changed once already, and a preprint that acquires its baseline and its closest related work between postings is a preprint whose next revision may move again.
+</WRAP>
+It evaluates five **open-weight, self-hosted** models (gpt-oss:20b, llama3.3:70b, mistral-small3.2:24b, phi4-mini:3.8b, qwen3:32b, all via Ollama, temperature 0) on three tasks with hand-built benchmarks, in two modes: URL string only, and URL plus live browsing through LangChain and Playwright.
 ^ Task ^ N ^ Classes ^ Best URL-only ^ Best URL + live visit ^
@@ Line 294: / Line 302: @@
 The number that matters for this page: on the same 13-category benchmark, **Homepage2Vec scores 0.60 Jaccard and 0.55 macro F1** — the paper's words are that "a general-purpose LLM like gpt-oss can label websites with higher accuracy than a classic machine learning model specifically trained for website categorization." A stock open-weight instruct model, with no fine-tuning, beat the field's reference specialist classifier.
+Two cautions before you repeat that sentence.
+  - **0.55 is not a refutation of the 0.90 in Homepage2Vec's abstract**, and anyone who puts the two side by side has compared different evaluation regimes. 0.90 is the balanced per-class setup; Homepage2Vec's own authors report **0.634** macro F1 on an unbalanced test set, which is the regime a benchmark like this one is in. See [[#Homepage2Vec]].
+  - **0.55 is nonetheless below that 0.634**, on a task that drops one of the 14 classes, and the preprint never states how it ran the baseline. Homepage2Vec emits independent //calibrated probabilities// per class, so turning it into a multi-label prediction requires a decision threshold — its own paper uses 0.5 on calibrated output — and the word "threshold" does not occur anywhere in the preprint.((Verified by full-text search of https://arxiv.org/html/2510.08101v3 on 2026-08-07: zero occurrences of "threshold".)) A head-to-head in which the winner ran the loser's model, without reporting the one hyperparameter that decides the loser's score, is a result to re-run before you build on it. Both models are free and the benchmark is small.
 Three practical consequences:
@@ Line 307: / Line 320: @@
   * **Label instability across seeds and temperature** for website classification specifically. The one systematic paper set temperature to 0 to sidestep the question rather than quantify it.
   * **Behaviour on parked, dead and holding-page domains** — a large fraction of any long-tail sample, and a case where "visit the site" gives the model a page that is not about anything.
-  * **A formal agreement statistic** (Cohen's or Fleiss' kappa) between LLM labels and human annotators for this task. Nobody has published one. What does exist, on those same 150 hand-relabelled sites, is a Jaccard similarity of **0.73 between the human labels and gpt-oss's predictions**, against **0.75 between the human labels and the benchmark's own ground truth** — the model is as close to careful humans as the benchmark is. Which is also the reason to read the 0.65 Jaccard above as a floor rather than a ceiling: a multi-label benchmark that under-labels punishes a model for being right. (For scale on how much under-labelling there is: the benchmark's own heuristic label-construction matched the human label set exactly on 86 of 150 sites (57%), where Lugeon et al.'s Curlie snapshot matched on 67 (45%). Both figures are about the ground truth, not about any model.)
+  * **A formal agreement statistic** (Cohen's or Fleiss' kappa) between LLM labels and human annotators for this task. Nobody has published one. What does exist, on those same 150 hand-relabelled sites, is a Jaccard similarity of **0.73 between the human labels and gpt-oss's predictions**, against **0.75 between the human labels and the benchmark's own ground truth** — the model is as close to careful humans as the benchmark is. Which is also the reason to read the 0.65 Jaccard above as a floor rather than a ceiling: a multi-label benchmark that under-labels punishes a model for being right. (For scale on how much under-labelling there is: the benchmark's own heuristic label-construction matched the human label set exactly on 86 of 150 sites (57%), where Lugeon et al.'s Curlie snapshot matched on 67 (45%). Both figures are about the ground truth, not about any model.) **And note what that pair cannot tell you:** those 150 sites were relabelled by the preprint's own authors, with no second annotator and no agreement statistic of any kind — no kappa, no Krippendorff's alpha, nothing.((Verified by full-text search of https://arxiv.org/html/2510.08101v3 on 2026-08-07: zero occurrences of "kappa", "inter-annotator", "Fleiss" or "Krippendorff".)) So 0.73 says the model tracks //that annotator's// labels. It does not say how fuzzy the task is, and a single-annotator relabelling cannot be used to argue that the remaining label noise is small — the disagreement you have not measured is exactly the quantity in question. If you relabel a sample, use two annotators and report the agreement; [[#Validating Your Labels]] is the whole procedure.
+===== Target country is not topic =====
+Studies that check compliance against a national law need to know which country a site is //for//, and this gets treated as a minor preprocessing step. It is not. There is no service that answers it, the available proxies measure different things, and none of them is a target-audience label:
+  * **The TLD.** Precise where it exists and absent where it matters — a ccTLD is strong evidence, but generic TLDs carry no country at all, and that is most of the head of any list.
+  * **CrUX country lists, via Tranco.** Tranco's list-generation API takes ''filterCRUX'', ''filterCRUXType'' (''global'' / ''country'' / ''region'' / ''subregion''), ''filterCRUXValue'' (e.g. a list of country codes) and ''filterCRUXMonth'', so you can generate a country-restricted list reproducibly, with a permalink, from a free account.((https://tranco-list.eu/api_documentation — fetched 2026-08-07. Basic Auth with your email as username and API token as password; the ''/configure'' page returns 401 without a login. CrUX datasets are CC BY 4.0 per Google's methodology, fetched 2026-08-27; Tranco's homepage still labels CrUX CC BY-SA 4.0, while the methodology page states no CrUX licence.)) This is the cleanest free source and it is under-used. **But it ranks by page loads from Chrome users in a country, which is popularity, not audience** — and the overlap is severe: the union of five country top-10k lists (China, Germany, Italy, Korea, Turkey) is only 18,718 domains rather than 50,000, with 4,017 domains common to all five, and a preliminary labelling built from these lists is incompatible with the site's own TLD in at least 25% of cases for every label {[bozzolan2026_llmweb]}.
+  * **Site language.** A good proxy, and cheap, but it splits badly on English, Spanish, Arabic and Portuguese, which is a large share of the web.
+  * **Host IP or CDN location.** Measures where bytes are served from. Behind a CDN — most of the head of any list — it tells you about the CDN. See [[Design:IP Classification]] and [[Design:Crawling Location]].
+  * **Cloudflare Radar's country field.** DNS-query volume by country against Cloudflare's resolver: 50.4% coverage and 39.7% agreement on the one benchmark that exists (see [[#Has Anything Changed Since 2020?]]).
+**So combine them and report the combination.** The one systematic evaluation on this task got its ground truth by starting from CrUX and then correcting with site language and host IP, and still hand-inspected the result — and an LLM given the URL alone beat the best single proxy on that benchmark. Whatever you do, say which proxies you combined and how you broke ties: "we classified sites by country" is not a method, and a compliance result is only as good as the jurisdiction assignment underneath it.
 ===== Taxonomies =====
-Pick the taxonomy before the service, because it is much harder to change later, and because 72% of the corpus papers that name a taxonomy at all name a bespoke one and do not even state how many classes it has (see [[#Use in Publications]]) — which makes their results incomparable with everyone else's.
+Pick the taxonomy before the service, because it is much harder to change later, and because 75% of the corpus papers that name a taxonomy at all name a bespoke one and do not even state how many classes it has (see [[#Use in Publications]]) — which makes their results incomparable with everyone else's.
   * **[[https://github.com/InteractiveAdvertisingBureau/Taxonomies|IAB Tech Lab Content Taxonomy]]** — the advertising industry's reference schema, and the right choice if your question is about ad targeting or if you want to be comparable to Webshrinker's output. Machine-readable TSV in the repo under ''Content Taxonomies/'', CC BY 3.0, up to four tiers with a relational ID system. Versions 1.0 through **3.1** sit side by side; 2.x → 3.0 is a **breaking change** and IAB says the two must not be used together. Category rows counted from the TSVs on 2026-08-07, excluding the two header lines: **2.2 has 1,196 rows, 3.0 has 703, 3.1 has 704** — so 3.1 is marginally //larger// than 3.0. Tier-1 rows: 37 in 3.1, but only 31 of those carry a numeric Unique ID; the other six (Communication, Entertainment, Events, Genres, Holidays, Sensitive Topics) are alphanumeric extension rows. **State which convention you counted with** — three independent counts of this one file in preparing this page produced three different numbers. Vendor marketing claiming that 3.x "expands to 1,500+ categories" is wrong in both directions — 3.x is roughly //40% smaller// than 2.2, because it consolidated and restructured rather than adding. Count the rows in the file yourself; it takes ten seconds and the summaries are not reliable.
-  * **Curlie's 15 top-level categories** — the right choice if you want free ground truth, an open model that already predicts into it, and a hierarchy you can prune to whatever depth your sample supports.
+  * **Curlie's 15 top-level categories** — the right choice if you want free ground truth, an open model that already predicts into it, and a hierarchy you can prune to whatever depth your sample supports. **Check that your classes are actually in it before you commit.** The 15 are Arts, Business, Computers, Games, Health, Home, News, Recreation, Reference, Regional, Science, Shopping, Society, Sports, and Kids and Teens — a directory taxonomy from the early 2000s. There is no top-level class for social media, finance and banking, education, government, or online services, and none at all for adult content; those questions land inside Business, Society, Reference or Computers, or nowhere. If your paper is about the privacy of banking sites, or trackers on education sites, or anything on social platforms, a Curlie-trained classifier will not give you the class you are measuring, and pruning to top level will hide that rather than fix it.
   * **A vendor's own list** — fine, but pin the version, and expect to discover categories that are not in the documentation. Vallina et al. found at least one undocumented category in eight of the nine services whose documentation they could compare against its output.
   * **A standard industry code** (NAICS, SIC, NACE) — the right choice for a question about firms rather than content, and the natural join key to the company datasets below. Exactly one paper in our corpus did this.
@@ Line 338: / Line 363: @@
     - Based on LinkedIn profiles that are self-reported — prone to adversarial data.
     - Only a subset of PeopleDataLabs' full dataset. The "22M of 70M rows" framing is longstanding on this page; the 22M is confirmed, the 70M total could not be re-confirmed in 2026 and PDL's marketing page now cites 23.8M+ without saying which corpus that is.
-  <wrap todo>TODO: cite ''Machine Learning Compliance Analysis for Email Regulation'' when it is public.</wrap>
+  <WRAP todo>TODO: cite ''Machine Learning Compliance Analysis for Email Regulation'' when it is public.</WRAP>
 ==== Crunchbase ====
@@ Line 353: / Line 378: @@
     - Focuses mostly on variables useful for investments and market competitiveness.
     - Academic access is no longer publicly documented; budget for a sales conversation.
-  <wrap todo>TODO: cite ''Machine Learning Compliance Analysis for Email Regulation'' when it is public.</wrap>
+  <WRAP todo>TODO: cite ''Machine Learning Compliance Analysis for Email Regulation'' when it is public.</WRAP>
 ==== Orbis ====
@@ Line 713: / Line 738: @@
 total                                1100      100
+wrote sheet_a.csv  (100 rows, category hidden, order shuffled)
+wrote sheet_b.csv  (100 rows, category hidden, order shuffled)
+wrote sample_key.csv     (the labels under test -- do not show this to annotators)
 note: +/-20% half-width at 90% precision needs 9 domains per category (normal approximation; the Wilson interval reported by `score` is wider near 100%)
 note: +/-10% half-width at 90% precision needs 35 domains per category (normal approximation; the Wilson interval reported by `score` is wider near 100%)
 note: +/-5% half-width at 90% precision needs 139 domains per category (normal approximation; the Wilson interval reported by `score` is wider near 100%)
 ==================================================================
 INTER-ANNOTATOR AGREEMENT
@@ Line 723: / Line 750: @@
 raw agreement            : 67/100 = 67.0%
 Cohen's kappa            : 0.564
+Report both. Kappa alone hides how many categories there were, and raw
+agreement alone is inflated when one category dominates.
 ==================================================================
@@ Line 739: / Line 768: @@
 ALL (disputed counted wrong)       63  100  63.0%  [53.2%, 71.8%]
+The true precision is between these two rows. If they are far apart your
+annotators disagreed too much for the number to mean anything -- fix the
+codebook and re-annotate rather than picking the flattering row.
 ==================================================================
@@ Line 744: / Line 776: @@
 ==================================================================
 /1100 domains got a label = 80.0% [77.5%, 82.3%]
+==================================================================
+PASTE THIS INTO YOUR METHODOLOGY SECTION
+==================================================================
+We categorised 1100 domains, of which 880 (80.0%) received a label. To validate the labels we drew a stratified random sample of 100 domains and had two authors independently assign a category while blind to the service's answer (Cohen's kappa = 0.56, raw agreement 67%). On the 67 domains where the annotators agreed, the service's micro-averaged precision was 94.0% (95% Wilson CI [85.6%, 97.7%]); counting every disputed domain as an error gives a lower bound of 63.0%. We discarded the 1 category that fell below 80% precision (Search Engines) and report results over the remaining 3.
 </code>
-Three things to read out of that:
+Four things to read out of that:
   - **It found the bad category.** Search Engines came back at 55.6% and got dropped, exactly as intended — and it did so from 9 usable annotations, which is why per-category stratification matters more than total sample size.
@@ Line 840: / Line 877: @@
 ===== What to Report =====
-A checklist, derived from what the 247 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply.
+A checklist, derived from what the 330 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply.
   - **Which service or model, and which version.** "McAfee" identified a different company before and after February 2024. "Homepage2Vec" without a commit hash is not reproducible; "GPT-4" without a dated snapshot is not either.
@@ Line 849: / Line 886: @@
   - **Validation**: sample size, how it was stratified, how many annotators, whether they were blind to the service's label, the agreement statistic, and per-category precision — not one global accuracy number. Publish the annotated sample as an [[:Artifacts|artefact]] //(page not yet written)//; it is small, and it is the only part of this a reader can check.
   - **The categories you dropped**, and the rule you dropped them by.
-  - **If you used more than one service, the disagreement between them.** Only 9 of 122 corpus papers that name a service name two, so this is nearly free novelty.
+  - **If you used more than one service, the disagreement between them.** Only 9 of 155 corpus papers that name a service name two, so this is nearly free novelty.
 <WRAP important>
@@ Line 857: / Line 894: @@
 ===== Use in Publications =====
-Two sources here. The historical survey is Vallina et al. {[vallina2020_misshapes]}, whose figures on service popularity are reproduced below. The rest is a fresh query over a structured extraction of **4,322 papers from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2024** — so every figure is a claim about those seven venues over that window, not about the field.
+Two sources here. The historical survey is Vallina et al. {[vallina2020_misshapes]}, whose figures on service popularity are reproduced below. The rest is a fresh query over a structured extraction of **5,859 papers from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026** — so every figure is a claim about those seven venues over that window, not about the field. The 2025 and 2026 venue-years are provisional: CCS and IMC 2026 have not been held, and IEEE S&P and WWW 2026 abstracts are not yet in the selection source.
 <WRAP right box>
@@ Line 866: / Line 903: @@
 ==== The population, and what "website category" hides ====
-The extraction marks 320 papers as classifying something with the target ''website-category''. **That number is not the population this page is about**, and the difference is instructive: the same label covers several unrelated tasks.
+The extraction marks 424 papers as classifying something with the target ''website-category''. **That number is not the population this page is about**, and the difference is instructive: the same label covers several unrelated tasks.
-^ What is actually being classified ^ Papers ^ Share of the 320 ^
+^ What is actually being classified ^ Papers ^ Share of the 424 ^
-| A topic, industry or sector label on a site or domain | 247 | 77.2% |
+| A topic, industry or sector label on a site or domain | 330 | 77.8% |
-| An abuse verdict — phishing, scam, malicious | 47 | 14.7% |
+| An abuse verdict — phishing, scam, malicious | 63 | 14.9% |
-| Website fingerprinting — which of N monitored sites produced this traffic trace | 22 | 6.9% |
+| Website fingerprinting — which of N monitored sites produced this traffic trace | 36 | 8.5% |
-| Hosting or CDN infrastructure | 7 | 2.2% |
+| Hosting or CDN infrastructure | 8 | 1.9% |
-| Page type — homepage, login page, sign-up | 6 | 1.9% |
+| Page type — homepage, login page, sign-up | 7 | 1.7% |
-| Which software runs the site | 6 | 1.9% |
+| Which software runs the site | 6 | 1.4% |
-| Third-party role — tracker, advertiser, analytics | 5 | 1.6% |
+| Third-party role — tracker, advertiser, analytics | 5 | 1.2% |
-Website fingerprinting is a traffic-analysis attack that shares no method, taxonomy or ground truth with topic categorisation; it is here because "classify a website" is the same English phrase. **Everything below uses the 247, which is 5.7% of the corpus.** The other 73 papers are counted, named, and excluded — see [[#Methodology and limitations of these figures]].
+Website fingerprinting is a traffic-analysis attack that shares no method, taxonomy or ground truth with topic categorisation; it is here because "classify a website" is the same English phrase. **Everything below uses the 330, which is 5.6% of the corpus.** The other 94 papers are counted, named, and excluded — see [[#Methodology and limitations of these figures]].
 ==== This is a Web Conference and IMC topic ====
-^ Venue ^ Papers ^ Share of the 247 ^ Share of that venue's papers ^
+^ Venue ^ Papers ^ Share of the 330 ^ Share of that venue's papers ^
-| TheWebConf | 59 | 23.9% | 8.3% |
+| TheWebConf | 71 | 21.5% | 8.4% |
-| IMC | 45 | 18.2% | 8.1% |
+| USENIX Security | 58 | 17.6% | 4.1% |
-| USENIX Security | 45 | 18.2% | 4.0% |
+| IMC | 50 | 15.2% | 7.8% |
-| CCS | 44 | 17.8% | 4.9% |
+| CCS | 48 | 14.5% | 4.8% |
-| PETS | 29 | 11.7% | 8.2% |
+| PETS | 43 | 13.0% | 8.4% |
-| NDSS | 22 | 8.9% | 5.3% |
+| NDSS | 35 | 10.6% | 5.0% |
-| IEEE S&P | 3 | 1.2% | 1.1% |
+| IEEE S&P | 25 | 7.6% | 3.3% |
-TheWebConf, IMC and PETS each devote about 8% of their programme to it; IEEE S&P essentially never does. The overall rate has been flat-to-falling: 5.1% of papers in 2010–2013, peaking at 7.5% in 2014–2017, back to 4.6% in 2022–2024.
+TheWebConf and PETS each devote about 8% of their programme to it, IMC nearly as much. The overall rate has been flat-to-falling: 5.5% of papers in 2010–2013, peaking at 7.3% in 2014–2017, 4.6% in 2022–2024 and 5.5% in the provisional 2025–2026 window.
+<WRAP info>
+**An earlier version of this table said IEEE S&P "essentially never" does this, at 3 papers and 1.2%.** That was an artefact of the corpus: IEEE S&P was then only 43% retrieved. It is now complete (780 of 780 selected papers), and IEEE S&P has 25 such papers, 3.3% of its programme. The old caveat named exactly the right suspect, and the fix confirmed it — which is the useful lesson: a venue that looks absent from a corpus should be checked for a retrieval gap before it is described as uninterested.
+</WRAP>
 ==== How they classify ====
-^ Method ^ Papers ^ Share of 247 ^
+^ Method ^ Papers ^ Share of 330 ^
-| Third-party service | 105 | 42.5% |
+| Third-party service | 131 | 39.7% |
-| Manual labelling | 53 | 21.5% |
+| Manual labelling | 68 | 20.6% |
-| Heuristic rules | 39 | 15.8% |
+| Heuristic rules | 59 | 17.9% |
-| Supervised ML | 35 | 14.2% |
+| Supervised ML | 46 | 13.9% |
-| Curated database | 31 | 12.6% |
+| Curated database | 46 | 13.9% |
-| Unsupervised ML | 11 | 4.5% |
+| Unsupervised ML | 14 | 4.2% |
-| Regex or signature | 9 | 3.6% |
+| Regex or signature | 9 | 2.7% |
+| **LLM** | **2** | **0.6%** |
+Multi-valued, so shares exceed 100%. Reliance on third-party services grew steadily for four windows — 25.0% of categorising papers in 2010–2013, 33.9%, 43.5%, **49.4% in 2022–2024** — while use of curated databases (DMOZ and friends) fell from 21.4% to 9.0% across the middle of that run. The field moved from open, inspectable label sources toward closed, unauditable ones, over exactly the period in which the only audit of those sources found them wanting.
-Multi-valued, so shares exceed 100%. Reliance on third-party services has grown steadily — 29.2% of categorising papers in 2010–2013, 35.8%, 44.4%, and **48.8% in 2022–2024** — while use of curated databases (DMOZ and friends) halved over the same period, from 16.7% to 8.8%. The field has moved from open, inspectable label sources toward closed, unauditable ones, over exactly the period in which the only audit of those sources found them wanting.
+**In the provisional 2025–2026 window both of those trends reverse**: third-party services drop to 32.3% and curated databases return to 18.5%, with hand-built heuristics up to 23.1% and the first two LLM papers appearing at 3.1%. Two incomplete venue-years cannot establish a turn, and this is exactly the kind of row not to quote as a trend — but it is the first movement against the closed-source drift in fifteen years, and it is worth re-checking when 2026 is complete.
 ==== Which services, folded ====
@@ Line 910: / Line 954: @@
 ^ Service ^ Papers ^ Distinct spellings folded ^
-| McAfee (incl. SmartFilter, TrustedSource, SiteAdvisor, URL Ticketing System) | 19 | **15** |
+| McAfee (incl. SmartFilter, TrustedSource, SiteAdvisor, URL Ticketing System) | 23 | **18** |
+| Symantec / Blue Coat (incl. WebPulse, RuleSpace, SiteReview) | 14 | **10** |
 | Alexa | 12 | 8 |
-| Symantec / Blue Coat (incl. WebPulse, RuleSpace, SiteReview) | 12 | **10** |
 | FortiGuard | 12 | 7 |
+| Media bias raters (AllSides, Media Bias/Fact Check, Science Feedback) | 10 | 9 |
+| Cloudflare domain categories | 10 | 6 |
+| Webshrinker | 9 | 2 |
 | DMOZ / Open Directory Project | 8 | 6 |
-| Webshrinker | 8 | 2 |
+| Google Cloud Natural Language | 8 | 6 |
-| Google Cloud Natural Language | 7 | 6 |
+| SimilarWeb | 7 | 2 |
+| VirusTotal | 7 | 3 |
 | Google Ads keyword and topic tools | 6 | 5 |
-| Cloudflare domain categories | 6 | 3 |
-| Media bias raters (AllSides, Media Bias/Fact Check) | 5 | 5 |
 | Trend Micro | 4 | 3 |
-| SimilarWeb | 4 | 2 |
 | Cisco (OpenDNS, IronPort) | 3 | 3 |
 | Cyren | 3 | 3 |
-| VirusTotal | 3 | 3 |
+| Citizen Lab test list | 3 | 3 |
+| Forcepoint / Websense | 3 | 2 |
+| Google Topics API | 3 | 3 |
-**122 of the 247 papers (49.4%) name at least one third-party service or dataset.** McAfee alone appears under fifteen different names — from bare "McAfee" to "McAfee's URL Ticketing System" to "McAfee SmartFilter Internet Database" — and Symantec under ten. Any count of these services built on exact string matching will be wrong by a factor of several, which is the same failure that has bitten geolocation and crawler-tool tallies on other pages of this site.
+**155 of the 330 papers (47.0%) name at least one third-party service or dataset.** McAfee alone appears under eighteen different names — from bare "McAfee" to "McAfee's URL Ticketing System" to "McAfee SmartFilter Internet Database" — and Symantec under ten. Any count of these services built on exact string matching will be wrong by a factor of several, which is the same failure that has bitten geolocation and crawler-tool tallies on other pages of this site.
-Everything that is not a named service, ranked: unnamed custom methods (38 papers), manual labelling by the authors or hired coders (32), custom keyword and regex rules (25), random forests (10), SVMs (9), clustering (8), then a long tail. Those first and third rows are one family: **61 papers (24.7%) classify websites with rules they wrote themselves, and 38 of them do not name the rules at all.**
+Everything that is not a named service, ranked: unnamed custom methods (58 papers), manual labelling by the authors or hired coders (41), custom keyword and regex rules (32), random forests (13), clustering (10), SVMs (9), then a long tail. Those first and third rows are one family: **87 papers (26.4%) classify websites with rules they wrote themselves, and 58 of them do not name the rules at all.**
 ==== Almost nobody cross-checks ====
 <WRAP important>
-Vallina et al.'s central recommendation is not to rely on a single service. **Nine of the 122 papers that name a service name two** — 7.4%. Five of those nine are from 2013–2015. The recommendation has not been taken up, and it remains one of the cheapest ways to strengthen a categorisation-based result.
+Vallina et al.'s central recommendation is not to rely on a single service. **Nine of the 155 papers that name a service name two** — 5.8%, down from 7.4% on the smaller corpus, because the count of papers naming two has not moved at all while the field has grown. Five of those nine are from 2013–2015. The recommendation has not been taken up, and it remains one of the cheapest ways to strengthen a categorisation-based result.
 </WRAP>
 ==== Most do not validate, and half do not say where their labels came from ====
-^ Question ^ Papers ^ Share of 247 ^
+^ Question ^ Papers ^ Share of 330 ^
-| Names the taxonomy it classified into | 184 | 74.5% |
+| Names the taxonomy it classified into | 248 | 75.2% |
-| Names a ground-truth source | 124 | 50.2% |
+| Names a ground-truth source | 169 | 51.2% |
-| Reports **any** validation of the labels | 95 | **38.5%** |
+| Reports **any** validation of the labels | 123 | **37.3%** |
-| Explicitly reports no validation | 69 | 27.9% |
+| Explicitly reports no validation | 98 | 29.7% |
-| Sentinel: the extraction marked validation ''not-applicable'' | 83 | 33.6% |
+| Sentinel: the extraction marked validation ''not-applicable'' | 109 | 33.0% |
 The last row is a sentinel and is shown because the page's own rule says sentinels are never folded into an answer. It means the extraction could not find a validation step and did not think one was called for — typically because the paper is reporting a distribution of vendor labels with no claim that depends on them being right. Read it as "no validation reported", not as "validation was unnecessary", unless you check the paper.
-Of those that do validate: 70 papers by manual validation, 18 by cross-validation, 9 by a held-out test set, 9 by comparison to another method. Note the shape — manual spot-checking is the dominant and correct instrument here, because a held-out test set only tells you about the labels you already had.
+Of those that do validate: 92 papers by manual validation, 21 by cross-validation, 13 by a held-out test set, 10 by comparison to another method. Note the shape — manual spot-checking is the dominant and correct instrument here, because a held-out test set only tells you about the labels you already had.
-The 50.2% figure is worth dwelling on: for **website** classification, the share naming a ground-truth source is well below the 71.7% the same extraction measures across all 3,241 papers that classify anything. Website categorisation is reported worse than classification in general, not better.
+The 51.2% figure is worth dwelling on: for **website** classification, the share naming a ground-truth source is well below the 72.9% the same extraction measures across all 4,439 papers that classify anything. Website categorisation is reported worse than classification in general, not better.
 ==== Taxonomies are bespoke, and usually not even sized ====
-Of the 184 papers that name a taxonomy at all:
+Of the 248 papers that name a taxonomy at all:
-^ Taxonomy ^ Papers ^ Share of 184 ^
+^ Taxonomy ^ Papers ^ Share of 248 ^
-| A bespoke scheme, class count not given | 133 | 72.3% |
+| A bespoke scheme, class count not given | 187 | 75.4% |
-| A vendor's own category list | 37 | 20.1% |
+| A vendor's own category list | 43 | 17.3% |
-| DMOZ / ODP / Curlie | 6 | 3.3% |
+| IAB | 9 | 3.6% |
-| IAB | 6 | 3.3% |
+| DMOZ / ODP / Curlie | 7 | 2.8% |
-| A bespoke scheme with a stated class count | 6 | 3.3% |
+| A bespoke scheme with a stated class count | 7 | 2.8% |
-| Google AdWords / Topics | 3 | 1.6% |
+| Google AdWords / Topics | 4 | 1.6% |
-| A standard industry code (NAICS, SIC, NACE) | 1 | 0.5% |
+| A standard industry code (NAICS, SIC, NACE) | 1 | 0.4% |
-Nearly three quarters invent a scheme and do not say how many classes it has. That single omission makes a result impossible to compare against any other paper, and it is free to fix.
+Three quarters invent a scheme and do not say how many classes it has. That single omission makes a result impossible to compare against any other paper, and it is free to fix.
 ==== Where LLMs actually appear ====
 ^ Year ^ Papers using an LLM to classify anything ^ Corpus that year ^ Share ^
-| 2023 | 2 | 630 | 0.3% |
+| 2023 | 2 | 719 | 0.3% |
-| 2024 | 27 | 628 | 4.3% |
+| 2024 | 27 | 690 | 3.9% |
+| 2025 //(provisional)// | 77 | 770 | 10.0% |
+| 2026 //(provisional)// | 71 | 415 | 17.1% |
+**177 papers, and the curve is no longer starting — it is the fastest-moving thing in this corpus.** From 2 papers in 2023 to one in six in 2026, and both of the last two venue-years are incomplete, so those shares are floors. They point the model at "other" targets (116 papers), user-generated text (32), vulnerabilities (14), privacy policies (12), mobile apps (6), website categories (5), and single-digit counts of domains, network traffic, consent notices, requests, IP addresses, cookies, email and dark patterns.
+They establish a norm worth copying: **148 of 177 (83.6%) validate the model's labels against something**, usually manual annotation of a sample — the same share as when this was 29 papers, which is the encouraging part.
+**Five papers now point an LLM at a website category, and two of those are genuine topic classification** rather than an abuse verdict:
-Twenty-nine papers across the corpus's fifteen years, twenty-seven of them in the final one — the curve is visibly starting, and the corpus ends before it goes anywhere. They point the model at "other" targets (19 papers), user-generated text (6), vulnerabilities (4), privacy policies (1) and domains (1). **Not one of the 29 uses an LLM to assign a website a topic category.**
+  * {[tsai2025_harmful]} (TheWebConf 2025) classifies Tranco sites as shopping or non-shopping with **GPT-4o prompted with the URL and a screenshot**, validates against 500 manually annotated sites, and reports **92% accuracy, "comparable to commercial website classification services"**. That sentence is the first thing in this corpus that makes an LLM a real alternative to the vendors catalogued above, and it is worth reading before you buy an API key.
+  * DarkGram (USENIX Security 2025) has GPT-4 label cybercriminal Telegram channels into 15 predefined categories, which the model itself extended to 19, validated by coders on 1,000 sampled posts. Categories of a channel, not of a website — count it as adjacent.
-They do, however, establish a norm worth copying: **24 of 29 (82.8%) validate the model's labels against something**, usually manual annotation of a sample. The single ''website-category'' LLM tuple in the whole corpus is a USENIX Security 2024 phishing detector using GPT-3.5-turbo-16k with a chain-of-thought prompt for a binary credential-page question, validated against three cybersecurity experts' annotations — a security verdict, not a topic.
+The other three are abuse verdicts: the USENIX Security 2024 phishing detector using GPT-3.5-turbo-16k for a binary credential-page question, a TheWebConf 2025 crypto-scam study using a Llama3:70b + GPT-4 hybrid, and an NDSS 2026 credential-theft phishing detector using GPT-4o with retrieval-augmented prompts. All three validate against manual annotation.
-See [[#LLM Classification]] for what has happened since the corpus ends.
+See [[#LLM Classification]] for what has happened outside the corpus.
 ==== Two examples worth reading first ====
@@ Line 985: / Line 1041: @@
 ==== Methodology and limitations of these figures ====
-  * **Denominators.** Every figure names its own. The corpus is 4,322 papers; 3,241 classify something; 320 carry a ''website-category'' target; **247 assign a topic, industry or sector label** and are this section's population. Never "of 4,322 papers".
+  * **Denominators.** Every figure names its own. The corpus is 5,859 papers; 4,439 classify something; 424 carry a ''website-category'' target; **330 assign a topic, industry or sector label** and are this section's population. Never "of 5,859 papers".
   * **Papers, not mentions.** A paper naming McAfee three times counts once.
   * **Sentinels are not answers.** ''not-stated'' and ''none-mentioned'' are excluded from every numerator and reported as their own row where they matter — which they do, since "does not validate" is the headline finding.
-  * **Free-text names are folded** into canonical families by an ordered regex list before counting, because exact strings agree run-to-run only about 20% of the time. The fold left **20 distinct strings unclassified**, each appearing once: among them ''Taobao main-business categories'', ''SNET pillar portals'', ''Rakuten Popshops'', ''multiple domain classification services'' and ''public classification service from the security vendor''. The last two are papers that used a service and declined to say which — a small, exact instance of the reporting gap this section measures.
+  * **Free-text names are folded** into canonical families by an ordered regex list before counting, because exact strings agree run-to-run only about 20% of the time. The fold left **33 distinct strings unclassified**, each appearing once: among them ''Taobao main-business categories'', ''SNET pillar portals'', ''Rakuten Popshops'', ''NAICSlite'', ''ORES topic routing'', ''multiple domain classification services'' and ''public classification service from the security vendor''. The last two are papers that used a service and declined to say which — a small, exact instance of the reporting gap this section measures. The residue was 40 strings before the fold was extended for this corpus; [[provenance:design:website_classification]] lists what was added and what was deliberately left out.
-  * **The task fold is imperfect and its exclusions are visible.** Eight side-channel and website-fingerprinting papers named their classifier only as "a 1D CNN" or "k-NN", leaving no signal in any extracted field; they were caught by matching the paper's own slug and are excluded by an explicit named list rather than by a general rule. The 73 excluded papers are counted in the table above rather than dropped.
+  * **The task fold is imperfect and its exclusions are visible.** Eight side-channel and website-fingerprinting papers named their classifier only as "a 1D CNN" or "k-NN", leaving no signal in any extracted field; they were caught by matching the paper's own slug and are excluded by an explicit named list rather than by a general rule. The 94 excluded papers are counted in the table above rather than dropped.
   * **Enum fields versus free text.** Method and validation are enums, stable enough to publish as rough shares (''classification.method'' agrees 58% run-to-run, so read those as a ranking). Service names and taxonomies are free text and are reported as rankings and folded families only.
-  * **Quotes were checked.** Every figure above traces to tuples carrying a verbatim evidence quote; a sample of these was re-located in the source PDFs. Of six quotes checked by hand, two initially "failed" a literal grep and turned out to be intact but split across a two-column break — normalise whitespace before concluding that a quote is not in the paper.
+  * **Quotes were checked.** Every figure above traces to tuples carrying a verbatim evidence quote; a sample of these was re-located in the source PDFs. Of six quotes checked by hand, two initially "failed" a literal grep and turned out to be intact but split across a two-column break — normalise whitespace before concluding that a quote is not in the paper. The five new LLM website-category tuples were re-checked individually on 2026-08-12; see [[provenance:design:website_classification]].
-  * **Coverage.** IEEE S&P is only 43% retrieved, which is the most likely explanation for its 1.2% share. EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS are absent entirely, and so is anything published after 2024. Notably, **{[vallina2020_misshapes]} itself is in the venue index but has no extracted full text** — the reference work for this page is not in the population the page measures.
+  * **Coverage.** Seven venues only, with 2025 and 2026 incomplete by construction rather than by relevance, so any row that reaches them is a floor; the scope and the selection funnel are on [[literature:corpus]]. Notably, **{[vallina2020_misshapes]} itself is in the venue index but has no extracted full text** — the reference work for this page is not in the population the page measures.
+  * **Every query behind this section, the report script and its unedited output** are on [[provenance:design:website_classification]]; corpus-level caveats are on [[literature:corpus]].
 ===== Open Questions =====
@@ Line 1009: / Line 1066: @@
   * [[Design:Website Selection]] — which domains to study, and popularity rankings.
   * [[Design:IP Classification]] — the same problem for addresses instead of names, with the same "which question are you actually asking" structure.
+  * [[Design:Crawling Location]] — where you crawl from, which is a different question from which country a site is aimed at ([[#Target country is not topic]]).
   * [[Privacy:Requests]] — tracker and advertiser classification, where filter lists beat categorisers.
   * [[Privacy:Cookies]], [[Privacy:Fingerprinting]], [[Privacy:JavaScript]] — classification of the other things a crawl collects.