Differences

This shows you the differences between two versions of the page.

--- design:website_classification [2026/08/07 10:54] – Website Categorify's domain no longer resolves (checked 2026-08-07) and Klazify is behind a Cloudflare interstitial that blocks both curl and a headless browser; report both honestly instead of linking a dead vendor. Authored by Claude. karel.kubicek.claude
+++ design:website_classification [2026/08/27 14:39] (current) – CrUX licence: Tranco homepage CC BY-SA 4.0, methodology silent, Google CC BY 4.0. Authored by Claude karel.kubicek.claude
@@ Line 5: / Line 5: @@
 This page is about turning a domain into a category you can defend. Its companion [[Design:Website Selection]] is about which domains to put on the list in the first place; popularity ranking lives there, category and company data live here.
-Before anything else: **"classify a website" is not one measurement.** It is at least five, they use different data, and they are not equally reliable.
+Before anything else: **"classify a website" is not one measurement.** It is at least six, they use different data, and they are not equally reliable.
 ^ Question ^ Where the answer comes from ^ How good it is ^ Free at scale? ^
 | What //topic// is this site about? | Commercial categorisation vendors, Curlie, an open model, an LLM | The subject of this page. Vendor coverage ranges from 94% to 0.5%; vendors agree with each other poorly | Rarely |
+| Which //country// is it aimed at? | TLD, CrUX country lists (via Tranco), site language, host IP, an LLM | None of these is a target-audience label; they are proxies that disagree. See [[#Target country is not topic]] | Yes |
 | What //industry// is the company behind it in? | Company datasets (PeopleDataLabs, Crunchbase, Orbis), not web categorisers | Decent for large firms, noisy in the long tail, and the domain↔company join is the weak link | Partly |
 | Is it //abusive// — phishing, malware, scam? | Reputation feeds: VirusTotal, Google Safe Browsing, PhishTank | A different question with its own literature. Do not read a security verdict as a topic label | Rate-limited |
@@ Line 14: / Line 15: @@
 | What //software// runs it? | Wappalyzer, WhatWeb, response fingerprints | Unrelated to topic. A crawler-instrumentation question, see [[Programming:Crawler]] | Yes |
-The rows are different measurements with different failure modes, and the extraction over our publication corpus shows papers routinely reaching for a topic categoriser to answer one of the other four (see [[#Use in Publications]]).
+These are different measurements with different failure modes, and they are easy to conflate — the structured extraction behind [[#Use in Publications]] files them all under one label, because papers describe them all as "classifying websites". The tracker row is where the conflation costs most, and unlike the others it has been measured: see [[#The trap that has caught the most papers]].
 <WRAP important>
 **A categorisation service is a measurement instrument, and you have to characterise it before you use it.** Not "we categorised domains using X" — that is not a method, it is a citation. What a reviewer needs is: which service and which version, how many of your domains it actually labelled, what its taxonomy is, and what precision it achieved //on your sample//, established by hand.
-Of the 247 papers in our corpus that assign topic categories to websites, **38.5% report any validation of the resulting labels at all.** That is the single largest methodological gap this page exists to close, and [[#Validating Your Labels]] gives you a script that closes it in an afternoon.
+Of the 330 papers in our corpus that assign topic categories to websites, **37.3% report any validation of the resulting labels at all.** That is the single largest methodological gap this page exists to close, and [[#Validating Your Labels]] gives you a script that closes it in an afternoon.
 </WRAP>
@@ Line 28: / Line 29: @@
 ===== The One Study Everyone Cites, and What It Found =====
-Vallina et al. {[vallina2020_misshapes]} studied 13 domain classification services and retrieved labels from 11 of them for **4,424,142 domains** (September 2019, Alexa rankings aggregated Tranco-style; DMOZ and Curlie were excluded from the label comparison as discovery directories, and three rate-limited services could only be measured on the top-10k). Six years on it is still the reference, so it is worth having its actual numbers rather than the vibe of them.
+Vallina et al. {[vallina2020_misshapes]} studied 13 domain classification services. DMOZ and Curlie were set aside as discovery directories rather than classifiers, leaving 11 in the label comparison. **Nine label sources** — OpenDNS, McAfee, FortiGuard, and Alexa, Bitdefender, Forcepoint, Dr.Web, Trend Micro and Websense as re-served through VirusTotal — were queried across the full **4,424,142 domains** (September 2019, Alexa rankings aggregated Tranco-style). **Three** — Symantec, Trend Micro queried directly, and Webshrinker — were rate-limited down to the top-10k. The case studies in §6 cover nine services. Trend Micro appears in both groups because it was queried two ways, and Websense appears only via VirusTotal; **counting "how many services" is genuinely ambiguous in this paper, which is why every figure below names its own group.** Six years on it is still the reference, so it is worth having its actual numbers rather than the vibe of them.
 <WRAP right box>
@@ Line 41: / Line 42: @@
 **Documented taxonomies do not match observed ones.** Of the services whose documentation could be compared against their output, every one except Symantec returned at least one category that is not in its own documentation. McAfee returned six categories that its own docs had marked deprecated since 2010. FortiGuard omits sensitive categories (e.g. //homosexuality//) from public documentation while still assigning them. Trend Micro's labels //via VirusTotal// matched its 2011 taxonomy better than its 2019 one, implying VirusTotal was sourcing from an older product.
-**Labels are multi-valued in ways you will not expect.** Six services — OpenDNS, McAfee, Dr.Web, Forcepoint, Trend Micro and Websense — can return more than one category per domain. In Dr.Web, **67%** of labelled domains have several; in Forcepoint and Websense, under 1%. If your pipeline assumes one label per domain, you are silently dropping data from some services and not others.
+**Labels are multi-valued in ways you will not expect.** Six of the label sources they observed — OpenDNS, McAfee, Dr.Web, Forcepoint, Trend Micro and Websense — can return more than one category per domain. In Dr.Web, **67%** of labelled domains have several; in Forcepoint and Websense, under 1%. If your pipeline assumes one label per domain, you are silently dropping data from some services and not others.
 **Human labelling is genuinely hard.** In the paper's own small manual experiment, two labellers using the OpenDNS taxonomy **disagreed on 35.5% of domains**, and reached only 71% accuracy against OpenDNS's own answer. Budget for this: it is the number your inter-annotator agreement is competing against, and it is worse than most people assume before they try it.
@@ Line 51: / Line 52: @@
 ^ Ground truth ^ N ^ Best service by //related// label ^ How the others did ^
 | Advertising and tracking domains (EasyList + EasyPrivacy, sanitised) | 24,825 | Forcepoint, **15.1%** | McAfee 3.7%, FortiGuard 7.7%, Dr.Web 0.0% — despite McAfee labelling 70.8% and FortiGuard 78.7% of them //something// |
-| Adult content (curated porn plus government gambling lists) | 3,519 | FortiGuard, **98.8%** | McAfee 97.6%, Forcepoint 94.9%, Dr.Web 92.4% — but Trend Micro 11.8%, Websense 4.5%, Alexa 0.1% |
+| Adult content (curated porn plus government gambling lists) | 3,519 | FortiGuard, **98.8%** | McAfee 97.6%, Forcepoint 94.9%, Dr.Web 92.4%, OpenDNS 88.0% — but Trend Micro 11.8%, Websense 4.5%, Alexa 0.1% |
 | CDN and hosting domains (CNAME matched against WebPageTest signatures) | 2,858 | McAfee, **84.7%** | FortiGuard 81.7%, everyone else below 30% |
@@ Line 65: / Line 66: @@
 <WRAP important>
-**Nobody has repeated Vallina et al.'s audit.** Searching outside our seven venues for 2021–2026 turns up no multi-service, multi-million-domain re-benchmark. Six years is a long time in this market — every vendor on the 2020 list has since been acquired, renamed, spun out or re-tiered — and the one modern data point below suggests the services have got //worse// relative to the alternatives, not better. **This is an open, fundable, well-scoped measurement paper and it is sitting there unwritten.**
+**Nobody has repeated Vallina et al.'s audit.** Searching outside our seven venues for 2021–2026 turns up no multi-service, multi-million-domain re-benchmark. Six years is a long time in this market: since the audit, McAfee's database has passed to Trellix and Skyhigh, OpenDNS's community tagging has been switched off, Alexa has been retired outright, and the free tiers of what remains have narrowed to the point of uselessness. The one modern data point below suggests the services have got //worse// relative to the alternatives, not better. **This is an open, fundable, well-scoped measurement paper and it is sitting there unwritten.**
 </WRAP>
 The one modern data point comes from a 2025/2026 arXiv preprint on LLM-assisted web measurement {[bozzolan2026_llmweb]}, whose Appendix B explicitly reuses Vallina et al.'s service list. Its findings:
-  * Of the surviving free tiers, **only Cloudflare and VirusTotal offer one at all**, and both are unusable at scale: Cloudflare's Domain Threat Intelligence API allows 100 requests //per month//, so 100k websites would take "roughly 30,000 days"; VirusTotal's 500/day cap puts the same job at "around 200 days".
+  * Of the services they surveyed, **three offer a free API at all** — Google Topics, Cloudflare Radar and VirusTotal — and Topics is deprecated and scheduled for removal, leaving two. The rate-limited ones are hopeless at scale: Cloudflare's Domain Threat Intelligence API allows 100 requests //per month//, so 100k websites would take "roughly 30,000 days"; VirusTotal's 500/day cap puts the same job at "around 200 days". **For topic labels specifically that leaves one free API, VirusTotal, at 200 days per 100k domains** — Cloudflare's unmetered endpoint is the other one, and it answers a different question (next bullet).
-  * The Cloudflare **Radar** API has no explicit rate limit, so they could benchmark it. Against a 7,780-site ground truth for target-audience country, Radar **returned a label for 3,925 sites (50.4%)** and, of those, **agreed with the ground truth in 39.7% of cases**. They hand-checked 50 disagreements: their own label was right in 49 (98%), Cloudflare's in 1 (2%).
+  * The Cloudflare **Radar** API has no explicit rate limit, so they could benchmark it — **but on country, not on topic**, and the distinction matters because the number below gets quoted as if it were a categorisation result. Against a 7,780-site ground truth for target-audience country, Radar **returned a label for 3,925 sites (50.4%)** and, of those, **agreed with the ground truth in 39.7% of cases**. They hand-checked 50 disagreements: their own label was right in 49 (98%), Cloudflare's in 1 (2%). The reason is mechanical: Radar assigns a domain the country that issues the most DNS queries for it against Cloudflare's resolver, which is a measure of //where the traffic is//, not of who the site is for. **No comparable 2020s coverage-or-accuracy number exists for anyone's topic labels.**
   * Every open-weight LLM they tested reached 100% coverage and at least 76% accuracy on the same data, without visiting the sites.
@@ Line 78: / Line 79: @@
 ===== The Service Catalogue =====
-Status of every service checked by direct fetch on **7 August 2026**. Where a lookup exists but no API does, that is stated plainly — it is the most common case, and it is what makes these services awkward for research.
+Each service below was checked by direct fetch on **7 August 2026**; where something could not be verified from a primary source, a footnote says so. Where a lookup exists but no API does, that is stated plainly — it is the most common case, and it is what makes these services awkward for research.
 ==== Security and content-filtering vendors ====
@@ Line 104: / Line 105: @@
 === Symantec — now Broadcom ===
-Broadcom acquired Symantec Enterprise in 2019; WebFilter and WebPulse are still operated under the Symantec brand and still shipping categorisation updates in 2026.
+Broadcom acquired Symantec Enterprise in 2019; WebFilter and WebPulse are still operated under the Symantec brand. Both the lookup tool and Broadcom's WebFilter/WebPulse documentation served HTTP 200 on 2026-08-07, which is the strongest statement we can make from outside — Broadcom publishes categorisation-update notices on its status site, but we could not fetch a specific one to date them.
   * **Lookup**: [[https://sitereview.symantec.com/|sitereview.symantec.com]] — alive. It is an Angular single-page app, so a plain ''curl'' returns a splash screen with a ''<noscript>'' notice; that is a JavaScript UI, not a dead service. Category definitions live in [[https://techdocs.broadcom.com/us/en/symantec-security-software/web-and-network-security/edge-swg/7-3/getting-started/page-help-administration/page-help-data-services/webfilter-webpulse.html|Broadcom TechDocs]].
@@ Line 131: / Line 132: @@
   * **Advantages**: minimalistic and consistent labels; recovered 92.4% of adult content, which is its whole design goal.
   * **Disadvantages**: very low coverage outside its niche — it returned //any// label for 9.0% of tracking domains and 0.3% of CDN domains, and a correctly related label for 0.0% of both; **67% of its labelled domains carry several categories**, the highest multi-labelling rate measured, which breaks single-label pipelines silently.
+=== Cyren — now Data443 ===
+Three corpus papers categorise domains with Cyren, so it is worth knowing where it went. As of 2026-08-07 both ''cyren.com'' and ''www.cyren.com'' 301 to **[[https://data443.com/cyren-threat-intelligence/|data443.com/cyren-threat-intelligence]]** — the brand survives as a Data443 threat-intelligence product line. If you are reproducing one of those papers, you are not querying the same organisation they did.
 === Two vendors the earlier version of this page missed ===
@@ Line 143: / Line 148: @@
 === SimilarWeb ===
-  * **Advantages**: high-quality data; industry, region- and origin-based popularity; the taxonomy used in practice is ~24 top-level website categories.
+  * **Advantages**: high-quality data; industry, region- and origin-based popularity. A PETS 2021 paper reports classifying into "24 possible categories" via the category feature; SimilarWeb does not publish a canonical taxonomy size, so pin the number you observe rather than quoting that one.
   * **Disadvantages**: the official [[https://developers.similarweb.com/|API v5]] is paid on a "Data Credits" model with no published price list — you get a sales quote. The unofficial endpoint is a **clear terms violation**: SimilarWeb's terms prohibit //"use any robot, spider, scraper, or other automated means to access or monitor the Platform, Site or Applications for any purpose, or to copy, access, collect, download, or otherwise obtain data, content or other information"//.((https://www.similarweb.com/corp/legal/terms/ — clause (v), fetched 2026-08-07.))
   * See [[Programming:SimilarWeb]] for documentation of the unofficial API, code and output example, and note the date it was last confirmed working.
@@ Line 158: / Line 163: @@
 Two more appear in the corpus, and their fates illustrate the risk of building on a small vendor.
-  * **[[https://klazify.com/|Klazify]]** — used by an IMC 2023 paper to map dead links to categories. The domain resolves and is served behind Cloudflare, but Cloudflare's bot interstitial returned HTTP 403 to both ''curl'' and a headless browser on 2026-08-07, so we could not confirm anything about the current product from the site itself.
+  * **[[https://klazify.com/|Klazify]]** — two corpus papers use it: an IMC 2023 study mapping dead links to categories, and a USENIX Security 2024 brand-impersonation study that took its **IAB classification** and manually validated the result. The domain resolves and is served behind Cloudflare, but Cloudflare's bot interstitial returned HTTP 403 to both ''curl'' and a headless browser on 2026-08-07, so we could not confirm anything about the current product from the site itself.
   * **Website Categorify** — used by a CCS 2020 paper on DNS hijacking, which reported 195 domains left uncategorised. As of 2026-08-07 ''websitecategorify.com'' **does not resolve at all**: the domain is gone, and with it any hope of reproducing that paper's categorisation.
@@ Line 181: / Line 186: @@
   * **Bulk dump**: [[https://curlie.org/directory-dl|curlie.org/directory-dl]] redirects to the current archive. Tab-separated text, gzipped. Fetched 2026-08-07: **169 MiB (177,289,960 bytes), last modified 2 February 2026**, covering ~2.9 million entries with the category hierarchy, URLs, titles and editorial descriptions. The filenames still say "RDF" for historical reasons; the format is TSV.
   * **Cadence**: Curlie's documentation says "we strive to pull a fresh copy from the Curlie database every month". The copy we fetched was six months old, so **check ''Last-Modified'' before you cite a snapshot date** rather than assuming monthly freshness.
-  * **Taxonomy**: 15 top-level categories in the English tree — Arts, Business, Computers, Games, Health, Home, News, Recreation, Reference, Regional, Science, Shopping, Society, Sports, Kids and Teens — deeply hierarchical below that, with 92 further language trees alongside. Counted from [[https://curlie.org/en/|curlie.org/en]] on 2026-08-07. Homepage2Vec predicts into 14 of these, dropping //Regional//.
+  * **Taxonomy**: 15 top-level categories in the English tree — Arts, Business, Computers, Games, Health, Home, News, Recreation, Reference, Regional, Science, Shopping, Society, Sports, Kids and Teens — deeply hierarchical below that, with 92 further language trees alongside. Counted from [[https://curlie.org/en/|curlie.org/en]] on 2026-08-07. Homepage2Vec predicts into 14 of these, dropping //Regional//; the LLM benchmark in [[#LLM Classification]] reduces the same taxonomy to 13. If three sources hand you 15, 14 and 13 Curlie categories, none of them is a typo — check which ones each one dropped before you compare their numbers.
   * **Advantages**: free, redistributable, human-curated, multilingual, and it is the training set behind essentially every open website classifier — including [[#Homepage2Vec|Homepage2Vec]].
   * **Disadvantages**: 2.9M URLs is tiny against the modern web and skews to older, established sites; many branches have not been touched in years. **The most important caveat is under-labelling**: editors assign one or two relevant categories, not all of them. When Homepage2Vec's authors had crowdworkers re-label 807 Curlie sites, Curlie had supplied 836 labels (1.04 per site) and the crowd assigned 2,088 (2.59 per site) — a 2.5× increase {[lugeon2022_homepage2vec]}. **Treated as ground truth, Curlie will make any multi-label classifier look far less precise than it is.**
@@ Line 190: / Line 195: @@
 === VirusTotal ===
-  * **API**: ''GET /api/v3/domains/{domain}'' returns a ''categories'' attribute, //"a dictionary mapping that relates categorisation services with the category it assigns the domain to"//. Documented at [[https://docs.virustotal.com/reference/domains-object|docs.virustotal.com]].
+  * **API**: ''GET /api/v3/domains/{domain}'' returns a ''categories'' attribute, //"a dictionary mapping that relates categorisation services with the category it assigns the domain to"//. Documented at [[https://docs.virustotal.com/reference/domains-object|docs.virustotal.com]]. **That is a topic label, not a maliciousness verdict.** The oracle use of the same API — engine disagreement, thresholds, URL versus file lookup, snapshot dating — is [[security:virustotal]].
   * **Which vendors**: Dr.Web, Forcepoint ThreatSeeker, BitDefender, Sophos, Trend Micro, Websense and (legacy) Alexa, among others — so several of the vendors above reach you through VirusTotal without you querying them.
-  * **Rate limit**: the free tier is **500 requests per day and 4 requests per minute**, unchanged in substance by the October 2025 retiering into Community / Contributor / Lite / Duet.((https://docs.virustotal.com/docs/public-vs-premium-api — fetched 2026-08-07.)) The same page adds that the Public API //"must not be used in commercial products or services"//. At 500/day, a 100k-domain study takes 200 days. Vallina et al. had an academic key at 20k/day; there is no self-serve academic application form today, so budget for a direct conversation with VirusTotal rather than assuming access.
+  * **Rate limit**: the free tier is **500 requests per day and 4 requests per minute**.((https://docs.virustotal.com/docs/public-vs-premium-api — fetched 2026-08-07; that page names the "Community" tier but does not mention the October 2025 retiering, so treat the tier names as unconfirmed and the numbers as current.)) The same page adds that the Public API //"must not be used in commercial products or services"//. At 500/day, a 100k-domain study takes 200 days. Vallina et al. had an academic key at 20k/day. As of 2026-08-27 there is still no standalone quota-application form; the public contact form has a subject "I have an academic research request" ([[https://www.virustotal.com/gui/contact-us/legal|virustotal.com/gui/contact-us/legal]]). Budget for that conversation rather than assuming 20k/day. Maliciousness-oracle use of the same API is [[security:virustotal]].
   * **Advantages**: aggregates many providers in a single call; widely used and easy to cite.
   * **Disadvantages**: **the integration is lossy**. In the 2020 audit, several services returned labels when queried directly but not through VirusTotal, and Trend Micro's VirusTotal labels tracked its 2011 taxonomy rather than its 2019 one. You do not control which product version you are reading, and "we used VirusTotal categories" does not identify the underlying source.
@@ Line 204: / Line 209: @@
 Two different products, easily conflated:
-  * **[[https://developers.cloudflare.com/radar/|Radar API]]** — free, no explicit rate limit, data licensed **CC BY-NC 4.0** (non-commercial only, which matters for industry co-authors). Its measured quality is poor: 50.4% coverage and 39.7% agreement on the one 2026 benchmark that exists {[bozzolan2026_llmweb]}.
+  * **[[https://developers.cloudflare.com/radar/|Radar API]]** — free, no explicit rate limit, data licensed **CC BY-NC 4.0** (non-commercial only, which matters for industry co-authors). **Radar is a popularity-ranking product, not a categoriser**, and this is the single most common misreading of it on this page's topic. It does return a ''categories'' array, but only from ''GET /radar/ranking/domain/{domain}'', only for the ''POPULAR'' ranking, and only for the most recent ranking — so a domain gets categories only if Radar ranks it, and Radar ranks an ordered top 100 (global and per country) plus unordered buckets up to one million.((https://developers.cloudflare.com/api/resources/radar/subresources/ranking/subresources/domain/ and https://developers.cloudflare.com/radar/investigate/domain-ranking-datasets/ — both fetched 2026-08-07. The rate limit is unstated rather than absent; treat it as undocumented, not infinite.)) There is no Radar endpoint that categorises an arbitrary domain. The 50.4% / 39.7% figures quoted above are Radar's **country** output, not its categories {[bozzolan2026_llmweb]}; **nobody has published a coverage or accuracy figure for Radar's topic categories at all.**
   * **Domain Intelligence, now under Security Center Intel** — ''GET /accounts/{account_id}/intel/domain?domain=…'' returns ''content_categories''. This is the one the corpus papers used: {[ruth2022_toppling]} labelled Cloudflare-managed sites with it, and {[ruth2022_world]} built a whole global browsing study on it. It needs an API token with Intel permission scoped to an account; whether it is available on the free plan is not stated in Cloudflare's public docs.((Checked 2026-08-07 against https://developers.cloudflare.com/api/resources/intel/subresources/domains/methods/get/ — the docs describe the permission requirement but not the plan tier.))
@@ Line 221: / Line 226: @@
 === Alexa ===
-Amazon retired **Alexa.com on 1 May 2022**, per its own end-of-service notice ("we will be retiring Alexa.com on May 1, 2022").((Captured on the alexa.com login page, https://web.archive.org/web/20220315000000/https://www.alexa.com/ — retrieved 2026-08-07. [[Design:Website Selection]] currently gives 1 August 2023; we could not find a primary source for that date, and ''alexa.com'' now redirects to the unrelated Amazon Alexa voice assistant.)) Its ranking service and its category service died together.
+Amazon retired **Alexa.com on 1 May 2022**, per its own end-of-service notice ("we will be retiring Alexa.com on May 1, 2022").((Captured on the alexa.com login page, https://web.archive.org/web/20220315000000/https://www.alexa.com/ — retrieved 2026-08-07. An earlier revision of [[Design:Website Selection]] gave 1 August 2023; that date is when Tranco dropped Alexa from the default list, not when Amazon switched the service off. The selection page now carries both dates. ''alexa.com'' redirects to the unrelated Amazon Alexa voice assistant.)) Its ranking service and its category service died together.
-  * **Why it still matters**: it appears in 12 of the 247 corpus papers that categorise websites, under eight different spellings, and papers published as late as 2024 still use it because their data collection predates the shutdown. If you are reading such a paper, the labels are from a dead service with a documented 0.53% coverage rate.
+  * **Why it still matters**: it appears in 12 of the 330 corpus papers that categorise websites, under eight different spellings, and papers published as late as 2024 still use it because their data collection predates the shutdown. If you are reading such a paper, the labels are from a dead service with a documented 0.53% coverage rate.
   * **What it was**: highly granular (7,557 distinct labels observed through VirusTotal, against 279,716 categories in Alexa's own directory, which was derived from DMOZ), and capped at 500 sites per category — hence the tiny coverage. Leaf labels came back in the language of the branch, so the same concept appeared as //Arts//, //Artes// and //Kultur//.
@@ Line 240: / Line 245: @@
 | SimilarWeb | Alive | Paid API only | — | ~24 categories | you also need traffic and industry data |
 | VirusTotal | Alive | 500/day free | Lower than direct | Per underlying vendor | your sample is small, or you want vendor disagreement |
-| Cloudflare Radar | Alive | Free, CC BY-NC | 50.4% (2026, one task) | Yes | you can validate per-category and drop what fails |
+| Cloudflare Radar | Alive | Free, CC BY-NC | **Unmeasured for topics** — ranked domains only | Yes | your domains are inside Radar's top-1M ranking |
+| Cloudflare Domain Intelligence | Alive | Token, ~100/month free | Unmeasured | 26 super / 114 categories | you can validate per-category and drop what fails |
 | Curlie | Alive | **Free bulk dump** | Small but open | Yes, 15 top-level | you need redistributable ground truth |
 | OpenDNS community | **Dead** | — | 15% | — | never |
@@ Line 279: / Line 285: @@
 <WRAP important>
-**This is where our corpus runs out, and you should know exactly where.** Of the 4,322 papers in the 2010–2024 corpus, 29 use an LLM to classify anything at all — 27 of them in 2024 — and they point it at privacy policies, app reviews, vulnerabilities, jailbreak prompts and content moderation. **Zero of them use an LLM to assign a website a topic category.** Any ranking of methods built from this literature will therefore tell you that third-party services and hand-built keyword rules are the state of the art. As of 2026 that is no longer true, and the corpus cannot see why.
+**The corpus now reaches 2026, and the picture changed.** Of the 5,859 papers in the 2010–2026 corpus, **177 use an LLM to classify something** — 2 in 2023, 27 in 2024, 77 in 2025 and 71 in the partial 2026 — and **two of them use one to assign a website a topic category**. The nearer of the two, {[tsai2025_harmful]}, reports **92% accuracy for GPT-4o on shopping-vs-non-shopping over 500 manually annotated Tranco sites, and says explicitly that this is "comparable to commercial website classification services"**. So a ranking of methods built from this literature no longer tells you that third-party services and keyword rules are the only options — but it still gives you a two-paper base for the topic task specifically, in the corpus's two thinnest and provisional years. Treat the peer-reviewed evidence as thin rather than absent, and read the rest of this section for what is outside the corpus.
 </WRAP>
-The evidence outside the corpus is a 2025/2026 preprint, {[bozzolan2026_llmweb]}, which by its authors' account is the first systematic study of LLMs for web-measurement classification. It evaluates five **open-weight, self-hosted** models (gpt-oss:20b, llama3.3:70b, mistral-small3.2:24b, phi4-mini:3.8b, qwen3:32b, all via Ollama, temperature 0) on three tasks with hand-built benchmarks, in two modes: URL string only, and URL plus live browsing through LangChain and Playwright.
+The evidence outside the corpus is a 2025/2026 preprint, {[bozzolan2026_llmweb]}, which by its authors' account is the first systematic study of LLMs for web-measurement classification.
+<WRAP important>
+**It is one unrefereed preprint, and it has moved under its own feet.** Everything in this section rests on a single arXiv posting that has not been through peer review, and whose results are not stable across versions. **v1** (9 October 2025) benchmarked a cloud model — Gemini 2.5 Flash — against Llama 4:109B and Gemma 3:27B, on a **14**-category Curlie task. **v3** (30 April 2026) drops the cloud model entirely, evaluates five open-weight models instead, reduces the topic task to **13** categories, and adds two things that appear nowhere in v1: a Homepage2Vec baseline, and {[vallina2020_misshapes]} — the audit this whole page is built on, which v1 does not cite once.((Version dates from https://arxiv.org/abs/2510.08101 (submission history); v1 text read from https://arxiv.org/html/2510.08101v1, which contains zero occurrences of Vallina et al. 2020 and one passing mention of Homepage2Vec with no comparison. Both fetched 2026-08-07.)) **This page reads v3.** Cite the version you read, and re-derive any number you lean on — this one changed once already, and a preprint that acquires its baseline and its closest related work between postings is a preprint whose next revision may move again.
+</WRAP>
+It evaluates five **open-weight, self-hosted** models (gpt-oss:20b, llama3.3:70b, mistral-small3.2:24b, phi4-mini:3.8b, qwen3:32b, all via Ollama, temperature 0) on three tasks with hand-built benchmarks, in two modes: URL string only, and URL plus live browsing through LangChain and Playwright.
 ^ Task ^ N ^ Classes ^ Best URL-only ^ Best URL + live visit ^
@@ Line 290: / Line 302: @@
 The number that matters for this page: on the same 13-category benchmark, **Homepage2Vec scores 0.60 Jaccard and 0.55 macro F1** — the paper's words are that "a general-purpose LLM like gpt-oss can label websites with higher accuracy than a classic machine learning model specifically trained for website categorization." A stock open-weight instruct model, with no fine-tuning, beat the field's reference specialist classifier.
+Two cautions before you repeat that sentence.
+  - **0.55 is not a refutation of the 0.90 in Homepage2Vec's abstract**, and anyone who puts the two side by side has compared different evaluation regimes. 0.90 is the balanced per-class setup; Homepage2Vec's own authors report **0.634** macro F1 on an unbalanced test set, which is the regime a benchmark like this one is in. See [[#Homepage2Vec]].
+  - **0.55 is nonetheless below that 0.634**, on a task that drops one of the 14 classes, and the preprint never states how it ran the baseline. Homepage2Vec emits independent //calibrated probabilities// per class, so turning it into a multi-label prediction requires a decision threshold — its own paper uses 0.5 on calibrated output — and the word "threshold" does not occur anywhere in the preprint.((Verified by full-text search of https://arxiv.org/html/2510.08101v3 on 2026-08-07: zero occurrences of "threshold".)) A head-to-head in which the winner ran the loser's model, without reporting the one hyperparameter that decides the loser's score, is a result to re-run before you build on it. Both models are free and the benchmark is small.
 Three practical consequences:
@@ Line 303: / Line 320: @@
   * **Label instability across seeds and temperature** for website classification specifically. The one systematic paper set temperature to 0 to sidestep the question rather than quantify it.
   * **Behaviour on parked, dead and holding-page domains** — a large fraction of any long-tail sample, and a case where "visit the site" gives the model a page that is not about anything.
-  * **A formal agreement statistic** (Cohen's or Fleiss' kappa) between LLM labels and human annotators for this task. Nobody has published one. The nearest available figures are a validation of the //benchmark's own ground truth//, not of an LLM: the authors hand-relabelled 150 of their 3,785 sites, and their automated label-construction procedure matched the human label set exactly on 86 (57%) with a further 47 (31%) a proper subset of it, where Lugeon et al.'s Curlie snapshot matched exactly on only 67 (45%) of the same sites. That is a statement about how good the ground truth is — and a reason to read the 0.65 Jaccard above as a floor on model quality rather than a ceiling, because a multi-label benchmark that under-labels punishes a model for being right.
+  * **A formal agreement statistic** (Cohen's or Fleiss' kappa) between LLM labels and human annotators for this task. Nobody has published one. What does exist, on those same 150 hand-relabelled sites, is a Jaccard similarity of **0.73 between the human labels and gpt-oss's predictions**, against **0.75 between the human labels and the benchmark's own ground truth** — the model is as close to careful humans as the benchmark is. Which is also the reason to read the 0.65 Jaccard above as a floor rather than a ceiling: a multi-label benchmark that under-labels punishes a model for being right. (For scale on how much under-labelling there is: the benchmark's own heuristic label-construction matched the human label set exactly on 86 of 150 sites (57%), where Lugeon et al.'s Curlie snapshot matched on 67 (45%). Both figures are about the ground truth, not about any model.) **And note what that pair cannot tell you:** those 150 sites were relabelled by the preprint's own authors, with no second annotator and no agreement statistic of any kind — no kappa, no Krippendorff's alpha, nothing.((Verified by full-text search of https://arxiv.org/html/2510.08101v3 on 2026-08-07: zero occurrences of "kappa", "inter-annotator", "Fleiss" or "Krippendorff".)) So 0.73 says the model tracks //that annotator's// labels. It does not say how fuzzy the task is, and a single-annotator relabelling cannot be used to argue that the remaining label noise is small — the disagreement you have not measured is exactly the quantity in question. If you relabel a sample, use two annotators and report the agreement; [[#Validating Your Labels]] is the whole procedure.
+===== Target country is not topic =====
+Studies that check compliance against a national law need to know which country a site is //for//, and this gets treated as a minor preprocessing step. It is not. There is no service that answers it, the available proxies measure different things, and none of them is a target-audience label:
+  * **The TLD.** Precise where it exists and absent where it matters — a ccTLD is strong evidence, but generic TLDs carry no country at all, and that is most of the head of any list.
+  * **CrUX country lists, via Tranco.** Tranco's list-generation API takes ''filterCRUX'', ''filterCRUXType'' (''global'' / ''country'' / ''region'' / ''subregion''), ''filterCRUXValue'' (e.g. a list of country codes) and ''filterCRUXMonth'', so you can generate a country-restricted list reproducibly, with a permalink, from a free account.((https://tranco-list.eu/api_documentation — fetched 2026-08-07. Basic Auth with your email as username and API token as password; the ''/configure'' page returns 401 without a login. CrUX datasets are CC BY 4.0 per Google's methodology, fetched 2026-08-27; Tranco's homepage still labels CrUX CC BY-SA 4.0, while the methodology page states no CrUX licence.)) This is the cleanest free source and it is under-used. **But it ranks by page loads from Chrome users in a country, which is popularity, not audience** — and the overlap is severe: the union of five country top-10k lists (China, Germany, Italy, Korea, Turkey) is only 18,718 domains rather than 50,000, with 4,017 domains common to all five, and a preliminary labelling built from these lists is incompatible with the site's own TLD in at least 25% of cases for every label {[bozzolan2026_llmweb]}.
+  * **Site language.** A good proxy, and cheap, but it splits badly on English, Spanish, Arabic and Portuguese, which is a large share of the web.
+  * **Host IP or CDN location.** Measures where bytes are served from. Behind a CDN — most of the head of any list — it tells you about the CDN. See [[Design:IP Classification]] and [[Design:Crawling Location]].
+  * **Cloudflare Radar's country field.** DNS-query volume by country against Cloudflare's resolver: 50.4% coverage and 39.7% agreement on the one benchmark that exists (see [[#Has Anything Changed Since 2020?]]).
+**So combine them and report the combination.** The one systematic evaluation on this task got its ground truth by starting from CrUX and then correcting with site language and host IP, and still hand-inspected the result — and an LLM given the URL alone beat the best single proxy on that benchmark. Whatever you do, say which proxies you combined and how you broke ties: "we classified sites by country" is not a method, and a compliance result is only as good as the jurisdiction assignment underneath it.
 ===== Taxonomies =====
-Pick the taxonomy before the service, because it is much harder to change later, and because 72% of the corpus papers that name a taxonomy at all name a bespoke one they invented (see [[#Use in Publications]]) — which makes their results incomparable with everyone else's.
+Pick the taxonomy before the service, because it is much harder to change later, and because 75% of the corpus papers that name a taxonomy at all name a bespoke one and do not even state how many classes it has (see [[#Use in Publications]]) — which makes their results incomparable with everyone else's.
-  * **[[https://github.com/InteractiveAdvertisingBureau/Taxonomies|IAB Tech Lab Content Taxonomy]]** — the advertising industry's reference schema, and the right choice if your question is about ad targeting or if you want to be comparable to Webshrinker's output. Machine-readable TSV in the repo under ''Content Taxonomies/'', CC BY 3.0, up to four tiers with a relational ID system. Versions 1.0 through **3.1** sit side by side; 2.x → 3.0 is a **breaking change** and IAB says the two must not be used together. Category rows counted from the TSVs on 2026-08-07: **2.2 has 1,196 rows under 37 tier-1 categories; 3.0 has 703 under 36; 3.1 has 704 under 37.** Vendor marketing claiming that 3.x "expands to 1,500+ categories" is wrong in both directions — 3.x is roughly //40% smaller// than 2.2, because it consolidated and restructured rather than adding. Count the rows in the file yourself; it takes ten seconds and the summaries are not reliable.
+  * **[[https://github.com/InteractiveAdvertisingBureau/Taxonomies|IAB Tech Lab Content Taxonomy]]** — the advertising industry's reference schema, and the right choice if your question is about ad targeting or if you want to be comparable to Webshrinker's output. Machine-readable TSV in the repo under ''Content Taxonomies/'', CC BY 3.0, up to four tiers with a relational ID system. Versions 1.0 through **3.1** sit side by side; 2.x → 3.0 is a **breaking change** and IAB says the two must not be used together. Category rows counted from the TSVs on 2026-08-07, excluding the two header lines: **2.2 has 1,196 rows, 3.0 has 703, 3.1 has 704** — so 3.1 is marginally //larger// than 3.0. Tier-1 rows: 37 in 3.1, but only 31 of those carry a numeric Unique ID; the other six (Communication, Entertainment, Events, Genres, Holidays, Sensitive Topics) are alphanumeric extension rows. **State which convention you counted with** — three independent counts of this one file in preparing this page produced three different numbers. Vendor marketing claiming that 3.x "expands to 1,500+ categories" is wrong in both directions — 3.x is roughly //40% smaller// than 2.2, because it consolidated and restructured rather than adding. Count the rows in the file yourself; it takes ten seconds and the summaries are not reliable.
-  * **Curlie's 15 top-level categories** — the right choice if you want free ground truth, an open model that already predicts into it, and a hierarchy you can prune to whatever depth your sample supports.
+  * **Curlie's 15 top-level categories** — the right choice if you want free ground truth, an open model that already predicts into it, and a hierarchy you can prune to whatever depth your sample supports. **Check that your classes are actually in it before you commit.** The 15 are Arts, Business, Computers, Games, Health, Home, News, Recreation, Reference, Regional, Science, Shopping, Society, Sports, and Kids and Teens — a directory taxonomy from the early 2000s. There is no top-level class for social media, finance and banking, education, government, or online services, and none at all for adult content; those questions land inside Business, Society, Reference or Computers, or nowhere. If your paper is about the privacy of banking sites, or trackers on education sites, or anything on social platforms, a Curlie-trained classifier will not give you the class you are measuring, and pruning to top level will hide that rather than fix it.
   * **A vendor's own list** — fine, but pin the version, and expect to discover categories that are not in the documentation. Vallina et al. found at least one undocumented category in eight of the nine services whose documentation they could compare against its output.
   * **A standard industry code** (NAICS, SIC, NACE) — the right choice for a question about firms rather than content, and the natural join key to the company datasets below. Exactly one paper in our corpus did this.
-Whatever you pick: **report the number of classes and the pruning depth.** "We categorised sites using IAB" is not reproducible: IAB 3.1 has 37 tier-1 categories and 704 rows in total, and reporting at one depth or the other is two very different measurements.
+Whatever you pick: **report the number of classes and the pruning depth.** "We categorised sites using IAB" is not reproducible: IAB 3.1 has 31–37 tier-1 categories depending on whether you count the extension rows, and 704 rows in total. Reporting at one depth or the other is two very different measurements.
 ===== Company Datasets =====
@@ Line 334: / Line 363: @@
     - Based on LinkedIn profiles that are self-reported — prone to adversarial data.
     - Only a subset of PeopleDataLabs' full dataset. The "22M of 70M rows" framing is longstanding on this page; the 22M is confirmed, the 70M total could not be re-confirmed in 2026 and PDL's marketing page now cites 23.8M+ without saying which corpus that is.
-  <wrap todo>TODO: cite ''Machine Learning Compliance Analysis for Email Regulation'' when it is public.</wrap>
+  <WRAP todo>TODO: cite ''Machine Learning Compliance Analysis for Email Regulation'' when it is public.</WRAP>
 ==== Crunchbase ====
@@ Line 349: / Line 378: @@
     - Focuses mostly on variables useful for investments and market competitiveness.
     - Academic access is no longer publicly documented; budget for a sales conversation.
-  <wrap todo>TODO: cite ''Machine Learning Compliance Analysis for Email Regulation'' when it is public.</wrap>
+  <WRAP todo>TODO: cite ''Machine Learning Compliance Analysis for Email Regulation'' when it is public.</WRAP>
 ==== Orbis ====
@@ Line 548: / Line 577: @@
     print(f"wrote {key}     (the labels under test -- do not show this to annotators)")
+    # Planning sizes use the normal approximation; the reported intervals use
+    # Wilson, which is wider near p = 1. Treat these as a floor, not a promise.
     for hw in (0.20, 0.10, 0.05):
         print(
             f"note: +/-{hw:.0%} half-width at 90% precision needs "
-            f"{sample_size_for(hw)} domains per category"
+            f"{sample_size_for(hw)} domains per category (normal approximation; "
+            f"the Wilson interval reported by `score` is wider near 100%)"
         )
@@ Line 706: / Line 738: @@
 total                                1100      100
-note: +/-20% half-width at 90% precision needs 9 domains per category
+wrote sheet_a.csv  (100 rows, category hidden, order shuffled)
-note: +/-10% half-width at 90% precision needs 35 domains per category
+wrote sheet_b.csv  (100 rows, category hidden, order shuffled)
-note: +/-5% half-width at 90% precision needs 139 domains per category
+wrote sample_key.csv     (the labels under test -- do not show this to annotators)
+note: +/-20% half-width at 90% precision needs 9 domains per category (normal approximation; the Wilson interval reported by `score` is wider near 100%)
+note: +/-10% half-width at 90% precision needs 35 domains per category (normal approximation; the Wilson interval reported by `score` is wider near 100%)
+note: +/-5% half-width at 90% precision needs 139 domains per category (normal approximation; the Wilson interval reported by `score` is wider near 100%)
 ==================================================================
 INTER-ANNOTATOR AGREEMENT
@@ Line 716: / Line 750: @@
 raw agreement            : 67/100 = 67.0%
 Cohen's kappa            : 0.564
+Report both. Kappa alone hides how many categories there were, and raw
+agreement alone is inflated when one category dominates.
 ==================================================================
@@ Line 732: / Line 768: @@
 ALL (disputed counted wrong)       63  100  63.0%  [53.2%, 71.8%]
+The true precision is between these two rows. If they are far apart your
+annotators disagreed too much for the number to mean anything -- fix the
+codebook and re-annotate rather than picking the flattering row.
 ==================================================================
@@ Line 737: / Line 776: @@
 ==================================================================
 /1100 domains got a label = 80.0% [77.5%, 82.3%]
+==================================================================
+PASTE THIS INTO YOUR METHODOLOGY SECTION
+==================================================================
+We categorised 1100 domains, of which 880 (80.0%) received a label. To validate the labels we drew a stratified random sample of 100 domains and had two authors independently assign a category while blind to the service's answer (Cohen's kappa = 0.56, raw agreement 67%). On the 67 domains where the annotators agreed, the service's micro-averaged precision was 94.0% (95% Wilson CI [85.6%, 97.7%]); counting every disputed domain as an error gives a lower bound of 63.0%. We discarded the 1 category that fell below 80% precision (Search Engines) and report results over the remaining 3.
 </code>
-Three things to read out of that:
+Four things to read out of that:
   - **It found the bad category.** Search Engines came back at 55.6% and got dropped, exactly as intended — and it did so from 9 usable annotations, which is why per-category stratification matters more than total sample size.
   - **Kappa 0.56 on a 5-way label set is realistic, not a failure.** Vallina et al.'s own labellers disagreed on 35.5% of domains. If yours agree on 95% you have probably let them see each other's sheets.
-  - **The two "ALL" rows disagree by 31 points**, and that gap is the honest measure of how much the annotation is carrying. Report both. Reporting only the agreed-subset row is how a 63%-precision pipeline gets published as 94%.
+  - **The two "ALL" rows disagree by 31 points**, and that gap is the honest measure of how much the annotation is carrying. The synthetic pipeline's true prevalence-weighted precision is about 88%: reporting only the agreed-subset row publishes it as 94%, reporting only the pessimistic row buries it at 63%, and reporting both brackets it. Report both.
+  - **The micro-average is a diagnostic, not an estimate of your population's precision.** The sample is stratified with equal //n// per category, so a small bad category is overweighted — Search Engines is 6.8% of the population here but 13% of the scored sample. Prevalence-weight before quoting a single number for the whole corpus, or say plainly that the row is per-category diagnostics rolled up.
 <file python category_validation_demo.py>
@@ Line 832: / Line 877: @@
 ===== What to Report =====
-A checklist, derived from what the 247 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply.
+A checklist, derived from what the 330 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply.
   - **Which service or model, and which version.** "McAfee" identified a different company before and after February 2024. "Homepage2Vec" without a commit hash is not reproducible; "GPT-4" without a dated snapshot is not either.
@@ Line 839: / Line 884: @@
   - **The taxonomy and its size.** Name it, give the number of classes, and say at which tier you pruned. If it is bespoke, say so and publish the codebook.
   - **Whether labels are multi-valued**, and how you collapsed them if they are.
-  - **Validation**: sample size, how it was stratified, how many annotators, whether they were blind to the service's label, the agreement statistic, and per-category precision — not one global accuracy number. Publish the annotated sample as an [[:Artifacts|artefact]]; it is small, and it is the only part of this a reader can check.
+  - **Validation**: sample size, how it was stratified, how many annotators, whether they were blind to the service's label, the agreement statistic, and per-category precision — not one global accuracy number. Publish the annotated sample as an [[:Artifacts|artefact]] //(page not yet written)//; it is small, and it is the only part of this a reader can check.
   - **The categories you dropped**, and the rule you dropped them by.
-  - **If you used more than one service, the disagreement between them.** Only 9 of 122 corpus papers that name a service name two, so this is nearly free novelty.
+  - **If you used more than one service, the disagreement between them.** Only 9 of 155 corpus papers that name a service name two, so this is nearly free novelty.
 <WRAP important>
@@ Line 849: / Line 894: @@
 ===== Use in Publications =====
-Two sources here. The historical survey is Vallina et al. {[vallina2020_misshapes]}, whose figures on service popularity are reproduced below. The rest is a fresh query over a structured extraction of **4,322 papers from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2024** — so every figure is a claim about those seven venues over that window, not about the field.
+Two sources here. The historical survey is Vallina et al. {[vallina2020_misshapes]}, whose figures on service popularity are reproduced below. The rest is a fresh query over a structured extraction of **5,859 papers from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026** — so every figure is a claim about those seven venues over that window, not about the field. The 2025 and 2026 venue-years are provisional: CCS and IMC 2026 have not been held, and IEEE S&P and WWW 2026 abstracts are not yet in the selection source.
 <WRAP right box>
@@ Line 858: / Line 903: @@
 ==== The population, and what "website category" hides ====
-The extraction marks 320 papers as classifying something with the target ''website-category''. **That number is not the population this page is about**, and the difference is instructive: the same label covers several unrelated tasks.
+The extraction marks 424 papers as classifying something with the target ''website-category''. **That number is not the population this page is about**, and the difference is instructive: the same label covers several unrelated tasks.
-^ What is actually being classified ^ Papers ^ Share of the 320 ^
+^ What is actually being classified ^ Papers ^ Share of the 424 ^
-| A topic, industry or sector label on a site or domain | 247 | 77.2% |
+| A topic, industry or sector label on a site or domain | 330 | 77.8% |
-| An abuse verdict — phishing, scam, malicious | 47 | 14.7% |
+| An abuse verdict — phishing, scam, malicious | 63 | 14.9% |
-| Website fingerprinting — which of N monitored sites produced this traffic trace | 22 | 6.9% |
+| Website fingerprinting — which of N monitored sites produced this traffic trace | 36 | 8.5% |
-| Hosting or CDN infrastructure | 7 | 2.2% |
+| Hosting or CDN infrastructure | 8 | 1.9% |
-| Page type — homepage, login page, sign-up | 6 | 1.9% |
+| Page type — homepage, login page, sign-up | 7 | 1.7% |
-| Which software runs the site | 6 | 1.9% |
+| Which software runs the site | 6 | 1.4% |
-| Third-party role — tracker, advertiser, analytics | 5 | 1.6% |
+| Third-party role — tracker, advertiser, analytics | 5 | 1.2% |
-Website fingerprinting is a traffic-analysis attack that shares no method, taxonomy or ground truth with topic categorisation; it is here because "classify a website" is the same English phrase. **Everything below uses the 247, which is 5.7% of the corpus.** The other 73 papers are counted, named, and excluded — see [[#Methodology and limitations of these figures]].
+Website fingerprinting is a traffic-analysis attack that shares no method, taxonomy or ground truth with topic categorisation; it is here because "classify a website" is the same English phrase. **Everything below uses the 330, which is 5.6% of the corpus.** The other 94 papers are counted, named, and excluded — see [[#Methodology and limitations of these figures]].
 ==== This is a Web Conference and IMC topic ====
-^ Venue ^ Papers ^ Share of the 247 ^ Share of that venue's papers ^
+^ Venue ^ Papers ^ Share of the 330 ^ Share of that venue's papers ^
-| TheWebConf | 59 | 23.9% | 8.3% |
+| TheWebConf | 71 | 21.5% | 8.4% |
-| IMC | 45 | 18.2% | 8.1% |
+| USENIX Security | 58 | 17.6% | 4.1% |
-| USENIX Security | 45 | 18.2% | 4.0% |
+| IMC | 50 | 15.2% | 7.8% |
-| CCS | 44 | 17.8% | 4.9% |
+| CCS | 48 | 14.5% | 4.8% |
-| PETS | 29 | 11.7% | 8.2% |
+| PETS | 43 | 13.0% | 8.4% |
-| NDSS | 22 | 8.9% | 5.3% |
+| NDSS | 35 | 10.6% | 5.0% |
-| IEEE S&P | 3 | 1.2% | 1.1% |
+| IEEE S&P | 25 | 7.6% | 3.3% |
-TheWebConf, IMC and PETS each devote about 8% of their programme to it; IEEE S&P essentially never does. The overall rate has been flat-to-falling: 5.1% of papers in 2010–2013, peaking at 7.5% in 2014–2017, back to 4.6% in 2022–2024.
+TheWebConf and PETS each devote about 8% of their programme to it, IMC nearly as much. The overall rate has been flat-to-falling: 5.5% of papers in 2010–2013, peaking at 7.3% in 2014–2017, 4.6% in 2022–2024 and 5.5% in the provisional 2025–2026 window.
+<WRAP info>
+**An earlier version of this table said IEEE S&P "essentially never" does this, at 3 papers and 1.2%.** That was an artefact of the corpus: IEEE S&P was then only 43% retrieved. It is now complete (780 of 780 selected papers), and IEEE S&P has 25 such papers, 3.3% of its programme. The old caveat named exactly the right suspect, and the fix confirmed it — which is the useful lesson: a venue that looks absent from a corpus should be checked for a retrieval gap before it is described as uninterested.
+</WRAP>
 ==== How they classify ====
-^ Method ^ Papers ^ Share of 247 ^
+^ Method ^ Papers ^ Share of 330 ^
-| Third-party service | 105 | 42.5% |
+| Third-party service | 131 | 39.7% |
-| Manual labelling | 53 | 21.5% |
+| Manual labelling | 68 | 20.6% |
-| Heuristic rules | 39 | 15.8% |
+| Heuristic rules | 59 | 17.9% |
-| Supervised ML | 35 | 14.2% |
+| Supervised ML | 46 | 13.9% |
-| Curated database | 31 | 12.6% |
+| Curated database | 46 | 13.9% |
-| Unsupervised ML | 11 | 4.5% |
+| Unsupervised ML | 14 | 4.2% |
-| Regex or signature | 9 | 3.6% |
+| Regex or signature | 9 | 2.7% |
+| **LLM** | **2** | **0.6%** |
+Multi-valued, so shares exceed 100%. Reliance on third-party services grew steadily for four windows — 25.0% of categorising papers in 2010–2013, 33.9%, 43.5%, **49.4% in 2022–2024** — while use of curated databases (DMOZ and friends) fell from 21.4% to 9.0% across the middle of that run. The field moved from open, inspectable label sources toward closed, unauditable ones, over exactly the period in which the only audit of those sources found them wanting.
-Multi-valued, so shares exceed 100%. Reliance on third-party services has grown steadily — 29.2% of categorising papers in 2010–2013, 35.8%, 44.4%, and **48.8% in 2022–2024** — while use of curated databases (DMOZ and friends) halved over the same period, from 16.7% to 8.8%. The field has moved from open, inspectable label sources toward closed, unauditable ones, over exactly the period in which the only audit of those sources found them wanting.
+**In the provisional 2025–2026 window both of those trends reverse**: third-party services drop to 32.3% and curated databases return to 18.5%, with hand-built heuristics up to 23.1% and the first two LLM papers appearing at 3.1%. Two incomplete venue-years cannot establish a turn, and this is exactly the kind of row not to quote as a trend — but it is the first movement against the closed-source drift in fifteen years, and it is worth re-checking when 2026 is complete.
 ==== Which services, folded ====
@@ Line 902: / Line 954: @@
 ^ Service ^ Papers ^ Distinct spellings folded ^
-| McAfee (incl. SmartFilter, TrustedSource, SiteAdvisor, URL Ticketing System) | 19 | **15** |
+| McAfee (incl. SmartFilter, TrustedSource, SiteAdvisor, URL Ticketing System) | 23 | **18** |
+| Symantec / Blue Coat (incl. WebPulse, RuleSpace, SiteReview) | 14 | **10** |
 | Alexa | 12 | 8 |
-| Symantec / Blue Coat (incl. WebPulse, RuleSpace, SiteReview) | 12 | **10** |
 | FortiGuard | 12 | 7 |
+| Media bias raters (AllSides, Media Bias/Fact Check, Science Feedback) | 10 | 9 |
+| Cloudflare domain categories | 10 | 6 |
+| Webshrinker | 9 | 2 |
 | DMOZ / Open Directory Project | 8 | 6 |
-| Webshrinker | 8 | 2 |
+| Google Cloud Natural Language | 8 | 6 |
-| Google Cloud Natural Language | 7 | 6 |
+| SimilarWeb | 7 | 2 |
+| VirusTotal | 7 | 3 |
 | Google Ads keyword and topic tools | 6 | 5 |
-| Cloudflare domain categories | 6 | 3 |
-| Media bias raters (AllSides, Media Bias/Fact Check) | 5 | 5 |
 | Trend Micro | 4 | 3 |
-| SimilarWeb | 4 | 2 |
 | Cisco (OpenDNS, IronPort) | 3 | 3 |
 | Cyren | 3 | 3 |
-| VirusTotal | 3 | 3 |
+| Citizen Lab test list | 3 | 3 |
+| Forcepoint / Websense | 3 | 2 |
+| Google Topics API | 3 | 3 |
-**122 of the 247 papers (49.4%) name at least one third-party service or dataset.** McAfee alone appears under fifteen different names — from bare "McAfee" to "McAfee's URL Ticketing System" to "McAfee SmartFilter Internet Database" — and Symantec under ten. Any count of these services built on exact string matching will be wrong by a factor of several, which is the same failure that has bitten geolocation and crawler-tool tallies on other pages of this site.
+**155 of the 330 papers (47.0%) name at least one third-party service or dataset.** McAfee alone appears under eighteen different names — from bare "McAfee" to "McAfee's URL Ticketing System" to "McAfee SmartFilter Internet Database" — and Symantec under ten. Any count of these services built on exact string matching will be wrong by a factor of several, which is the same failure that has bitten geolocation and crawler-tool tallies on other pages of this site.
-Everything that is not a named service, ranked: unnamed custom methods (38 papers), manual labelling by the authors or hired coders (32), custom keyword and regex rules (25), random forests (10), SVMs (9), clustering (8), then a long tail. Those first and third rows are one family: **61 papers (24.7%) classify websites with rules they wrote themselves, and 38 of them do not name the rules at all.**
+Everything that is not a named service, ranked: unnamed custom methods (58 papers), manual labelling by the authors or hired coders (41), custom keyword and regex rules (32), random forests (13), clustering (10), SVMs (9), then a long tail. Those first and third rows are one family: **87 papers (26.4%) classify websites with rules they wrote themselves, and 58 of them do not name the rules at all.**
 ==== Almost nobody cross-checks ====
 <WRAP important>
-Vallina et al.'s central recommendation is not to rely on a single service. **Nine of the 122 papers that name a service name two** — 7.4%. Five of those nine are from 2013–2015. The recommendation has not been taken up, and it remains one of the cheapest ways to strengthen a categorisation-based result.
+Vallina et al.'s central recommendation is not to rely on a single service. **Nine of the 155 papers that name a service name two** — 5.8%, down from 7.4% on the smaller corpus, because the count of papers naming two has not moved at all while the field has grown. Five of those nine are from 2013–2015. The recommendation has not been taken up, and it remains one of the cheapest ways to strengthen a categorisation-based result.
 </WRAP>
 ==== Most do not validate, and half do not say where their labels came from ====
-^ Question ^ Papers ^ Share of 247 ^
+^ Question ^ Papers ^ Share of 330 ^
-| Names the taxonomy it classified into | 184 | 74.5% |
+| Names the taxonomy it classified into | 248 | 75.2% |
-| Names a ground-truth source | 124 | 50.2% |
+| Names a ground-truth source | 169 | 51.2% |
-| Reports **any** validation of the labels | 95 | **38.5%** |
+| Reports **any** validation of the labels | 123 | **37.3%** |
-| Reports no validation at all | 69 | 27.9% |
+| Explicitly reports no validation | 98 | 29.7% |
+| Sentinel: the extraction marked validation ''not-applicable'' | 109 | 33.0% |
-Of those that do validate: 70 papers by manual validation, 18 by cross-validation, 9 by a held-out test set, 9 by comparison to another method. Note the shape — manual spot-checking is the dominant and correct instrument here, because a held-out test set only tells you about the labels you already had.
+The last row is a sentinel and is shown because the page's own rule says sentinels are never folded into an answer. It means the extraction could not find a validation step and did not think one was called for — typically because the paper is reporting a distribution of vendor labels with no claim that depends on them being right. Read it as "no validation reported", not as "validation was unnecessary", unless you check the paper.
-The 50.2% figure is worth dwelling on: for **website** classification, the share naming a ground-truth source is well below the 71.7% the same extraction measures across all 3,241 papers that classify anything. Website categorisation is reported worse than classification in general, not better.
+Of those that do validate: 92 papers by manual validation, 21 by cross-validation, 13 by a held-out test set, 10 by comparison to another method. Note the shape — manual spot-checking is the dominant and correct instrument here, because a held-out test set only tells you about the labels you already had.
+The 51.2% figure is worth dwelling on: for **website** classification, the share naming a ground-truth source is well below the 72.9% the same extraction measures across all 4,439 papers that classify anything. Website categorisation is reported worse than classification in general, not better.
 ==== Taxonomies are bespoke, and usually not even sized ====
-Of the 184 papers that name a taxonomy at all:
+Of the 248 papers that name a taxonomy at all:
-^ Taxonomy ^ Papers ^ Share of 184 ^
+^ Taxonomy ^ Papers ^ Share of 248 ^
-| A bespoke scheme, class count not given | 133 | 72.3% |
+| A bespoke scheme, class count not given | 187 | 75.4% |
-| A vendor's own category list | 37 | 20.1% |
+| A vendor's own category list | 43 | 17.3% |
-| DMOZ / ODP / Curlie | 6 | 3.3% |
+| IAB | 9 | 3.6% |
-| IAB | 6 | 3.3% |
+| DMOZ / ODP / Curlie | 7 | 2.8% |
-| A bespoke scheme with a stated class count | 6 | 3.3% |
+| A bespoke scheme with a stated class count | 7 | 2.8% |
-| Google AdWords / Topics | 3 | 1.6% |
+| Google AdWords / Topics | 4 | 1.6% |
-| A standard industry code (NAICS, SIC, NACE) | 1 | 0.5% |
+| A standard industry code (NAICS, SIC, NACE) | 1 | 0.4% |
-Nearly three quarters invent a scheme and do not say how many classes it has. That single omission makes a result impossible to compare against any other paper, and it is free to fix.
+Three quarters invent a scheme and do not say how many classes it has. That single omission makes a result impossible to compare against any other paper, and it is free to fix.
 ==== Where LLMs actually appear ====
 ^ Year ^ Papers using an LLM to classify anything ^ Corpus that year ^ Share ^
-| 2023 | 2 | 630 | 0.3% |
+| 2023 | 2 | 719 | 0.3% |
-| 2024 | 27 | 628 | 4.3% |
+| 2024 | 27 | 690 | 3.9% |
+| 2025 //(provisional)// | 77 | 770 | 10.0% |
+| 2026 //(provisional)// | 71 | 415 | 17.1% |
+**177 papers, and the curve is no longer starting — it is the fastest-moving thing in this corpus.** From 2 papers in 2023 to one in six in 2026, and both of the last two venue-years are incomplete, so those shares are floors. They point the model at "other" targets (116 papers), user-generated text (32), vulnerabilities (14), privacy policies (12), mobile apps (6), website categories (5), and single-digit counts of domains, network traffic, consent notices, requests, IP addresses, cookies, email and dark patterns.
+They establish a norm worth copying: **148 of 177 (83.6%) validate the model's labels against something**, usually manual annotation of a sample — the same share as when this was 29 papers, which is the encouraging part.
+**Five papers now point an LLM at a website category, and two of those are genuine topic classification** rather than an abuse verdict:
-Twenty-nine papers across the corpus's fifteen years, twenty-seven of them in the final one — the curve is visibly starting, and the corpus ends before it goes anywhere. They point the model at "other" targets (19 papers), user-generated text (6), vulnerabilities (4), privacy policies (1) and domains (1). **Not one of the 29 uses an LLM to assign a website a topic category.**
+  * {[tsai2025_harmful]} (TheWebConf 2025) classifies Tranco sites as shopping or non-shopping with **GPT-4o prompted with the URL and a screenshot**, validates against 500 manually annotated sites, and reports **92% accuracy, "comparable to commercial website classification services"**. That sentence is the first thing in this corpus that makes an LLM a real alternative to the vendors catalogued above, and it is worth reading before you buy an API key.
+  * DarkGram (USENIX Security 2025) has GPT-4 label cybercriminal Telegram channels into 15 predefined categories, which the model itself extended to 19, validated by coders on 1,000 sampled posts. Categories of a channel, not of a website — count it as adjacent.
-They do, however, establish a norm worth copying: **24 of 29 (82.8%) validate the model's labels against something**, usually manual annotation of a sample. The single ''website-category'' LLM tuple in the whole corpus is a USENIX Security 2024 phishing detector using GPT-3.5-turbo-16k with a chain-of-thought prompt for a binary credential-page question, validated against three cybersecurity experts' annotations — a security verdict, not a topic.
+The other three are abuse verdicts: the USENIX Security 2024 phishing detector using GPT-3.5-turbo-16k for a binary credential-page question, a TheWebConf 2025 crypto-scam study using a Llama3:70b + GPT-4 hybrid, and an NDSS 2026 credential-theft phishing detector using GPT-4o with retrieval-augmented prompts. All three validate against manual annotation.
-See [[#LLM Classification]] for what has happened since the corpus ends.
+See [[#LLM Classification]] for what has happened outside the corpus.
 ==== Two examples worth reading first ====
@@ Line 974: / Line 1041: @@
 ==== Methodology and limitations of these figures ====
-  * **Denominators.** Every figure names its own. The corpus is 4,322 papers; 3,241 classify something; 320 carry a ''website-category'' target; **247 assign a topic, industry or sector label** and are this section's population. Never "of 4,322 papers".
+  * **Denominators.** Every figure names its own. The corpus is 5,859 papers; 4,439 classify something; 424 carry a ''website-category'' target; **330 assign a topic, industry or sector label** and are this section's population. Never "of 5,859 papers".
   * **Papers, not mentions.** A paper naming McAfee three times counts once.
   * **Sentinels are not answers.** ''not-stated'' and ''none-mentioned'' are excluded from every numerator and reported as their own row where they matter — which they do, since "does not validate" is the headline finding.
-  * **Free-text names are folded** into canonical families by an ordered regex list before counting, because exact strings agree run-to-run only about 20% of the time. The fold left **21 distinct strings unclassified**, each appearing once: among them ''Taobao main-business categories'', ''SNET pillar portals'', ''Rakuten Popshops'', ''multiple domain classification services'' and ''public classification service from the security vendor''. The last two are papers that used a service and declined to say which — a small, exact instance of the reporting gap this section measures.
+  * **Free-text names are folded** into canonical families by an ordered regex list before counting, because exact strings agree run-to-run only about 20% of the time. The fold left **33 distinct strings unclassified**, each appearing once: among them ''Taobao main-business categories'', ''SNET pillar portals'', ''Rakuten Popshops'', ''NAICSlite'', ''ORES topic routing'', ''multiple domain classification services'' and ''public classification service from the security vendor''. The last two are papers that used a service and declined to say which — a small, exact instance of the reporting gap this section measures. The residue was 40 strings before the fold was extended for this corpus; [[provenance:design:website_classification]] lists what was added and what was deliberately left out.
-  * **The task fold is imperfect and its exclusions are visible.** Eight side-channel and website-fingerprinting papers named their classifier only as "a 1D CNN" or "k-NN", leaving no signal in any extracted field; they were caught by matching the paper's own title and are excluded by name rather than by rule. The 73 excluded papers are counted in the table above rather than dropped.
+  * **The task fold is imperfect and its exclusions are visible.** Eight side-channel and website-fingerprinting papers named their classifier only as "a 1D CNN" or "k-NN", leaving no signal in any extracted field; they were caught by matching the paper's own slug and are excluded by an explicit named list rather than by a general rule. The 94 excluded papers are counted in the table above rather than dropped.
   * **Enum fields versus free text.** Method and validation are enums, stable enough to publish as rough shares (''classification.method'' agrees 58% run-to-run, so read those as a ranking). Service names and taxonomies are free text and are reported as rankings and folded families only.
-  * **Quotes were checked.** Every figure above traces to tuples carrying a verbatim evidence quote; a sample of these was re-located in the source PDFs. Of six quotes checked by hand, two initially "failed" a literal grep and turned out to be intact but split across a two-column break — normalise whitespace before concluding that a quote is not in the paper.
+  * **Quotes were checked.** Every figure above traces to tuples carrying a verbatim evidence quote; a sample of these was re-located in the source PDFs. Of six quotes checked by hand, two initially "failed" a literal grep and turned out to be intact but split across a two-column break — normalise whitespace before concluding that a quote is not in the paper. The five new LLM website-category tuples were re-checked individually on 2026-08-12; see [[provenance:design:website_classification]].
-  * **Coverage.** IEEE S&P is only 43% retrieved, which is the most likely explanation for its 1.2% share. EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS are absent entirely, and so is anything published after 2024. Notably, **{[vallina2020_misshapes]} itself is in the venue index but has no extracted full text** — the reference work for this page is not in the population the page measures.
+  * **Coverage.** Seven venues only, with 2025 and 2026 incomplete by construction rather than by relevance, so any row that reaches them is a floor; the scope and the selection funnel are on [[literature:corpus]]. Notably, **{[vallina2020_misshapes]} itself is in the venue index but has no extracted full text** — the reference work for this page is not in the population the page measures.
+  * **Every query behind this section, the report script and its unedited output** are on [[provenance:design:website_classification]]; corpus-level caveats are on [[literature:corpus]].
 ===== Open Questions =====
@@ Line 987: / Line 1055: @@
 Genuinely unresolved, and each is a paper:
-  - **Redo Vallina et al.** Thirteen services, millions of domains, 2026 ownership. Every vendor has changed hands; the only modern data point suggests the free tiers have collapsed and quality has not improved. This is the obvious one.
+  - **Redo Vallina et al.** Thirteen services, millions of domains, 2026 ownership. Two of the thirteen are dead, two more have changed hands, the free tiers have collapsed, and the only modern data point suggests quality has not improved. This is the obvious one.
   - **Benchmark open models against commercial services on a shared, public ground truth.** Homepage2Vec versus gpt-oss has been done on 3,785 sites; neither has been compared against FortiGuard or McAfee at all.
   - **Measure LLM label stability** across seeds, temperature and model versions for website classification. Everyone currently sets temperature to 0 and hopes.
@@ Line 998: / Line 1066: @@
   * [[Design:Website Selection]] — which domains to study, and popularity rankings.
   * [[Design:IP Classification]] — the same problem for addresses instead of names, with the same "which question are you actually asking" structure.
+  * [[Design:Crawling Location]] — where you crawl from, which is a different question from which country a site is aimed at ([[#Target country is not topic]]).
   * [[Privacy:Requests]] — tracker and advertiser classification, where filter lists beat categorisers.
   * [[Privacy:Cookies]], [[Privacy:Fingerprinting]], [[Privacy:JavaScript]] — classification of the other things a crawl collects.