| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| design:website_classification [2026/08/07 14:48] – Corrections from Karel's review notes: separate Cloudflare Radar (rankings/country) from Domain Intelligence (topic categories) and flag that the 50.4%/39.7% figures are a country benchmark; add version/peer-review caveat for arXiv:2510.08101 (v1 vs v3) a karel.kubicek.claude | design:website_classification [2026/08/27 14:39] (current) – CrUX licence: Tranco homepage CC BY-SA 4.0, methodology silent, Google CC BY 4.0. Authored by Claude karel.kubicek.claude |
|---|
| **A categorisation service is a measurement instrument, and you have to characterise it before you use it.** Not "we categorised domains using X" — that is not a method, it is a citation. What a reviewer needs is: which service and which version, how many of your domains it actually labelled, what its taxonomy is, and what precision it achieved //on your sample//, established by hand. | **A categorisation service is a measurement instrument, and you have to characterise it before you use it.** Not "we categorised domains using X" — that is not a method, it is a citation. What a reviewer needs is: which service and which version, how many of your domains it actually labelled, what its taxonomy is, and what precision it achieved //on your sample//, established by hand. |
| |
| Of the 247 papers in our corpus that assign topic categories to websites, **38.5% report any validation of the resulting labels at all.** That is the single largest methodological gap this page exists to close, and [[#Validating Your Labels]] gives you a script that closes it in an afternoon. | Of the 330 papers in our corpus that assign topic categories to websites, **37.3% report any validation of the resulting labels at all.** That is the single largest methodological gap this page exists to close, and [[#Validating Your Labels]] gives you a script that closes it in an afternoon. |
| </WRAP> | </WRAP> |
| |
| === VirusTotal === | === VirusTotal === |
| |
| * **API**: ''GET /api/v3/domains/{domain}'' returns a ''categories'' attribute, //"a dictionary mapping that relates categorisation services with the category it assigns the domain to"//. Documented at [[https://docs.virustotal.com/reference/domains-object|docs.virustotal.com]]. | * **API**: ''GET /api/v3/domains/{domain}'' returns a ''categories'' attribute, //"a dictionary mapping that relates categorisation services with the category it assigns the domain to"//. Documented at [[https://docs.virustotal.com/reference/domains-object|docs.virustotal.com]]. **That is a topic label, not a maliciousness verdict.** The oracle use of the same API — engine disagreement, thresholds, URL versus file lookup, snapshot dating — is [[security:virustotal]]. |
| * **Which vendors**: Dr.Web, Forcepoint ThreatSeeker, BitDefender, Sophos, Trend Micro, Websense and (legacy) Alexa, among others — so several of the vendors above reach you through VirusTotal without you querying them. | * **Which vendors**: Dr.Web, Forcepoint ThreatSeeker, BitDefender, Sophos, Trend Micro, Websense and (legacy) Alexa, among others — so several of the vendors above reach you through VirusTotal without you querying them. |
| * **Rate limit**: the free tier is **500 requests per day and 4 requests per minute**.((https://docs.virustotal.com/docs/public-vs-premium-api — fetched 2026-08-07; that page names the "Community" tier but does not mention the October 2025 retiering, so treat the tier names as unconfirmed and the numbers as current.)) The same page adds that the Public API //"must not be used in commercial products or services"//. At 500/day, a 100k-domain study takes 200 days. Vallina et al. had an academic key at 20k/day; we could find no self-serve academic application form in 2026, so budget for a direct conversation with VirusTotal rather than assuming access. | * **Rate limit**: the free tier is **500 requests per day and 4 requests per minute**.((https://docs.virustotal.com/docs/public-vs-premium-api — fetched 2026-08-07; that page names the "Community" tier but does not mention the October 2025 retiering, so treat the tier names as unconfirmed and the numbers as current.)) The same page adds that the Public API //"must not be used in commercial products or services"//. At 500/day, a 100k-domain study takes 200 days. Vallina et al. had an academic key at 20k/day. As of 2026-08-27 there is still no standalone quota-application form; the public contact form has a subject "I have an academic research request" ([[https://www.virustotal.com/gui/contact-us/legal|virustotal.com/gui/contact-us/legal]]). Budget for that conversation rather than assuming 20k/day. Maliciousness-oracle use of the same API is [[security:virustotal]]. |
| * **Advantages**: aggregates many providers in a single call; widely used and easy to cite. | * **Advantages**: aggregates many providers in a single call; widely used and easy to cite. |
| * **Disadvantages**: **the integration is lossy**. In the 2020 audit, several services returned labels when queried directly but not through VirusTotal, and Trend Micro's VirusTotal labels tracked its 2011 taxonomy rather than its 2019 one. You do not control which product version you are reading, and "we used VirusTotal categories" does not identify the underlying source. | * **Disadvantages**: **the integration is lossy**. In the 2020 audit, several services returned labels when queried directly but not through VirusTotal, and Trend Micro's VirusTotal labels tracked its 2011 taxonomy rather than its 2019 one. You do not control which product version you are reading, and "we used VirusTotal categories" does not identify the underlying source. |
| === Alexa === | === Alexa === |
| |
| Amazon retired **Alexa.com on 1 May 2022**, per its own end-of-service notice ("we will be retiring Alexa.com on May 1, 2022").((Captured on the alexa.com login page, https://web.archive.org/web/20220315000000/https://www.alexa.com/ — retrieved 2026-08-07. [[Design:Website Selection]] currently gives 1 August 2023; we could not find a primary source for that date, and ''alexa.com'' now redirects to the unrelated Amazon Alexa voice assistant.)) Its ranking service and its category service died together. | Amazon retired **Alexa.com on 1 May 2022**, per its own end-of-service notice ("we will be retiring Alexa.com on May 1, 2022").((Captured on the alexa.com login page, https://web.archive.org/web/20220315000000/https://www.alexa.com/ — retrieved 2026-08-07. An earlier revision of [[Design:Website Selection]] gave 1 August 2023; that date is when Tranco dropped Alexa from the default list, not when Amazon switched the service off. The selection page now carries both dates. ''alexa.com'' redirects to the unrelated Amazon Alexa voice assistant.)) Its ranking service and its category service died together. |
| |
| * **Why it still matters**: it appears in 12 of the 247 corpus papers that categorise websites, under eight different spellings, and papers published as late as 2024 still use it because their data collection predates the shutdown. If you are reading such a paper, the labels are from a dead service with a documented 0.53% coverage rate. | * **Why it still matters**: it appears in 12 of the 330 corpus papers that categorise websites, under eight different spellings, and papers published as late as 2024 still use it because their data collection predates the shutdown. If you are reading such a paper, the labels are from a dead service with a documented 0.53% coverage rate. |
| * **What it was**: highly granular (7,557 distinct labels observed through VirusTotal, against 279,716 categories in Alexa's own directory, which was derived from DMOZ), and capped at 500 sites per category — hence the tiny coverage. Leaf labels came back in the language of the branch, so the same concept appeared as //Arts//, //Artes// and //Kultur//. | * **What it was**: highly granular (7,557 distinct labels observed through VirusTotal, against 279,716 categories in Alexa's own directory, which was derived from DMOZ), and capped at 500 sites per category — hence the tiny coverage. Leaf labels came back in the language of the branch, so the same concept appeared as //Arts//, //Artes// and //Kultur//. |
| |
| |
| <WRAP important> | <WRAP important> |
| **This is where our corpus runs out, and you should know exactly where.** Of the 4,322 papers in the 2010–2024 corpus, 29 use an LLM to classify anything at all — 27 of them in 2024 — and they point it at privacy policies, app reviews, vulnerabilities, jailbreak prompts and content moderation. **Zero of them use an LLM to assign a website a topic category.** Any ranking of methods built from this literature will therefore tell you that third-party services and hand-built keyword rules are the state of the art. As of 2026 that is no longer true, and the corpus cannot see why. | **The corpus now reaches 2026, and the picture changed.** Of the 5,859 papers in the 2010–2026 corpus, **177 use an LLM to classify something** — 2 in 2023, 27 in 2024, 77 in 2025 and 71 in the partial 2026 — and **two of them use one to assign a website a topic category**. The nearer of the two, {[tsai2025_harmful]}, reports **92% accuracy for GPT-4o on shopping-vs-non-shopping over 500 manually annotated Tranco sites, and says explicitly that this is "comparable to commercial website classification services"**. So a ranking of methods built from this literature no longer tells you that third-party services and keyword rules are the only options — but it still gives you a two-paper base for the topic task specifically, in the corpus's two thinnest and provisional years. Treat the peer-reviewed evidence as thin rather than absent, and read the rest of this section for what is outside the corpus. |
| </WRAP> | </WRAP> |
| |
| |
| * **The TLD.** Precise where it exists and absent where it matters — a ccTLD is strong evidence, but generic TLDs carry no country at all, and that is most of the head of any list. | * **The TLD.** Precise where it exists and absent where it matters — a ccTLD is strong evidence, but generic TLDs carry no country at all, and that is most of the head of any list. |
| * **CrUX country lists, via Tranco.** Tranco's list-generation API takes ''filterCRUX'', ''filterCRUXType'' (''global'' / ''country'' / ''region'' / ''subregion''), ''filterCRUXValue'' (e.g. a list of country codes) and ''filterCRUXMonth'', so you can generate a country-restricted list reproducibly, with a permalink, from a free account.((https://tranco-list.eu/api_documentation — fetched 2026-08-07. Basic Auth with your email as username and API token as password; the ''/configure'' page returns 401 without a login. CrUX itself is CC BY-SA 4.0.)) This is the cleanest free source and it is under-used. **But it ranks by page loads from Chrome users in a country, which is popularity, not audience** — and the overlap is severe: the union of five country top-10k lists (China, Germany, Italy, Korea, Turkey) is only 18,718 domains rather than 50,000, with 4,017 domains common to all five, and a preliminary labelling built from these lists is incompatible with the site's own TLD in at least 25% of cases for every label {[bozzolan2026_llmweb]}. | * **CrUX country lists, via Tranco.** Tranco's list-generation API takes ''filterCRUX'', ''filterCRUXType'' (''global'' / ''country'' / ''region'' / ''subregion''), ''filterCRUXValue'' (e.g. a list of country codes) and ''filterCRUXMonth'', so you can generate a country-restricted list reproducibly, with a permalink, from a free account.((https://tranco-list.eu/api_documentation — fetched 2026-08-07. Basic Auth with your email as username and API token as password; the ''/configure'' page returns 401 without a login. CrUX datasets are CC BY 4.0 per Google's methodology, fetched 2026-08-27; Tranco's homepage still labels CrUX CC BY-SA 4.0, while the methodology page states no CrUX licence.)) This is the cleanest free source and it is under-used. **But it ranks by page loads from Chrome users in a country, which is popularity, not audience** — and the overlap is severe: the union of five country top-10k lists (China, Germany, Italy, Korea, Turkey) is only 18,718 domains rather than 50,000, with 4,017 domains common to all five, and a preliminary labelling built from these lists is incompatible with the site's own TLD in at least 25% of cases for every label {[bozzolan2026_llmweb]}. |
| * **Site language.** A good proxy, and cheap, but it splits badly on English, Spanish, Arabic and Portuguese, which is a large share of the web. | * **Site language.** A good proxy, and cheap, but it splits badly on English, Spanish, Arabic and Portuguese, which is a large share of the web. |
| * **Host IP or CDN location.** Measures where bytes are served from. Behind a CDN — most of the head of any list — it tells you about the CDN. See [[Design:IP Classification]] and [[Design:Crawling Location]]. | * **Host IP or CDN location.** Measures where bytes are served from. Behind a CDN — most of the head of any list — it tells you about the CDN. See [[Design:IP Classification]] and [[Design:Crawling Location]]. |
| ===== Taxonomies ===== | ===== Taxonomies ===== |
| |
| Pick the taxonomy before the service, because it is much harder to change later, and because 72% of the corpus papers that name a taxonomy at all name a bespoke one and do not even state how many classes it has (see [[#Use in Publications]]) — which makes their results incomparable with everyone else's. | Pick the taxonomy before the service, because it is much harder to change later, and because 75% of the corpus papers that name a taxonomy at all name a bespoke one and do not even state how many classes it has (see [[#Use in Publications]]) — which makes their results incomparable with everyone else's. |
| |
| * **[[https://github.com/InteractiveAdvertisingBureau/Taxonomies|IAB Tech Lab Content Taxonomy]]** — the advertising industry's reference schema, and the right choice if your question is about ad targeting or if you want to be comparable to Webshrinker's output. Machine-readable TSV in the repo under ''Content Taxonomies/'', CC BY 3.0, up to four tiers with a relational ID system. Versions 1.0 through **3.1** sit side by side; 2.x → 3.0 is a **breaking change** and IAB says the two must not be used together. Category rows counted from the TSVs on 2026-08-07, excluding the two header lines: **2.2 has 1,196 rows, 3.0 has 703, 3.1 has 704** — so 3.1 is marginally //larger// than 3.0. Tier-1 rows: 37 in 3.1, but only 31 of those carry a numeric Unique ID; the other six (Communication, Entertainment, Events, Genres, Holidays, Sensitive Topics) are alphanumeric extension rows. **State which convention you counted with** — three independent counts of this one file in preparing this page produced three different numbers. Vendor marketing claiming that 3.x "expands to 1,500+ categories" is wrong in both directions — 3.x is roughly //40% smaller// than 2.2, because it consolidated and restructured rather than adding. Count the rows in the file yourself; it takes ten seconds and the summaries are not reliable. | * **[[https://github.com/InteractiveAdvertisingBureau/Taxonomies|IAB Tech Lab Content Taxonomy]]** — the advertising industry's reference schema, and the right choice if your question is about ad targeting or if you want to be comparable to Webshrinker's output. Machine-readable TSV in the repo under ''Content Taxonomies/'', CC BY 3.0, up to four tiers with a relational ID system. Versions 1.0 through **3.1** sit side by side; 2.x → 3.0 is a **breaking change** and IAB says the two must not be used together. Category rows counted from the TSVs on 2026-08-07, excluding the two header lines: **2.2 has 1,196 rows, 3.0 has 703, 3.1 has 704** — so 3.1 is marginally //larger// than 3.0. Tier-1 rows: 37 in 3.1, but only 31 of those carry a numeric Unique ID; the other six (Communication, Entertainment, Events, Genres, Holidays, Sensitive Topics) are alphanumeric extension rows. **State which convention you counted with** — three independent counts of this one file in preparing this page produced three different numbers. Vendor marketing claiming that 3.x "expands to 1,500+ categories" is wrong in both directions — 3.x is roughly //40% smaller// than 2.2, because it consolidated and restructured rather than adding. Count the rows in the file yourself; it takes ten seconds and the summaries are not reliable. |
| - Based on LinkedIn profiles that are self-reported — prone to adversarial data. | - Based on LinkedIn profiles that are self-reported — prone to adversarial data. |
| - Only a subset of PeopleDataLabs' full dataset. The "22M of 70M rows" framing is longstanding on this page; the 22M is confirmed, the 70M total could not be re-confirmed in 2026 and PDL's marketing page now cites 23.8M+ without saying which corpus that is. | - Only a subset of PeopleDataLabs' full dataset. The "22M of 70M rows" framing is longstanding on this page; the 22M is confirmed, the 70M total could not be re-confirmed in 2026 and PDL's marketing page now cites 23.8M+ without saying which corpus that is. |
| <wrap todo>TODO: cite ''Machine Learning Compliance Analysis for Email Regulation'' when it is public.</wrap> | <WRAP todo>TODO: cite ''Machine Learning Compliance Analysis for Email Regulation'' when it is public.</WRAP> |
| |
| ==== Crunchbase ==== | ==== Crunchbase ==== |
| - Focuses mostly on variables useful for investments and market competitiveness. | - Focuses mostly on variables useful for investments and market competitiveness. |
| - Academic access is no longer publicly documented; budget for a sales conversation. | - Academic access is no longer publicly documented; budget for a sales conversation. |
| <wrap todo>TODO: cite ''Machine Learning Compliance Analysis for Email Regulation'' when it is public.</wrap> | <WRAP todo>TODO: cite ''Machine Learning Compliance Analysis for Email Regulation'' when it is public.</WRAP> |
| |
| ==== Orbis ==== | ==== Orbis ==== |
| ===== What to Report ===== | ===== What to Report ===== |
| |
| A checklist, derived from what the 247 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply. | A checklist, derived from what the 330 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply. |
| |
| - **Which service or model, and which version.** "McAfee" identified a different company before and after February 2024. "Homepage2Vec" without a commit hash is not reproducible; "GPT-4" without a dated snapshot is not either. | - **Which service or model, and which version.** "McAfee" identified a different company before and after February 2024. "Homepage2Vec" without a commit hash is not reproducible; "GPT-4" without a dated snapshot is not either. |
| - **Validation**: sample size, how it was stratified, how many annotators, whether they were blind to the service's label, the agreement statistic, and per-category precision — not one global accuracy number. Publish the annotated sample as an [[:Artifacts|artefact]] //(page not yet written)//; it is small, and it is the only part of this a reader can check. | - **Validation**: sample size, how it was stratified, how many annotators, whether they were blind to the service's label, the agreement statistic, and per-category precision — not one global accuracy number. Publish the annotated sample as an [[:Artifacts|artefact]] //(page not yet written)//; it is small, and it is the only part of this a reader can check. |
| - **The categories you dropped**, and the rule you dropped them by. | - **The categories you dropped**, and the rule you dropped them by. |
| - **If you used more than one service, the disagreement between them.** Only 9 of 122 corpus papers that name a service name two, so this is nearly free novelty. | - **If you used more than one service, the disagreement between them.** Only 9 of 155 corpus papers that name a service name two, so this is nearly free novelty. |
| |
| <WRAP important> | <WRAP important> |
| ===== Use in Publications ===== | ===== Use in Publications ===== |
| |
| Two sources here. The historical survey is Vallina et al. {[vallina2020_misshapes]}, whose figures on service popularity are reproduced below. The rest is a fresh query over a structured extraction of **4,322 papers from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2024** — so every figure is a claim about those seven venues over that window, not about the field. | Two sources here. The historical survey is Vallina et al. {[vallina2020_misshapes]}, whose figures on service popularity are reproduced below. The rest is a fresh query over a structured extraction of **5,859 papers from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026** — so every figure is a claim about those seven venues over that window, not about the field. The 2025 and 2026 venue-years are provisional: CCS and IMC 2026 have not been held, and IEEE S&P and WWW 2026 abstracts are not yet in the selection source. |
| |
| <WRAP right box> | <WRAP right box> |
| ==== The population, and what "website category" hides ==== | ==== The population, and what "website category" hides ==== |
| |
| The extraction marks 320 papers as classifying something with the target ''website-category''. **That number is not the population this page is about**, and the difference is instructive: the same label covers several unrelated tasks. | The extraction marks 424 papers as classifying something with the target ''website-category''. **That number is not the population this page is about**, and the difference is instructive: the same label covers several unrelated tasks. |
| |
| ^ What is actually being classified ^ Papers ^ Share of the 320 ^ | ^ What is actually being classified ^ Papers ^ Share of the 424 ^ |
| | A topic, industry or sector label on a site or domain | 247 | 77.2% | | | A topic, industry or sector label on a site or domain | 330 | 77.8% | |
| | An abuse verdict — phishing, scam, malicious | 47 | 14.7% | | | An abuse verdict — phishing, scam, malicious | 63 | 14.9% | |
| | Website fingerprinting — which of N monitored sites produced this traffic trace | 22 | 6.9% | | | Website fingerprinting — which of N monitored sites produced this traffic trace | 36 | 8.5% | |
| | Hosting or CDN infrastructure | 7 | 2.2% | | | Hosting or CDN infrastructure | 8 | 1.9% | |
| | Page type — homepage, login page, sign-up | 6 | 1.9% | | | Page type — homepage, login page, sign-up | 7 | 1.7% | |
| | Which software runs the site | 6 | 1.9% | | | Which software runs the site | 6 | 1.4% | |
| | Third-party role — tracker, advertiser, analytics | 5 | 1.6% | | | Third-party role — tracker, advertiser, analytics | 5 | 1.2% | |
| |
| Website fingerprinting is a traffic-analysis attack that shares no method, taxonomy or ground truth with topic categorisation; it is here because "classify a website" is the same English phrase. **Everything below uses the 247, which is 5.7% of the corpus.** The other 73 papers are counted, named, and excluded — see [[#Methodology and limitations of these figures]]. | Website fingerprinting is a traffic-analysis attack that shares no method, taxonomy or ground truth with topic categorisation; it is here because "classify a website" is the same English phrase. **Everything below uses the 330, which is 5.6% of the corpus.** The other 94 papers are counted, named, and excluded — see [[#Methodology and limitations of these figures]]. |
| |
| ==== This is a Web Conference and IMC topic ==== | ==== This is a Web Conference and IMC topic ==== |
| |
| ^ Venue ^ Papers ^ Share of the 247 ^ Share of that venue's papers ^ | ^ Venue ^ Papers ^ Share of the 330 ^ Share of that venue's papers ^ |
| | TheWebConf | 59 | 23.9% | 8.3% | | | TheWebConf | 71 | 21.5% | 8.4% | |
| | IMC | 45 | 18.2% | 8.1% | | | USENIX Security | 58 | 17.6% | 4.1% | |
| | USENIX Security | 45 | 18.2% | 4.0% | | | IMC | 50 | 15.2% | 7.8% | |
| | CCS | 44 | 17.8% | 4.9% | | | CCS | 48 | 14.5% | 4.8% | |
| | PETS | 29 | 11.7% | 8.2% | | | PETS | 43 | 13.0% | 8.4% | |
| | NDSS | 22 | 8.9% | 5.3% | | | NDSS | 35 | 10.6% | 5.0% | |
| | IEEE S&P | 3 | 1.2% | 1.1% | | | IEEE S&P | 25 | 7.6% | 3.3% | |
| |
| TheWebConf, IMC and PETS each devote about 8% of their programme to it; IEEE S&P essentially never does. The overall rate has been flat-to-falling: 5.1% of papers in 2010–2013, peaking at 7.5% in 2014–2017, back to 4.6% in 2022–2024. | TheWebConf and PETS each devote about 8% of their programme to it, IMC nearly as much. The overall rate has been flat-to-falling: 5.5% of papers in 2010–2013, peaking at 7.3% in 2014–2017, 4.6% in 2022–2024 and 5.5% in the provisional 2025–2026 window. |
| | |
| | <WRAP info> |
| | **An earlier version of this table said IEEE S&P "essentially never" does this, at 3 papers and 1.2%.** That was an artefact of the corpus: IEEE S&P was then only 43% retrieved. It is now complete (780 of 780 selected papers), and IEEE S&P has 25 such papers, 3.3% of its programme. The old caveat named exactly the right suspect, and the fix confirmed it — which is the useful lesson: a venue that looks absent from a corpus should be checked for a retrieval gap before it is described as uninterested. |
| | </WRAP> |
| |
| ==== How they classify ==== | ==== How they classify ==== |
| |
| ^ Method ^ Papers ^ Share of 247 ^ | ^ Method ^ Papers ^ Share of 330 ^ |
| | Third-party service | 105 | 42.5% | | | Third-party service | 131 | 39.7% | |
| | Manual labelling | 53 | 21.5% | | | Manual labelling | 68 | 20.6% | |
| | Heuristic rules | 39 | 15.8% | | | Heuristic rules | 59 | 17.9% | |
| | Supervised ML | 35 | 14.2% | | | Supervised ML | 46 | 13.9% | |
| | Curated database | 31 | 12.6% | | | Curated database | 46 | 13.9% | |
| | Unsupervised ML | 11 | 4.5% | | | Unsupervised ML | 14 | 4.2% | |
| | Regex or signature | 9 | 3.6% | | | Regex or signature | 9 | 2.7% | |
| | | **LLM** | **2** | **0.6%** | |
| | |
| | Multi-valued, so shares exceed 100%. Reliance on third-party services grew steadily for four windows — 25.0% of categorising papers in 2010–2013, 33.9%, 43.5%, **49.4% in 2022–2024** — while use of curated databases (DMOZ and friends) fell from 21.4% to 9.0% across the middle of that run. The field moved from open, inspectable label sources toward closed, unauditable ones, over exactly the period in which the only audit of those sources found them wanting. |
| |
| Multi-valued, so shares exceed 100%. Reliance on third-party services has grown steadily — 29.2% of categorising papers in 2010–2013, 35.8%, 44.4%, and **48.8% in 2022–2024** — while use of curated databases (DMOZ and friends) halved over the same period, from 16.7% to 8.8%. The field has moved from open, inspectable label sources toward closed, unauditable ones, over exactly the period in which the only audit of those sources found them wanting. | **In the provisional 2025–2026 window both of those trends reverse**: third-party services drop to 32.3% and curated databases return to 18.5%, with hand-built heuristics up to 23.1% and the first two LLM papers appearing at 3.1%. Two incomplete venue-years cannot establish a turn, and this is exactly the kind of row not to quote as a trend — but it is the first movement against the closed-source drift in fifteen years, and it is worth re-checking when 2026 is complete. |
| |
| ==== Which services, folded ==== | ==== Which services, folded ==== |
| |
| ^ Service ^ Papers ^ Distinct spellings folded ^ | ^ Service ^ Papers ^ Distinct spellings folded ^ |
| | McAfee (incl. SmartFilter, TrustedSource, SiteAdvisor, URL Ticketing System) | 19 | **15** | | | McAfee (incl. SmartFilter, TrustedSource, SiteAdvisor, URL Ticketing System) | 23 | **18** | |
| | | Symantec / Blue Coat (incl. WebPulse, RuleSpace, SiteReview) | 14 | **10** | |
| | Alexa | 12 | 8 | | | Alexa | 12 | 8 | |
| | Symantec / Blue Coat (incl. WebPulse, RuleSpace, SiteReview) | 12 | **10** | | |
| | FortiGuard | 12 | 7 | | | FortiGuard | 12 | 7 | |
| | | Media bias raters (AllSides, Media Bias/Fact Check, Science Feedback) | 10 | 9 | |
| | | Cloudflare domain categories | 10 | 6 | |
| | | Webshrinker | 9 | 2 | |
| | DMOZ / Open Directory Project | 8 | 6 | | | DMOZ / Open Directory Project | 8 | 6 | |
| | Webshrinker | 8 | 2 | | | Google Cloud Natural Language | 8 | 6 | |
| | Google Cloud Natural Language | 7 | 6 | | | SimilarWeb | 7 | 2 | |
| | | VirusTotal | 7 | 3 | |
| | Google Ads keyword and topic tools | 6 | 5 | | | Google Ads keyword and topic tools | 6 | 5 | |
| | Cloudflare domain categories | 6 | 3 | | |
| | Media bias raters (AllSides, Media Bias/Fact Check) | 5 | 5 | | |
| | Trend Micro | 4 | 3 | | | Trend Micro | 4 | 3 | |
| | SimilarWeb | 4 | 2 | | |
| | Cisco (OpenDNS, IronPort) | 3 | 3 | | | Cisco (OpenDNS, IronPort) | 3 | 3 | |
| | Cyren | 3 | 3 | | | Cyren | 3 | 3 | |
| | VirusTotal | 3 | 3 | | | Citizen Lab test list | 3 | 3 | |
| | | Forcepoint / Websense | 3 | 2 | |
| | | Google Topics API | 3 | 3 | |
| |
| **122 of the 247 papers (49.4%) name at least one third-party service or dataset.** McAfee alone appears under fifteen different names — from bare "McAfee" to "McAfee's URL Ticketing System" to "McAfee SmartFilter Internet Database" — and Symantec under ten. Any count of these services built on exact string matching will be wrong by a factor of several, which is the same failure that has bitten geolocation and crawler-tool tallies on other pages of this site. | **155 of the 330 papers (47.0%) name at least one third-party service or dataset.** McAfee alone appears under eighteen different names — from bare "McAfee" to "McAfee's URL Ticketing System" to "McAfee SmartFilter Internet Database" — and Symantec under ten. Any count of these services built on exact string matching will be wrong by a factor of several, which is the same failure that has bitten geolocation and crawler-tool tallies on other pages of this site. |
| |
| Everything that is not a named service, ranked: unnamed custom methods (38 papers), manual labelling by the authors or hired coders (32), custom keyword and regex rules (25), random forests (10), SVMs (9), clustering (8), then a long tail. Those first and third rows are one family: **61 papers (24.7%) classify websites with rules they wrote themselves, and 38 of them do not name the rules at all.** | Everything that is not a named service, ranked: unnamed custom methods (58 papers), manual labelling by the authors or hired coders (41), custom keyword and regex rules (32), random forests (13), clustering (10), SVMs (9), then a long tail. Those first and third rows are one family: **87 papers (26.4%) classify websites with rules they wrote themselves, and 58 of them do not name the rules at all.** |
| |
| ==== Almost nobody cross-checks ==== | ==== Almost nobody cross-checks ==== |
| |
| <WRAP important> | <WRAP important> |
| Vallina et al.'s central recommendation is not to rely on a single service. **Nine of the 122 papers that name a service name two** — 7.4%. Five of those nine are from 2013–2015. The recommendation has not been taken up, and it remains one of the cheapest ways to strengthen a categorisation-based result. | Vallina et al.'s central recommendation is not to rely on a single service. **Nine of the 155 papers that name a service name two** — 5.8%, down from 7.4% on the smaller corpus, because the count of papers naming two has not moved at all while the field has grown. Five of those nine are from 2013–2015. The recommendation has not been taken up, and it remains one of the cheapest ways to strengthen a categorisation-based result. |
| </WRAP> | </WRAP> |
| |
| ==== Most do not validate, and half do not say where their labels came from ==== | ==== Most do not validate, and half do not say where their labels came from ==== |
| |
| ^ Question ^ Papers ^ Share of 247 ^ | ^ Question ^ Papers ^ Share of 330 ^ |
| | Names the taxonomy it classified into | 184 | 74.5% | | | Names the taxonomy it classified into | 248 | 75.2% | |
| | Names a ground-truth source | 124 | 50.2% | | | Names a ground-truth source | 169 | 51.2% | |
| | Reports **any** validation of the labels | 95 | **38.5%** | | | Reports **any** validation of the labels | 123 | **37.3%** | |
| | Explicitly reports no validation | 69 | 27.9% | | | Explicitly reports no validation | 98 | 29.7% | |
| | Sentinel: the extraction marked validation ''not-applicable'' | 83 | 33.6% | | | Sentinel: the extraction marked validation ''not-applicable'' | 109 | 33.0% | |
| |
| The last row is a sentinel and is shown because the page's own rule says sentinels are never folded into an answer. It means the extraction could not find a validation step and did not think one was called for — typically because the paper is reporting a distribution of vendor labels with no claim that depends on them being right. Read it as "no validation reported", not as "validation was unnecessary", unless you check the paper. | The last row is a sentinel and is shown because the page's own rule says sentinels are never folded into an answer. It means the extraction could not find a validation step and did not think one was called for — typically because the paper is reporting a distribution of vendor labels with no claim that depends on them being right. Read it as "no validation reported", not as "validation was unnecessary", unless you check the paper. |
| |
| Of those that do validate: 70 papers by manual validation, 18 by cross-validation, 9 by a held-out test set, 9 by comparison to another method. Note the shape — manual spot-checking is the dominant and correct instrument here, because a held-out test set only tells you about the labels you already had. | Of those that do validate: 92 papers by manual validation, 21 by cross-validation, 13 by a held-out test set, 10 by comparison to another method. Note the shape — manual spot-checking is the dominant and correct instrument here, because a held-out test set only tells you about the labels you already had. |
| |
| The 50.2% figure is worth dwelling on: for **website** classification, the share naming a ground-truth source is well below the 71.7% the same extraction measures across all 3,241 papers that classify anything. Website categorisation is reported worse than classification in general, not better. | The 51.2% figure is worth dwelling on: for **website** classification, the share naming a ground-truth source is well below the 72.9% the same extraction measures across all 4,439 papers that classify anything. Website categorisation is reported worse than classification in general, not better. |
| |
| ==== Taxonomies are bespoke, and usually not even sized ==== | ==== Taxonomies are bespoke, and usually not even sized ==== |
| |
| Of the 184 papers that name a taxonomy at all: | Of the 248 papers that name a taxonomy at all: |
| |
| ^ Taxonomy ^ Papers ^ Share of 184 ^ | ^ Taxonomy ^ Papers ^ Share of 248 ^ |
| | A bespoke scheme, class count not given | 133 | 72.3% | | | A bespoke scheme, class count not given | 187 | 75.4% | |
| | A vendor's own category list | 37 | 20.1% | | | A vendor's own category list | 43 | 17.3% | |
| | DMOZ / ODP / Curlie | 6 | 3.3% | | | IAB | 9 | 3.6% | |
| | IAB | 6 | 3.3% | | | DMOZ / ODP / Curlie | 7 | 2.8% | |
| | A bespoke scheme with a stated class count | 6 | 3.3% | | | A bespoke scheme with a stated class count | 7 | 2.8% | |
| | Google AdWords / Topics | 3 | 1.6% | | | Google AdWords / Topics | 4 | 1.6% | |
| | A standard industry code (NAICS, SIC, NACE) | 1 | 0.5% | | | A standard industry code (NAICS, SIC, NACE) | 1 | 0.4% | |
| |
| Nearly three quarters invent a scheme and do not say how many classes it has. That single omission makes a result impossible to compare against any other paper, and it is free to fix. | Three quarters invent a scheme and do not say how many classes it has. That single omission makes a result impossible to compare against any other paper, and it is free to fix. |
| |
| ==== Where LLMs actually appear ==== | ==== Where LLMs actually appear ==== |
| |
| ^ Year ^ Papers using an LLM to classify anything ^ Corpus that year ^ Share ^ | ^ Year ^ Papers using an LLM to classify anything ^ Corpus that year ^ Share ^ |
| | 2023 | 2 | 630 | 0.3% | | | 2023 | 2 | 719 | 0.3% | |
| | 2024 | 27 | 628 | 4.3% | | | 2024 | 27 | 690 | 3.9% | |
| | | 2025 //(provisional)// | 77 | 770 | 10.0% | |
| | | 2026 //(provisional)// | 71 | 415 | 17.1% | |
| | |
| | **177 papers, and the curve is no longer starting — it is the fastest-moving thing in this corpus.** From 2 papers in 2023 to one in six in 2026, and both of the last two venue-years are incomplete, so those shares are floors. They point the model at "other" targets (116 papers), user-generated text (32), vulnerabilities (14), privacy policies (12), mobile apps (6), website categories (5), and single-digit counts of domains, network traffic, consent notices, requests, IP addresses, cookies, email and dark patterns. |
| | |
| | They establish a norm worth copying: **148 of 177 (83.6%) validate the model's labels against something**, usually manual annotation of a sample — the same share as when this was 29 papers, which is the encouraging part. |
| | |
| | **Five papers now point an LLM at a website category, and two of those are genuine topic classification** rather than an abuse verdict: |
| |
| Twenty-nine papers across the corpus's fifteen years, twenty-seven of them in the final one — the curve is visibly starting, and the corpus ends before it goes anywhere. They point the model at "other" targets (19 papers), user-generated text (6), vulnerabilities (4), privacy policies (1) and domains (1). **Not one of the 29 uses an LLM to assign a website a topic category.** | * {[tsai2025_harmful]} (TheWebConf 2025) classifies Tranco sites as shopping or non-shopping with **GPT-4o prompted with the URL and a screenshot**, validates against 500 manually annotated sites, and reports **92% accuracy, "comparable to commercial website classification services"**. That sentence is the first thing in this corpus that makes an LLM a real alternative to the vendors catalogued above, and it is worth reading before you buy an API key. |
| | * DarkGram (USENIX Security 2025) has GPT-4 label cybercriminal Telegram channels into 15 predefined categories, which the model itself extended to 19, validated by coders on 1,000 sampled posts. Categories of a channel, not of a website — count it as adjacent. |
| |
| They do, however, establish a norm worth copying: **24 of 29 (82.8%) validate the model's labels against something**, usually manual annotation of a sample. The single ''website-category'' LLM tuple in the whole corpus is a USENIX Security 2024 phishing detector using GPT-3.5-turbo-16k with a chain-of-thought prompt for a binary credential-page question, validated against three cybersecurity experts' annotations — a security verdict, not a topic. | The other three are abuse verdicts: the USENIX Security 2024 phishing detector using GPT-3.5-turbo-16k for a binary credential-page question, a TheWebConf 2025 crypto-scam study using a Llama3:70b + GPT-4 hybrid, and an NDSS 2026 credential-theft phishing detector using GPT-4o with retrieval-augmented prompts. All three validate against manual annotation. |
| |
| See [[#LLM Classification]] for what has happened since the corpus ends. | See [[#LLM Classification]] for what has happened outside the corpus. |
| |
| ==== Two examples worth reading first ==== | ==== Two examples worth reading first ==== |
| ==== Methodology and limitations of these figures ==== | ==== Methodology and limitations of these figures ==== |
| |
| * **Denominators.** Every figure names its own. The corpus is 4,322 papers; 3,241 classify something; 320 carry a ''website-category'' target; **247 assign a topic, industry or sector label** and are this section's population. Never "of 4,322 papers". | * **Denominators.** Every figure names its own. The corpus is 5,859 papers; 4,439 classify something; 424 carry a ''website-category'' target; **330 assign a topic, industry or sector label** and are this section's population. Never "of 5,859 papers". |
| * **Papers, not mentions.** A paper naming McAfee three times counts once. | * **Papers, not mentions.** A paper naming McAfee three times counts once. |
| * **Sentinels are not answers.** ''not-stated'' and ''none-mentioned'' are excluded from every numerator and reported as their own row where they matter — which they do, since "does not validate" is the headline finding. | * **Sentinels are not answers.** ''not-stated'' and ''none-mentioned'' are excluded from every numerator and reported as their own row where they matter — which they do, since "does not validate" is the headline finding. |
| * **Free-text names are folded** into canonical families by an ordered regex list before counting, because exact strings agree run-to-run only about 20% of the time. The fold left **20 distinct strings unclassified**, each appearing once: among them ''Taobao main-business categories'', ''SNET pillar portals'', ''Rakuten Popshops'', ''multiple domain classification services'' and ''public classification service from the security vendor''. The last two are papers that used a service and declined to say which — a small, exact instance of the reporting gap this section measures. | * **Free-text names are folded** into canonical families by an ordered regex list before counting, because exact strings agree run-to-run only about 20% of the time. The fold left **33 distinct strings unclassified**, each appearing once: among them ''Taobao main-business categories'', ''SNET pillar portals'', ''Rakuten Popshops'', ''NAICSlite'', ''ORES topic routing'', ''multiple domain classification services'' and ''public classification service from the security vendor''. The last two are papers that used a service and declined to say which — a small, exact instance of the reporting gap this section measures. The residue was 40 strings before the fold was extended for this corpus; [[provenance:design:website_classification]] lists what was added and what was deliberately left out. |
| * **The task fold is imperfect and its exclusions are visible.** Eight side-channel and website-fingerprinting papers named their classifier only as "a 1D CNN" or "k-NN", leaving no signal in any extracted field; they were caught by matching the paper's own slug and are excluded by an explicit named list rather than by a general rule. The 73 excluded papers are counted in the table above rather than dropped. | * **The task fold is imperfect and its exclusions are visible.** Eight side-channel and website-fingerprinting papers named their classifier only as "a 1D CNN" or "k-NN", leaving no signal in any extracted field; they were caught by matching the paper's own slug and are excluded by an explicit named list rather than by a general rule. The 94 excluded papers are counted in the table above rather than dropped. |
| * **Enum fields versus free text.** Method and validation are enums, stable enough to publish as rough shares (''classification.method'' agrees 58% run-to-run, so read those as a ranking). Service names and taxonomies are free text and are reported as rankings and folded families only. | * **Enum fields versus free text.** Method and validation are enums, stable enough to publish as rough shares (''classification.method'' agrees 58% run-to-run, so read those as a ranking). Service names and taxonomies are free text and are reported as rankings and folded families only. |
| * **Quotes were checked.** Every figure above traces to tuples carrying a verbatim evidence quote; a sample of these was re-located in the source PDFs. Of six quotes checked by hand, two initially "failed" a literal grep and turned out to be intact but split across a two-column break — normalise whitespace before concluding that a quote is not in the paper. | * **Quotes were checked.** Every figure above traces to tuples carrying a verbatim evidence quote; a sample of these was re-located in the source PDFs. Of six quotes checked by hand, two initially "failed" a literal grep and turned out to be intact but split across a two-column break — normalise whitespace before concluding that a quote is not in the paper. The five new LLM website-category tuples were re-checked individually on 2026-08-12; see [[provenance:design:website_classification]]. |
| * **Coverage.** IEEE S&P is only 43% retrieved, which is the most likely explanation for its 1.2% share. EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS are absent entirely, and so is anything published after 2024. Notably, **{[vallina2020_misshapes]} itself is in the venue index but has no extracted full text** — the reference work for this page is not in the population the page measures. | * **Coverage.** Seven venues only, with 2025 and 2026 incomplete by construction rather than by relevance, so any row that reaches them is a floor; the scope and the selection funnel are on [[literature:corpus]]. Notably, **{[vallina2020_misshapes]} itself is in the venue index but has no extracted full text** — the reference work for this page is not in the population the page measures. |
| | * **Every query behind this section, the report script and its unedited output** are on [[provenance:design:website_classification]]; corpus-level caveats are on [[literature:corpus]]. |
| |
| ===== Open Questions ===== | ===== Open Questions ===== |
