You have a list of domains — a Tranco slice, the third parties a crawl touched, the sites that set a cookie before consent — and a reviewer wants to know what kind of sites they are. Are the offenders news sites or shops? Does the effect hold outside adult content? Is the sample dominated by one sector?

This page is about turning a domain into a category you can defend. Its companion Website Selection is about which domains to put on the list in the first place; popularity ranking lives there, category and company data live here.

Before anything else: “classify a website” is not one measurement. It is at least five, they use different data, and they are not equally reliable.

The rows are different measurements with different failure modes, and the extraction over our publication corpus shows papers routinely reaching for a topic categoriser to answer one of the other four (see Use in Publications).

If you read one thing before starting, make it [1Vallina, Pelayo; Le Pochat, Victor; Feal, ´Alvaro; Paraschiv, Marius; Gamba, Julien; Burke, Tim; Hohlfeld, Oliver; Tapiador, Juan; Vallina-Rodriguez, Narseo (2020): "Mis-shapes, Mistakes, Misfits: An Analysis of Domain Classification Services", in: Proceedings of the ACM Internet Measurement Conference, pp. 598–618. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] (IMC 2020). It is the only large-scale independent audit of these services that exists, it is six years old, and nobody has redone it — which is itself something you should know before you build a result on top of one.

Note that several of these services are not meant for large-scale data mining, and querying them at research scale may violate their terms. MeasureTheWeb authors take no responsibility for your use of these services. Check with your IRB and legal department if you want to be sure that you can use them.

Vallina et al. [1Vallina, Pelayo; Le Pochat, Victor; Feal, ´Alvaro; Paraschiv, Marius; Gamba, Julien; Burke, Tim; Hohlfeld, Oliver; Tapiador, Juan; Vallina-Rodriguez, Narseo (2020): "Mis-shapes, Mistakes, Misfits: An Analysis of Domain Classification Services", in: Proceedings of the ACM Internet Measurement Conference, pp. 598–618. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] studied 13 domain classification services and retrieved labels from 11 of them for 4,424,142 domains (September 2019, Alexa rankings aggregated Tranco-style; DMOZ and Curlie were excluded from the label comparison as discovery directories, and three rate-limited services could only be measured on the top-10k). Six years on it is still the reference, so it is worth having its actual numbers rather than the vibe of them.

Coverage varies by two orders of magnitude. FortiGuard and McAfee labelled around 94% of the 4.4M domains. OpenDNS managed 15% — its human submit-and-vote pipeline cannot keep up with domain registration. Alexa managed 0.53%. Combining services buys less than you would hope: the union of FortiGuard and McAfee reaches just over 98%, because their intersection was already large.

Coverage collapses outside the head of the distribution. McAfee and FortiGuard hold at ≥93% from the top-1k down into the long tail past rank 1M. Bitdefender falls from 93% to 27% over the same range, and Forcepoint from 98% to 48%. Symantec, Trend Micro (queried directly) and Webshrinker all exceed 96% on the top-10k — but their rate limits make anything larger infeasible, which is why the paper could only measure them on the top-10k in the first place. If your population is not a top list, the vendor's headline coverage number does not apply to you.

Documented taxonomies do not match observed ones. Of the services whose documentation could be compared against their output, every one except Symantec returned at least one category that is not in its own documentation. McAfee returned six categories that its own docs had marked deprecated since 2010. FortiGuard omits sensitive categories (e.g. homosexuality) from public documentation while still assigning them. Trend Micro's labels via VirusTotal matched its 2011 taxonomy better than its 2019 one, implying VirusTotal was sourcing from an older product.

Labels are multi-valued in ways you will not expect. Six services — OpenDNS, McAfee, Dr.Web, Forcepoint, Trend Micro and Websense — can return more than one category per domain. In Dr.Web, 67% of labelled domains have several; in Forcepoint and Websense, under 1%. If your pipeline assumes one label per domain, you are silently dropping data from some services and not others.

Human labelling is genuinely hard. In the paper's own small manual experiment, two labellers using the OpenDNS taxonomy disagreed on 35.5% of domains, and reached only 71% accuracy against OpenDNS's own answer. Budget for this: it is the number your inter-annotator agreement is competing against, and it is worse than most people assume before they try it.

Vallina et al.'s case studies are the part of the paper that should change what you do. They took ground-truth sets and asked whether the services could recover them.

Adult content is the case where the services do work well — four of the nine measured recovered it at over 90%, and a fifth (OpenDNS) at 88%. That is a real, usable result, and it is the strongest single reason a topic categoriser earns its place in a measurement pipeline.

Mostly the corporate ownership, and not in your favour.

The one modern data point comes from a 2025/2026 arXiv preprint on LLM-assisted web measurement [2Bozzolan, Simone; Calzavara, Stefano; Cazzaro, Lorenzo (2026): "LLM-Assisted Web Measurements". arXiv:2510.08101, v3, 30 April 2026 (Link)], whose Appendix B explicitly reuses Vallina et al.'s service list. Its findings:

• Of the surviving free tiers, only Cloudflare and VirusTotal offer one at all, and both are unusable at scale: Cloudflare's Domain Threat Intelligence API allows 100 requests per month, so 100k websites would take “roughly 30,000 days”; VirusTotal's 500/day cap puts the same job at “around 200 days”.
• The Cloudflare Radar API has no explicit rate limit, so they could benchmark it. Against a 7,780-site ground truth for target-audience country, Radar returned a label for 3,925 sites (50.4%) and, of those, agreed with the ground truth in 39.7% of cases. They hand-checked 50 disagreements: their own label was right in 49 (98%), Cloudflare's in 1 (2%).
• Every open-weight LLM they tested reached 100% coverage and at least 76% accuracy on the same data, without visiting the sites.

That is one service on one task, not a re-audit. But it is the only 2020s number available, and it points the same way the 2020 numbers did.

Status of every service checked by direct fetch on 7 August 2026. Where a lookup exists but no API does, that is stated plainly — it is the most common case, and it is what makes these services awkward for research.

These sell web filtering to enterprises. Their taxonomies are risk-oriented (Proxy Avoidance, Potentially Unwanted Program) with a general-interest tail bolted on. High coverage, coarse labels, essentially no bulk access.

The database papers cite as “McAfee URL categorization”, “TrustedSource”, “SmartFilter” or the “Customer URL Ticketing System” is one thing, and it no longer belongs to McAfee. McAfee Enterprise split into Trellix and Skyhigh Security; the URL-categorisation service migrated to them on 20 February 2024.¹⁾

• Lookup: trustedsource.org — live, footer reads “Copyright © Musarubra US LLC” (Trellix's legal entity), and the product dropdown offers both Trellix products (Real-Time Database, Endpoint Security Web Control, SmartFilter) and Skyhigh's Secure Web Gateway. One database, two companies.
• API: none. The page states: “this Web-based service is designed for occasional use by individuals. If you have hundreds or thousand of URLs you would like to test, please write to us about your needs… If we see heavy use of this Web-based service by automated programs, we reserve the right to deny access.” Bulk requests go to sites@trellix.com.
• Advantages: joint-highest measured coverage of any service (~94% in 2020, tied with FortiGuard), holding into the long tail; multi-label; automated with manual oversight.
• Disadvantages: coarse labels, unsuitable for marketing or behavioural analysis; documentation carries categories deprecated since 2010 that the service still returns; no programmatic access, so at research scale you are either asking permission or violating the notice above.

• Lookup: fortiguard.com/webfilter (single URL or IP, with rating history). Published taxonomy: fortiguard.com/webfilter/categories — 93 named categories in 7 groups (Adult/Mature Content, Bandwidth Consuming, General Interest – Business, General Interest – Personal, Potentially Liable, Security Risk, Unrated), each with a one-sentence definition. Counted from the live page on 2026-08-07.
• API: no public bulk API. FortiOS exposes /api/v2/cmdb/webfilter/ftgd-local-rating but that requires a licensed FortiGate appliance, not a research credential.
• Advantages: the joint-best coverage in the 2020 audit (~94%, stable into the long tail); the most completely and publicly documented taxonomy of any vendor here, which makes it the best choice when you need to name categories in a paper; real-time classification.
• Disadvantages: security-shaped granularity; documentation omits some sensitive categories the service nonetheless assigns; no bulk access; recovered only 7.7% of a tracking ground truth.

Broadcom acquired Symantec Enterprise in 2019; WebFilter and WebPulse are still operated under the Symantec brand and still shipping categorisation updates in 2026.

• Lookup: sitereview.symantec.com — alive. It is an Angular single-page app, so a plain curl returns a splash screen with a <noscript> notice; that is a JavaScript UI, not a dead service. Category definitions live in Broadcom TechDocs.
• API: none public. Single-URL lookup plus a re-review request workflow.
• Advantages: high accuracy on top-tier domains; the only service in the 2020 audit that returned no undocumented categories; good for threat assessment and content filtering.
• Disadvantages: less diverse taxonomy than the marketing-oriented services; rate-limited hard enough that the 2020 audit could only measure it on the top-10k; poor long-tail coverage.

• Lookup: Site Safety Center — alive. Taxonomy documented at docs.trendmicro.com: 7 top-level groups, roughly 80–90 sub-categories.
• API: none public; same occasional-individual-use posture as Trellix.
• Advantages: high coverage for popular domains, rapid updates, labels aligned with threat intelligence.
• Disadvantages — the earlier version of this page asked whether there were any, and the audit answers it: coverage is poor everywhere except the very top of the list. Trend Micro assigned an adult-related label to only 11.8% of a curated adult-content set (it returned any label at all for just 12.1% of them) and a tracking-related label to 0.9% of a tracking set. Its labels through VirusTotal also correspond to an older taxonomy than the one it documents, so which route you query by changes the answer.

• Lookup: not fully public — the Site Lookup Tool sits behind a support.forcepoint.com login, and the category list is published inside the Cloud portal help. Forcepoint's product page claims “more than 95 URL categories in more than 50 languages”.
• API: a Management API exists for administering a deployment you already own, not for third-party lookups. An XLabs submission tool accepts up to 10,000 URLs for classification requests, not retrieval.
• Advantages: fast, scalable label updates; the best performer in the whole 2020 audit on tracking domains (15.1% correct) and strong on adult content (94.9%).
• Disadvantages: needing an account to look a single domain up makes it the least accessible service here; coverage drops from 98% to 48% between the top-1k and the long tail; multi-labelling exists but fires on under 1% of domains.

• Lookup: online.drweb.com (“Check URL”) and vms.drweb.com/online. Parental-control categories (Adult content, Violence, Weapons, Gambling, Drugs, Terrorism, Obscene language, plus optional ones) are documented in fragments rather than as one canonical list.²⁾
• API: none.
• Advantages: minimalistic and consistent labels; recovered 92.4% of adult content, which is its whole design goal.
• Disadvantages: very low coverage outside its niche — it returned any label for 9.0% of tracking domains and 0.3% of CDN domains, and a correctly related label for 0.0% of both; 67% of its labelled domains carry several categories, the highest multi-labelling rate measured, which breaks single-label pipelines silently.

• Zvelo (zveloDB / zveloCAT) — a current, actively marketed competitor to the six above. Self-reported ~500 categories over 200+ languages, delivered as API, SDK or raw feed. Sales-quoted only, no public pricing or free tier, and its coverage claims are self-reported and have never been independently audited.
• Palo Alto Networks PAN-DB — free public single-site lookup (“Test A Site”), up to four categories per URL, with per-category benign test URLs published for verification. The total category count varies by PAN-OS version (figures between 57 and 90+ circulate), so quote it against a version rather than in the abstract.

Finer taxonomies aimed at advertising rather than filtering, so the labels are the ones an IAB-shaped research question actually wants — at the cost of coverage, and of money.

• Advantages: high-quality data; industry, region- and origin-based popularity; the taxonomy used in practice is ~24 top-level website categories.
• Disadvantages: the official API v5 is paid on a “Data Credits” model with no published price list — you get a sales quote. The unofficial endpoint is a clear terms violation: SimilarWeb's terms prohibit “use any robot, spider, scraper, or other automated means to access or monitor the Platform, Site or Applications for any purpose, or to copy, access, collect, download, or otherwise obtain data, content or other information”.³⁾
• See SimilarWeb for documentation of the unofficial API, code and output example, and note the date it was last confirmed working.

Still alive, and not renamed. DNSFilter acquired Web Shrinker on 17 April 2018;⁴⁾ eight years on, webshrinker.com is live with its own branding and product line (Offline Database, Domain API Feed, Screenshot API Feed) sold through DNSFilter's pricing page. Its homepage claims 380 million websites categorised with a five-day re-scan cycle.

• Advantages: offers both a “Simplified Categories” scheme and an IAB-aligned taxonomy, which is what makes it attractive for advertising research; automated real-time updates; the widest label vocabulary of the audited services (299 observed labels against 401 documented).
• Disadvantages: precision and granularity vary; 103 of its documented categories were never observed in the 2020 audit, so the documentation over-promises; rate limits confined the audit to the top-10k; commercial terms are now DNSFilter's.

Two more appear in the corpus, and their fates illustrate the risk of building on a small vendor.

• Klazify — used by an IMC 2023 paper to map dead links to categories. The domain resolves and is served behind Cloudflare, but Cloudflare's bot interstitial returned HTTP 403 to both curl and a headless browser on 2026-08-07, so we could not confirm anything about the current product from the site itself.
• Website Categorify — used by a CCS 2020 paper on DNS hijacking, which reported 195 domains left uncategorised. As of 2026-08-07 websitecategorify.com does not resolve at all: the domain is gone, and with it any hope of reproducing that paper's categorisation.

Neither has been independently audited. Treat the accuracy of a small vendor as unknown until you measure it on your own sample — and treat its continued existence as unknown too, which is an argument for publishing your label file rather than the query script.

The living successor is Cisco Umbrella Investigate, which offers domain/IP categorisation through a documented API — but it is an enterprise product with no published price list and no free tier, sold through a sales process. This is a strict downgrade for researchers: the community system was free and transparent, and what replaced it is neither.

For the historical record, OpenDNS's properties in the 2020 audit remain instructive about crowd-curation in general: 15% coverage, human submission and voting as the bottleneck, five security categories closed to community submission, and domains with as many as 58 rejected label proposals.

DMOZ / the Open Directory Project closed in 2017; Curlie is its volunteer-run continuation and is the one free bulk source of human-curated website categories that still exists.

• Bulk dump: curlie.org/directory-dl redirects to the current archive. Tab-separated text, gzipped. Fetched 2026-08-07: 169 MiB (177,289,960 bytes), last modified 2 February 2026, covering ~2.9 million entries with the category hierarchy, URLs, titles and editorial descriptions. The filenames still say “RDF” for historical reasons; the format is TSV.
• Cadence: Curlie's documentation says “we strive to pull a fresh copy from the Curlie database every month”. The copy we fetched was six months old, so check Last-Modified before you cite a snapshot date rather than assuming monthly freshness.
• Taxonomy: 15 top-level categories in the English tree — Arts, Business, Computers, Games, Health, Home, News, Recreation, Reference, Regional, Science, Shopping, Society, Sports, Kids and Teens — deeply hierarchical below that, with 92 further language trees alongside. Counted from curlie.org/en on 2026-08-07. Homepage2Vec predicts into 14 of these, dropping Regional.
• Advantages: free, redistributable, human-curated, multilingual, and it is the training set behind essentially every open website classifier — including Homepage2Vec.
• Disadvantages: 2.9M URLs is tiny against the modern web and skews to older, established sites; many branches have not been touched in years. The most important caveat is under-labelling: editors assign one or two relevant categories, not all of them. When Homepage2Vec's authors had crowdworkers re-label 807 Curlie sites, Curlie had supplied 836 labels (1.04 per site) and the crowd assigned 2,088 (2.59 per site) — a 2.5× increase [3Lugeon, Sylvain; Piccardi, Tiziano; West, Robert (2022): "Homepage2Vec: Language-Agnostic Website Embedding and Classification", in: Proceedings of the International AAAI Conference on Web and Social Media, pp. 1285-1291. (DOI)]. Treated as ground truth, Curlie will make any multi-label classifier look far less precise than it is.
• Vallina et al. deliberately excluded DMOZ and Curlie from their comparison, on the grounds that a discovery directory and a classification service are answering different questions. That distinction is worth keeping.

• API: GET /api/v3/domains/{domain} returns a categories attribute, “a dictionary mapping that relates categorisation services with the category it assigns the domain to”. Documented at docs.virustotal.com.
• Which vendors: Dr.Web, Forcepoint ThreatSeeker, BitDefender, Sophos, Trend Micro, Websense and (legacy) Alexa, among others — so several of the vendors above reach you through VirusTotal without you querying them.
• Rate limit: the free tier is 500 requests per day and 4 requests per minute, unchanged in substance by the October 2025 retiering into Community / Contributor / Lite / Duet.⁵⁾ The same page adds that the Public API “must not be used in commercial products or services”. At 500/day, a 100k-domain study takes 200 days. Vallina et al. had an academic key at 20k/day; there is no self-serve academic application form today, so budget for a direct conversation with VirusTotal rather than assuming access.
• Advantages: aggregates many providers in a single call; widely used and easy to cite.
• Disadvantages: the integration is lossy. In the 2020 audit, several services returned labels when queried directly but not through VirusTotal, and Trend Micro's VirusTotal labels tracked its 2011 taxonomy rather than its 2019 one. You do not control which product version you are reading, and “we used VirusTotal categories” does not identify the underlying source.

None of these existed when the older parts of this page were written, and all three appear in corpus papers from 2019 onward.

Two different products, easily conflated:

• Radar API — free, no explicit rate limit, data licensed CC BY-NC 4.0 (non-commercial only, which matters for industry co-authors). Its measured quality is poor: 50.4% coverage and 39.7% agreement on the one 2026 benchmark that exists [2Bozzolan, Simone; Calzavara, Stefano; Cazzaro, Lorenzo (2026): "LLM-Assisted Web Measurements". arXiv:2510.08101, v3, 30 April 2026 (Link)].
• Domain Intelligence, now under Security Center Intel — GET /accounts/{account_id}/intel/domain?domain=… returns content_categories. This is the one the corpus papers used: [4Ruth, Kimberly; Kumar, Deepak; Wang, Brandon; Valenta, Luke; Durumeric, Zakir (2022): "Toppling Top Lists: Evaluating the Accuracy of Popular Website Lists", in: Proceedings of the 22nd ACM Internet Measurement Conference, pp. 374–387. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] labelled Cloudflare-managed sites with it, and [5Ruth, Kimberly; Fass, Aurore; Azose, Jonathan; Pearson, Mark; Thomas, Emma; Sadowski, Caitlin; Durumeric, Zakir (2022): "A World Wide View of Browsing the World Wide Web", in: Proceedings of the ACM Internet Measurement Conference, pp. 317-336. (DOI)] built a whole global browsing study on it. It needs an API token with Intel permission scoped to an account; whether it is available on the free plan is not stated in Cloudflare's public docs.⁶⁾

The best worked example of using it well is [5Ruth, Kimberly; Fass, Aurore; Azose, Jonathan; Pearson, Mark; Thomas, Emma; Sadowski, Caitlin; Durumeric, Zakir (2022): "A World Wide View of Browsing the World Wide Web", in: Proceedings of the ACM Internet Measurement Conference, pp. 317-336. (DOI)] (IMC 2022): the API places domains into 26 super-categories and 114 categories; the authors manually validated ten random websites per category and kept only categories that reached 80% accuracy, excluding 19 of them and merging others, ending at 22 super-categories and 61 categories. Notably, Search Engines and Social Networks were among the categories that failed — two of the most important classes on the web. Copy this procedure.

Classifies text, not domains, so you have to fetch and extract the page yourself — but the taxonomy is the advertising-shaped one many privacy questions actually need. Papers in the corpus report it variously as 27 level-1 categories, 35 categories with 400 subcategories, and ~620 AdWords-style buckets, depending on the year and API version; state the version and the date you queried. Priced per unit of text, no free research tier of consequence.

The browser-side interest taxonomy (349 topics, assigned by a model Google trained on a manually curated set of 10,000 domains) is a measurement target rather than a tool — [6Jha, Nikhil; Trevisan, Martino; Leonardi, Emilio; Mellia, Marco (2023): "On the Robustness of Topics API to a Re-Identification Attack", Proceedings on Privacy Enhancing Technologies 2023(4):66-78. (DOI)] attacks its re-identification properties. Treat it as an object of study, not a categoriser you can call; the 2026 survey of classification services reports it as deprecated and scheduled for removal [2Bozzolan, Simone; Calzavara, Stefano; Cazzaro, Lorenzo (2026): "LLM-Assisted Web Measurements". arXiv:2510.08101, v3, 30 April 2026 (Link)].

Amazon retired Alexa.com on 1 May 2022, per its own end-of-service notice (“we will be retiring Alexa.com on May 1, 2022”).⁷⁾ Its ranking service and its category service died together.

• Why it still matters: it appears in 12 of the 247 corpus papers that categorise websites, under eight different spellings, and papers published as late as 2024 still use it because their data collection predates the shutdown. If you are reading such a paper, the labels are from a dead service with a documented 0.53% coverage rate.
• What it was: highly granular (7,557 distinct labels observed through VirusTotal, against 279,716 categories in Alexa's own directory, which was derived from DMOZ), and capped at 500 sites per category — hence the tiny coverage. Leaf labels came back in the language of the branch, so the same concept appeared as Arts, Artes and Kultur.

Ranked for a researcher who needs topic labels for tens of thousands of domains. Coverage figures are from [1Vallina, Pelayo; Le Pochat, Victor; Feal, ´Alvaro; Paraschiv, Marius; Gamba, Julien; Burke, Tim; Hohlfeld, Oliver; Tapiador, Juan; Vallina-Rodriguez, Narseo (2020): "Mis-shapes, Mistakes, Misfits: An Analysis of Domain Classification Services", in: Proceedings of the ACM Internet Measurement Conference, pp. 598–618. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)], measured in 2019 on 4.4M domains, and are the most recent independent numbers that exist.

The service catalogue above is a list of things you cannot get at scale, cannot redistribute, and cannot reproduce. Running your own classifier avoids all three problems, and since 2022 the open options have been good enough that this is the default recommendation for a topic-labelling task.

[3Lugeon, Sylvain; Piccardi, Tiziano; West, Robert (2022): "Homepage2Vec: Language-Agnostic Website Embedding and Classification", in: Proceedings of the International AAAI Conference on Web and Social Media, pp. 1285-1291. (DOI)] (ICWSM 2022, EPFL) is the reference open model. It takes a homepage — HTML text, metadata, optionally a screenshot — and emits both a 100-dimensional embedding and independent multi-label probabilities over Curlie's 14 top-level categories. It is language-agnostic by construction, built on XLM-R sentence embeddings, and trained on more than two million Curlie-labelled sites in 92 languages (April 2021 snapshot).

• Code: github.com/epfl-dlab/homepage2vec, MIT licensed, pip install homepage2vec. Not actively maintained — last commit 22 January 2024, no tagged releases.
• Training data: the curated Curlie snapshot is released separately at figshare under CC BY 3.0. This is the practical way to get an aligned, labelled Curlie set without parsing the raw dump.

Its documented weaknesses: rare classes are much worse than frequent ones (Kids_and_Teens is ~1% of the data, Business ~27%); the visual branch requires a full Selenium and Chrome pipeline for a limited gain the authors themselves call out; and the Curlie under-labelling problem above depresses its apparent precision — crowd re-labelling raised measured macro precision on the affected subset from 0.734 to 0.873.

“Multilingual Taxonomic Web Page Categorization Through Ensemble Knowledge Distillation” [7Ye, Eric; Bai, Xiao; O'Hare, Neil; Asgarieh, Eliyar; Thadani, Kapil; Perez-Sorrosal, Francisco; Adiga, Sujyothi (2024): "Multilingual Taxonomic Web Page Categorization Through Ensemble Knowledge Distillation", IEEE Transactions on Knowledge and Data Engineering 36(11):6614-6627. (DOI)] (IEEE TKDE 36(11):6614–6627) is Yahoo Research's production contextual-ad-targeting classifier, an extended version of a KDD 2022 paper. A unified multilingual Transformer classifies a page from either full content or the URL alone — URL-only matters because ad bid requests arrive as bare URLs — with class-based loss re-weighting for label skew and an ensemble of teacher models distilled into one lightweight student.

Its taxonomy is Yahoo's proprietary 442-category, 5-tier Interest Category scheme over five languages (English, Spanish, French, Portuguese, Traditional Chinese), trained on ~248k editorially annotated pages. It reports “at least 2.6% improvement in mean average precision across 5 languages” over single-teacher distillation and “at least 37% better mAP than the legacy tree-based models” in production.

Neither the model, the data nor the taxonomy is released. Read it for the method — the URL-only-and-content-in-one-model trick, and the distillation setup — not as something you can use. It also does not compare against Homepage2Vec, and could not sensibly: 442 proprietary categories versus 14 public ones is not a shared benchmark.

• WebOrganizer (ICML 2025) — two 24-category taxonomies (Topic and Format) produced by prompting Llama-3.1-405B over Common Crawl and distilling into 140M-parameter classifiers, released on HuggingFace with -NoURL variants. Built for pre-training data curation, so it classifies pages, not sites — you would have to aggregate per-page predictions to a domain yourself.
• SoAC (DocEng 2025) — 195,495 websites collected in 2024, labelled with 10 coarse sectors and 148 fine-grained industry categories, MIT licensed, including extracted site text. Their own classifier reaches 72.6% accuracy on it. This is the closest thing to an open industry-sector ground truth, which the company datasets below otherwise monopolise.
• WebKB (CMU, cs.cmu.edu/~webkb) — still cited, but it is 7 classes of university CS-department pages in English. It is a text-classification benchmark, not website-category ground truth. Do not use it to justify a claim about the web.

The evidence outside the corpus is a 2025/2026 preprint, [2Bozzolan, Simone; Calzavara, Stefano; Cazzaro, Lorenzo (2026): "LLM-Assisted Web Measurements". arXiv:2510.08101, v3, 30 April 2026 (Link)], which by its authors' account is the first systematic study of LLMs for web-measurement classification. It evaluates five open-weight, self-hosted models (gpt-oss:20b, llama3.3:70b, mistral-small3.2:24b, phi4-mini:3.8b, qwen3:32b, all via Ollama, temperature 0) on three tasks with hand-built benchmarks, in two modes: URL string only, and URL plus live browsing through LangChain and Playwright.

The number that matters for this page: on the same 13-category benchmark, Homepage2Vec scores 0.60 Jaccard and 0.55 macro F1 — the paper's words are that “a general-purpose LLM like gpt-oss can label websites with higher accuracy than a classic machine learning model specifically trained for website categorization.” A stock open-weight instruct model, with no fine-tuning, beat the field's reference specialist classifier.

Three practical consequences:

1. URL-only works for coarse questions and fails for topics. Every model reached at least 0.86 accuracy on the binary government task from the URL string alone, and exceeded 0.75 on both accuracy and macro F1 for the 10-way country task. Topic classification is where live access earns its cost: 0.51 → 0.65 Jaccard for the best model. If all you have is a domain list with no crawl, budget accordingly.
2. Self-hosting is free in dollars and expensive in wall-clock. On one A40 GPU, median under 2 s and mean under 4 s per site: ~3 days for 100k sites URL-only, ~4 days 9 hours with live access, and ~30–44 days for the Tranco top 1M. A hosted small model is cheaper in time — at August 2026 list prices for a nano-tier model and ~300 input / 20 output tokens per domain, URL-only classification of 100k domains costs on the order of a few dollars, roughly halved by a batch API.⁸⁾ The tradeoff is reproducibility: an open-weight model you pin by digest is reproducible five years from now, and gpt-4o-2024-08-06 is not.
3. Ensembling helps because single models are unstable. A 2025 evaluation of ten LLMs zero-shot against the IAB taxonomy on 8,660 human-annotated samples names “inconsistency, hallucination, category inflation, and misclassification” as the individual-model failure modes, and reports up to 65% F1 improvement from majority-vote ensembling over the strongest single model.⁹⁾

Constrain the output. Every failure mode above gets worse when the model is free to invent a category name. Give it the taxonomy in the prompt, force a structured output (a JSON schema or a constrained-decoding grammar), reject anything outside the enum, and count the rejections — that count is a result.

Three things nobody has measured yet, so do not assert them and consider measuring them yourself:

• Label instability across seeds and temperature for website classification specifically. The one systematic paper set temperature to 0 to sidestep the question rather than quantify it.
• Behaviour on parked, dead and holding-page domains — a large fraction of any long-tail sample, and a case where “visit the site” gives the model a page that is not about anything.
• A formal agreement statistic (Cohen's or Fleiss' kappa) between LLM labels and human annotators for this task. Nobody has published one. The nearest available figures are a validation of the benchmark's own ground truth, not of an LLM: the authors hand-relabelled 150 of their 3,785 sites, and their automated label-construction procedure matched the human label set exactly on 86 (57%) with a further 47 (31%) a proper subset of it, where Lugeon et al.'s Curlie snapshot matched exactly on only 67 (45%) of the same sites. That is a statement about how good the ground truth is — and a reason to read the 0.65 Jaccard above as a floor on model quality rather than a ceiling, because a multi-label benchmark that under-labels punishes a model for being right.

Pick the taxonomy before the service, because it is much harder to change later, and because 72% of the corpus papers that name a taxonomy at all name a bespoke one they invented (see Use in Publications) — which makes their results incomparable with everyone else's.

• IAB Tech Lab Content Taxonomy — the advertising industry's reference schema, and the right choice if your question is about ad targeting or if you want to be comparable to Webshrinker's output. Machine-readable TSV in the repo under Content Taxonomies/, CC BY 3.0, up to four tiers with a relational ID system. Versions 1.0 through 3.1 sit side by side; 2.x → 3.0 is a breaking change and IAB says the two must not be used together. Category rows counted from the TSVs on 2026-08-07: 2.2 has 1,196 rows under 37 tier-1 categories; 3.0 has 703 under 36; 3.1 has 704 under 37. Vendor marketing claiming that 3.x “expands to 1,500+ categories” is wrong in both directions — 3.x is roughly 40% smaller than 2.2, because it consolidated and restructured rather than adding. Count the rows in the file yourself; it takes ten seconds and the summaries are not reliable.
• Curlie's 15 top-level categories — the right choice if you want free ground truth, an open model that already predicts into it, and a hierarchy you can prune to whatever depth your sample supports.
• A vendor's own list — fine, but pin the version, and expect to discover categories that are not in the documentation. Vallina et al. found at least one undocumented category in eight of the nine services whose documentation they could compare against its output.
• A standard industry code (NAICS, SIC, NACE) — the right choice for a question about firms rather than content, and the natural join key to the company datasets below. Exactly one paper in our corpus did this.

Whatever you pick: report the number of classes and the pruning depth. “We categorised sites using IAB” is not reproducible: IAB 3.1 has 37 tier-1 categories and 704 rows in total, and reporting at one depth or the other is two very different measurements.

Compared to the services above, the following datasets are company-oriented instead of website-oriented. Some include the company's website, but this matching might be incomplete and might cause the following issues:

• If a company owns multiple websites:
  • Likely only the main website will be listed.
  • This is especially pronounced with international versions of the website.
• Likewise, the dataset may contain multiple companies for a given website:
  • Because of sister companies in a corporate group.
  • Many small businesses list social media as their website. Sometimes this link does not include the full path, so a single-person company might indicate facebook.com as its domain.

• Advantages:
  1. Freely available from PeopleDataLabs' website, which also includes good documentation. No registration is needed. CC BY 4.0, updated quarterly.
  2. Confirmed 2026-08-07: over 22 million companies, exactly 10 fields — country, founded, id, industry, linkedin_url, locality, name, region, size, website.
• Disadvantages:
  1. Based on LinkedIn profiles that are self-reported — prone to adversarial data.
  2. Only a subset of PeopleDataLabs' full dataset. The “22M of 70M rows” framing is longstanding on this page; the 22M is confirmed, the 70M total could not be re-confirmed in 2026 and PDL's marketing page now cites 23.8M+ without saying which corpus that is.

TODO: cite Machine Learning Compliance Analysis for Email Regulation when it is public.

Crunchbase is proprietary. You can check individual entries (e.g. Google) on their website in visual form, which is useful for a quick content check.

• Advantages:
  1. Very rich data: website, rank, region, industries, financial data.
• Disadvantages:
  1. URLs are extremely noisy — they are not the priority.¹⁰⁾
  2. Focuses mostly on variables useful for investments and market competitiveness.
  3. Academic access is no longer publicly documented; budget for a sales conversation.

TODO: cite Machine Learning Compliance Analysis for Email Regulation when it is public.

Orbis (Bureau van Dijk, acquired by Moody's in 2017 and now marketed as Moody's Orbis) is in many ways similar to Crunchbase. It is proprietary but offers academic licensing, and Moody's claims coverage of more than 635 million companies. It is primarily company-focused but contains URLs, and it covers the longer tail of private companies — which also means the data is noisier.

• Advantages:
  1. Longer tail data than Crunchbase.
  2. Academic access is real and still available — but it is arranged through your institution's library as a site licence or SSO entitlement, not through a public request form. Check your library's database list before contacting Moody's.
• Disadvantages:
  1. Extremely noisy.
  2. Fewer fields than Crunchbase.

There are many scraping services for datasets like LinkedIn, Glassdoor, Yahoo Finance business information, Yelp businesses, Indeed, etc. Their use might violate the terms and conditions of the primary data sources, but the whole industry is built on scraping each other's data, so you might be causing very limited harm. Nevertheless, check with your IRB and potentially your legal department before using these services.

• Coresignal: runs an explicit, application-based pro bono programme for academic and NGO researchers, with enriched company, job and workplace datasets given free to selected applicants; it expects the resulting research to be published within 12–24 months. This is the only concretely documented academic programme among the scraping services listed here. A free developer tier (200 Collect + 400 Search credits) exists for testing.
• BrightData: confirmed free tier of 5,000 credits per month plus a free-samples page. Its academic-licensing offer is informally reported rather than documented — do not plan around it.
• Amazon AWS Marketplace offers various datasets through external services.
• Kaggle: often contains smaller samples of various company datasets.

Multiple lists exist, mostly maintained for child protection in routers and similar. They are simple to access, and you can directly download a large list of URLs. For these specific categories, prefer them to a general classification service. That is what Vallina et al.'s case studies show: taking curated lists as ground truth, no commercial service recovered more than 15% of the tracking domains, and only five of nine recovered the adult-content domains well.

• https://github.com/Bon-Appetit/porn-domains: daily updated list of porn websites.
• https://github.com/hagezi/dns-blocklists: DNS blocklists for adult content, piracy, gambling, security (scam, malware), and privacy (ads, tracking).
• https://github.com/search?q=pihole%20blocklist&type=repositories: search for pi-hole blocklists for specific uses.
• https://oisd.nl/, https://firebog.net/
• https://dsi.ut-capitole.fr/blacklists/index_en.php contains filtering designed for blocking university traffic. Contains many lists, from porn to malicious or misinformation content.
• Citizen Lab test lists: per-country lists of politically and socially sensitive URLs with a documented category scheme, maintained for censorship measurement and used as such in the corpus.

Visit individual privacy-oriented pages for more details regarding classification of Requests, Cookies, Fingerprinting, and JavaScript.

• Martech provides data on 17k companies in the advertising industry. They are easy to download after registering on https://martechmap.com/; search for martech_data_N.json files in the network tab of your browser's dev tools.
• IAB Europe lists advertising and tracking third parties. On https://iabeurope.eu/vendor-list-tcf/, open the source code and find the table <table id=“tablepress-72” class=“tablepress tablepress-id-72”>. Copy that table into an HTML-table-to-CSV converter.

Used by several corpus papers on misinformation and political advertising, and worth knowing exists rather than rebuilding: Media Bias/Fact Check and AllSides both publish per-outlet bias and factuality ratings. Both are human-rated by small teams with their own editorial stance — report which one you used and treat the ratings as one organisation's judgement, not ground truth.

This is the part most papers skip, and it costs an afternoon.

The protocol is the one [5Ruth, Kimberly; Fass, Aurore; Azose, Jonathan; Pearson, Mark; Thomas, Emma; Sadowski, Caitlin; Durumeric, Zakir (2022): "A World Wide View of Browsing the World Wide Web", in: Proceedings of the ACM Internet Measurement Conference, pp. 317-336. (DOI)] used and the one Vallina et al. recommend: draw a stratified random sample, have two people label it independently while blind to the service's answer, report agreement, report per-category precision with a confidence interval, and drop the categories that do not clear a floor.

category_validation.py

#!/usr/bin/env python3
"""Validate a website categoriser's output the way a reviewer will ask you to.
 
Whatever assigned your categories -- a vendor API, Homepage2Vec, an LLM -- you
are claiming that a label means what its name says. This script produces the
evidence for that claim, and it is deliberately boring: stratified sample,
blind double annotation, per-category precision with a confidence interval,
inter-annotator agreement, and a drop rule for categories that do not hold up.
 
Standard library only: no numpy, no scipy, no network.
 
    # 1. draw the sample and write two blind annotation sheets
    python3 category_validation.py sample labels.csv --per-category 25 --seed 20260807
 
    # 2. annotate sheet_a.csv and sheet_b.csv by hand (fill the `verdict` column
    #    with the category you think is right, or `none` if nothing fits)
 
    # 3. score
    python3 category_validation.py score labels.csv sheet_a.csv sheet_b.csv --floor 0.80
 
`labels.csv` is `domain,category` with one row per domain. Use the literal
category `unlabelled` for domains the service returned nothing for -- coverage
is a result, not a preprocessing step, and dropping those rows silently is the
single most common way this measurement goes wrong.
"""
 
import argparse
import csv
import math
import random
import sys
from collections import Counter, defaultdict
 
UNLABELLED = "unlabelled"
 
 
# --------------------------------------------------------------------------
# statistics
# --------------------------------------------------------------------------
def wilson(k, n, z=1.959963984540054):
    """95% Wilson score interval for k successes in n trials.
 
    Wilson rather than the normal approximation because per-category samples
    are small (25-50) and precisions are often near 1.0, where the normal
    interval runs past 100% and stops being an interval.
    """
    if n == 0:
        return (0.0, 0.0, 1.0)
    p = k / n
    d = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / d
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return (p, max(0.0, centre - half), min(1.0, centre + half))
 
 
def cohen_kappa(pairs):
    """Cohen's kappa over (annotator_a, annotator_b) label pairs."""
    n = len(pairs)
    if n == 0:
        return float("nan")
    agree = sum(1 for a, b in pairs if a == b)
    po = agree / n
    ca = Counter(a for a, _ in pairs)
    cb = Counter(b for _, b in pairs)
    pe = sum(ca[k] * cb[k] for k in set(ca) | set(cb)) / (n * n)
    if pe == 1.0:
        return float("nan")
    return (po - pe) / (1 - pe)
 
 
def sample_size_for(halfwidth, p=0.9, z=1.959963984540054):
    """How many domains per category for a target CI half-width at precision p."""
    return math.ceil(z * z * p * (1 - p) / (halfwidth * halfwidth))
 
 
# --------------------------------------------------------------------------
# io
# --------------------------------------------------------------------------
def read_labels(path):
    with open(path, newline="", encoding="utf-8") as fh:
        rdr = csv.DictReader(fh)
        if rdr.fieldnames is None or "domain" not in rdr.fieldnames or "category" not in rdr.fieldnames:
            sys.exit(f"{path}: need a header row with columns `domain` and `category`")
        return [(r["domain"].strip(), r["category"].strip()) for r in rdr]
 
 
def read_sheet(path):
    with open(path, newline="", encoding="utf-8") as fh:
        rdr = csv.DictReader(fh)
        out = {}
        for r in rdr:
            verdict = r["verdict"].strip()
            if not verdict:
                sys.exit(f"{path}: domain {r['domain']} has an empty verdict; annotate every row")
            out[r["domain"].strip()] = verdict
        return out
 
 
# --------------------------------------------------------------------------
# commands
# --------------------------------------------------------------------------
def cmd_sample(args):
    rows = read_labels(args.labels)
    by_cat = defaultdict(list)
    n_unlabelled = 0
    for domain, cat in rows:
        # Unlabelled domains are a coverage result, measured over the whole
        # population in `score`. Asking an annotator to check the precision of
        # "no answer" is not a question, so they are held out of the sample.
        if cat.lower() == UNLABELLED:
            n_unlabelled += 1
            continue
        by_cat[cat].append(domain)
 
    rng = random.Random(args.seed)
    picked = []
    print(f"population: {len(rows)} domains, {n_unlabelled} unlabelled, "
          f"{len(rows) - n_unlabelled} in {len(by_cat)} categories")
    print(f"seed: {args.seed}   target per category: {args.per_category}\n")
    print(f"{'category':<32} {'size':>8} {'sampled':>8}")
    print("-" * 50)
    for cat in sorted(by_cat, key=lambda c: -len(by_cat[c])):
        pool = sorted(by_cat[cat])
        take = min(args.per_category, len(pool))
        chosen = rng.sample(pool, take)
        picked.extend((d, cat) for d in chosen)
        flag = "  <- undersized" if take < args.per_category else ""
        print(f"{cat:<32} {len(pool):>8} {take:>8}{flag}")
    print("-" * 50)
    print(f"{'total':<32} {len(rows):>8} {len(picked):>8}\n")
 
    # Blind: the sheet carries no category, and the order is shuffled so an
    # annotator cannot infer it from adjacency.
    order = list(picked)
    rng.shuffle(order)
    for sheet in ("a", "b"):
        path = f"sheet_{sheet}.csv"
        with open(path, "w", newline="", encoding="utf-8") as fh:
            w = csv.writer(fh)
            w.writerow(["domain", "verdict"])
            for domain, _ in order:
                w.writerow([domain, ""])
        print(f"wrote {path}  ({len(order)} rows, category hidden, order shuffled)")
 
    key = "sample_key.csv"
    with open(key, "w", newline="", encoding="utf-8") as fh:
        w = csv.writer(fh)
        w.writerow(["domain", "category"])
        w.writerows(picked)
    print(f"wrote {key}     (the labels under test -- do not show this to annotators)")
 
    for hw in (0.20, 0.10, 0.05):
        print(
            f"note: +/-{hw:.0%} half-width at 90% precision needs "
            f"{sample_size_for(hw)} domains per category"
        )
 
 
def cmd_score(args):
    truth_labels = dict(read_labels(args.labels))
    a = read_sheet(args.sheet_a)
    b = read_sheet(args.sheet_b)
 
    common = sorted(set(a) & set(b))
    if not common:
        sys.exit("the two sheets share no domains")
    if set(a) != set(b):
        print(f"warning: sheets differ; scoring the {len(common)} shared domains\n")
 
    pairs = [(a[d], b[d]) for d in common]
    kappa = cohen_kappa(pairs)
    n_agree = sum(1 for x, y in pairs if x == y)
 
    print("=" * 66)
    print("INTER-ANNOTATOR AGREEMENT")
    print("=" * 66)
    print(f"domains double-annotated : {len(common)}")
    print(f"raw agreement            : {n_agree}/{len(common)} = {n_agree / len(common):.1%}")
    print(f"Cohen's kappa            : {kappa:.3f}")
    print(
        "Report both. Kappa alone hides how many categories there were, and raw\n"
        "agreement alone is inflated when one category dominates.\n"
    )
 
    # Precision is scored only where the two annotators agree. Where they do not,
    # the ground truth is genuinely unknown and forcing an adjudication would
    # bury the disagreement that is itself the result.
    scored = [d for d in common if a[d] == b[d]]
    disputed = len(common) - len(scored)
 
    per_cat = defaultdict(lambda: [0, 0])  # category -> [correct, n]
    for d in scored:
        service = truth_labels[d]
        per_cat[service][1] += 1
        if a[d] == service:
            per_cat[service][0] += 1
 
    print("=" * 66)
    print("PER-CATEGORY PRECISION")
    print("=" * 66)
    print(f"scored on the {len(scored)} domains the annotators agreed on; "
          f"{disputed} disputed and excluded\n")
    print(f"{'category':<32} {'ok':>4} {'n':>4} {'prec':>7}  {'95% CI':>16}  verdict")
    print("-" * 82)
    keep, drop = [], []
    for cat in sorted(per_cat, key=lambda c: (per_cat[c][0] / per_cat[c][1] if per_cat[c][1] else 0)):
        ok, n = per_cat[cat]
        p, lo, hi = wilson(ok, n)
        # The rule from Ruth et al., IMC 2022: keep a category only if it clears
        # the floor. Applied to the point estimate, with the CI shown so a reader
        # can see how much the sample size is actually supporting.
        verdict = "keep" if p >= args.floor else "DROP"
        (keep if p >= args.floor else drop).append(cat)
        print(f"{cat:<32} {ok:>4} {n:>4} {p:>6.1%}  [{lo:>5.1%}, {hi:>5.1%}]  {verdict}")
    print("-" * 82)
 
    tot_ok = sum(v[0] for v in per_cat.values())
    tot_n = sum(v[1] for v in per_cat.values())
    p, lo, hi = wilson(tot_ok, tot_n)
    print(f"{'ALL (micro-average)':<32} {tot_ok:>4} {tot_n:>4} {p:>6.1%}  [{lo:>5.1%}, {hi:>5.1%}]\n")
 
    # Scoring only where the annotators agreed biases precision UPWARD, because
    # a domain two people find easy to label is also a domain the service finds
    # easy. Report the pessimistic bound alongside it -- every disputed domain
    # counted as a service error -- and let the gap between the two speak.
    lo_ok = tot_ok
    lo_n = tot_n + disputed
    pp, plo, phi = wilson(lo_ok, lo_n)
    print(
        f"{'ALL (disputed counted wrong)':<32} {lo_ok:>4} {lo_n:>4} {pp:>6.1%}  "
        f"[{plo:>5.1%}, {phi:>5.1%}]"
    )
    print(
        "The true precision is between these two rows. If they are far apart your\n"
        "annotators disagreed too much for the number to mean anything -- fix the\n"
        "codebook and re-annotate rather than picking the flattering row.\n"
    )
 
    # Coverage over the whole population, not the sample.
    total = len(truth_labels)
    unlabelled = sum(1 for c in truth_labels.values() if c.lower() == UNLABELLED)
    cov, clo, chi = wilson(total - unlabelled, total)
    print("=" * 66)
    print("COVERAGE (whole population, not the sample)")
    print("=" * 66)
    print(f"{total - unlabelled}/{total} domains got a label = {cov:.1%} [{clo:.1%}, {chi:.1%}]\n")
 
    print("=" * 66)
    print("PASTE THIS INTO YOUR METHODOLOGY SECTION")
    print("=" * 66)
    print(
        f"We categorised {total} domains, of which {total - unlabelled} ({cov:.1%}) received a "
        f"label. To validate the labels we drew a stratified random sample of "
        f"{len(common)} domains and had two authors independently assign a category "
        f"while blind to the service's answer (Cohen's kappa = {kappa:.2f}, raw agreement "
        f"{n_agree / len(common):.0%}). On the {len(scored)} domains where the annotators "
        f"agreed, the service's micro-averaged precision was {p:.1%} "
        f"(95% Wilson CI [{lo:.1%}, {hi:.1%}]); counting every disputed domain as an error "
        f"gives a lower bound of {pp:.1%}. "
        + (
            f"We discarded the {len(drop)} categor{'y' if len(drop) == 1 else 'ies'} that fell "
            f"below {args.floor:.0%} precision ({', '.join(drop)}) and report results over the "
            f"remaining {len(keep)}."
            if drop
            else f"All {len(keep)} categories cleared our {args.floor:.0%} precision floor."
        )
    )
 
 
def main():
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    sub = ap.add_subparsers(dest="cmd", required=True)
 
    s = sub.add_parser("sample", help="draw a stratified sample and write blind annotation sheets")
    s.add_argument("labels")
    s.add_argument("--per-category", type=int, default=25)
    s.add_argument("--seed", type=int, required=True, help="record this in the paper")
    s.set_defaults(func=cmd_sample)
 
    c = sub.add_parser("score", help="score two completed annotation sheets")
    c.add_argument("labels")
    c.add_argument("sheet_a")
    c.add_argument("sheet_b")
    c.add_argument("--floor", type=float, default=0.80, help="per-category precision floor")
    c.set_defaults(func=cmd_score)
 
    args = ap.parse_args()
    args.func(args)
 
 
if __name__ == "__main__":
    main()

A worked example on synthetic data, where one category (“Search Engines”) was deliberately built to be only 45% correct — the category [5Ruth, Kimberly; Fass, Aurore; Azose, Jonathan; Pearson, Mark; Thomas, Emma; Sadowski, Caitlin; Durumeric, Zakir (2022): "A World Wide View of Browsing the World Wide Web", in: Proceedings of the ACM Internet Measurement Conference, pp. 317-336. (DOI)] found unreliable in Cloudflare's taxonomy. Real output, reproducible with the demo script below:

population: 1100 domains, 220 unlabelled, 880 in 4 categories
seed: 20260807   target per category: 25

category                             size  sampled
--------------------------------------------------
News                                  400       25
Shopping                              300       25
Adult                                 120       25
Search Engines                         60       25
--------------------------------------------------
total                                1100      100

note: +/-20% half-width at 90% precision needs 9 domains per category
note: +/-10% half-width at 90% precision needs 35 domains per category
note: +/-5% half-width at 90% precision needs 139 domains per category

==================================================================
INTER-ANNOTATOR AGREEMENT
==================================================================
domains double-annotated : 100
raw agreement            : 67/100 = 67.0%
Cohen's kappa            : 0.564

==================================================================
PER-CATEGORY PRECISION
==================================================================
scored on the 67 domains the annotators agreed on; 33 disputed and excluded

category                           ok    n    prec            95% CI  verdict
----------------------------------------------------------------------------------
Search Engines                      5    9  55.6%  [26.7%, 81.1%]  DROP
News                               19   19 100.0%  [83.2%, 100.0%]  keep
Shopping                           16   16 100.0%  [80.6%, 100.0%]  keep
Adult                              23   23 100.0%  [85.7%, 100.0%]  keep
----------------------------------------------------------------------------------
ALL (micro-average)                63   67  94.0%  [85.6%, 97.7%]

ALL (disputed counted wrong)       63  100  63.0%  [53.2%, 71.8%]

==================================================================
COVERAGE (whole population, not the sample)
==================================================================
880/1100 domains got a label = 80.0% [77.5%, 82.3%]

Three things to read out of that:

1. It found the bad category. Search Engines came back at 55.6% and got dropped, exactly as intended — and it did so from 9 usable annotations, which is why per-category stratification matters more than total sample size.
2. Kappa 0.56 on a 5-way label set is realistic, not a failure. Vallina et al.'s own labellers disagreed on 35.5% of domains. If yours agree on 95% you have probably let them see each other's sheets.
3. The two “ALL” rows disagree by 31 points, and that gap is the honest measure of how much the annotation is carrying. Report both. Reporting only the agreed-subset row is how a 63%-precision pipeline gets published as 94%.

category_validation_demo.py

#!/usr/bin/env python3
"""Reproduce the worked example shown on design:website_classification.
 
Builds a synthetic 1,100-domain labelling with one deliberately bad category
("Search Engines", 45% correct -- the category Ruth et al. found unreliable in
Cloudflare's taxonomy), runs `category_validation.py sample`, simulates two
annotators, and runs `score`. Stdlib only.
 
    python3 category_validation_demo.py
"""
 
import csv
import pathlib
import random
import subprocess
import sys
import tempfile
 
HERE = pathlib.Path(__file__).resolve().parent
TOOL = HERE / "category_validation.py"
SEED = 20260807
 
# category -> (population size, share of labels that are actually correct)
TRUTH = {
    "News": (400, 0.92),
    "Shopping": (300, 0.88),
    "Adult": (120, 0.96),
    "Search Engines": (60, 0.45),
    "unlabelled": (220, 1.0),
}
VERDICTS = ["News", "Shopping", "Adult", "Search Engines", "none"]
# per-annotator probability of slipping to a different verdict
ANNOTATOR_NOISE = {"sheet_a.csv": 0.02, "sheet_b.csv": 0.05}
 
 
def run(work):
    rng = random.Random(SEED)
 
    rows, i = [], 0
    for cat, (n, _) in TRUTH.items():
        for _ in range(n):
            i += 1
            rows.append((f"site{i:04d}.example", cat))
    with open(work / "labels.csv", "w", newline="", encoding="utf-8") as fh:
        w = csv.writer(fh)
        w.writerow(["domain", "category"])
        w.writerows(rows)
 
    subprocess.run(
        [sys.executable, str(TOOL), "sample", "labels.csv",
         "--per-category", "25", "--seed", str(SEED)],
        cwd=work, check=True,
    )
 
    key = {r["domain"]: r["category"]
           for r in csv.DictReader(open(work / "sample_key.csv", encoding="utf-8"))}
    rng = random.Random(11)
    for sheet, noise in ANNOTATOR_NOISE.items():
        annotated = list(csv.DictReader(open(work / sheet, encoding="utf-8")))
        for r in annotated:
            true = key[r["domain"]]
            v = true if rng.random() < TRUTH[true][1] else rng.choice(
                [c for c in VERDICTS if c != true])
            if rng.random() < noise:
                v = rng.choice([c for c in VERDICTS if c != v])
            r["verdict"] = v
        with open(work / sheet, "w", newline="", encoding="utf-8") as fh:
            w = csv.DictWriter(fh, ["domain", "verdict"])
            w.writeheader()
            w.writerows(annotated)
 
    print()
    subprocess.run(
        [sys.executable, str(TOOL), "score", "labels.csv",
         "sheet_a.csv", "sheet_b.csv", "--floor", "0.80"],
        cwd=work, check=True,
    )
 
 
if __name__ == "__main__":
    with tempfile.TemporaryDirectory() as tmp:
        run(pathlib.Path(tmp))

A checklist, derived from what the 247 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply.

1. Which service or model, and which version. “McAfee” identified a different company before and after February 2024. “Homepage2Vec” without a commit hash is not reproducible; “GPT-4” without a dated snapshot is not either.
2. The date you queried. These are live services whose labels change without notice.
3. Coverage on your population, as a fraction, before any filtering. How many of your domains got no label — and what you did with them. Dropping them silently changes the denominator of every downstream percentage.
4. The taxonomy and its size. Name it, give the number of classes, and say at which tier you pruned. If it is bespoke, say so and publish the codebook.
5. Whether labels are multi-valued, and how you collapsed them if they are.
6. Validation: sample size, how it was stratified, how many annotators, whether they were blind to the service's label, the agreement statistic, and per-category precision — not one global accuracy number. Publish the annotated sample as an artefact; it is small, and it is the only part of this a reader can check.
7. The categories you dropped, and the rule you dropped them by.
8. If you used more than one service, the disagreement between them. Only 9 of 122 corpus papers that name a service name two, so this is nearly free novelty.

Two sources here. The historical survey is Vallina et al. [1Vallina, Pelayo; Le Pochat, Victor; Feal, ´Alvaro; Paraschiv, Marius; Gamba, Julien; Burke, Tim; Hohlfeld, Oliver; Tapiador, Juan; Vallina-Rodriguez, Narseo (2020): "Mis-shapes, Mistakes, Misfits: An Analysis of Domain Classification Services", in: Proceedings of the ACM Internet Measurement Conference, pp. 598–618. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)], whose figures on service popularity are reproduced below. The rest is a fresh query over a structured extraction of 4,322 papers from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2024 — so every figure is a claim about those seven venues over that window, not about the field.

The extraction marks 320 papers as classifying something with the target website-category. That number is not the population this page is about, and the difference is instructive: the same label covers several unrelated tasks.

Website fingerprinting is a traffic-analysis attack that shares no method, taxonomy or ground truth with topic categorisation; it is here because “classify a website” is the same English phrase. Everything below uses the 247, which is 5.7% of the corpus. The other 73 papers are counted, named, and excluded — see Methodology and limitations of these figures.

TheWebConf, IMC and PETS each devote about 8% of their programme to it; IEEE S&P essentially never does. The overall rate has been flat-to-falling: 5.1% of papers in 2010–2013, peaking at 7.5% in 2014–2017, back to 4.6% in 2022–2024.

Multi-valued, so shares exceed 100%. Reliance on third-party services has grown steadily — 29.2% of categorising papers in 2010–2013, 35.8%, 44.4%, and 48.8% in 2022–2024 — while use of curated databases (DMOZ and friends) halved over the same period, from 16.7% to 8.8%. The field has moved from open, inspectable label sources toward closed, unauditable ones, over exactly the period in which the only audit of those sources found them wanting.

Free-text service names agree run-to-run on only about 20% of exact strings, so these are folded into canonical families and counted by paper, never by mention. The “spellings” column is why this matters.

122 of the 247 papers (49.4%) name at least one third-party service or dataset. McAfee alone appears under fifteen different names — from bare “McAfee” to “McAfee's URL Ticketing System” to “McAfee SmartFilter Internet Database” — and Symantec under ten. Any count of these services built on exact string matching will be wrong by a factor of several, which is the same failure that has bitten geolocation and crawler-tool tallies on other pages of this site.

Everything that is not a named service, ranked: unnamed custom methods (38 papers), manual labelling by the authors or hired coders (32), custom keyword and regex rules (25), random forests (10), SVMs (9), clustering (8), then a long tail. Those first and third rows are one family: 61 papers (24.7%) classify websites with rules they wrote themselves, and 38 of them do not name the rules at all.

Of those that do validate: 70 papers by manual validation, 18 by cross-validation, 9 by a held-out test set, 9 by comparison to another method. Note the shape — manual spot-checking is the dominant and correct instrument here, because a held-out test set only tells you about the labels you already had.

The 50.2% figure is worth dwelling on: for website classification, the share naming a ground-truth source is well below the 71.7% the same extraction measures across all 3,241 papers that classify anything. Website categorisation is reported worse than classification in general, not better.

Of the 184 papers that name a taxonomy at all:

Nearly three quarters invent a scheme and do not say how many classes it has. That single omission makes a result impossible to compare against any other paper, and it is free to fix.

Twenty-nine papers across the corpus's fifteen years, twenty-seven of them in the final one — the curve is visibly starting, and the corpus ends before it goes anywhere. They point the model at “other” targets (19 papers), user-generated text (6), vulnerabilities (4), privacy policies (1) and domains (1). Not one of the 29 uses an LLM to assign a website a topic category.

They do, however, establish a norm worth copying: 24 of 29 (82.8%) validate the model's labels against something, usually manual annotation of a sample. The single website-category LLM tuple in the whole corpus is a USENIX Security 2024 phishing detector using GPT-3.5-turbo-16k with a chain-of-thought prompt for a binary credential-page question, validated against three cybersecurity experts' annotations — a security verdict, not a topic.

See LLM Classification for what has happened since the corpus ends.

• [5Ruth, Kimberly; Fass, Aurore; Azose, Jonathan; Pearson, Mark; Thomas, Emma; Sadowski, Caitlin; Durumeric, Zakir (2022): "A World Wide View of Browsing the World Wide Web", in: Proceedings of the ACM Internet Measurement Conference, pp. 317-336. (DOI)] (IMC 2022) is the best worked example of using a categorisation API responsibly that the corpus contains: state the taxonomy sizes, validate ten random sites per category, keep only categories above 80% accuracy, report what you dropped and why. It is one paragraph of a large paper and it is the paragraph to imitate.
• [1Vallina, Pelayo; Le Pochat, Victor; Feal, ´Alvaro; Paraschiv, Marius; Gamba, Julien; Burke, Tim; Hohlfeld, Oliver; Tapiador, Juan; Vallina-Rodriguez, Narseo (2020): "Mis-shapes, Mistakes, Misfits: An Analysis of Domain Classification Services", in: Proceedings of the ACM Internet Measurement Conference, pp. 598–618. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] (IMC 2020) is the audit itself. Read §4.2 (coverage), §4.3 (documented versus observed taxonomies) and §6 (the case studies) before you commit to a service.

• Denominators. Every figure names its own. The corpus is 4,322 papers; 3,241 classify something; 320 carry a website-category target; 247 assign a topic, industry or sector label and are this section's population. Never “of 4,322 papers”.
• Papers, not mentions. A paper naming McAfee three times counts once.
• Sentinels are not answers. not-stated and none-mentioned are excluded from every numerator and reported as their own row where they matter — which they do, since “does not validate” is the headline finding.
• Free-text names are folded into canonical families by an ordered regex list before counting, because exact strings agree run-to-run only about 20% of the time. The fold left 21 distinct strings unclassified, each appearing once: among them Taobao main-business categories, SNET pillar portals, Rakuten Popshops, multiple domain classification services and public classification service from the security vendor. The last two are papers that used a service and declined to say which — a small, exact instance of the reporting gap this section measures.
• The task fold is imperfect and its exclusions are visible. Eight side-channel and website-fingerprinting papers named their classifier only as “a 1D CNN” or “k-NN”, leaving no signal in any extracted field; they were caught by matching the paper's own title and are excluded by name rather than by rule. The 73 excluded papers are counted in the table above rather than dropped.
• Enum fields versus free text. Method and validation are enums, stable enough to publish as rough shares (classification.method agrees 58% run-to-run, so read those as a ranking). Service names and taxonomies are free text and are reported as rankings and folded families only.
• Quotes were checked. Every figure above traces to tuples carrying a verbatim evidence quote; a sample of these was re-located in the source PDFs. Of six quotes checked by hand, two initially “failed” a literal grep and turned out to be intact but split across a two-column break — normalise whitespace before concluding that a quote is not in the paper.
• Coverage. IEEE S&P is only 43% retrieved, which is the most likely explanation for its 1.2% share. EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS are absent entirely, and so is anything published after 2024. Notably, [1Vallina, Pelayo; Le Pochat, Victor; Feal, ´Alvaro; Paraschiv, Marius; Gamba, Julien; Burke, Tim; Hohlfeld, Oliver; Tapiador, Juan; Vallina-Rodriguez, Narseo (2020): "Mis-shapes, Mistakes, Misfits: An Analysis of Domain Classification Services", in: Proceedings of the ACM Internet Measurement Conference, pp. 598–618. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] itself is in the venue index but has no extracted full text — the reference work for this page is not in the population the page measures.

Genuinely unresolved, and each is a paper:

1. Redo Vallina et al. Thirteen services, millions of domains, 2026 ownership. Every vendor has changed hands; the only modern data point suggests the free tiers have collapsed and quality has not improved. This is the obvious one.
2. Benchmark open models against commercial services on a shared, public ground truth. Homepage2Vec versus gpt-oss has been done on 3,785 sites; neither has been compared against FortiGuard or McAfee at all.
3. Measure LLM label stability across seeds, temperature and model versions for website classification. Everyone currently sets temperature to 0 and hopes.
4. Measure what any of these do with parked, dead and holding-page domains — a large share of any long-tail sample, and the case where “read the homepage” reads a registrar's advert.
5. Publish a formal agreement statistic between LLM labels and expert human annotators for this task. Nobody has, and the script above computes one.
6. A crosswalk between vendor taxonomies. Vallina et al. named this as future work in 2020 and pointed at AVClass as the model. It still does not exist, and it is what would make combining two services something other than a mess.

• Website Selection — which domains to study, and popularity rankings.
• IP Classification — the same problem for addresses instead of names, with the same “which question are you actually asking” structure.
• Requests — tracker and advertiser classification, where filter lists beat categorisers.
• Cookies, Fingerprinting, JavaScript — classification of the other things a crawl collects.
• SimilarWeb — API documentation, code and output example for the one service on this page that has a page of its own.
• Sampling, Biases (both not yet written) — drawing a sample you can generalise from, and what a categoriser's uneven coverage does to a downstream estimate.
• Tranco, Cloudflare Radar (both not yet written) — API documentation for two services this page references.

[1]

Vallina, Pelayo; Le Pochat, Victor; Feal, ´Alvaro; Paraschiv, Marius; Gamba, Julien; Burke, Tim; Hohlfeld, Oliver; Tapiador, Juan; Vallina-Rodriguez, Narseo (2020): "Mis-shapes, Mistakes, Misfits: An Analysis of Domain Classification Services", in: Proceedings of the ACM Internet Measurement Conference, pp. 598–618. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)

[2]

Bozzolan, Simone; Calzavara, Stefano; Cazzaro, Lorenzo (2026): "LLM-Assisted Web Measurements". arXiv:2510.08101, v3, 30 April 2026 (Link)

[3]

Lugeon, Sylvain; Piccardi, Tiziano; West, Robert (2022): "Homepage2Vec: Language-Agnostic Website Embedding and Classification", in: Proceedings of the International AAAI Conference on Web and Social Media, pp. 1285-1291. (DOI)

[4]

Ruth, Kimberly; Kumar, Deepak; Wang, Brandon; Valenta, Luke; Durumeric, Zakir (2022): "Toppling Top Lists: Evaluating the Accuracy of Popular Website Lists", in: Proceedings of the 22nd ACM Internet Measurement Conference, pp. 374–387. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)

[5]

Ruth, Kimberly; Fass, Aurore; Azose, Jonathan; Pearson, Mark; Thomas, Emma; Sadowski, Caitlin; Durumeric, Zakir (2022): "A World Wide View of Browsing the World Wide Web", in: Proceedings of the ACM Internet Measurement Conference, pp. 317-336. (DOI)

[6]

Jha, Nikhil; Trevisan, Martino; Leonardi, Emilio; Mellia, Marco (2023): "On the Robustness of Topics API to a Re-Identification Attack", Proceedings on Privacy Enhancing Technologies 2023(4):66-78. (DOI)

[7]

Ye, Eric; Bai, Xiao; O'Hare, Neil; Asgarieh, Eliyar; Thadani, Kapil; Perez-Sorrosal, Francisco; Adiga, Sujyothi (2024): "Multilingual Taxonomic Web Page Categorization Through Ensemble Knowledge Distillation", IEEE Transactions on Knowledge and Data Engineering 36(11):6614-6627. (DOI)