Differences

This shows you the differences between two versions of the page.

--- design:website_selection [2025/02/19 10:13] – highlighted todos karelkubicek
+++ design:website_selection [2026/08/27 14:39] (current) – Rewrite toward IP-classification shape: decision table, live rank comparison, 2026 status table, quotes, open questions. Kept human catalogue. Authored by Claude karel.kubicek.claude
@@ Line 1: / Line 1: @@
-===== Overview =====
+====== Website selection ======
-Selecting a representative sample of websites is a crucial step in web measurement studies. The choice of website lists can significantly affect research outcomes, making it essential to understand their strengths, limitations, and best practices.
-This page examines various top website lists, outlines best practices for their usage, and provides detailed documentation for each service, including API access, via linked pages. Note that this page overlaps with [[Design:Website Classification]], as popularity is among the most important categories used in research. This page focuses more in detail on the services that provide ranking as primary data, while [[Design:Website Classification]] focuses more on categories such as website industry or company data.
+There is no register of the web, so every measurement substitutes a **list** for the population it wants to talk about. This page is about **which list**: what each ranking actually measures, what it is biased towards, and which ones the field still uses. [[Design:Sampling]] is about **how you draw** from a list you have already chosen — top-//n// versus stratified, sample size, unit, versioning. Read them together. A reader can pick CrUX, know its biases, and still produce an unreproducible top-1,000 crawl with no list id.
-===== Popular Website Lists =====
+This page overlaps with [[Design:Website classification]] only in that popularity is one label people attach to sites. Classification is about topic, industry and company data. Rankings are the sampling frame.
-A variety of top website lists are commonly utilized in research. Below is an overview of the most widely referenced options:
-==== 1. CrUX (Chrome User Experience Report) ====
+The single most useful thing to internalise is that **"a top-sites list" is not one instrument.** The vendors measure different things, they disagree on the head, and a Tranco row is a mixture of those instruments rather than a ground truth:
-[[https://developer.chrome.com/docs/crux|CrUX]] provides rank order magnitude buckets (e.g., 1k, 5k, 10k, ..., 5M) of popular websites based on user-initiated page loads observed by Google Chrome users with the "Help improve Chrome's features and performance" setting enabled.
-  * **Advantages**: The most representative rankings according to {[ruth2022_toppling]}; derived from real user browsing data; includes rich country- and device-specific insights.
+^ If your claim is about ^ Use ^ Do not use as if it measured that ^
-  * **Limitations**: Rankings are aggregated into broad rank buckets (e.g., Top 1K, 10K).
+| Pages people actually load | **CrUX** (origins, monthly buckets) | Umbrella, Radar, Farsight, Majestic. Tranco is not a substitute: CrUX is one of five inputs |
-  * **More details, API**: [[Programming:CrUX]]
+| DNS / infrastructure / non-browser clients | Umbrella, Radar, SecRank, Farsight-in-Tranco | CrUX |
+| Link structure | Majestic | any traffic or DNS list |
+| "The web" | None of these. Say which frame you had. | A top-//n// described as a sample of the web |
-==== 2. Tranco ====
+The short version, from 1,153 papers in our corpus of seven security and privacy venues that drew a population of websites, domains or web pages (see [[#Use in Publications]]):
-[[https://tranco-list.eu/|Tranco]] is a research-oriented, hardened top sites ranking designed to enhance stability and resilience against manipulation. It combines data from Alexa, Cisco Umbrella, and Majestic over a 30-day period. It was introduced in NDSS 2019 {[LePochat2019_tranco]}.
-  * **Advantages**: Resilient to popularity manipulation; research-oriented with robust versioning; configurable upon registration.
+<WRAP important>
-  * **Limitations**: Dependent on the quality of its constituent lists; subject to change due to the discontinuation of older services.
+**Pick the list that matches the claim; then pin it.** Of 133 papers published in 2025 that sampled the web, **75 (56.4%) named Tranco** and **12 (9.0%) named CrUX** — Tranco is current practice, not a page-load ranking. Ruth et al. measured Alexa, Majestic, Umbrella, Tranco and CrUX against Cloudflare HTTP logs in **February 2022** and found **CrUX the most accurate across all metrics** {[ruth2022_toppling]}. That is 2022 evidence, not a re-run on the five-provider Tranco list you download now. Alexa.com was retired on **1 May 2022** (APIs 15 December 2022); **66 papers from 2023–2026 still name it**, 34 of them with no version at all. Do not write "the Tranco top 1M" without a list id: the default provider set changed on 1 August 2023. Pin the id with [[Programming:Tranco]]'s ''pin_tranco.py''.
-  * **More details, API**: [[Programming:Tranco]]
+</WRAP>
-==== 3. Cloudflare Radar ====
+If you read three papers before starting: {[scheitle2018_long]} (IMC 2018) on overlap, churn and head-vs-zone bias; {[LePochat2019_tranco]} (NDSS 2019) on manipulation and why Tranco exists; {[ruth2022_toppling]} (IMC 2022) on which list matched HTTP logs. {[galloway2024_practical]} (IEEE S&P 2024) is the current attack paper, and it is about Radar and Tranco-via-Radar, not about CrUX.
-[[https://radar.cloudflare.com/domains|Cloudflare Radar]] provides rankings based on DNS query data and traffic on Cloudflare-operated websites.
-  * **Advantages**: <wrap todo>TODO</wrap>
+===== The lists are different instruments =====
-  * **Limitations**: <wrap todo>TODO</wrap>
-  * **More details, API**: [[Programming:Cloudflare_Radar]]
-==== 4. Cisco Umbrella ====
+Each list ranks an **observable** that people then treat as popularity. The observable is the whole design:
-[[https://umbrella-static.s3-us-west-1.amazonaws.com/index.html|Cisco Umbrella]] ranks domains based on DNS query traffic to its resolvers.
-  * **Advantages**: Captures non-browser-based traffic.
+  * **Page loads** (CrUX): user-initiated navigations from eligible Chrome installs. Closest to "sites people visit".
-  * **Limitations**: DNS-based methodology can introduce biases (see [[#Limitations of DNS-Based Lists]]); may include invalid domains (e.g., typos, internal domains).
+  * **DNS queries** (Umbrella, Radar, Farsight, SecRank): names resolvers looked up. Includes software updates, telemetry, CDNs and names no human opened.
+  * **Backlinks** (Majestic): the link graph. Independent of visits; cheap to manipulate relative to a panel {[LePochat2019_tranco]}.
+  * **An aggregate** (Tranco): Dowdall over a month of several of the above. More stable than a single daily source; not a fourth kind of popularity, and not CrUX.
-==== 5. Majestic Million ====
+Tranco's daily list of 2026-08-26 (id ''46W9X'') mixes CrUX with three DNS rankings and Majestic. Treating that id as "the CrUX top 1M" is a methods error; treating it as "websites" is the same error one layer down. The script below looks the same names up on that Tranco list, Umbrella and Majestic (fetched 2026-08-27T14:26:07Z):
-Majestic ranks domains based on backlinks.
-  * **Advantages**: Valuable for analyzing network link structures.
+^ Name ^ Tranco (PLD, 46W9X) ^ Cisco Umbrella (hostname) ^ Majestic Million (backlinks) ^
-  * **Limitations**: Does not strictly reflect actual user traffic.
+| ''google.com'' | 1 | 1 | 1 |
+| ''facebook.com'' | 4 | 44 | 2 |
+| ''wikipedia.org'' | 31 | 2,219 | 11 |
+| ''baidu.com'' | 81 | 2,115 | 68 |
+| ''nytimes.com'' | 158 | 4,093 | 67 |
+| ''microsoft.com'' | 5 | 4 | 9 |
+| ''gstatic.com'' | 3 | 2 | 1,232 |
+| ''windowsupdate.com'' | 60 | 30 | 23,840 |
+| ''windowsupdate.microsoft.com'' | absent | 79,026 | absent |
+| ''googleusercontent.com'' | 35 | 26 | 178 |
-==== 6. Other ====
+<WRAP important>
-  * ''SimilarWeb'': Offers high-quality data but is paywalled (some exceptions exist). See [[Programming:SimilarWeb]].
+**''gstatic.com'' is rank 3 on Tranco list ''46W9X'' (2026-08-26).** It is rank 2 on Umbrella and rank 1,232 on Majestic. A static-content hostname sits in the head of the list the field treats as "top websites" because Tranco's default inputs include DNS rankings. ''windowsupdate.com'' is rank 60 on Tranco and rank 23,840 on Majestic. ''windowsupdate.microsoft.com'' exists only on Umbrella — Tranco and Majestic are pay-level domains. Wikipedia is rank 31 on Tranco and rank 2,219 on Umbrella. These are not permutations of one ordering.
-  * ''Quantcast'': Primarily focused on US traffic.
+</WRAP>
-  * ''[[https://en.wikipedia.org/wiki/Alexa_Internet|Alexa]]'': Discontinued as of August 1, 2023; previously widely used in research. Rankings were based on page visits from a user panel and tracking scripts, making them more reliable than DNS-based lists but highly volatile.
-  * ''[[https://www.domaintools.com/resources/blog/mirror-mirror-on-the-wall-whos-the-fairest-website-of-them-all|Farsight]]'': <wrap todo>TODO</wrap>
-  * ''[[https://secrank.cn/|SecRank]]'': List based on Chinese DNS data, introduced in USENIX Security 2022 {[xie2022_building]}.
-===== Best Practices =====
+Umbrella still publishes a **separate** TLD file (''top-1m-TLD.csv.zip'', 10,905 rows on 2026-08-27: ''com'' = 1, ''net'' = 2, ''org'' = 6). Bare TLDs are **not** in the 2026-08-27 top-1m.csv — an earlier version of this page described the popularity list as beginning ''1,com'' / ''2,net''. That is the TLD file, not the list you crawl.
-Based on recent studies {[LePochat2019_tranco,ruth2022_toppling]}, the following recommendations can enhance the representativeness and reliability of website selection:
-  - **Use Aggregated Data**: Majority of research does not require the popularity indices of individual websites, but rather aggregated ranks (e.g., top 10k). This reduces [[#Temporal Stability and Manipulations]] limitation.
+==== A script ====
-  - **Understand Use-Cases and Biases**: Use a list that is collected by methods similar to your target study audience. E.g., if you study mobile websites, use CrUX collected from mobile users, while if you study internet requests in China, use Chinese-DNS-based SecRank. Be aware of biases inherent in DNS-based lists (e.g., Cisco Umbrella) and traffic-based lists (e.g., SimilarWeb).
-  - **Document and Archive**: Record the version of the list and the date of access to ensure reproducibility. Publish the exact list in the publication's [[Research_practices|artefact]].
-  - **Avoid Single Sources**: If possible, report your results according to multiple lists, such as Tranco and CrUX, for better representativeness.
-===== Detailed Analysis =====
+Looks the same names up on the three lists you can fetch without an account. Stdlib only. Rerun it on the day you freeze a sample; the table above is the latest Tranco daily list as of the fetch (**dated 2026-08-26, fetched 2026-08-27T14:26:07Z**), plus that day's Umbrella zip and Majestic CSV.
-==== Limitations of DNS-Based Lists ====
-DNS-based lists, such as **Cisco Umbrella**, **Farsight**, **SecRank**, and partially **Cloudflare Radar** and **Tranco**, rank domains based on DNS query volumes. While this approach provides insights into overall domain popularity, it introduces several limitations:
-DNS lists capture queries from devices and applications that automatically resolve domain names without user interaction. This can lead to overrepresentation of domains related to software updates, network configuration, or telemetry, rather than user-driven web browsing. For example:
+<file python compare_ranks.py>
-  * ''windowsupdate.microsoft.com'' is frequently queried for updates but does not represent typical user-visited websites.
+#!/usr/bin/env python3
-  * Internal or infrastructure-related domains, such as ''ec2.internal'', may also appear on these lists.
+"""Look the same names up on today's Tranco daily list, Cisco Umbrella, and
+Majestic Million, and print the ranks side by side.
-==== Temporal Stability and Manipulations ====
+The point is not the ranks. It is that they disagree, and that Umbrella ranks
+TLDs and infrastructure hostnames a page-load list does not contain.
-**Alexa**, a historically prominent ranking source, relied on data from users who installed its browser toolbar. This methodology presented two primary issues:
+    python3 compare_ranks.py
-  * **Manipulations**: Adversaries could artificially inflate website popularity by automating visits with the installed toolbar. Given economic interest for ranking high, there was real-world incentive for such manipulation.
+    python3 compare_ranks.py --wiki
-  * **Stability**: Due to the high number of websites, a limited user base of the toolbar, and daily updates, the published lists exhibited significant volatility.
-Tranco was designed to reduce the impact of these issues by aggregating diverse ranking sources over a 30-day period. Similarly, CrUX, which organizes rankings into broader buckets, offers greater stability. While CrUX is theoretically susceptible to manipulation, the vast user base of Chrome (billions of users sharing browsing data) and the high cost of such an attack provide significant resistance.
+Stdlib only. Caches the bulk CSVs under /tmp/measuretheweb-compare-ranks.
+Tranco is a ranks API (1 query/second). Dated in the output; rerun before you
+cite the table.
+"""
+from __future__ import annotations
+import argparse
+import csv
+import hashlib
+import io
+import json
+import sys
+import time
+import urllib.request
+import zipfile
+from datetime import datetime, timezone
+from pathlib import Path
+UA = "measuretheweb-compare-ranks/1.0"
+CACHE = Path("/tmp/measuretheweb-compare-ranks")
+CACHE.mkdir(parents=True, exist_ok=True)
+TRANCO_RANKS = "https://tranco-list.eu/api/ranks/domain/{domain}"
+TRANCO_LATEST = "https://tranco-list.eu/api/lists/date/latest"
+UMBRELLA_ZIP = "https://s3-us-west-1.amazonaws.com/umbrella-static/top-1m.csv.zip"
+UMBRELLA_TLD_ZIP = "https://s3-us-west-1.amazonaws.com/umbrella-static/top-1m-TLD.csv.zip"
+MAJESTIC_CSV = "https://downloads.majestic.com/majestic_million.csv"
+# Names chosen to hit the disagreements the rest of the page argues: a page
+# people visit, a news site, a Chinese head, DNS/update infrastructure, and a
+# hostname vs its registrable domain. Bare TLDs are a separate Umbrella file
+# (top-1m-TLD.csv.zip); they are not in the top-1m this script loads.
+NAMES = [
+    "google.com",
+    "facebook.com",
+    "wikipedia.org",
+    "baidu.com",
+    "nytimes.com",
+    "microsoft.com",
+    "gstatic.com",
+    "windowsupdate.com",
+    "windowsupdate.microsoft.com",
+    "googleusercontent.com",
+]
+def fetch(url: str) -> bytes:
+    req = urllib.request.Request(url, headers={"User-Agent": UA})
+    with urllib.request.urlopen(req, timeout=180) as resp:
+        return resp.read()
+def cached(url: str, name: str) -> Path:
+    path = CACHE / name
+    if path.exists() and path.stat().st_size > 0:
+        return path
+    path.write_bytes(fetch(url))
+    return path
+def load_rank_csv(path: Path, domain_col: int, rank_col: int, skip_header: bool) -> dict[str, int]:
+    raw = path.read_bytes()
+    if raw[:2] == b"PK":
+        with zipfile.ZipFile(io.BytesIO(raw)) as zf:
+            text = zf.read(zf.namelist()[0]).decode()
+    else:
+        text = raw.decode(errors="replace")
+    out: dict[str, int] = {}
+    reader = csv.reader(io.StringIO(text))
+    if skip_header:
+        next(reader)
+    for row in reader:
+        if len(row) <= max(domain_col, rank_col):
+            raise RuntimeError(f"{path}: short row {row!r}")
+        domain = row[domain_col].strip().lower()
+        rank = int(row[rank_col])
+        if domain in out:
+            continue
+        out[domain] = rank
+    if len(out) < 1000:
+        raise RuntimeError(f"{path}: only {len(out)} rows — file is not a top list")
+    return out
+def tranco_today(domain: str) -> int | None:
+    raw = fetch(TRANCO_RANKS.format(domain=domain))
+    body = json.loads(raw.decode())
+    ranks = body["ranks"]
+    if len(ranks) == 0:
+        return None
+    rank = ranks[0]["rank"]
+    if rank is None:
+        return None
+    return int(rank)
+def fmt(rank: int | None) -> str:
+    if rank is None:
+        return "absent"
+    return f"{rank:,}"
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--wiki", action="store_true", help="DokuWiki table instead of TSV")
+    args = ap.parse_args()
+    meta = json.loads(fetch(TRANCO_LATEST).decode())
+    list_id = meta["list_id"]
+    cfg = meta["configuration"]
+    fetched_at = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+    umbrella_path = cached(UMBRELLA_ZIP, "umbrella_top1m.csv.zip")
+    majestic_path = cached(MAJESTIC_CSV, "majestic_million.csv")
+    umbrella = load_rank_csv(umbrella_path, domain_col=1, rank_col=0, skip_header=False)
+    majestic = load_rank_csv(majestic_path, domain_col=2, rank_col=0, skip_header=True)
+    rows: list[tuple[str, int | None, int | None, int | None]] = []
+    for i, name in enumerate(NAMES):
+        if i:
+            time.sleep(1.05)
+        t = tranco_today(name)
+        u = umbrella[name] if name in umbrella else None
+        m = majestic[name] if name in majestic else None
+        rows.append((name, t, u, m))
+    if rows[0][0] != "google.com" or rows[0][1] is None:
+        raise RuntimeError("google.com missing from Tranco — the ranks API changed")
+    if "gstatic.com" not in umbrella:
+        raise RuntimeError("gstatic.com missing from Umbrella — the zip changed shape")
+    tld_path = cached(UMBRELLA_TLD_ZIP, "umbrella_top1m_tld.csv.zip")
+    tld = load_rank_csv(tld_path, domain_col=1, rank_col=0, skip_header=False)
+    if "com" not in tld:
+        raise RuntimeError("TLD 'com' missing from Umbrella top-1m-TLD.csv.zip")
+    print(f"fetched_at {fetched_at}")
+    print(f"tranco {list_id} providers {','.join(cfg['providers'])} {cfg['combinationMethod']}")
+    print(f"umbrella sha256 {hashlib.sha256(umbrella_path.read_bytes()).hexdigest()[:12]} n={len(umbrella)}")
+    print(f"majestic sha256 {hashlib.sha256(majestic_path.read_bytes()).hexdigest()[:12]} n={len(majestic)}")
+    print(
+        "umbrella TLD file: "
+        f"com={tld['com']} net={tld['net']} org={tld['org']} n={len(tld)}"
+    )
+    print()
+    if args.wiki:
+        print("^ Name ^ Tranco (PLD, " + list_id + ") ^ Cisco Umbrella (hostname) ^ Majestic Million (backlinks) ^")
+        for name, t, u, m in rows:
+            print(f"| ''{name}'' | {fmt(t)} | {fmt(u)} | {fmt(m)} |")
+    else:
+        print("name\ttranco\tumbrella\tmajestic")
+        for name, t, u, m in rows:
+            print(f"{name}\t{fmt(t)}\t{fmt(u)}\t{fmt(m)}")
+    return 0
+if __name__ == "__main__":
+    sys.exit(main())
+</file>
+To pin a Tranco id rather than compare ranks, use ''python3 pin_tranco.py'' on [[Programming:Tranco]].
+===== Which list in 2026 =====
+Checked against each vendor's own documentation and APIs on **2026-08-27**. Dated deliberately.
+^ Source ^ What it measures ^ Pin with ^ Status 2026-08-27 ^ Papers of 1,153 ^
+| **CrUX** | Page-load buckets of **origins** from eligible Chrome | month + bucket | Current. Docs live. Google: CC BY 4.0. Tranco homepage still labels CrUX CC BY-SA 4.0; methodology page states no CrUX licence | 36 (3.1%) |
+| **Tranco** | Aggregate of CrUX, Farsight, Majestic, Radar, Umbrella | list id (''46W9X'' dated 2026-08-26) | Current. Daily list; Farsight is default-list-only | 262 (22.7%) |
+| **Cloudflare Radar** | DNS to 1.1.1.1, pay-level domains | snapshot date (no permanent id) | Current. Ordered top 100; unordered buckets below. CC BY-NC 4.0 | 4 (0.3%) |
+| **Cisco Umbrella** | DNS to OpenDNS, hostnames | dated zip | Current. top-1m.csv.zip still published | 24 (2.1%) |
+| **Majestic Million** | Backlinks | download date + file hash | Current. CSV advertised | 26 (2.3%) |
+| **SecRank** | Chinese DNS, voting | snapshot date | Current. secrank.cn live | 4 (0.3%) |
+| **SimilarWeb** | Commercial panel + crawl | only if the contract lets you keep the exact export | Current, paywalled. No public dated dump or permanent id | 12 (1.0%) |
+| **Farsight** (as ranking) | DNSDB cache-misses, 1M PLD | a Tranco daily id | Current **only inside Tranco's default list** | **0** |
+| **Quantcast** | Was a Tranco input | — | **Historical as a public ranking.** Measure still exists as publisher analytics | 8 (0.7%) |
+| **Alexa** | Toolbar / panel visits | snapshot, if you still have one | **Dead.** Site 1 May 2022; APIs 15 December 2022 | 463 (40.2%) |
+==== CrUX ====
+[[https://developer.chrome.com/docs/crux|CrUX]] publishes rank-magnitude buckets (1k, 5k, 10k, …, 5M) of popular **origins** based on user-initiated page loads from eligible Chrome installs — usage-statistic reporting, history sync without a passphrase, and a supported platform (not Chrome on iOS).(([[https://developer.chrome.com/docs/crux/methodology|CrUX methodology]], fetched 2026-08-27: user eligibility; datasets licensed **CC BY 4.0**. Tranco's homepage still labels CrUX [[https://creativecommons.org/licenses/by-sa/4.0/|CC BY-SA 4.0]]; the methodology page states no CrUX licence. Google's page is the primary source.)) There is no rank 5,000. Country- and device-specific slices exist; they measure Chrome page loads in that country, not "sites for that audience" — see [[Design:Website classification]]. API detail: [[Programming:CrUX]]. Galloway et al. did not evaluate it {[galloway2024_practical]}.
+==== Tranco ====
+[[https://tranco-list.eu/|Tranco]] is a research-oriented generator with permanent list ids, introduced in NDSS 2019 {[LePochat2019_tranco]} combining Alexa, Umbrella, Majestic and Quantcast. **Its inputs have changed three times** (Quantcast dropped after 2020-04-01, Farsight joined 2022-05-01, Alexa replaced by CrUX and Radar on 2023-08-01). The daily list of 2026-08-26 (id ''46W9X'') is CrUX, Farsight, Majestic, Radar and Umbrella, Dowdall over 30 days, ''filterPLD: on'', prefix 1,000,000.((Read from Tranco's own API on 2026-08-27: ''https://tranco-list.eu/api/lists/date/latest''. Describe composition from the live API rather than from the 2019 paper.)) A 2019 id and a 2026 id are different instruments. Radar's CC BY-NC 4.0 rides along on the default list. API, Farsight trap, ''pin_tranco.py'': [[Programming:Tranco]].
+==== Cloudflare Radar ====
+[[https://radar.cloudflare.com/domains|Cloudflare Radar Domain Rankings]] ranks **pay-level domains** from DNS queries to 1.1.1.1, not from HTTP hits on Cloudflare-operated websites. Ordered top 100 (daily, global and per country) plus unordered buckets up to 1M (weekly). Details: [[Programming:Cloudflare Radar]]. Manipulable with a cheap VPN {[galloway2024_practical]}.
+==== Cisco Umbrella ====
+[[https://umbrella-static.s3-us-west-1.amazonaws.com/index.html|Cisco Umbrella]] ranks names by DNS query traffic to OpenDNS resolvers. The daily top-1m zip was still published on 2026-08-27. Today's file is hostnames: ''1,google.com'' / ''2,gstatic.com'' / ''3,www.google.com''. The vendor page still says the list "includes root domains, subdomains in addition to TLDs"; the TLD ranking is a **different zip** (above). Typos and infrastructure hostnames survive. Still a Tranco default input.
+==== Majestic Million ====
+[[https://majestic.com/reports/majestic-million|Majestic]] ranks domains by backlinks. The Million page still advertised free search and download on 2026-08-27 (''downloads.majestic.com/majestic_million.csv''). CC BY 3.0, per Tranco. Does not measure visits. Link-graph rankings are cheap to manipulate relative to a toolbar or panel list {[LePochat2019_tranco]}. That paper did not evaluate CrUX.
+==== Farsight (DomainTools) ====
+A **1-million pay-level-domain ranking** built from cache-misses in Farsight's DNSDB — "above-resolver" DNS that organisations share, so a name that stays in a recursive cache is invisible. Tranco has used it as a default input since **1 May 2022**, the same day Alexa.com was retired.((Tranco [[https://tranco-list.eu/methodology|methodology]], fetched 2026-08-27: "Farsight Security, part of DomainTools, provides a daily updated list consisting of one million entries. … Only pay-level domains are ranked. This list is available since May 1, 2022." Ranks "are based on passive DNS traffic from its DNSDB dataset" of cache misses. DomainTools' own post [[https://www.domaintools.com/blog/mirror-mirror-on-the-wall-whos-the-fairest-website-of-them-all|Mirror, Mirror]] is the ranking's public description.)) It is **not a public CSV**. You get it only as part of Tranco's daily list: the custom-list API's ''providers'' enum is ''crux / majestic / radar / umbrella / alexa / quantcast'' — Farsight, which **is** in the daily list, is **not** in that enum. See [[Programming:Tranco]]. **Zero papers in this corpus name the Farsight ranking as a ''sourceList''.** The 13 papers that name Farsight or DNSDB used the passive-DNS **dataset**, which is a different instrument.
+==== Alexa, Quantcast, SimilarWeb, SecRank ====
+  * ''[[https://en.wikipedia.org/wiki/Alexa_Internet|Alexa]]'': **Discontinued — alexa.com was retired on 1 May 2022 and the Alexa Top Sites and Web Information Service APIs on 15 December 2022.**((Alexa Support, "We retired Alexa.com on May 1, 2022", archived at [[https://web.archive.org/web/20221126115049/https://support.alexa.com/hc/en-us/articles/4410503838999-We-retired-Alexa-com-on-May-1-2022|web.archive.org]]. The date **1 August 2023** is when Tranco dropped Alexa from the default list and folded in CrUX and Cloudflare Radar — not when Amazon switched Alexa off.)) Rankings came from a browser toolbar / panel {[scheitle2018_long]}. **Do not start a new crawl from an Alexa snapshot** unless the study is a reproduction of a pre-2022 paper, in which case name the snapshot.
+  * ''Quantcast'': Was a Tranco default input until 1 April 2020. The Measure product still exists as publisher analytics (''quantcast.com/measure/'' redirects to ''/publisher/measure''); it is not a public top-sites ranking you can pin. Named by 8 papers, none in 2025–2026.
+  * ''SimilarWeb'': Commercial panel and crawl estimates; paywalled (some research exceptions exist). See [[Programming:SimilarWeb]]. Named by 12 of 1,153 web-sampling papers.
+  * ''[[https://secrank.cn/|SecRank]]'': Voting-based ranking from Chinese DNS, introduced in USENIX Security 2022 {[xie2022_building]}. Site live on 2026-08-27. Named by 4 papers. Use it when the audience is Chinese resolver traffic; it is not a drop-in Alexa replacement for a global crawl.
+===== They disagree, and that is the finding =====
+==== Rank-magnitude, not a shuffled head ====
+Ruth et al. compared public lists to Cloudflare HTTP logs in February 2022 {[ruth2022_toppling]}:
+> of the 1,790 domains we measure in the Alexa top 10K, 70% of them are ranked by Cloudflare in a lower rank-magnitude bucket, and 27.2% of them are ranked by Cloudflare in a bucket two or more orders of magnitude less popular.
+On the Alexa top 1K the same paper reports **87.1%** overranked (average of 210 sites) and **56.7%** by two or more orders. CrUX is the exception they found: **47.1%** of 1,410 CrUX top-10K domains overranked against Cloudflare, and **only 1%** by two or more orders. "Inaccurate" is not a small permutation of the head; it is a different web. They did not evaluate the five-provider Tranco list current in 2026.
+Scheitle et al. measured agreement between the lists themselves {[scheitle2018_long]}: for the Top 1M, Alexa and Majestic shared **285k** domains on average in their 2017–2018 joint window — well under a third. Daily churn reached **50%** of domains on some lists. An unused test domain reached rank **22k** on Umbrella.
+==== The head is a technological outlier ====
+Scheitle et al. compared list domains against the general ''com''/''net''/''org'' population {[scheitle2018_long]}:
+> the HTTP/2 adoption of all com/net/org domains is 7.84% on average and thus significantly lower than for domains listed in Top 1M lists, (up to 26.6% for Alexa)
+Top 1k lists showed adoption around **35%** or more. The same paper's IPv6 and CDN gaps went the same way. On that indicator the head was a technological outlier; it is not a theorem that every later adoption measure will. How you //draw// from the head is [[Design:Sampling]].
+==== DNS lists are not lists of websites ====
+DNS-based lists — **Umbrella**, **Farsight**, **SecRank**, and parts of **Radar** and **Tranco** — count resolver observations. That over-represents names that devices look up without a user opening a page. The table above is the current evidence: ''gstatic.com'' and ''windowsupdate.com'' sit in Tranco's head; ''windowsupdate.microsoft.com'' is an Umbrella hostname that a PLD list drops. An earlier draft of this page used ''ec2.internal'' as the running example; it is **not** in the 2026-08-27 Umbrella top-1m.
+Galloway et al. generated names, queried them through a **$10/month VPN**, and "consistently achieved a ranking in the top 100,000" on **Radar**; Tranco then placed the same names at rank 1 million within 10 days and inside 500k within 14 days {[galloway2024_practical]}. They also sat consistently under rank 250,000 on Umbrella. On the Radar API as of 2026-08-27 that top 100,000 is an **unordered bucket**, not a total order — see [[Programming:Cloudflare Radar]]. That is a finding about Radar (and Tranco via that input), not about every DNS ranking, and not about CrUX.
+==== Stability and manipulation ====
+**Alexa**, when it was live, ranked from a toolbar panel. Two problems, both measured:
+  * **Manipulations**: ranks could be altered with as little as a single HTTP request {[LePochat2019_tranco]}. There was a real-world incentive to do this.
+  * **Stability**: a small panel, a large web, and a daily publish produced high churn.
+Tranco was designed to reduce both by aggregating sources over 30 days. CrUX publishes buckets rather than a total order, so a name that moves inside a magnitude stays in the same published bucket; this page does not re-measure that as a stability rate. {[LePochat2019_tranco]} costed attacking **Alexa** as cheap; it did not evaluate CrUX. Galloway et al. also did not evaluate CrUX {[galloway2024_practical]}.
 <WRAP right 50% box>
@@ Line 75: / Line 300: @@
 <div>Possible manipulations of lists and their estimated cost according to {[LePochat2019_tranco]}.</div>
 </WRAP>
-==== Representativeness ====
-  * CrUX ranks websites based on actual page loads, making it more reflective of real user behavior compared to Majestic, which relies on backlink analysis.
 ===== Use in Publications =====
-Several publications have surveyed the usage of various ranking lists in academic research. The figures below illustrate findings from Scheitle et al. {[scheitle2018_long]} and Xie et al. {[xie2024_crawling]}, noting, however, that recent trends are not fully captured due to the lag in the research process. For instance, Alexa, though discontinued in 2023, is still used in 2024 publications due to sampling occurring at the start of studies. Also, the publication survey ends in 2022.
+The figures below come from a structured extraction over **5,859 full-text papers** from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026. The population for this page is the **1,153 papers that drew at least one study population whose unit is websites, domains or web pages** — 19.7% of the corpus, the same population as [[Design:Sampling]]. Only those web-unit tuples are counted. Free-text ''sourceList'' is folded into vendor families by ''scripts/rank_fold.mjs'' before counting (multi-label: a string naming Tranco and Radar counts for both). Sentinel values are silence. 2025 and 2026 are provisional venue-years.
+Of the 1,143 WEB papers that name any source, **764 (66.8%)** name a popularity ranking and **257 (22.5%)** name a custom or ad-hoc seed list — the same frame-kind fold as [[Design:Sampling]]. This page is about the 763 of those ranking papers that name a **vendor**. A custom seed is a different instrument; it does not go in the table below.
+==== Folded vendors, not exact strings ====
+^ Vendor ^ Papers of 1,153 ^ Share ^ Exact name only ^ Spellings folded ^ Exact-string undercount ^
+| Alexa | 463 | 40.2% | 50 | 355 | **89.2%** |
+| Tranco | 262 | 22.7% | 178 | 91 | 32.1% |
+| CrUX | 36 | 3.1% | 9 | 23 | 75.0% |
+| Majestic | 26 | 2.3% | 4 | 17 | 84.6% |
+| Cisco Umbrella | 24 | 2.1% | 2 | 23 | 91.7% |
+| SimilarWeb | 12 | 1.0% | 9 | 5 | 25.0% |
+| Quantcast | 8 | 0.7% | 4 | 5 | 50.0% |
+| Cloudflare Radar | 4 | 0.3% | 3 | 4 | 25.0% |
+| SecRank | 4 | 0.3% | 2 | 4 | 50.0% |
+**763 papers (66.2%) name at least one of these vendors.** Family counts do not sum to 763: 55 papers (7.2% of vendor-users) name two or more. Farsight-as-ranking: **0 papers**. Radar as a named source: 4, all in 2025–2026 — it is a Tranco input, not a list people cite. The full fold, including commercial also-rans (Semrush 3, BuiltWith 2, Open PageRank 2, Ahrefs / Moz / Netcraft / Trexa 1 each) and the unnamed-top-N residue (4 strings), is on the [[provenance:design:website_selection|provenance page]].
+Counting the exact string "Alexa" on all 5,712 papers that sampled //anything// finds 51; folding on WEB finds 463. The same un-folded method on ALL ranks "custom seed list" (516), Tranco (180) and Google Play (143) above Alexa (51). Google Play is 281 papers after folding ''/google play/i'' over ALL, and **0** over the 1,153 that sampled the web. Restrict to web units and fold spellings, or you will publish an app store as a website ranking.
+==== The list changed under the field ====
+Papers naming each list, as a share of the 1,153 in that year:
+^ Year ^ Papers ^ Alexa ^ Tranco ^ CrUX ^
+| 2021 | 87 | 47.1% | 31.0% | 0.0% |
+| 2022 | 106 | 37.7% | 26.4% | 1.9% |
+| 2023 | 122 | 24.6% | 38.5% | 4.1% |
+| 2024 | 122 | 17.2% | 45.9% | 9.8% |
+| 2025 //(provisional)// | 133 | 8.3% | **56.4%** | 9.0% |
+| 2026 //(provisional)// | 57 | 7.0% | 38.6% | 7.0% |
+Tranco overtook Alexa in 2023, the first full year after Alexa.com died. The lag is compatible with a submission cycle; we did not read why each of those papers still named Alexa (cached CSV, inherited list, or a crawl that started before May 2022 are all possible). CrUX peaked at 9.8% in 2024 and did not keep climbing. Read 2026 as incomplete: CCS and IMC 2026 have not been held.
+Of the **66 papers from 2023–2026 that still name Alexa**, 32 (48.5%) state a version or date and 34 (51.5%) do not. The undated 34 cannot be pinned to a snapshot from the paper. A mirror, a cached CSV, or a list inherited from earlier work would all look like this; this corpus does not distinguish them.
+Umbrella, Majestic, Radar, SimilarWeb and SecRank stay in the low single digits in every year they appear. They are specialised instruments and Tranco inputs, not a second mainstream ranking.
+==== What the pre-corpus surveys said ====
+Scheitle et al. {[scheitle2018_long]} and Xie et al. {[xie2024_crawling]} surveyed ranking-list usage in web-measurement papers by hand. They are why this page had figures before this corpus existed. Scheitle ends in 2016-era crawls; Xie's survey of crawling papers ends around 2022 (PAM 2024; outside the seven venues). Neither includes 2023–2026, which is the window in which Tranco became the default and Alexa became residue. **Alexa was not "discontinued in 2023"** — that sentence was on an earlier version of this page and confused Amazon's 2022 retirement with Tranco's 2023 provider swap.
 <WRAP center 100% box>
@@ Line 90: / Line 354: @@
 <div>Popularity of website lists in web measurement publications according to {[xie2024_crawling]}.</div>
 </WRAP>
+==== Methodology and limitations of these figures ====
+  * **How they were produced.** One structured record per paper; each ''population[]'' tuple carries a verbatim evidence quote. The population is the extraction's ''unit'' enum (stable). The free-text ''sourceList'' is not, and is folded before counting.
+  * **Folding.** ''scripts/rank_fold.mjs'' maps a string onto every vendor it names. Alexa undercounts by 89.2% if you count the exact string "Alexa". Farsight/DNSDB strings are a hand map of 12 spellings classified as a passive-DNS **dataset**, not a ranking; the report fails if a new spelling appears. Residue of unnamed "top N websites" strings: 4.
+  * **This is not the 5,712-paper ''sampled'' population.** That denominator is 97.5% of a corpus that is mostly not web measurement. Google Play, Prolific and MNIST live in it.
+  * **Sampling's 764 "popularity-ranking" papers** are a frame-kind fold, not a vendor fold; the vendor union here is 763. The one-paper gap is an unnamed top-N string that names no vendor. [[Design:Sampling]] reports the frame kind; this page reports the vendor. Custom-seed 257 is on that page too.
+  * **Posters.** 23 of 1,153 WEB papers are posters; 10 of 763 vendor-users. Trimmed shares (posters and ≤4-page records removed, 1,113 papers) move Alexa 40.2% → 41.0% and Tranco 22.7% → 22.8%. The headline is not a silence rate.
+  * **Venue coverage.** Seven venues only. 2025 and 2026 are incomplete by construction. Xie 2024 is PAM, cited as a survey, not counted in the tables.
+  * **Every query, the report script and its unedited output** are on [[provenance:design:website_selection]]; corpus-level caveats are on [[literature:corpus]].
+===== What to report =====
+A methods sentence a reader can act on names, in this order:
+  - **Which list, with its identity** — Tranco id, CrUX month, Umbrella dated zip, Majestic download date, Radar snapshot date. Not "the top 1M".
+  - **What that list measures** — page loads, DNS queries, cache-misses, backlinks, or an aggregate of those.
+  - **The draw** — top-//n//, stratified, random, … — which is [[Design:Sampling]], not this page.
+  - **The file** — archived, hashed, and deposited. See [[Artifacts]].
+Pin **one dated frame**, not an informal average of several daily lists. Tranco's own daily id already aggregates 30 days (Dowdall); that is a defined procedure with a permalink. CrUX is a monthly bucket snapshot — cite the month. {[LePochat2019_tranco]} is the stability argument for that kind of aggregation, not a licence to mix unpublished daily CSVs. Do not treat Tranco as CrUX. If you can, report on more than one list: 55 of 763 ranking-users (7.2%) named two or more vendors; Ruth et al. showed CrUX and Tranco are different instruments against HTTP logs {[ruth2022_toppling]}, and they did not evaluate reporting both. 52.0% of papers that named a ranking vendor in this corpus stated no version on that tuple (397 of 763).
+===== Open Questions =====
+<WRAP todo>
+  * **Ruth et al. has not been re-run on the five-provider Tranco list.** The 2022 comparison predates Radar-in-Tranco and the Alexa→CrUX swap. Whether the 2026 default list is closer to CrUX or to Umbrella is an unmeasured claim, and the ''gstatic.com'' rank above is a reason to measure it.
+  * **How much does swapping Tranco for CrUX move a published web-measurement figure?** Several papers could be re-crawled on both; we found none that reports the same measurement on both as its contribution. The IP-classification analogue — four geolocation databases on one address set — is the template.
+  * **No public download of the Farsight ranking**, and zero papers name it as a ''sourceList''. Tranco's methodology describes it; DomainTools does not publish a CSV equivalent of Umbrella's. Whether that input is doing work in the daily list is opaque from outside.
+  * **The 34 undated 2023–2026 Alexa papers.** A full-text pass that distinguished cached CSVs, inherited lists, and crawls that started before May 2022 would turn a residual count into a finding.
+</WRAP>
+===== Related pages =====
+  * [[Design:Sampling]] — how to draw, sample size, unit, versioning. Assumes you have chosen a list. Custom-seed lists live there.
+  * [[Design:Longitudinal]] — why a 2019 Tranco id and a 2026 Tranco id are not commensurable.
+  * [[Programming:Tranco]] — ids, API, the Farsight-is-default-list-only trap, ''pin_tranco.py''.
+  * [[Programming:Cloudflare Radar]] — buckets are not ranks; CC BY-NC 4.0; Galloway.
+  * [[Programming:CrUX]] — magnitude buckets, BigQuery, country/device slices.
+  * [[Programming:SimilarWeb]] — the commercial panel.
+  * [[Design:Website classification]] — topic labels, not popularity ranks. Alexa died as a categoriser on the same day it died as a ranking. CrUX country lists are popularity, not audience.
+  * [[Design:IP classification]] — the servers a crawl connects to are mostly CDNs; geolocating them is a different question from picking the crawl's seed list.
+  * [[Artifacts]] — depositing the pinned list.
+  * [[literature:corpus]] — how the 5,859-paper extraction was built.
+  * [[provenance:design:website_selection]] — every query on this page, the fold, the residue, the review log.
 ====== References ======
@@ Line 96: / Line 404: @@
 ~~DISCUSSION~~