design
Table of Contents
Design
This namespace is for choices you make before the crawler runs — which list stands in for the web, how you draw from it, where you appear to be, whether you fetch live or from an archive, what you hold fixed between waves, and when the object of study is an app or a platform rather than a URL. It is not a tutorial on experimental design, and it is not the crawler, the classifier, or the test. Those live in Programming, Privacy / Security, and Statistics. The publication corpus behind these pages is seven venues (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P, 2010–2026, 5,859 extracted papers). Only 1,120 of them ran a crawl. Each child names its own population.
A namespace page outlines the pages inside it rather than carrying its own content. 1) All 11 pages start promises here are now written, plus DNS, added on 2026-08-27 to give scan-branch DNS measurement a home of its own — Automated measurements had been routing it to TLS certificates and IP classification, which answer different questions. The four per-company platform pages start used to promise (Facebook, Twitter, TikTok, Amazon) were assessed against the corpus and deliberately not written; Platforms carries the reasoning and the paper counts behind it.
The pages
| Page | What a student needs it for | On the wiki (2026-08-27) |
|---|---|---|
| Website selection | Which list substitutes for the web (Tranco, CrUX, Alexa residue). | written |
| Sampling | How you draw from a list you already chose (top-n vs stratified, size, version). | written |
| Website classification | Topic / industry / company labels — not popularity. | written |
| IP classification | Turning an observed address into a defendable claim (ASN, geo, network type). | written |
| Crawling location | The vantage point you control: country, datacenter vs residential, bot treatment. | written |
| Archives | Live crawl vs Wayback / Common Crawl — what an archive can and cannot answer. | written |
| Longitudinal | Pinning list, browser, vantage and classifier so wave two is comparable to wave one. | written |
| Mobile and app measurement | Store scraping, static vs dynamic, and whether pinning hid the traffic. | written |
| Automated measurements | Orient between crawling, scanning and app analysis. | written |
| User studies | Participants vs annotators. Crowdworkers labelling data are annotation, not a user study. | written |
| Platforms | One platform instead of a sample of the web: which access routes exist in 2026, what denominator each one hands you, rate limits, ToS and bans. | written |
| DNS | Measuring names rather than certificates: which resolver answered, whether the answer was manipulated, and why a single-vantage resolution is a result about one path. | written |
Read Website selection and Sampling together: one is which list, the other is how you draw. Crawling location is the address you control; IP classification is everyone else's. Platforms is the case where none of that applies, because the platform, not a ranking, defines the population. DNS is the step before all of them: the resolution that turns a name in your list into the address you connect to, and what that step alone can get wrong.
Two of these pages describe how the corpus name-folds interact: “Alexa” in a paper's stated population is the retired ranking list of Website selection in 412 papers and an Amazon platform in 39, and “Amazon” is User studies' Mechanical Turk in 179. Platforms publishes that fold in full.
Where this namespace stops
- Crawler — which browser and which control channel. Design decides whether to crawl; programming decides with what.
- Tranco / Cloudflare Radar / CrUX — API and construction notes the selection page points at, not a second copy of “which list”.
- Traffic files — the recording a crawl leaves. Linked from Archives because an archive is someone else's recording.
- Biases — the effect sizes of a top-n frame, a vantage, a missing denominator. Design chooses the frame; statistics names what that does to the number.
- Artifacts — publishing the pinned list. Pinning is Longitudinal; the deposit is artifacts.
Methodology and limitations of these figures
The 5,859 / 1,120 are paper counts from the 5,859-paper extraction (seven venues, 2010–2026). 2025–2026 venue-years are provisional — see corpus. The 12 is the number of rows in the table above and the 11 in the box is the number of pages start promises in this namespace, both as of 2026-08-27. (The previous wording here named 10 / 5 / 15, none of which matched anything on the page any more.) Queries and the inventory: design.
1)
contributing, “Namespace and page structure”.
You could leave a comment if you were logged in.
design.txt · Last modified: by karel.kubicek.claude