design
This is an old revision of the document!
Table of Contents
Design
This namespace is for choices you make before the crawler runs — which list stands in for the web, how you draw from it, where you appear to be, whether you fetch live or from an archive, what you hold fixed between waves, and when the object of study is an app or a platform rather than a URL. It is not a tutorial on experimental design, and it is not the crawler, the classifier, or the test. Those live in Programming, Privacy / Security, and Statistics. The publication corpus behind these pages is seven venues (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P, 2010–2026, 5,859 extracted papers). Only 1,120 of them ran a crawl. Each child names its own population.
A namespace page outlines the pages inside it rather than carrying its own content. 1) All 11 pages start promises here are now written. The four per-company platform pages start used to promise (Facebook, Twitter, TikTok, Amazon) were assessed against the corpus and deliberately not written; Platforms carries the reasoning and the paper counts behind it.
The pages
| Page | What a student needs it for | On the wiki (2026-08-27) |
|---|---|---|
| Website selection | Which list substitutes for the web (Tranco, CrUX, Alexa residue). | written |
| Sampling | How you draw from a list you already chose (top-n vs stratified, size, version). | written |
| Website classification | Topic / industry / company labels — not popularity. | written |
| IP classification | Turning an observed address into a defendable claim (ASN, geo, network type). | written |
| Crawling location | The vantage point you control: country, datacenter vs residential, bot treatment. | written |
| Archives | Live crawl vs Wayback / Common Crawl — what an archive can and cannot answer. | written |
| Longitudinal | Pinning list, browser, vantage and classifier so wave two is comparable to wave one. | written |
| Mobile and app measurement | Store scraping, static vs dynamic, and whether pinning hid the traffic. | written |
| Automated measurements | Orient between crawling, scanning and app analysis. | written |
| User studies | Participants vs annotators. Crowdworkers labelling data are annotation, not a user study. | written |
| Platforms | One platform instead of a sample of the web: which access routes exist in 2026, what denominator each one hands you, rate limits, ToS and bans. | written |
Read Website selection and Sampling together: one is which list, the other is how you draw. Crawling location is the address you control; IP classification is everyone else's. Platforms is the case where none of that applies, because the platform, not a ranking, defines the population.
Two of these pages describe how the corpus name-folds interact: “Alexa” in a paper's stated population is the retired ranking list of Website selection in 412 papers and an Amazon platform in 39, and “Amazon” is User studies' Mechanical Turk in 179. Platforms publishes that fold in full.
Where this namespace stops
- Crawler — which browser and which control channel. Design decides whether to crawl; programming decides with what.
- Tranco / Cloudflare Radar / CrUX — API and construction notes the selection page points at, not a second copy of “which list”.
- Traffic files — the recording a crawl leaves. Linked from Archives because an archive is someone else's recording.
- Biases — the effect sizes of a top-n frame, a vantage, a missing denominator. Design chooses the frame; statistics names what that does to the number.
- Artifacts — publishing the pinned list. Pinning is Longitudinal; the deposit is artifacts.
Methodology and limitations of these figures
The 5,859 / 1,120 are paper counts from the 5,859-paper extraction (seven venues, 2010–2026). 2025–2026 venue-years are provisional — see corpus. The 10 / 5 / 15 are counts of wiki pages as of 2026-08-27. Queries and the inventory: design.
1)
contributing, “Namespace and page structure”.
You could leave a comment if you were logged in.
design.1787862128.txt.gz · Last modified: by karel.kubicek.claude