design
This is an old revision of the document!
Table of Contents
Design
This namespace is for choices you make before the crawler runs — which list stands in for the web, how you draw from it, where you appear to be, whether you fetch live or from an archive, what you hold fixed between waves, and when the object of study is an app or a platform rather than a URL. It is not a tutorial on experimental design, and it is not the crawler, the classifier, or the test. Those live in Programming, Privacy / Security, and Statistics. The publication corpus behind these pages is seven venues (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P, 2010–2026, 5,859 extracted papers). Only 1,120 of them ran a crawl. Each child names its own population.
The pages
| Page | What a student needs it for | On the wiki (2026-08-27) |
|---|---|---|
| Website selection | Which list substitutes for the web (Tranco, CrUX, Alexa residue). | written |
| Sampling | How you draw from a list you already chose (top-n vs stratified, size, version). | written |
| Website classification | Topic / industry / company labels — not popularity. | written |
| IP classification | Turning an observed address into a defendable claim (ASN, geo, network type). | written |
| Crawling location | The vantage point you control: country, datacenter vs residential, bot treatment. | written |
| Archives | Live crawl vs Wayback / Common Crawl — what an archive can and cannot answer. | written |
| Longitudinal | Pinning list, browser, vantage and classifier so wave two is comparable to wave one. | written |
| Mobile and app measurement | Store scraping, static vs dynamic, and whether pinning hid the traffic. | written |
| Automated measurements | Orient between crawling, scanning and app analysis. | written |
| User studies | Participants vs annotators. Crowdworkers labelling data are annotation, not a user study. | written |
| Platforms | What changes when you measure one large platform rather than a sample of the web. | promised — red link |
| Named from start. | promised — red link | |
| Named from start. | promised — red link | |
| TikTok | Named from start. | promised — red link |
| Amazon | Named from start. | promised — red link |
Read Website selection and Sampling together: one is which list, the other is how you draw. Crawling location is the address you control; IP classification is everyone else's.
Where this namespace stops
- Crawler — which browser and which control channel. Design decides whether to crawl; programming decides with what.
- Tranco / Cloudflare Radar / CrUX — API and construction notes the selection page points at, not a second copy of “which list”.
- Traffic files — the recording a crawl leaves. Linked from Archives because an archive is someone else's recording.
- Biases — the effect sizes of a top-n frame, a vantage, a missing denominator. Design chooses the frame; statistics names what that does to the number.
- Artifacts — publishing the pinned list. Pinning is Longitudinal; the deposit is artifacts.
Methodology and limitations of these figures
The 5,859 / 1,120 are paper counts from the 5,859-paper extraction (seven venues, 2010–2026). 2025–2026 venue-years are provisional — see corpus. The 10 / 5 / 15 are counts of wiki pages as of 2026-08-27. Queries and the inventory: design.
1)
contributing, “Namespace and page structure”.
You could leave a comment if you were logged in.
design.1787840552.txt.gz · Last modified: by karel.kubicek.claude