| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| start [2026/08/27 20:21] – design:platforms is written; drop the four per-company red links (Facebook/Twitter/TikTok/Amazon) — the parent page explains why they are not worth writing. Authored by Claude karel.kubicek.claude | start [2026/08/31 19:48] (current) – Link the new Privacy:Browser storage page from the Privacy section. Authored by Claude karel.kubicek.claude |
|---|
| * [[Design:Website selection]] and [[Design:Sampling|Representative sampling methods]], [[Design:Website classification]], and [[Design:IP classification]] | * [[Design:Website selection]] and [[Design:Sampling|Representative sampling methods]], [[Design:Website classification]], and [[Design:IP classification]] |
| * [[Design:Crawling location]] and [[Design:Archives|Crawling live or using archives]] ([[Programming:Traffic files|using traffic files, e.g., HAR]]) | * [[Design:Crawling location]] and [[Design:Archives|Crawling live or using archives]] ([[Programming:Traffic files|using traffic files, e.g., HAR]]) |
| | * [[Design:DNS|Measuring DNS]] — which resolver answered, whether the answer was manipulated, and what encrypted DNS changed. Three public resolvers gave completely disjoint answers for 28 of the Tranco top 100 on 2026-08-27. |
| | * [[Design:Existing datasets|Measuring from an existing dataset]] — when the measurement is a query (Censys, a CT log, AndroZoo, last year's crawl) rather than a crawl you ran: which snapshot, which query, which join key, and what a producer's port set and vantage point do to your denominator |
| * [[Design:Longitudinal|Repeating a measurement over time]] — pinning the list, the browser, the vantage point and the classifier so wave two is comparable to wave one | * [[Design:Longitudinal|Repeating a measurement over time]] — pinning the list, the browser, the vantage point and the classifier so wave two is comparable to wave one |
| * [[Design:Mobile and app measurement|Measuring mobile apps]] — store scraping, static and dynamic analysis, and the certificate-pinning problem that decides whether you saw the traffic at all | * [[Design:Mobile and app measurement|Measuring mobile apps]] — store scraping, static and dynamic analysis, and the certificate-pinning problem that decides whether you saw the traffic at all |
| * [[Programming]] | * [[Programming]] |
| * [[Programming:Crawler|Comparison of crawling libraries]] such as [[Programming:Crawler:OpenWPM]], [[Programming:Crawler:Tracker Radar Collector]], [[Programming:Crawler:PageGraph]], etc. | * [[Programming:Crawler|Comparison of crawling libraries]] such as [[Programming:Crawler:OpenWPM]], [[Programming:Crawler:Tracker Radar Collector]], [[Programming:Crawler:PageGraph]], etc. |
| | * [[Programming:Crawler Detection|When the website notices your crawler]] — bot management, headless and CDP leaks, CAPTCHAs and challenge pages, and what a blocked crawl does to the headline number |
| | * [[Programming:Crawler:LLM Agents|Crawling with an LLM agent]] — Browser Use, BrowserGym/AgentLab and the Computer-Use APIs as a measurement instrument: what five 2026 papers measured about completion rates and per-site cost, what nobody has measured about run-to-run variance, and why this is not yet the default |
| | * [[Programming:Filter Lists|Filter lists]] — EasyList, EasyPrivacy, Disconnect and the regional lists as the field's shared labelling instrument //and// its ground truth: which list, which commit, what they miss off the anglophone desktop web, and why "blocked implies tracker" is circular |
| * [[Programming:Multilingual support]] | * [[Programming:Multilingual support]] |
| * [[Programming:Interaction|Interaction with websites]] | * [[Programming:Interaction|Interaction with websites]] |
| * [[Programming:Registration|Automating login and registration]] | * [[Programming:Registration|Automating login and registration]] |
| | * [[Programming:Deployment|Running a measurement for weeks]] — the crawl as a process that must survive: checkpoints, retries, monitoring, cost, and what to report when the machine dies on day four |
| * Multitude of pages linked from elsewhere documenting specific technologies, e.g., [[Programming:Tranco]], [[Programming:Cloudflare Radar]], [[Programming:Docker]], [[Programming:CrUX]], [[Programming:Similarweb]]. | * Multitude of pages linked from elsewhere documenting specific technologies, e.g., [[Programming:Tranco]], [[Programming:Cloudflare Radar]], [[Programming:Docker]], [[Programming:CrUX]], [[Programming:Similarweb]]. |
| * [[Privacy]] | * [[Privacy]] |
| * Classifying [[Privacy:Requests|Web requests]], [[Privacy:Cookies]], [[Privacy:Fingerprinting]], or [[Privacy:JavaScript]] | * Classifying [[Privacy:Requests|Web requests]], [[Privacy:Cookies]], [[Privacy:Fingerprinting]], or [[Privacy:JavaScript]] |
| | * [[Privacy:Browser storage|Storage beyond cookies]] — localStorage, IndexedDB, service workers and the caches: what your crawler records, what it silently does not, and why "we cleared cookies" is not a reset |
| * [[Privacy:Server side tracking|Measuring server-side tracking]] — when the tracking request never reaches the tracker, and every request-level method quietly stops working | * [[Privacy:Server side tracking|Measuring server-side tracking]] — when the tracking request never reaches the tracker, and every request-level method quietly stops working |
| * [[Privacy:Cookie syncing|Measuring cookie and ID syncing]] — how third parties learn that their two identifiers are the same person, and the crawl configuration that decides whether you see it at all | * [[Privacy:Cookie syncing|Measuring cookie and ID syncing]] — how third parties learn that their two identifiers are the same person, and the crawl configuration that decides whether you see it at all |
