| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| start [2026/08/27 12:00] – Link Programming:Cloudflare Radar next to Tranco. Authored by Claude. karel.kubicek.claude | start [2026/08/31 19:48] (current) – Link the new Privacy:Browser storage page from the Privacy section. Authored by Claude karel.kubicek.claude |
|---|
| |
| * [[Design|Research design]] clarifies choices needed to conduct the measurements. Example pages: | * [[Design|Research design]] clarifies choices needed to conduct the measurements. Example pages: |
| * [[Design:Automated measurements]] or [[Design:User studies]] | * [[Design:Automated measurements]] — crawl, scan, or unpack an app; the instrument is the design decision — or [[Design:User studies]] (Mechanical Turk shuts 30 September 2026) |
| * [[Design:Website selection]] and [[Design:Sampling|Representative sampling methods]], [[Design:Website classification]], and [[Design:IP classification]] | * [[Design:Website selection]] and [[Design:Sampling|Representative sampling methods]], [[Design:Website classification]], and [[Design:IP classification]] |
| * [[Design:Crawling location]] and [[Design:Archives|Crawling live or using archives]] ([[Programming:Traffic files|using traffic files, e.g., HAR]]) | * [[Design:Crawling location]] and [[Design:Archives|Crawling live or using archives]] ([[Programming:Traffic files|using traffic files, e.g., HAR]]) |
| | * [[Design:DNS|Measuring DNS]] — which resolver answered, whether the answer was manipulated, and what encrypted DNS changed. Three public resolvers gave completely disjoint answers for 28 of the Tranco top 100 on 2026-08-27. |
| | * [[Design:Existing datasets|Measuring from an existing dataset]] — when the measurement is a query (Censys, a CT log, AndroZoo, last year's crawl) rather than a crawl you ran: which snapshot, which query, which join key, and what a producer's port set and vantage point do to your denominator |
| * [[Design:Longitudinal|Repeating a measurement over time]] — pinning the list, the browser, the vantage point and the classifier so wave two is comparable to wave one | * [[Design:Longitudinal|Repeating a measurement over time]] — pinning the list, the browser, the vantage point and the classifier so wave two is comparable to wave one |
| * [[Design:Mobile and app measurement|Measuring mobile apps]] — store scraping, static and dynamic analysis, and the certificate-pinning problem that decides whether you saw the traffic at all | * [[Design:Mobile and app measurement|Measuring mobile apps]] — store scraping, static and dynamic analysis, and the certificate-pinning problem that decides whether you saw the traffic at all |
| * [[Design:Platforms|Research of specific large platforms]] such as [[Design:Platforms:Facebook]], [[Design:Platforms:Twitter]], [[Design:Platforms:TikTok]], [[Design:Platforms:Amazon]] | * [[Design:Platforms|Research of specific large platforms]] — when the population is one platform rather than a sample of the web, the **access route is the design decision** and it fixes your denominator. Which routes still exist in 2026, what they cost, and why there are no per-company sub-pages. |
| * [[Programming]] | * [[Programming]] |
| * [[Programming:Crawler|Comparison of crawling libraries]] such as [[Programming:Crawler:OpenWPM]], [[Programming:Crawler:Tracker Radar Collector]], [[Programming:Crawler:PageGraph]], etc. | * [[Programming:Crawler|Comparison of crawling libraries]] such as [[Programming:Crawler:OpenWPM]], [[Programming:Crawler:Tracker Radar Collector]], [[Programming:Crawler:PageGraph]], etc. |
| | * [[Programming:Crawler Detection|When the website notices your crawler]] — bot management, headless and CDP leaks, CAPTCHAs and challenge pages, and what a blocked crawl does to the headline number |
| | * [[Programming:Crawler:LLM Agents|Crawling with an LLM agent]] — Browser Use, BrowserGym/AgentLab and the Computer-Use APIs as a measurement instrument: what five 2026 papers measured about completion rates and per-site cost, what nobody has measured about run-to-run variance, and why this is not yet the default |
| | * [[Programming:Filter Lists|Filter lists]] — EasyList, EasyPrivacy, Disconnect and the regional lists as the field's shared labelling instrument //and// its ground truth: which list, which commit, what they miss off the anglophone desktop web, and why "blocked implies tracker" is circular |
| * [[Programming:Multilingual support]] | * [[Programming:Multilingual support]] |
| * [[Programming:Interaction|Interaction with websites]] | * [[Programming:Interaction|Interaction with websites]] |
| * Multitude of pages linked from elsewhere documenting specific technologies, e.g., [[Programming:Tranco]], [[Programming:Cloudflare Radar]], [[Programming:CrUX]], [[Programming:Similarweb]]. | * [[Programming:Registration|Automating login and registration]] |
| | * [[Programming:Deployment|Running a measurement for weeks]] — the crawl as a process that must survive: checkpoints, retries, monitoring, cost, and what to report when the machine dies on day four |
| | * Multitude of pages linked from elsewhere documenting specific technologies, e.g., [[Programming:Tranco]], [[Programming:Cloudflare Radar]], [[Programming:Docker]], [[Programming:CrUX]], [[Programming:Similarweb]]. |
| * [[Privacy]] | * [[Privacy]] |
| * Classifying [[Privacy:Requests|Web requests]], [[Privacy:Cookies]], [[Privacy:Fingerprinting]], or [[Privacy:JavaScript]] | * Classifying [[Privacy:Requests|Web requests]], [[Privacy:Cookies]], [[Privacy:Fingerprinting]], or [[Privacy:JavaScript]] |
| | * [[Privacy:Browser storage|Storage beyond cookies]] — localStorage, IndexedDB, service workers and the caches: what your crawler records, what it silently does not, and why "we cleared cookies" is not a reset |
| * [[Privacy:Server side tracking|Measuring server-side tracking]] — when the tracking request never reaches the tracker, and every request-level method quietly stops working | * [[Privacy:Server side tracking|Measuring server-side tracking]] — when the tracking request never reaches the tracker, and every request-level method quietly stops working |
| * [[Privacy:Cookie syncing|Measuring cookie and ID syncing]] — how third parties learn that their two identifiers are the same person, and the crawl configuration that decides whether you see it at all | * [[Privacy:Cookie syncing|Measuring cookie and ID syncing]] — how third parties learn that their two identifiers are the same person, and the crawl configuration that decides whether you see it at all |
| * [[Privacy:TCF consent strings|Decoding TCF consent strings]] — the IAB TC string and Google's Additional Consent string as measurement artefacts, and what a decoded string does and does not prove | * [[Privacy:TCF consent strings|Decoding TCF consent strings]] — the IAB TC string and Google's Additional Consent string as measurement artefacts, and what a decoded string does and does not prove |
| * [[Privacy:darkpatterns|Dark patterns]] in the interfaces you measure — and in the way they defeat your crawler | * [[Privacy:darkpatterns|Dark patterns]] in the interfaces you measure — and in the way they defeat your crawler |
| * [[Security]] | * [[Security]] — measuring web security as it is deployed, not how to break it: |
| * Statistics: [[Statistics:Study preregistration]], [[Statistics:Hypothesis testing]] suitable for web measurements, [[Statistics:Regression]], [[Statistics:Pvalue corrections|P-value corrections]], [[Statistics:Biases]], [[Statistics:Interrater agreement|Inter-rater agreement]] for hand-coded data | * [[Security:TLS certificates|TLS and the web PKI]] |
| | * [[Security:Headers|CSP, HSTS and the other security headers]] |
| | * [[Security:VirusTotal|VirusTotal as a label source]] |
| | * [[Security:Phishing]] |
| | * [[Security:Web vulnerabilities|Measuring web vulnerabilities in the wild]] |
| | * [[Statistics]]: [[Statistics:Study preregistration]], [[Statistics:Hypothesis testing]] suitable for web measurements, [[Statistics:Regression]], [[Statistics:Pvalue corrections|P-value corrections]], [[Statistics:Biases]], [[Statistics:Interrater agreement|Inter-rater agreement]] for hand-coded data |
| * [[Writing]] | * [[Writing]] |
| * [[Writing:Conferences]] suitable for web measurements in the field of security and privacy. | * [[Writing:Conferences]] — where to send a web-measurement paper, and the current CFP dates. |
| * [[Writing:Literature review]] | * [[Writing:Literature review]] — the keyword-derived denominator trap, measured on the SoKs in this corpus. |
| * [[Artifacts]] explains how to ensure your research to be reproducible. | * [[Artifacts]] — a GitHub URL is not an archival artifact. What to put in a crawl deposit, where 2026 Available badges will accept it, and what to withhold when the data has people in it. |
| * Pages whose figures come from the publication corpus carry a working log under ''provenance:'' — every query with its denominator, the report script and its unedited output, the folds and their residue, and what could not be established. The corpus-wide caveats are on [[Literature:Corpus]]; the per-page logs mirror the page id, e.g. [[provenance:privacy:fingerprinting]] for [[Privacy:Fingerprinting]]. | * Pages whose figures come from the publication corpus carry a working log under ''provenance:'' — every query with its denominator, the report script and its unedited output, the folds and their residue, and what could not be established. The corpus-wide caveats are on [[Literature:Corpus]]; the per-page logs mirror the page id, e.g. [[provenance:privacy:fingerprinting]] for [[Privacy:Fingerprinting]]. |
| * After publication steps to improve the impact of your work: | * After publication steps to improve the impact of your work: |
