Differences

This shows you the differences between two versions of the page.

--- start [2026/08/27 14:23] – Link [[Statistics]] namespace landing; security children are written. Authored by Claude. karel.kubicek.claude
+++ start [2026/08/31 19:48] (current) – Link the new Privacy:Browser storage page from the Privacy section. Authored by Claude karel.kubicek.claude
@@ Line 16: / Line 16: @@
     * [[Design:Website selection]] and [[Design:Sampling|Representative sampling methods]], [[Design:Website classification]], and [[Design:IP classification]]
     * [[Design:Crawling location]] and [[Design:Archives|Crawling live or using archives]] ([[Programming:Traffic files|using traffic files, e.g., HAR]])
+    * [[Design:DNS|Measuring DNS]] — which resolver answered, whether the answer was manipulated, and what encrypted DNS changed. Three public resolvers gave completely disjoint answers for 28 of the Tranco top 100 on 2026-08-27.
+    * [[Design:Existing datasets|Measuring from an existing dataset]] — when the measurement is a query (Censys, a CT log, AndroZoo, last year's crawl) rather than a crawl you ran: which snapshot, which query, which join key, and what a producer's port set and vantage point do to your denominator
     * [[Design:Longitudinal|Repeating a measurement over time]] — pinning the list, the browser, the vantage point and the classifier so wave two is comparable to wave one
     * [[Design:Mobile and app measurement|Measuring mobile apps]] — store scraping, static and dynamic analysis, and the certificate-pinning problem that decides whether you saw the traffic at all
-    * [[Design:Platforms|Research of specific large platforms]] such as [[Design:Platforms:Facebook]], [[Design:Platforms:Twitter]], [[Design:Platforms:TikTok]], [[Design:Platforms:Amazon]]
+    * [[Design:Platforms|Research of specific large platforms]] — when the population is one platform rather than a sample of the web, the **access route is the design decision** and it fixes your denominator. Which routes still exist in 2026, what they cost, and why there are no per-company sub-pages.
   * [[Programming]]
     * [[Programming:Crawler|Comparison of crawling libraries]] such as [[Programming:Crawler:OpenWPM]], [[Programming:Crawler:Tracker Radar Collector]], [[Programming:Crawler:PageGraph]], etc.
+    * [[Programming:Crawler Detection|When the website notices your crawler]] — bot management, headless and CDP leaks, CAPTCHAs and challenge pages, and what a blocked crawl does to the headline number
+    * [[Programming:Crawler:LLM Agents|Crawling with an LLM agent]] — Browser Use, BrowserGym/AgentLab and the Computer-Use APIs as a measurement instrument: what five 2026 papers measured about completion rates and per-site cost, what nobody has measured about run-to-run variance, and why this is not yet the default
+    * [[Programming:Filter Lists|Filter lists]] — EasyList, EasyPrivacy, Disconnect and the regional lists as the field's shared labelling instrument //and// its ground truth: which list, which commit, what they miss off the anglophone desktop web, and why "blocked implies tracker" is circular
     * [[Programming:Multilingual support]]
     * [[Programming:Interaction|Interaction with websites]]
     * [[Programming:Registration|Automating login and registration]]
+    * [[Programming:Deployment|Running a measurement for weeks]] — the crawl as a process that must survive: checkpoints, retries, monitoring, cost, and what to report when the machine dies on day four
     * Multitude of pages linked from elsewhere documenting specific technologies, e.g., [[Programming:Tranco]], [[Programming:Cloudflare Radar]], [[Programming:Docker]], [[Programming:CrUX]], [[Programming:Similarweb]].
   * [[Privacy]]
     * Classifying [[Privacy:Requests|Web requests]], [[Privacy:Cookies]], [[Privacy:Fingerprinting]], or [[Privacy:JavaScript]]
+    * [[Privacy:Browser storage|Storage beyond cookies]] — localStorage, IndexedDB, service workers and the caches: what your crawler records, what it silently does not, and why "we cleared cookies" is not a reset
     * [[Privacy:Server side tracking|Measuring server-side tracking]] — when the tracking request never reaches the tracker, and every request-level method quietly stops working
     * [[Privacy:Cookie syncing|Measuring cookie and ID syncing]] — how third parties learn that their two identifiers are the same person, and the crawl configuration that decides whether you see it at all