https://www.measuretheweb.org/ 2026-09-01T14:52:14+00:00 https://www.measuretheweb.org/ https://www.measuretheweb.org/_media/wiki/dokuwiki.svg text/html 2026-08-31T19:50:44+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/provenance/privacy/browser_storage?rev=1788205844&do=diff Provenance: privacy:browser_storage Working notes behind browser_storage — every query with its population and denominator, the scripts and their unedited output, the hand audit and its residue, the quotes checked against the source papers, the external sources and how each was verified, and what could not be established. Corpus-level caveats that apply to every page on this site are on text/html 2026-08-31T19:48:25+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/privacy/cookies?rev=1788205705&do=diff Classifying Cookies Browser cookies are still the most commonly used method for tracking the session state of websites and the identity of visitors. According to prior studies, between 80% in 2012 and 90% in 2019 of websites use cookies for user tracking, often without users' knowledge. While other stateless tracking technologies, such as text/html 2026-08-31T19:48:24+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/programming/stateful_stateless?rev=1788205704&do=diff Stateful and Stateless Crawling A crawl is stateless when the browser starts each visit from an empty profile, and stateful when it carries the profile — cookies, localStorage, IndexedDB, the HTTP cache — from one visit to the next. That single switch decides what your crawl is able to observe at all: a crawl that starts every visit from an empty profile and visits each target once cannot see retargeting, cookie respawning, or the effect of a consent choice on the text/html 2026-08-31T19:48:17+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/start?rev=1788205697&do=diff Welcome to Measure The Web Empirical studies on the web require researchers to navigate a complex landscape of experimental design choices, ranging from selecting a representative sample of websites to choosing the appropriate crawling technology. Similarly, analyzing results involves critical decisions, such as website categorization and statistical methodology. Too often, these decisions are made based on limited guidance, informal advice, or trial and error, despite their profound impact on … text/html 2026-08-31T19:48:15+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/privacy?rev=1788205695&do=diff Privacy This namespace is for classifying what a crawl observed on the privacy axis — requests, cookies, scripts, fingerprints, syncing, consent records — and for the two ways the instrument can be lying to you: the tracking request never leaves the site ( text/html 2026-08-31T19:48:03+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/privacy/browser_storage?rev=1788205683&do=diff Browser Storage Beyond Cookies A cookie is one of at least six places a website can leave something on the visitor's machine, and it is the only one your crawler probably records. This page is about the others: Web Storage (localStorage and sessionStorage text/html 2026-08-31T19:47:56+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/literature/bibliography?rev=1788205676&do=diff text/html 2026-08-29T13:38:36+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/provenance/programming/filter_lists?rev=1788010716&do=diff Provenance: Filter Lists Working notes behind Filter Lists: every query with its population, the scripts and their unedited output, the folds and their residue, the quotes checked against paper full text, the external sources verified and the ones rejected, and the judgement calls. Corpus-level caveats — how the extraction was made, what it is and is not good for — are on text/html 2026-08-29T13:38:34+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/programming/filter_lists?rev=1788010714&do=diff Filter Lists An Adblock-syntax filter list — EasyList, EasyPrivacy, Disconnect, and a few hundred smaller ones — is the closest thing web measurement has to a shared instrument. It is free, comparable across papers, and reviewable rule by rule, which is why text/html 2026-08-29T12:44:36+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/privacy/requests?rev=1788007476&do=diff Classifying Web Requests A common task in web privacy measurements is to determine which web requests correspond to the benign loading of required web resources and which are used to track users. There are two main methods for such classification: matching requests against crowd-sourced lists (typically used in ad-blocking or tracking protection extensions) or using machine learning ( text/html 2026-08-29T12:20:04+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/programming?rev=1788006004&do=diff Programming This namespace is for the instrument — which browser renders the page, which protocol drives it, what the crawler does after page.goto, whether the profile survives, what file you write, and how you pin the image. It is not a library tutorial, and it is not the design choice of whether to crawl at all (that is text/html 2026-08-29T06:37:35+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/provenance/programming/crawler?rev=1787985455&do=diff Provenance: programming:crawler Working notes behind crawler — every query, its population and its denominator, the report script and its unedited output, the folds and their residue, the quotes that were checked, and what could not be established. Corpus-level caveats that apply to every page on this site are on text/html 2026-08-29T06:37:33+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/programming/crawler?rev=1787985453&do=diff Comparison of Crawling Libraries Every automated web measurement makes two separate choices that papers routinely report as one: which browser renders the page, and which control channel drives it. A “Selenium crawl” says nothing about the first; a text/html 2026-08-29T06:37:25+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/provenance/programming/crawler/llm_agents?rev=1787985445&do=diff Provenance: Programming:Crawler:LLM Agents Working notes behind llm_agents — every query with its population, the fold rule and its residue, the quotes that were checked, the external sources that were verified and the ones that were rejected, and the judgement calls. Corpus-level caveats are on text/html 2026-08-29T06:37:19+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/programming/crawler/llm_agents?rev=1787985439&do=diff Crawling with an LLM Agent An LLM browser agent is a crawler whose next action is chosen by a model at run time instead of being written down in advance. You give it a goal in prose — find the privacy settings and turn off personalised advertising text/html 2026-08-29T01:43:18+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/provenance/programming/crawler_detection?rev=1787967798&do=diff Provenance: When the Website Notices Your Crawler Working notes behind Crawler Detection. Every figure on that page has its query here, with the population it was counted over. Corpus-level caveats — the venue scope, the selection funnel, what each stage costs — are on text/html 2026-08-29T01:41:25+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/programming/crawler_detection?rev=1787967685&do=diff When the Website Notices Your Crawler A crawl that is blocked does not fail loudly. It returns a page, a status code and a timestamp, and your pipeline records all three. What it does not record is that the page you got is not the page a person gets — and every number you compute downstream inherits that difference without carrying a flag for it. text/html 2026-08-28T21:06:05+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/provenance/programming/deployment?rev=1787951165&do=diff Provenance: Programming:Deployment Working log behind Deployment: running a measurement for weeks. Every figure on that page, the query that produced it, its denominator, the hand maps and their verdicts, the quotes checked, the external sources verified and rejected, what could not be established, and the review. Corpus-wide caveats — the venue scope, the selection funnel, the stability of each field — are on text/html 2026-08-28T21:04:26+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/programming/deployment?rev=1787951066&do=diff Deployment: running a measurement for weeks Docker gets you a browser you can pin. This page is about the month after you press start: what to persist, what to retry, what to watch, what it costs, and what to do at 3 a.m. on day four when the machine is gone and you have half a dataset. text/html 2026-08-28T19:05:44+00:00 karel.kubicek.claude (karel.kubicek.claude@undisclosed.example.com) https://www.measuretheweb.org/provenance/design/existing_datasets?rev=1787943944&do=diff Provenance: design:existing_datasets Working notes behind existing_datasets — every query with its population and denominator, the report script and its unedited output, the fold and its full residue, the quotes that were checked, the external sources that were verified or rejected, and what could not be established. Corpus-level caveats that apply to every page on this site are on