Measure The Web

User Tools

Site Tools

Trace:
programming:crawler:openwpm

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
programming:crawler:openwpm [2026/08/14 10:26] – Add two pitfalls from practice: OpenWPM is fork-shaped not library-shaped (monkey-patch over a pinned release instead), and the pinned unbranded Firefox expires from Mozilla CI after ~1 year (mirror it; container images are the escape hatch). Authored by karel.kubicek.claudeprogramming:crawler:openwpm [2026/08/21 08:32] (current) – [Open Questions] karelkubicek
Line 3: Line 3:
 OpenWPM {[englehardt2016online]} is the closest thing web privacy measurement has to a standard instrument: a Python platform that drives an **unbranded Firefox** through **Selenium**, and records what the browser did through a **privileged WebExtension** rather than through the automation protocol. It is the most widely shared specialised crawler in our corpus: **60 papers used or extended it**, against 21 each for the next two, on the mention-matching count over all 5,859 papers in [[Programming:Crawler#Which specialised crawlers actually get used|the comparison page]]. 59 of those 60 are inside the 1,120 papers that ran a crawl; the exception re-analysed someone else's OpenWPM data. OpenWPM {[englehardt2016online]} is the closest thing web privacy measurement has to a standard instrument: a Python platform that drives an **unbranded Firefox** through **Selenium**, and records what the browser did through a **privileged WebExtension** rather than through the automation protocol. It is the most widely shared specialised crawler in our corpus: **60 papers used or extended it**, against 21 each for the next two, on the mention-matching count over all 5,859 papers in [[Programming:Crawler#Which specialised crawlers actually get used|the comparison page]]. 59 of those 60 are inside the 1,120 papers that ran a crawl; the exception re-analysed someone else's OpenWPM data.
  
-This page is about running it and reading papers that ran it: which instrumentation surfaces exist and which are silent by default, what stateful and stateless mean //in OpenWPM's implementation//, what the version number commits you to, and what its maintenance looks like today. For the choice between OpenWPM and Playwright, Tracker Radar Collector or a patched browser, see [[Programming:Crawler]]. For the design question of whether to keep browser state at all, see [[Programming:Stateful stateless]] — currently a notes stub, so treat it as a reading list rather than an answer.+This page is about running it and reading papers that ran it: which instrumentation surfaces exist and which are silent by default, what stateful and stateless mean //in OpenWPM's implementation//, what the version number commits you to, and what its maintenance looks like today. For the choice between OpenWPM and Playwright, Tracker Radar Collector or a patched browser, see [[Programming:Crawler]]. For the design question of whether to keep browser state at all, see [[Programming:Stateful stateless]], which also covers what a "reset" actually resets and why the browser's own partitioning defaults now decide part of the answer.
  
 <WRAP important> <WRAP important>
Line 592: Line 592:
 ===== Open Questions ===== ===== Open Questions =====
  
-<wrap todo>+<WRAP todo>
   * Nobody has published what OpenWPM's default ''collection_fingerprinting'' misses on a modern fingerprinting corpus. The list has not changed since 2020 and the surface has; measuring the gap is a small paper.   * Nobody has published what OpenWPM's default ''collection_fingerprinting'' misses on a modern fingerprinting corpus. The list has not changed since 2020 and the surface has; measuring the gap is a small paper.
   * How much the unbranded Firefox build differs from a stock Firefox of the same version, in fingerprint and in behaviour, is unmeasured — the same question [[Programming:Crawler]] raises for Playwright's patched builds.   * How much the unbranded Firefox build differs from a stock Firefox of the same version, in fingerprint and in behaviour, is unmeasured — the same question [[Programming:Crawler]] raises for Playwright's patched builds.
   * Whether the per-browser cookie-jar partition in a stateful multi-browser crawl changes published results, and by how much. Every stateful OpenWPM crawl with ''num_browsers > 1'' has this property; no paper we found reports ''num_browsers'' alongside a stateful claim.   * Whether the per-browser cookie-jar partition in a stateful multi-browser crawl changes published results, and by how much. Every stateful OpenWPM crawl with ''num_browsers > 1'' has this property; no paper we found reports ''num_browsers'' alongside a stateful claim.
   * Where ARM support would come from: Mozilla's ''add-on-devel'' build matrix, or an OpenWPM-side path that does not need an unbranded build.   * Where ARM support would come from: Mozilla's ''add-on-devel'' build matrix, or an OpenWPM-side path that does not need an unbranded build.
-</wrap>+</WRAP>
  
 ====== References ====== ====== References ======
programming/crawler/openwpm.1786703170.txt.gz · Last modified: by karel.kubicek.claude
Except where otherwise noted, content on this wiki is licensed under the following license: CC BY-NC-SA 4.0
CC BY-NC-SA 4.0 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki