| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| programming:crawler:openwpm [2026/08/14 08:54] – Apply the generic review pass: fix a data-loss bug in the published smoke script (save_content with no unstructured storage provider drops every body), correct MD5 -> SHA-256 for content_hash, drop the false Xvfb-extraction claim for what the data shows, karel.kubicek.claude | programming:crawler:openwpm [2026/08/21 08:32] (current) – [Open Questions] karelkubicek |
|---|
| OpenWPM {[englehardt2016online]} is the closest thing web privacy measurement has to a standard instrument: a Python platform that drives an **unbranded Firefox** through **Selenium**, and records what the browser did through a **privileged WebExtension** rather than through the automation protocol. It is the most widely shared specialised crawler in our corpus: **60 papers used or extended it**, against 21 each for the next two, on the mention-matching count over all 5,859 papers in [[Programming:Crawler#Which specialised crawlers actually get used|the comparison page]]. 59 of those 60 are inside the 1,120 papers that ran a crawl; the exception re-analysed someone else's OpenWPM data. | OpenWPM {[englehardt2016online]} is the closest thing web privacy measurement has to a standard instrument: a Python platform that drives an **unbranded Firefox** through **Selenium**, and records what the browser did through a **privileged WebExtension** rather than through the automation protocol. It is the most widely shared specialised crawler in our corpus: **60 papers used or extended it**, against 21 each for the next two, on the mention-matching count over all 5,859 papers in [[Programming:Crawler#Which specialised crawlers actually get used|the comparison page]]. 59 of those 60 are inside the 1,120 papers that ran a crawl; the exception re-analysed someone else's OpenWPM data. |
| |
| This page is about running it and reading papers that ran it: which instrumentation surfaces exist and which are silent by default, what stateful and stateless mean //in OpenWPM's implementation//, what the version number commits you to, and what its maintenance looks like today. For the choice between OpenWPM and Playwright, Tracker Radar Collector or a patched browser, see [[Programming:Crawler]]. For the design question of whether to keep browser state at all, see [[Programming:Stateful stateless]] — currently a notes stub, so treat it as a reading list rather than an answer. | This page is about running it and reading papers that ran it: which instrumentation surfaces exist and which are silent by default, what stateful and stateless mean //in OpenWPM's implementation//, what the version number commits you to, and what its maintenance looks like today. For the choice between OpenWPM and Playwright, Tracker Radar Collector or a patched browser, see [[Programming:Crawler]]. For the design question of whether to keep browser state at all, see [[Programming:Stateful stateless]], which also covers what a "reset" actually resets and why the browser's own partitioning defaults now decide part of the answer. |
| |
| <WRAP important> | <WRAP important> |
| ===== An OpenWPM version is a Firefox version, by default ===== | ===== An OpenWPM version is a Firefox version, by default ===== |
| |
| Every OpenWPM release pins one unbranded Firefox build, in one line of ''scripts/install-firefox.sh''. Reading that line out of every ''v*'' tag gives the mapping. The table below omits seven point releases for width (0.11.0, 0.14.1, 0.16.0, 0.19.0, 0.21.0, 0.24.0, 0.33.0); the complete map is in ''report_openwpm.mjs''. It is the reason the README asks for the version number: | Every OpenWPM release pins one unbranded Firefox build, in one line of ''scripts/install-firefox.sh''. Reading that line out of every ''v*'' tag gives the mapping. Note before you read it as an installation recipe: for every release below v0.32.0 the build that line points at **no longer exists** — see [[#Known pitfalls]]. The table below omits seven point releases for width (0.11.0, 0.14.1, 0.16.0, 0.19.0, 0.21.0, 0.24.0, 0.33.0); the complete map is in ''report_openwpm.mjs''. It is the reason the README asks for the version number: |
| |
| ^ OpenWPM ^ Tagged ^ Firefox ^ ^ OpenWPM ^ Tagged ^ Firefox ^ | ^ OpenWPM ^ Tagged ^ Firefox ^ ^ OpenWPM ^ Tagged ^ Firefox ^ |
| ===== Known pitfalls ===== | ===== Known pitfalls ===== |
| |
| | * **It is written to be forked, not imported — and a fork does not survive an upgrade.** There is no ''openwpm'' package on PyPI and no stable public API surface: the documented path clones the repository, builds a conda environment inside it, and expects your crawl script to live in that tree. The obvious thing to do next is to edit the tree, and it is the mistake. Every subsequent upgrade is then a merge against a codebase that has repeatedly rewritten the parts you edited (the whole architecture at v0.10.0, the command and storage interfaces at v0.14.0), so in practice the upgrade never happens: **a crawler built by editing OpenWPM stops being buildable roughly when its pinned Firefox stops being downloadable** (next bullet), which is why a three-year-old artefact so often cannot be run by its own authors. Contrast [[https://github.com/duckduckgo/tracker-radar-collector|DuckDuckGo's Tracker Radar Collector]], which is a library you depend on.\\ **Do this instead: treat it as a library it does not claim to be.** Pin a tagged release as a submodule or a ''git clone --branch v0.xx.0'' in your Dockerfile, keep every line you write in your own repository, and express changes as ''BaseCommand'' subclasses where the API allows and **monkey patches** where it does not — so an upgrade is a version bump plus whatever patches broke, and the patches are a short, reviewable list of exactly where you diverge from upstream. A worked example is the ALSAcnc crawler {[bouhoula2024_automated]}, whose ''cookie_crawler/utils/monkey_patches/'' is five files and 583 lines rebinding ''BrowserManager'', ''BrowserManagerHandle'', ''CommandSequence'', the storage-controller shutdown and the default screen resolution; on that footing it tracked upstream through v0.23.0 → v0.29.0 → v0.31.0 in its public history alone((''bouhoula/alsacnc'', ''docker/crawler/Dockerfile'', commits ''235a510'' (2024-06-22, v0.23.0), ''d5c449a'' (2024-09-24, v0.29.0) and ''3c6cfd1'' (2025-01-29, v0.31.0); the project began on v0.21.x before that repository was published. Read 2026-08-14.)) — while a fork of any of those releases would by now be unbuildable. Upstream agrees this is the right shape and says the project has not made it easy: "//Iirc other people keep the OpenWPM code and their code separated by using a git submodule. Tbh I always assumed that people would just check out one version and build on top of it and never upgrade. […] We should provide a better story here.//"((Maintainer ''vringar'' in [[https://github.com/openwpm/OpenWPM/issues/964|issue #964]], 2021-12-16, which also notes that PRs #743 and #753, released in v0.14.0, made keeping the two codebases separate easier. Read 2026-08-14.)) |
| | * **The pinned Firefox build is deleted after about a year — mirror it or your crawl is not reproducible.** ''scripts/install-firefox.sh'' fetches the unbranded build from Mozilla's **CI index**, not from a release archive, and those artefacts expire: the index entry for the build 0.35.0 pins reports ''expires: 2027-06-10''. We resolved the pinned URL for all 29 releases from v0.10.0 on 2026-08-14; **only the four 2026 releases (Firefox 148–152) still download. Everything up to v0.31.0 — including the Firefox 134 pinned as recently as January 2025 — returns HTTP 404.** ''install.sh'' therefore fails on any older release, which is the state most published artefacts are in; the failure has been [[https://github.com/openwpm/OpenWPM/issues/964|open since 2021]], and the current script at least names it in the error message. The branded build of the same version is still on ''archive.mozilla.org'' and is **not** a substitute — the instrumenting extension needs privileged APIs that only an unbranded build will load.\\ **Do this instead:** archive the ''target.tar.*'' you actually crawled with, alongside your data, and give its version and checksum in the paper. If you are reviving someone else's crawl, the practical escape hatch is the container image — ''docker pull openwpm/openwpm:0.17.0'' still works today (tags go back to 0.15.0, ''linux/amd64'' only) and the image bakes the Firefox binary in at ''/opt/firefox-bin''. Failing both, ask around: a handful of groups keep private copies of these tarballs and pass them between each other, which is not a reproducibility story anyone should be content with. |
| * **An empty database looks exactly like a complete one.** A crawl in which every browser failed to launch still produces a ''crawl-data.sqlite'' containing all 13 of OpenWPM's tables (14 with SQLite's own ''sqlite_sequence''), all of them empty — including ''incomplete_visits'', because a visit that never started is never recorded as incomplete. We produced exactly that file (see [[#Installing it, and what we could and could not run]]). **Print row counts per table and check ''crawl_history.command_status'' before you analyse anything.** A crawl that half-failed is worse than one that failed: it looks like a low-tracking result. | * **An empty database looks exactly like a complete one.** A crawl in which every browser failed to launch still produces a ''crawl-data.sqlite'' containing all 13 of OpenWPM's tables (14 with SQLite's own ''sqlite_sequence''), all of them empty — including ''incomplete_visits'', because a visit that never started is never recorded as incomplete. We produced exactly that file (see [[#Installing it, and what we could and could not run]]). **Print row counts per table and check ''crawl_history.command_status'' before you analyse anything.** A crawl that half-failed is worse than one that failed: it looks like a low-tracking result. |
| * **You are detectable, and specifically as OpenWPM.** Krumnow et al. {[krumnow2022_gullible]} analyse how detectable OpenWPM is in the wild, how resilient its recording is against a page that fights back, and how widespread OpenWPM-specific detection is. A tool 60 papers share is worth writing a detector for. ''bot_mitigation = True'' is not an answer to this: it performs up to ten random mouse moves, one scroll, and a random 1–7 second sleep, and its own documentation says these "aren't comprehensive and automated interaction with the site will still appear very bot-like". | * **You are detectable, and specifically as OpenWPM.** Krumnow et al. {[krumnow2022_gullible]} analyse how detectable OpenWPM is in the wild, how resilient its recording is against a page that fights back, and how widespread OpenWPM-specific detection is. A tool 60 papers share is worth writing a detector for. ''bot_mitigation = True'' is not an answer to this: it performs up to ten random mouse moves, one scroll, and a random 1–7 second sleep, and its own documentation says these "aren't comprehensive and automated interaction with the site will still appear very bot-like". |
| * **Extended JS instrumentation.** "No boundaries" {[acar2020_noboundaries]} extended OpenWPM's JavaScript instrumentation to a further set of DOM properties and function calls — the standard move when ''collection_fingerprinting'' is not enough, and the one to copy. | * **Extended JS instrumentation.** "No boundaries" {[acar2020_noboundaries]} extended OpenWPM's JavaScript instrumentation to a further set of DOM properties and function calls — the standard move when ''collection_fingerprinting'' is not enough, and the one to copy. |
| * Papers also describe "a lightly modified version of OpenWPM" without saying what was modified. If you extend it, publish the diff with your artefact; a modified instrument with no published modification is not a shared instrument. | * Papers also describe "a lightly modified version of OpenWPM" without saying what was modified. If you extend it, publish the diff with your artefact; a modified instrument with no published modification is not a shared instrument. |
| | * **How you modify it decides whether your crawler outlives the paper.** A fork you edit in place cannot practically be upgraded; the same changes written as ''BaseCommand'' subclasses and monkey patches over a pinned release can be, and have been. This is the first entry under [[#Known pitfalls]], with a worked example. |
| |
| **33 of the 60 (55.0%) release an artefact publicly**, against 51.8% of other crawling papers — no better than the field, for a tool whose whole value is shared instrumentation. | **33 of the 60 (55.0%) release an artefact publicly**, against 51.8% of other crawling papers — no better than the field, for a tool whose whole value is shared instrumentation. |
| - **''tp_cookies'', ''donottrack'', and any ''prefs''** you set. The defaults are a policy choice, and ''tp_cookies'' defaults to accepting everything. | - **''tp_cookies'', ''donottrack'', and any ''prefs''** you set. The defaults are a policy choice, and ''tp_cookies'' defaults to accepting everything. |
| - **Failure accounting**: how many visits are in ''incomplete_visits'', how many commands have a non-''ok'' ''command_status'', and what you did about them. This is the number nobody publishes and every reviewer should ask for. | - **Failure accounting**: how many visits are in ''incomplete_visits'', how many commands have a non-''ok'' ''command_status'', and what you did about them. This is the number nobody publishes and every reviewer should ask for. |
| | - **The Firefox binary itself**, archived with your artefact and checksummed. Mozilla's CI deletes the unbranded build about a year after it is produced, so a version number alone does not let anyone rebuild your crawler (see [[#Known pitfalls]]). |
| - **Your diff**, if you modified the instrumentation, with the artefact (see [[:Artifacts]], a page this wiki still owes you). | - **Your diff**, if you modified the instrumentation, with the artefact (see [[:Artifacts]], a page this wiki still owes you). |
| - **The citation the project asks for**, which is the CCS 2016 paper {[englehardt2016online]} — its README says so — in the same sentence as the version, so a reader gets the method and the build together. | - **The citation the project asks for**, which is the CCS 2016 paper {[englehardt2016online]} — its README says so — in the same sentence as the version, so a reader gets the method and the build together. |
| ===== Open Questions ===== | ===== Open Questions ===== |
| |
| <wrap todo> | <WRAP todo> |
| * Nobody has published what OpenWPM's default ''collection_fingerprinting'' misses on a modern fingerprinting corpus. The list has not changed since 2020 and the surface has; measuring the gap is a small paper. | * Nobody has published what OpenWPM's default ''collection_fingerprinting'' misses on a modern fingerprinting corpus. The list has not changed since 2020 and the surface has; measuring the gap is a small paper. |
| * How much the unbranded Firefox build differs from a stock Firefox of the same version, in fingerprint and in behaviour, is unmeasured — the same question [[Programming:Crawler]] raises for Playwright's patched builds. | * How much the unbranded Firefox build differs from a stock Firefox of the same version, in fingerprint and in behaviour, is unmeasured — the same question [[Programming:Crawler]] raises for Playwright's patched builds. |
| * Whether the per-browser cookie-jar partition in a stateful multi-browser crawl changes published results, and by how much. Every stateful OpenWPM crawl with ''num_browsers > 1'' has this property; no paper we found reports ''num_browsers'' alongside a stateful claim. | * Whether the per-browser cookie-jar partition in a stateful multi-browser crawl changes published results, and by how much. Every stateful OpenWPM crawl with ''num_browsers > 1'' has this property; no paper we found reports ''num_browsers'' alongside a stateful claim. |
| * Where ARM support would come from: Mozilla's ''add-on-devel'' build matrix, or an OpenWPM-side path that does not need an unbranded build. | * Where ARM support would come from: Mozilla's ''add-on-devel'' build matrix, or an OpenWPM-side path that does not need an unbranded build. |
| </wrap> | </WRAP> |
| |
| ====== References ====== | ====== References ====== |
