Differences

This shows you the differences between two versions of the page.

--- programming:crawler [2026/08/06 07:23] – Playwright: separate the shipped-Chromium/patched-Firefox distinction, add the headless-shell and channel caveats with a docs citation; OpenWPM storage backends and the no-Windows footnote. Authored by Claude. karel.kubicek.claude
+++ programming:crawler [2026/08/29 06:37] (current) – LLM-agent fold: new family row (4 papers), bespoke own-name 184->181, union 321->318 (28.7%->28.4%), 75->74, residue 204/210->199/205; new Agent-Driven Crawling section pointing at programming:crawler:llm_agents. Authored by Claude karel.kubicek.claude
@@ Line 3: / Line 3: @@
 Every automated web measurement makes two separate choices that papers routinely report as one: **which browser** renders the page, and **which control channel** drives it. A "Selenium crawl" says nothing about the first; a "Chrome crawl" says nothing about the second. They fail differently, they are detected differently, and they give you access to different data.
-This page compares the options. It covers the generic automation libraries ([[#Selenium|Selenium]], [[#Puppeteer|Puppeteer]], [[#Playwright|Playwright]], [[#Plain CDP|plain CDP]]) and then the specialised privacy and security crawlers built on top of them, each of which has — or will have — its own page: [[Programming:Crawler:OpenWPM]], [[Programming:Crawler:webXray]], [[Programming:Crawler:Tracker Radar Collector]], [[Programming:Crawler:PageGraph]], [[Programming:Crawler:Foxhound]], [[Programming:Crawler:PanoptiChrome]].
+This page compares the options. It covers the generic automation libraries ([[#Selenium|Selenium]], [[#Puppeteer|Puppeteer]], [[#Playwright|Playwright]], [[#Plain CDP|plain CDP]]) and then the specialised privacy and security crawlers built on top of them, each of which has — or will have — its own page: [[Programming:Crawler:OpenWPM]], [[Programming:Crawler:webXray]], [[Programming:Crawler:Tracker Radar Collector]], [[Programming:Crawler:PageGraph]], [[Programming:Crawler:Foxhound]], [[Programming:Crawler:PanoptiChrome]]. A newer child, [[Programming:Crawler:LLM Agents]], covers the one instrument on this page that is not a control channel at all: an agent that decides the next click with a model.
 It pairs with [[Design:Automated measurements]] (whether to crawl at all), [[Design:Crawling location]] (where from), [[Programming:Stateful stateless]] (with or without a profile), [[Programming:Interaction]] (what to do on the page), and [[Privacy:Consent]] (what to do with the banner).
 <WRAP important>
-The single most consequential thing on this page is not which library is best. It is that **only 11.6% of the papers in our corpus that name an automation tool also state its version** (see [[#Use in Publications]]). "We used Selenium" spans fifteen years of incompatible releases and two different wire protocols. Report the version.
+The single most consequential thing on this page is not which library is best. It is that **only 12.0% of the papers in our corpus that name an automation tool also state its version** (see [[#Use in Publications]]). "We used Selenium" spans fifteen years of incompatible releases and two different wire protocols. Report the version.
 </WRAP>
@@ Line 15: / Line 15: @@
   * **The browser** is what the website sees: a rendering engine, a JavaScript engine, a TLS stack, a fingerprint. Chrome and Firefox disagree about cookie partitioning, about tracking protection defaults, and about which APIs exist at all, so they see different tracking. OmniCrawl {[cassel2022_omnicrawl]} drove 42 non-emulated desktop and mobile browsers simultaneously precisely because a single-browser crawl is a single-browser result.
   * **The control channel** is how your code talks to it. There are exactly three in practice:
-    * **W3C WebDriver** — an HTTP protocol spoken by a per-browser driver process (''chromedriver'', ''geckodriver''). Portable across browsers; deliberately models a *user*, not a *debugger*.
+    * **W3C WebDriver** — an HTTP protocol spoken by a per-browser driver process (''chromedriver'', ''geckodriver''). Portable across browsers; classic WebDriver deliberately models a //user//, not a //debugger//, and has no network commands at all. Its successor **WebDriver BiDi** adds them, over a WebSocket, standardised and cross-browser — see [[#Selenium and WebDriver BiDi]].
     * **The Chrome DevTools Protocol (CDP)** — a WebSocket protocol into Chromium's internals. Gives you the network layer, the JS engine and the cookie jar. Chromium only.((Firefox implements a partial CDP shim and, more actively, WebDriver BiDi. Playwright drives Firefox and WebKit through its own patched builds rather than through CDP.))
     * **Nothing** — a patched browser build that instruments itself and writes its own logs. This is what [[#Specialised Measurement Crawlers|the specialised crawlers]] do.
-The practical consequence is a hard boundary: **WebDriver cannot capture network requests.** There is no such command in the specification. If your research question involves third-party requests, cookies set by responses, or JavaScript API calls — which is to say, most privacy measurement — WebDriver alone will not answer it, and you will end up adding a proxy, a browser extension, or a CDP side-channel. That gap is the reason the specialised crawlers exist.
+For most of the period this page's corpus covers, that produced a hard boundary: **classic WebDriver cannot capture network requests.** There is no such command in the specification. If your research question involves third-party requests, cookies set by responses, or JavaScript API calls — which is to say, most privacy measurement — classic WebDriver will not answer it, and you end up adding a proxy, a browser extension, or a CDP side-channel. That gap is a large part of why the specialised crawlers exist, and it is why a paper that says only "we used Selenium" leaves its instrument unidentified.
+WebDriver BiDi closes the boundary, and it is new enough that essentially none of the corpus could have used it. If you are starting a crawler today it changes the recommendation; it does not change how to read the literature.
 ===== The Generic Libraries =====
 ^ ^ Selenium 4 ^ Puppeteer ^ Playwright ^ Plain CDP ^
-^ Control channel | W3C WebDriver (+ optional CDP bridge on Chromium) | CDP | CDP (Chromium), patched builds (Firefox, WebKit) | CDP |
+^ Control channel | classic WebDriver, **WebDriver BiDi**, or a CDP bridge on Chromium | CDP | CDP (Chromium), own protocol for its Firefox/WebKit builds | CDP |
-^ Browsers | Chrome, Firefox, Edge, Safari, and anything with a driver | Chrome, Chromium, Firefox //(experimental)// | Chromium, Firefox, WebKit — Playwright's **own** builds; real Chrome/Edge via ''channel'' | Chromium only |
+^ Browsers | Chrome, Firefox, Edge, Safari, and anything with a driver | Chrome, Chromium, and Firefox — production-ready since v23, over WebDriver BiDi | Chromium, Firefox, WebKit — Playwright's **own** builds; real Chrome/Edge via ''channel'' | Chromium only |
 ^ Languages | Java, Python, JS, C#, Ruby, Rust… | JS/TS (''pyppeteer'' unmaintained) | JS/TS, Python, Java, .NET | any language with a WebSocket client |
-^ Network interception | ✗ not in WebDriver | ✓ | ✓ | ✓ |
+^ Network interception | ✗ classic WebDriver; ✓ via BiDi or the CDP bridge | ✓ | ✓ | ✓ |
-^ Response bodies | ✗ | ✓ | ✓ | ✓ |
+^ Response bodies | ✗ classic; via BiDi/CDP otherwise | ✓ | ✓ | ✓ |
-^ All cookies incl. ''HttpOnly'' | partial — per-domain, via the driver | ✓ (''Network.getAllCookies'') | ✓ | ✓ |
+^ All cookies incl. ''HttpOnly'' | ✓ — the driver returns them, unlike in-page JS | ✓ (''Network.getAllCookies'') | ✓ | ✓ |
 ^ Waits for the network to settle | ✗ you sleep | ✓ ''networkidle0'' | ✓ ''networkidle'' | ✗ you sleep |
 ^ Multiple isolated profiles per browser | ✗ one session per browser process | contexts | contexts, cheap and first-class | ''Target.createBrowserContext'' |
 ^ Ships its own browser | ✗ — Selenium Manager fetches a matching //driver//, and will fetch a browser if it finds none | ✓ pinned //Chrome for Testing// build | ✓ pinned, patched builds | ✗ |
-^ Best for | multi-browser and multi-language work; anything Firefox | Chromium-only measurement; the base of most modern tooling | reproducibility, parallelism, cross-engine work | minimal dependencies, exotic domains, other languages |
+^ Best for | multi-language work; an existing Selenium codebase | Chromium measurement; the base of most modern research tooling | reproducibility, parallelism, cross-engine work | minimal dependencies, exotic CDP domains, other languages |
 ==== Selenium ====
-Selenium is the field's default: **21.8% of all crawling papers in the corpus name it**, and it has been the most-used tool in every four-year window since 2014. Its virtues are real — it drives Firefox properly, it has first-class bindings in Python and Java, and code written against it in 2016 mostly still runs.
+Selenium is the field's default: **21.6% of all crawling papers in the corpus name it**, and it has been the most-used off-the-shelf library in every four-year window since 2014. Its virtues are real — it drives Firefox properly, it has first-class bindings in Python and Java, and code written against it in 2016 mostly still runs.
-Its limitation is architectural. WebDriver was designed to test websites, so it exposes what a user can do and hides what a debugger can see. In Selenium 4 the escape hatch is a CDP session tunnelled over the WebDriver connection, which works but is Chromium-only and gives up the portability you chose Selenium for. This is worth stating plainly because it is the most common silent methodological compromise in the corpus: a paper says "Selenium" and reports third-party requests, and the reader cannot tell whether those came from a proxy, an extension, ''performance.getEntriesByType("resource")'', or CDP — four instruments with four different blind spots.
+Its limitation was architectural. Classic WebDriver was designed to test websites, so it exposes what a user can do and hides what a debugger can see. The older escape hatch is a CDP session tunnelled over the WebDriver connection, which works but is Chromium-only and gives up the portability you chose Selenium for; the modern answer is [[#Selenium and WebDriver BiDi|WebDriver BiDi]]. This is worth stating plainly because it is the most common silent methodological compromise in the corpus: a paper says "Selenium" and reports third-party requests, and the reader cannot tell whether those came from a proxy, an extension, ''performance.getEntriesByType("resource")'', or CDP — four instruments with four different blind spots.
 <file javascript selenium.mjs>
 // Selenium 4 (W3C WebDriver) driving Chromium through chromedriver.
 //
-// The point of this script is what is missing from it. WebDriver has no
+// The point of this script is what is missing from it. Classic WebDriver has no
 // network-interception command, so there is no `page.on('request')` to write:
 // cookies and the DOM you can have, the request log you cannot. What follows is
-// the two escape hatches researchers actually use.
+// the two escape hatches researchers actually used before WebDriver BiDi.
 import { Builder, Browser } from 'selenium-webdriver';
 import chrome from 'selenium-webdriver/chrome.js';
@@ Line 95: / Line 97: @@
 <WRAP alert>
-That CDP bridge is racy. In five runs of the script above against an identical local page, four recorded all four requests and **one recorded only the main document** — the listener attaches asynchronously, and ''Network.enable'' is not synchronised with the navigation that follows.((Measured in the sandbox described in [[#What We Ran]], five consecutive runs, 2026-08-06.)) If you build on this, assert on the number of requests you expected rather than trusting the log.
+That CDP bridge is racy. In five runs of the script above against an identical local page, four recorded all four requests and **one recorded only the main document**. We have not isolated the mechanism — the listener is attached before the navigation, so this is not simply "subscribed too late" — but the outcome reproduced on a second machine, and it is a silent 75% data loss with no error.((Measured in the sandbox described in [[#What We Ran]], five consecutive runs, 2026-08-06; independently reproduced (4, 4, 1, 4, 4) in a separate five-run sequence during review of this page.)) If you build on this, assert on the number of requests you expected rather than trusting the log.
 </WRAP>
+==== Selenium and WebDriver BiDi ====
+The paragraph above is the historical picture, and it is the one the literature was written under. It is no longer the whole story: **WebDriver BiDi** is a bidirectional WebSocket protocol standardised alongside WebDriver, and Selenium exposes it directly. Turn it on with ''options.enableBidi()'' and you subscribe to network events without CDP and without giving up cross-browser portability, because BiDi is a W3C protocol that Firefox implements too.
+<file javascript selenium_bidi.mjs>
+// Selenium 4 using WebDriver BiDi rather than classic WebDriver.
+//
+// MODE = 'req' | 'resp' | 'both'. With 'both', every callback receives both event
+// types in selenium-webdriver 4.46 -- see the table below the listing. Nothing
+// here is asserted: the response data is read off the events.
+import { Builder, Browser } from 'selenium-webdriver';
+import chrome from 'selenium-webdriver/chrome.js';
+import { Network } from 'selenium-webdriver/bidi/network.js';
+const MODE = process.argv[2] ?? 'both';
+const options = new chrome.Options();
+options.setChromeBinaryPath('/path/to/chrome');
+options.addArguments('--headless', '--no-sandbox', '--disable-dev-shm-usage');
+options.enableBidi(); // sets the webSocketUrl capability
+const service = new chrome.ServiceBuilder('/path/to/chromedriver');
+const driver = await new Builder()
+  .forBrowser(Browser.CHROME)
+  .setChromeOptions(options)
+  .setChromeService(service)
+  .build();
+const requestEvents = [];
+const responseEvents = [];
+try {
+  const network = await Network(driver);
+  if (MODE !== 'resp')
+    await network.beforeRequestSent((e) => requestEvents.push({ method: e.request.method, url: e.request.url }));
+  if (MODE !== 'req')
+    await network.responseCompleted((e) =>
+      // e.response is undefined on the cross-delivered request events, so read
+      // what is actually there rather than what should be.
+      responseEvents.push({ url: e.request.url, status: e.response === undefined ? null : e.response.status })
+    );
+  await driver.get('http://127.0.0.1:8099/');
+  await driver.sleep(1500);
+  console.log({
+    requestEvents: requestEvents.length,
+    responseEvents: responseEvents.length,
+    responseEventsWithData: responseEvents.filter((r) => r.status !== null).length,
+    statuses: responseEvents.map((r) => r.status),
+    uniqueUrls: [...new Set([...requestEvents, ...responseEvents].map((r) => r.url))],
+  });
+} finally {
+  await driver.quit();
+}
+</file>
+Subscribe to **one** network event and it is exactly right. Subscribe to **two** and, in ''selenium-webdriver'' 4.46, every callback starts receiving //both// event types. Measured on the fixture page, whose four requests we know:
+^ Handlers registered ^ ''beforeRequestSent'' fired ^ ''responseCompleted'' fired ^ …of which carried response data ^ Statuses seen ^
+| ''beforeRequestSent'' only | 4 | — | — | — |
+| ''responseCompleted'' only | — | 4 | 4 | 200, 200, 200, 204 |
+| **both** | **8** | **8** | **4** | ''null'', 200, ''null'', ''null'', 200, 200, ''null'', 204 |
+Both counts double, and half the "responses" are request events wearing a response event's clothing: ''e.response'' is ''undefined'' for them, so the obvious ''e.response.status'' throws on roughly every other event. Nothing warns you.
+<WRAP alert>
+Two consequences, in order of how badly they bite:
+  - **Do not paper over this by deduplicating on URL.** Of the two copies of each event, one carries the response data and one does not, and a URL-keyed dedupe has no way to prefer the right one — you can keep the empty copy and conclude the status was unavailable. We did exactly that, and published the wrong claim before catching it. Filter on the field you need (''e.response !== undefined'') instead.
+  - **Register one handler, or verify the counts.** The straightforward workaround is to subscribe only to ''responseCompleted'', which carries the request //and// the response. If you need both events, assert your event count against a page whose requests you already know before you point the crawler at a hundred thousand sites.
+</WRAP>
+So BiDi is the right direction and, in this binding, not yet a drop-in replacement. The protocol is not at fault: the W3C specification defines a ''response'' field of type ''network.ResponseData'' on ''network.responseCompleted'', and Chromium delivers it. Check what //your// binding does with it.
 ==== Puppeteer ====
-Puppeteer is a thin, well-documented wrapper over CDP maintained by the Chrome team. It appears in **6.3% of crawling papers**, first in 2018, and it is what most of the modern research tooling is built on — DuckDuckGo's [[#Tracker Radar Collector]] and Brave's ''pagegraph-crawl'' are both Puppeteer programs. Recent versions download and pin a //Chrome for Testing// build, which fixes the "which Chrome was that?" reproducibility problem for free, provided you record the version.
+Puppeteer is a thin, well-documented wrapper over CDP maintained by the Chrome team. It appears in **6.8% of crawling papers**, first in 2018, and it is what most of the modern research tooling is built on — DuckDuckGo's [[Programming:Crawler:Tracker Radar Collector|Tracker Radar Collector]] and Brave's ''pagegraph-crawl'' are both Puppeteer programs. Recent versions download and pin a //Chrome for Testing// build, which fixes the "which Chrome was that?" reproducibility problem for free, provided you record the version.
-The trade-off is that Puppeteer is Chromium-and-CDP all the way down. Firefox support exists but is not the reason anyone picks Puppeteer.
+Puppeteer used to be Chromium-and-CDP all the way down. It is not any more: it supports Firefox over WebDriver BiDi, and when you launch Firefox, BiDi is the default. Chrome still defaults to CDP "since not all CDP features are supported by WebDriver BiDi yet", and a Puppeteer feature that BiDi cannot express throws ''UnsupportedOperation''.((Puppeteer documentation, [[https://pptr.dev/webdriver-bidi|WebDriver BiDi support]], checked 2026-08-06.)) So Puppeteer is now a plausible cross-engine choice, with the caveat that the two engines do not give you the same API surface — check that the calls your measurement depends on work on both before you report a cross-browser comparison.
 <file javascript puppeteer.mjs>
@@ Line 133: / Line 209: @@
 Playwright is the newest of the three and the only one that drives three different engines — Chromium, Firefox and WebKit — through **browser builds it ships itself**, pinned per Playwright version. For a measurement study that is the interesting property: the browser binary is part of the dependency, so naming the Playwright version pins the engine too, and a cross-engine comparison stops requiring three separate harnesses.
-It is still rare in the literature — **10 papers, all of them 2022 or later**, the youngest tool on this page — but it is the only one whose adoption curve is still going up steeply, and the small group using it reports its version far more often than anyone else (see [[#Use in Publications]]).
+It is the youngest tool on this page — **34 papers (3.0% of crawling papers), all of them 2022 or later** — and the only one whose adoption curve is still going up steeply: in the provisional 2025–2026 window it **overtakes Puppeteer**, 9.6% against 8.1%. Its users also report its version better than most, though the margin is narrow: 26.5%, just ahead of OpenWPM's 25.9% and more than double everyone else (see [[#Use in Publications]]).
 Three caveats for measurement work.
@@ Line 165: / Line 241: @@
 ==== Plain CDP ====
-You can skip the libraries entirely: launch Chromium with ''--remote-debugging-port'', open a WebSocket, enable the domains you want, and read the events. **35 papers (4.1%) did**, most of them between 2018 and 2021 — a window that closes as Puppeteer matures.
+You can skip the libraries entirely: launch Chromium with ''--remote-debugging-port'', open a WebSocket, enable the domains you want, and read the events. **42 papers (3.8%) did**, most of them between 2018 and 2021 — a window that closes as Puppeteer matures.
 Reach for it when your analysis lives in a language with no good automation binding, when you need a CDP domain the wrapper does not expose, or when you want to know exactly what your instrument is doing. The cost, visible in the code below, is that everything a library gives you is now yours to write: waiting for the port, waiting for the load, waiting for the network to settle, and reconnecting when a target dies.
@@ Line 222: / Line 298: @@
 ==== What We Ran ====
-The four scripts above were run against an identical local fixture page — one HTML document, one image, one script that sets a cookie and fires a ''fetch()'', one ''Set-Cookie'' response header on each — so that the differences in the table below are differences between instruments and not between page loads. Chromium 151.0.7922.34 (Playwright's build), Node 22.23, aarch64 Linux, 2026-08-06.((Our own run for this page, in a Linux container, 2026-08-06. The four scripts are exactly the code shown above; the fixture is a 40-line Node HTTP server. Numbers are one representative run of five. See the page history for who ran it.))
+The four scripts above were run against an identical local fixture page — one HTML document, one image, one script that sets a cookie and fires a ''fetch()'', one ''Set-Cookie'' response header on each — so that the differences in the table below are differences between instruments and not between page loads. Chromium 151.0.7922.34 (Playwright's build), Node 22.23, aarch64 Linux, 2026-08-06.((Our own runs for this page, in a Linux container, 2026-08-06. The listings above are the programs, with the sandbox's absolute paths replaced by placeholders and a little JSON-shaping trimmed; the fixture is a 40-line Node HTTP server. Numbers are one representative run of five, except where a row says otherwise. See the page history for who ran them.))
-^ Library ^ Lines of code ^ Requests seen ^ Response statuses ^ Cookies ^ Saw the ''HttpOnly'' cookie ^ Wall clock ^
+^ Library ^ Lines((Non-comment, non-blank lines of the listing above, which is the whole program.)) ^ Requests seen ^ Response statuses ^ Cookies ^ Saw the ''HttpOnly'' cookie ^ Wall clock ^
 | Playwright | 15 | 4 | 4 | 3 | yes | 779 ms |
 | Puppeteer | 18 | 4 | 4 | 3 | yes | 877 ms |
-| Plain CDP | 49 | 4 | 4 | 3 | yes | 1329 ms |
+| Selenium 4, WebDriver BiDi | 36 | 4 //unique, from 8 events// | 4 //of 8 response events carried one// | 3 | yes | 2038 ms |
-| Selenium 4 | 39 | 4 //(via the CDP bridge; 1 in one run of five)// | 0 //(not exposed)// | 3 | yes | 1838 ms |
+| Selenium 4, CDP bridge | 33 | 4 //(1 in one run of five)// | 0 //(not exposed)// | 3 | yes | 1838 ms |
+| Plain CDP | 39 | 4 | 4 | 3 | yes | 1329 ms |
 Three things in that table are worth more than the timings:
-  - **Selenium needed 39 lines to get less data.** The requests came from the CDP bridge, not from Selenium; response status codes were not available at all. Its portable fallback, the Resource Timing API, returned **3** entries rather than 4 — it does not report the main document, and a page can erase it with ''performance.clearResourceTimings()''.
+  - **Both Selenium paths cost more code and delivered less.** Neither returned response status codes: the CDP bridge does not surface them without extra handlers, and the BiDi response event does not carry them at all. Selenium's portable fallback, the Resource Timing API, returned **3** entries rather than 4 — it does not report the main document, and a page can erase it with ''performance.clearResourceTimings()''.
   - **All four saw the ''HttpOnly'' cookie**, which is the one thing a JavaScript-based instrument (''document.cookie'', an in-page script) cannot see. If a paper reports cookie counts collected in-page, its numbers exclude session cookies by construction.
   - **The wall-clock ordering is mostly setup cost**, not throughput. Do not choose a library on these numbers; choose it on the columns of the [[#The Generic Libraries|comparison table]].
@@ Line 242: / Line 319: @@
 ===== Specialised Measurement Crawlers =====
-The generic libraries give you the network layer and the DOM. They do not tell you *which script* set a cookie, *which* fingerprinting API was called with what arguments, or *how* a value flowed from ''document.cookie'' into a request URL. Getting that means instrumenting the browser itself, and that is the entire business of the tools below.
+The generic libraries give you the network layer and the DOM. They do not tell you //which script// set a cookie, //which// fingerprinting API was called with what arguments, or //how// a value flowed from ''document.cookie'' into a request URL. Getting that means instrumenting the browser itself, and that is the entire business of the tools below.
-^ Tool ^ Base ^ Automation ^ What it adds over a generic library ^ Output ^ Setup ^ Papers in corpus ^
+^ Tool ^ Base ^ Automation ^ What it adds over a generic library ^ Output ^ Setup ^ Papers((All from the same source: the mention-matching table in [[#Which specialised crawlers actually get used]], over all 5,859 papers. Do not compare these against the folded framework table, whose population is the 1,120 crawling papers.)) ^
-| [[Programming:Crawler:OpenWPM|OpenWPM]] {[englehardt2016online]} | unbranded Firefox | Selenium + geckodriver | JS API call monitoring, HTTP request/response, navigation, callstack, DNS, cookie instruments; per-browser config; profile management | SQLite or Parquet for structured data, LevelDB or gzip for response bodies, S3/GCS remotes | Python/conda, one install script; **no Windows**((''OpenWPM does not support windows'' — README, and [[https://github.com/openwpm/OpenWPM/issues/503|issue #503]].)) | **51** |
+| [[Programming:Crawler:OpenWPM|OpenWPM]] {[englehardt2016online]} | unbranded Firefox | Selenium + geckodriver | JS API call monitoring, HTTP request/response, navigation, DNS and cookie instruments, configurable per browser((The ''callstack_instrument'', which recorded JS call stacks for HTTP requests, is **currently broken** and raises a ''ConfigError'' if enabled — ''docs/Configuration.md'' and [[https://github.com/openwpm/OpenWPM/issues/557|issue #557]].)); profile management | SQLite or Parquet for structured data, LevelDB or gzip for response bodies, S3/GCS remotes | Python/conda, one install script; **no Windows**((''OpenWPM does not support windows'' — README, and [[https://github.com/openwpm/OpenWPM/issues/503|issue #503]].)) | **60** |
-| [[Programming:Crawler:Tracker Radar Collector|Tracker Radar Collector]] | Chromium | Puppeteer | modular collectors (requests, cookies, API calls, screenshots, cookie popups); built-in autoconsent opt-in/opt-out; mobile emulation; SOCKS proxy | one JSON per site + ''metadata.json'' | ''npm i''; easiest on this list | **8** |
+| [[Programming:Crawler:Tracker Radar Collector|Tracker Radar Collector]] | Chromium | Puppeteer | modular collectors (requests, cookies, API calls, screenshots, cookie popups); built-in autoconsent opt-in/opt-out; mobile emulation; SOCKS proxy | one JSON per site + ''metadata.json'' | ''npm i''; easiest on this list | **21** |
-| [[Programming:Crawler:PageGraph|Brave PageGraph]] | patched Brave (Blink + V8) | ''pagegraph-crawl'' (Node) | a causal **graph**: every DOM mutation, script execution and request attributed to its cause, plus Shields filter-rule effects; the production successor to AdGraph {[iqbal2020_adgraph]} | GraphML | in Brave ≥ 1.46, but JS-API recording needs Brave **Nightly** | **4** |
+| [[Programming:Crawler:PageGraph|Brave PageGraph]] | patched Brave (Blink + V8) | ''pagegraph-crawl'' (Node) | a causal **graph**: every DOM mutation, script execution and request attributed to its cause, plus Shields filter-rule effects; the production successor to AdGraph {[iqbal2020_adgraph]} | GraphML | in Brave ≥ 1.46, but JS-API recording needs Brave **Nightly** | **8** |
-| VisibleV8 {[jueckstock2019_visiblev8]} | patched Chromium (V8) | any CDP library | native-code JS API tracing inside V8, below the reach of page JavaScript | append-only trace logs | custom Chromium build | **8** |
+| VisibleV8 {[jueckstock2019_visiblev8]} | patched Chromium (V8) | any CDP library | native-code JS API tracing inside V8, below the reach of page JavaScript | append-only trace logs | custom Chromium build | **9** |
-| [[Programming:Crawler:Foxhound|SAP Project Foxhound]] | patched Firefox (Gecko + SpiderMonkey) | Playwright or Selenium | **dynamic taint tracking**: string-level flows from sources to sinks | ''__taintreport'' DOM events | full Firefox build toolchain, or prebuilt binaries | **2** |
+| [[Programming:Crawler:Foxhound|SAP Project Foxhound]] | patched Firefox (Gecko + SpiderMonkey) | Playwright((The only automation layer the repository documents; it carries its own ''.PLAYWRIGHT_VERSION'' and a CI workflow for it.)) | **dynamic taint tracking**: string-level flows from sources to sinks | ''__taintreport'' DOM events | full Firefox build toolchain, or prebuilt binaries | **8** |
-| [[Programming:Crawler:PanoptiChrome|PanoptiChrome]] {[kanyal2024_panoptichrome]} | patched Chromium/V8, pinned to Chrome 116 | — | dynamic taint tracking in Chromium, arbitrary sources and sinks | //see the paper// | build a pinned Chrome 116 fork from patches | **1** |
+| [[Programming:Crawler:PanoptiChrome|PanoptiChrome]] {[kanyal2024_panoptichrome]} | patched Chromium/V8, pinned to Chrome 116 | — | dynamic taint tracking in Chromium, on objects rather than strings, including implicit flows | one plain-text log per V8 isolate, in VisibleV8's format | build a pinned Chrome 116 fork from patches — the patch published with the paper **does not compile**; use the completed one, or a third party's prebuilt binary | **2** |
-| [[Programming:Crawler:webXray|webXray]] {[libert2015_invisible]} | PhantomJS //(historically)// | — | third-party request detection plus **domain-to-company attribution** | MySQL, CSV reports | //see below// | **6** |
+| [[Programming:Crawler:webXray|webXray]] {[libert2015_invisible]} | PhantomJS in 2015; **consumer Chrome over raw CDP** in the last public version | — | third-party request detection plus **domain-to-company attribution** | SQLite or Postgres, CSV reports | //see below// | **7** |
 Notes that matter more than the table:
   * **OpenWPM is still Firefox and still Selenium.** Its README's first paragraph says so, and it pins an unbranded Firefox build.(([[https://github.com/openwpm/OpenWPM|github.com/openwpm/OpenWPM]], README and ''scripts/install-firefox.sh'', checked 2026-08-06.)) It is actively maintained. If your study is about Chrome's behaviour, OpenWPM is measuring a different browser — a point Krumnow et al. {[krumnow2022_gullible]} press further by analysing how detectable OpenWPM is, how resilient its recording is, and how widespread OpenWPM-specific detection is in the wild.
-  * **PageGraph does not track your automation.** Its own documentation warns that "PageGraph currently does not track puppeteer / automation scripts, and so modifying or interacting with the document through devtools/puppeteer while recording a PageGraph file will likely fail".((''pagegraph-crawl'' README, [[https://github.com/brave/pagegraph-crawl|github.com/brave/pagegraph-crawl]], checked 2026-08-06.)) It is a recorder of what the page did, not a driver of interaction. Its own limitations list is unusually honest and worth reading before you rely on it: no WebSocket tracking, no worker tracking, request headers not recorded (responses only), no ''style='' mutation tracking.((PageGraph wiki, [[https://github.com/brave/brave-browser/wiki/PageGraph|"Would Be Nice / Someday / Known Limitations"]], checked 2026-08-06. Query tooling lives under the ''brave-experiments'' organisation: ''pagegraph-query'' (Python), ''pagegraph-rust''.))
+  * **PageGraph does not track your automation.** Its own documentation warns that "PageGraph currently does not track puppeteer / automation scripts, and so modifying or interacting with the document through devtools/puppeteer while recording a PageGraph file will likely fail".((''pagegraph-crawl'' README, [[https://github.com/brave/pagegraph-crawl|github.com/brave/pagegraph-crawl]], checked 2026-08-06.)) It is a recorder of what the page did, not a driver of interaction. Its own limitations list is unusually honest and worth reading before you rely on it: no WebSocket tracking, no worker tracking, request headers not recorded (responses only), no ''style='' mutation tracking.((PageGraph wiki, [[https://github.com/brave/brave-browser/wiki/PageGraph|"Would Be Nice / Someday / Known Limitations"]], checked 2026-08-06. Query tooling: ''pagegraph-query'' (Python) under ''brave-experiments'', ''pagegraph-rust'' under ''brave''.))
   * **Tracker Radar Collector caps concurrency at 38** crawlers, injects a simple anti-bot-detection script into every frame by default, and can be pointed at a Selenium Hub or a specific Chromium version.((''tracker-radar-collector'' README, [[https://github.com/duckduckgo/tracker-radar-collector|github.com/duckduckgo/tracker-radar-collector]], checked 2026-08-06.)) It has no canonical academic paper; cite the repository and the Tracker Radar dataset.
-  * **webXray needs care.** The repository its own documentation points at, ''github.com/timlib/webXray'', **no longer exists** — the account has no public repositories at all, and there is no ''webxray'' package on PyPI. What survives on GitHub are stale third-party mirrors, the newest of which was last pushed in 2015 and targets PhantomJS, whose own maintainer suspended development in 2018.(([[https://github.com/ariya/phantomjs/issues/15344|ariya/phantomjs#15344]], "Archiving the project: suspending the development", opened 2018-03-03.)) ''webxray.org'' is a live landing page with no source link.((Checked 2026-08-06: ''api.github.com/repos/timlib/webXray'' → HTTP 404; ''api.github.com/users/timlib/repos'' → empty; ''pypi.org/pypi/webxray/json'' → not found.)) Treat the 2015 paper {[libert2015_invisible]} as the citation for the *method* — third-party request measurement with company attribution — and not as a tool you can install today. <wrap todo>If you know where webXray is currently developed, please correct this.</wrap>
+  * **webXray cannot be installed, and the reason is now known.** The repository its own documentation points at, ''github.com/timlib/webXray'', **no longer exists**, the ''timlib'' account has no public repositories, and there is no ''webxray'' package on PyPI. webXray is now a **commercial** product, ''webxray.ai'', run by webXray LLC with Libert as founder and CEO; ''webxray.org'' has been reduced to a placeholder.((Checked 2026-08-17: ''api.github.com/repos/timlib/webXray'' → HTTP 404, ''users/timlib'' → HTTP 200 with ''public_repos: 0'' and ''company: webXray.ai''; ''pypi.org/pypi/webxray/json'' → 404; ''https://timlibert.me/'' states "(Dr.) Timothy Libert is founder and CEO of webXray LLC".)) The most complete surviving copy is ''thezedwards/webXray'', last pushed **2021-03-04** — webXray 3.x, which drives **consumer Chrome over the DevTools protocol**, not PhantomJS; its licence is PolyForm Strict 1.0.0, which permits noncommercial use but **forbids redistribution**, so there is no lawful route to the code now that upstream is gone. Treat {[libert2015_invisible]} and {[libert2018_automated]} as citations for the //method// — third-party request measurement with company attribution — and see [[Programming:Crawler:webXray]] for what survives of it: the ownership database, and how it compares to Tracker Radar and Disconnect today.
-  * **Taint tracking is a different question.** Foxhound and PanoptiChrome answer "did this value reach that sink?", not "what did the page load". If your question is about tracking prevalence, they are the wrong instrument and cost you a browser build; if it is about how data escapes — client-side XSS, DOM-based leaks, fingerprinting inputs — nothing else answers it. Note that Foxhound has no single canonical citation: the project points at a list of papers using it rather than naming one, and Klein et al. {[klein2022_handsanitizers]} is one of the earlier and most cited.
+  * **Taint tracking is a different question.** Foxhound and PanoptiChrome answer "did this value reach that sink?", not "what did the page load". If your question is about tracking prevalence, they are the wrong instrument and cost you a browser build; if it is about how data escapes — client-side XSS, DOM-based leaks, fingerprinting inputs — nothing else answers it. **Between the two, start with Foxhound**: the one head-to-head evaluation measured PanoptiChrome at 50% compatibility and 36.7× overhead against Foxhound's 95% and 1.4×, and no paper in this corpus has used PanoptiChrome as an instrument {[calzavara2025_dynamic]} — see [[Programming:Crawler:PanoptiChrome]] for when it is nevertheless the only option. Foxhound tells you what to cite: its README's "Cite us!" section asks for the EuroS&P paper in which the browser is described, Klein et al. {[klein2022_handsanitizers]}, and its wiki separately lists the papers that have used it.
-===== Being Detected =====
+===== Agent-Driven Crawling =====
-Whatever you drive, the website may notice. This is a measurement-validity problem, not just an engineering nuisance: a bot-managed site serves your crawler a different page, and that difference is silently attributed to whatever you were studying.
+Since 2025 there is a fifth option that does not fit the two-layer model above: a
+**crawler whose next action is chosen by a model** rather than scripted. The open-source
+ones — Browser Use, BrowserGym/AgentLab, Skyvern — still drive Chromium through CDP or
+Playwright, so they add no new control channel; the vendor computer-use APIs act on
+screenshots and coordinates and leave the channel to you. What all of them change is that
+the sequence of actions is non-deterministic, the completion rate becomes an empirical
+quantity, and the cost per site is measured in cents rather than fractions of one.
-  * **Automation is detectable in the browser.** Vastel et al. {[vastel2018_scanner]} show that fingerprint inconsistencies distinguish instrumented and spoofed browsers from ordinary ones; headless Chrome and driver-injected properties are among the easiest signals to read.
+**Five of the 1,120 crawling papers here drove a crawl this way, all of them in 2026.**((Four
-  * **Specific frameworks are detectable specifically.** Krumnow et al. {[krumnow2022_gullible]} study OpenWPM detection in the wild. A tool that 51 papers in this corpus share is a tool worth writing a detector for.
+in the table below, which counts only ''tools[]'' entries in an automation category, as
-  * **Crawls differ from humans even when nobody is detecting you.** Zeber et al. {[zeber2020representativeness]} quantified how far automated crawls diverge from real browsing on common tracking and fingerprinting metrics, and found crawls fail to capture the diversity of user environments.
+every other row of that table does. A fifth paper drove its crawl with Claude's Computer
-  * **The vantage point compounds this**, since datacenter IP ranges are themselves a low-trust signal {[jueckstock2021_realistic]} — see [[Design:Crawling location]].
+Use API, which the extraction filed under category ''llm''; it is counted on the child
-  * **Repeating a crawl is not free either.** Demir et al. {[demir2022_reproducibility]} found substantial variation between repetitions of the same web measurement, so a single crawl of a single configuration is a point estimate with unstated error bars.
+page and explained there.)) That is not enough to call it practice, and this page does not
+recommend it as a default. It is enough for a page of its own, because those five — and
+four neighbours that measure agents rather than crawl with them — have already published
+completion rates, per-site costs and (on benchmark tasks) run-to-run variance that nobody
+starting an agent crawl should have to rediscover.
+**[[Programming:Crawler:LLM Agents]]** has those numbers, the tool currency, and what to
+report.
-Five papers in the corpus reach for explicit anti-detection patches (''puppeteer-extra-plugin-stealth'', ''undetected-chromedriver''), all of them 2021 or later. Two warnings if you follow them. They are an arms race you will lose quietly and without notice, so any result that depends on them needs a validity check that does not. And evading bot management is a decision with an ethics dimension — see [[Practices:Ethics]] — because you are deliberately overriding a site operator's expressed access preference.
+===== Being Detected =====
+Whatever you drive, the website may notice, and a bot-managed site serves your crawler a
+different page — a measurement-validity problem, not an engineering nuisance.
+**[[Programming:Crawler Detection]] covers it in full**: what each layer of the stack
+gives you away with, why cloaking is the shape that ends up in your abstract, how to
+measure the obstruction instead of fighting it, and what the corpus says about how rarely
+anyone reports it. Two points from there bear on the choice made //on this page//:
+  * **The framework is detectable, not just automation in general.** Vastel et al. {[vastel2018_scanner]} show fingerprint inconsistencies distinguish instrumented browsers from ordinary ones, and Krumnow et al. {[krumnow2022_gullible]} show OpenWPM specifically has detectors deployed against it in the wild.
+  * **The anti-detection patches in the tables below are an arms race you lose quietly**, and most of those packages are now unmaintained or have moved to a successor under a different name; [[Programming:Crawler Detection]] dates each of them. Using them is also an ethics decision — see [[Practices:Ethics]] — because you are overriding a site operator's expressed access preference.
 ===== Use in Publications =====
-The figures below come from a structured extraction over **4,322 full-text papers** from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2024. Unless stated otherwise the population is the **859 papers that ran a crawl**, and sentinel values (''not-stated'') are counted as what they are rather than as answers. Methodology and limitations are at the end of this section.
+The figures below come from a structured extraction over **5,859 full-text papers** from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026. Unless stated otherwise the population is the **1,120 papers that ran a crawl**, and sentinel values (''not-stated'') are counted as what they are rather than as answers. The 2025 and 2026 venue-years are provisional — CCS and IMC 2026 have not been held, and IEEE S&P and WWW 2026 abstracts are not yet in the selection source — so any per-year row that reaches them is under-represented by construction. Methodology and limitations are at the end of this section.
 ==== Most papers do not say what drove the browser ====
-^ What the paper states ^ Papers ^ Share of 859 ^
+^ What the paper states ^ Papers ^ Share of 1,120 ^
-| Names an automation framework or library | 544 | 63.3% |
+| Names an automation framework or library | 723 | 64.6% |
-| Names a browser | 395 | 46.0% |
+| Names a browser | 529 | 47.2% |
-| Names both | 328 | 38.2% |
+| Names both | 443 | 39.6% |
-| **Names neither** | 248 | 28.9% |
+| **Names neither** | 311 | 27.8% |
 More than a quarter of crawling papers describe neither layer. Naming the browser is //rarer// than naming the library, which is the wrong way round: the browser is what the website reacts to.
@@ Line 291: / Line 390: @@
 Folded into families (see [[#Methodology and limitations of these figures|the note on folding]]) and counted by paper:
-^ Family ^ Papers ^ Share of 859 ^
+^ Family ^ Papers ^ Share of 1,120 ^
-| Selenium | 187 | 21.8% |
+| Selenium | 242 | 21.6% |
-| Puppeteer | 54 | 6.3% |
+| //Bespoke crawler, given its own name// (''SSOScan'', ''AdFisher'', ''PhishPrint'', ''CryptoScamTracker''…) | 181 | 16.2% |
-| OpenWPM | 51 | 5.9% |
+| //Bespoke crawler, described generically// ("our crawler", "a custom Python crawler", "a crawling extension") | 147 | 13.1% |
-| //Unnamed bespoke crawler// ("our crawler", "a custom Python crawler") | 44 | 5.1% |
+| Puppeteer | 76 | 6.8% |
-| Chrome DevTools Protocol, used directly | 35 | 4.1% |
+| OpenWPM | 58 | 5.2% |
-| PhantomJS and other legacy scriptable browsers | 30 | 3.5% |
+| Chrome DevTools Protocol, used directly | 42 | 3.8% |
-| Vulnerability and state-space crawlers (Crawljax, Black Widow, ZAP…) | 23 | 2.7% |
+| Playwright | 34 | 3.0% |
-| App-store and platform scrapers | 17 | 2.0% |
+| PhantomJS and other legacy scriptable browsers | 32 | 2.9% |
-| Scrapy | 17 | 2.0% |
+| Vulnerability and state-space crawlers (Crawljax, Black Widow, ZAP…) | 29 | 2.6% |
-| Tor Browser Crawler / ''tbselenium'' | 17 | 2.0% |
+| Scrapy | 21 | 1.9% |
-| Web-performance harnesses (WebPageTest, Browsertime, Lighthouse) | 14 | 1.6% |
+| Tor Browser Crawler / ''tbselenium'' | 20 | 1.8% |
-| Browser extensions used as the instrument | 13 | 1.5% |
+| App-store and platform scrapers | 18 | 1.6% |
-| HTTP-level scrapers, no browser (BeautifulSoup, ''requests'') | 13 | 1.5% |
+| HTTP-level scrapers, no browser (BeautifulSoup, ''requests'') | 17 | 1.5% |
-| Archival crawlers (Heritrix, HTTrack, ''wget -r'') | 11 | 1.3% |
+| Web-performance harnesses (WebPageTest, Browsertime, Lighthouse) | 16 | 1.4% |
-| Playwright | 10 | 1.2% |
+| Third-party browser extension used as the instrument (Ghostery, uBlock Origin, NoScript…)((An extension the authors wrote themselves is a bespoke crawler and is counted in the two bespoke rows instead, because that is the distinction that decides whether a reader can identify the instrument.)) | 14 | 1.3% |
-| Ad and tracker collection crawlers | 9 | 1.0% |
+| OS-level GUI automation (''pyautogui'', ''xdotool'') | 14 | 1.3% |
-| OS-level GUI automation (''pyautogui'', ''xdotool'') | 9 | 1.0% |
+| Archival crawlers (Heritrix, HTTrack, ''wget -r'') | 12 | 1.1% |
-| Fuzzers and monkey testers | 5 | 0.6% |
+| Tracker Radar Collector | 10 | 0.9% |
-| Tracker Radar Collector | 5 | 0.6% |
+| Consent-interaction crawlers (BannerClick, Consent-O-Matic, Priv-Accept) | 10 | 0.9% |
-| Anti-detection patches (''stealth'', ''undetected-chromedriver'') | 5 | 0.6% |
+| Ad and tracker collection crawlers | 9 | 0.8% |
-| Consent-interaction crawlers (BannerClick, Consent-O-Matic, Priv-Accept) | 4 | 0.5% |
+| Anti-detection patches (''stealth'', ''undetected-chromedriver'') | 9 | 0.8% |
+| Fuzzers and monkey testers | 5 | 0.4% |
+| [[Programming:Crawler:LLM Agents|LLM browser agents]] (Browser Use, BrowserGym/AgentLab)((Added as a family on 2026-08-29. All four papers are 2026, i.e. entirely inside the provisional slice. The family is tested //before// Playwright and Puppeteer because several of these agents are built on those libraries, so a name matched in the wrong order would land in the library's row instead. A fifth crawling paper drove its crawl with an agent the extraction filed under category ''llm'', outside this table's population; [[Programming:Crawler:LLM Agents]] counts it and explains the boundary.)) | 4 | 0.4% |
 | webXray | 1 | 0.1% |
+<WRAP important>
+Put the two bespoke rows together — 10 papers are in both — and **318 of 1,120 crawling papers (28.4%) crawled with something home-grown or too obscure to have a family here.** That is more than used Selenium (242), and more than the number naming any of Puppeteer, Playwright, direct CDP, Scrapy or OpenWPM (215). Of the 181 that gave their crawler a proper name, **74 named nothing else at all** — the name is the only identification of the instrument in the paper, and it means nothing to a reader who does not have the code.
+</WRAP>
 ==== And when ====
-^ Framework ^ 2010–2013 (n=89) ^ 2014–2017 (n=158) ^ 2018–2021 (n=298) ^ 2022–2024 (n=314) ^
+^ Framework ^ 2010–2013 (n=102) ^ 2014–2017 (n=167) ^ 2018–2021 (n=308) ^ 2022–2024 (n=345) ^ 2025–2026 //(provisional)// (n=198) ^
-| Selenium | 4 (4.5%) | 31 (19.6%) | 74 (24.8%) | 78 (24.8%) |
+| Selenium | 4 (3.9%) | 33 (19.8%) | 76 (24.7%) | 81 (23.5%) | 48 (24.2%) |
-| Puppeteer | 0 | 0 | 20 (6.7%) | 34 (10.8%) |
+| Puppeteer | 0 | 0 | 21 (6.8%) | 39 (11.3%) | 16 (8.1%) |
-| Playwright | 0 | 0 | 0 | 10 (3.2%) |
+| Playwright | 0 | 0 | 0 | 15 (4.3%) | 19 (9.6%) |
-| CDP used directly | 0 | 1 (0.6%) | 22 (7.4%) | 12 (3.8%) |
+| CDP used directly | 0 | 1 (0.6%) | 23 (7.5%) | 15 (4.3%) | 3 (1.5%) |
-| PhantomJS and other legacy scriptable browsers | 3 (3.4%) | 16 (10.1%) | 10 (3.4%) | 1 (0.3%) |
+| PhantomJS and other legacy scriptable browsers | 3 (2.9%) | 16 (9.6%) | 10 (3.2%) | 1 (0.3%) | 2 (1.0%) |
-| OpenWPM | 0 | 7 (4.4%) | 18 (6.0%) | 26 (8.3%) |
+| OpenWPM | 0 | 7 (4.2%) | 18 (5.8%) | 26 (7.5%) | 7 (3.5%) |
-The shape of the field, in one table. **Selenium has been flat at a quarter of crawling papers for a decade** — it is not being displaced, it is being supplemented. **PhantomJS collapsed** from 10.1% to 0.3% once headless Chrome shipped in 2017, which is the cleanest example available of a measurement tool becoming a liability: papers using it were measuring a WebKit engine that no user ran. **Direct CDP use peaked in 2018–2021 and then halved**, absorbed by Puppeteer. **OpenWPM has grown every window.** Playwright's first appearance in this corpus is 2022.
+The shape of the field, in one table. **Selenium rose to a fifth of crawling papers by 2014–2017 and has sat at a quarter ever since** — it is not being displaced, it is being supplemented. **PhantomJS collapsed** from 9.6% to 0.3% once headless Chrome shipped in 2017, which is the cleanest example available of a measurement tool becoming a liability: papers using it were measuring a WebKit engine that no user ran. **Direct CDP use peaked in 2018–2021 and has fallen away since**, absorbed by Puppeteer. Playwright's first appearance in this corpus is 2022, and in 2025–2026 it **overtakes Puppeteer** (9.6% against 8.1%) — the one clear reordering the extended corpus produced. Read that last column with the provisional caveat above: it rests on two venue-years that are incomplete by construction.
 ==== Which browser ====
-Of the 859 crawling papers, 829 have a recorded crawl configuration, and **395 (46.0%) name at least one browser**:
+**529 of the 1,120 crawling papers (47.2%) name at least one browser.** (1,080 of the 1,120 have a crawl-configuration record at all, so the missing 40 could not have named one; the denominator below stays 1,120 either way, because a paper that says nothing counts as saying nothing.)
-^ Browser family ^ Papers ^ Share of 395 naming a browser ^
+^ Browser family ^ Papers ^ Share of 529 naming a browser ^
-| Chrome / Chromium | 225 | 57.0% |
+| Chrome / Chromium | 304 | 57.5% |
-| Firefox | 121 | 30.6% |
+| Firefox | 155 | 29.3% |
-| Tor Browser | 32 | 8.1% |
+| //Names a library, not a browser// ("Selenium", "Puppeteer", "Playwright-controlled browser") | 49 | 9.3% |
-| PhantomJS and other headless shells | 24 | 6.1% |
+| Tor Browser | 38 | 7.2% |
-| //Names a library, not a browser// ("Selenium", "Puppeteer") | 23 | 5.8% |
+| PhantomJS and other headless shells | 24 | 4.5% |
-| //Unnamed or custom browser// ("a real browser", "instrumented Chromium") | 14 | 3.5% |
+| Internet Explorer | 16 | 3.0% |
-| Internet Explorer | 12 | 3.0% |
+| //Unnamed or custom browser// ("a real browser", "instrumented Chromium") | 16 | 3.0% |
-| Edge | 11 | 2.8% |
+| Safari / WebKit | 16 | 3.0% |
-| Safari / WebKit | 10 | 2.5% |
+| Edge | 15 | 2.8% |
-| Brave | 8 | 2.0% |
+| Brave | 10 | 1.9% |
-| Opera | 7 | 1.8% |
+| Opera | 9 | 1.7% |
-| Mobile device or WebView | 6 | 1.5% |
+| Mobile device or WebView | 8 | 1.5% |
-| Other named browser (Whale, Kiwi, QQ, UC…) | 4 | 1.0% |
+| Other named browser (Whale, Kiwi, QQ, UC…) | 5 | 0.9% |
-| Messenger web client (WhatsApp, Telegram, WeChat) | 2 | 0.5% |
+| Messenger web client (WhatsApp, Telegram, WeChat) | 2 | 0.4% |
-Two rows are findings rather than data. **23 papers answered "which browser?" with the name of a library**, which is exactly the conflation this page opened with. And **Safari/WebKit is 2.5%** of the papers that name a browser, far below any published estimate of its real-world usage share — the engine that behaves least like the other two, and whose tracking protection is on by default, is the one the field almost never measures. OmniCrawl {[cassel2022_omnicrawl]} is the notable exception.
+Two rows are findings rather than data. **49 papers answered "which browser?" with the name of a library**, which is exactly the conflation this page opened with — and the row grew faster than the corpus did, because the 2025–2026 papers write "a Playwright-controlled browser" where older ones wrote "Chrome". And **Safari/WebKit is 3.0%** of the papers that name a browser, far below any published estimate of its real-world usage share — the engine that behaves least like the other two, and whose tracking protection is on by default, is the one the field almost never measures. OmniCrawl {[cassel2022_omnicrawl]} is the notable exception.
 ==== Almost nobody reports the version ====
 ^ Family ^ Papers naming it ^ of which state a version ^ Share ^
-| Selenium | 187 | 18 | 9.6% |
+| Selenium | 242 | 27 | 11.2% |
-| Puppeteer | 54 | 7 | 13.0% |
+| Puppeteer | 76 | 10 | 13.2% |
-| Playwright | 10 | 5 | 50.0% |
+| Playwright | 34 | 9 | 26.5% |
-| CDP used directly | 35 | 2 | 5.7% |
+| CDP used directly | 42 | 2 | 4.8% |
-| PhantomJS and other legacy scriptable browsers | 30 | 5 | 16.7% |
+| PhantomJS and other legacy scriptable browsers | 32 | 5 | 15.6% |
-| OpenWPM | 51 | 13 | 25.5% |
+| OpenWPM | 58 | 15 | 25.9% |
-| Tor Browser Crawler / ''tbselenium'' | 17 | 3 | 17.6% |
+| Tor Browser Crawler / ''tbselenium'' | 20 | 3 | 15.0% |
-| **Any automation tool** | **544** | **63** | **11.6%** |
+| **Any automation tool** | **723** | **87** | **12.0%** |
-This is the largest reporting gap on the page and the cheapest to fix. OpenWPM users are two and a half times better than average at it, plausibly because OpenWPM's own releases are numbered and cited; Playwright's small cohort is best of all. Everyone else writes "we used Selenium".
+This is the largest reporting gap on the page and the cheapest to fix. Playwright's cohort is best at it (26.5%) and OpenWPM's is a fraction behind (25.9%) — both more than double the 12.0% average, and in OpenWPM's case plausibly because its own releases are numbered and cited. On the previous corpus Playwright led by a wide margin on a base of ten papers; with 34 the two are effectively tied. Everyone else writes "we used Selenium".
 ==== Configuration reporting, and whether the tool predicts it ====
-^ Configuration detail ^ States it ^ Share of 859 ^
+^ Configuration detail ^ States it ^ Share of 1,120 ^
-| Headless or headful | 111 | 12.9% |
+| Headless or headful | 140 | 12.5% |
-| Stateful or stateless | 165 | 19.2% |
+| Stateful or stateless | 219 | 19.6% |
-| Consent action | 270 | 31.4% |
+| Consent action | 349 | 31.2% |
-| Interaction depth | 652 | 75.9% |
+| Interaction depth | 841 | 75.1% |
-| Authentication | 593 | 69.0% |
+| Authentication | 779 | 69.6% |
 ^ Family ^ N ^ States headless ^ States statefulness ^ States consent action ^ Public artifact ^
-| Selenium | 187 | 20.9% | 28.9% | 35.3% | 52.9% |
+| Selenium | 242 | 21.1% | 27.7% | 36.0% | 57.0% |
-| Puppeteer | 54 | 40.7% | 33.3% | 64.8% | 72.2% |
+| Puppeteer | 76 | 35.5% | 35.5% | 57.9% | 67.1% |
-| Playwright | 10 | 30.0% | 30.0% | 60.0% | 80.0% |
+| Playwright | 34 | 26.5% | 35.3% | 47.1% | 70.6% |
-| CDP used directly | 35 | 25.7% | 28.6% | 60.0% | 77.1% |
+| CDP used directly | 42 | 21.4% | 33.3% | 59.5% | 69.0% |
-| PhantomJS and other legacy scriptable browsers | 30 | 43.3% | 23.3% | 40.0% | 46.7% |
+| PhantomJS and other legacy scriptable browsers | 32 | 40.6% | 21.9% | 37.5% | 50.0% |
-| OpenWPM | 51 | 17.6% | 54.9% | 52.9% | 49.0% |
+| OpenWPM | 58 | 17.2% | 55.2% | 51.7% | 55.2% |
-| Tor Browser Crawler / ''tbselenium'' | 17 | 11.8% | 23.5% | 29.4% | 64.7% |
+| Tor Browser Crawler / ''tbselenium'' | 20 | 10.0% | 20.0% | 30.0% | 70.0% |
-Read the rows against each other, not against 100%. **OpenWPM papers report statefulness at nearly twice the rate of everyone else** (54.9%), which is what you would expect from a tool whose configuration file makes the stateful/stateless choice explicit and whose documentation names it — the instrument shapes what gets written down. Conversely OpenWPM papers report //headless// least often (17.6%), and its configuration suggests why: OpenWPM's ''display_mode'' takes three values — ''native'', ''headless'' and ''xvfb''((OpenWPM ''README.md'' and ''docs/Configuration.md''; the ''xvfb'' mode runs a real, non-headless Firefox inside a virtual X display.)) — so a paper reporting "we ran under Xvfb" has stated something the headless/headful dichotomy does not have a slot for, and the extractor scores it as unstated.
+Read the rows against each other, not against 100%. **OpenWPM papers report statefulness at nearly twice the rate of everyone else** (55.2%), which is what you would expect from a tool whose configuration file makes the stateful/stateless choice explicit and whose documentation names it — the instrument shapes what gets written down. Conversely OpenWPM papers report //headless// least often (17.2%), and its configuration suggests why: OpenWPM's ''display_mode'' takes three values — ''native'', ''headless'' and ''xvfb''((OpenWPM ''README.md'' and ''docs/Configuration.md''; the ''xvfb'' mode runs a real, non-headless Firefox inside a virtual X display.)) — so a paper reporting "we ran under Xvfb" has stated something the headless/headful dichotomy does not have a slot for, and the extractor scores it as unstated.
-The **Public artifact** column is confounded by year and should not be read as an effect of the tool: Playwright and Puppeteer papers are recent, and artifact release rose from 26.4% to 66.2% across the corpus for reasons that have nothing to do with crawler choice.
+The **Public artifact** column is confounded by year and should not be read as an effect of the tool: Playwright and Puppeteer papers are recent, and among crawling papers the share releasing a public artifact rose from **21.6% (2010–2013) to 61.2% (2022–2024)**, and to 71.7% in the provisional 2025–2026 window, for reasons that have nothing to do with crawler choice.
 ==== Which specialised crawlers actually get used ====
-Matched by name across every tool mention in all 4,322 papers, so these counts include uses outside a formal crawl configuration and are not restricted to the 859:
+Matched by name across every tool mention in all 5,859 papers, so these counts include uses outside a formal crawl configuration and are not restricted to the 1,120. The two rows that also appear in the folded framework table use the same patterns there, so the two tables agree:
 ^ Tool ^ Papers using or producing it ^ Papers only citing it ^ Years used ^
-| OpenWPM | 53 | 2 | 2015–2024 |
+| OpenWPM | 60 | 3 | 2015–2026 |
-| ''tbselenium'' / tor-browser-crawler | 16 | 0 | 2017–2024 |
+| ''tbselenium'' / tor-browser-crawler | 21 | 0 | 2017–2026 |
-| Tracker Radar Collector | 8 | 1 | 2021–2024 |
+| Tracker Radar Collector | 21 | 1 | 2021–2026 |
-| VisibleV8 | 8 | 0 | 2019–2024 |
+| Anti-detection patches | 11 | 1 | 2021–2026 |
-| webXray | 6 | 0 | 2018–2022 |
+| VisibleV8 | 9 | 0 | 2019–2026 |
-| Anti-detection patches | 5 | 1 | 2021–2024 |
+| Brave PageGraph | 8 | 0 | 2020–2025 |
-| Brave PageGraph | 4 | 0 | 2020–2022 |
+| SAP Project Foxhound | 8 | 1 | 2024–2026 |
-| SAP Project Foxhound | 2 | 0 | 2024 |
+| webXray | 7 | 0 | 2018–2022 |
-| PanoptiChrome | 1 | 0 | 2024 |
+| PanoptiChrome | 2 | 0 | 2024–2025 |
-**OpenWPM is the field's only shared instrument**, by an order of magnitude. Everything else is either niche, new, or a tool used mainly by the group that built it — which is the honest reading of PageGraph's four papers and Foxhound's two. Newer instruments appear only at the end of the window, so these counts under-report current adoption: the 2024 boundary cuts them off mid-growth.
+**OpenWPM is still the field's only broadly shared instrument**, but by less than it was: extending the corpus to 2026 roughly **tripled Tracker Radar Collector (8 → 21) and quadrupled Foxhound (2 → 8)** while OpenWPM grew 53 → 60. The 2024 cut-off in the earlier version of this table was doing real work — it caught the newer instruments mid-growth and made them look more marginal than they are. Everything below OpenWPM is still either niche, new, or a tool used mainly by the group that built it.
 ==== Methodology and limitations of these figures ====
-  * **How they were produced.** One structured record per paper was extracted from full text, each tuple carrying a verbatim evidence quote and its section, so any figure here can be traced to the sentence that supports it. The script that produces every number on this page, with its denominators, is ''report_crawler.mjs''; the folding rules are in ''tool_fold.mjs''.
+  * **How they were produced.** One structured record per paper was extracted from full text, each tuple carrying a verbatim evidence quote and its section, so any figure here can be traced to the sentence that supports it. The script that produces every number on this page, with its denominators, is ''report_crawler.mjs''; the folding rules are in ''tool_fold.mjs''. Every query, the script's unedited output and the full residue are on [[provenance:programming:crawler]]; corpus-level caveats are on [[literature:corpus]].
-  * **How names were folded.** Tool names are free text and agree run-to-run on only about a fifth of exact strings, so nothing here is counted by exact string. Names were folded into the families shown by an explicit, ordered list of regular expressions — specific tools before the generic libraries they are built on, so ''puppeteer-extra-plugin-stealth'' lands in //Anti-detection patches// and not in //Puppeteer//. Folding matters: the exact string ''Selenium'' appears in 147 papers, while the folded family covers 187, so counting exact strings would undercount Selenium by 21% — the difference is ''Selenium WebDriver'', ''Selenium Webdriver'', ''Python Selenium WebDriver'', ''selenium'', ''ChromeDriver'' and ''Selenium's ChromeDriver''. Of **789 tool mentions across 378 distinct strings** in the crawling population, **10 distinct strings across 11 mentions could not be classified** and are excluded —''Dagger'', ''MyPageKeeper'', ''Trajectory'', ''BrowserStack'', ''Less is More'', ''MAPS'', ''MediaStream API'', ''Ruby watir-webdriver'', ''Tor BC'', ''Chrome Developer Protocol''. Browser names folded to 1 unclassified string out of 395 papers (''Ghostery'', which is an extension, not a browser).
+  * **How names were folded.** Tool names are free text and agree run-to-run on only about a fifth of exact strings, so nothing here is counted by exact string. Names were folded into the families shown by an explicit, ordered list of regular expressions — specific tools before the generic libraries they are built on, so ''puppeteer-extra-plugin-stealth'' lands in //Anti-detection patches// and not in //Puppeteer//. Folding matters: the exact string ''Selenium'' appears in 194 papers, while the folded family covers 242, so counting exact strings would undercount Selenium by 19.8% — the difference is ''Selenium WebDriver'', ''Selenium Webdriver'', ''Python Selenium WebDriver'', ''selenium'', ''ChromeDriver'' and ''Selenium's ChromeDriver''. Of **1,075 tool mentions across 501 distinct strings** in the crawling population, **199 distinct strings across 205 mentions match no family**. They are not discarded — they are the //Bespoke crawler, given its own name// row, because almost all of them are one paper's own tool (''SSOScan'', ''AdFisher'', ''Formlock'', ''CryptoScamTracker'', ''Spider-Scents'': 181 papers, essentially one name each). A handful are third-party tools we chose not to give a family of their own (''OmniCrawl'', ''JAW'', ''BrowserStack'', ''MetaMask automator'', ''Headless Chromium'', the ''measurement framework of Demir et al.''), so read that row as "home-grown or obscure" rather than strictly "home-grown". ''report_crawler.mjs'' prints the full list, so nothing vanishes. Browser names folded to 1 unclassified string out of 529 papers (''Ghostery'', which is an extension, not a browser).
   * **A paper counts once per family**, never once per mention, and shares do not sum to 100% because a paper can name several tools. A family's share is of the population named in its heading.
   * **Silence is not absence.** "Does not name a framework" means the paper did not say, not that the authors used none. These are reporting figures.
   * **''used'' and ''produced'' both count** as driving a crawl — a paper that built its own crawler crawled with it. Tools only //compared// or //cited// do not, which is why the last table separates the two.
-  * **Quotes were spot-checked.** Of 66 evidence quotes behind the OpenWPM and Playwright figures, 44 matched the source text character-for-character and the remainder differed only by line-break hyphenation ("Fire-fox"); none was unsupported.
+  * **Quotes were spot-checked**, by ''scripts/quote_check.mjs'' rather than by hand. Of the **106** evidence quotes attached to an OpenWPM or Playwright tool mention anywhere in the corpus, 58 match the source text exactly once whitespace and line-break hyphenation are normalised, and 32 more match on at least 60% of their five-word windows. The remaining 16 were then read by hand against the full text: **all sixteen are present in the paper and none was unsupported.** The mismatch is always either a bracketed citation marker the extraction dropped (''the Open-WPM platform [52] on the Firefox browser'') or the two-column reading order still interleaving mid-sentence (''we created a Playwright**constrains the valid child elements, and everything else is moved based** [18] crawler'').
-  * **Venue coverage.** IEEE S&P is only 43% retrieved (paywall), so it is under-represented. EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS are absent entirely; any claim here is a claim about seven venues. Notably, several of the tool papers this page recommends were published //outside// them — Krumnow et al. at CoNEXT, Klein et al. at EuroS&P, Libert in a communication journal — so a tool's paper count in this table is a lower bound on its standing.
+  * **Venue coverage.** Seven venues only; the scope, the selection funnel and what each stage of it costs are on [[literature:corpus]]. Notably, several of the tool papers this page recommends were published //outside// them — Krumnow et al. at CoNEXT, Klein et al. at EuroS&P, Libert in a communication journal — so a tool's paper count in this table is a lower bound on its standing.
-  * **''crawled'' is defined** as a paper whose crawl configuration was recorded or whose study types include an automated web crawl (859 papers, 19.9% of the corpus). This corpus is seven broad security venues, not a web-measurement corpus, so shares of all 4,322 papers would be meaningless here.
+  * **''crawled'' is defined** as a paper whose crawl configuration was recorded or whose study types include an automated web crawl (1,120 papers, 19.1% of the corpus). This corpus is seven broad security venues, not a web-measurement corpus, so shares of all 5,859 papers would be meaningless here.
 ===== Recommendations =====
-  - **Report both layers and both versions.** "Chromium 121 driven by Playwright 1.41" is a complete answer; "Selenium" is not. This costs one sentence and is the difference between a replicable and an unreplicable crawl. It is also the field's biggest gap: 11.6%.
+  - **Report both layers and both versions.** "Chromium 121 driven by Playwright 1.41" is a complete answer; "Selenium" is not. This costs one sentence and is the difference between a replicable and an unreplicable crawl. It is also the field's biggest gap: 12.0%.
   - **Pick the instrument from the question, not the habit.**
-    * Third-party requests, cookies, response bodies → Puppeteer, Playwright or CDP. Not bare Selenium.
+    * Third-party requests, cookies, response bodies → Puppeteer, Playwright or CDP. With Selenium, use WebDriver BiDi and verify what your binding actually delivers; never classic WebDriver alone.
     * Which script did it, and why → PageGraph or VisibleV8.
     * Did this value reach that sink → Foxhound or PanoptiChrome.
@@ Line 423: / Line 528: @@
     * Multi-browser or multi-language, or a lab already fluent in it → Selenium.
     * The lowest-effort modern tracking crawl → Tracker Radar Collector.
-  - **Do not write a new crawler for a solved problem.** 44 papers report an unnamed bespoke crawler, and that is 44 instruments nobody can inspect, compare or reuse. If you must, publish it.
+  - **Do not write a new crawler for a solved problem.** Nearly three in ten crawling papers used a home-grown one, and 75 of them identify it by nothing but a name they invented. If you must build one, say what it is built on, and publish it.
-  - **State headless or headful and justify it.** Headless is the most detectable configuration you can choose {[vastel2018_scanner]} and only 12.9% of papers say which they used.
+  - **State headless or headful and justify it.** Headless is the most detectable configuration you can choose {[vastel2018_scanner]} and only 12.5% of papers say which they used.
-  - **Pin the browser binary, not just the library.** Puppeteer and Playwright do this for you; Selenium does not. Record the exact build and archive it with your [[Artifacts|artefact]].
+  - **Pin the browser binary, not just the library.** Puppeteer and Playwright do this for you; Selenium does not. Record the exact build and archive it with your research artefact (see [[:Artifacts]], a page this wiki still owes you).
   - **Validate against something that is not your crawler.** A manual visit to a sample, a second browser engine, or a second vantage point {[jueckstock2021_realistic]}. Crawls diverge from human browsing in ways your crawl cannot see {[zeber2020representativeness]}, and repeated crawls diverge from each other {[demir2022_reproducibility]}.
   - **Run more than once.** A single crawl is a point estimate. Report the spread.
@@ Line 431: / Line 536: @@
 ===== Open Questions =====
-<wrap todo>
+<WRAP todo>
   * A head-to-head comparison of what OpenWPM, Tracker Radar Collector and a plain Playwright crawl each detect on the same sample, with the same vantage point and the same date, does not exist in the literature we found. It would settle a design question the whole field guesses at.
   * Playwright's patched Firefox and WebKit builds are used as stand-ins for the real browsers. How far the patching moves the fingerprint, and whether it changes what trackers do, is unmeasured.
-  * Where webXray is developed today (see [[#Specialised Measurement Crawlers|above]]).
+  * Whether Selenium's steady quarter-share reflects a considered choice or institutional inertia — a question for a survey of authors, not for this corpus.
-  * Whether the flat quarter-share of Selenium reflects a considered choice or institutional inertia — a question for a survey of authors, not for this corpus.
+</WRAP>
-</wrap>
 ====== References ======