Differences

This shows you the differences between two versions of the page.

--- programming:crawler [2026/08/17 07:54] – Correct the webXray paragraph: the newest surviving copy is 2021 and drives consumer Chrome over CDP, not a 2015 PhantomJS mirror; webXray is now the commercial webxray.ai, which closes the open TODO; add its PolyForm Strict no-redistribution licence and karel.kubicek.claude
+++ programming:crawler [2026/08/29 06:37] (current) – LLM-agent fold: new family row (4 papers), bespoke own-name 184->181, union 321->318 (28.7%->28.4%), 75->74, residue 204/210->199/205; new Agent-Driven Crawling section pointing at programming:crawler:llm_agents. Authored by Claude karel.kubicek.claude
@@ Line 3: / Line 3: @@
 Every automated web measurement makes two separate choices that papers routinely report as one: **which browser** renders the page, and **which control channel** drives it. A "Selenium crawl" says nothing about the first; a "Chrome crawl" says nothing about the second. They fail differently, they are detected differently, and they give you access to different data.
-This page compares the options. It covers the generic automation libraries ([[#Selenium|Selenium]], [[#Puppeteer|Puppeteer]], [[#Playwright|Playwright]], [[#Plain CDP|plain CDP]]) and then the specialised privacy and security crawlers built on top of them, each of which has — or will have — its own page: [[Programming:Crawler:OpenWPM]], [[Programming:Crawler:webXray]], [[Programming:Crawler:Tracker Radar Collector]], [[Programming:Crawler:PageGraph]], [[Programming:Crawler:Foxhound]], [[Programming:Crawler:PanoptiChrome]].
+This page compares the options. It covers the generic automation libraries ([[#Selenium|Selenium]], [[#Puppeteer|Puppeteer]], [[#Playwright|Playwright]], [[#Plain CDP|plain CDP]]) and then the specialised privacy and security crawlers built on top of them, each of which has — or will have — its own page: [[Programming:Crawler:OpenWPM]], [[Programming:Crawler:webXray]], [[Programming:Crawler:Tracker Radar Collector]], [[Programming:Crawler:PageGraph]], [[Programming:Crawler:Foxhound]], [[Programming:Crawler:PanoptiChrome]]. A newer child, [[Programming:Crawler:LLM Agents]], covers the one instrument on this page that is not a control channel at all: an agent that decides the next click with a model.
 It pairs with [[Design:Automated measurements]] (whether to crawl at all), [[Design:Crawling location]] (where from), [[Programming:Stateful stateless]] (with or without a profile), [[Programming:Interaction]] (what to do on the page), and [[Privacy:Consent]] (what to do with the banner).
@@ Line 327: / Line 327: @@
 | VisibleV8 {[jueckstock2019_visiblev8]} | patched Chromium (V8) | any CDP library | native-code JS API tracing inside V8, below the reach of page JavaScript | append-only trace logs | custom Chromium build | **9** |
 | [[Programming:Crawler:Foxhound|SAP Project Foxhound]] | patched Firefox (Gecko + SpiderMonkey) | Playwright((The only automation layer the repository documents; it carries its own ''.PLAYWRIGHT_VERSION'' and a CI workflow for it.)) | **dynamic taint tracking**: string-level flows from sources to sinks | ''__taintreport'' DOM events | full Firefox build toolchain, or prebuilt binaries | **8** |
-| [[Programming:Crawler:PanoptiChrome|PanoptiChrome]] {[kanyal2024_panoptichrome]} | patched Chromium/V8, pinned to Chrome 116 | — | dynamic taint tracking in Chromium, arbitrary sources and sinks | //see the paper// | build a pinned Chrome 116 fork from patches | **2** |
+| [[Programming:Crawler:PanoptiChrome|PanoptiChrome]] {[kanyal2024_panoptichrome]} | patched Chromium/V8, pinned to Chrome 116 | — | dynamic taint tracking in Chromium, on objects rather than strings, including implicit flows | one plain-text log per V8 isolate, in VisibleV8's format | build a pinned Chrome 116 fork from patches — the patch published with the paper **does not compile**; use the completed one, or a third party's prebuilt binary | **2** |
 | [[Programming:Crawler:webXray|webXray]] {[libert2015_invisible]} | PhantomJS in 2015; **consumer Chrome over raw CDP** in the last public version | — | third-party request detection plus **domain-to-company attribution** | SQLite or Postgres, CSV reports | //see below// | **7** |
@@ Line 336: / Line 336: @@
   * **Tracker Radar Collector caps concurrency at 38** crawlers, injects a simple anti-bot-detection script into every frame by default, and can be pointed at a Selenium Hub or a specific Chromium version.((''tracker-radar-collector'' README, [[https://github.com/duckduckgo/tracker-radar-collector|github.com/duckduckgo/tracker-radar-collector]], checked 2026-08-06.)) It has no canonical academic paper; cite the repository and the Tracker Radar dataset.
   * **webXray cannot be installed, and the reason is now known.** The repository its own documentation points at, ''github.com/timlib/webXray'', **no longer exists**, the ''timlib'' account has no public repositories, and there is no ''webxray'' package on PyPI. webXray is now a **commercial** product, ''webxray.ai'', run by webXray LLC with Libert as founder and CEO; ''webxray.org'' has been reduced to a placeholder.((Checked 2026-08-17: ''api.github.com/repos/timlib/webXray'' → HTTP 404, ''users/timlib'' → HTTP 200 with ''public_repos: 0'' and ''company: webXray.ai''; ''pypi.org/pypi/webxray/json'' → 404; ''https://timlibert.me/'' states "(Dr.) Timothy Libert is founder and CEO of webXray LLC".)) The most complete surviving copy is ''thezedwards/webXray'', last pushed **2021-03-04** — webXray 3.x, which drives **consumer Chrome over the DevTools protocol**, not PhantomJS; its licence is PolyForm Strict 1.0.0, which permits noncommercial use but **forbids redistribution**, so there is no lawful route to the code now that upstream is gone. Treat {[libert2015_invisible]} and {[libert2018_automated]} as citations for the //method// — third-party request measurement with company attribution — and see [[Programming:Crawler:webXray]] for what survives of it: the ownership database, and how it compares to Tracker Radar and Disconnect today.
-  * **Taint tracking is a different question.** Foxhound and PanoptiChrome answer "did this value reach that sink?", not "what did the page load". If your question is about tracking prevalence, they are the wrong instrument and cost you a browser build; if it is about how data escapes — client-side XSS, DOM-based leaks, fingerprinting inputs — nothing else answers it. Foxhound tells you what to cite: its README's "Cite us!" section asks for the EuroS&P paper in which the browser is described, Klein et al. {[klein2022_handsanitizers]}, and its wiki separately lists the papers that have used it.
+  * **Taint tracking is a different question.** Foxhound and PanoptiChrome answer "did this value reach that sink?", not "what did the page load". If your question is about tracking prevalence, they are the wrong instrument and cost you a browser build; if it is about how data escapes — client-side XSS, DOM-based leaks, fingerprinting inputs — nothing else answers it. **Between the two, start with Foxhound**: the one head-to-head evaluation measured PanoptiChrome at 50% compatibility and 36.7× overhead against Foxhound's 95% and 1.4×, and no paper in this corpus has used PanoptiChrome as an instrument {[calzavara2025_dynamic]} — see [[Programming:Crawler:PanoptiChrome]] for when it is nevertheless the only option. Foxhound tells you what to cite: its README's "Cite us!" section asks for the EuroS&P paper in which the browser is described, Klein et al. {[klein2022_handsanitizers]}, and its wiki separately lists the papers that have used it.
-===== Being Detected =====
+===== Agent-Driven Crawling =====
-Whatever you drive, the website may notice. This is a measurement-validity problem, not just an engineering nuisance: a bot-managed site serves your crawler a different page, and that difference is silently attributed to whatever you were studying.
+Since 2025 there is a fifth option that does not fit the two-layer model above: a
+**crawler whose next action is chosen by a model** rather than scripted. The open-source
+ones — Browser Use, BrowserGym/AgentLab, Skyvern — still drive Chromium through CDP or
+Playwright, so they add no new control channel; the vendor computer-use APIs act on
+screenshots and coordinates and leave the channel to you. What all of them change is that
+the sequence of actions is non-deterministic, the completion rate becomes an empirical
+quantity, and the cost per site is measured in cents rather than fractions of one.
+**Five of the 1,120 crawling papers here drove a crawl this way, all of them in 2026.**((Four
+in the table below, which counts only ''tools[]'' entries in an automation category, as
+every other row of that table does. A fifth paper drove its crawl with Claude's Computer
+Use API, which the extraction filed under category ''llm''; it is counted on the child
+page and explained there.)) That is not enough to call it practice, and this page does not
+recommend it as a default. It is enough for a page of its own, because those five — and
+four neighbours that measure agents rather than crawl with them — have already published
+completion rates, per-site costs and (on benchmark tasks) run-to-run variance that nobody
+starting an agent crawl should have to rediscover.
+**[[Programming:Crawler:LLM Agents]]** has those numbers, the tool currency, and what to
+report.
+===== Being Detected =====
-  * **Automation is detectable in the browser.** Vastel et al. {[vastel2018_scanner]} show that fingerprint inconsistencies distinguish instrumented and spoofed browsers from ordinary ones; headless Chrome and driver-injected properties are among the easiest signals to read.
+Whatever you drive, the website may notice, and a bot-managed site serves your crawler a
-  * **Specific frameworks are detectable specifically.** Krumnow et al. {[krumnow2022_gullible]} study OpenWPM detection in the wild. A tool that 58 papers in this corpus share is a tool worth writing a detector for.
+different page — a measurement-validity problem, not an engineering nuisance.
-  * **Crawls differ from humans even when nobody is detecting you.** Zeber et al. {[zeber2020representativeness]} quantified how far automated crawls diverge from real browsing on common tracking and fingerprinting metrics, and found crawls fail to capture the diversity of user environments.
+**[[Programming:Crawler Detection]] covers it in full**: what each layer of the stack
-  * **The vantage point compounds this**, since datacenter IP ranges are themselves a low-trust signal {[jueckstock2021_realistic]} — see [[Design:Crawling location]].
+gives you away with, why cloaking is the shape that ends up in your abstract, how to
-  * **Repeating a crawl is not free either.** Demir et al. {[demir2022_reproducibility]} found substantial variation between repetitions of the same web measurement, so a single crawl of a single configuration is a point estimate with unstated error bars.
+measure the obstruction instead of fighting it, and what the corpus says about how rarely
+anyone reports it. Two points from there bear on the choice made //on this page//:
-Nine papers in the corpus reach for explicit anti-detection patches (''puppeteer-extra-plugin-stealth'', ''undetected-chromedriver''), all of them 2021 or later. Two warnings if you follow them. They are an arms race you will lose quietly and without notice, so any result that depends on them needs a validity check that does not. And evading bot management is a decision with an ethics dimension — see [[Practices:Ethics]] — because you are deliberately overriding a site operator's expressed access preference.
+  * **The framework is detectable, not just automation in general.** Vastel et al. {[vastel2018_scanner]} show fingerprint inconsistencies distinguish instrumented browsers from ordinary ones, and Krumnow et al. {[krumnow2022_gullible]} show OpenWPM specifically has detectors deployed against it in the wild.
+  * **The anti-detection patches in the tables below are an arms race you lose quietly**, and most of those packages are now unmaintained or have moved to a successor under a different name; [[Programming:Crawler Detection]] dates each of them. Using them is also an ethics decision — see [[Practices:Ethics]] — because you are overriding a site operator's expressed access preference.
 ===== Use in Publications =====
@@ Line 370: / Line 392: @@
 ^ Family ^ Papers ^ Share of 1,120 ^
 | Selenium | 242 | 21.6% |
-| //Bespoke crawler, given its own name// (''SSOScan'', ''AdFisher'', ''PhishPrint'', ''CryptoScamTracker''…) | 184 | 16.4% |
+| //Bespoke crawler, given its own name// (''SSOScan'', ''AdFisher'', ''PhishPrint'', ''CryptoScamTracker''…) | 181 | 16.2% |
 | //Bespoke crawler, described generically// ("our crawler", "a custom Python crawler", "a crawling extension") | 147 | 13.1% |
 | Puppeteer | 76 | 6.8% |
@@ Line 391: / Line 413: @@
 | Anti-detection patches (''stealth'', ''undetected-chromedriver'') | 9 | 0.8% |
 | Fuzzers and monkey testers | 5 | 0.4% |
+| [[Programming:Crawler:LLM Agents|LLM browser agents]] (Browser Use, BrowserGym/AgentLab)((Added as a family on 2026-08-29. All four papers are 2026, i.e. entirely inside the provisional slice. The family is tested //before// Playwright and Puppeteer because several of these agents are built on those libraries, so a name matched in the wrong order would land in the library's row instead. A fifth crawling paper drove its crawl with an agent the extraction filed under category ''llm'', outside this table's population; [[Programming:Crawler:LLM Agents]] counts it and explains the boundary.)) | 4 | 0.4% |
 | webXray | 1 | 0.1% |
 <WRAP important>
-Put the two bespoke rows together — 10 papers are in both — and **321 of 1,120 crawling papers (28.7%) crawled with something home-grown or too obscure to have a family here.** That is more than used Selenium (242), and more than the number naming any of Puppeteer, Playwright, direct CDP, Scrapy or OpenWPM (215). Of the 184 that gave their crawler a proper name, **75 named nothing else at all** — the name is the only identification of the instrument in the paper, and it means nothing to a reader who does not have the code.
+Put the two bespoke rows together — 10 papers are in both — and **318 of 1,120 crawling papers (28.4%) crawled with something home-grown or too obscure to have a family here.** That is more than used Selenium (242), and more than the number naming any of Puppeteer, Playwright, direct CDP, Scrapy or OpenWPM (215). Of the 181 that gave their crawler a proper name, **74 named nothing else at all** — the name is the only identification of the instrument in the paper, and it means nothing to a reader who does not have the code.
 </WRAP>
@@ Line 475: / Line 498: @@
 | ''tbselenium'' / tor-browser-crawler | 21 | 0 | 2017–2026 |
 | Tracker Radar Collector | 21 | 1 | 2021–2026 |
-| Anti-detection patches | 12 | 2 | 2021–2026 |
+| Anti-detection patches | 11 | 1 | 2021–2026 |
 | VisibleV8 | 9 | 0 | 2019–2026 |
 | Brave PageGraph | 8 | 0 | 2020–2025 |
@@ Line 487: / Line 510: @@
   * **How they were produced.** One structured record per paper was extracted from full text, each tuple carrying a verbatim evidence quote and its section, so any figure here can be traced to the sentence that supports it. The script that produces every number on this page, with its denominators, is ''report_crawler.mjs''; the folding rules are in ''tool_fold.mjs''. Every query, the script's unedited output and the full residue are on [[provenance:programming:crawler]]; corpus-level caveats are on [[literature:corpus]].
-  * **How names were folded.** Tool names are free text and agree run-to-run on only about a fifth of exact strings, so nothing here is counted by exact string. Names were folded into the families shown by an explicit, ordered list of regular expressions — specific tools before the generic libraries they are built on, so ''puppeteer-extra-plugin-stealth'' lands in //Anti-detection patches// and not in //Puppeteer//. Folding matters: the exact string ''Selenium'' appears in 194 papers, while the folded family covers 242, so counting exact strings would undercount Selenium by 19.8% — the difference is ''Selenium WebDriver'', ''Selenium Webdriver'', ''Python Selenium WebDriver'', ''selenium'', ''ChromeDriver'' and ''Selenium's ChromeDriver''. Of **1,075 tool mentions across 501 distinct strings** in the crawling population, **204 distinct strings across 210 mentions match no family**. They are not discarded — they are the //Bespoke crawler, given its own name// row, because almost all of them are one paper's own tool (''SSOScan'', ''AdFisher'', ''Formlock'', ''CryptoScamTracker'', ''Spider-Scents'': 184 papers, essentially one name each). A handful are third-party tools we chose not to give a family of their own (''OmniCrawl'', ''JAW'', ''BrowserStack'', ''MetaMask automator'', ''Headless Chromium'', the ''measurement framework of Demir et al.''), so read that row as "home-grown or obscure" rather than strictly "home-grown". ''report_crawler.mjs'' prints the full list, so nothing vanishes. Browser names folded to 1 unclassified string out of 529 papers (''Ghostery'', which is an extension, not a browser).
+  * **How names were folded.** Tool names are free text and agree run-to-run on only about a fifth of exact strings, so nothing here is counted by exact string. Names were folded into the families shown by an explicit, ordered list of regular expressions — specific tools before the generic libraries they are built on, so ''puppeteer-extra-plugin-stealth'' lands in //Anti-detection patches// and not in //Puppeteer//. Folding matters: the exact string ''Selenium'' appears in 194 papers, while the folded family covers 242, so counting exact strings would undercount Selenium by 19.8% — the difference is ''Selenium WebDriver'', ''Selenium Webdriver'', ''Python Selenium WebDriver'', ''selenium'', ''ChromeDriver'' and ''Selenium's ChromeDriver''. Of **1,075 tool mentions across 501 distinct strings** in the crawling population, **199 distinct strings across 205 mentions match no family**. They are not discarded — they are the //Bespoke crawler, given its own name// row, because almost all of them are one paper's own tool (''SSOScan'', ''AdFisher'', ''Formlock'', ''CryptoScamTracker'', ''Spider-Scents'': 181 papers, essentially one name each). A handful are third-party tools we chose not to give a family of their own (''OmniCrawl'', ''JAW'', ''BrowserStack'', ''MetaMask automator'', ''Headless Chromium'', the ''measurement framework of Demir et al.''), so read that row as "home-grown or obscure" rather than strictly "home-grown". ''report_crawler.mjs'' prints the full list, so nothing vanishes. Browser names folded to 1 unclassified string out of 529 papers (''Ghostery'', which is an extension, not a browser).
   * **A paper counts once per family**, never once per mention, and shares do not sum to 100% because a paper can name several tools. A family's share is of the population named in its heading.
   * **Silence is not absence.** "Does not name a framework" means the paper did not say, not that the authors used none. These are reporting figures.
@@ Line 513: / Line 536: @@
 ===== Open Questions =====
-<wrap todo>
+<WRAP todo>
   * A head-to-head comparison of what OpenWPM, Tracker Radar Collector and a plain Playwright crawl each detect on the same sample, with the same vantage point and the same date, does not exist in the literature we found. It would settle a design question the whole field guesses at.
   * Playwright's patched Firefox and WebKit builds are used as stand-ins for the real browsers. How far the patching moves the fingerprint, and whether it changes what trackers do, is unmeasured.
   * Whether Selenium's steady quarter-share reflects a considered choice or institutional inertia — a question for a survey of authors, not for this corpus.
-</wrap>
+</WRAP>
 ====== References ======