| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| programming:crawler [2026/08/29 06:37] – LLM-agent fold: new family row (4 papers), bespoke own-name 184->181, union 321->318 (28.7%->28.4%), 75->74, residue 204/210->199/205; new Agent-Driven Crawling section pointing at programming:crawler:llm_agents. Authored by Claude karel.kubicek.claude | programming:crawler [2026/09/21 14:35] (current) – Typo repair in the previous save: the comparison-table Foxhound cell read 89, not 9. Authored by Claude karel.kubicek.claude |
|---|
| ==== Plain CDP ==== | ==== Plain CDP ==== |
| |
| You can skip the libraries entirely: launch Chromium with ''--remote-debugging-port'', open a WebSocket, enable the domains you want, and read the events. **42 papers (3.8%) did**, most of them between 2018 and 2021 — a window that closes as Puppeteer matures. | You can skip the libraries entirely: launch Chromium with ''%%--remote-debugging-port%%'', open a WebSocket, enable the domains you want, and read the events. **42 papers (3.8%) did**, most of them between 2018 and 2021 — a window that closes as Puppeteer matures. |
| |
| Reach for it when your analysis lives in a language with no good automation binding, when you need a CDP domain the wrapper does not expose, or when you want to know exactly what your instrument is doing. The cost, visible in the code below, is that everything a library gives you is now yours to write: waiting for the port, waiting for the load, waiting for the network to settle, and reconnecting when a target dies. | Reach for it when your analysis lives in a language with no good automation binding, when you need a CDP domain the wrapper does not expose, or when you want to know exactly what your instrument is doing. The cost, visible in the code below, is that everything a library gives you is now yours to write: waiting for the port, waiting for the load, waiting for the network to settle, and reconnecting when a target dies. |
| |
| <WRAP info> | <WRAP info> |
| **Sandbox gotchas we hit, since they cost half a day.** On this aarch64 container, //every full Chrome/Chromium binary// — Debian's ''/usr/bin/chromium'', Chrome for Testing, and Playwright's own ''chromium-*'' build — died at startup with ''chrome_crashpad_handler: --database is required'' and never opened the DevTools port. Only Playwright's ''chromium_headless_shell'' build started. Chrome for Testing ships no aarch64 build (the downloaded ''linux_arm'' binary is x86-64 and fails under emulation), and neither does ''chromedriver''; a working aarch64 ''chromedriver'' came from Debian's ''chromium-driver'' package, extracted with ''dpkg -x'' without root. Firefox 153 + geckodriver 0.37.1 never handed over its Marionette port in this container, with or without Xvfb, so the Selenium example is the Chromium one. None of this is a property of the libraries — it is a property of running x86-64-first browser tooling on ARM in a container, which is increasingly where CI lives. | **Sandbox gotchas we hit, since they cost half a day.** On this aarch64 container, //every full Chrome/Chromium binary// — Debian's ''/usr/bin/chromium'', Chrome for Testing, and Playwright's own ''chromium-*'' build — died at startup with ''chrome_crashpad_handler: %%--database%% is required'' and never opened the DevTools port. Only Playwright's ''chromium_headless_shell'' build started. Chrome for Testing ships no aarch64 build (the downloaded ''linux_arm'' binary is x86-64 and fails under emulation), and neither does ''chromedriver''; a working aarch64 ''chromedriver'' came from Debian's ''chromium-driver'' package, extracted with ''dpkg -x'' without root. Firefox 153 + geckodriver 0.37.1 never handed over its Marionette port in this container, with or without Xvfb, so the Selenium example is the Chromium one. None of this is a property of the libraries — it is a property of running x86-64-first browser tooling on ARM in a container, which is increasingly where CI lives. |
| </WRAP> | </WRAP> |
| |
| | [[Programming:Crawler:PageGraph|Brave PageGraph]] | patched Brave (Blink + V8) | ''pagegraph-crawl'' (Node) | a causal **graph**: every DOM mutation, script execution and request attributed to its cause, plus Shields filter-rule effects; the production successor to AdGraph {[iqbal2020_adgraph]} | GraphML | in Brave ≥ 1.46, but JS-API recording needs Brave **Nightly** | **8** | | | [[Programming:Crawler:PageGraph|Brave PageGraph]] | patched Brave (Blink + V8) | ''pagegraph-crawl'' (Node) | a causal **graph**: every DOM mutation, script execution and request attributed to its cause, plus Shields filter-rule effects; the production successor to AdGraph {[iqbal2020_adgraph]} | GraphML | in Brave ≥ 1.46, but JS-API recording needs Brave **Nightly** | **8** | |
| | VisibleV8 {[jueckstock2019_visiblev8]} | patched Chromium (V8) | any CDP library | native-code JS API tracing inside V8, below the reach of page JavaScript | append-only trace logs | custom Chromium build | **9** | | | VisibleV8 {[jueckstock2019_visiblev8]} | patched Chromium (V8) | any CDP library | native-code JS API tracing inside V8, below the reach of page JavaScript | append-only trace logs | custom Chromium build | **9** | |
| | [[Programming:Crawler:Foxhound|SAP Project Foxhound]] | patched Firefox (Gecko + SpiderMonkey) | Playwright((The only automation layer the repository documents; it carries its own ''.PLAYWRIGHT_VERSION'' and a CI workflow for it.)) | **dynamic taint tracking**: string-level flows from sources to sinks | ''__taintreport'' DOM events | full Firefox build toolchain, or prebuilt binaries | **8** | | | [[Programming:Crawler:Foxhound|SAP Project Foxhound]] | patched Firefox (Gecko + SpiderMonkey) | Playwright((The only automation layer the repository documents; it carries its own ''.PLAYWRIGHT_VERSION'' and a CI workflow for it.)) | **dynamic taint tracking**: string-level flows from sources to sinks | ''__taintreport'' DOM events | full Firefox build toolchain, or prebuilt binaries | **9** | |
| | [[Programming:Crawler:PanoptiChrome|PanoptiChrome]] {[kanyal2024_panoptichrome]} | patched Chromium/V8, pinned to Chrome 116 | — | dynamic taint tracking in Chromium, on objects rather than strings, including implicit flows | one plain-text log per V8 isolate, in VisibleV8's format | build a pinned Chrome 116 fork from patches — the patch published with the paper **does not compile**; use the completed one, or a third party's prebuilt binary | **2** | | | [[Programming:Crawler:PanoptiChrome|PanoptiChrome]] {[kanyal2024_panoptichrome]} | patched Chromium/V8, pinned to Chrome 116 | — | dynamic taint tracking in Chromium, on objects rather than strings, including implicit flows | one plain-text log per V8 isolate, in VisibleV8's format | build a pinned Chrome 116 fork from patches — the patch published with the paper **does not compile**; use the completed one, or a third party's prebuilt binary | **2** | |
| | [[Programming:Crawler:webXray|webXray]] {[libert2015_invisible]} | PhantomJS in 2015; **consumer Chrome over raw CDP** in the last public version | — | third-party request detection plus **domain-to-company attribution** | SQLite or Postgres, CSV reports | //see below// | **7** | | | [[Programming:Crawler:webXray|webXray]] {[libert2015_invisible]} | PhantomJS in 2015; **consumer Chrome over raw CDP** in the last public version | — | third-party request detection plus **domain-to-company attribution** | SQLite or Postgres, CSV reports | //see below// | **7** | |
| * **OpenWPM is still Firefox and still Selenium.** Its README's first paragraph says so, and it pins an unbranded Firefox build.(([[https://github.com/openwpm/OpenWPM|github.com/openwpm/OpenWPM]], README and ''scripts/install-firefox.sh'', checked 2026-08-06.)) It is actively maintained. If your study is about Chrome's behaviour, OpenWPM is measuring a different browser — a point Krumnow et al. {[krumnow2022_gullible]} press further by analysing how detectable OpenWPM is, how resilient its recording is, and how widespread OpenWPM-specific detection is in the wild. | * **OpenWPM is still Firefox and still Selenium.** Its README's first paragraph says so, and it pins an unbranded Firefox build.(([[https://github.com/openwpm/OpenWPM|github.com/openwpm/OpenWPM]], README and ''scripts/install-firefox.sh'', checked 2026-08-06.)) It is actively maintained. If your study is about Chrome's behaviour, OpenWPM is measuring a different browser — a point Krumnow et al. {[krumnow2022_gullible]} press further by analysing how detectable OpenWPM is, how resilient its recording is, and how widespread OpenWPM-specific detection is in the wild. |
| * **PageGraph does not track your automation.** Its own documentation warns that "PageGraph currently does not track puppeteer / automation scripts, and so modifying or interacting with the document through devtools/puppeteer while recording a PageGraph file will likely fail".((''pagegraph-crawl'' README, [[https://github.com/brave/pagegraph-crawl|github.com/brave/pagegraph-crawl]], checked 2026-08-06.)) It is a recorder of what the page did, not a driver of interaction. Its own limitations list is unusually honest and worth reading before you rely on it: no WebSocket tracking, no worker tracking, request headers not recorded (responses only), no ''style='' mutation tracking.((PageGraph wiki, [[https://github.com/brave/brave-browser/wiki/PageGraph|"Would Be Nice / Someday / Known Limitations"]], checked 2026-08-06. Query tooling: ''pagegraph-query'' (Python) under ''brave-experiments'', ''pagegraph-rust'' under ''brave''.)) | * **PageGraph does not track your automation.** Its own documentation warns that "PageGraph currently does not track puppeteer / automation scripts, and so modifying or interacting with the document through devtools/puppeteer while recording a PageGraph file will likely fail".((''pagegraph-crawl'' README, [[https://github.com/brave/pagegraph-crawl|github.com/brave/pagegraph-crawl]], checked 2026-08-06.)) It is a recorder of what the page did, not a driver of interaction. Its own limitations list is unusually honest and worth reading before you rely on it: no WebSocket tracking, no worker tracking, request headers not recorded (responses only), no ''style='' mutation tracking.((PageGraph wiki, [[https://github.com/brave/brave-browser/wiki/PageGraph|"Would Be Nice / Someday / Known Limitations"]], checked 2026-08-06. Query tooling: ''pagegraph-query'' (Python) under ''brave-experiments'', ''pagegraph-rust'' under ''brave''.)) |
| * **Tracker Radar Collector caps concurrency at 38** crawlers, injects a simple anti-bot-detection script into every frame by default, and can be pointed at a Selenium Hub or a specific Chromium version.((''tracker-radar-collector'' README, [[https://github.com/duckduckgo/tracker-radar-collector|github.com/duckduckgo/tracker-radar-collector]], checked 2026-08-06.)) It has no canonical academic paper; cite the repository and the Tracker Radar dataset. | * **Tracker Radar Collector's README is stale, and one of its stale numbers was on this page.** It still claims "a hard limit of 38" concurrent crawlers; ''38'' appears nowhere in the code and the cap was removed in commit ''963f9ec'' (2025-04-16). The default is ''Math.floor(cores * 0.8)'', then capped at the number of URLs. The README's ''maxLoadTimeMs'' default of 30 s is stale the same way — the code fallback is 60 s. TRC does inject an anti-bot script into every frame by default (''-a'' disables it) and can be pointed at a Selenium Hub or a specific Chromium version.((Checked against a clone at commit ''8b64006691a1ce3929cfdfeeb425e7cc64be6543'' on 2026-09-03: ''crawlerConductor.js'' concurrency, ''crawler.js'' ''maxLoadTimeMs || 60000'', and ''git log -S38 -- crawlerConductor.js''. [[https://github.com/duckduckgo/tracker-radar-collector|github.com/duckduckgo/tracker-radar-collector]].)) It has no canonical academic paper; cite the repository and the Tracker Radar dataset — and see [[Programming:Crawler:Tracker Radar Collector]] for the rest of the README-versus-code divergences. |
| * **webXray cannot be installed, and the reason is now known.** The repository its own documentation points at, ''github.com/timlib/webXray'', **no longer exists**, the ''timlib'' account has no public repositories, and there is no ''webxray'' package on PyPI. webXray is now a **commercial** product, ''webxray.ai'', run by webXray LLC with Libert as founder and CEO; ''webxray.org'' has been reduced to a placeholder.((Checked 2026-08-17: ''api.github.com/repos/timlib/webXray'' → HTTP 404, ''users/timlib'' → HTTP 200 with ''public_repos: 0'' and ''company: webXray.ai''; ''pypi.org/pypi/webxray/json'' → 404; ''https://timlibert.me/'' states "(Dr.) Timothy Libert is founder and CEO of webXray LLC".)) The most complete surviving copy is ''thezedwards/webXray'', last pushed **2021-03-04** — webXray 3.x, which drives **consumer Chrome over the DevTools protocol**, not PhantomJS; its licence is PolyForm Strict 1.0.0, which permits noncommercial use but **forbids redistribution**, so there is no lawful route to the code now that upstream is gone. Treat {[libert2015_invisible]} and {[libert2018_automated]} as citations for the //method// — third-party request measurement with company attribution — and see [[Programming:Crawler:webXray]] for what survives of it: the ownership database, and how it compares to Tracker Radar and Disconnect today. | * **webXray cannot be installed, and the reason is now known.** The repository its own documentation points at, ''github.com/timlib/webXray'', **no longer exists**, the ''timlib'' account has no public repositories, and there is no ''webxray'' package on PyPI. webXray is now a **commercial** product, ''webxray.ai'', run by webXray LLC with Libert as founder and CEO; ''webxray.org'' has been reduced to a placeholder.((Checked 2026-08-17: ''api.github.com/repos/timlib/webXray'' → HTTP 404, ''users/timlib'' → HTTP 200 with ''public_repos: 0'' and ''company: webXray.ai''; ''pypi.org/pypi/webxray/json'' → 404; ''https://timlibert.me/'' states "(Dr.) Timothy Libert is founder and CEO of webXray LLC".)) The most widely mirrored copy is ''thezedwards/webXray'', last pushed **2021-03-04** — webXray 3.x, which drives **consumer Chrome over the DevTools protocol**, not PhantomJS. It is **not** the copy to take, and its licence is the reason to be careful rather than the reason to give up: that snapshot is PolyForm Strict 1.0.0, which permits noncommercial use but forbids redistribution — but Libert relicensed the project **back to GPLv3 in 2021 and then to MIT in 2023**, and that history survives in ''peterjoles/webXray'', 36 commits ahead. There **is** a lawful route to the code; it is just not the copy most mirrors carry. Treat {[libert2015_invisible]} and {[libert2018_automated]} as citations for the //method// — third-party request measurement with company attribution — and see [[Programming:Crawler:webXray]] for the tool's remains. The part that outlived it, the **domain-ownership database**, and how it compares to the two live alternatives today, moved on 2026-09-11 to [[Design:Ownership resolution]]: that is the page to read if your question is "whose domain is this?" rather than "which crawler should I run?". |
| * **Taint tracking is a different question.** Foxhound and PanoptiChrome answer "did this value reach that sink?", not "what did the page load". If your question is about tracking prevalence, they are the wrong instrument and cost you a browser build; if it is about how data escapes — client-side XSS, DOM-based leaks, fingerprinting inputs — nothing else answers it. **Between the two, start with Foxhound**: the one head-to-head evaluation measured PanoptiChrome at 50% compatibility and 36.7× overhead against Foxhound's 95% and 1.4×, and no paper in this corpus has used PanoptiChrome as an instrument {[calzavara2025_dynamic]} — see [[Programming:Crawler:PanoptiChrome]] for when it is nevertheless the only option. Foxhound tells you what to cite: its README's "Cite us!" section asks for the EuroS&P paper in which the browser is described, Klein et al. {[klein2022_handsanitizers]}, and its wiki separately lists the papers that have used it. | * **Taint tracking is a different question.** Foxhound and PanoptiChrome answer "did this value reach that sink?", not "what did the page load". If your question is about tracking prevalence, they are the wrong instrument and cost you a browser build; if it is about how data escapes — client-side XSS, DOM-based leaks, fingerprinting inputs — nothing else answers it. **Between the two, start with Foxhound**: the one head-to-head evaluation measured PanoptiChrome at 50% compatibility and 36.7× overhead against Foxhound's 95% and 1.4×, and no paper in this corpus has used PanoptiChrome as an instrument {[calzavara2025_dynamic]} — see [[Programming:Crawler:PanoptiChrome]] for when it is nevertheless the only option. Foxhound tells you what to cite: its README's "Cite us!" section asks for the EuroS&P paper in which the browser is described, Klein et al. {[klein2022_handsanitizers]}, and its wiki separately lists the papers that have used it. |
| |
| ===== Agent-Driven Crawling ===== | ===== Agent-Driven Crawling ===== |
| |
| Since 2025 there is a fifth option that does not fit the two-layer model above: a | Since 2026 there is an option that does not fit the two-layer model above: a |
| **crawler whose next action is chosen by a model** rather than scripted. The open-source | **crawler whose next action is chosen by a model** rather than scripted. The open-source |
| ones — Browser Use, BrowserGym/AgentLab, Skyvern — still drive Chromium through CDP or | ones — Browser Use, BrowserGym/AgentLab, Skyvern — still drive Chromium through CDP or |
| gives you away with, why cloaking is the shape that ends up in your abstract, how to | gives you away with, why cloaking is the shape that ends up in your abstract, how to |
| measure the obstruction instead of fighting it, and what the corpus says about how rarely | measure the obstruction instead of fighting it, and what the corpus says about how rarely |
| anyone reports it. Two points from there bear on the choice made //on this page//: | anyone reports it. Three points bear on the choice made //on this page//: |
| |
| * **The framework is detectable, not just automation in general.** Vastel et al. {[vastel2018_scanner]} show fingerprint inconsistencies distinguish instrumented browsers from ordinary ones, and Krumnow et al. {[krumnow2022_gullible]} show OpenWPM specifically has detectors deployed against it in the wild. | * **The framework is detectable, not just automation in general.** Vastel et al. {[vastel2018_scanner]} show fingerprint inconsistencies distinguish instrumented browsers from ordinary ones, and Krumnow et al. {[krumnow2022_gullible]} show OpenWPM specifically has detectors deployed against it in the wild. |
| * **The anti-detection patches in the tables below are an arms race you lose quietly**, and most of those packages are now unmaintained or have moved to a successor under a different name; [[Programming:Crawler Detection]] dates each of them. Using them is also an ethics decision — see [[Practices:Ethics]] — because you are overriding a site operator's expressed access preference. | * **The anti-detection patches in the tables below are an arms race you lose quietly**, and most of those packages are now unmaintained or have moved to a successor under a different name; [[Programming:Crawler Detection]] dates each of them. Using them is also an ethics decision — see [[Practices:Ethics]] — because you are overriding a site operator's expressed access preference. |
| | * **If you take the agent option above, you are on a different detection surface, not just a more visible one.** Bot management now scores an //agent acting on a user's behalf// separately from both a human and a crawler collecting training data; [[Programming:Crawler Detection]] names the vendor categories and dates their defaults, which were still moving through 2026. The rules you are judged by are those, and they are not the two above: [[Programming:Crawler:LLM Agents#being_detected_and_the_ethics_of_not_being|the child page's detection section]] has them, including three 2026 papers that took three different positions on whether to evade at all. |
| |
| ===== Use in Publications ===== | ===== Use in Publications ===== |
| | Headless or headful | 140 | 12.5% | | | Headless or headful | 140 | 12.5% | |
| | Stateful or stateless | 219 | 19.6% | | | Stateful or stateless | 219 | 19.6% | |
| | Consent action | 349 | 31.2% | | | Consent action field populated (schema; not an audited paper claim) | 349 | 31.2% | |
| | Interaction depth | 841 | 75.1% | | | Interaction depth | 841 | 75.1% | |
| | Authentication | 779 | 69.6% | | | Authentication | 779 | 69.6% | |
| |
| ^ Family ^ N ^ States headless ^ States statefulness ^ States consent action ^ Public artifact ^ | The consent-action count above is field population, not proof that 349 papers stated a choice. The 2026-09-05 audit found **55 of 1,120 crawling papers (4.9%)** with an audited consent-action claim and **279 of 313 (89.1%)** unsupported ''no-interaction'' labels. The family percentages below retain the schema-field denominator and should be read the same way. |
| | |
| | ^ Family ^ N ^ States headless ^ States statefulness ^ Consent field populated (schema) ^ Public artifact ^ |
| | Selenium | 242 | 21.1% | 27.7% | 36.0% | 57.0% | | | Selenium | 242 | 21.1% | 27.7% | 36.0% | 57.0% | |
| | Puppeteer | 76 | 35.5% | 35.5% | 57.9% | 67.1% | | | Puppeteer | 76 | 35.5% | 35.5% | 57.9% | 67.1% | |
| | OpenWPM | 60 | 3 | 2015–2026 | | | OpenWPM | 60 | 3 | 2015–2026 | |
| | ''tbselenium'' / tor-browser-crawler | 21 | 0 | 2017–2026 | | | ''tbselenium'' / tor-browser-crawler | 21 | 0 | 2017–2026 | |
| | Tracker Radar Collector | 21 | 1 | 2021–2026 | | | Tracker Radar Collector((This row is the whole ''tracker.?radar'' fold, and the family is named after the crawler. Only **10** of the 21 ran the Collector; the other **11** used the //Tracker Radar dataset//, the entity list, the entity map or the wiki, filed by the extraction as a ''classification-service'' (9 papers), a ''cookie-database'' (1) or ''other'' (1). The folded-framework row above (10 papers, 0.9%) is the crawler-only count and agrees with [[Programming:Crawler:Tracker Radar Collector]]. Split by ''trc_index_reconcile.mjs''; the 11 papers are listed on [[provenance:programming:crawler:tracker_radar_collector]].)) | 21 | 1 | 2021–2026 | |
| | Anti-detection patches | 11 | 1 | 2021–2026 | | | Anti-detection patches | 11 | 1 | 2021–2026 | |
| | VisibleV8 | 9 | 0 | 2019–2026 | | | VisibleV8 | 9 | 0 | 2019–2026 | |
| | Brave PageGraph | 8 | 0 | 2020–2025 | | | Brave PageGraph | 8 | 0 | 2020–2025 | |
| | SAP Project Foxhound | 8 | 1 | 2024–2026 | | | SAP Project Foxhound((**9, not the 8 this row showed until 2026-09-21.** The column is computed from extracted ''tools[]'' tuples by ''usedOrMentioned'', and that filed Khodayari et al.'s NDSS 2025 open-redirection paper (cited on [[Programming:Crawler:Foxhound]]) under ''compared'' — the paper ran Foxhound as one of six baseline detectors over 42,288 pages, which is running it. The per-paper role audit on [[Programming:Crawler:Foxhound]] read all 13 full-text matches and its verdicts now drive this row, so //Papers only citing it// is 0: the two papers that cite Foxhound without running it never reached ''tools[]'' at all and are counted on that page, not here.)) | 9 | 0 | 2024–2026 | |
| | webXray | 7 | 0 | 2018–2022 | | | webXray | 7 | 0 | 2018–2022 | |
| | PanoptiChrome | 2 | 0 | 2024–2025 | | | PanoptiChrome | 2 | 0 | 2024–2025 | |
| |
| **OpenWPM is still the field's only broadly shared instrument**, but by less than it was: extending the corpus to 2026 roughly **tripled Tracker Radar Collector (8 → 21) and quadrupled Foxhound (2 → 8)** while OpenWPM grew 53 → 60. The 2024 cut-off in the earlier version of this table was doing real work — it caught the newer instruments mid-growth and made them look more marginal than they are. Everything below OpenWPM is still either niche, new, or a tool used mainly by the group that built it. | **OpenWPM is still the field's only broadly shared instrument**, but by less than it was: extending the corpus to 2026 roughly **tripled the Tracker Radar family (8 → 21) and more than quadrupled Foxhound (2 → 9)** while OpenWPM grew 53 → 60. Read the Tracker Radar growth with its footnote: most of this family is people //using the dataset//, and the crawler itself is the 10-paper row in the folded table above. The 2024 cut-off in the earlier version of this table was doing real work — it caught the newer instruments mid-growth and made them look more marginal than they are. Everything below OpenWPM is still either niche, new, or a tool used mainly by the group that built it. |
| |
| ==== Methodology and limitations of these figures ==== | ==== Methodology and limitations of these figures ==== |
| * **Silence is not absence.** "Does not name a framework" means the paper did not say, not that the authors used none. These are reporting figures. | * **Silence is not absence.** "Does not name a framework" means the paper did not say, not that the authors used none. These are reporting figures. |
| * **''used'' and ''produced'' both count** as driving a crawl — a paper that built its own crawler crawled with it. Tools only //compared// or //cited// do not, which is why the last table separates the two. | * **''used'' and ''produced'' both count** as driving a crawl — a paper that built its own crawler crawled with it. Tools only //compared// or //cited// do not, which is why the last table separates the two. |
| * **Quotes were spot-checked**, by ''scripts/quote_check.mjs'' rather than by hand. Of the **106** evidence quotes attached to an OpenWPM or Playwright tool mention anywhere in the corpus, 58 match the source text exactly once whitespace and line-break hyphenation are normalised, and 32 more match on at least 60% of their five-word windows. The remaining 16 were then read by hand against the full text: **all sixteen are present in the paper and none was unsupported.** The mismatch is always either a bracketed citation marker the extraction dropped (''the Open-WPM platform [52] on the Firefox browser'') or the two-column reading order still interleaving mid-sentence (''we created a Playwright**constrains the valid child elements, and everything else is moved based** [18] crawler''). | * **Quotes were spot-checked**, by ''scripts/quote_check.mjs'' rather than by hand. Of the **106** evidence quotes attached to an OpenWPM or Playwright tool mention anywhere in the corpus, 58 match the source text exactly once whitespace and line-break hyphenation are normalised, and 32 more match on at least 60% of their five-word windows. Of the remaining 16, **13 are located in an independent ''pypdf'' rendering of the same PDF** — a defect in the stored text rather than in the extraction — and **3** are below threshold in both renderings. All 16 were also read by hand against the full text: **all sixteen are present in the paper and none was unsupported.** The mismatch is always either a bracketed citation marker the extraction dropped (''the Open-WPM platform [52] on the Firefox browser''), which no rendering can repair and which is what the 3 are, or the two-column reading order still interleaving mid-sentence (''we created a Playwright**constrains the valid child elements, and everything else is moved based** [18] crawler''), which is what the 13 are. |
| * **Venue coverage.** Seven venues only; the scope, the selection funnel and what each stage of it costs are on [[literature:corpus]]. Notably, several of the tool papers this page recommends were published //outside// them — Krumnow et al. at CoNEXT, Klein et al. at EuroS&P, Libert in a communication journal — so a tool's paper count in this table is a lower bound on its standing. | * **Venue coverage.** Seven venues only; the scope, the selection funnel and what each stage of it costs are on [[literature:corpus]]. Notably, several of the tool papers this page recommends were published //outside// them — Krumnow et al. at CoNEXT, Klein et al. at EuroS&P, Libert in a communication journal — so a tool's paper count in this table is a lower bound on its standing. |
| * **''crawled'' is defined** as a paper whose crawl configuration was recorded or whose study types include an automated web crawl (1,120 papers, 19.1% of the corpus). This corpus is seven broad security venues, not a web-measurement corpus, so shares of all 5,859 papers would be meaningless here. | * **''crawled'' is defined** as a paper whose crawl configuration was recorded or whose study types include an automated web crawl (1,120 papers, 19.1% of the corpus). This corpus is seven broad security venues, not a web-measurement corpus, so shares of all 5,859 papers would be meaningless here. |
| * Multi-browser or multi-language, or a lab already fluent in it → Selenium. | * Multi-browser or multi-language, or a lab already fluent in it → Selenium. |
| * The lowest-effort modern tracking crawl → Tracker Radar Collector. | * The lowest-effort modern tracking crawl → Tracker Radar Collector. |
| - **Do not write a new crawler for a solved problem.** Nearly three in ten crawling papers used a home-grown one, and 75 of them identify it by nothing but a name they invented. If you must build one, say what it is built on, and publish it. | - **Do not write a new crawler for a solved problem.** Nearly three in ten crawling papers used a home-grown one, and 74 of them identify it by nothing but a name they invented. If you must build one, say what it is built on, and publish it. |
| - **State headless or headful and justify it.** Headless is the most detectable configuration you can choose {[vastel2018_scanner]} and only 12.5% of papers say which they used. | - **State headless or headful and justify it.** Headless is the most detectable configuration you can choose {[vastel2018_scanner]} and only 12.5% of papers say which they used. |
| - **Pin the browser binary, not just the library.** Puppeteer and Playwright do this for you; Selenium does not. Record the exact build and archive it with your research artefact (see [[:Artifacts]], a page this wiki still owes you). | - **Pin the browser binary, not just the library.** Puppeteer and Playwright do this for you; Selenium does not. Record the exact build and archive it with your research artefact (see [[:Artifacts]]). |
| - **Validate against something that is not your crawler.** A manual visit to a sample, a second browser engine, or a second vantage point {[jueckstock2021_realistic]}. Crawls diverge from human browsing in ways your crawl cannot see {[zeber2020representativeness]}, and repeated crawls diverge from each other {[demir2022_reproducibility]}. | - **Validate against something that is not your crawler.** A manual visit to a sample, a second browser engine, or a second vantage point {[jueckstock2021_realistic]}. Crawls diverge from human browsing in ways your crawl cannot see {[zeber2020representativeness]}, and repeated crawls diverge from each other {[demir2022_reproducibility]}. |
| - **Run more than once.** A single crawl is a point estimate. Report the spread. | - **Run more than once.** A single crawl is a point estimate. Report the spread. |
