Differences

This shows you the differences between two versions of the page.

--- programming:crawler [2026/09/17 07:22] – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude
+++ programming:crawler [2026/09/21 14:35] (current) – Typo repair in the previous save: the comparison-table Foxhound cell read 89, not 9. Authored by Claude karel.kubicek.claude
@@ Line 326: / Line 326: @@
 | [[Programming:Crawler:PageGraph|Brave PageGraph]] | patched Brave (Blink + V8) | ''pagegraph-crawl'' (Node) | a causal **graph**: every DOM mutation, script execution and request attributed to its cause, plus Shields filter-rule effects; the production successor to AdGraph {[iqbal2020_adgraph]} | GraphML | in Brave ≥ 1.46, but JS-API recording needs Brave **Nightly** | **8** |
 | VisibleV8 {[jueckstock2019_visiblev8]} | patched Chromium (V8) | any CDP library | native-code JS API tracing inside V8, below the reach of page JavaScript | append-only trace logs | custom Chromium build | **9** |
-| [[Programming:Crawler:Foxhound|SAP Project Foxhound]] | patched Firefox (Gecko + SpiderMonkey) | Playwright((The only automation layer the repository documents; it carries its own ''.PLAYWRIGHT_VERSION'' and a CI workflow for it.)) | **dynamic taint tracking**: string-level flows from sources to sinks | ''__taintreport'' DOM events | full Firefox build toolchain, or prebuilt binaries | **8** |
+| [[Programming:Crawler:Foxhound|SAP Project Foxhound]] | patched Firefox (Gecko + SpiderMonkey) | Playwright((The only automation layer the repository documents; it carries its own ''.PLAYWRIGHT_VERSION'' and a CI workflow for it.)) | **dynamic taint tracking**: string-level flows from sources to sinks | ''__taintreport'' DOM events | full Firefox build toolchain, or prebuilt binaries | **9** |
 | [[Programming:Crawler:PanoptiChrome|PanoptiChrome]] {[kanyal2024_panoptichrome]} | patched Chromium/V8, pinned to Chrome 116 | — | dynamic taint tracking in Chromium, on objects rather than strings, including implicit flows | one plain-text log per V8 isolate, in VisibleV8's format | build a pinned Chrome 116 fork from patches — the patch published with the paper **does not compile**; use the completed one, or a third party's prebuilt binary | **2** |
 | [[Programming:Crawler:webXray|webXray]] {[libert2015_invisible]} | PhantomJS in 2015; **consumer Chrome over raw CDP** in the last public version | — | third-party request detection plus **domain-to-company attribution** | SQLite or Postgres, CSV reports | //see below// | **7** |
@@ Line 474: / Line 474: @@
 | Headless or headful | 140 | 12.5% |
 | Stateful or stateless | 219 | 19.6% |
-| Consent action | 349 | 31.2% |
+| Consent action field populated (schema; not an audited paper claim) | 349 | 31.2% |
 | Interaction depth | 841 | 75.1% |
 | Authentication | 779 | 69.6% |
-^ Family ^ N ^ States headless ^ States statefulness ^ States consent action ^ Public artifact ^
+The consent-action count above is field population, not proof that 349 papers stated a choice. The 2026-09-05 audit found **55 of 1,120 crawling papers (4.9%)** with an audited consent-action claim and **279 of 313 (89.1%)** unsupported ''no-interaction'' labels. The family percentages below retain the schema-field denominator and should be read the same way.
+^ Family ^ N ^ States headless ^ States statefulness ^ Consent field populated (schema) ^ Public artifact ^
 | Selenium | 242 | 21.1% | 27.7% | 36.0% | 57.0% |
 | Puppeteer | 76 | 35.5% | 35.5% | 57.9% | 67.1% |
@@ Line 502: / Line 504: @@
 | VisibleV8 | 9 | 0 | 2019–2026 |
 | Brave PageGraph | 8 | 0 | 2020–2025 |
-| SAP Project Foxhound | 8 | 1 | 2024–2026 |
+| SAP Project Foxhound((**9, not the 8 this row showed until 2026-09-21.** The column is computed from extracted ''tools[]'' tuples by ''usedOrMentioned'', and that filed Khodayari et al.'s NDSS 2025 open-redirection paper (cited on [[Programming:Crawler:Foxhound]]) under ''compared'' — the paper ran Foxhound as one of six baseline detectors over 42,288 pages, which is running it. The per-paper role audit on [[Programming:Crawler:Foxhound]] read all 13 full-text matches and its verdicts now drive this row, so //Papers only citing it// is 0: the two papers that cite Foxhound without running it never reached ''tools[]'' at all and are counted on that page, not here.)) | 9 | 0 | 2024–2026 |
 | webXray | 7 | 0 | 2018–2022 |
 | PanoptiChrome | 2 | 0 | 2024–2025 |
-**OpenWPM is still the field's only broadly shared instrument**, but by less than it was: extending the corpus to 2026 roughly **tripled the Tracker Radar family (8 → 21) and quadrupled Foxhound (2 → 8)** while OpenWPM grew 53 → 60. Read the Tracker Radar growth with its footnote: most of this family is people //using the dataset//, and the crawler itself is the 10-paper row in the folded table above. The 2024 cut-off in the earlier version of this table was doing real work — it caught the newer instruments mid-growth and made them look more marginal than they are. Everything below OpenWPM is still either niche, new, or a tool used mainly by the group that built it.
+**OpenWPM is still the field's only broadly shared instrument**, but by less than it was: extending the corpus to 2026 roughly **tripled the Tracker Radar family (8 → 21) and more than quadrupled Foxhound (2 → 9)** while OpenWPM grew 53 → 60. Read the Tracker Radar growth with its footnote: most of this family is people //using the dataset//, and the crawler itself is the 10-paper row in the folded table above. The 2024 cut-off in the earlier version of this table was doing real work — it caught the newer instruments mid-growth and made them look more marginal than they are. Everything below OpenWPM is still either niche, new, or a tool used mainly by the group that built it.
 ==== Methodology and limitations of these figures ====
@@ Line 515: / Line 517: @@
   * **Silence is not absence.** "Does not name a framework" means the paper did not say, not that the authors used none. These are reporting figures.
   * **''used'' and ''produced'' both count** as driving a crawl — a paper that built its own crawler crawled with it. Tools only //compared// or //cited// do not, which is why the last table separates the two.
-  * **Quotes were spot-checked**, by ''scripts/quote_check.mjs'' rather than by hand. Of the **106** evidence quotes attached to an OpenWPM or Playwright tool mention anywhere in the corpus, 58 match the source text exactly once whitespace and line-break hyphenation are normalised, and 32 more match on at least 60% of their five-word windows. The remaining 16 were then read by hand against the full text: **all sixteen are present in the paper and none was unsupported.** The mismatch is always either a bracketed citation marker the extraction dropped (''the Open-WPM platform [52] on the Firefox browser'') or the two-column reading order still interleaving mid-sentence (''we created a Playwright**constrains the valid child elements, and everything else is moved based** [18] crawler'').
+  * **Quotes were spot-checked**, by ''scripts/quote_check.mjs'' rather than by hand. Of the **106** evidence quotes attached to an OpenWPM or Playwright tool mention anywhere in the corpus, 58 match the source text exactly once whitespace and line-break hyphenation are normalised, and 32 more match on at least 60% of their five-word windows. Of the remaining 16, **13 are located in an independent ''pypdf'' rendering of the same PDF** — a defect in the stored text rather than in the extraction — and **3** are below threshold in both renderings. All 16 were also read by hand against the full text: **all sixteen are present in the paper and none was unsupported.** The mismatch is always either a bracketed citation marker the extraction dropped (''the Open-WPM platform [52] on the Firefox browser''), which no rendering can repair and which is what the 3 are, or the two-column reading order still interleaving mid-sentence (''we created a Playwright**constrains the valid child elements, and everything else is moved based** [18] crawler''), which is what the 13 are.
   * **Venue coverage.** Seven venues only; the scope, the selection funnel and what each stage of it costs are on [[literature:corpus]]. Notably, several of the tool papers this page recommends were published //outside// them — Krumnow et al. at CoNEXT, Klein et al. at EuroS&P, Libert in a communication journal — so a tool's paper count in this table is a lower bound on its standing.
   * **''crawled'' is defined** as a paper whose crawl configuration was recorded or whose study types include an automated web crawl (1,120 papers, 19.1% of the corpus). This corpus is seven broad security venues, not a web-measurement corpus, so shares of all 5,859 papers would be meaningless here.
@@ Line 531: / Line 533: @@
   - **Do not write a new crawler for a solved problem.** Nearly three in ten crawling papers used a home-grown one, and 74 of them identify it by nothing but a name they invented. If you must build one, say what it is built on, and publish it.
   - **State headless or headful and justify it.** Headless is the most detectable configuration you can choose {[vastel2018_scanner]} and only 12.5% of papers say which they used.
-  - **Pin the browser binary, not just the library.** Puppeteer and Playwright do this for you; Selenium does not. Record the exact build and archive it with your research artefact (see [[:Artifacts]], a page this wiki still owes you).
+  - **Pin the browser binary, not just the library.** Puppeteer and Playwright do this for you; Selenium does not. Record the exact build and archive it with your research artefact (see [[:Artifacts]]).
   - **Validate against something that is not your crawler.** A manual visit to a sample, a second browser engine, or a second vantage point {[jueckstock2021_realistic]}. Crawls diverge from human browsing in ways your crawl cannot see {[zeber2020representativeness]}, and repeated crawls diverge from each other {[demir2022_reproducibility]}.
   - **Run more than once.** A single crawl is a point estimate. Report the spread.