Differences

This shows you the differences between two versions of the page.

--- programming:crawler [2026/09/17 10:45] – Clarify consentAction schema statistic; Authored by Claude karel.kubicek.claude
+++ programming:crawler [2026/09/21 14:35] (current) – Typo repair in the previous save: the comparison-table Foxhound cell read 89, not 9. Authored by Claude karel.kubicek.claude
@@ Line 326: / Line 326: @@
 | [[Programming:Crawler:PageGraph|Brave PageGraph]] | patched Brave (Blink + V8) | ''pagegraph-crawl'' (Node) | a causal **graph**: every DOM mutation, script execution and request attributed to its cause, plus Shields filter-rule effects; the production successor to AdGraph {[iqbal2020_adgraph]} | GraphML | in Brave ≥ 1.46, but JS-API recording needs Brave **Nightly** | **8** |
 | VisibleV8 {[jueckstock2019_visiblev8]} | patched Chromium (V8) | any CDP library | native-code JS API tracing inside V8, below the reach of page JavaScript | append-only trace logs | custom Chromium build | **9** |
-| [[Programming:Crawler:Foxhound|SAP Project Foxhound]] | patched Firefox (Gecko + SpiderMonkey) | Playwright((The only automation layer the repository documents; it carries its own ''.PLAYWRIGHT_VERSION'' and a CI workflow for it.)) | **dynamic taint tracking**: string-level flows from sources to sinks | ''__taintreport'' DOM events | full Firefox build toolchain, or prebuilt binaries | **8** |
+| [[Programming:Crawler:Foxhound|SAP Project Foxhound]] | patched Firefox (Gecko + SpiderMonkey) | Playwright((The only automation layer the repository documents; it carries its own ''.PLAYWRIGHT_VERSION'' and a CI workflow for it.)) | **dynamic taint tracking**: string-level flows from sources to sinks | ''__taintreport'' DOM events | full Firefox build toolchain, or prebuilt binaries | **9** |
 | [[Programming:Crawler:PanoptiChrome|PanoptiChrome]] {[kanyal2024_panoptichrome]} | patched Chromium/V8, pinned to Chrome 116 | — | dynamic taint tracking in Chromium, on objects rather than strings, including implicit flows | one plain-text log per V8 isolate, in VisibleV8's format | build a pinned Chrome 116 fork from patches — the patch published with the paper **does not compile**; use the completed one, or a third party's prebuilt binary | **2** |
 | [[Programming:Crawler:webXray|webXray]] {[libert2015_invisible]} | PhantomJS in 2015; **consumer Chrome over raw CDP** in the last public version | — | third-party request detection plus **domain-to-company attribution** | SQLite or Postgres, CSV reports | //see below// | **7** |
@@ Line 504: / Line 504: @@
 | VisibleV8 | 9 | 0 | 2019–2026 |
 | Brave PageGraph | 8 | 0 | 2020–2025 |
-| SAP Project Foxhound | 8 | 1 | 2024–2026 |
+| SAP Project Foxhound((**9, not the 8 this row showed until 2026-09-21.** The column is computed from extracted ''tools[]'' tuples by ''usedOrMentioned'', and that filed Khodayari et al.'s NDSS 2025 open-redirection paper (cited on [[Programming:Crawler:Foxhound]]) under ''compared'' — the paper ran Foxhound as one of six baseline detectors over 42,288 pages, which is running it. The per-paper role audit on [[Programming:Crawler:Foxhound]] read all 13 full-text matches and its verdicts now drive this row, so //Papers only citing it// is 0: the two papers that cite Foxhound without running it never reached ''tools[]'' at all and are counted on that page, not here.)) | 9 | 0 | 2024–2026 |
 | webXray | 7 | 0 | 2018–2022 |
 | PanoptiChrome | 2 | 0 | 2024–2025 |
-**OpenWPM is still the field's only broadly shared instrument**, but by less than it was: extending the corpus to 2026 roughly **tripled the Tracker Radar family (8 → 21) and quadrupled Foxhound (2 → 8)** while OpenWPM grew 53 → 60. Read the Tracker Radar growth with its footnote: most of this family is people //using the dataset//, and the crawler itself is the 10-paper row in the folded table above. The 2024 cut-off in the earlier version of this table was doing real work — it caught the newer instruments mid-growth and made them look more marginal than they are. Everything below OpenWPM is still either niche, new, or a tool used mainly by the group that built it.
+**OpenWPM is still the field's only broadly shared instrument**, but by less than it was: extending the corpus to 2026 roughly **tripled the Tracker Radar family (8 → 21) and more than quadrupled Foxhound (2 → 9)** while OpenWPM grew 53 → 60. Read the Tracker Radar growth with its footnote: most of this family is people //using the dataset//, and the crawler itself is the 10-paper row in the folded table above. The 2024 cut-off in the earlier version of this table was doing real work — it caught the newer instruments mid-growth and made them look more marginal than they are. Everything below OpenWPM is still either niche, new, or a tool used mainly by the group that built it.
 ==== Methodology and limitations of these figures ====
@@ Line 517: / Line 517: @@
   * **Silence is not absence.** "Does not name a framework" means the paper did not say, not that the authors used none. These are reporting figures.
   * **''used'' and ''produced'' both count** as driving a crawl — a paper that built its own crawler crawled with it. Tools only //compared// or //cited// do not, which is why the last table separates the two.
-  * **Quotes were spot-checked**, by ''scripts/quote_check.mjs'' rather than by hand. Of the **106** evidence quotes attached to an OpenWPM or Playwright tool mention anywhere in the corpus, 58 match the source text exactly once whitespace and line-break hyphenation are normalised, and 32 more match on at least 60% of their five-word windows. The remaining 16 were then read by hand against the full text: **all sixteen are present in the paper and none was unsupported.** The mismatch is always either a bracketed citation marker the extraction dropped (''the Open-WPM platform [52] on the Firefox browser'') or the two-column reading order still interleaving mid-sentence (''we created a Playwright**constrains the valid child elements, and everything else is moved based** [18] crawler'').
+  * **Quotes were spot-checked**, by ''scripts/quote_check.mjs'' rather than by hand. Of the **106** evidence quotes attached to an OpenWPM or Playwright tool mention anywhere in the corpus, 58 match the source text exactly once whitespace and line-break hyphenation are normalised, and 32 more match on at least 60% of their five-word windows. Of the remaining 16, **13 are located in an independent ''pypdf'' rendering of the same PDF** — a defect in the stored text rather than in the extraction — and **3** are below threshold in both renderings. All 16 were also read by hand against the full text: **all sixteen are present in the paper and none was unsupported.** The mismatch is always either a bracketed citation marker the extraction dropped (''the Open-WPM platform [52] on the Firefox browser''), which no rendering can repair and which is what the 3 are, or the two-column reading order still interleaving mid-sentence (''we created a Playwright**constrains the valid child elements, and everything else is moved based** [18] crawler''), which is what the 13 are.
   * **Venue coverage.** Seven venues only; the scope, the selection funnel and what each stage of it costs are on [[literature:corpus]]. Notably, several of the tool papers this page recommends were published //outside// them — Krumnow et al. at CoNEXT, Klein et al. at EuroS&P, Libert in a communication journal — so a tool's paper count in this table is a lower bound on its standing.
   * **''crawled'' is defined** as a paper whose crawl configuration was recorded or whose study types include an automated web crawl (1,120 papers, 19.1% of the corpus). This corpus is seven broad security venues, not a web-measurement corpus, so shares of all 5,859 papers would be meaningless here.