| Next revision | Previous revision |
| provenance:programming:crawler [2026/08/12 09:17] – Create provenance page for programming:crawler: queries, denominators, fold changes and residue, quote verification, and the 4,322 -> 5,859 refresh diff. Partly reconstructed after the fact; marked as such. Authored by Claude. karel.kubicek.claude | provenance:programming:crawler [2026/08/29 06:37] (current) – Record the LLM-agent fold: 5.3 resolved (MCP Server Crawler correctly excluded, it is not an agent), five figures in section 4 updated, links to the new child page and its provenance. Authored by Claude karel.kubicek.claude |
|---|
| </code> | </code> |
| |
| ''check_page_numbers.mjs'' left four figures unaccounted on 2026-08-12, all deliberate: ''100'' ("not against 100%"), ''19.1'' (the crawled share of the corpus, from ''data/extract/OVERVIEW.md''), ''53'' (the **old** OpenWPM count, quoted on the page as history), and ''93'' — which was a stale number from an intermediate run of the quote check and was corrected to 106 before saving. That last one is the guard doing exactly the job it exists for. | ''check_page_numbers.mjs'' left four figures unaccounted on 2026-08-12. **Three are permanent and legitimate** and are named here so the next run does not re-investigate them: ''100'' ("not against 100%"), ''19.1'' (the crawled share of the corpus, from ''data/extract/OVERVIEW.md'') and ''106'' (the quote-check total, from ''out/new/quote_check_crawler.txt'', not from this report). A fourth, ''53'', is the **old** OpenWPM count quoted on the page as history. The guard also caught a real error: the page briefly carried ''93'' from an intermediate run of the quote check, corrected to 106 before saving. Run it whole-page too (§10). |
| |
| ===== 4. What the refresh changed ===== | ===== 4. What the refresh changed ===== |
| | Names neither | 248 (28.9%) | 311 (27.8%) | | | Names neither | 248 (28.9%) | 311 (27.8%) | |
| | Selenium | 187 (21.8%) | 242 (21.6%) | | | Selenium | 187 (21.8%) | 242 (21.6%) | |
| | Bespoke, own name | 98 (11.4%) | 184 (16.4%) | | | Bespoke, own name | 98 (11.4%) | 184 (16.4%) → **181 (16.2%)** on 2026-08-29((The LLM-agent fold, §5.3.)) | |
| | Bespoke, generic | 130 (15.1%) | 147 (13.1%) | | | Bespoke, generic | 130 (15.1%) | 147 (13.1%) | |
| | Bespoke, union | 221 (25.7%) | 321 (28.7%) | | | Bespoke, union | 221 (25.7%) | 321 (28.7%) → **318 (28.4%)** on 2026-08-29 | |
| | …of which named nothing else | 35 | 75 | | | …of which named nothing else | 35 | 75 → **74** on 2026-08-29 | |
| | Puppeteer | 54 (6.3%) | 76 (6.8%) | | | Puppeteer | 54 (6.3%) | 76 (6.8%) | |
| | OpenWPM | 51 (5.9%) | 58 (5.2%) | | | OpenWPM | 51 (5.9%) | 58 (5.2%) | |
| | Any tool states its version | 63 of 544 (11.6%) | 87 of 723 (12.0%) | | | Any tool states its version | 63 of 544 (11.6%) | 87 of 723 (12.0%) | |
| | Distinct tool strings / mentions | 378 / 789 | 501 / 1,075 | | | Distinct tool strings / mentions | 378 / 789 | 501 / 1,075 | |
| | Unmapped residue | 104 strings / 106 mentions | 204 / 210 | | | Unmapped residue | 104 strings / 106 mentions | 204 / 210 → **199 / 205** on 2026-08-29 | |
| | Exact string "Selenium" vs folded | 147 vs 187 | 194 vs 242 | | | Exact string "Selenium" vs folded | 147 vs 187 | 194 vs 242 | |
| | Artifact share, 2010–2013 → 2022–2024 | 23.6% → 62.1% | 21.6% → 61.2% | | | Artifact share, 2010–2013 → 2022–2024 | 23.6% → 62.1% | 21.6% → 61.2% | |
| * **Tracker Radar Collector 8 → 21 and Foxhound 2 → 8 in the all-papers mention table.** The old page's closing sentence for that table already warned that "the 2024 boundary cuts them off mid-growth". It did, and by a factor of two to four. The sentence was rewritten to say so rather than to warn about it. | * **Tracker Radar Collector 8 → 21 and Foxhound 2 → 8 in the all-papers mention table.** The old page's closing sentence for that table already warned that "the 2024 boundary cuts them off mid-growth". It did, and by a factor of two to four. The sentence was rewritten to say so rather than to warn about it. |
| * **"Names a library, not a browser" 23 → 49**, growing about twice as fast as the corpus. The new strings are the reason: 2025–2026 papers write "a Playwright-controlled browser", "Playwright-derived browser", "Puppeteer/TRC desktop browser". | * **"Names a library, not a browser" 23 → 49**, growing about twice as fast as the corpus. The new strings are the reason: 2025–2026 papers write "a Playwright-controlled browser", "Playwright-derived browser", "Puppeteer/TRC desktop browser". |
| * **Bespoke, own name 98 → 184.** Corpus growth plus the same effect the README already documents: including ''produced'' in the tool query surfaces one-off names. | * **Bespoke, own name 98 → 184, then 181.** Corpus growth plus the same effect the README already documents: including ''produced'' in the tool query surfaces one-off names. On **2026-08-29** the LLM-agent family (§5.3) claimed five of those strings, moving the row to 181 and the union to 318. |
| * **Artifact share 23.6% → 21.6% for 2010–2013.** A genuine change in the same window, from the newly retrieved IEEE S&P papers landing in the early buckets. | * **Artifact share 23.6% → 21.6% for 2010–2013.** A genuine change in the same window, from the newly retrieved IEEE S&P papers landing in the early buckets. |
| |
| Exact string "Selenium": 194 papers. Folded Selenium family: 242 papers | Exact string "Selenium": 194 papers. Folded Selenium family: 242 papers |
| (exact-string counting would undercount by 19.8%). | (exact-string counting would undercount by 19.8%). |
| Bespoke: 147 generic + 184 own-name, 10 in both, union 321 of 1120 (28.7%). | Bespoke: 147 generic + 181 own-name, 10 in both, union 318 of 1120 (28.4%). |
| Papers naming any of Puppeteer, Playwright, Chrome DevTools Protocol (direct), | Papers naming any of Puppeteer, Playwright, Chrome DevTools Protocol (direct), |
| Scrapy, OpenWPM: 215. | Scrapy, OpenWPM: 215. |
| </code> | </code> |
| |
| The 204-string residue is printed in full at the end of ''out/new/report_crawler.txt''. It is too long to reproduce here and it is genuinely one-name-per-paper; the interesting part is what is **new** in it, below. | The residue — 204 strings until 2026-08-29, 199 after the LLM-agent fold — is printed in full at the end of ''out/new/report_crawler.txt''. It is too long to reproduce here and it is genuinely one-name-per-paper; the interesting part is what is **new** in it, below. |
| |
| ==== 5.2 Browser families — the fold that had gone stale ==== | ==== 5.2 Browser families — the fold that had gone stale ==== |
| ==== 5.3 What is new in the framework residue, and what was deliberately not done with it ==== | ==== 5.3 What is new in the framework residue, and what was deliberately not done with it ==== |
| |
| //Recorded.// The 2025–2026 slice introduced a cluster of one-off names that are not bespoke crawlers in the old sense but **LLM-agent browser automation**: ''Browser-Use'' / ''browser-use'' (two spellings, two papers), ''AgentLab'', ''BrowserGym'', ''MCP Server Crawler''. A sixth paper, ''USENIX/2026/the-state-of-passkeys…'', drives Playwright //through// Browser Use. | //Recorded 2026-08-12; **resolved 2026-08-29**.// The 2025–2026 slice introduced a cluster of one-off names that are not bespoke crawlers in the old sense but **LLM-agent browser automation**: ''Browser-Use'' / ''browser-use'' / ''Browser Use'' (three spellings, three papers), ''AgentLab'', ''BrowserGym''. A sixth string, ''MCP Server Crawler'', was listed with them on 2026-08-12 and **that was wrong** — see below. |
| |
| **This was deliberately not folded into a family and not written up on the page.** The refresh brief for this run is a mechanical re-derivation, and "does the field now drive browsers with LLM agents" is a new claim needing its own population, its own dating and its own reading of the papers — five or six papers in the thinnest, provisional years of the corpus is not enough to call it current practice. It is logged as a separate work item instead. Anyone picking it up should start from this residue list and from ''programming:crawler''’s //Being Detected// section, since an agent-driven browser is a different detection surface. | The 2026-08-12 run deliberately did not fold it, on the grounds that "does the field now drive browsers with LLM agents" is a new claim needing its own population, its own dating and its own reading of the papers. That work was done on **2026-08-29**. The outcome: |
| | |
| | * A family **''LLM browser agents''** now exists in ''tool_fold.mjs'', ordered //before// Playwright and Puppeteer, because Browser Use and Skyvern drive Playwright underneath and would otherwise be counted as Playwright. |
| | * It claims **4 of the 1,120 crawling papers (0.4%)**, all of them 2026. That is the row now on [[programming:crawler]], and the five figures it moved are in §4 above. |
| | * ''MCP Server Crawler'' was **excluded**, and the 2026-08-12 note that grouped it with the agents is corrected here. Reading IEEE-SP/2026 //Parasites in the Toolchain// shows it is an ordinary crawler over MCP directory listings and GitHub links — "we implemented the MCP Server Crawler to leverage these links as the entry point for data collection". The name describes what it crawls, not what drives it. It stays in the printed residue. ''WebAgentDriver'' (NDSS/2026 //ChameleoScan//), the Appium XCUITest driver on jailbroken iOS, is excluded for the same kind of reason. |
| | * The dating question was answered rather than deferred: **9 papers in 5,859 name an agent framework at all, 1 in 2025 and 8 in 2026**, and six of the nine //measure// agents rather than crawl with them. It is not current practice and the new page says so. |
| | * The page is [[programming:crawler:llm_agents]] and its own provenance is [[provenance:programming:crawler:llm_agents]], which carries the full query log, the fold rule, the residue and the quote adjudication. |
| |
| A second, smaller cluster in the same residue: consent-automation tools used as instruments — ''autoconsent'', ''DuckDuckGo autoconsent'', ''Super Agent-Automatic Cookie Consent'', ''ConsentChk''. Those belong to [[privacy:consent]] rather than here. | A second, smaller cluster in the same residue: consent-automation tools used as instruments — ''autoconsent'', ''DuckDuckGo autoconsent'', ''Super Agent-Automatic Cookie Consent'', ''ConsentChk''. Those belong to [[privacy:consent]] rather than here. |
| * **The sandbox measurements were not re-run.** The Selenium/Puppeteer/Playwright/CDP comparison, the WebDriver BiDi ''responseCompleted'' finding and the "two network handlers cross-deliver" bug are from 2026-08-06 against ''selenium-webdriver'' 4.46. Library versions have moved since. Nothing in this refresh touched them, and nothing in this refresh verified them. | * **The sandbox measurements were not re-run.** The Selenium/Puppeteer/Playwright/CDP comparison, the WebDriver BiDi ''responseCompleted'' finding and the "two network handlers cross-deliver" bug are from 2026-08-06 against ''selenium-webdriver'' 4.46. Library versions have moved since. Nothing in this refresh touched them, and nothing in this refresh verified them. |
| |
| ===== 9. Run log ===== | ===== 10. Review pass, 2026-08-12 ===== |
| | |
| | //Recorded.// The refresh was reviewed by a second model (Claude Fable 5), told explicitly that the summary it was given might not be exhaustive, with instructions to hunt stale numbers. It found a **systematic defect, not a scatter of typos**, and it is worth stating because it will recur on the next refresh: |
| | |
| | * ''check_page_numbers.mjs'' was run with a **heading window** — normally ''Use in Publications'' to the next section — so it audited only the corpus section. Every corpus figure repeated in a page's **intro, tooling section, recommendations, footnotes, Related Pages or an embedded code block** was outside the window and stayed at its 4,322-corpus value. Across the six pages **29 such figures** survived the first pass. |
| | * The matcher was **substring**, not word-boundary, so ''report.includes('59')'' was satisfied by ''11.59 bits''. One genuinely stale figure sat inside a checked window and passed for that reason. |
| | |
| | Both are fixed in ''scripts/check_page_numbers.mjs'': matching is now anchored with lookarounds, ISO dates and URLs are stripped before scanning, ''--code'' opts into scanning ''<file>'' blocks, and omitting the heading markers checks the whole page. **Run it windowed //and// whole-page.** The whole-page run is noisy — a page's non-corpus half is full of figures quoted from other papers — so read its output rather than expecting it to exit clean. |
| | Fixed on this page's content page as a result: **11.6% → 12.0%** (twice), Selenium **21.8% → 21.6%**, Puppeteer **6.3% → 6.8%**, direct CDP **35 papers (4.1%) → 42 (3.8%)**, headless **12.9% → 12.5%**, and unnamed bespoke crawlers **34 → 75**. Two prose corrections the numbers forced: |
| | |
| | * The Playwright section still said "still rare in the literature — 10 papers", two screens from a corpus table saying 34 and a sentence saying Playwright overtakes Puppeteer. Rewritten. |
| | * "OpenWPM users are more than twice as good as average … Playwright's cohort is next best" **ranked the two backwards** against the table directly above it (Playwright 26.5%, OpenWPM 25.9%). On the old corpus Playwright led 50.0% to 25.5% on a base of ten papers; with 34 they are effectively tied, and the sentence now says so. |
| | * The framework table had drifted out of descending order after the refresh moved three rows. Restored. |
| | |
| | **''106'' is a permanent legitimate unaccounted figure** on this page: it is the quote-check total, from ''out/new/quote_check_crawler.txt'', not from ''report_crawler.mjs''. Recorded here so the next run does not re-investigate it. §3 above previously said "four … all deliberate" while listing one figure (''93'') that was a real error — corrected. |
| | |
| | ===== 11. Run log ===== |
| |
| ^ ^ ^ | ^ ^ ^ |
| | Mistake caught in review | The page briefly carried "231" for that combined count — arrived at by summing family counts, which double-counts papers naming two of them. The real union is 215. ''check_page_numbers.mjs'' flagged it as unaccounted, which is how it was caught; it was then computed in the report script rather than by hand. | | | Mistake caught in review | The page briefly carried "231" for that combined count — arrived at by summing family counts, which double-counts papers naming two of them. The real union is 215. ''check_page_numbers.mjs'' flagged it as unaccounted, which is how it was caught; it was then computed in the report script rather than by hand. | |
| | Deferred | LLM-agent browser automation in the 2025–2026 residue (§5.3), logged as a separate work item. | | | Deferred | LLM-agent browser automation in the 2025–2026 residue (§5.3), logged as a separate work item. | |
| | | Review | Reviewed by Claude Fable 5 on 2026-08-12 with the instruction that the summary might not be exhaustive. It found the windowed-guard defect in §10 and 7 stale figures plus a mis-ordered table and a backwards ranking sentence on this page. All fixes were applied and re-saved the same day. | |
| |
| [[programming:crawler|← back to the content page]] · [[literature:corpus|corpus-level provenance]] | [[programming:crawler|← back to the content page]] · [[literature:corpus|corpus-level provenance]] |
| |
