Differences

This shows you the differences between two versions of the page.

--- provenance:programming:crawler [2026/08/12 10:07] – Add section 10: the Fable review pass, the windowed-guard defect it exposed (29 corpus figures stale outside the audited heading window across six pages), the hardened check_page_numbers.mjs, and the page-specific fixes. Authored by Claude. karel.kubicek.claude
+++ provenance:programming:crawler [2026/08/29 06:37] (current) – Record the LLM-agent fold: 5.3 resolved (MCP Server Crawler correctly excluded, it is not an agent), five figures in section 4 updated, links to the new child page and its provenance. Authored by Claude karel.kubicek.claude
@@ Line 55: / Line 55: @@
 | Names neither | 248 (28.9%) | 311 (27.8%) |
 | Selenium | 187 (21.8%) | 242 (21.6%) |
-| Bespoke, own name | 98 (11.4%) | 184 (16.4%) |
+| Bespoke, own name | 98 (11.4%) | 184 (16.4%) → **181 (16.2%)** on 2026-08-29((The LLM-agent fold, §5.3.)) |
 | Bespoke, generic | 130 (15.1%) | 147 (13.1%) |
-| Bespoke, union | 221 (25.7%) | 321 (28.7%) |
+| Bespoke, union | 221 (25.7%) | 321 (28.7%) → **318 (28.4%)** on 2026-08-29 |
-| …of which named nothing else | 35 | 75 |
+| …of which named nothing else | 35 | 75 → **74** on 2026-08-29 |
 | Puppeteer | 54 (6.3%) | 76 (6.8%) |
 | OpenWPM | 51 (5.9%) | 58 (5.2%) |
@@ Line 71: / Line 71: @@
 | Any tool states its version | 63 of 544 (11.6%) | 87 of 723 (12.0%) |
 | Distinct tool strings / mentions | 378 / 789 | 501 / 1,075 |
-| Unmapped residue | 104 strings / 106 mentions | 204 / 210 |
+| Unmapped residue | 104 strings / 106 mentions | 204 / 210 → **199 / 205** on 2026-08-29 |
 | Exact string "Selenium" vs folded | 147 vs 187 | 194 vs 242 |
 | Artifact share, 2010–2013 → 2022–2024 | 23.6% → 62.1% | 21.6% → 61.2% |
@@ Line 87: / Line 87: @@
   * **Tracker Radar Collector 8 → 21 and Foxhound 2 → 8 in the all-papers mention table.** The old page's closing sentence for that table already warned that "the 2024 boundary cuts them off mid-growth". It did, and by a factor of two to four. The sentence was rewritten to say so rather than to warn about it.
   * **"Names a library, not a browser" 23 → 49**, growing about twice as fast as the corpus. The new strings are the reason: 2025–2026 papers write "a Playwright-controlled browser", "Playwright-derived browser", "Puppeteer/TRC desktop browser".
-  * **Bespoke, own name 98 → 184.** Corpus growth plus the same effect the README already documents: including ''produced'' in the tool query surfaces one-off names.
+  * **Bespoke, own name 98 → 184, then 181.** Corpus growth plus the same effect the README already documents: including ''produced'' in the tool query surfaces one-off names. On **2026-08-29** the LLM-agent family (§5.3) claimed five of those strings, moving the row to 181 and the union to 318.
   * **Artifact share 23.6% → 21.6% for 2010–2013.** A genuine change in the same window, from the newly retrieved IEEE S&P papers landing in the early buckets.
@@ Line 104: / Line 104: @@
 Exact string "Selenium": 194 papers. Folded Selenium family: 242 papers
   (exact-string counting would undercount by 19.8%).
-Bespoke: 147 generic + 184 own-name, 10 in both, union 321 of 1120 (28.7%).
+Bespoke: 147 generic + 181 own-name, 10 in both, union 318 of 1120 (28.4%).
 Papers naming any of Puppeteer, Playwright, Chrome DevTools Protocol (direct),
   Scrapy, OpenWPM: 215.
 </code>
-The 204-string residue is printed in full at the end of ''out/new/report_crawler.txt''. It is too long to reproduce here and it is genuinely one-name-per-paper; the interesting part is what is **new** in it, below.
+The residue — 204 strings until 2026-08-29, 199 after the LLM-agent fold — is printed in full at the end of ''out/new/report_crawler.txt''. It is too long to reproduce here and it is genuinely one-name-per-paper; the interesting part is what is **new** in it, below.
 ==== 5.2 Browser families — the fold that had gone stale ====
@@ Line 130: / Line 130: @@
 ==== 5.3 What is new in the framework residue, and what was deliberately not done with it ====
-//Recorded.// The 2025–2026 slice introduced a cluster of one-off names that are not bespoke crawlers in the old sense but **LLM-agent browser automation**: ''Browser-Use'' / ''browser-use'' (two spellings, two papers), ''AgentLab'', ''BrowserGym'', ''MCP Server Crawler''. A sixth paper, ''USENIX/2026/the-state-of-passkeys…'', drives Playwright //through// Browser Use.
+//Recorded 2026-08-12; **resolved 2026-08-29**.// The 2025–2026 slice introduced a cluster of one-off names that are not bespoke crawlers in the old sense but **LLM-agent browser automation**: ''Browser-Use'' / ''browser-use'' / ''Browser Use'' (three spellings, three papers), ''AgentLab'', ''BrowserGym''. A sixth string, ''MCP Server Crawler'', was listed with them on 2026-08-12 and **that was wrong** — see below.
-**This was deliberately not folded into a family and not written up on the page.** The refresh brief for this run is a mechanical re-derivation, and "does the field now drive browsers with LLM agents" is a new claim needing its own population, its own dating and its own reading of the papers — five or six papers in the thinnest, provisional years of the corpus is not enough to call it current practice. It is logged as a separate work item instead. Anyone picking it up should start from this residue list and from ''programming:crawler''’s //Being Detected// section, since an agent-driven browser is a different detection surface.
+The 2026-08-12 run deliberately did not fold it, on the grounds that "does the field now drive browsers with LLM agents" is a new claim needing its own population, its own dating and its own reading of the papers. That work was done on **2026-08-29**. The outcome:
+  * A family **''LLM browser agents''** now exists in ''tool_fold.mjs'', ordered //before// Playwright and Puppeteer, because Browser Use and Skyvern drive Playwright underneath and would otherwise be counted as Playwright.
+  * It claims **4 of the 1,120 crawling papers (0.4%)**, all of them 2026. That is the row now on [[programming:crawler]], and the five figures it moved are in §4 above.
+  * ''MCP Server Crawler'' was **excluded**, and the 2026-08-12 note that grouped it with the agents is corrected here. Reading IEEE-SP/2026 //Parasites in the Toolchain// shows it is an ordinary crawler over MCP directory listings and GitHub links — "we implemented the MCP Server Crawler to leverage these links as the entry point for data collection". The name describes what it crawls, not what drives it. It stays in the printed residue. ''WebAgentDriver'' (NDSS/2026 //ChameleoScan//), the Appium XCUITest driver on jailbroken iOS, is excluded for the same kind of reason.
+  * The dating question was answered rather than deferred: **9 papers in 5,859 name an agent framework at all, 1 in 2025 and 8 in 2026**, and six of the nine //measure// agents rather than crawl with them. It is not current practice and the new page says so.
+  * The page is [[programming:crawler:llm_agents]] and its own provenance is [[provenance:programming:crawler:llm_agents]], which carries the full query log, the fold rule, the residue and the quote adjudication.
 A second, smaller cluster in the same residue: consent-automation tools used as instruments — ''autoconsent'', ''DuckDuckGo autoconsent'', ''Super Agent-Automatic Cookie Consent'', ''ConsentChk''. Those belong to [[privacy:consent]] rather than here.