Old Revisions

These are the older revisons of the current document. To revert to an old revision, select it from below, click Edit this page and save it.

2026/08/29 06:37 programming:crawler – LLM-agent fold: new family row (4 papers), bespoke own-name 184->181, union 321->318 (28.7%->28.4%), 75->74, residue 204/210->199/205; new Agent-Driven Crawling section pointing at programming:crawler:llm_agents. Authored by Claude karel.kubicek.claude +2.2 KB (current)
2026/08/29 01:42 programming:crawler – Fix rendering of the new pointer section: DokuWiki treats an indented continuation line as preformatted, so the two wrapped bullets rendered literally. Authored by Claude karel.kubicek.claude -28 B
2026/08/29 01:41 programming:crawler – Shrink 'Being Detected' to a pointer at the new programming:crawler_detection, which now covers it in full; keep the two points specific to choosing a library. Also correct the anti-detection patches row from 12/2 to 11/1: tool_fold.mjs matched 'Stealth A karel.kubicek.claude -739 B
2026/08/21 08:34 programming:crawler – [Open Questions] karelkubicek ±0 B
2026/08/17 21:45 programming:crawler – Update the PanoptiChrome row and taint-tracking note now that Programming:Crawler:PanoptiChrome exists: real output format, the published patch does not compile, and start with Foxhound between the two. Authored by Claude karel.kubicek.claude +544 B
2026/08/17 07:54 programming:crawler – Correct the webXray paragraph: the newest surviving copy is 2021 and drives consumer Chrome over CDP, not a 2015 PhantomJS mirror; webXray is now the commercial webxray.ai, which closes the open TODO; add its PolyForm Strict no-redistribution licence and karel.kubicek.claude +288 B
2026/08/12 15:45 programming:crawler – Methodology section: replaced the generic corpus caveats (absent venues, provisional 2025-2026) with a pointer to the new literature:corpus page, keeping the page-specific consequence. Authored by Claude. karel.kubicek.claude +1 B
2026/08/12 10:05 programming:crawler – Review pass (Fable): fix stale figures that sat OUTSIDE the corpus section and so were missed by the windowed staleness guard. 11.6->12.0%, 21.8->21.6%, 6.3->6.8%, 4.1->3.8%, 12.9->12.5%, 34->75 unnamed bespoke crawlers; Playwright rewritten (10 -> 34 pap karel.kubicek.claude +324 B
2026/08/12 09:16 programming:crawler – Refresh all corpus figures for the extended 2010-2026 extraction (4,322 -> 5,859 papers; crawled 859 -> 1,120). Adds a provisional 2025-2026 column; Playwright overtakes Puppeteer there. Browser fold extended (15 unmapped strings -> 1). Report script now karel.kubicek.claude +1.6 KB
2026/08/06 07:44 programming:crawler – Correct the BiDi finding: responseCompleted DOES carry the status; the real defect is that registering two network handlers cross-delivers both event types (8/8 events, only 4 with response data), which makes URL-keyed deduplication lose the status. Also: karel.kubicek.claude +2 KB
2026/08/06 07:37 programming:crawler – Two wording fixes: Puppeteer is no longer Chromium-only, and Selenium is the top off-the-shelf library rather than the top tool. Authored by Claude. karel.kubicek.claude +20 B
2026/08/06 07:36 programming:crawler – Clarify that the browser-extension family is third-party extensions only; an author-written extension counts as bespoke. Counts unchanged. Authored by Claude. karel.kubicek.claude +255 B
2026/08/06 07:35 programming:crawler – Soften the stated mechanism for the Selenium CDP-bridge race; note it reproduced independently. Authored by Claude. karel.kubicek.claude +214 B
2026/08/06 07:34 programming:crawler – Review fixes: real residue accounting (the fold now separates generically-described from self-named bespoke crawlers; 25.7% of crawling papers rolled their own), script-derived artifact-release figures, unified population for the specialised-crawler table karel.kubicek.claude +2.5 KB
2026/08/06 07:28 programming:crawler – Fix a broken in-page anchor. Authored by Claude. karel.kubicek.claude +43 B
2026/08/06 07:27 programming:crawler – Correct the WebDriver network claim: classic WebDriver has no network commands, but WebDriver BiDi does and Selenium exposes it. Add a measured BiDi section (duplicate events, no status code in the response event) and a fifth row in the measured compariso karel.kubicek.claude +3.7 KB
2026/08/06 07:23 programming:crawler – Playwright: separate the shipped-Chromium/patched-Firefox distinction, add the headless-shell and channel caveats with a docs citation; OpenWPM storage backends and the no-Windows footnote. Authored by Claude. karel.kubicek.claude +887 B
2026/08/06 07:20 programming:crawler – Correct attribution of the sandbox measurement footnote. Authored by Claude. karel.kubicek.claude +37 B
2026/08/06 07:18 programming:crawler – New page: comparison of crawling libraries (Selenium/Puppeteer/Playwright/CDP) and specialised measurement crawlers, with corpus figures and sandbox-verified code. Authored by Claude. karel.kubicek.claude +39.3 KB