Differences

This shows you the differences between two versions of the page.

--- programming [2026/09/11 18:00] – Carve-out drift: the domain-to-company ownership material moved from Programming:Crawler:webXray to the new Design:Ownership resolution on 2026-09-11; repoint the link. Authored by Claude Also rewrites the webXray row's description, which described the pa karel.kubicek.claude
+++ programming [2026/09/21 23:12] (current) – Reconcile the namespace table with the namespace: add Crawler detection, Crawler:LLM agents and Deployment (19 rows vs 22 pages); count now states both what it counts and that the script asserts set equality. Authored by Claude karel.kubicek.claude
@@ Line 4: / Line 4: @@
 <WRAP important>
-**A namespace page outlines the pages inside it rather than carrying its own content.** (([[contributing]], "Namespace and page structure".)) All **19** children below are written. **2** of them ([[Programming:CrUX]], [[Programming:Similarweb]]) are short instrument notes that have **not** been re-derived against the 2010–2026 extraction — treat them as API notes, not as evidence-backed siblings of [[Programming:Crawler]].
+**A namespace page outlines the pages inside it rather than carrying its own content.** (([[contributing]], "Namespace and page structure".)) All **22** pages in this namespace are listed below and written — this table, not [[start]], is the authority on what is here. **2** of them ([[Programming:CrUX]], [[Programming:Similarweb]]) are short instrument notes that have **not** been re-derived against the 2010–2026 extraction — treat them as API notes, not as evidence-backed siblings of [[Programming:Crawler]].
 </WRAP>
@@ Line 17: / Line 17: @@
 | [[Programming:Crawler:Foxhound]] | Firefox fork that taints strings. When a taint report is a flow and when it is a vulnerability. |
 | [[Programming:Crawler:PanoptiChrome]] | V8 object taint in Chromium. When it is the instrument and when it is not. |
+| [[Programming:Crawler:LLM agents]] | A crawler whose next action is chosen by a model at run time rather than written down in advance: Browser Use, BrowserGym/AgentLab, Skyvern, Stagehand and the Computer-Use APIs. What becomes non-deterministic, and what that costs a measurement. |
 ===== After the page loads =====
@@ Line 25: / Line 26: @@
 | [[Programming:Stateful stateless]] | Whether the browser profile survives between visits, and what a reset actually resets. |
-===== What you record, and how you pin it =====
+===== When the site notices =====
+^ Page ^ What a student needs it for ^
+| [[Programming:Crawler detection]] | What makes an automated client identifiable, what a site does about it, and why a blocked crawl does not fail loudly — it returns a page, a status code and a timestamp that every downstream number then inherits. |
+===== What you record, how you pin it, and how you run it =====
 ^ Page ^ What a student needs it for ^
 | [[Programming:Traffic files]] | HAR vs pcap vs proxy vs CDP — choosing where to tap. |
 | [[Programming:Docker]] | Pinning a crawler image (digest, shared memory). |
+| [[Programming:Deployment]] | The month after you press start: what to persist, what to retry, what to watch, what it costs, and what a paper has to say about a run that lasted weeks. |
 ===== Probing the network =====
@@ Line 45: / Line 52: @@
 These are the APIs [[Design:Website selection]] points at. They are not a second copy of "which list".
-^ Page ^ What a student needs it for ^ On the wiki (2026-08-27) ^
+^ Page ^ What a student needs it for ^ On the wiki (2026-09-21) ^
 | [[Programming:Tranco]] | Tranco API and construction; permanent list ids. | written, corpus-backed |
 | [[Programming:Cloudflare Radar]] | Radar Domain Rankings API; ordered head vs unordered buckets. | written, corpus-backed |
@@ Line 53: / Line 60: @@
 ===== Other =====
-^ Page ^ What a student needs it for ^ On the wiki (2026-08-27) ^
+^ Page ^ What a student needs it for ^ On the wiki (2026-09-21) ^
 | [[Programming:Multilingual support]] | Language detection, locale switching, translation in a crawl. | written, corpus-backed |
@@ Line 66: / Line 73: @@
 ===== Methodology and limitations of these figures =====
-The **5,859** / **1,120** are paper counts from the 5,859-paper extraction (seven venues, 2010–2026). 2025–2026 venue-years are provisional — see [[literature:corpus]]. The **18** / **2** and the stub byte counts (CrUX, Similarweb) are wiki-page counts, the stub sizes as of 2026-08-27 and the child count as of 2026-09-10 (when [[Programming:Internet scanning]] was added; it was 18 after [[Programming:Filter Lists]] on 2026-08-29). Queries: [[provenance:programming]]. Joint sitting: [[provenance:design]].
+The **5,859** / **1,120** are paper counts from the 5,859-paper extraction (seven venues, 2010–2026). 2025–2026 venue-years are provisional — see [[literature:corpus]].
+The **22** is **both** counts at once, and that is the point: it is the number of rows across the tables above //and// the number of pages whose id begins ''programming:'' in the live wiki index, and ''scripts/report_namespace_overviews.mjs'' now refuses to pass unless those two are the same **set**, not merely the same size. Re-derived 2026-09-21. It read **19** in the box and **18** in this sentence against a 22-page namespace: [[Programming:Crawler detection]], [[Programming:Crawler:LLM agents]] and [[Programming:Deployment]] were live pages here with no row in any table, and the two published counts did not even agree with each other. The **2** is the number of those 22 that are short instrument notes rather than corpus-backed pages, with their live byte counts in the sampling-frame table; it is a judgement about depth, not a measurement. Queries: [[provenance:programming]]. Joint sitting: [[provenance:design]].
 <bibtex bibliography></bibtex>