Differences

This shows you the differences between two versions of the page.

--- programming [2026/08/27 15:26] – Multilingual support is no longer a stub: 3 stubs -> 2 (CrUX, Similarweb). Authored by Claude. karel.kubicek.claude
+++ programming [2026/09/21 23:12] (current) – Reconcile the namespace table with the namespace: add Crawler detection, Crawler:LLM agents and Deployment (19 rows vs 22 pages); count now states both what it counts and that the script asserts set equality. Authored by Claude karel.kubicek.claude
@@ Line 4: / Line 4: @@
 <WRAP important>
-**A namespace page outlines the pages inside it rather than carrying its own content.** (([[contributing]], "Namespace and page structure".)) All **17** children below are written. **2** of them ([[Programming:CrUX]], [[Programming:Similarweb]]) are short instrument notes that have **not** been re-derived against the 2010–2026 extraction — treat them as API notes, not as evidence-backed siblings of [[Programming:Crawler]].
+**A namespace page outlines the pages inside it rather than carrying its own content.** (([[contributing]], "Namespace and page structure".)) All **22** pages in this namespace are listed below and written — this table, not [[start]], is the authority on what is here. **2** of them ([[Programming:CrUX]], [[Programming:Similarweb]]) are short instrument notes that have **not** been re-derived against the 2010–2026 extraction — treat them as API notes, not as evidence-backed siblings of [[Programming:Crawler]].
 </WRAP>
@@ Line 14: / Line 14: @@
 | [[Programming:Crawler:Tracker Radar Collector]] | DuckDuckGo's Chromium/CDP collector, vs the Tracker Radar dataset / detector / entity map. |
 | [[Programming:Crawler:PageGraph]] | Brave page-execution recorder: which actor caused this request. |
-| [[Programming:Crawler:webXray]] | Domain-to-company ownership lists, and what remains of the tool. |
+| [[Programming:Crawler:webXray]] | What remains of the tool: architecture, availability, its three licences, and its frozen ownership file. The ownership //question// — which source to use now — is [[Design:Ownership resolution]]. |
 | [[Programming:Crawler:Foxhound]] | Firefox fork that taints strings. When a taint report is a flow and when it is a vulnerability. |
 | [[Programming:Crawler:PanoptiChrome]] | V8 object taint in Chromium. When it is the instrument and when it is not. |
+| [[Programming:Crawler:LLM agents]] | A crawler whose next action is chosen by a model at run time rather than written down in advance: Browser Use, BrowserGym/AgentLab, Skyvern, Stagehand and the Computer-Use APIs. What becomes non-deterministic, and what that costs a measurement. |
 ===== After the page loads =====
@@ Line 25: / Line 26: @@
 | [[Programming:Stateful stateless]] | Whether the browser profile survives between visits, and what a reset actually resets. |
-===== What you record, and how you pin it =====
+===== When the site notices =====
+^ Page ^ What a student needs it for ^
+| [[Programming:Crawler detection]] | What makes an automated client identifiable, what a site does about it, and why a blocked crawl does not fail loudly — it returns a page, a status code and a timestamp that every downstream number then inherits. |
+===== What you record, how you pin it, and how you run it =====
 ^ Page ^ What a student needs it for ^
 | [[Programming:Traffic files]] | HAR vs pcap vs proxy vs CDP — choosing where to tap. |
 | [[Programming:Docker]] | Pinning a crawler image (digest, shared memory). |
+| [[Programming:Deployment]] | The month after you press start: what to persist, what to retry, what to watch, what it costs, and what a paper has to say about a run that lasted weeks. |
+===== Probing the network =====
+^ Page ^ What a student needs it for ^
+| [[Programming:Internet scanning]] | The other branch's instrument. ZMap/ZGrab invocation, probe rate and what it costs in wall clock, the exclusion file and where opt-out requests go, source-address hygiene, and why IPv6 needs a hitlist rather than an enumeration. |
+===== Labelling instruments =====
+^ Page ^ What a student needs it for ^
+| [[Programming:Filter Lists]] | EasyList / EasyPrivacy / Disconnect and the regional lists: which list, which commit, which engine, what they miss, and why the list is the field's ground truth as well as its instrument. |
 ===== Sampling-frame instruments =====
@@ Line 35: / Line 52: @@
 These are the APIs [[Design:Website selection]] points at. They are not a second copy of "which list".
-^ Page ^ What a student needs it for ^ On the wiki (2026-08-27) ^
+^ Page ^ What a student needs it for ^ On the wiki (2026-09-21) ^
 | [[Programming:Tranco]] | Tranco API and construction; permanent list ids. | written, corpus-backed |
 | [[Programming:Cloudflare Radar]] | Radar Domain Rankings API; ordered head vs unordered buckets. | written, corpus-backed |
@@ Line 43: / Line 60: @@
 ===== Other =====
-^ Page ^ What a student needs it for ^ On the wiki (2026-08-27) ^
+^ Page ^ What a student needs it for ^ On the wiki (2026-09-21) ^
 | [[Programming:Multilingual support]] | Language detection, locale switching, translation in a crawl. | written, corpus-backed |
@@ Line 56: / Line 73: @@
 ===== Methodology and limitations of these figures =====
-The **5,859** / **1,120** are paper counts from the 5,859-paper extraction (seven venues, 2010–2026). 2025–2026 venue-years are provisional — see [[literature:corpus]]. The **17** / **2** and the stub byte counts (CrUX, Similarweb) are wiki-page counts as of 2026-08-27. Queries: [[provenance:programming]]. Joint sitting: [[provenance:design]].
+The **5,859** / **1,120** are paper counts from the 5,859-paper extraction (seven venues, 2010–2026). 2025–2026 venue-years are provisional — see [[literature:corpus]].
+The **22** is **both** counts at once, and that is the point: it is the number of rows across the tables above //and// the number of pages whose id begins ''programming:'' in the live wiki index, and ''scripts/report_namespace_overviews.mjs'' now refuses to pass unless those two are the same **set**, not merely the same size. Re-derived 2026-09-21. It read **19** in the box and **18** in this sentence against a 22-page namespace: [[Programming:Crawler detection]], [[Programming:Crawler:LLM agents]] and [[Programming:Deployment]] were live pages here with no row in any table, and the two published counts did not even agree with each other. The **2** is the number of those 22 that are short instrument notes rather than corpus-backed pages, with their live byte counts in the sampling-frame table; it is a judgement about depth, not a measurement. Queries: [[provenance:programming]]. Joint sitting: [[provenance:design]].
 <bibtex bibliography></bibtex>