programming
Table of Contents
Programming
This namespace is for the instrument — which browser renders the page, which protocol drives it, what the crawler does after page.goto, whether the profile survives, what file you write, and how you pin the image. It is not a library tutorial, and it is not the design choice of whether to crawl at all (that is Design, and Automated measurements). The publication corpus behind the corpus-backed pages here is seven venues (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P, 2010–2026, 5,859 extracted papers). Only 1,120 of them ran a crawl. Each child names its own population.
A namespace page outlines the pages inside it rather than carrying its own content. 1) All 22 pages in this namespace are listed below and written — this table, not start, is the authority on what is here. 2 of them (CrUX, Similarweb) are short instrument notes that have not been re-derived against the 2010–2026 extraction — treat them as API notes, not as evidence-backed siblings of Crawler.
Driving the browser
| Page | What a student needs it for |
|---|---|
| Crawler | Browser vs control channel. Selenium / Puppeteer / Playwright / CDP, then the specialised crawlers. |
| OpenWPM | Unbranded Firefox + privileged WebExtension. Which version, and which instruments were on. |
| Tracker Radar Collector | DuckDuckGo's Chromium/CDP collector, vs the Tracker Radar dataset / detector / entity map. |
| PageGraph | Brave page-execution recorder: which actor caused this request. |
| webXray | What remains of the tool: architecture, availability, its three licences, and its frozen ownership file. The ownership question — which source to use now — is Ownership resolution. |
| Foxhound | Firefox fork that taints strings. When a taint report is a flow and when it is a vulnerability. |
| PanoptiChrome | V8 object taint in Chromium. When it is the instrument and when it is not. |
| LLM agents | A crawler whose next action is chosen by a model at run time rather than written down in advance: Browser Use, BrowserGym/AgentLab, Skyvern, Stagehand and the Computer-Use APIs. What becomes non-deterministic, and what that costs a measurement. |
After the page loads
| Page | What a student needs it for |
|---|---|
| Interaction | What the crawler does after page.goto: depth, scroll, click. |
| Registration | Login, account creation, newsletter subscribe. |
| Stateful stateless | Whether the browser profile survives between visits, and what a reset actually resets. |
When the site notices
| Page | What a student needs it for |
|---|---|
| Crawler detection | What makes an automated client identifiable, what a site does about it, and why a blocked crawl does not fail loudly — it returns a page, a status code and a timestamp that every downstream number then inherits. |
What you record, how you pin it, and how you run it
| Page | What a student needs it for |
|---|---|
| Traffic files | HAR vs pcap vs proxy vs CDP — choosing where to tap. |
| Docker | Pinning a crawler image (digest, shared memory). |
| Deployment | The month after you press start: what to persist, what to retry, what to watch, what it costs, and what a paper has to say about a run that lasted weeks. |
Probing the network
| Page | What a student needs it for |
|---|---|
| Internet scanning | The other branch's instrument. ZMap/ZGrab invocation, probe rate and what it costs in wall clock, the exclusion file and where opt-out requests go, source-address hygiene, and why IPv6 needs a hitlist rather than an enumeration. |
Labelling instruments
| Page | What a student needs it for |
|---|---|
| Filter Lists | EasyList / EasyPrivacy / Disconnect and the regional lists: which list, which commit, which engine, what they miss, and why the list is the field's ground truth as well as its instrument. |
Sampling-frame instruments
These are the APIs Website selection points at. They are not a second copy of “which list”.
| Page | What a student needs it for | On the wiki (2026-09-21) |
|---|---|---|
| Tranco | Tranco API and construction; permanent list ids. | written, corpus-backed |
| Cloudflare Radar | Radar Domain Rankings API; ordered head vs unordered buckets. | written, corpus-backed |
| CrUX | Chrome UX Report access (BigQuery). | written, stub (8,394 bytes) |
| Similarweb | Similarweb extension/API note. | written, stub (4,514 bytes) |
Other
| Page | What a student needs it for | On the wiki (2026-09-21) |
|---|---|---|
| Multilingual support | Language detection, locale switching, translation in a crawl. | written, corpus-backed |
Where this namespace stops
- Website selection / Sampling / Crawling location / Longitudinal — which list, how you draw, where from, what you pin across waves.
- Archives — reading someone else's recording; Traffic files is writing your own.
- Consent — the banner is a treatment. Interaction is how you click; consent is which click and what it means.
- Requests — classifying the requests the instrument captured.
- Artifacts — archiving the pin. Docker is how you pin; artifacts is where a 2026 Available badge will accept it.
Methodology and limitations of these figures
The 5,859 / 1,120 are paper counts from the 5,859-paper extraction (seven venues, 2010–2026). 2025–2026 venue-years are provisional — see corpus.
The 22 is both counts at once, and that is the point: it is the number of rows across the tables above and the number of pages whose id begins programming: in the live wiki index, and scripts/report_namespace_overviews.mjs now refuses to pass unless those two are the same set, not merely the same size. Re-derived 2026-09-21. It read 19 in the box and 18 in this sentence against a 22-page namespace: Crawler detection, LLM agents and Deployment were live pages here with no row in any table, and the two published counts did not even agree with each other. The 2 is the number of those 22 that are short instrument notes rather than corpus-backed pages, with their live byte counts in the sampling-frame table; it is a judgement about depth, not a measurement. Queries: programming. Joint sitting: design.
1)
contributing, “Namespace and page structure”.
You could leave a comment if you were logged in.
programming.txt · Last modified: by karel.kubicek.claude