Differences

This shows you the differences between two versions of the page.

--- programming:traffic_files [2026/08/14 03:19] – Fix [[Artifacts]] -> [[:Artifacts]] (DokuWiki resolves unqualified links inside the current namespace). Authored by Claude. karel.kubicek.claude
+++ programming:traffic_files [2026/08/21 14:50] (current) – Boxes: <wrap> renders a span, use uppercase <WRAP>; one box per list, not per bullet karel.kubicek.claude
@@ Line 6: / Line 6: @@
 <WRAP important>
-**The one thing to take away.** The three places you can tap — the packet stream, an intercepting proxy, the browser's own internals — are **not nested**, and neither is a superset of the others. Measured on a single page load of a local fixture ([[#What We Ran]]):
+**The one thing to take away.** The three places you can tap — the packet stream, an intercepting proxy, the browser's own internals — are **not nested**, and neither is a superset of the others. Measured on one page load of a local fixture recorded three ways at once, plus one re-run against a remote origin ([[#What We Ran]]):
   * the browser's HAR contains a request that a proxy in front of it **never saw**, because Content-Security-Policy stopped the browser from sending it;
-  * the proxy's HAR contains the origin server's IP address, which the browser's HAR **cannot** contain — behind a proxy, every ''serverIPAddress'' in the browser's file is the proxy's;
+  * the proxy's HAR contains the origin server's IP address, which the browser's HAR **cannot** contain — behind a proxy, every ''serverIPAddress'' in the browser's file is the proxy's (this one needs the remote re-run: on a loopback fixture both sides read ''127.0.0.1'' and the difference is invisible);
   * **neither** contains the JavaScript call stack that made the request, which the Chrome DevTools Protocol hands you for free — 0 of 12 HAR entries carried an initiator, against 11 of 11 CDP events, 5 of them with a JS stack frame;
   * the proxy's HAR has **no page boundaries at all** — 0 ''pageref'' fields and an empty ''log.pages'' — so you cannot tell from it which page load a request belongs to.
@@ Line 30: / Line 30: @@
 ^ TLS/QUIC handshake detail, timing at packet granularity | ✓ | partial | ✗ |
 ^ Changes what the site sees | no | **yes** — new TLS stack, new certificate | mostly no |
-^ Volume per 1,000 sites | tens of GB | GB | hundreds of MB without bodies |
+^ Volume per 1,000 sites((Order of magnitude, not a measurement: our fixture recorded single page loads, and no paper in the corpus reports capture volume in a comparable form. Treat it as a planning estimate and measure your own on a pilot.)) | tens of GB | GB | hundreds of MB without bodies |
 Three consequences that decide most projects:
@@ Line 42: / Line 42: @@
 HAR — HTTP Archive format — is a JSON log of HTTP transactions. It is the //lingua franca// of the browser layer: every major browser exports it from DevTools, and every automation library can write one.
-**Its specification is abandoned, and this matters more than it sounds.** HAR 1.2 was written in 2011–2012 by Jan Odvarko and taken up by the W3C Web Performance Working Group, which never published it. The W3C copy now opens with a banner reading //"\*DO NOT USE\* This document was never published by the W3C Web Performance Working Group and has been abandoned."//((Fetched from ''https://w3c.github.io/web-performance/specs/HAR/Overview.html'' on 14 August 2026 by ''scripts/external_checks_traffic_files.sh''.)) Six papers in our corpus cite one of the two W3C URLs as //the// HAR specification — and the older ''dvcs.w3.org'' copy several of them point at does **not** carry the banner, so following the citation does not tell you the document was withdrawn. There has been no successor: a community repository carries a "proposed" 1.3 that has not moved since 2019. In practice the specification is whatever the writers agree on, which is why the differences below exist at all. Cite Odvarko's original at ''softwareishard.com/blog/har-12-spec/'' if you must cite something, and //name your writer//.
+**Its specification is abandoned, and this matters more than it sounds.** HAR 1.2 was written in 2011–2012 by Jan Odvarko and taken up by the W3C Web Performance Working Group, which never published it. The W3C copy now opens with a banner reading //"\*DO NOT USE\* This document was never published by the W3C Web Performance Working Group and has been abandoned."//((Fetched from ''https://w3c.github.io/web-performance/specs/HAR/Overview.html'' on 14 August 2026 by ''scripts/external_checks_traffic_files.sh''.)) Six papers in our corpus cite one of the two W3C URLs as //the// HAR specification — and the older ''dvcs.w3.org'' copy that two of them point at does **not** carry the banner, so following the citation does not tell you the document was withdrawn. There has been no successor: a community repository carries a "proposed" 1.3 that has been dormant for years. In practice the specification is whatever the writers agree on, which is why the differences below exist at all. Cite Odvarko's original at ''softwareishard.com/blog/har-12-spec/'' if you must cite something, and //name your writer//.
 <WRAP tip>
-**Two different things are called "HTTP Archive"**, and conflating them is the most common error in this literature. **HAR** is a file format. **The HTTP Archive** (''httparchive.org'') is a monthly crawl of millions of sites, queryable in BigQuery, and it is a //dataset//, not a file you made. In our corpus, 14 papers analyse the dataset and 32 use the format, and only one paper does both ({[trevisan2019_cookielaw]}). See [[#Use in Publications]].
+**Two different things are called "HTTP Archive"**, and conflating them is the most common error in this literature. **HAR** is a file format. **The HTTP Archive** (''httparchive.org'') is a monthly crawl of millions of sites, queryable in BigQuery, and it is a //dataset//, not a file you made. In our corpus, 32 papers use the format and 14 further papers analyse only the dataset. One paper does both — {[trevisan2019_cookielaw]} dumps its own HAR files //and// reanalyses the httparchive.org corpus — and our classification is single-label, so it is counted under //format// and not in the 14. See [[#Use in Publications]].
 </WRAP>
@@ Line 87: / Line 87: @@
   * **No initiator, no call stack.** This is the big one. HAR 1.2 has no field for //who asked for this resource//. CDP's ''Network.requestWillBeSent'' carries ''initiator'', including a JavaScript stack. If your question is "which script caused this third-party request" — the question behind most of [[Privacy:Requests]] and all of [[Privacy:JavaScript]] — a HAR cannot answer it and you need the browser's event stream, or a crawler that records it for you ([[Programming:Crawler:OpenWPM]], [[Programming:Crawler:PageGraph]]).
-  * **Bodies are opt-in, and expensive.** With ''content: 'omit''' the file was 20,566 B; with ''content: 'embed''' it was 226,719 B — **11× larger**, for one 200 KiB response. Bodies are base64-encoded when they are not text, so budget roughly the raw byte count plus a third on top of it. At crawl scale this is the difference between a dataset you can release and one you cannot.
+  * **Bodies are opt-in, and expensive.** With ''content: 'omit''' the file was 20,566 B; with ''content: 'embed''' it was 226,719 B — **11× larger**, for one 200 KiB response. A text body embeds roughly 1:1 — the run above measured 100.2% of the raw body bytes — while a binary body is base64-encoded and costs about a third more. Budget from whichever your crawl is mostly made of. At crawl scale this is the difference between a dataset you can release and one you cannot.
   * **Failure has no vocabulary.** Both requests that never completed appear with ''status: -1''. //Why// they failed is not in the standard: Playwright adds a non-standard ''response._failureText'' (it read ''"csp"'' for the blocked image, and nothing at all for the request left in flight). CDP told us ''blockedReason: "csp"'' for one and, for the other, nothing — it had emitted no terminal event by the time the context closed.
   * **The two recordings disagree about which requests failed.** On the same load, the HAR marked the never-answered ''/slow'' request failed and recorded the 204 beacon as complete; CDP emitted ''loadingFailed net::ERR_ABORTED'' for the //beacon// and no terminal event at all for ''/slow''. Neither is a superset. **In-flight requests at the moment you stop recording are recorded inconsistently, so drain the page before you close the context and report how long you waited.**
   * **A HAR is a transaction log, not a cookie jar.** The main document's ''response.cookies'' correctly contained the ''HttpOnly'' cookie — that part of the folklore is wrong, the flag hides a cookie from ''document.cookie'', not from the recorder. But at the end of the load the browser jar held **4** cookies and one of them, set by ''document.cookie'' in a script, appears in **no** HAR entry, because it never crossed the network. Counting cookies from a HAR counts //cookies observed in HTTP headers//, which is a different quantity from //cookies in the jar//. Say which one you mean. [[Privacy:Cookies]] is the page for the rest of that problem.
+  * **The WebSocket rows are about the writer, not the format.** HAR 1.2 has no representation for a WebSocket at all. Playwright records the connection as an entry and the frames under a non-standard ''_webSocketMessages'', and it only started doing so in version 1.61 (June 2026) — so a HAR written by an older Playwright, or by a different tool, may show nothing. Check rather than assume.
   * **Everything interesting is a vendor extension.** The spec reserves ''_''-prefixed names for custom fields, and the useful ones all live there. Playwright 1.62.1 emitted exactly ''_resourceType'', ''_securityDetails'', ''_serverPort'', ''_webSocketMessages'', ''response._failureText'' and ''response._transferSize''. Chrome DevTools emits a //different// set. So an analysis script written against one writer's HAR quietly produces empty columns against another's.
@@ Line 98: / Line 99: @@
 | Chrome / Edge DevTools | Network panel → //Save all as HAR (sanitized)// or //(with sensitive data)// | current; the sanitized variant is the default since Chrome 130 — see [[#Before You Publish The File]] |
 | Firefox DevTools | Network panel → //Save All As HAR//; ''devtools.netmonitor.har.enableAutoExportToFile'' auto-saves every load | current; the pref is how {[wang2021_just]} automated it. Read the neighbouring prefs too |
-| Playwright | ''browser.newContext({recordHar: {path, content}})'', where ''content'' is ''omit'', ''embed'' or ''attach'' | current; one HAR per context, and ''content'' is the 11× size difference |
+| Playwright | ''browser.newContext({recordHar: {path, content, mode, urlFilter}})'', or ''context.tracing.startHar(path)'' / ''stopHar()'' since 1.60 | current; ''content'' (''omit''/''embed''/''attach'') is the 11× size difference, ''mode'' is ''full'' or ''minimal''. **Moving**: the tracing API is new in 1.60 (May 2026) and WebSocket requests only started appearing in Playwright's HAR in 1.61 |
 | Puppeteer | no built-in writer — use a library over CDP | ''chrome-har-capturer'' (npm 0.14.4, 2026-01-27) is maintained and is what the corpus uses {[muthuraj2024_replication,zhang2024_quic]} |
-| Selenium | none; classic WebDriver has no network commands at all | this is why [[Programming:Crawler|Selenium crawls]] end up with a proxy bolted on. ''selenium-wire'', the usual bridge, was **archived in January 2024** |
+| Selenium | none built in; //classic// WebDriver has no network commands at all | this is why [[Programming:Crawler|Selenium crawls]] historically end up with a proxy bolted on, and ''selenium-wire'', the usual bridge, was **archived in January 2024**. WebDriver BiDi has since given Selenium native network interception — with [[Programming:Crawler|its own caveats]] — but still no HAR writer |
 | mitmproxy | ''mitmdump --set hardump=out.har'' | current (12.2.3, 2026-05-12); proxy-layer HAR — bodies by default, no page boundaries |
-| WebPageTest | records a HAR alongside its own waterfall | run by Catchpoint since 2020 {[chaqfeh2020_jscleaner]} |
+| WebPageTest | records a HAR alongside its own waterfall {[chaqfeh2020_jscleaner]} | current; run by Catchpoint since 2020 |
-| HARExportTrigger | a Firefox extension that triggers export from inside the page | **deprecated and archived**; last release 0.6.1, May 2018. Papers that used it {[borgolte2020_performance]} cannot be reproduced with it today — use the pref instead |
+| HARExportTrigger | a Firefox extension that triggers export from inside the page | **deprecated and archived**, description "DEPRECATED"; last release 0.6.1, May 2018. It is what {[borgolte2020_performance]} automated Firefox with; for new work use the pref above instead |
-| BrowserMob Proxy | a Java proxy that writes HAR | effectively dead: last tagged release 2016, last commit 2024 |
+| BrowserMob Proxy | a Java proxy that writes HAR | effectively dead: the last GitHub //Release// is 2.1.4 (December 2016); a later ''2.1.5'' tag carries no release notes, and the repository was last pushed in May 2024 |
 **Firefox's defaults are not Chrome's, and the prefs say so.** Read straight out of ''browser/app/profile/firefox.js'' in current Firefox: ''includeResponseBodies'' defaults to **''true''**, so a Firefox HAR carries bodies unless you turn them off; ''multiple-pages'' defaults to **''false''**, so an auto-exported HAR covers **one** page; and ''pageLoadedTimeout'' is **1500** ms, which is how long the exporter waits after load before writing — anything still in flight after that is what §//What a HAR Loses// is about. Chrome's sanitized-by-default export strips headers, not bodies. Two files both labelled HAR 1.2, three different defaults.
+==== When the Browser Layer Is Not Enough: NetLog ====
+A HAR stops at HTTP. Chrome's **NetLog** goes below it: DNS resolution and its stalls, socket-pool waits, TCP connect attempts, proxy resolution, disk-cache reads and writes, QUIC session events, and the errors that a HAR can only render as ''status: -1''. You capture it from ''chrome://net-export'' or with ''--log-net-log=FILE'', and read it in the standalone ''netlog-viewer''. It appears in 3 papers in our corpus.
+Reach for it when your question is "//why// was this slow or broken" rather than "what was requested" — a QUIC negotiation that silently fell back, a DNS lookup that stalled, a connection the pool refused to reuse. It is Chromium-only, it is verbose, and its schema is an implementation detail rather than a standard, so it is a debugging instrument rather than a dataset format. But it is the one browser-native answer to a below-HTTP question, and the alternative is leaving the browser layer entirely for a packet capture.
 ===== Intercepting Proxies =====
@@ Line 112: / Line 119: @@
 An intercepting proxy terminates TLS in the middle: it presents your own CA's certificate to the client, opens its own connection to the origin, and sees plaintext in both directions. **mitmproxy** is the field's default — 98 papers in our corpus, against 18 for Burp Suite and 15 for Fiddler ([[#Use in Publications]]) — and ''mitmdump'' is its non-interactive form, which is what a crawl wants. It is actively maintained (12.2.3, May 2026); of the alternatives named in the corpus, Fiddler Classic is no longer in active development and Charles is commercial.
-Its two research-relevant outputs are its own ''.flows'' stream (''-w file'') and a HAR (''--set hardump=file''). We ran the same fixture load through ''mitmdump'' and recorded the browser's HAR at the same time (excerpt of the run's output — the rest is on [[provenance:programming:traffic_files]]):
+Its two research-relevant outputs are its own ''.flows'' stream (''-w file'') and a HAR (''--set hardump=file''). We ran the same fixture load through ''mitmdump'' and recorded the browser's HAR at the same time — with **mitmproxy 11.0.2**, two major versions behind the current release, which by this page's own argument is exactly the kind of thing to state (excerpt of the run's output — the rest is on [[provenance:programming:traffic_files]]):
 <code>
@@ Line 137: / Line 144: @@
   * **The WebSocket is in both files under different URLs** — ''ws://…/ws'' in the browser's, ''http://…/ws'' (the upgrade) in the proxy's. Joining the two files on URL silently loses it.
   * **Bodies by default.** 10 of 11 entries carried response text, and the file is 12.8× the browser's body-less HAR. This is what you want for content analysis and what you do not want for a 100k-site crawl.
-  * **Every ''serverIPAddress'' the browser records becomes the proxy's.** Re-running the same script against ''https://example.com/'' the browser HAR recorded ''127.0.0.1'' and the mitmproxy HAR recorded ''172.66.147.243''. If you need origin addresses and you are proxying, take them from the proxy side. [[Design:IP classification]] is where that matters.
+  * **Every ''serverIPAddress'' the browser records becomes the proxy's.** Re-running the same script against ''https://example.com/'' on 14 August 2026, the browser HAR recorded ''127.0.0.1'' — the proxy — and the mitmproxy HAR recorded ''104.20.23.154'', the origin. (That second address is not a constant: ''example.com'' sits on Cloudflare's anycast network and the edge address rotates. The point is which //side// each recorder sees, not the octets.) If you need origin addresses and you are proxying, take them from the proxy side. [[Design:IP classification]] is where that matters.
-**What the proxy buys you** that the browser layer does not: traffic from a mobile app or an IoT device that has no DevTools at all — which is why the mobile-privacy literature is built on it {[kollnig2022_iphones,figueira2024_diffaudit]} — and full request and response bodies without the browser's cooperation.
+**What the proxy buys you** that the browser layer does not: traffic from a mobile app or an IoT device that has no DevTools at all — which is why much of the mobile-privacy literature is built on it {[kollnig2022_iphones]} — and full request and response bodies without the browser's cooperation. It is not the only way onto a phone: {[figueira2024_diffaudit]} goes to the //packet// layer there instead, with an on-device capture app, and keeps the browser layer for the web half of the same study.
 **What it costs.** Certificate pinning breaks. TLS fingerprinting sees mitmproxy, not Chrome. QUIC and HTTP/3 interception is newer and less complete than HTTP/1.1 and HTTP/2 — mitmproxy enables it by default but documents the support as limited, and browsers do not accept a custom CA over QUIC the way they do over TLS — so a proxied crawl can silently downgrade the protocol under measurement. That is fatal when the protocol //is// the measurement, which is why the QUIC website-fingerprinting work captures at the browser and packet layers instead {[siby2023_quic]}. And you have added a component that can fail: a flow the proxy drops looks exactly like a request the site did not make.
@@ Line 175: / Line 182: @@
 **A raw traffic file is a credential store.** By design it holds ''Authorization'' headers, ''Cookie'' and ''Set-Cookie'' headers, session tokens in query strings, CSRF tokens and login POST bodies, and — if you enabled bodies — whatever the responses contained.
-This is not theoretical, and the one well-documented case is worth knowing in detail. Between 28 September and 17 October 2023 an attacker had access to Okta's customer support case-management system. Okta's own root-cause post says: //"Some of these files were HAR files that contained session tokens which could in turn be used for session hijacking attacks"//, and that //"the threat actor was able to use these session tokens to hijack the legitimate Okta sessions of 5 customers"//. The initial disclosure said 134 customers' files were accessed; a follow-up on 29 November 2023 revised the scope to essentially all Workforce and Customer Identity customers.((''https://sec.okta.com/articles/2023/11/unauthorized-access-oktas-support-case-management-system-root-cause/'' and ''https://sec.okta.com/articles/october-security-incident-recommended-actions/'', both fetched 14 August 2026.)) The files were uploaded by customers, at Okta's request, to help debug a login problem. Nobody involved thought they were sending a credential.
+This is not theoretical, and the one well-documented case is worth knowing in detail. Between 28 September and 17 October 2023 an attacker had access to Okta's customer support case-management system. Okta's own root-cause post says: //"Some of these files were HAR files that contained session tokens which could in turn be used for session hijacking attacks"//, and that //"the threat actor was able to use these session tokens to hijack the legitimate Okta sessions of 5 customers"//. The initial disclosure said 134 customers' files were accessed; a follow-up on 29 November 2023 revised the scope to essentially all Workforce and Customer Identity customers.((''https://sec.okta.com/articles/2023/11/unauthorized-access-oktas-support-case-management-system-root-cause/'' and ''https://sec.okta.com/articles/october-security-incident-recommended-actions/'', both fetched 14 August 2026.)) An independent forensic investigation was reported closed on 8 February 2024 with no further malicious activity found, so the figures above stand.((''https://sec.okta.com/articles/harfiles/'', "Okta October 2023 Security Incident Investigation Closure". Despite the URL, that page carries no HAR-specific text.)) The files were uploaded by customers, at Okta's request, to help debug a login problem. Nobody involved thought they were sending a credential.
 Treat every traffic file you did not personally sanitise as a live secret.
 </WRAP>
-**There is currently no maintained general-purpose HAR sanitiser.** Both of the tools written in response to that incident — Google's ''har-sanitizer'' and Cloudflare's — are now archived read-only on GitHub (checked 14 August 2026), as is the Chrome-extension one. What survives is vendor-specific: Okta's own sanitiser strips all cookies and the //signature// of ID/access/SAML tokens but explicitly //"There is no attempt to remove user information as part of the sanitization process"// and leaves JWT claims and SAML attributes intact, and recommends capturing with a test account instead. If you need sanitisation in a crawl pipeline, you are writing it.
+**There is currently no maintained general-purpose HAR sanitiser**, and this is a claim we tried to disprove rather than assert. Every candidate we could find is archived or dormant as of 14 August 2026: Google's ''har-sanitizer'' and Cloudflare's ''har-sanitizer'' are both archived read-only, as is the ''sanitizhar'' Chrome extension; Beyond Identity's ''har-sanitize'' has not been touched since November 2023; ''harmor'' was last committed in April 2025; and Edgio's ''har-tools'' is orphaned, its publisher having shut down in January 2025. If one of these has revived since, it is a better answer than the paragraph below — check before you write your own. What survives is vendor-specific: Okta's own sanitiser strips all cookies and the //signature// of ID/access/SAML tokens but explicitly //"There is no attempt to remove user information as part of the sanitization process"// and leaves JWT claims and SAML attributes intact, and recommends capturing with a test account instead. If you need sanitisation in a crawl pipeline, you are writing it.
 Before a traffic file goes into an artefact, a support ticket, or a shared drive:
@@ Line 192: / Line 199: @@
 ===== What to Report =====
-A reviewer cannot check a traffic-file measurement without these, and in our corpus most papers give none of them:
+A reviewer cannot check a traffic-file measurement without these. We can only measure how often the first one is given — 12.7% of capture-tool uses state a version — so treat the rest as a checklist rather than as a measured gap:
-  - **Which layer** — browser, proxy, or packets — and **which writer**, by name and version. "We collected HAR files" identifies nothing: only **12.7%** of capture-tool mentions in the corpus state a version at all, and for ''tcpdump'' it is **3.4%**.
+  - **Which layer** — browser, proxy, or packets — and **which writer**, by name and version. "We collected HAR files" identifies nothing: only **12.7%** of capture-tool uses in the corpus state a version at all, and for ''tcpdump'' it is **3.4%**.
   - **Whether bodies were recorded**, because it changes what your numbers can mean and whether anyone can re-derive them.
   - **How a request was attributed to a page**, especially from a proxy capture where the file carries no page boundaries.
@@ Line 201: / Line 208: @@
   - **What was stripped before release**, field by field.
-<wrap todo>
+<WRAP todo>
 Wanted: a measurement of how much a browser HAR and a proxy HAR of the //same// crawl disagree at scale, on real sites rather than a fixture. Our fixture shows the mechanisms; nobody in this corpus has quantified the gap over a top-list crawl. If you run it, please add it here.
-</wrap>
+</WRAP>
 ===== Use in Publications =====
@@ Line 213: / Line 220: @@
 The corpus records capture tooling in two categories. Taking only tuples marked //used// or //produced//: **509 papers** name a ''traffic-capture'' tool and **206** name a ''proxy-interception'' tool, overlapping on 36, for a union of **679**. Of those 679, **217** measured the web platform and **178** ran an automated web crawl.
-That raw union is not the page's population. On a broad security corpus the ''traffic-capture'' category is dominated by instruments that produce no web traffic file at all — software-defined radios, Bluetooth and 802.15.4 sniffers, oscilloscopes and logic analysers, screen recorders, micro-architectural side-channel probes, load generators, and commercial VPN services that are vantage points rather than instruments. Folding the 457 distinct tool names into families and excluding those explicitly leaves **511 papers** naming an on-topic instrument; **165 papers** name only off-topic ones. The fold and its residue are on [[provenance:programming:traffic_files]].
+That raw union is not the page's population. On a broad security corpus the ''traffic-capture'' category is dominated by instruments that produce no web traffic file at all — software-defined radios, Bluetooth and 802.15.4 sniffers, oscilloscopes and logic analysers, screen recorders, micro-architectural side-channel probes, load generators, and commercial VPN services that are vantage points rather than instruments. Folding the 457 distinct tool names into families and excluding those explicitly leaves **514 papers** naming an on-topic instrument and **162** naming only off-topic ones. The remaining **3** name only tools the fold could not identify at all — "a browser add-on we designed", "custom addon script", "custom tracing plugin" — which is the honest verdict when a paper does not say what it used. The fold and its residue are on [[provenance:programming:traffic_files]].
 ^ Family ^ Layer ^ Papers ^ Share of 679 ^ Distinct spellings ^
@@ Line 224: / Line 231: @@
 | record-and-replay / web archive | browser | 9 | 1.3% | 9 |
 | browser-native HTTP log (HAR / NetLog) | browser | 8 | 1.2% | 8 |
+| forward / caching proxy, self-hosted (Squid, Privoxy) | proxy | 7 | 1.0% | 6 |
 The named instruments, folded across spellings and counted by paper:
@@ Line 243: / Line 251: @@
 | OWASP ZAP | 2 | 0.3% |
-**The layer flips when you restrict to the web.** Of the 511 on-topic papers, 62.2% name a packet-capture tool and 37.0% a proxy. Of the **186** whose platform includes //web//, it is **49.5% proxy** against **46.2% packet** — the packet layer's dominance is an artefact of the venues, not advice about web measurement.
+**The layer flips when you restrict to the web.** Of the 514 on-topic papers, 61.9% name a packet-capture tool and 37.7% a proxy. Of the **188** whose platform includes //web//, it is **50.5% proxy** against **45.7% packet** — the packet layer's dominance is an artefact of the venues, not advice about web measurement.
 ==== HAR Is Invisible to a Tool Query ====
-If you ask the extraction which papers used HAR, you get almost nothing: HAR appears as a named tool in **10** papers. So we swept the full text of all 5,855 papers with stored text for ''/HAR|HTTP Archive|httparchive/'' and hand-classified all **95** hits. The sweep cannot be automated, because both names are homographs.
+If you ask the extraction which papers used HAR, you get almost nothing: HAR appears as a named tool in **10** papers. So we swept the full text of all 5,855 papers with stored text for HAR, ''.har'', "HTTP Archive" and "httparchive" — the exact regex, which is word-boundary anchored so that //SHARE// and //CHART// do not match, is on [[provenance:programming:traffic_files]] — and hand-classified all **95** hits. The sweep cannot be automated, because both names are homographs.
 ^ Verdict ^ Papers ^ Share of the 95 ^
@@ Line 260: / Line 268: @@
 **Of the 32 papers that use HAR as an instrument, only 9 (28.1%) name it in any tool field, and only 16 (50.0%) have any traffic-capture or proxy tuple at all.** A structured tool query would have found at most half of them. That is a fact about how the field reports its instruments, not about the extraction: writing HAR files is treated as too ordinary to name.
-Where they are, by year bucket: 2010–2013 **1**, 2014–2017 **3**, 2018–2021 **12**, 2022–2024 **9**, 2025–2026 **7** (2025–2026 is provisional — CCS 2026 and IMC 2026 have not been held). By venue: IMC 9, PoPETs 8, TheWebConf 7, IEEE S&P 4, USENIX Security 3, NDSS 1. The absence of CCS is real: no CCS paper in the corpus uses HAR as an instrument.
+Where they are, by year bucket: 2010–2013 **1**, 2014–2017 **3**, 2018–2021 **12**, 2022–2024 **9**, 2025–2026 **7** (2025–2026 is provisional — CCS 2026 and IMC 2026 have not been held). By venue: IMC 9, PoPETs 8, TheWebConf 7, IEEE S&P 4, USENIX Security 3, NDSS 1. No CCS paper in the corpus uses HAR as an instrument, and that is not a small-//n// artefact: CCS contributes **163** crawling papers, more than IMC's 132, and IMC contributes 9 of the 32.
-And it stays a minority practice inside crawling. Of the **1,120** papers that ran an automated web crawl, **155 (13.8%)** name an on-topic capture instrument and **26 (2.3%)** use HAR; the union is **171 (15.3%)**. Six papers in seven that crawl the web either never wrote a traffic file or never said so.
+And it stays a minority practice inside crawling. Of the **1,120** papers that ran an automated web crawl, **156 (13.9%)** name an on-topic capture instrument and **26 (2.3%)** use HAR; the union is **172 (15.4%)**. Six papers in seven that crawl the web either never wrote a traffic file or never said so.
 ==== The Instrument Goes Unidentified ====
-Of the 662 tool tuples in on-topic capture families, **84 (12.7%)** state a version.
+Of the 669 tool tuples in on-topic capture families, **85 (12.7%)** state a version.
 ^ Tool ^ Papers ^ Tuples ^ Tuples with a version ^ Share ^
@@ Line 273: / Line 281: @@
 | mitmproxy | 98 | 101 | 25 | 24.8% |
-mitmproxy is reported three times better than ''wireshark'' and seven times better than ''tcpdump'', which is the right way round — mitmproxy's behaviour changes between major versions in ways that matter, and researchers seem to know it.
+mitmproxy is reported twice as often as Wireshark and seven times as often as ''tcpdump'', which is the right way round — mitmproxy's behaviour changes between major versions in ways that matter, and researchers seem to know it.
 For HAR the equivalent question is which //writer// produced the file, and it can only be answered as an upper bound: a keyword probe over the full text of the 32 instrument papers finds a recognisable writer in 25 (78.1%), but it counts any mention of Selenium or DevTools anywhere in the paper, so the true figure is lower. Seven of the 32 name no recognisable writer anywhere at all.
@@ Line 290: / Line 298: @@
 ===== What We Ran =====
-The two comparisons on this page are reproducible. ''fixture_traffic.mjs'' serves a local page containing a redirect chain, script-initiated requests, a POST body, a 200 KiB response, a request that is never answered, a WebSocket, and an image blocked by the page's own CSP. ''har_vs_cdp.mjs'' records one load of it as a HAR and as a CDP event stream simultaneously; ''mitm_vs_browser.mjs'' records one load through ''mitmdump'' while the browser writes its own HAR.
+The two comparisons on this page are reproducible. ''fixture_traffic.mjs'' serves a local page containing a redirect chain, script-initiated requests, a POST body, a 200 KiB response, a request that is never answered, a WebSocket, and an image blocked by the page's own CSP. ''har_vs_cdp.mjs'' records one load of it as a HAR and as a CDP event stream simultaneously; ''mitm_vs_browser.mjs'' records one load through ''mitmdump'' while the browser writes its own HAR. ''mitm_vs_browser.mjs'' was run **twice**: once against the fixture, and once against ''https://example.com/'', because the ''serverIPAddress'' claim cannot be shown on a loopback target where both recorders read ''127.0.0.1''.
 <file javascript har_vs_cdp_core.mjs>
@@ Line 310: / Line 318: @@
 await cdp.send('Network.enable');
-await page.goto('http://127.0.0.1:8098/', { waitUntil: 'load' }).catch(() => {});
+// Log the failure -- do NOT swallow it. In a crawl, a silently failed goto()
+// becomes a site recorded as having made no requests, which is indistinguishable
+// from a site that made none.
+await page.goto('http://127.0.0.1:8098/', { waitUntil: 'load' }).catch((e) => console.error('goto failed:', e.message));
 await page.waitForTimeout(2500);   // drain in-flight requests before closing
 await context.close();             // the HAR is only written on context.close()
@@ Line 338: / Line 349: @@
   * **The HAR population is a full-text sweep, not a schema query.** It finds papers that write the string; a paper that captured HAR files and never used the word is invisible. The 32 is therefore a floor.
-  * **The capture-tool fold is a judgement call.** Deciding that a software-defined radio is off-topic for a page about web traffic files and that ''selenium-wire'' is on-topic is a decision, not a measurement; the family list and the 165 excluded papers are published so you can disagree with it.
+  * **The capture-tool fold is a judgement call.** Deciding that a software-defined radio is off-topic for a page about web traffic files and that ''selenium-wire'' is on-topic is a decision, not a measurement; the family list and the 162 excluded papers are published so you can disagree with it.
-  * **The fixture comparisons are one page load of a synthetic page in one browser.** They demonstrate mechanisms — an initiator field that is absent, a CSP-blocked request a proxy cannot see — not rates. Nothing on this page claims a HAR loses //x%// of anything on real sites.
+  * **The fixture comparisons are one page load of a synthetic page in one browser.** They demonstrate mechanisms — an initiator field that is absent, a CSP-blocked request a proxy cannot see — not rates. **The fixture claims no percentages at all**; the only per-cent figures on this page about real sites are Hantke et al.'s, over 8,544 origins, and they are theirs rather than ours.
 ===== Related Pages =====