Differences

This shows you the differences between two versions of the page.

--- programming:internet_scanning [2026/09/10 21:04] – New page: the instrument page for the scan branch of design:automated_measurements — ZMap/ZGrab invocation, rate arithmetic, exclusion files, source-address hygiene, IPv6 hitlists, and the join to design:ip_classification. Authored by Claude. karel.kubicek.claude
+++ programming:internet_scanning [2026/09/17 02:37] (current) – Review passes applied: Spoki's ZMap is a closed veth loop, so ACTIVE 242->241; fixed a 174/173 self-contradiction and a 46-vs-44 denominator. Authored by Claude karel.kubicek.claude
@@ Line 6: / Line 6: @@
 **A scan is a claim about hosts, not about the web.** It sees open ports, banners, certificates and reachability. It cannot see a cookie, a script or a consent banner — those need a crawl ([[Programming:Crawler]]). And it will not tell you //whose// host answered: an address is not an organisation, and turning one into the other is [[Design:IP classification]], not a scan result.
-Three things a reviewer will ask for that a first draft usually does not have: the **rate** you sent at, the **exclusion list** you honoured and where opt-out requests went, and the **source addresses** you scanned from with their reverse DNS. Measured below on the papers in this corpus that ran an active scan, each is stated by roughly a fifth to a quarter of them.
+Three things a reviewer will ask for that a first draft usually does not have: the **rate** you sent at, the **exclusion list** you honoured and where opt-out requests went, and the **source addresses** you scanned from with their reverse DNS. Measured below on the papers in this corpus that ran an active scan, the best estimate for each is **somewhere between a seventh and a quarter** of them. Those are probe estimates on hand-read samples of ten to twelve papers — read them as "a minority, and a small one", not as percentages.
 </WRAP>
@@ Line 15: / Line 15: @@
 ^ Name ^ Definition ^ Papers ^
 | **SCAN** | ''studyTypes'' contains ''network-scan-or-probe'' — the branch population [[Design:Automated measurements]] routes with | **930** |
-| **ACTIVE** | names, as a tool it **used or produced**, an instrument that sends probes to addresses the authors chose: an address-space scanner, a banner grabber, a reachability sweep, a remote censorship probe, or an IPv6 target generator | **245** |
+| **ACTIVE** | names, as a tool it **used or produced**, an instrument //of active scanning//: an address-space scanner, a banner grabber, a reachability sweep, a remote censorship-probing platform, or an IPv6 target generator. Two edges of that rule are judgement calls, stated here rather than buried: a **censorship platform** (OONI, Censored Planet, Quack, Satellite — 23 papers) counts whether the paper ran the probes or re-analysed the platform's, because the platform //is// the instrument and the corpus rarely says which; and an **IPv6 target generator** counts although it sends nothing, because in IPv6 it is the step that decides what gets probed. Both are argued on [[provenance:programming:internet_scanning]] | **241** |
-**ACTIVE** is this page's population and it is a **floor**, not a census. It is decided by the instrument, not by the extractor's ''tools[].category'': a scanner filed under ''other'' or ''infrastructure'' still counts (that rule adds **14** papers). But one-off names are only hand-read inside ''category == "network-scanner"'', so a paper that scanned and named its scanner something unique under some other category is missed. **235** of the 245 are inside SCAN; the other **10** scanned without being tagged as a scan study.
+**ACTIVE** is this page's population and it is a **floor**, not a census. It is decided by the instrument, not by the extractor's ''tools[].category'': a scanner filed under ''other'' or ''infrastructure'' still counts (that rule adds **14** papers). But one-off names are only hand-read inside ''category == "network-scanner"'', so a paper that scanned and named its scanner something unique under some other category is missed. **232** of the 241 are inside SCAN; the other **9** scanned without being tagged as a scan study.
-The gap between the two numbers is itself a result. **534 of the 930 SCAN papers (57.4%) never name a ''network-scanner'' tool at all**, and 416 (44.7%) name no instrument this page's fold recognises anywhere in ''tools[]''. Some of those queried a dataset rather than scanning; some scanned and did not say with what. The fold, its 174 hand verdicts and the full residue are on [[provenance:programming:internet_scanning]].
+The gap between the two numbers is itself a result. **534 of the 930 SCAN papers (57.4%) never name a ''network-scanner'' tool at all**, and 416 (44.7%) name no instrument this page's fold recognises anywhere in ''tools[]''. Some of those queried a dataset rather than scanning; some scanned and did not say with what. The fold, its 173 hand verdicts and the full residue are on [[provenance:programming:internet_scanning]].
 ===== Which instrument =====
-Folded families over the 245 ACTIVE papers. The share column is of ACTIVE, and a paper can name several.
+Folded families over the 241 ACTIVE papers. The share column is of ACTIVE, and a paper can name several.
-^ Family ^ What it does ^ Papers ^ Share of 245 ^ Spellings in the corpus ^
+^ Family ^ What it does ^ Papers ^ Share of 241 ^ Spellings in the corpus ^
-| **ZMap** | stateless IPv4/IPv6 address-space scanner: sends one probe per target and never keeps per-connection state | **95** | 38.8% | 9 |
+| **ZMap** | stateless IPv4 address-space scanner: sends one probe per target and keeps no per-connection state, so a single machine can walk the whole space. IPv6 comes from forks and successors — the ''ZMap6''/''ZMapv6'' spellings in the corpus are one of them | **94** | 39.0% | 10 |
-| nmap | host and service discovery on a target list; stateful, feature-rich, orders of magnitude slower | 46 | 18.8% | 10 |
+| nmap | host and service discovery on a target list; keeps per-target state and does far more per host, so it does not scale to the address space — see the 1300x figure below | 45 | 18.7% | 10 |
-| **ZGrab / ZGrab2** | application-layer follow-up: completes the handshake ZMap only started, and records the transcript | **32** | 13.1% | 7 |
+| **ZGrab / ZGrab2** | application-layer follow-up: opens its own connection to the hosts ZMap found (ZMap itself is stateless and RSTs), completes the protocol handshake and records the transcript | **32** | 13.3% | 7 |
-| //a home-grown scanner// | the paper wrote its own, or called it "a custom scanner" | 25 | 10.2% | 21 |
+| //a home-grown scanner// | the paper wrote its own, or called it "a custom scanner" | 25 | 10.4% | 21 |
-| ping / fping / hping | reachability sweeps | 18 | 7.3% | 7 |
+| ping / fping / hping | reachability sweeps | 18 | 7.5% | 7 |
-| **XMap** | ZMap-family scanner built for IPv6 as well as IPv4 | **16** | 6.5% | 3 |
+| **XMap** | ZMap-family scanner built for IPv6 as well as IPv4 | **16** | 6.6% | 3 |
-| TLS banner grabbers | sslscan, sslyze, testssl.sh, TLS-Scanner, TLS-Crawler | 11 | 4.5% | 8 |
+| TLS banner grabbers | sslscan, sslyze, testssl.sh, TLS-Scanner, TLS-Crawler | 11 | 4.6% | 8 |
 | OONI | volunteer-run censorship measurement | 7 | 2.9% | 3 |
 | Geneva | evolves packet sequences that evade a censor | 6 | 2.4% | 1 |
-| Masscan | asynchronous IPv4 scanner, ZMap's contemporary | 5 | 2.0% | 1 |
+| Masscan | asynchronous scanner, ZMap's contemporary; IPv4 in the papers here, IPv6 in later releases | 4 | 1.7% | 1 |
 | Censored Planet / Quack / Hyperquack / Satellite / Augur / Iris | remote censorship probing from hosts you do not control | 4 / 4 / 3 / 2 / 2 / 2 | | |
-| IPv6 Hitlist Service / 6Gen / Entropy-IP / addr6 / 6Sense / 6Graph / 6Tree | IPv6 target generation — see below | 4 / 3 / 2 / 2 / 1 / 1 / 1 | | |
+| IPv6 Hitlist Service / 6Gen / Entropy/IP / addr6 / 6Sense / 6Graph / 6Tree | IPv6 target generation — see below | 4 / 3 / 2 / 2 / 1 / 1 / 1 | | |
 | Goscanner, LZR, QScanner, Karma | Go and QUIC grabbers, IoT scanners | 2 each | | |
-**Forty more families are named by exactly one paper each.** Counting those together with the generic "custom scanner" strings, **45 of 245 (18.4%) rolled their own**. That is the scanning analogue of the home-grown-crawler row on [[Programming:Crawler]], and it has the same cause: the scanner is a hundred lines around a raw socket, so people write one instead of citing one.
+**Thirty-seven more families are named by exactly one paper each** — but only **20** of those are a tool the paper itself //produced//; the other 17 are third-party tools that happen to appear once (''nc'', ''ike-scan'', ''smbmap'', ''JARM'', ''WhatWeb'' and the like). Those 20 home-grown ones plus the 25 papers whose scanner is only ever called "a custom scanner", minus the 5 that are both, give **45 of 241 (18.7%) that rolled their own**. That is the scanning analogue of the home-grown-crawler row on [[Programming:Crawler]].
 <WRAP tip>
-**Not every "scanner" in a scan paper is a scanner.** Of the 930 SCAN papers, **97 (10.4%)** name a scan **dataset** (Censys, Shodan, FOFA, ZoomEye, Rapid7's Sonar data, GreyNoise, LeakIX) and **95 (10.2%)** name a **topology** instrument (traceroute, scamper, Yarrp, alias resolution). Naming Censys is not running ZMap. If your related-work section says "N papers scanned", check which of those three things each one did.
+**Not every "scanner" in a scan paper is a scanner.** Of the 930 SCAN papers, **97 (10.4%)** name a scan **dataset** (Censys, Shodan, FOFA, ZoomEye, Rapid7's Sonar data, GreyNoise, LeakIX) and **94 (10.1%)** name a **topology** instrument (traceroute, scamper, Yarrp, alias resolution). Naming Censys is not running ZMap. If your related-work section says "N papers scanned", check which of those three things each one did.
 </WRAP>
+**These counts are corpus-wide, and they agree with [[Design:DNS]].** ZMap and XMap are one family there and two here, so that page reports **110 papers** where this one reports 94 + 16; the sets are identical, and so are the two pages' 32 for ZGrab. They did not always agree — until 2026-09-17 this page said 95 + 16 = 111 against that page's 112, and the whole gap was **one** CCS 2016 paper whose ''tools[]'' entry is the single compound string ''%%ZGrab/ZMap%%'', which two ordered first-match-wins folds sent to different families. It was not an overlap artefact: **no paper in the corpus names both ZMap and XMap**. Both folds now split that one hand-read string, and ''scripts/reconcile_zmap.mjs'' fails if the two ever diverge again or if a new compound name appears whose parts a fold loses.
 ==== Current, historical, superseded — dated 2026-09-10 ====
@@ Line 52: / Line 54: @@
 ^ Window ^ ACTIVE ^ ZMap ^ ZGrab ^ nmap ^ XMap ^ Masscan ^ home-grown ^
 | 2010–2013 | 12 | 1 (8.3%) | 0 | 5 (41.7%) | 0 | 0 | 3 (25.0%) |
-| 2014–2017 | 45 | 26 (57.8%) | 4 (8.9%) | 8 (17.8%) | 0 | 1 | 7 (15.6%) |
+| 2014–2017 | 45 | 27 (60.0%) | 4 (8.9%) | 8 (17.8%) | 0 | 1 | 7 (15.6%) |
-| 2018–2021 | 56 | 21 (37.5%) | 9 (16.1%) | 15 (26.8%) | 0 | 0 | 5 (8.9%) |
+| 2018–2021 | 55 | 21 (38.2%) | 9 (16.4%) | 15 (27.3%) | 0 | 0 | 5 (9.1%) |
-| 2022–2024 | 86 | 26 (30.2%) | 9 (10.5%) | 14 (16.3%) | 10 (11.6%) | 4 | 6 (7.0%) |
+| 2022–2024 | 85 | 24 (28.2%) | 9 (10.6%) | 13 (15.3%) | 10 (11.8%) | 3 | 6 (7.1%) |
-| 2025–2026* | 46 | 21 (45.7%) | 10 (21.7%) | 4 (8.7%) | 6 (13.0%) | 0 | 4 (8.7%) |
+| 2025–2026* | 44 | 21 (47.7%) | 10 (22.7%) | 4 (9.1%) | 6 (13.6%) | 0 | 4 (9.1%) |
   * **ZMap and ZGrab2 are current practice, and this is not only a corpus artefact.** ZMap's first appearance in ACTIVE is **2013**, the year of the tool paper {[durumeric2013_zmap]} — which is itself **not in this extraction** (missing from both the USENIX 2013 index and ''data/fulltext''), so the in-corpus paper to read is the ten-year retrospective {[durumeric2024_years]}. It reports that //"over 33% of all Internet-wide IPv4 scan traffic can be fingerprinted as coming from ZMap"//, and, for Q1 2024, //"35.4% of all Internet-wide IPv4 TCP scan packets originated from ZMap"//. The repository ''zmap/zmap'' is not archived, was pushed on **2026-08-28**, and released **v4.4.0** on 2026-05-29; ''zmap/zgrab2'' released **v1.0.0** on 2025-12-04 and was pushed on 2026-09-07.((Checked 2026-09-10 via the GitHub REST API; the commands are on [[provenance:programming:internet_scanning]].))
-  * **nmap is not the Internet-wide instrument and the corpus shows it stopping being one.** It leads the 2010–2013 bucket (5 of 12) and is 4 of 46 in 2025–2026*. nmap is stateful and per-host; it is the right tool for a few thousand targets and the wrong one for 3.7 billion. nmap 7.991 is current on ''nmap.org''.
+  * **nmap is not the Internet-wide instrument and the corpus shows it stopping being one.** It leads the 2010–2013 bucket (5 of 12) and is 4 of 44 in 2025–2026*. nmap is stateful and per-host; it is the right tool for a few thousand targets and the wrong one for 3.7 billion. The original ZMap paper measured the gap directly: ZMap is //"capable of scanning the IPv4 public address space over 1300 times faster than the most aggressive Nmap default settings, with equivalent accuracy"// {[durumeric2013_zmap]}. nmap is not obsolete — it is a different instrument, and 7.991 is current on ''nmap.org''.
-  * **Masscan is historical in this corpus.** Five papers, none after 2024, and its newest GitHub Release is **1.3.2, from 2021-01-31** (the repository is not archived and was pushed 2026-04-23). {[durumeric2024_years]} also records a measured reason to prefer ZMap: //"Masscan finds notably fewer hosts than ZMap, likely due to biases in its randomization algorithm"//.
+  * **Masscan is historical in this corpus.** Four papers, none after 2024, and its newest GitHub Release is **1.3.2, from 2021-01-31** (the repository is not archived and was pushed 2026-04-23). {[durumeric2024_years]} also relays a measured reason to prefer ZMap, from Adrian et al.: //"Masscan […] finds notably fewer hosts than ZMap, likely due to biases in its randomization algorithm"//.
   * **XMap is the newcomer, and it is an IPv6 story.** Sixteen papers, **all of them 2023 or later**, 6 in the provisional window. It exists because the ZMap design does not extend to a 128-bit address space unchanged.
-  * **Rolling your own has not gone away** — 4 of the 46 papers in the provisional window still do — but it has fallen from a quarter of the earliest bucket to under a tenth.
+  * **Rolling your own has not gone away** — 4 of the 44 papers in the provisional window still do — but it has fallen from a quarter of the earliest bucket to under a tenth.
 ===== The invocation a reviewer accepts =====
@@ Line 93: / Line 95: @@
   * ''%%--source-ip%%'' as a range, not a single address, and a range whose reverse DNS you control. See //Source addresses//.
   * ''%%--metadata-file%%''. Writes the run's own parameters and counters as JSON. Ship it with the dataset: it is the cheapest possible answer to "what exactly did you run".
+**Most first scans are not Internet-wide, and should not be.** If your targets are a list — the addresses behind a domain sample, one AS, the hosts a previous crawl resolved — pass them with ''%%-w/--allowlist-file%%'' (CIDRs) or ''%%-I/--list-of-ips-file%%'' (individual addresses, for more than about 10 million); the blocklist still applies on top, and everything in this page about rate, exclusions and source addresses applies unchanged. And ZMap's probe module is a choice, not a constant: ''%%--probe-module%%'' does UDP and ICMP echo as well as the default TCP SYN, which changes both what a non-answer means and the packet size the wire-rate arithmetic below assumes.
 ZMap tells you an address answered on a port. It does not tell you what is there. The second half is ZGrab2, which //"performs in-depth, follow-up L7 handshakes"// on the hosts ZMap found:((''zmap/zgrab2'' ''README.md'' at tag ''v1.0.0'', read 2026-09-10.))
@@ Line 135: / Line 139: @@
 </code>
-Read that as three facts. **ZMap's default rate takes 4.29 days for one port.** A single gigabit link finishes the same scan in **41.5 minutes**. And **ports and probes multiply**: ''%%--ports 2 --probes 2%%'' is four times every number in that table — **17.14 days** at the default rate — and a "quick" eight-port sweep is **34.28 days**. Both of those come out of the same script, with ''%%--ports%%'' and ''%%--probes%%''.
+Read that as three facts. **ZMap's default rate takes 4.29 days for one port.** A single gigabit link finishes the same scan in **41.5 minutes**. And **ports and probes multiply the wall clock, not the wire rate**: ''%%--ports 2 --probes 2%%'' is four times the packets and therefore four times every figure in the //wall clock// column — **17.14 days** at the default rate — while the //wire rate// column is unchanged, because that depends only on ''%%--rate%%''. A "quick" eight-port sweep is **34.28 days** and still 7 Mb/s. Both figures come out of the same script, with ''%%--ports%%'' and ''%%--probes%%''.
-The wire-rate column assumes **84 bytes of wire time per probe** — a 60-byte minimum Ethernet frame plus FCS, preamble and interframe gap — which is what a bare TCP SYN costs. If your probe module sends a larger packet, say so and rerun the arithmetic; the script takes the assumption as a constant you can see.
+**A cross-check on that arithmetic.** The ZMap paper reports the same scan measured rather than computed: a single machine //"surveying the entire IPv4 address space in under 45 minutes from user space"//, at //"over 97% of the theoretical maximum speed of gigabit Ethernet"//, with //"an estimated 98% coverage of publicly available hosts"// {[durumeric2013_zmap]}. The table's 41.5 minutes is the theoretical maximum for a 1 Gb/s link; 41.5 / 0.97 = 42.8, and the paper's ceiling is 45. The two agree, which is the point of publishing the script rather than a number.((That paper is **not** in this corpus's extraction. The copy read on 2026-09-10 is the authors' own at ''zmap.io/paper.pdf'' (''Last-Modified: 2025-06-22''), extracted with ''pypdf''; all four quotes were matched against it.))
-**What the literature reports.** Two probes over the 245 ACTIVE papers' full text, both upper bounds: a loose regex for any rate or bandwidth term matches **127 (51.8%)**, and a first-person one requiring "we"/"our" near the term matches **84 (34.3%)**. Reading a deterministic 12-paper sample of that second list, **6 of 12** are actually about the authors' own probe rate — the rest are a target's ICMP rate limiting, a censor's throttling, or a defence being recommended. So the honest estimate is **about 42 papers, roughly one in six**, state the rate they scanned at. The ones that do are unambiguous about it: //"a low scanning rate of 10,000 packets per second, total, across all networks"// {[williams2024_6sense]}; //"a low probing rate of 8,000 packets per second, i.e., around 500KB/s from a single vantage point"// {[holzbauer2025_tracking]}.
+The wire-rate column assumes **84 bytes of wire time per probe** — a bare TCP SYN is 54 bytes, ZMap's default ''synscan'' module sends no TCP options, and the frame is padded to the 60-byte Ethernet minimum, plus FCS, preamble and interframe gap. If your probe module sends a larger packet, say so and rerun the arithmetic; the script takes the assumption as a named constant you can see and change.
+**What the literature reports.** Two probes over the 241 ACTIVE papers' full text, both upper bounds: a loose regex for any rate or bandwidth term matches **124 (51.5%)**, and a first-person one requiring "we"/"our" near the term matches **81 (33.6%)**. Reading a pinned 12-paper sample of that second list, **5 of 12** are actually about the authors' own probe rate — the rest are a target's ICMP rate limiting, a censor's throttling, a lab attack rate, or a defence being recommended. So the honest estimate is **about 34 papers, roughly one in seven**, state the rate they scanned at. The ones that do are unambiguous about it: //"a low scanning rate of 10,000 packets per second, total, across all networks"// {[williams2024_6sense]}; //"a low probing rate of 8,000 packets per second, i.e., around 500KB/s from a single vantage point"// {[holzbauer2025_tracking]}.
 <file python scan_budget.py>
@@ Line 163: / Line 169: @@
      a four-day one.
-Stdlib only. `--self-test` runs eight checks including two exact hand-computed
+Stdlib only. `--self-test` runs nine checks including two exact hand-computed
-cases and a mutation guard on the CIDR arithmetic.
+cases, a mutation guard on the CIDR arithmetic, and boundary coverage for the
+unit formatter (added after a review pass mutated its thresholds and the
+self-test still passed, because nothing called it).
-Frame size: a bare TCP SYN with an MSS option is 58 bytes of Ethernet frame,
+Frame size: a bare TCP SYN is 54 bytes of Ethernet frame -- ZMap's default
-padded to the 60-byte minimum, plus a 4-byte FCS, plus the 12-byte interframe
+synscan module sends no TCP options -- padded up to the 60-byte minimum, plus a
-gap and the 8-byte preamble/SFD -- 84 bytes of wire time per probe. That is
+-byte FCS, plus the 12-byte interframe gap and the 8-byte preamble/SFD: 84
-the number that gives 1,488,095 pps on a 1 GbE link, and it is an assumption
+bytes of wire time per probe. The padding is why adding options does not
-you should restate if your probe module sends anything bigger.
+change the number until they push the frame past 60 bytes. 84 bytes is what
+gives 1,488,095 pps on a 1 GbE link, and it is an assumption to restate if your
+probe module sends anything bigger.
 """
@@ Line 317: / Line 327: @@
     assert round(line_rate_pps(1)) == 1_488_095
     assert round(line_rate_pps(10)) == 14_880_952
+    checks += 1
+    # 9. human_time() picks the right unit on both sides of every boundary.
+    #    Added after a review pass mutated the thresholds from 120/7200/172800
+    #    to 60/3600/86400 and the self-test still said OK: nothing called it.
+    for seconds, want_unit in [
+        (119.9, "s"), (120.0, "min"),
+        (7199.0, "min"), (7200.0, "h"),
+        (171999.0, "h"), (172800.0, "days"),
+    ]:
+        got = human_time(seconds)
+        assert got.endswith(want_unit), f"human_time({seconds}) = {got!r}, want a value in {want_unit}"
+    #    …and that the unit conversion itself is right, not just the bucket.
+    assert human_time(3600.0) == "60.0 min"
+    assert human_time(86400.0) == "24.0 h"
+    assert human_time(172800.0) == "2.00 days"
     checks += 1
@@ Line 362: / Line 388: @@
   - **Your own list**: every prefix whose operator asked you to stop, plus anything your institution excludes. This file has to outlive the study. An opt-out honoured for one paper and forgotten for the next is worse than none, because the operator now believes they opted out.
-ZMap has one ''%%--blocklist-file%%'' flag, so those two go into the same file and you keep the inputs separately. Three helper binaries ship alongside ''zmap'' and are worth knowing: **''ziterate''** //"will produce IPv4 addresses in a pseudorandom order similar to how ZMap generates random addresses to be scanned"// — run it with your seed and blocklist to see exactly what you are about to probe, without probing it; **''zblocklist''** filters an existing address list through the same rules; and **''ztee''** is the buffer //"used whenever ZMap is piped into an application scanner"//.
+ZMap has one ''%%--blocklist-file%%'' flag, so those two go into the same file and you keep the inputs separately. Three helper binaries ship alongside ''zmap'' and are worth knowing: **''ziterate''** //"will produce IPv4 addresses in a psuedorandom [sic] order similar to how ZMap generates random addresses to be scanned"// — run it with your seed and blocklist to see exactly what you are about to probe, without probing it; **''zblocklist''** filters an existing address list through the same rules; and **''ztee''** is the buffer //"used whenever ZMap is piped into an application scanner"//.
 {[durumeric2024_years]}'s own recommendation is explicit about both halves: //"Provide a simple mechanism for operators to request exclusion from future scans. Indicate the IP ranges you use for scanning so that operators can drop research traffic themselves."//
-**How rare is this?** The loose blocklist probe fires on **106 of 245 ACTIVE papers (43.3%)** and the first-person one on **90 (36.7%)**, but hand-reading a 12-paper sample of the latter gives **6 of 12** — the other six are a censor's blocklist, a threat-intelligence feed used as input data, or an analysis-time exclusion. Estimate: **about 45 papers, roughly one in five**. The opt-out probe is tighter and cleaner: **69 papers (28.2%)** match a first-person opt-out pattern and **9 of a 10-paper sample** are genuine, giving **about 62 papers, one in four**.
+**How rare is this?** The loose blocklist probe fires on **103 of 241 ACTIVE papers (42.7%)** and the first-person one on **88 (36.5%)**, but hand-reading an 11-paper sample of the latter gives **6 of 11** — the other five are a censor's blocklist, a threat-intelligence feed used as input data, or an analysis-time exclusion. Estimate: **about 48 papers, roughly one in five**. The opt-out probe is tighter and cleaner: **70 papers (29.0%)** match a first-person opt-out pattern and **9 of a 10-paper sample** are genuine, giving **about 63 papers, one in four**.
-Two figures worth carrying. Durumeric et al. {[durumeric2014_view]} report that after years of running the largest academic scanning operation of its day, //"only 208 organizations have requested that we exclude their networks from our scans, reducing the IPv4 address space for study by only 0.15%"//. And Beverly et al. {[beverly2018_beholder]}, on a six-month IPv6 topology campaign: //"we received two opt-out requests with which we immediately complied"//. The opt-out volume is small. That is an argument for offering it, not against.
+Two figures worth carrying. Durumeric et al. {[durumeric2014_view]} report that after years of running the largest academic scanning operation of its day, //"only 208 organizations have requested that we exclude their networks from our scans, reducing the IPv4 address space for study by only 0.15%"//. And Beverly et al. {[beverly2018_beholder]}, on a run of IPv6 topology campaigns between February and May 2018: //"we received two opt-out requests with which we immediately complied"//. The opt-out volume is small. That is an argument for offering it, not against.
 The **norms** around this — what your NREN requires, whether an ethics board will look at it, what operators say they will tolerate — are on [[Practices:Ethics]], which quotes SURF's and Jisc's actual policy text and Hantke et al.'s operator interviews. Do not re-derive them here.
@@ Line 374: / Line 400: @@
 ===== Source addresses and abuse contacts =====
-The address you scan from is the only channel a scanned operator has to reach you, and the only evidence they have that you are not an attacker. The convention this literature has converged on is four things on the scanning host itself:
+The address you scan from is the only channel a scanned operator has to reach you, and the only evidence they have that you are not an attacker. The convention the ZMap-era methods papers set — and, on the estimate below, about a quarter of scanning papers follow — is four things on the scanning host itself:
   * A **PTR (reverse DNS) record** that says what it is, on every address in the ''%%--source-ip%%'' range. Not ''host-10.example.edu''.
@@ Line 381: / Line 407: @@
   * A **stable, dedicated range** — so a repeat scan comes from the same place and an operator's own blocklist keeps working.
-{[durumeric2015_search]} describes exactly this for Censys' own infrastructure: //"All of the scanning hosts have WHOIS records and reverse DNS entries that describe the intent of the scanning. Further, each scanning host runs a simple website on port 80 that describes the goals of the research"//. {[kaspereit2024_landscape]} adds the fourth party most papers forget: //"On the scanning IPs, a website gives project details and contact information for opt-out requests. We honored all such requests. Our ISP, national CERT, and university network administrator cleared all scans."// Telling your own network operations team **before** the scan is the difference between a conversation and a ticket.
+{[durumeric2015_search]} describes exactly this for Censys' own infrastructure: //"All of the scanning hosts have WHOIS records and reverse DNS entries that describe the intent of the scanning. Further, each scanning host runs a simple website on port 80 that describes the goals of the research"//. {[kaspereit2024_landscape]} adds the people almost nothing in the corpus mentions — your own operators: //"On the scanning IPs, a website gives project details and contact information for opt-out requests. We honored all such requests. Our ISP, national CERT, and university network administrator cleared all scans."// Telling your own network operations team **before** the scan is the difference between a conversation and a ticket.
-**In the corpus**: the loose source-address probe matches **170 of 245 (69.4%)** and the first-person one **114 (46.5%)**, with **6 of a 10-paper sample** genuine — about **68 papers, a little over a quarter**, describe any of this. Abuse-complaint handling is discussed by **46 (18.8%)** on the first-person probe (upper bound; no precision measured). And **56 (22.9%)** name an IRB or ethics board at all, against **174 (71.0%)** that cite Durumeric, Partridge and Allman, the Menlo Report, or the phrase "good Internet citizen" — citing the norm is four times more common than being reviewed against it.
+**In the corpus**: the loose source-address probe matches **167 of 241 (69.3%)** and the first-person one **112 (46.5%)**, with **6 of a 10-paper sample** genuine — about **67 papers, a little over a quarter**, describe any of this. Abuse-complaint handling is discussed by **45 (18.7%)** on the first-person probe (upper bound; no precision measured). Two more probes, both loose and neither hand-audited, so read them as upper bounds and not against each other: **55 (22.8%)** match an IRB / ethics-board pattern and **172 (71.4%)** cite Durumeric, Partridge and Allman, the Menlo Report, or the phrase "good Internet citizen". Do **not** read the 55 as "only 55 had ethics review": the extraction's own ''ethics.reviewOutcome'' field is stated on **92 (38.2%)** of the same 241 papers, and that is the number to quote.
 <WRAP tip>
@@ Line 401: / Line 427: @@
 ^ Query ^ Population ^ Papers ^ Share ^
 | mentions IPv6 anywhere | SCAN 930 | 281 | 30.2% |
-| mentions IPv6 anywhere | ACTIVE 245 | 108 | 44.1% |
+| mentions IPv6 anywhere | ACTIVE 241 | 107 | 44.4% |
 | names an IPv6-targeting instrument in ''tools[]'' | SCAN 930 | **11** | 1.2% |
 | full text says "hitlist" / "hit list" | SCAN 930 | 62 | 6.7% |
@@ Line 408: / Line 434: @@
 | discusses aliased prefixes or dealiasing | SCAN 930 | **10** | 1.1% |
-Almost a third of scan papers mention IPv6; **one in a hundred discusses the aliasing problem that decides whether their IPv6 counts mean anything**. The eleven papers naming a targeting instrument are, in full: the IPv6 Hitlist Service (4), 6Gen (3), Entropy/IP (2), addr6 (2), 6Graph, 6Tree and 6Sense (1 each) — and every one of those eleven is IMC, NDSS, PETS or USENIX, none earlier than 2016. IPv6 mention inside SCAN rises from 12.5% in 2010 to 44.3% in 2025.
+Almost a third of scan papers mention IPv6. The aliasing row needs its own denominator, though: 10 of 930 is 1.1%, but most of those 930 never touch IPv6. Against the **42** papers that either name an IPv6-targeting instrument or say "hitlist" near "IPv6", it is **8 of 42 (19.0%)** — still four in five of the papers doing IPv6 targeting not discussing the thing that decides whether their counts mean anything, and a far more defensible way to say it. The eleven papers naming a targeting instrument are, in full: the IPv6 Hitlist Service (4), 6Gen (3), Entropy/IP (2), addr6 (2), 6Graph, 6Tree and 6Sense (1 each) — and every one of those eleven is IMC, NDSS, PETS or USENIX, none earlier than 2016. IPv6 mention inside SCAN is up over the window, but not monotonically: 12.5% in 2010, 0% in 2013, 35.2% in 2018, 29.2% in 2024, 44.3% in 2025 and 26.1% in the provisional 2026.
-If you are scanning IPv6, the sentence a reviewer wants is: **which hitlist or generator, which snapshot date, and whether you de-aliased.** Almost nobody writes it.
+If you are scanning IPv6, the sentence a reviewer wants is: **which hitlist or generator, which snapshot date, and whether you de-aliased.** Four in five of the papers that look like they are doing this do not write the de-aliasing half.
 ===== From scan output to a claim =====
@@ Line 416: / Line 442: @@
 ZMap hands you a column of source addresses. That is not a result, and the step that turns it into one is where scanning papers most often overreach.
-  * **An address is not an organisation.** The [[Design:IP classification]] page exists for the join — ASN, geolocation, network type, hosting vs eyeball, reputation. **77 of the 245 ACTIVE papers (31.4%)** carry a ''classification'' tuple whose target is ''ip-address'', against 22.9% of SCAN and 5.0% of the corpus; the raw resource names they reach for are 116 distinct free-text strings, led by MaxMind and IPinfo. Fold them before you count anything — that page publishes the fold.
+  * **An address is not an organisation.** The [[Design:IP classification]] page exists for the join — ASN, geolocation, network type, hosting vs eyeball, reputation. **75 of the 241 ACTIVE papers (31.1%)** carry a ''classification'' tuple whose target is ''ip-address'', against 22.9% of SCAN and 5.0% of the corpus; the raw resource names they reach for are 110 distinct free-text strings, led by MaxMind and IPinfo. Fold them before you count anything — that page publishes the fold.
   * **An address is not a name.** Reverse DNS is not injective and is often absent; a certificate's SAN list, a Host header or a hitlist's provenance are usually better joins than a PTR.
   * **A responding address is not a distinct machine.** In IPv6 this is aliasing, above. In IPv4 it is anycast, load balancers and CDNs: the same content answers from thousands of addresses, and "N hosts run X" quietly becomes "N addresses answered".
@@ Line 424: / Line 450: @@
 ===== What to report =====
-The methods paragraph a reviewer should be able to find, and how often the 245 ACTIVE papers actually contain each part. Schema fields are counted from the extraction; probe rows are the hand-corrected estimates from the sections above and are marked.
+The methods paragraph a reviewer should be able to find, and how often the 241 ACTIVE papers actually contain each part. Schema fields are counted from the extraction; probe rows are the hand-corrected estimates from the sections above and are marked.
-^ What ^ ACTIVE (245) ^ SCAN (930) ^
+^ What ^ ACTIVE (241) ^ SCAN (930) ^
-| The **scanner and its version** — version of the scanner itself, not of some tool | **20 (8.2%)** | 20 (2.2%) |
+| The **scanner and its version** — version of the scanner itself, not of some tool | **20 (8.3%)** | 20 (2.2%) |
-| Any used-or-produced tool version at all | 118 (48.2%) | 401 (43.1%) |
+| Any used-or-produced tool version at all | 115 (47.7%) | 401 (43.1%) |
-| **Vantage location** you scanned from | 136 (55.5%) | 487 (52.4%) |
+| **Vantage location** you scanned from | 134 (55.6%) | 487 (52.4%) |
-| Vantage infrastructure (cloud, university, residential) | 168 (68.6%) | 612 (65.8%) |
+| Vantage infrastructure (cloud, university, residential) | 164 (68.0%) | 612 (65.8%) |
-| **Probe rate** //(probe estimate)// | ~42 (~17%) | — |
+| **Probe rate** //(probe estimate)// | ~34 (~14%) | — |
-| **Exclusion list honoured** //(probe estimate)// | ~45 (~18%) | — |
+| **Exclusion list honoured** //(probe estimate)// | ~48 (~20%) | — |
-| **Opt-out offered** //(probe estimate)// | ~62 (~25%) | — |
+| **Opt-out offered** //(probe estimate)// | ~63 (~26%) | — |
-| **Source-address hygiene** //(probe estimate)// | ~68 (~28%) | — |
+| **Source-address hygiene** //(probe estimate)// | ~67 (~28%) | — |
-| Ethics review outcome | 93 (38.0%) | 290 (31.2%) |
+| Ethics review outcome | 92 (38.2%) | 290 (31.2%) |
-| Notified affected parties | 186 (75.9%) | 560 (60.2%) |
+| Notified affected parties | 184 (76.3%) | 560 (60.2%) |
-| Harm mitigation described | 212 (86.5%) | 700 (75.3%) |
+| Harm mitigation described | 208 (86.3%) | 700 (75.3%) |
-| Measurement start date | 196 (80.0%) | 656 (70.5%) |
+| Measurement start date | 192 (79.7%) | 656 (70.5%) |
-| Own artifact URL | 146 (59.6%) | 506 (54.4%) |
+| Own artifact URL | 144 (59.8%) | 506 (54.4%) |
 **Eight per cent name the version of the scanner they ran.** That is the worst row on the table and the easiest to fix: ZMap's behaviour has changed materially inside the version range this corpus covers — {[durumeric2024_years]} records that //"In early 2024, ZMap changed its default behavior to use random per-probe IP IDs"//, which is exactly the kind of change that moves an IP-ID side-channel result — and ''%%zmap --version%%'' costs nothing. Also add: your **port set**, your **seed**, your **probes-per-target**, the **wall-clock span** (''temporal.spanStart'' is missing on a fifth of ACTIVE papers), and, for IPv6, **which hitlist snapshot and whether you de-aliased**.
@@ Line 459: / Line 485: @@
 | Wu et al., NDSS 2025, //Revealing the Black Box of Device Search Engine// {[wu2025_revealing]} | What commercial scanning does that academic scanning should not. Also on [[Practices:Ethics]]. |
-Leonard and Loguinov's IRLscanner {[leonard2010_demystifying]} is the pre-ZMap ancestor and is worth reading for its complaint analysis, which is still the most detailed in this literature. Kaspereit et al. {[kaspereit2024_landscape]} is a recent methods section to imitate.
+Leonard and Loguinov's IRLscanner {[leonard2010_demystifying]} is the pre-ZMap ancestor, and its complaint analysis — what people actually wrote back, and about what — is the most detailed one this page found. Kaspereit et al. {[kaspereit2024_landscape]} is a recent methods section to imitate.
 ===== Someone else's scan is often the right answer =====
@@ Line 472: / Line 498: @@
 ===== Methodology and limitations of these figures =====
-Every number above is a count of **papers**, from the 5,859-paper extraction, with its denominator in the same sentence or table header. Sentinels (''not-stated'', ''none-mentioned'') are never counted as answers, and an absent ''ethics'' or ''artifacts'' object is counted as neither (7 and 3 ACTIVE papers). Free-text instrument names are folded before counting: the fold is ''scripts/scan_fold.mjs'', it decides **746** used-or-produced ''network-scanner'' tuples with **zero** unmapped strings, and its 174 hand verdicts and full residue are published on the provenance page. ZMap alone appears under **9** spellings.
+Every number above is a count of **papers**, from the 5,859-paper extraction, with its denominator in the same sentence or table header. Sentinels (''not-stated'', ''none-mentioned'') are never counted as answers, and an absent ''ethics'' or ''artifacts'' object is counted as neither (7 and 3 ACTIVE papers). Free-text instrument names are folded before counting: the fold is ''scripts/scan_fold.mjs'', it decides **741** used-or-produced ''network-scanner'' tuples with **zero** unmapped strings, and its 173 hand verdicts and full residue are published on the provenance page. ZMap alone appears under **10** spellings.
 Four limitations to carry:
-  * **ACTIVE is a floor.** A paper that scanned and named its scanner something unique outside ''tools[].category == "network-scanner"'' is not counted. The corresponding over-count risk is the opposite of small: the ''ZMap'' regex ''%%/z-?map/i%%'' matches //WizMap//, an unrelated visualisation tool, and is guarded against it explicitly.
+  * **ACTIVE is a floor.** A paper that scanned and named its scanner something unique outside ''tools[].category == "network-scanner"'' is not counted. The over-count risk is guarded rather than assumed: the ''ZMap'' regex ''%%/z-?map/i%%'' matches //WizMap//, an unrelated visualisation tool that appears in one paper, and the fold names and excludes it explicitly.
-  * **The four practice rows are probe estimates on samples of 10–12.** Read "about a fifth", not "18.4%". The probes, the samples, the per-paper verdicts and the deciding sentence for each are on the provenance page.
+  * **The four practice rows are probe estimates on samples of 10–12.** Read "about a fifth", not "19.9%". The probes, the samples, the per-paper verdicts and the deciding sentence for each are on the provenance page.
   * **''studyTypes'' is the least stable field in the schema** (57% run-to-run on a 100-paper sample, not re-measured on this corpus). SCAN = 930 is a ranking-grade number, so treat every //share of 930// as approximate. ACTIVE does not depend on it.
-  * **2025–2026 venue-years are provisional** and the corpus is seven venues. Network-measurement work also appears at PAM, TMA, ANRW, CoNEXT and ACSAC, **none of which are in this corpus** — for scanning specifically that is a bigger gap than for the web pages on this site, and several instruments above (Yarrp, MIDAR, scamper) have their tool papers at those venues.
+  * **The hand verdicts have been read once, by one reader.** All 173 were audited against each tool's ''purpose'' text and its paper's title and full text on 2026-09-17; five kinds changed, three of them taking a paper out of ACTIVE and one putting one in, and five tuples across two papers were suppressed — four because a network-telescope paper fingerprinted the scanners rather than running them, one because a ZMap run never left its own machine. ACTIVE moved from 245 to **241**. Every change, with its deciding sentence, is on the provenance page, and so is the review pass that checked them: it confirmed all five kind changes and all six borderlines against the papers, and found the fifth suppression, which this run had missed.
+  * **2025–2026 venue-years are provisional** and the corpus is seven venues. Network-measurement work also appears at PAM, TMA, ANRW, CoNEXT and ACSAC, **none of which are in this corpus** — for scanning specifically that is a bigger gap than for the web pages on this site, and several instrument papers this literature leans on are published there. Do not take that as a claim about any specific tool above — Yarrp's own paper, for one, is IMC 2016 and //is// in this corpus.
 The report script is ''scripts/report_internet_scanning.mjs''. It exits 1 if the corpus size, the SCAN population, the fold coverage, a stale hand verdict, or a hand-read sample that has drifted disagrees with the contracts it encodes. Full query log, folds, residue, quote checks, rejected sources and the review log: [[provenance:programming:internet_scanning]]. Corpus-level caveats: [[Literature:Corpus]].