Measure The Web

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Internet Scanning ======

You are about to probe addresses instead of loading pages. That is the **scan** branch of [[Design:Automated measurements]], and this is its instrument page — the counterpart of [[Programming:Crawler]] for the other branch. It covers the invocation, the rate, the exclusion file, the source addresses you scan from, why IPv6 is a different problem, and what you have to do to the output before it becomes a claim. The **norms** — whether to scan at all, what your NREN's policy says, how to write the ethics section — are on [[Practices:Ethics]], which has a scanning checklist this page does not repeat.

<WRAP important>
**A scan is a claim about hosts, not about the web.** It sees open ports, banners, certificates and reachability. It cannot see a cookie, a script or a consent banner — those need a crawl ([[Programming:Crawler]]). And it will not tell you //whose// host answered: an address is not an organisation, and turning one into the other is [[Design:IP classification]], not a scan result.

Three things a reviewer will ask for that a first draft usually does not have: the **rate** you sent at, the **exclusion list** you honoured and where opt-out requests went, and the **source addresses** you scanned from with their reverse DNS. Measured below on the papers in this corpus that ran an active scan, the best estimate for each is **somewhere between a seventh and a quarter** of them. Those are probe estimates on hand-read samples of ten to twelve papers — read them as "a minority, and a small one", not as percentages.
</WRAP>

===== Where the figures on this page come from =====

Everything below about "the literature" is a claim about **seven venues** — CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026, **5,859** papers with extracted full text ([[Literature:Corpus]]). Two populations, never mixed:

^ Name ^ Definition ^ Papers ^
| **SCAN** | ''studyTypes'' contains ''network-scan-or-probe'' — the branch population [[Design:Automated measurements]] routes with | **930** |
| **ACTIVE** | names, as a tool it **used or produced**, an instrument //of active scanning//: an address-space scanner, a banner grabber, a reachability sweep, a remote censorship-probing platform, or an IPv6 target generator. Two edges of that rule are judgement calls, stated here rather than buried: a **censorship platform** (OONI, Censored Planet, Quack, Satellite — 23 papers) counts whether the paper ran the probes or re-analysed the platform's, because the platform //is// the instrument and the corpus rarely says which; and an **IPv6 target generator** counts although it sends nothing, because in IPv6 it is the step that decides what gets probed. Both are argued on [[provenance:programming:internet_scanning]] | **241** |

**ACTIVE** is this page's population and it is a **floor**, not a census. It is decided by the instrument, not by the extractor's ''tools[].category'': a scanner filed under ''other'' or ''infrastructure'' still counts (that rule adds **14** papers). But one-off names are only hand-read inside ''category == "network-scanner"'', so a paper that scanned and named its scanner something unique under some other category is missed. **232** of the 241 are inside SCAN; the other **9** scanned without being tagged as a scan study.

The gap between the two numbers is itself a result. **534 of the 930 SCAN papers (57.4%) never name a ''network-scanner'' tool at all**, and 416 (44.7%) name no instrument this page's fold recognises anywhere in ''tools[]''. Some of those queried a dataset rather than scanning; some scanned and did not say with what. The fold, its 173 hand verdicts and the full residue are on [[provenance:programming:internet_scanning]].

===== Which instrument =====

Folded families over the 241 ACTIVE papers. The share column is of ACTIVE, and a paper can name several.

^ Family ^ What it does ^ Papers ^ Share of 241 ^ Spellings in the corpus ^
| **ZMap** | stateless IPv4 address-space scanner: sends one probe per target and keeps no per-connection state, so a single machine can walk the whole space. IPv6 comes from forks and successors — the ''ZMap6''/''ZMapv6'' spellings in the corpus are one of them | **94** | 39.0% | 10 |
| nmap | host and service discovery on a target list; keeps per-target state and does far more per host, so it does not scale to the address space — see the 1300x figure below | 45 | 18.7% | 10 |
| **ZGrab / ZGrab2** | application-layer follow-up: opens its own connection to the hosts ZMap found (ZMap itself is stateless and RSTs), completes the protocol handshake and records the transcript | **32** | 13.3% | 7 |
| //a home-grown scanner// | the paper wrote its own, or called it "a custom scanner" | 25 | 10.4% | 21 |
| ping / fping / hping | reachability sweeps | 18 | 7.5% | 7 |
| **XMap** | ZMap-family scanner built for IPv6 as well as IPv4 | **16** | 6.6% | 3 |
| TLS banner grabbers | sslscan, sslyze, testssl.sh, TLS-Scanner, TLS-Crawler | 11 | 4.6% | 8 |
| OONI | volunteer-run censorship measurement | 7 | 2.9% | 3 |
| Geneva | evolves packet sequences that evade a censor | 6 | 2.4% | 1 |
| Masscan | asynchronous scanner, ZMap's contemporary; IPv4 in the papers here, IPv6 in later releases | 4 | 1.7% | 1 |
| Censored Planet / Quack / Hyperquack / Satellite / Augur / Iris | remote censorship probing from hosts you do not control | 4 / 4 / 3 / 2 / 2 / 2 | | |
| IPv6 Hitlist Service / 6Gen / Entropy/IP / addr6 / 6Sense / 6Graph / 6Tree | IPv6 target generation — see below | 4 / 3 / 2 / 2 / 1 / 1 / 1 | | |
| Goscanner, LZR, QScanner, Karma | Go and QUIC grabbers, IoT scanners | 2 each | | |

**Thirty-seven more families are named by exactly one paper each** — but only **20** of those are a tool the paper itself //produced//; the other 17 are third-party tools that happen to appear once (''nc'', ''ike-scan'', ''smbmap'', ''JARM'', ''WhatWeb'' and the like). Those 20 home-grown ones plus the 25 papers whose scanner is only ever called "a custom scanner", minus the 5 that are both, give **45 of 241 (18.7%) that rolled their own**. That is the scanning analogue of the home-grown-crawler row on [[Programming:Crawler]].

<WRAP tip>
**Not every "scanner" in a scan paper is a scanner.** Of the 930 SCAN papers, **97 (10.4%)** name a scan **dataset** (Censys, Shodan, FOFA, ZoomEye, Rapid7's Sonar data, GreyNoise, LeakIX) and **94 (10.1%)** name a **topology** instrument (traceroute, scamper, Yarrp, alias resolution). Naming Censys is not running ZMap. If your related-work section says "N papers scanned", check which of those three things each one did.
</WRAP>

**These counts are corpus-wide, and they agree with [[Design:DNS]].** ZMap and XMap are one family there and two here, so that page reports **110 papers** where this one reports 94 + 16; the sets are identical, and so are the two pages' 32 for ZGrab. They did not always agree — until 2026-09-17 this page said 95 + 16 = 111 against that page's 112, and the whole gap was **one** CCS 2016 paper whose ''tools[]'' entry is the single compound string ''%%ZGrab/ZMap%%'', which two ordered first-match-wins folds sent to different families. It was not an overlap artefact: **no paper in the corpus names both ZMap and XMap**. Both folds now split that one hand-read string, and ''scripts/reconcile_zmap.mjs'' fails if the two ever diverge again or if a new compound name appears whose parts a fold loses.

==== Current, historical, superseded — dated 2026-09-10 ====

Membership families by year bucket, as a share of that bucket's own ACTIVE papers. **2025–2026 is provisional**: CCS 2026 and IMC 2026 have not been held and IEEE S&P/WWW 2026 abstracts are under-selected, so the last column is a thin slice, not a completed window.

^ Window ^ ACTIVE ^ ZMap ^ ZGrab ^ nmap ^ XMap ^ Masscan ^ home-grown ^
| 2010–2013 | 12 | 1 (8.3%) | 0 | 5 (41.7%) | 0 | 0 | 3 (25.0%) |
| 2014–2017 | 45 | 27 (60.0%) | 4 (8.9%) | 8 (17.8%) | 0 | 1 | 7 (15.6%) |
| 2018–2021 | 55 | 21 (38.2%) | 9 (16.4%) | 15 (27.3%) | 0 | 0 | 5 (9.1%) |
| 2022–2024 | 85 | 24 (28.2%) | 9 (10.6%) | 13 (15.3%) | 10 (11.8%) | 3 | 6 (7.1%) |
| 2025–2026* | 44 | 21 (47.7%) | 10 (22.7%) | 4 (9.1%) | 6 (13.6%) | 0 | 4 (9.1%) |

  * **ZMap and ZGrab2 are current practice, and this is not only a corpus artefact.** ZMap's first appearance in ACTIVE is **2013**, the year of the tool paper {[durumeric2013_zmap]} — which is itself **not in this extraction** (missing from both the USENIX 2013 index and ''data/fulltext''), so the in-corpus paper to read is the ten-year retrospective {[durumeric2024_years]}. It reports that //"over 33% of all Internet-wide IPv4 scan traffic can be fingerprinted as coming from ZMap"//, and, for Q1 2024, //"35.4% of all Internet-wide IPv4 TCP scan packets originated from ZMap"//. The repository ''zmap/zmap'' is not archived, was pushed on **2026-08-28**, and released **v4.4.0** on 2026-05-29; ''zmap/zgrab2'' released **v1.0.0** on 2025-12-04 and was pushed on 2026-09-07.((Checked 2026-09-10 via the GitHub REST API; the commands are on [[provenance:programming:internet_scanning]].))
  * **nmap is not the Internet-wide instrument and the corpus shows it stopping being one.** It leads the 2010–2013 bucket (5 of 12) and is 4 of 44 in 2025–2026*. nmap is stateful and per-host; it is the right tool for a few thousand targets and the wrong one for 3.7 billion. The original ZMap paper measured the gap directly: ZMap is //"capable of scanning the IPv4 public address space over 1300 times faster than the most aggressive Nmap default settings, with equivalent accuracy"// {[durumeric2013_zmap]}. nmap is not obsolete — it is a different instrument, and 7.991 is current on ''nmap.org''.
  * **Masscan is historical in this corpus.** Four papers, none after 2024, and its newest GitHub Release is **1.3.2, from 2021-01-31** (the repository is not archived and was pushed 2026-04-23). {[durumeric2024_years]} also relays a measured reason to prefer ZMap, from Adrian et al.: //"Masscan […] finds notably fewer hosts than ZMap, likely due to biases in its randomization algorithm"//.
  * **XMap is the newcomer, and it is an IPv6 story.** Sixteen papers, **all of them 2023 or later**, 6 in the provisional window. It exists because the ZMap design does not extend to a 128-bit address space unchanged.
  * **Rolling your own has not gone away** — 4 of the 44 papers in the provisional window still do — but it has fallen from a quarter of the earliest bucket to under a tenth.

===== The invocation a reviewer accepts =====

A ZMap command that will survive review looks like this. Every flag is doing work for a reader, not for you.

<code bash>
# ZMap takes exactly ONE --blocklist-file. Build it, and keep the inputs.
cat /etc/zmap/blocklist.conf our-opt-outs.conf > scan-blocklist.conf

zmap \
  --target-ports=443 \
  --blocklist-file=./scan-blocklist.conf \
  --rate=10000 \
  --probes=1 \
  --seed=20260910 \
  --source-ip=198.51.100.10-198.51.100.13 \
  --sender-threads=4 \
  --cooldown-time=8 \
  --output-module=csv \
  --output-fields=saddr,sport,classification,success,timestamp-str \
  --output-file=scan-443.csv \
  --log-file=scan-443.log \
  --metadata-file=scan-443.json
</code>

  * ''%%--rate%%'' / ''%%--bandwidth%%''. ZMap's default is **10,000 pps**; ''%%--rate=0%%'' scans at full line rate.((''src/zmap.1.ronn'' at tag ''v4.4.0''; the defaults there are identical to ''main'' as of 2026-09-10.)) State the number you used. See the next section for what it costs you.
  * ''%%--blocklist-file%%'', built from two inputs. There is **one** such flag — no separate "extra" file — so merging the shipped IANA list with your own opt-out list is your job, and keeping both inputs under version control is what makes the merge auditable. See //The exclusion file// below.
  * ''%%--seed%%''. ZMap walks the address space in a pseudorandom permutation. Recording the seed is what makes "the same scan" mean something: with the same seed and the same blocklist, the target order is reproducible, and ''%%--shards%%'' / ''%%--shard%%'' (which //require// a seed) let you split one scan across machines without overlapping.
  * ''%%--probes%%''. Default **1**. Raising it to 2 or 3 finds hosts that dropped the first packet — and doubles or triples your packet count and therefore your wall clock. {[durumeric2024_years]} reports the better trade: //"the best way to mitigate transient drop is to scan from 2-3 geographically and topologically diverse vantages, rather than to send multiple probes from a single scanner, since both probes are oftentimes lost"//. That makes it a [[Design:Crawling location]] decision as much as a scanner one.
  * ''%%--source-ip%%'' as a range, not a single address, and a range whose reverse DNS you control. See //Source addresses//.
  * ''%%--metadata-file%%''. Writes the run's own parameters and counters as JSON. Ship it with the dataset: it is the cheapest possible answer to "what exactly did you run".

**Most first scans are not Internet-wide, and should not be.** If your targets are a list — the addresses behind a domain sample, one AS, the hosts a previous crawl resolved — pass them with ''%%-w/--allowlist-file%%'' (CIDRs) or ''%%-I/--list-of-ips-file%%'' (individual addresses, for more than about 10 million); the blocklist still applies on top, and everything in this page about rate, exclusions and source addresses applies unchanged. And ZMap's probe module is a choice, not a constant: ''%%--probe-module%%'' does UDP and ICMP echo as well as the default TCP SYN, which changes both what a non-answer means and the packet size the wire-rate arithmetic below assumes.

ZMap tells you an address answered on a port. It does not tell you what is there. The second half is ZGrab2, which //"performs in-depth, follow-up L7 handshakes"// on the hosts ZMap found:((''zmap/zgrab2'' ''README.md'' at tag ''v1.0.0'', read 2026-09-10.))

<code bash>
zmap --target-ports=443 --rate=10000 --output-fields=saddr --output-file=- \
  | ztee found.csv \
  | zgrab2 tls --port=443 --output-file=tls.json
</code>

ZGrab2 reads ''IP, DOMAIN, TAG, PORT'' CSV on stdin, and the ''DOMAIN'' column matters: //"If both IP and DOMAIN are provided, scanners connect to IP but use DOMAIN in protocol-specific contexts, such as the HTTP HOST header and TLS SNI extension."// A TLS scan without SNI and a TLS scan with SNI reach different virtual hosts on the same address, so which column you filled is part of your method, not a detail. For more than one module in a run, ZGrab2 wants a ''.ini'' and the ''multiple'' module rather than a pipeline of processes.

<WRAP tip>
**Your port list is a population choice, and the obvious one is wrong.** Izhikevich et al. {[izhikevich2021_identifying]} scanned beyond the IANA-assigned ports and found that //"only 3% of HTTP and 6% of TLS services run on ports 80 and 443, respectively"//, and that //"services on non-standard ports are more likely to be insecure"//. If your paper says "we measured HTTPS deployment" and you scanned 443, say so in those words — you measured HTTPS deployment on port 443.
</WRAP>

===== Rate limiting, and the wall clock it buys =====

The scan-rate decision is not an ethics footnote; it is the schedule. Below is the real output of ''scan_budget.py'', published in full at the foot of this section, run on 2026-09-10 against ZMap's own shipped exclusion file:

<code>
blocklist            https://raw.githubusercontent.com/zmap/zmap/v4.4.0/conf/blocklist.conf
  CIDR entries       16
  addresses removed  592,708,864  (13.8% of IPv4)
  addresses in scope 3,702,258,432

plan: 1 port(s) x 1 probe(s) per target = 3,702,258,432 packets

    rate (pps)    wall clock     wire rate
--------------  ------------  ------------
        10,000     4.29 days          7 Mb/s
       100,000        10.3 h         67 Mb/s
     1,488,095      41.5 min      1,000 Mb/s
    14,880,952       4.1 min     10,000 Mb/s

Reference rates: ZMap's default --rate is 10,000 pps; 1 GbE line rate is
1,488,095 pps and 10 GbE is 14,880,952 pps at 84 bytes of wire time per probe.

What this file does NOT tell you: whether anyone asked to be excluded.
The shipped blocklist is IANA special-purpose space. Opt-out entries are
yours to add, and to keep across runs.
</code>

Read that as three facts. **ZMap's default rate takes 4.29 days for one port.** A single gigabit link finishes the same scan in **41.5 minutes**. And **ports and probes multiply the wall clock, not the wire rate**: ''%%--ports 2 --probes 2%%'' is four times the packets and therefore four times every figure in the //wall clock// column — **17.14 days** at the default rate — while the //wire rate// column is unchanged, because that depends only on ''%%--rate%%''. A "quick" eight-port sweep is **34.28 days** and still 7 Mb/s. Both figures come out of the same script, with ''%%--ports%%'' and ''%%--probes%%''.

**A cross-check on that arithmetic.** The ZMap paper reports the same scan measured rather than computed: a single machine //"surveying the entire IPv4 address space in under 45 minutes from user space"//, at //"over 97% of the theoretical maximum speed of gigabit Ethernet"//, with //"an estimated 98% coverage of publicly available hosts"// {[durumeric2013_zmap]}. The table's 41.5 minutes is the theoretical maximum for a 1 Gb/s link; 41.5 / 0.97 = 42.8, and the paper's ceiling is 45. The two agree, which is the point of publishing the script rather than a number.((That paper is **not** in this corpus's extraction. The copy read on 2026-09-10 is the authors' own at ''zmap.io/paper.pdf'' (''Last-Modified: 2025-06-22''), extracted with ''pypdf''; all four quotes were matched against it.))

The wire-rate column assumes **84 bytes of wire time per probe** — a bare TCP SYN is 54 bytes, ZMap's default ''synscan'' module sends no TCP options, and the frame is padded to the 60-byte Ethernet minimum, plus FCS, preamble and interframe gap. If your probe module sends a larger packet, say so and rerun the arithmetic; the script takes the assumption as a named constant you can see and change.

**What the literature reports.** Two probes over the 241 ACTIVE papers' full text, both upper bounds: a loose regex for any rate or bandwidth term matches **124 (51.5%)**, and a first-person one requiring "we"/"our" near the term matches **81 (33.6%)**. Reading a pinned 12-paper sample of that second list, **5 of 12** are actually about the authors' own probe rate — the rest are a target's ICMP rate limiting, a censor's throttling, a lab attack rate, or a defence being recommended. So the honest estimate is **about 34 papers, roughly one in seven**, state the rate they scanned at. The ones that do are unambiguous about it: //"a low scanning rate of 10,000 packets per second, total, across all networks"// {[williams2024_6sense]}; //"a low probing rate of 8,000 packets per second, i.e., around 500KB/s from a single vantage point"// {[holzbauer2025_tracking]}.

<file python scan_budget.py>
#!/usr/bin/env python3
"""Size an Internet-wide scan before you run it, and before a reviewer asks.

Three questions this answers, none of which anyone should be estimating in
their head:

  1. How many addresses are actually in scope, once the exclusions are gone?
     ZMap's shipped conf/blocklist.conf is IANA special-purpose space only --
     RFC 1918, loopback, multicast, the documentation ranges. It contains no
     opt-out entries at all, so "we used the default blocklist" is a statement
     about IANA, not about anyone who asked you to stop.

  2. How long does one probe to each of them take at a given rate, and what
     rate does that imply on the wire? Wall-clock is the number that decides
     whether the study is possible; bandwidth is the number your network
     operator will ask about.

  3. What does a port set or a --probes retry do to both? Multiplying the
     target set by ports and probes is where a four-hour scan quietly becomes
     a four-day one.

Stdlib only. `--self-test` runs nine checks including two exact hand-computed
cases, a mutation guard on the CIDR arithmetic, and boundary coverage for the
unit formatter (added after a review pass mutated its thresholds and the
self-test still passed, because nothing called it).

Frame size: a bare TCP SYN is 54 bytes of Ethernet frame -- ZMap's default
synscan module sends no TCP options -- padded up to the 60-byte minimum, plus a
4-byte FCS, plus the 12-byte interframe gap and the 8-byte preamble/SFD: 84
bytes of wire time per probe. The padding is why adding options does not
change the number until they push the frame past 60 bytes. 84 bytes is what
gives 1,488,095 pps on a 1 GbE link, and it is an assumption to restate if your
probe module sends anything bigger.
"""

from __future__ import annotations

import argparse
import ipaddress
import sys
import urllib.request

# ZMap's shipped exclusion file, on the tag you are running. Pinning the tag
# matters: this file is not versioned independently of ZMap.
BLOCKLIST_URL = "https://raw.githubusercontent.com/zmap/zmap/v4.4.0/conf/blocklist.conf"

WIRE_OVERHEAD_BYTES = 84  # 60-byte minimum frame + FCS + preamble/SFD + IFG
IPV4_SPACE = 2 ** 32


def parse_blocklist(text: str) -> list[ipaddress.IPv4Network]:
    """CIDR lines from a ZMap blocklist file. A malformed line is an error."""
    nets = []
    for lineno, raw in enumerate(text.splitlines(), 1):
        line = raw.split("#")[0].strip()
        if not line:
            continue
        try:
            nets.append(ipaddress.ip_network(line))
        except ValueError as exc:
            raise ValueError(f"{BLOCKLIST_URL}:{lineno}: {raw!r}: {exc}") from exc
    return nets


def excluded_addresses(nets: list[ipaddress.IPv4Network]) -> int:
    """Addresses removed, counting overlapping entries once.

    collapse_addresses() is the whole point: the shipped file happens not to
    overlap, but a site blocklist with an opt-out section usually does, and
    summing num_addresses over an overlapping list double-counts silently.
    """
    return sum(n.num_addresses for n in ipaddress.collapse_addresses(nets))


def scan_plan(targets: int, ports: int, probes: int, rate_pps: float) -> dict:
    packets = targets * ports * probes
    seconds = packets / rate_pps
    bits_per_second = rate_pps * WIRE_OVERHEAD_BYTES * 8
    return {
        "targets": targets,
        "ports": ports,
        "probes": probes,
        "packets": packets,
        "rate_pps": rate_pps,
        "seconds": seconds,
        "hours": seconds / 3600,
        "days": seconds / 86400,
        "wire_bps": bits_per_second,
        "wire_mbps": bits_per_second / 1e6,
    }


def line_rate_pps(gigabits: float) -> float:
    return gigabits * 1e9 / (WIRE_OVERHEAD_BYTES * 8)


def human_time(seconds: float) -> str:
    if seconds < 120:
        return f"{seconds:.1f} s"
    if seconds < 7200:
        return f"{seconds / 60:.1f} min"
    if seconds < 172800:
        return f"{seconds / 3600:.1f} h"
    return f"{seconds / 86400:.2f} days"


def fetch_blocklist() -> str:
    with urllib.request.urlopen(BLOCKLIST_URL, timeout=30) as fh:
        return fh.read().decode("utf-8")


def report(ports: int, probes: int, rates: list[float], blocklist_text: str, source: str) -> None:
    nets = parse_blocklist(blocklist_text)
    removed = excluded_addresses(nets)
    targets = IPV4_SPACE - removed
    print(f"blocklist            {source}")
    print(f"  CIDR entries       {len(nets)}")
    print(f"  addresses removed  {removed:,}  ({100 * removed / IPV4_SPACE:.1f}% of IPv4)")
    print(f"  addresses in scope {targets:,}")
    print()
    print(f"plan: {ports} port(s) x {probes} probe(s) per target = {targets * ports * probes:,} packets")
    print()
    print(f"{'rate (pps)':>14}  {'wall clock':>12}  {'wire rate':>12}")
    print(f"{'-' * 14}  {'-' * 12}  {'-' * 12}")
    for rate in rates:
        p = scan_plan(targets, ports, probes, rate)
        print(f"{rate:>14,.0f}  {human_time(p['seconds']):>12}  {p['wire_mbps']:>9,.0f} Mb/s")
    print()
    print("Reference rates: ZMap's default --rate is 10,000 pps; 1 GbE line rate is")
    print(f"{line_rate_pps(1):,.0f} pps and 10 GbE is {line_rate_pps(10):,.0f} pps at {WIRE_OVERHEAD_BYTES} bytes of wire time per probe.")
    print()
    print("What this file does NOT tell you: whether anyone asked to be excluded.")
    print("The shipped blocklist is IANA special-purpose space. Opt-out entries are")
    print("yours to add, and to keep across runs.")


def self_test() -> None:
    checks = 0

    # 1-2. Two hand-computed CIDR cases.
    assert excluded_addresses([ipaddress.ip_network("10.0.0.0/8")]) == 16777216
    checks += 1
    assert excluded_addresses([ipaddress.ip_network("0.0.0.0/0")]) == IPV4_SPACE
    checks += 1

    # 3. Overlap must collapse, not sum. This is the mutation guard: replacing
    #    collapse_addresses() with a bare sum makes this 33554432.
    overlapping = [ipaddress.ip_network("10.0.0.0/8"), ipaddress.ip_network("10.1.0.0/16")]
    assert excluded_addresses(overlapping) == 16777216, "overlapping entries must be counted once"
    checks += 1

    # 4. And a non-overlapping pair must still add up, or "collapse" could be
    #    doing nothing at all and check 3 would still pass.
    disjoint = [ipaddress.ip_network("10.0.0.0/8"), ipaddress.ip_network("192.168.0.0/16")]
    assert excluded_addresses(disjoint) == 16777216 + 65536
    checks += 1

    # 5. A malformed line is an error, not a skipped line.
    try:
        parse_blocklist("10.0.0.0/8\nnot-a-network\n")
    except ValueError:
        pass
    else:  # pragma: no cover
        raise AssertionError("a malformed blocklist line must raise")
    checks += 1

    # 6. Comments and blank lines are not networks.
    assert len(parse_blocklist("# header\n\n10.0.0.0/8  # private\n")) == 1
    checks += 1

    # 7. Wall clock scales with ports and probes.
    one = scan_plan(1_000_000, 1, 1, 10_000)
    four = scan_plan(1_000_000, 2, 2, 10_000)
    assert four["seconds"] == 4 * one["seconds"]
    checks += 1

    # 8. Line rate, to the packet.
    assert round(line_rate_pps(1)) == 1_488_095
    assert round(line_rate_pps(10)) == 14_880_952
    checks += 1

    # 9. human_time() picks the right unit on both sides of every boundary.
    #    Added after a review pass mutated the thresholds from 120/7200/172800
    #    to 60/3600/86400 and the self-test still said OK: nothing called it.
    for seconds, want_unit in [
        (119.9, "s"), (120.0, "min"),
        (7199.0, "min"), (7200.0, "h"),
        (171999.0, "h"), (172800.0, "days"),
    ]:
        got = human_time(seconds)
        assert got.endswith(want_unit), f"human_time({seconds}) = {got!r}, want a value in {want_unit}"
    #    …and that the unit conversion itself is right, not just the bucket.
    assert human_time(3600.0) == "60.0 min"
    assert human_time(86400.0) == "24.0 h"
    assert human_time(172800.0) == "2.00 days"
    checks += 1

    print(f"self-test OK ({checks} checks)")


def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("--ports", type=int, default=1, help="number of ports per target (default 1)")
    ap.add_argument("--probes", type=int, default=1, help="probes per target/port pair (ZMap --probes, default 1)")
    ap.add_argument("--rate", type=float, action="append", help="a send rate in pps; repeatable")
    ap.add_argument("--blocklist-file", help="read a local blocklist instead of fetching ZMap's")
    ap.add_argument("--self-test", action="store_true")
    args = ap.parse_args()

    if args.self_test:
        self_test()
        return 0

    rates = args.rate or [10_000, 100_000, line_rate_pps(1), line_rate_pps(10)]
    if args.blocklist_file:
        with open(args.blocklist_file, encoding="utf-8") as fh:
            text = fh.read()
        source = args.blocklist_file
    else:
        text = fetch_blocklist()
        source = BLOCKLIST_URL
    report(args.ports, args.probes, rates, text, source)
    return 0


if __name__ == "__main__":
    sys.exit(main())
</file>

===== The exclusion file, and where opt-out requests go =====

This is the single most-misread thing about ZMap. **The blocklist it ships is not an opt-out list.**

''conf/blocklist.conf'' at tag v4.4.0 contains **16 CIDR entries** and nothing else: RFC 1918 private space, loopback, link-local, the RFC 5737 documentation ranges, multicast, and the rest of the IANA special-purpose registry. Its header comment is dated **2013**. It removes 592,708,864 addresses — 13.8% of IPv4 — and it removes them because probing them is meaningless, not because anyone asked.

So the exclusion story has two files and you need both:

  - The **shipped IANA list**, so you do not scan 10.0.0.0/8.
  - **Your own list**: every prefix whose operator asked you to stop, plus anything your institution excludes. This file has to outlive the study. An opt-out honoured for one paper and forgotten for the next is worse than none, because the operator now believes they opted out.

ZMap has one ''%%--blocklist-file%%'' flag, so those two go into the same file and you keep the inputs separately. Three helper binaries ship alongside ''zmap'' and are worth knowing: **''ziterate''** //"will produce IPv4 addresses in a psuedorandom [sic] order similar to how ZMap generates random addresses to be scanned"// — run it with your seed and blocklist to see exactly what you are about to probe, without probing it; **''zblocklist''** filters an existing address list through the same rules; and **''ztee''** is the buffer //"used whenever ZMap is piped into an application scanner"//.

{[durumeric2024_years]}'s own recommendation is explicit about both halves: //"Provide a simple mechanism for operators to request exclusion from future scans. Indicate the IP ranges you use for scanning so that operators can drop research traffic themselves."//

**How rare is this?** The loose blocklist probe fires on **103 of 241 ACTIVE papers (42.7%)** and the first-person one on **88 (36.5%)**, but hand-reading an 11-paper sample of the latter gives **6 of 11** — the other five are a censor's blocklist, a threat-intelligence feed used as input data, or an analysis-time exclusion. Estimate: **about 48 papers, roughly one in five**. The opt-out probe is tighter and cleaner: **70 papers (29.0%)** match a first-person opt-out pattern and **9 of a 10-paper sample** are genuine, giving **about 63 papers, one in four**.

Two figures worth carrying. Durumeric et al. {[durumeric2014_view]} report that after years of running the largest academic scanning operation of its day, //"only 208 organizations have requested that we exclude their networks from our scans, reducing the IPv4 address space for study by only 0.15%"//. And Beverly et al. {[beverly2018_beholder]}, on a run of IPv6 topology campaigns between February and May 2018: //"we received two opt-out requests with which we immediately complied"//. The opt-out volume is small. That is an argument for offering it, not against.

The **norms** around this — what your NREN requires, whether an ethics board will look at it, what operators say they will tolerate — are on [[Practices:Ethics]], which quotes SURF's and Jisc's actual policy text and Hantke et al.'s operator interviews. Do not re-derive them here.

===== Source addresses and abuse contacts =====

The address you scan from is the only channel a scanned operator has to reach you, and the only evidence they have that you are not an attacker. The convention the ZMap-era methods papers set — and, on the estimate below, about a quarter of scanning papers follow — is four things on the scanning host itself:

  * A **PTR (reverse DNS) record** that says what it is, on every address in the ''%%--source-ip%%'' range. Not ''host-10.example.edu''.
  * A **WHOIS / RDAP entry** for the range with a working abuse contact.
  * A **web server on port 80** serving one page: who is scanning, why, what is collected, how to opt out.
  * A **stable, dedicated range** — so a repeat scan comes from the same place and an operator's own blocklist keeps working.

{[durumeric2015_search]} describes exactly this for Censys' own infrastructure: //"All of the scanning hosts have WHOIS records and reverse DNS entries that describe the intent of the scanning. Further, each scanning host runs a simple website on port 80 that describes the goals of the research"//. {[kaspereit2024_landscape]} adds the people almost nothing in the corpus mentions — your own operators: //"On the scanning IPs, a website gives project details and contact information for opt-out requests. We honored all such requests. Our ISP, national CERT, and university network administrator cleared all scans."// Telling your own network operations team **before** the scan is the difference between a conversation and a ticket.

**In the corpus**: the loose source-address probe matches **167 of 241 (69.3%)** and the first-person one **112 (46.5%)**, with **6 of a 10-paper sample** genuine — about **67 papers, a little over a quarter**, describe any of this. Abuse-complaint handling is discussed by **45 (18.7%)** on the first-person probe (upper bound; no precision measured). Two more probes, both loose and neither hand-audited, so read them as upper bounds and not against each other: **55 (22.8%)** match an IRB / ethics-board pattern and **172 (71.4%)** cite Durumeric, Partridge and Allman, the Menlo Report, or the phrase "good Internet citizen". Do **not** read the 55 as "only 55 had ethics review": the extraction's own ''ethics.reviewOutcome'' field is stated on **92 (38.2%)** of the same 241 papers, and that is the number to quote.

<WRAP tip>
**Scanning from a cloud VM is a vantage-point decision, not a convenience.** The provider's acceptable-use policy governs it, the address has the provider's reverse DNS and not yours, and {[durumeric2024_years]} found that //"none of the top 100 ASes that emit the most ZMap traffic belong to universities; rather, most traffic originates from security companies and cloud providers"// — so a target that filters cloud scan traffic filters you along with everyone else. [[Design:Crawling location]] is the page for what a vantage point does to a result.
</WRAP>

===== IPv6: the address space is not enumerable =====

Everything above assumes you can walk the target space. In IPv6 you cannot, and no rate will fix it: 2^128 addresses against 2^32, so the four-day full scan in the table becomes longer than the universe has existed. **There is no IPv6 equivalent of "scan the whole Internet."** What exists instead is three techniques, and each is a denominator problem.

  - **Hitlists.** A published list of addresses already known to respond. The TUM **IPv6 Hitlist Service** {[gasser2018_clusters]} is the de-facto standard and is live: its open snapshot of **2026-09-05** carries **29,050,771 responsive addresses**, alongside **24,476,266 aliased prefixes** and **222,836,153 non-aliased prefixes**.((Downloaded from ''alcatraz.net.in.tum.de/ipv6-hitlist-service/open/'' on 2026-09-10; line counts of the decompressed files. A registration-first tier adds daily input addresses and raw ZMap output.)) Twenty-nine million is a large list and it is not a sample of the IPv6 Internet — it is a sample of what previous scans found.
  - **Aliased-prefix detection.** A single machine can answer for an entire prefix, so a naive responsive-address count is inflated by orders of magnitude. {[gasser2018_clusters]} developed detection that //"identifies 1.5 % of our prefixes as aliased, pertaining to about half of our target addresses"//. And the de-aliasing step has a cost most papers do not report: {[zirngibl2022_rusty]} found that //"More than 98 % of all IPv6 addresses announced by Fastly were labeled as aliased and Cloudflare prefixes hosting more than 10 M domains were excluded"//. Removing aliased prefixes removes the CDNs. Whether that is right depends entirely on your question.
  - **Target generation.** Learn the structure of allocated address plans and generate plausible candidates: Entropy/IP, 6Gen {[murdock2017_target]}, 6Graph, 6Tree, and most recently 6Sense {[williams2024_6sense]}. These produce candidates, not a frame, so a hit rate is a property of the generator as much as of the Internet.

**How thin is this in the corpus?** Thin, and the thinness is the finding.

^ Query ^ Population ^ Papers ^ Share ^
| mentions IPv6 anywhere | SCAN 930 | 281 | 30.2% |
| mentions IPv6 anywhere | ACTIVE 241 | 107 | 44.4% |
| names an IPv6-targeting instrument in ''tools[]'' | SCAN 930 | **11** | 1.2% |
| full text says "hitlist" / "hit list" | SCAN 930 | 62 | 6.7% |
| "hitlist" within 80 characters of "IPv6" | SCAN 930 | **39** | 4.2% |
| names a target-generation algorithm | SCAN 930 | 20 | 2.2% |
| discusses aliased prefixes or dealiasing | SCAN 930 | **10** | 1.1% |

Almost a third of scan papers mention IPv6. The aliasing row needs its own denominator, though: 10 of 930 is 1.1%, but most of those 930 never touch IPv6. Against the **42** papers that either name an IPv6-targeting instrument or say "hitlist" near "IPv6", it is **8 of 42 (19.0%)** — still four in five of the papers doing IPv6 targeting not discussing the thing that decides whether their counts mean anything, and a far more defensible way to say it. The eleven papers naming a targeting instrument are, in full: the IPv6 Hitlist Service (4), 6Gen (3), Entropy/IP (2), addr6 (2), 6Graph, 6Tree and 6Sense (1 each) — and every one of those eleven is IMC, NDSS, PETS or USENIX, none earlier than 2016. IPv6 mention inside SCAN is up over the window, but not monotonically: 12.5% in 2010, 0% in 2013, 35.2% in 2018, 29.2% in 2024, 44.3% in 2025 and 26.1% in the provisional 2026.

If you are scanning IPv6, the sentence a reviewer wants is: **which hitlist or generator, which snapshot date, and whether you de-aliased.** Four in five of the papers that look like they are doing this do not write the de-aliasing half.

===== From scan output to a claim =====

ZMap hands you a column of source addresses. That is not a result, and the step that turns it into one is where scanning papers most often overreach.

  * **An address is not an organisation.** The [[Design:IP classification]] page exists for the join — ASN, geolocation, network type, hosting vs eyeball, reputation. **75 of the 241 ACTIVE papers (31.1%)** carry a ''classification'' tuple whose target is ''ip-address'', against 22.9% of SCAN and 5.0% of the corpus; the raw resource names they reach for are 110 distinct free-text strings, led by MaxMind and IPinfo. Fold them before you count anything — that page publishes the fold.
  * **An address is not a name.** Reverse DNS is not injective and is often absent; a certificate's SAN list, a Host header or a hitlist's provenance are usually better joins than a PTR.
  * **A responding address is not a distinct machine.** In IPv6 this is aliasing, above. In IPv4 it is anycast, load balancers and CDNs: the same content answers from thousands of addresses, and "N hosts run X" quietly becomes "N addresses answered".
  * **Absence of a response is three different things** — filtered, rate-limited, down — and only your own experiment can separate them. {[durumeric2024_years]} notes that scan loss //"is typically driven by a handful of small service and cloud providers"//, so a per-AS breakdown of non-response is worth more than a single completion rate.
  * **The scanning population itself moves under you.** Griffioen et al. {[griffioen2024_have]}, from ten years of a network telescope, report that //"Internet scanning has increased 30-fold over the last ten years"// — 11 million scan packets a day in 2015 to 345 million in 2024 — and that //"the ecosystem is extremely volatile, where targeted ports and geographical scanner locations drastically change at the level of weeks or months"//. Any claim about who scans, or about background radiation, is dated the day you make it.

===== What to report =====

The methods paragraph a reviewer should be able to find, and how often the 241 ACTIVE papers actually contain each part. Schema fields are counted from the extraction; probe rows are the hand-corrected estimates from the sections above and are marked.

^ What ^ ACTIVE (241) ^ SCAN (930) ^
| The **scanner and its version** — version of the scanner itself, not of some tool | **20 (8.3%)** | 20 (2.2%) |
| Any used-or-produced tool version at all | 115 (47.7%) | 401 (43.1%) |
| **Vantage location** you scanned from | 134 (55.6%) | 487 (52.4%) |
| Vantage infrastructure (cloud, university, residential) | 164 (68.0%) | 612 (65.8%) |
| **Probe rate** //(probe estimate)// | ~34 (~14%) | — |
| **Exclusion list honoured** //(probe estimate)// | ~48 (~20%) | — |
| **Opt-out offered** //(probe estimate)// | ~63 (~26%) | — |
| **Source-address hygiene** //(probe estimate)// | ~67 (~28%) | — |
| Ethics review outcome | 92 (38.2%) | 290 (31.2%) |
| Notified affected parties | 184 (76.3%) | 560 (60.2%) |
| Harm mitigation described | 208 (86.3%) | 700 (75.3%) |
| Measurement start date | 192 (79.7%) | 656 (70.5%) |
| Own artifact URL | 144 (59.8%) | 506 (54.4%) |

**Eight per cent name the version of the scanner they ran.** That is the worst row on the table and the easiest to fix: ZMap's behaviour has changed materially inside the version range this corpus covers — {[durumeric2024_years]} records that //"In early 2024, ZMap changed its default behavior to use random per-probe IP IDs"//, which is exactly the kind of change that moves an IP-ID side-channel result — and ''%%zmap --version%%'' costs nothing. Also add: your **port set**, your **seed**, your **probes-per-target**, the **wall-clock span** (''temporal.spanStart'' is missing on a fifth of ACTIVE papers), and, for IPv6, **which hitlist snapshot and whether you de-aliased**.

Do not put the ethics narrative here. Put it where [[Practices:Ethics]] says to put it, and make this paragraph the reproducible half.

===== What to read first =====

^ Paper ^ Why ^
| Durumeric, Wustrow and Halderman, USENIX Security 2013, //ZMap: Fast Internet-Wide Scanning and Its Security Applications// {[durumeric2013_zmap]} | The instrument paper. **Not in this extraction** — cited from the USENIX page. |
| Durumeric et al., IMC 2024, //Ten Years of ZMap// {[durumeric2024_years]} | What changed, what the authors would do differently, the five-point practice list, and the 33% traffic-share figure. The single most useful paper on this page. |
| Durumeric, Bailey and Halderman, USENIX Security 2014, //An Internet-Wide View of Internet-Wide Scanning// {[durumeric2014_view]} | What being scanned looks like from the other side, and how few operators ever ask you to stop. |
| Griffioen et al., IMC 2024, //Have you SYN me?// {[griffioen2024_have]} | Ten years of telescope data: the scanning population you join is volatile and 30x larger than it was. |
| Izhikevich, Teixeira and Durumeric, USENIX Security 2021, //LZR: Identifying Unexpected Internet Services// {[izhikevich2021_identifying]} | Why your port list is a population choice. 3% of HTTP is on port 80. |
| Gasser et al., IMC 2018, //Clusters in the Expanse// {[gasser2018_clusters]} | The IPv6 hitlist and aliased-prefix detection — read before any IPv6 scan. |
| Zirngibl et al., IMC 2022, //Rusty Clusters?// {[zirngibl2022_rusty]} | What de-aliasing removes: 98% of Fastly, and Cloudflare prefixes hosting 10M domains. |
| Williams et al., USENIX Security 2024, //6Sense// {[williams2024_6sense]} | Current IPv6 target generation, with a methods section that states its rate and its opt-out. |
| Durumeric et al., CCS 2015, //A Search Engine Backed by Internet-Wide Scanning// {[durumeric2015_search]} | Censys: the scan you query instead of running, and a model description of scanner-host hygiene. |
| Wu et al., NDSS 2025, //Revealing the Black Box of Device Search Engine// {[wu2025_revealing]} | What commercial scanning does that academic scanning should not. Also on [[Practices:Ethics]]. |

Leonard and Loguinov's IRLscanner {[leonard2010_demystifying]} is the pre-ZMap ancestor, and its complaint analysis — what people actually wrote back, and about what — is the most detailed one this page found. Kaspereit et al. {[kaspereit2024_landscape]} is a recent methods section to imitate.

===== Someone else's scan is often the right answer =====

The corpus names scan **datasets** in 97 of 930 SCAN papers. Running your own scan when a current public one answers the question is work you did not need to do — {[durumeric2024_years]}'s first recommendation is blunt: //"Consider whether existing open source datasets provide the data you need."//. But the access terms move, so check before you plan around one:

  * **Censys** {[durumeric2015_search]} — ''censys.com'' answers, but ''search.censys.io'' returns **HTTP 403** to automated clients including with a browser User-Agent. That is a bot wall, not an outage; the API is the supported route and is tiered.
  * **Rapid7 Project Sonar** — **the free Open Data programme is gone.** ''opendata.rapid7.com'' now redirects to ''sonardata.rapid7.com'', which says it //"provides commercial access to data from Project Sonar"//, signs in existing accounts only, and directs new users to a commercial contact. The data itself is still being produced (8 datasets, 47,457 files, 60.7 TB, most recent files dated within the last week).((Read 2026-09-10 with a browser User-Agent. A corpus paper from 2021 uses "Rapid7 Labs Open Data" for full-IPv4 liveness checks; that route no longer exists as described.))
  * **The IPv6 Hitlist Service** is the exception that is still openly downloadable, above.
  * **Shodan, FOFA, ZoomEye, GreyNoise, LeakIX** appear in the corpus as query targets. Each is a commercial index with its own crawl policy, refresh interval and coverage — which means its denominator is not yours and is not published. [[Design:Existing datasets]] is the page for that problem in general.

===== Methodology and limitations of these figures =====

Every number above is a count of **papers**, from the 5,859-paper extraction, with its denominator in the same sentence or table header. Sentinels (''not-stated'', ''none-mentioned'') are never counted as answers, and an absent ''ethics'' or ''artifacts'' object is counted as neither (7 and 3 ACTIVE papers). Free-text instrument names are folded before counting: the fold is ''scripts/scan_fold.mjs'', it decides **741** used-or-produced ''network-scanner'' tuples with **zero** unmapped strings, and its 173 hand verdicts and full residue are published on the provenance page. ZMap alone appears under **10** spellings.

Four limitations to carry:

  * **ACTIVE is a floor.** A paper that scanned and named its scanner something unique outside ''tools[].category == "network-scanner"'' is not counted. The over-count risk is guarded rather than assumed: the ''ZMap'' regex ''%%/z-?map/i%%'' matches //WizMap//, an unrelated visualisation tool that appears in one paper, and the fold names and excludes it explicitly.
  * **The four practice rows are probe estimates on samples of 10–12.** Read "about a fifth", not "19.9%". The probes, the samples, the per-paper verdicts and the deciding sentence for each are on the provenance page.
  * **''studyTypes'' is the least stable field in the schema** (57% run-to-run on a 100-paper sample, not re-measured on this corpus). SCAN = 930 is a ranking-grade number, so treat every //share of 930// as approximate. ACTIVE does not depend on it.
  * **The hand verdicts have been read once, by one reader.** All 173 were audited against each tool's ''purpose'' text and its paper's title and full text on 2026-09-17; five kinds changed, three of them taking a paper out of ACTIVE and one putting one in, and five tuples across two papers were suppressed — four because a network-telescope paper fingerprinted the scanners rather than running them, one because a ZMap run never left its own machine. ACTIVE moved from 245 to **241**. Every change, with its deciding sentence, is on the provenance page, and so is the review pass that checked them: it confirmed all five kind changes and all six borderlines against the papers, and found the fifth suppression, which this run had missed.
  * **2025–2026 venue-years are provisional** and the corpus is seven venues. Network-measurement work also appears at PAM, TMA, ANRW, CoNEXT and ACSAC, **none of which are in this corpus** — for scanning specifically that is a bigger gap than for the web pages on this site, and several instrument papers this literature leans on are published there. Do not take that as a claim about any specific tool above — Yarrp's own paper, for one, is IMC 2016 and //is// in this corpus.

The report script is ''scripts/report_internet_scanning.mjs''. It exits 1 if the corpus size, the SCAN population, the fold coverage, a stale hand verdict, or a hand-read sample that has drifted disagrees with the contracts it encodes. Full query log, folds, residue, quote checks, rejected sources and the review log: [[provenance:programming:internet_scanning]]. Corpus-level caveats: [[Literature:Corpus]].

===== Related pages =====

  * [[Design:Automated measurements]] — crawl vs scan vs app analysis, and how big each branch is.
  * [[Practices:Ethics]] — the scanning norms, NREN policies, and the operator interviews. Read it before you run anything.
  * [[Practices:Notifying websites]] — telling the operator what you found.
  * [[Design:IP classification]] — turning the addresses you collected into a claim.
  * [[Design:DNS]] — the scan branch when the question is names. ZDNS is the ZMap-family tool there.
  * [[Security:TLS certificates]] — the scan branch when the question is the web PKI.
  * [[Security:Email authentication]] — a scan-shaped measurement over DNS records rather than ports.
  * [[Design:Crawling location]] — the vantage point, which for a scan is also your source address.
  * [[Design:Existing datasets]] — querying someone else's scan.
  * [[Programming:Crawler]] — the other branch's instrument page.
  * [[Literature:Corpus]] — venue scope, funnel, provisional years.

<bibtex bibliography></bibtex>

~~DISCUSSION~~