Back to Internet scanning. Corpus-wide selection and extraction notes are on corpus. This is the page-specific query log: every query with its population, the fold and its residue, the probes and the hand verdicts behind them, the external sources and the ones rejected, and what could not be established.

No ~~DISCUSSION~~ block here. Comments belong on the content page; this is a working log.

• Run date: 2026-09-10 (UTC). Single sitting.
• Authoring agent: Claude Opus 5 (Claude Code), working the wiki-measuretheweb item “scanning has no instrument page”, section A2 of the 2026-09-02 brainstorm pass.
• Corpus at run time: 5,859 extracted papers, 2010–2026, CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P. Read-only inputs under /workspace/publications_dataset/data/. No write to the publication mount.
• Creating, not extending. Before writing: node scripts/sitemap.mjs (168 existing pages, 7 promised-but-missing, all 7 declared on roadmap), then exported and read automated_measurements, design, programming, ip_classification, dns, ethics, email_authentication, crawler and roadmap.
• Wiki saves through scripts/dw.mjs (JSON-RPC) with –if-rev on every existing page.
• Pages saved this sitting: literature:bibliography (rev 1789063332 → 1789074254, 8 entries appended), programming:internet_scanning (new), roadmap (1789062297 →), programming (1788006004 →), design:automated_measurements (1787943845 →), security:email_authentication (1788954595 →), and this page.

A follow-up run on 2026-09-17 worked the three items the 2026-09-10 review left open. It changed the page's population from 245 to 241 and refreshed about forty figures. Everything below that carries a number is the state after that run; the run itself is written up in Follow-up run, 2026-09-17, which also says which of the three items is still open.

• Run date: 2026-09-17 (UTC). Single sitting. Authoring agent: Claude Opus 5 (Claude Code), item “internet_scanning: three follow-ups the review left open”.
• Corpus unchanged: 5,859 extracted papers. No re-extraction, no new venue-years.
• Pages saved: programming:internet_scanning, design:dns, roadmap, this page and provenance:design:dns. Each was saved twice: once at ACTIVE=242, then again at 241 after the review passes returned. No bibliography change — no new citekeys.
• Scripts added: scripts/reconcile_zmap.mjs, scripts/tool_use_overrides.mjs, scripts/audit_hand_verdicts.mjs. Scripts changed: scripts/scan_fold.mjs, scripts/dns_fold.mjs, scripts/report_internet_scanning.mjs, scripts/report_dns.mjs, scripts/verify_scanning_figures.mjs. All are published in full at the foot of this page or of dns.

Three items were left open by the 2026-09-10 review. Two are closed; the third is not.

The reviewer's hypothesis was an overlap artefact — a paper naming both ZMap and XMap, counted once on one page and twice on the other. It is not. scripts/reconcile_zmap.mjs computes both paper sets from the same extraction and prints the symmetric difference:

The one paper is CCS 2016, A Surfeit of SSH Cipher Suites. Its tools[] entry is the single string ZGrab/ZMap, with evidence.quote “We ran two IPv4-address-space-wide scans for SSH servers using ZGrab/ZMap [13].” — matched verbatim in data/fulltext/2016/CCS/a-surfeit-of-ssh-cipher-suites/paper.cols.txt. The paper used both tools. Both folds are ordered first-match-wins and single-valued, so a compound string can only land in one family, and the two folds happen to test in opposite orders: dns_fold.mjs tests ZMap / XMap before ZGrab, scan_fold.mjs tests ZGrab before ZMap. Same string, two families, one paper of disagreement.

Neither page was right. Both folds now carry an identical, hand-read COMPOUND_SPLIT map that splits this one string into its two components, each folded through the normal path:

• scan_fold.mjs: ZMap 95 → 96, ZGrab 32 (already had it).
• dns_fold.mjs: ZMap / XMap 112 (unchanged), ZGrab 31 → 32.

After that, and after the tuple suppression in item B below, the two pages agree exactly: ZMap ∪ XMap = 111 on both, ZGrab = 32 on both, symmetric difference 0.

The recall question — how many other compound names are there? A blanket split would be wrong (miekg/dns is a GitHub path, HTTP/OneProbe is a product name), so the sweep is a guard rather than a rule. It looks at every used-or-produced tools[] name containing /, '' + '', '' and '', '' & '' or a comma — 263 tuples — and reports each one whose parts fold to a family the whole string does not. There are four, and only the ZMap one touches a published figure:

reconcile_zmap.mjs exits 1 if a new compound name appears whose lost part is a membership kind (this page) or a DNS_KIND (design:dns). Mutation-tested: removing the zgrab/zmap entry from both maps produced three failures, including “unsplit compound \”ZGrab/ZMap\“ loses membership kind ZMap/active-scan on the scanning page”.

The HAND map in scan_fold.mjs is one verdict per distinct category == “network-scanner” string the FAMILIES list does not fold. The coverage and staleness guards prove every string has a verdict and every verdict matches a string; they prove nothing about whether the kind is right. A wrong kind on a one-off scanner name moves ACTIVE by one paper and nothing fires.

scripts/audit_hand_verdicts.mjs dumps every entry with the evidence needed to judge it — the tool's purpose, the extractor's evidence.quote, the category and the paper's own title — and its full output is published at the foot of this page. All 174 entries were read on 2026-09-17 (the map is 173 afterwards, because one entry was removed). Where the name and purpose did not settle it, the paper's paper.cols.txt was read as well.

Five verdicts changed. Each is annotated inline in the fold source with the sentence that decided it.

Net from the five kind changes: 245 → 242, and one correction in each direction rather than all one way. A sixth change, found by the review pass and not by this audit, took it to 241; it is in the review log below and in the suppression section that follows.

Reading Unicorn exposed a defect the HAND map cannot fix, because it is not about the name. IMC 2024 Have you SYN me? is a telescope study of other people's scanning, and the extraction records ZMap, NMap, Masscan and Unicorn as used because the paper's sentences name them in the first person. The extraction's own purpose fields give it away — “Fingerprint ZMap-generated scanning traffic”, “Fingerprint NMap-generated scanning traffic”, “Test Unicorn packet fingerprints”. A name fold cannot see the difference: the string is ZMap either way.

Three of those four are families, not HAND entries, so the correction has to be per (paper, tool). scripts/tool_use_overrides.mjs holds it, and is imported by both report scripts and by the reconciler, because “the paper did not run this” is a fact about the tuple and not about either page.

The rule, and an honest account of how it changed mid-run:

• (a) Suppress when the paper did not run the tool — it fingerprinted it, or read someone else's output. Written before the first entry.
• (b) Suppress when the packets never left the host onto the public Internet — a closed local loop or a synthetic load generator. Added on 2026-09-17 after the review pass read Spoki; clause (a) alone would have kept it, and one published figure was wrong by one paper because the rule was incomplete.
• Do not suppress when the paper ran the tool and the packets did cross the Internet, even if aimed somewhere unusual — its own honeypot, its own hosts, a cloud VM it controls. “The target was ours” is a different claim from “no packet was sent”, and relitigating the former per paper has no stopping rule.

The recall probe for this defect. A title/slug sweep for /telescope|darknet|darkspace|network tele|scanning traffic|characteriz\w* .*scanning|honeypot|backscatter|unsolicited traffic|scan traffic/i returns 20 papers, of which 4 are in ACTIVE. All four were read:

That is a bounded sweep over a title pattern, not over the population, so it is a floor. A paper that fingerprints scanners without saying so in its title would be missed.

These are the judgement calls a second reader would most plausibly decide the other way. All five are load-bearing — each one alone puts its paper in ACTIVE — and all five were kept.

• It read 174 strings, not 174 papers. Where the name and purpose were unambiguous — iperf → perf, Airmon-ng → local-link, the eleven web-application scanners from one 2023 paper → vuln-scanner — no paper was opened. Roughly 20 of the 174 needed the full text.
• One reader, one pass. No second opinion was sought on any of the five changes or the five borderlines.
• It cannot reach the families. ZMap, nmap, Masscan and the other 60-odd FAMILIES entries are matched by regex with no per-paper reading at all. The telescope sweep above is the only check that has ever been run on them, and it is a title pattern.

Two sonnet passes, both handed the page text, the report script, its unedited output, the reconciler output and the 173-entry evidence sheet, and both told explicitly that the run's context might not be exhaustive. No fable generic pass was run — the item said not to re-run it — and that remains a gap.

What the second pass cost to ignore would have been high, and one of its findings is about this page lying about itself. Both passes are recorded in full because the rejections matter as much as the fixes: two findings (TorFlow, the Nuclei/vuln-scanner tension) were deliberately not acted on, and the reasons are above.

Not re-reviewed. The figures pass ran against the 242-state snapshot. Accepting its findings and the Spoki finding together moved ACTIVE to 241 and changed roughly forty cells again, and that final state has not been through a figures pass. It has been through check_page_numbers.mjs –code on all four pages, every report and verify script, and the reconciler — but a fresh adversarial read of the 241 state is the obvious next thing to do.

No review sub-agents were run on 2026-09-17. The item said explicitly not to re-run the generic pass, and the run changed figures rather than prose. That is a gap: the figure-vs-script check that a sonnet reviewer would do was done by check_page_numbers.mjs and by re-reading the diff, both by the same agent that made the change.

Counts are papers, never tuples. Sentinels are never answers. An absent ethics or artifacts object is counted as neither a stated value nor a sentinel and is printed as its own row.

Poster sensitivity. Dropping the 6 short records moves “vantage location stated” from 55.6% to 56.2% — 0.6 points. The corpus-wide poster caveat (posters have methods compressed out and bias every silence figure upward) applies, but ACTIVE is 1.7% posters against SCAN's 3.7%, so it bites less here than on most pages.

scripts/scan_fold.mjs. The written boundary is at the top of the file and was fixed before the first count.

• Coverage is a contract, not an aspiration. Every category == “network-scanner” used-or-produced string must resolve to a family or to a HAND verdict. 741 tuples, 0 unmapped, 0 stale HAND entries. The report exits 1 in either direction — an unmapped string and a hand verdict that nothing matches, because a shadowed verdict is a lie about what was read.
• Kinds, not one ranking. Twenty kinds. Five are membership (active-scan, grabber, reachability, censorship-probe, ipv6-targeting); the rest are reported beside ACTIVE and routed elsewhere (dataset, topology, dns, vantage-platform, routing, vuln-scanner, passive, local-link, perf, simulation, p2p-crawler, name-enumeration, packet-crafting, registry, lab-protocol-test, host-tool).
• One negative guard. /z-?map/i matches WizMap, an embedding-visualisation tool that appears in one paper under category: other. Without the guard it would have entered ACTIVE as a ZMap use. This is the \b-does-not-protect-an-acronym trap in a different costume, and it was found only by printing the strings that matched outside network-scanner.
• Generic families. custom scanner and ping are flagged generic and only fire inside category == “network-scanner”. Measured cost of not doing this: ACTIVE went from 245 to 708, with custom scanner alone at 464 papers. (Measured on 2026-09-10, when ACTIVE was 245; not re-measured.)
• ZMap appears under 9 spellings in ACTIVE: TCP Options module for ZMap, ZMap, ZMap-based script, ZMap6, ZMapv6, Zmap, zmap, zmap2, zmap6. nmap has 10, ZGrab 7, TLS banner grabbers 8, and the generic “custom” strings 21.
• Self-tests: 74 cases plus 6 structural assertions, including both directions of the generic/category rule and the WizMap guard. The module throws at import if any fails, so every script that imports it inherits the check.

The full residue — which is empty — and the complete 173-entry hand map with its kinds are in the fold source at the foot of this page. Every one of the 173 was read against its tool's purpose text, its paper's title and, where those did not settle it, the paper's full text on 2026-09-17; see B. Hand audit of the 173 HAND verdicts.

All probes read paper.cols.txt with whitespace collapsed (a PDF line break inside a phrase otherwise silently undercounts). Three of the 930 SCAN papers have no paper.cols.txt; the report prints the missing count for every probe.

Each practice gets two widths. The loose regex is the recall claim; the first-person regex requires “we”/“our” near the term. Neither is a population, and the width decides the claim — publishing either alone would have been wrong in a different direction.

Samples were every k-th paper of the sorted hit list. That made the sample a function of ACTIVE, and on 2026-09-17 a three-paper change to ACTIVE reshuffled all four strides and failed 46 of these checks on papers that were still perfectly good hits — the reads were invalidated by arithmetic, not by evidence. The sample is now the verdict map itself: a pinned list of papers, each with the sentence that decided it, drawn originally as every k-th of a hit list of the length recorded in the script. Three guards keep that honest, and all three were mutation-tested on 2026-09-17:

own = the matched sentence is about the authors' own scan. other = the term is in the paper for a different reason.

scripts/verify_scanning_figures.mjs looks up every quoted sentence and per-paper figure on the content page in the cited paper's paper.cols.txt with whitespace collapsed, falling back to a pypdf re-extraction of paper.pdf. 30 needles, 0 not located, 0 weak needles under 20 characters. 26 are in-corpus and all 26 resolved in .cols, so the PDF fallback never had to fire for those. The other 4 are from [1Durumeric, Zakir; Wustrow, Eric; Halderman, J. Alex (2013): "ZMap: Fast Internet-wide Scanning and Its Security Applications", in: Proceedings of the USENIX Security Symposium. (Link)], which is not in the extraction: they are checked against out/scan/zmap2013_paper.txt (72,192 bytes), a pypdf extraction of the authors' own copy at zmap.io/paper.pdf. That check needs its own de-hyphenation step — pypdf keeps the printed hyphen at a line break, so accu- racy and theoreti- cal are two words in the extraction and one in the PDF, and two of the four quotes failed until it was added.

Two things that went wrong and were fixed:

• A misread digit. A first pass read the Durumeric et al. USENIX Security 2014 sentence off a 180-character context window as “908 organizations have requested that we exclude their networks”. The paper says 208, in both paper.cols.txt and an independent pypdf extraction of paper.pdf. The wrong number never reached the page, but it did reach a hand-verdict comment in the report script and had to be corrected there. A narrow context window is not a source.
• Two quotes failed on capitalisation. “Consider whether existing open source datasets…” and “In early 2024, ZMap changed…” are sentence-initial in the paper, and the draft had lower-cased them to fit mid-sentence. The page now quotes them with the source's capitalisation. A quote check that had been case-insensitive would have passed both silently.

pages/scan_budget.py is embedded in the page as a <file python> block. Verified after saving:

• The embedded block is byte-identical to pages/scan_budget.py.
• Downloading it back off the rendered page (/_export/code/programming:internet_scanning?codeblock=3) and diffing gives one difference, a stripped trailing newline. The downloaded copy runs its own –self-test: 9 checks, OK.
• The <code> block beside it is the whole output of a real run, compared line-by-line for exact equality — not a containment check, which passes on a truncated block.
• The self-test has a mutation guard: replacing collapse_addresses() with a bare sum makes the overlapping-CIDR case return 33,554,432 instead of 16,777,216 and check 3 fails. Check 4 exists so that “collapse does nothing at all” cannot also pass — a disjoint pair must still add up.
• The collapse is doing real work on the real file: 255.255.255.255/32 is inside 240.0.0.0/4, so a naive sum reports 592,708,865 removed against the correct 592,708,864.

scripts/external_checks_internet_scanning.sh re-fetches every non-corpus fact and prints FAILED explicitly per check. Its unedited output is at the foot of this page. Honours GH_TOKEN via ${GH_TOKEN:+x} — see the exposure note above for the form that leaks.

• How many scan papers actually rate-limited, honoured an exclusion list, or offered an opt-out. Still open after 2026-09-17. Only estimates, from a regex probe with a hand-read precision sample of 10–12. Closing this needs the same treatment harm_fold.mjs got on ethics: a loose candidate sweep plus a per-paper verdict over the whole hit list. The current hit lists are 81 / 88 / 70 / 112 papers — 351 reads, not the ~90 per practice the original note estimated — and the report now says so in its own output. What the 2026-09-17 run did do is make the existing sample survive a population change instead of dissolving into it; the four estimates are still estimates.
• Whether the 534 SCAN papers that name no network-scanner tool scanned without saying with what, or queried a dataset. The schema cannot separate those two, and neither can a name fold. A full-text pass over a sample would.
• The recall of ACTIVE outside category == “network-scanner”. One-off scanner names filed under other are not hand-read — that would be the whole 16,737-name tool vocabulary. ACTIVE is a floor and the page says so.
• Venue coverage. PAM, TMA, ANRW, CoNEXT and ACSAC are not in this corpus, and for network scanning that is a larger gap than for the web-measurement pages: Yarrp, MIDAR, scamper and several IPv6 target generators have their tool papers at those venues. Any “the literature does X” sentence on this page should be read as “these seven venues do X”. Nothing here can fix that; it needs the corpus extended.
• Whether ZMap's 33% traffic-share figure still holds. It is [2Durumeric, Zakir; Adrian, David; Stephens, Phillip; Wustrow, Eric; Halderman, J. Alex (2024): "Ten Years of ZMap", in: Proceedings of the ACM Internet Measurement Conference. (DOI)]'s Q1 2024 measurement of scan traffic seen at one telescope, not a corpus figure and not a 2026 figure. No newer public measurement was found.
• Whether the roadmap's original “143” was ever right. It re-derives to 141 in SCAN / 145 corpus-wide under this fold and 136/137 under am_fold.mjs. The difference is entirely in how loosely the nmap regex is anchored. No reconstruction of the original query exists.

Four passes, all told explicitly that the author's context might not be exhaustive, and all handed the page text, the report script and its unedited output, the fold, the published code and the provenance notes. The three focused passes were spawned in parallel against the first published revision (programming:internet_scanning rev 1789074265); the generic pass was spawned afterwards, against the corrected page, so it saw a different snapshot. That matters for reading their findings and is recorded here rather than left to be inferred.

Two of the three focused passes had to be prompted to return partial findings — they were still fetching after a long wall-clock stretch. All three returned in full in the end.

Re-ran all four scripts, confirmed they reproduce the committed outputs byte-for-byte and that the provenance page's embedded copies are byte-identical to fresh runs. Mutation-tested excluded_addresses, line_rate_pps, scan_plan, both directions of the section-8 sample check and the section-0 fold-coverage check, and confirmed each fires. Found no figure that mixes the SCAN/ACTIVE denominators. Four findings:

All 15 citekeys resolve exactly once in a fresh export; bib_dedup_scan.py finds 0 definite duplicates; check_attributions.mjs passes 10/10; every quoted sentence checked matched verbatim; both hand-fetched USENIX author lists match the PDF header in order, including the last author and the umlauts; the published code contains no names at all. One finding:

Independently re-fetched and confirmed: both ZMap man pages (tag and main, byte-identical, 332 lines each), every published flag and default, the absence of --blocklist-file-extra, the three helper binaries, conf/blocklist.conf (16 entries, 2013 header, no opt-out language) with the CIDR arithmetic recomputed by hand including the 255.255.255.255/32-inside-240.0.0.0/4 overlap, all ZGrab2 README quotes, every repository state and release date, nmap 7.991 against nmap.org's own changelog, all three IPv6 Hitlist line counts downloaded and counted independently, the Rapid7 Sonar redirect and the 8 / 47,457 / 60.7 TB figures (summed by hand from the eight per-dataset counts), and both USENIX author lists. Two findings:

It returned after the “did not return” note above had already been written and published, in response to a second prompt for partial findings. Fifteen findings on the already-corrected page, and it earned its slot: five of them are things the three focused passes structurally could not see, and two are wrong numbers the number guard passed because the digits occur elsewhere in the report. This is the fourth page in a row where the generic pass found the worst item.

It also read the provenance probe table and found the IRB row filed under the “first-person” column although the report calls it loose. That is correct and the row is now labelled. Its own NOT READ list is long — the embedded fold source, the external-checks output, `practices:ethics` and `design_dns.txt` for duplication, and a possible disagreement between this page's ZMap 95 + XMap 16 and dns' “ZMap and XMap (112 papers)” that it could not resolve. That last one is unresolved. A follow-up item (internet_scanning: three follow-ups the review left open) is filed for it, together with the two gaps below.

Each pass named its own gaps and they are worth recording, because they are the parts of this page nothing checked:

• The 112-vs-111 question above. dns reports “ZMap and XMap (112 papers)” against this page's ZMap 95 + XMap 16 = 111. That may be an overlap artefact (a paper naming both) or a real disagreement between two folds. Not resolved.
• Nobody hand-audited the 174 HAND verdicts in scan_fold.mjs for judgement (only the coverage and staleness guards were mutation-tested). A wrong kind on a one-off scanner name moves ACTIVE by one paper and nothing would fire.
• Nobody re-read the 44 probe papers to confirm the own/other verdicts in section 8 independently of the author's reading. Those verdicts carry the four practice estimates.
• The figures pass did not re-fetch the external facts; the currency pass did not re-derive the corpus figures. That split is deliberate, but it means no single pass read an external figure and a corpus figure in the same sentence.
• The citations pass flagged that nobody had verified [1Durumeric, Zakir; Wustrow, Eric; Halderman, J. Alex (2013): "ZMap: Fast Internet-wide Scanning and Its Security Applications", in: Proceedings of the USENIX Security Symposium. (Link)]'s claimed absence from the corpus. Closed by the author on 2026-09-10: 0 extraction records, 0 data/fulltext/2013/USENIX directories and 0 of the 28 corpus2/.meta/USENIX-2013.json records match /zmap|internet-wide-scanning/i. Twenty-eight records is itself thin for that venue-year — a corpus-level observation, recorded on corpus' territory rather than here.

report_internet_scanning-output.txt

==============================================================================
0. FOLD COVERAGE — every category=network-scanner name must be decided
==============================================================================
category='network-scanner' used/produced tuples: 741
unmapped strings: 0
HAND entries: 173, matched: 173, stale: 0
 
==============================================================================
0b. IS THE ZMAP TOOL PAPER IN THE CORPUS? (the page says no, twice)
==============================================================================
extraction records matching /zmap|internet-wide-scanning/i in USENIX 2013: 0
data/fulltext/2013/USENIX directories matching /zmap|scann/i:            0
corpus2/.meta/USENIX-2013.json records matching /zmap/i:                 0 of 28
PUBLISHED_ZMAP2013_ABSENT yes
PUBLISHED_USENIX2013_INDEX 28
 
==============================================================================
1. POPULATIONS
==============================================================================
Population                                                  Papers  Share
----------------------------------------------------------  ------  --------------
corpus (7 venues, 2010–2026)                                5859    100%
SCAN — studyTypes 'network-scan-or-probe'                   930     15.9%
ACTIVE — names an active-scan instrument (page population)  241     4.1% of corpus
  … of those, inside SCAN                                   232     24.9% of SCAN
  … of those, outside SCAN                                  9       3.7% of ACTIVE
SCAN papers naming ANY category='network-scanner' tool      396     42.6%
SCAN papers naming NO category='network-scanner' tool       534     57.4%
PUBLISHED_CORPUS 5859
PUBLISHED_SCAN 930
PUBLISHED_ACTIVE 241
PUBLISHED_ACTIVE_IN_SCAN 232
PUBLISHED_ACTIVE_IN_SCAN_PCT 24.9%
PUBLISHED_ACTIVE_OUTSIDE 9
PUBLISHED_NAMES_NS 396
PUBLISHED_NO_NS 534 57.4%
 
category-restricted membership would give 227; the any-category rule adds 14
PUBLISHED_NS_ONLY 227
  +CCS/2015/iris-vetting-private-api-abuse-in-ios-applications  Iris[program-analysis]
  +IMC/2016/entropy-ip-uncovering-structure-in-ipv6-addresses  Entropy/IP[other]
  +IMC/2018/digging-into-browser-based-crypto-mining  ZGrab[http-client]
  +IMC/2018/where-the-light-gets-in-analyzing-web-censorship-mechanisms-in-india  OONI[other]
  +CCS/2019/geneva-evolving-censorship-evasion-strategies  Geneva[other]
  +IMC/2020/flashroute-efficient-traceroute-on-a-massive-scale  IPv6 Hitlist Service[other]
  +USENIX/2022/get-out-automated-discovery-of-application-layer-censorship-evasion-strategies  Geneva[other]
  +IMC/2023/replication-towards-a-publicly-available-internet-scale-ip-geolocation-dataset  IPv6 Hitlist Service[other]
  +PETS/2023/certainty-detecting-dns-manipulation-at-scale-using-tls-certificates  Censored Planet[infrastructure]
  +IMC/2023/inferring-changes-in-daily-human-activity-from-internet-response  Trinocular[infrastructure]
  +USENIX/2023/deresistor-toward-detection-resistant-probing-for-evasion-of-internet-censorship  Geneva[other]
  +NDSS/2024/modeling-and-detecting-internet-censorship-events  Censored Planet[other], OONI[other]
  +IMC/2024/poster-traffic-engineering-security-implications  IPv6 Hitlist Service[other]
  +USENIX/2024/with-great-power-come-great-side-channels-statistical-timing-side-channel-analys  TLS banner grabbers[other]
 
==============================================================================
2. KIND UNION over SCAN (a paper can name several kinds)
==============================================================================
kind               membership?  Papers  Share of 930  routed to
-----------------  -----------  ------  ------------  ----------------------------
active-scan        YES          177     19.0%
dataset                         97      10.4%         design:existing_datasets
topology                        94      10.1%         design:ip_classification
passive                         85      9.1%
vantage-platform                74      8.0%          design:crawling_location
routing                         72      7.7%          design:ip_classification
dns                             72      7.7%          design:dns
grabber            YES          53      5.7%
censorship-probe   YES          23      2.5%
reachability       YES          19      2.0%
vuln-scanner                    15      1.6%          security:web_vulnerabilities
ipv6-targeting     YES          11      1.2%
local-link                      9       1.0%
perf                            7       0.8%
p2p-crawler                     6       0.6%
name-enumeration                3       0.3%
packet-crafting                 3       0.3%
simulation                      3       0.3%
lab-protocol-test               3       0.3%
registry                        1       0.1%
 
SCAN papers whose tools[] name no instrument this fold recognises: 416 (44.7%)
PUBLISHED_SCAN_NO_INSTRUMENT 416 44.7%
 
==============================================================================
3. THE INSTRUMENTS — membership families over ACTIVE
==============================================================================
Family                kind              Papers  Share of ACTIVE  distinct spellings
--------------------  ----------------  ------  ---------------  ------------------
ZMap                  active-scan       94      39.0%            10
nmap                  active-scan       45      18.7%            10
ZGrab                 grabber           32      13.3%            7
custom scanner        active-scan       25      10.4%            21
ping                  reachability      18      7.5%             7
XMap                  active-scan       16      6.6%             3
TLS banner grabbers   grabber           11      4.6%             8
OONI                  censorship-probe  7       2.9%             3
Geneva                censorship-probe  6       2.5%             1
Censored Planet       censorship-probe  4       1.7%             1
IPv6 Hitlist Service  ipv6-targeting    4       1.7%             5
Masscan               active-scan       4       1.7%             1
Quack                 censorship-probe  4       1.7%             1
6Gen                  ipv6-targeting    3       1.2%             1
Hyperquack            censorship-probe  3       1.2%             1
Trinocular            reachability      3       1.2%             1
addr6                 ipv6-targeting    2       0.8%             1
Augur                 censorship-probe  2       0.8%             1
Entropy/IP            ipv6-targeting    2       0.8%             1
GeoInspector          censorship-probe  2       0.8%             1
Goscanner             grabber           2       0.8%             1
Iris                  censorship-probe  2       0.8%             2
Karma                 active-scan       2       0.8%             1
LZR                   grabber           2       0.8%             1
QScanner              grabber           2       0.8%             1
Satellite             censorship-probe  2       0.8%             1
 
families named by exactly one paper: 37 — 6Graph, 6Sense, 6Tree, AmpMap, BreadCrumb, CnCHunter, dMAP (DPI Mapper), EAST, ERIS, exitmap, forwarders-based scanner, gateway checker, gQUIC-module, ICMP_Sonar, ike-scan, IRBlock, IRLscanner, JARM, LanDscAPe, LFP, Lumscan, Mint, modified ntpdedup, Monocle, nc, Netcat, node probing program, onesixtyone, onion-grab, OSAVRoute, PLCScan, Python script, quicgrabber, RAT-Scan, smbmap, TMC, WhatWeb
PUBLISHED_ONEOFF_FAMILIES 37
PUBLISHED_FAM_ZMAP 94
PUBLISHED_FAM_NMAP 45
PUBLISHED_FAM_ZGRAB 32
PUBLISHED_FAM_CUSTOM_SCANNER 25
PUBLISHED_FAM_PING 18
PUBLISHED_FAM_XMAP 16
PUBLISHED_FAM_TLS_BANNER_GRABBERS 11
PUBLISHED_FAM_OONI 7
 
 
ZMap|ZGrab|Masscan|nmap, used or produced: 143 corpus-wide, 139 inside SCAN (the roadmap row said 143)
PUBLISHED_CORE_FOUR_ALL 143
PUBLISHED_CORE_FOUR_SCAN 139
ZMap spellings: TCP Options module for ZMap | ZGrab/ZMap | ZMap | ZMap-based script | ZMap6 | ZMapv6 | Zmap | zmap | zmap2 | zmap6
PUBLISHED_ZMAP_SPELLINGS 10
ACTIVE papers whose scanner is home-grown: 45 (18.7%)
  = generic 'custom scanner' string:                    25
  + a one-paper name the paper itself PRODUCED:         20
  - overlap:                                            0
  (the other 17 one-paper families are third-party tools only this corpus's one paper used)
PUBLISHED_HOMEGROWN 45 18.7%
PUBLISHED_HOMEGROWN_CUSTOM 25
PUBLISHED_HOMEGROWN_PRODUCED 20
PUBLISHED_ONEOFF_THIRDPARTY 17
 
==============================================================================
4. CURRENCY — membership families by year bucket, share of that bucket's ACTIVE papers
==============================================================================
Window      ACTIVE papers  ZMap        ZGrab       nmap        XMap        Masscan   custom scanner  ping
----------  -------------  ----------  ----------  ----------  ----------  --------  --------------  ---------
2010–2013   12             1 (8.3%)    0 (0.0%)    5 (41.7%)   0 (0.0%)    0 (0.0%)  3 (25.0%)       3 (25.0%)
2014–2017   45             27 (60.0%)  4 (8.9%)    8 (17.8%)   0 (0.0%)    1 (2.2%)  7 (15.6%)       2 (4.4%)
2018–2021   55             21 (38.2%)  9 (16.4%)   15 (27.3%)  0 (0.0%)    0 (0.0%)  5 (9.1%)        3 (5.5%)
2022–2024   85             24 (28.2%)  9 (10.6%)   13 (15.3%)  10 (11.8%)  3 (3.5%)  6 (7.1%)        9 (10.6%)
2025–2026*  44             21 (47.7%)  10 (22.7%)  4 (9.1%)    6 (13.6%)   0 (0.0%)  4 (9.1%)        1 (2.3%)
 
Per year, ZMap and the whole ACTIVE population (2026 is provisional):
Year  corpus  SCAN  ACTIVE  ZMap
----  ------  ----  ------  ----
2010  119     32    6       0
2011  116     23    5       0
2012  151     19    0       0
2013  125     15    1       1
2014  166     33    9       6
2015  190     40    13      6
2016  182     46    12      6
2017  231     54    11      9
2018  254     54    17      6
2019  402     65    12      3
2020  404     71    12      7
2021  379     65    14      5
2022  546     70    24      9
2023  719     79    30      7
2024  690     96    31      8
2025  770     122   33      17
2026  415     46    11      4
first ZMap year in ACTIVE: 2013
PUBLISHED_ZMAP_FIRST_YEAR 2013
 
==============================================================================
5. WHERE SCANNING PUBLISHES — venue
==============================================================================
Venue    Papers  SCAN  SCAN share of venue  ACTIVE  ACTIVE share of venue
-------  ------  ----  -------------------  ------  ---------------------
CCS      990     132   13.3%                23      2.3%
IEEE-SP  767     87    11.3%                18      2.3%
IMC      638     296   46.4%                79      12.4%
NDSS     701     117   16.7%                34      4.9%
PETS     510     49    9.6%                 8       1.6%
USENIX   1410    199   14.1%                68      4.8%
WWW      843     50    5.9%                 11      1.3%
 
==============================================================================
6. WHAT A SCAN PAPER REPORTS — schema fields, denominator ACTIVE and SCAN
==============================================================================
Field                           ACTIVE (241)  share  SCAN (930)  share
------------------------------  ------------  -----  ----------  -----
vantage location                134           55.6%  487         52.4%
vantage infrastructure          164           68.0%  612         65.8%
any used/produced tool version  115           47.7%  401         43.1%
version of the SCANNER itself   20            8.3%   20          2.2%
own artifact URL                144           59.8%  506         54.4%
ethics review outcome           92            38.2%  290         31.2%
notified affected parties       184           76.3%  560         60.2%
harm mitigation described       208           86.3%  700         75.3%
robots.txt                      11            4.6%   22          2.4%
temporal span start             192           79.7%  656         70.5%
 
ACTIVE papers with NO ethics record at all: 7 (2.9%) — not counted as stated, not counted as a sentinel
ACTIVE papers with NO artifacts record at all: 3 (1.2%)
PUBLISHED_NO_ETHICS_RECORD 7 2.9%
PUBLISHED_NO_ARTIFACT_RECORD 3 1.2%
PUBLISHED_REPORT_VANTAGE_LOCATION 134 55.6% | SCAN 487 52.4%
PUBLISHED_REPORT_VANTAGE_INFRASTRUCTURE 164 68.0% | SCAN 612 65.8%
PUBLISHED_REPORT_ANY_USED_PRODUCED_TOOL_VERSION 115 47.7% | SCAN 401 43.1%
PUBLISHED_REPORT_VERSION_OF_THE_SCANNER_ITSELF 20 8.3% | SCAN 20 2.2%
PUBLISHED_REPORT_OWN_ARTIFACT_URL 144 59.8% | SCAN 506 54.4%
PUBLISHED_REPORT_ETHICS_REVIEW_OUTCOME 92 38.2% | SCAN 290 31.2%
PUBLISHED_REPORT_NOTIFIED_AFFECTED_PARTIES 184 76.3% | SCAN 560 60.2%
PUBLISHED_REPORT_HARM_MITIGATION_DESCRIBED 208 86.3% | SCAN 700 75.3%
PUBLISHED_REPORT_ROBOTS_TXT 11 4.6% | SCAN 22 2.4%
PUBLISHED_REPORT_TEMPORAL_SPAN_START 192 79.7% | SCAN 656 70.5%
 
==============================================================================
7. FULL-TEXT PROBES over ACTIVE — a loose upper bound and a tight estimate
==============================================================================
PUBLISHED_PROBE_STATES_ITS_OWN_PROBE_RATE_OR_RATE_LIMIT_LOOSE 124 51.5%
PUBLISHED_PROBE_STATES_ITS_OWN_PROBE_RATE_OR_RATE_LIMIT_TIGHT 81 33.6%
PUBLISHED_PROBE_HONOURS_A_BLOCKLIST_EXCLUSION_LIST_LOOSE 103 42.7%
PUBLISHED_PROBE_HONOURS_A_BLOCKLIST_EXCLUSION_LIST_TIGHT 88 36.5%
PUBLISHED_PROBE_OFFERS_AN_OPT_OUT_LOOSE 74 30.7%
PUBLISHED_PROBE_OFFERS_AN_OPT_OUT_TIGHT 70 29.0%
PUBLISHED_PROBE_SOURCE_ADDRESS_HYGIENE_DEDICATED_IPS_RDNS_PTR_WHOIS_WEB_PAGE__LOOSE 167 69.3%
PUBLISHED_PROBE_SOURCE_ADDRESS_HYGIENE_DEDICATED_IPS_RDNS_PTR_WHOIS_WEB_PAGE__TIGHT 112 46.5%
PUBLISHED_PROBE_ABUSE_COMPLAINTS_RECEIVED_OR_HANDLED_LOOSE 58 24.1%
PUBLISHED_PROBE_ABUSE_COMPLAINTS_RECEIVED_OR_HANDLED_TIGHT 45 18.7%
Practice                                                           loose probe (upper bound)  first-person probe
-----------------------------------------------------------------  -------------------------  ------------------
states its own probe rate or rate limit                            124 (51.5%)                81 (33.6%)
honours a blocklist / exclusion list                               103 (42.7%)                88 (36.5%)
offers an opt-out                                                  74 (30.7%)                 70 (29.0%)
source-address hygiene (dedicated IPs, rDNS/PTR, WHOIS, web page)  167 (69.3%)                112 (46.5%)
abuse complaints received or handled                               58 (24.1%)                 45 (18.7%)
 
Neither column is a population. The loose regex fires on any sentence in
the paper, including one about somebody else's rate limit or somebody
else's blocklist; the first-person regex requires "we"/"our" near the term
and still admits sentences about the authors' own *subject*. Section 8
hand-reads a deterministic sample of each first-person hit list and gives
the measured precision.
 
ACTIVE papers naming an IRB / ethics board: 55 (22.8%) [loose, no precision measured]
ACTIVE papers citing Durumeric / Partridge-Allman / Menlo / "good Internet citizen": 172 (71.4%) [loose]
PUBLISHED_IRB 55 22.8%
PUBLISHED_GOODPRACTICE 172 71.4%
 
==============================================================================
8. HAND-READ PRECISION — a PINNED sample of each first-person hit list
==============================================================================
 
--- own probe rate: 81 first-person hits of 241 ACTIVE; pinned sample of 12
    sample drawn on 2026-09-10 as every-k-th of a hit list that was then 84 long
    other    CCS/2015/a-search-engine-backed-by-internet-wide-scanning
    other    CCS/2025/poster-eris-evaluating-rov-via-icmpv6-rate-limiting-side-channels
    other    IEEE-SP/2023/pcspoof-compromising-the-safety-of-time-triggered-ethernet
    other    IMC/2016/on-the-free-bridge-across-the-digital-divide-assessing-the-quality-of-facebooks
    own      IMC/2022/saving-brians-privacy-the-perils-of-privacy-exposure-through-reverse-dns
    own      IMC/2025/chaos-in-the-chain-evaluate-deployment-and-construction-compliance-of-web-pki-ce
    own      NDSS/2020/measuring-the-deployment-of-network-censorship-filters-at-global-scale
    other    NDSS/2024/proof-of-backhaul-trustfree-measurement-of-broadband-bandwidth
    other    USENIX/2011/toward-secure-embedded-web-interfaces
    other    USENIX/2021/accurately-measuring-global-risk-of-amplification-attacks-using-ampmap
    own      USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications
    own      USENIX/2025/exposing-and-circumventing-sni-based-quic-censorship-of-the-great-firewall-of-ch
    hit list 81 now vs 84 at draw time — drift 3.6%
    sample positions in the sorted hit list: 0, 7, 14, 21, 28, 34, 41, 48, 55, 62, 68, 75 (spread 75 of 81)
    precision 5/12; 81 x 5/12 = about 34 papers (14.1% of ACTIVE)
PUBLISHED_PRECISION_OWN_PROBE_RATE 5/12 -> ~34 14.1%
 
--- own blocklist / exclusion: 88 first-person hits of 241 ACTIVE; pinned sample of 11
    sample drawn on 2026-09-10 as every-k-th of a hit list that was then 90 long
    own      CCS/2018/practical-state-recovery-attacks-against-legacy-rng-implementations
    other    IEEE-SP/2017/augur-internet-wide-detection-of-connectivity-disruptions
    other    IMC/2011/detecting-validating-and-characterizing-computer-infections-in-the-wild
    own      IMC/2016/measuring-the-security-harm-of-tls-crypto-shortcuts
    own      IMC/2018/clusters-in-the-expanse-understanding-and-unbiasing-ipv6-hitlists
    other    IMC/2022/deep-dive-into-the-iot-backend-ecosystem
    own      IMC/2023/ecn-with-quic-challenges-in-the-wild
    other    NDSS/2020/decentralized-control-a-case-study-of-russia
    other    NDSS/2025/revealing-the-black-box-of-device-search-engine-scanning-assets-strategies-and-ethical-consideration
    own      USENIX/2014/an-internet-wide-view-of-internet-wide-scanning
    own      USENIX/2018/quack-scalable-remote-measurement-of-application-layer-censorship
    hit list 88 now vs 90 at draw time — drift 2.2%
    sample positions in the sorted hit list: 0, 7, 14, 21, 28, 35, 42, 48, 55, 62, 69 (spread 69 of 88)
    precision 6/11; 88 x 6/11 = about 48 papers (19.9% of ACTIVE)
PUBLISHED_PRECISION_OWN_BLOCKLIST_EXCLUSION 6/11 -> ~48 19.9%
 
--- own opt-out offer: 70 first-person hits of 241 ACTIVE; pinned sample of 10
    sample drawn on 2026-09-10 as every-k-th of a hit list that was then 69 long
    own      CCS/2015/a-search-engine-backed-by-internet-wide-scanning
    other    IEEE-SP/2020/meddling-middlemen-empirical-analysis-of-the-risks-of-data-saving-mobile-browser
    own      IMC/2010/demystifying-service-discovery-implementing-an-internet-wide-scanner
    own      IMC/2018/in-the-ip-of-the-beholder-strategies-for-active-ipv6-topology-discovery
    own      IMC/2022/deep-dive-into-the-iot-backend-ecosystem
    own      IMC/2024/destination-reachable-what-icmpv6-error-messages-reveal-about-their-sources
    own      NDSS/2024/understanding-the-implementation-and-security-implications-of-protective-dns-services
    own      PETS/2023/certainty-detecting-dns-manipulation-at-scale-using-tls-certificates
    own      USENIX/2018/quack-scalable-remote-measurement-of-application-layer-censorship
    own      USENIX/2024/landscape-exploring-ldap-weaknesses-and-data-leaks-at-internet-scale
    hit list 70 now vs 69 at draw time — drift 1.4%
    sample positions in the sorted hit list: 0, 6, 12, 18, 24, 31, 37, 43, 49, 55 (spread 55 of 70)
    precision 9/10; 70 x 9/10 = about 63 papers (26.1% of ACTIVE)
PUBLISHED_PRECISION_OWN_OPT_OUT_OFFER 9/10 -> ~63 26.1%
 
--- own source-address hygiene: 112 first-person hits of 241 ACTIVE; pinned sample of 10
    sample drawn on 2026-09-10 as every-k-th of a hit list that was then 114 long
    own      CCS/2015/a-search-engine-backed-by-internet-wide-scanning
    other    IEEE-SP/2010/investigation-of-triangular-spamming-a-stealthy-and-efficient-spamming-technique
    own      IMC/2010/demystifying-service-discovery-implementing-an-internet-wide-scanner
    other    IMC/2016/identifying-and-aggregating-homogeneous-ipv4-24-blocks-with-hobbit
    own      IMC/2019/an-end-to-end-large-scale-measurement-of-dns-over-encryption-how-far-have-we-com
    own      IMC/2023/does-it-spin-on-the-adoption-and-use-of-quics-spin-bit
    own      NDSS/2020/on-using-application-layer-middlebox-protocols-for-peeking-behind-nat-gateways
    other    NDSS/2026/osavroute-advancing-outbound-source-address-validation-deployment-detection-with-non-cooperative-measurement
    other    USENIX/2021/lzr-identifying-unexpected-internet-services
    own      USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications
    hit list 112 now vs 114 at draw time — drift 1.8%
    sample positions in the sorted hit list: 0, 11, 22, 33, 44, 55, 66, 77, 88, 98 (spread 98 of 112)
    precision 6/10; 112 x 6/10 = about 67 papers (27.8% of ACTIVE)
PUBLISHED_PRECISION_OWN_SOURCE_ADDRESS_HYGIENE 6/10 -> ~67 27.8%
 
Practice                    first-person probe  hand-read precision  point estimate
--------------------------  ------------------  -------------------  --------------
own probe rate              81 (33.6%)          5/12                 ~34 (14.1%)
own blocklist / exclusion   88 (36.5%)          6/11                 ~48 (19.9%)
own opt-out offer           70 (29.0%)          9/10                 ~63 (26.1%)
own source-address hygiene  112 (46.5%)         6/10                 ~67 (27.8%)
 
The point estimate is probe count x sample precision. It is an estimate on a
sample of 10-12, so read it as "about a fifth" rather than as a percentage.
Replacing these four estimates with four numbers needs a per-paper verdict
over the whole hit list — about 400 reads. That work is NOT done.
 
==============================================================================
9. IPv6 — the address space is not enumerable
==============================================================================
Query                                          Population    Papers  Share
---------------------------------------------  ------------  ------  -----
mentions IPv6 anywhere                         SCAN 930      281     30.2%
mentions IPv6 anywhere                         ACTIVE 241    107     44.4%
names an IPv6-targeting instrument in tools[]  SCAN 930      11      1.2%
names an IPv6-targeting instrument in tools[]  corpus 5,859  11      0.2%
full text says "hitlist"/"hit list" (loose)    SCAN 930      62      6.7%
"hitlist" within 80 chars of "IPv6"            SCAN 930      39      4.2%
names a target-generation algorithm            SCAN 930      20      2.2%
discusses aliased prefixes / dealiasing        SCAN 930      10      1.1%
PUBLISHED_V6_SCAN 281 30.2%
PUBLISHED_V6_ACTIVE 107 44.4%
PUBLISHED_V6_INSTR_SCAN 11
PUBLISHED_V6_INSTR_ALL 11
PUBLISHED_HITLIST_LOOSE 62 6.7%
PUBLISHED_HITLIST_V6 39 4.2%
REJECTED_PROBE /Gasser/ over SCAN: 93 (10.0%) — surname is not unique, hits are reference-list entries; not published
PUBLISHED_TARGETGEN 20 2.2%
PUBLISHED_ALIASED 10 1.1%
 
IPv6-SCANNING SUBSET (hitlist-near-IPv6 UNION names an IPv6-targeting instrument): 42 papers
  of those, discuss aliased prefixes / dealiasing: 8 (19.0%)
  the same 10 papers against all of SCAN would be 1.1% — the wrong denominator, most of SCAN never touches IPv6
PUBLISHED_V6_SUBSET 42
PUBLISHED_ALIAS_IN_SUBSET 8 19.0%
 
IPv6-targeting instruments named, corpus-wide:
    4 IPv6 Hitlist Service  (IMC/2020/flashroute-efficient-traceroute-on-a-massive-scale, IMC/2023/replication-towards-a-publicly-available-internet-scale-ip-geolocation-dataset, IMC/2024/poster-traffic-engineering-security-implications, PETS/2025/path-to-encrypted-dns-with-ddr-adoption-configuration-patterns-and-privacy-impli)
    3 6Gen  (IMC/2017/target-generation-for-internet-wide-ipv6-scanning, IMC/2018/clusters-in-the-expanse-understanding-and-unbiasing-ipv6-hitlists, IMC/2018/in-the-ip-of-the-beholder-strategies-for-active-ipv6-topology-discovery)
    2 Entropy/IP  (IMC/2016/entropy-ip-uncovering-structure-in-ipv6-addresses, IMC/2018/clusters-in-the-expanse-understanding-and-unbiasing-ipv6-hitlists)
    2 addr6  (IMC/2018/in-the-ip-of-the-beholder-strategies-for-active-ipv6-topology-discovery, NDSS/2026/on-borrowed-time-measurement-informed-understanding-of-the-ntp-pools-robustness-to-monopoly-attacks)
    1 6Graph  (IMC/2022/rusty-clusters-dusting-an-ipv6-research-foundation)
    1 6Tree  (IMC/2022/rusty-clusters-dusting-an-ipv6-research-foundation)
    1 6Sense  (USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications)
 
Per-year IPv6 mention rate inside SCAN (2026 provisional):
Year  SCAN  mentions IPv6  share
----  ----  -------------  -----
2010  32    4              12.5%
2011  23    2              8.7%
2012  19    1              5.3%
2013  15    0              0.0%
2014  33    6              18.2%
2015  40    11             27.5%
2016  46    12             26.1%
2017  54    13             24.1%
2018  54    19             35.2%
2019  65    21             32.3%
2020  71    21             29.6%
2021  65    22             33.8%
2022  70    24             34.3%
2023  79    31             39.2%
2024  96    28             29.2%
2025  122   54             44.3%
2026  46    12             26.1%
 
==============================================================================
10. THE JOIN TO design:ip_classification
==============================================================================
Population  N     classification[].target == 'ip-address'  share
----------  ----  ---------------------------------------  -----
ACTIVE      241   75                                       31.1%
SCAN        930   213                                      22.9%
corpus      5859  295                                      5.0%
PUBLISHED_IPCLS_ACTIVE 75 31.1%
PUBLISHED_IPCLS_SCAN 213 22.9%
PUBLISHED_IPCLS_CORPUS 295 5.0%
 
distinct raw resourceName strings on ACTIVE ip-address classifications: 110 (free text; folded on design:ip_classification, not here)
    5 MaxMind
    4 MaxMind GeoLite2
    3 IPinfo
    3 VirusTotal
    3 Censys
    2 MaxMind GeoIP
    2 regular expressions (custom)
    2 ipinfo.io
    2 ASdb
    1 custom response categorization
    1 Snacktime (custom implementation)
    1 five public and one commercial blacklist
    1 12 IP reputation blacklists
    1 MaxMind GeoLiteCity
    1 WHOIS
 
==============================================================================
11. POSTERS AND SHORT RECORDS
==============================================================================
ACTIVE posters: 4 (1.7%); SCAN posters: 34 (3.7%)
ACTIVE records of <=4 pages: 6 (2.5%)
vantage location stated: all ACTIVE 55.6%, excluding <=4-page records 56.2%
PUBLISHED_ACTIVE_POSTERS 4
PUBLISHED_VANTAGE_FULLONLY 56.2%
 
==============================================================================
12. MEASURED RESULTS available for the page (detection[].prevalence on ACTIVE)
==============================================================================
ACTIVE papers with at least one detection[].prevalence: 240 (99.6%)
PUBLISHED_ACTIVE_PREVALENCE 240 99.6%
 
==============================================================================
Z. NON-CORPUS FIGURES ON THE PAGE (checked by external_checks_internet_scanning.sh)
==============================================================================
ZMap v4.4.0, released 2026-05-29; repo zmap/zmap not archived, pushed 2026-08-28
ZGrab2 v1.0.0, released 2025-12-04; repo zmap/zgrab2 pushed 2026-09-07
ZDNS v2.1.1, released 2026-05-28; repo zmap/zdns pushed 2026-09-09
masscan latest GitHub Release 1.3.2, 2021-01-31; repo pushed 2026-04-23, not archived
XMap 2.0.6, 2026-03-10; repo idealeer/xmap pushed 2026-03-10
LZR repo stanford-esrg/lzr pushed 2026-01-06, not archived
nmap 7.991 is the newest tarball on nmap.org/dist
ZMap v4.4.0 has ONE --blocklist-file flag (-b); there is no --blocklist-file-extra. The port flag is -p/--target-ports. ziterate, zblocklist and ztee ship in src/ alongside zmap
ZGrab2 input is CSV "IP, DOMAIN, TAG, PORT"; the multiple module takes a .ini
ZMap changed to random per-probe IP IDs in early 2024 (durumeric2024_years section 3)
ZMap defaults (v4.4.0 and main agree): --rate 10000 pps, --probes 1, --cooldown-time 8, --retries 10, --batch 64
ZMap conf/blocklist.conf ships 16 CIDR entries, all IANA special-purpose and multicast ranges, header dated 2013
IPv6 Hitlist Service open files, snapshot 2026-09-05: responsive-addresses 29,050,771 lines; aliased-prefixes 24,476,266; non-aliased-prefixes 222,836,153
Rapid7 Project Sonar: opendata.rapid7.com now redirects to sonardata.rapid7.com, which is commercial access only (8 datasets, 47,457 files, 60.7 TB)
search.censys.io returns HTTP 403 to automated clients; censys.com returns 200
2^32 = 4,294,967,296 IPv4 addresses; the shipped blocklist removes 592,708,864 (13.8%), leaving 3,702,258,432 — computed by pages/scan_budget.py from the live file, collapsing overlaps (255.255.255.255/32 is inside 240.0.0.0/4, so a naive sum gives one address more)
pages/scan_budget.py --self-test: 9 checks (check 9 covers human_time() boundaries). Default run: 4.29 days at 10,000 pps, 10.3 h at 100,000 pps, 41.5 min at 1 GbE line rate (1,488,095 pps), 4.1 min at 10 GbE (14,880,952 pps), on 84 bytes of wire time per probe
 
==============================================================================
RESULT
==============================================================================
OK — all contracts hold

verify_scanning_figures-output.txt

==============================================================================
A. QUOTES AND PER-PAPER FIGURES
==============================================================================
  OK   [.cols] IMC/2024/ten-years-of-zmap
         over 33% of all Internet-wide IPv4 scan traffic can be fingerprinted as coming from ZMap
  OK   [.cols] IMC/2024/ten-years-of-zmap
         35.4% of all Internet-
  OK   [.cols] IMC/2024/ten-years-of-zmap
         TCP scan packets originated from ZMap
  OK   [.cols] IMC/2024/ten-years-of-zmap
         the best way to mitigate transient drop is to scan from 2-3 geographically and topologically div…
  OK   [.cols] IMC/2024/ten-years-of-zmap
         Provide a simple mechanism for operators to request exclusion from future scans. Indicate the IP…
  OK   [.cols] IMC/2024/ten-years-of-zmap
         Consider whether existing open source datasets provide the data you need
  OK   [.cols] IMC/2024/ten-years-of-zmap
         In early 2024, ZMap changed its default behavior to use random per-probe IP IDs
  OK   [.cols] IMC/2024/ten-years-of-zmap
         is typically driven by a handful of small service and cloud providers
  OK   [.cols] IMC/2024/ten-years-of-zmap
         Masscan [48] finds notably fewer hosts than ZMap, likely due to biases in its randomization algo…
  OK   [.cols] IMC/2024/ten-years-of-zmap
         None of the top 100 ASes that emit the most ZMap
  OK   [.cols] USENIX/2014/an-internet-wide-view-of-internet-wide-scanning
         only 208 organizations have requested that we exclude their networks from our scans, reducing th…
  OK   [.cols] IMC/2018/in-the-ip-of-the-beholder-strategies-for-active-ipv6-topology-discovery
         we received two opt-out requests with
  OK   [.cols] IMC/2018/in-the-ip-of-the-beholder-strategies-for-active-ipv6-topology-discovery
         which we immediately complied
  OK   [.cols] USENIX/2021/lzr-identifying-unexpected-internet-services
         only 3% of HTTP and 6% of TLS services run on ports 80 and 443, respectively
  OK   [.cols] USENIX/2021/lzr-identifying-unexpected-internet-services
         Services on non-standard ports are more likely to be insecure
  OK   [.cols] USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications
         a low scanning rate of 10,000 packets per second, total, across all networks
  OK   [.cols] IMC/2025/tracking-internet-disruptions-in-ukraine-insights-from-three-years-of-active-ful
         a low probing rate of 8,000 packets per second, i.e., around 500KB/s from a single vantage point
  OK   [.cols] CCS/2015/a-search-engine-backed-by-internet-wide-scanning
         All of the scanning hosts have WHOIS records and reverse DNS entries that describe the intent of…
  OK   [.cols] CCS/2015/a-search-engine-backed-by-internet-wide-scanning
         each scanning host runs a simple website on port 80 that describes the goals of the research
  OK   [.cols] USENIX/2024/landscape-exploring-ldap-weaknesses-and-data-leaks-at-internet-scale
         On the scanning IPs, a website gives project details and contact information for opt-out request…
  OK   [.cols] IMC/2018/clusters-in-the-expanse-understanding-and-unbiasing-ipv6-hitlists
         identifies 1.5 % of our prefixes as aliased, pertaining to about half of our target addresses
  OK   [.cols] IMC/2022/rusty-clusters-dusting-an-ipv6-research-foundation
         More than 98 % of all IPv6 addresses announced by Fastly were labeled as aliased and Cloudflare …
  OK   [.cols] IMC/2024/have-you-syn-me-characterizing-ten-years-of-internet-scanning
         Internet scanning has increased 30-fold over the last ten years
  OK   [.cols] IMC/2024/have-you-syn-me-characterizing-ten-years-of-internet-scanning
         the ecosystem is extremely volatile, where targeted ports and geographical scanner locations dra…
  OK   [.cols] IMC/2024/have-you-syn-me-characterizing-ten-years-of-internet-scanning
         this number has increased to 345 million in 2024
  OK   [.cols] IMC/2024/have-you-syn-me-characterizing-ten-years-of-internet-scanning
         in 2015 we observed 11 million scanning
 
A2. OFF-CORPUS PAPER (fetched copy, not in the extraction)
  OK   [out/scan/zmap2013_paper.txt] capable of scanning the IPv4 public address space over 1300 times faster than the most agg…
  OK   [out/scan/zmap2013_paper.txt] surveying the entire IPv4 address space in under 45 minutes from user space
  OK   [out/scan/zmap2013_paper.txt] over 97% of the theoretical maximum speed of gigabit Ethernet
  OK   [out/scan/zmap2013_paper.txt] an estimated 98% coverage of publicly available hosts
 
30 needles, 0 not located
 
==============================================================================
B. EXTERNAL FIGURES (re-fetched by external_checks_internet_scanning.sh)
==============================================================================
  ZMap v4.4.0 / 2026-05-29 / pushed 2026-08-28 / 6,374 stars
      GitHub REST API repos/zmap/zmap and /releases/latest, 2026-09-10
  ZGrab2 v1.0.0 / 2025-12-04 / pushed 2026-09-07
      GitHub REST API repos/zmap/zgrab2, 2026-09-10
  masscan 1.3.2 / 2021-01-31 / pushed 2026-04-23
      GitHub REST API repos/robertdavidgraham/masscan, 2026-09-10
  XMap 2.0.6 / 2026-03-10
      GitHub REST API repos/idealeer/xmap, 2026-09-10
  nmap 7.991
      newest nmap-*.tar.bz2 on https://nmap.org/dist/, 2026-09-10
  ZMap defaults: --rate 10000 pps, --probes 1, --cooldown-time 8
      src/zmap.1.ronn at tag v4.4.0 (byte-identical defaults to main), 2026-09-10
  1300x faster than the most aggressive Nmap default settings; under 45 minutes; 97% of theoretical maximum; 98% coverage
      Durumeric, Wustrow and Halderman, USENIX Security 2013 — the paper is NOT in this extraction; read from the authors' copy at zmap.io/paper.pdf (Last-Modified 2025-06-22) on 2026-09-10, extracted with pypdf. Needles A27-A30.
  41.5 / 0.97 = 42.8
      the page's own arithmetic reconciling scan_budget.py's theoretical 1 GbE figure with the ZMap paper's measured 45-minute ceiling
  16 CIDR entries; 592,708,864 removed (13.8%); 3,702,258,432 in scope
      conf/blocklist.conf at tag v4.4.0, computed by pages/scan_budget.py
  4.29 days / 10.3 h / 41.5 min / 4.1 min; 7, 67, 1,000 and 10,000 Mb/s; 1,488,095 and 14,880,952 pps; 84 bytes
      pages/scan_budget.py run 2026-09-10, whole output quoted verbatim on the page
  17.14 days (--ports 2 --probes 2) and 34.28 days (--ports 8)
      pages/scan_budget.py with those flags, 2026-09-10
  0.15% of the IPv4 address space
      inside the durumeric2014_view quote, checked in needle A11
  IPv6 Hitlist open snapshot 2026-09-05: 29,050,771 / 24,476,266 / 222,836,153 lines
      alcatraz.net.in.tum.de/ipv6-hitlist-service/open/*.txt.xz, decompressed line counts, 2026-09-10
  Rapid7 Sonar: 8 datasets, 47,457 files, 60.7 TB, commercial access only
      sonardata.rapid7.com and /about with a browser User-Agent, 2026-09-10
  search.censys.io HTTP 403; censys.com HTTP 200
      curl with a browser User-Agent, 2026-09-10
 
==============================================================================
C. CODE CONSTANTS in pages/scan_budget.py and the example invocations
==============================================================================
  84
      WIRE_OVERHEAD_BYTES in scan_budget.py: 60-byte minimum frame + 4 FCS + 8 preamble/SFD + 12 IFG
  32 / 128
      IPv4 and IPv6 address widths in bits
  16777216 / 65536 / 33554432
      hand-computed /8, /16 and the mutation-guard double count in scan_budget.py self-test
  120 / 120.0 / 7199.0 / 7200 / 7200.0 / 171999.0 / 172800 / 172800.0 / 3600 / 3600.0 / 86400 / 86400.0 / 60 / 60.0 / 119.9 / 24.0 / 2.00
      second thresholds, divisors and the human_time() boundary cases in scan_budget.py --self-test check 9
  1e9 / 1e6
      bits per gigabit and per megabit in line_rate_pps() and scan_plan()
  8
      bits per byte in scan_plan(); also ZMap --cooldown-time default
  443 / 80 / 22 / 8080 / 53
      ports in the example invocations and in the ZGrab2 .ini example
  20260910
      the example --seed value: today's date, chosen so it is obviously arbitrary
  198.51.100.10 / 198.51.100.13 / 10.0.0.0 / 10.1.0.0 / 192.168.0.0 / 0.0.0.0
      RFC 5737 and RFC 1918 literals in the example command and the self-test
  4
      example --sender-threads; also the "four times" arithmetic for 2 ports x 2 probes
  1
      example --probes; ZMap default
  10000
      example --rate; ZMap default
  9 checks
      pages/scan_budget.py --self-test count
 
==============================================================================
D. NOT FIGURES
==============================================================================
  1918 / 5737 / 1122 / 6598 / 3927
      RFC numbers named in blocklist.conf and in the page text
  2013
      the year in the blocklist.conf header comment, and ZMap's release year
  1.0.0 / 4.4.0 / 2.0.6 / 1.3.2 / 7.991 / 2.1.1
      version strings
  30-fold / 2-3
      ratios inside quoted sentences
 
==============================================================================
E. DERIVED ARITHMETIC (recomputed from out/scan/rep.txt, not described)
==============================================================================
  OK   cites-the-norm over named-an-IRB: PUBLISHED_GOODPRACTICE=172 / PUBLISHED_IRB=55 = 3.127, page states 3.1
 
RESULT: OK

external_checks_internet_scanning-output.txt

=== GitHub repository state ===
  zmap/zmap                        archived=False pushed=2026-08-28T04:15:23Z stars=6374
  zmap/zgrab2                      archived=False pushed=2026-09-07T09:29:47Z stars=2173
  zmap/zdns                        archived=False pushed=2026-09-09T17:12:25Z stars=1135
  robertdavidgraham/masscan        archived=False pushed=2026-04-23T20:34:38Z stars=25996
  idealeer/xmap                    archived=False pushed=2026-03-10T14:15:08Z stars=491
  stanford-esrg/lzr                archived=False pushed=2026-01-06T17:27:02Z stars=196
 
=== Latest release AND newest tag (a tag-only repo 404s on /releases/latest; /tags is unsorted) ===
  zmap/zmap                        release=v4.4.0     published=2026-05-29T06:37:41Z   tags[0:6]=v4.4.0,v4.3.4,v4.3.3,v4.3.2,v4.3.1,v4.3.0
  zmap/zgrab2                      release=v1.0.0     published=2025-12-04T23:04:38Z   tags[0:6]=v1.0.0,v1.0.0-RC3,v1.0.0-RC2,v1.0.0-RC1,v0.2.0,v0.1.8
  zmap/zdns                        release=v2.1.1     published=2026-05-28T23:11:26Z   tags[0:6]=v2.1.1,v2.1.0,v2.0.5,v2.0.4,v2.0.3,v2.0.2
  robertdavidgraham/masscan        release=1.3.2      published=2021-01-31T09:19:32Z   tags[0:6]=v2,v1,kali_1_0,1.3.2,1.3.1,1.3.0
  idealeer/xmap                    release=2.0.6      published=2026-03-10T14:15:08Z   tags[0:6]=2.0.6,2.0.5,2.0.4,2.0.3,2.0.2,2.0.1
 
=== ZMap v4.4.0 man page: the flags and defaults the page publishes ===
     * `-p`, `--target-ports=port(s)`:
     * `-b`, `--blocklist-file=path`:
     * `-w`, `--allowlist-file=path`:
     * `-r`, `--rate=pps`:
     * `-B`, `--bandwidth=bps`:
     * `-c`, `--cooldown-time=secs`:
     * `-e`, `--seed=n`:
     * `-P`, `--probes=n`:
       sequentially for a given IP/Port, so if --probes=4, we'll send 4 probe packets
     * `--shards=N`:
     * `-S`, `--source-ip=ip|range`:
     * `-O`, `--output-module=name`:
     * `-f`, `--output-fields=fields`:
     * `-m`, `--metadata-file=filename`:
     * `-T`, `--sender-threads=n`:
  DEFAULT      at full line rate. Default: 10000 pps.
  DEFAULT      How long to continue receiving after sending has completed (default=8)
  DEFAULT      Number of probes to send to each IP/Port pair (default=1). Probes are sent
  DEFAULT      zmap (default=1). When sharding, **--seed** is required.
  OK: no --blocklist-file-extra (the page's claim)
 
=== ZMap ships ziterate / zblocklist / ztee ===
  ziterate.c       present
  zblocklist.c     present
  ztee.c           present
  zmap.c           present
 
=== conf/blocklist.conf at v4.4.0 and the arithmetic the page publishes ===
  lines mentioning "Updated 2013": 2
  OK: no opt-out entries in the shipped file (the page's claim)
  blocklist            /tmp/bl.conf
    CIDR entries       16
    addresses removed  592,708,864  (13.8% of IPv4)
    addresses in scope 3,702,258,432
 
  plan: 1 port(s) x 1 probe(s) per target = 3,702,258,432 packets
 
      rate (pps)    wall clock     wire rate
  --------------  ------------  ------------
          10,000     4.29 days          7 Mb/s
         100,000        10.3 h         67 Mb/s
       1,488,095      41.5 min      1,000 Mb/s
      14,880,952       4.1 min     10,000 Mb/s
 
  Reference rates: ZMap's default --rate is 10,000 pps; 1 GbE line rate is
  1,488,095 pps and 10 GbE is 14,880,952 pps at 84 bytes of wire time per probe.
 
  What this file does NOT tell you: whether anyone asked to be excluded.
  The shipped blocklist is IANA special-purpose space. Opt-out entries are
  yours to add, and to keep across runs.
  self-test OK (9 checks)
  plan: 2 port(s) x 2 probe(s) per target = 14,809,033,728 packets
 
      rate (pps)    wall clock     wire rate
  --------------  ------------  ------------
          10,000    17.14 days          7 Mb/s
  plan: 8 port(s) x 1 probe(s) per target = 29,618,067,456 packets
 
      rate (pps)    wall clock     wire rate
  --------------  ------------  ------------
          10,000    34.28 days          7 Mb/s
 
=== ZGrab2 README at v1.0.0: the input format the page quotes ===
  115:IP, DOMAIN, TAG, PORT
  118:Each line must specify `IP`, `DOMAIN`, or both.  If only `DOMAIN` is provided, scanners perform a DNS hostname lookup to determine the IP address.  If both `IP` and `DOMAIN` are provided, scanners connect to `IP` but use `DOMAIN` in protocol-specific contexts, such as the HTTP HOST header and TLS SNI extension.
  4:ZGrab is a fast, modular application-layer network scanner designed for completing large Internet-wide surveys. ZGrab is built to work with ZMap (ZMap identifies L4 responsive hosts, ZGrab performs in-depth, follow-up L7 handshakes). Unlike many other network scanners, ZGrab outputs detailed transcripts of network handshakes (e.g., all messages exchanged in a TLS handshake) for offline analysis.  
 
=== nmap: newest tarball on nmap.org/dist ===
  nmap-7.991.tar.bz2
 
=== IPv6 Hitlist Service (TUM) — open tier ===
  responsive-addresses   lines=29050771     Last-Modified: Sat, 05 Sep 2026 11:37:21 GMT
  aliased-prefixes       lines=24476266     Last-Modified: Sat, 05 Sep 2026 11:43:49 GMT
  non-aliased-prefixes   lines=222836153    Last-Modified: Sat, 05 Sep 2026 11:49:08 GMT
  ipv6hitlist.github.io http=200
 
=== Scan datasets: access terms ===
  opendata.rapid7.com   http=200 final=https://sonardata.rapid7.com/
  sonardata about: provides commercial access to data from Project Sonar
  Sign In (existing accounts only) | Datasets: 8 | Files: 47,457 | Total size: 60.7 TB
  search.censys.io      http=403
  censys.com            http=200
  shodan.io             http=200
 
=== The two USENIX author lists in the bibliography (venue records carry no authors) ===
  usenixsecurity24/presentation/williams
    Grant Williams
    Mert Erdemir
    Amanda Hsu
    Shraddha Bhat
    Abhishek Bhaskar
    Frank Li
    Paul Pearce
  usenixsecurity24/presentation/kaspereit
    Jonas Kaspereit
    Gurur Öndarö
    Gustavo Luvizotto Cesar
    Simon Ebbers
    Fabian Ising
    Christoph Saatjohann
    Mattijs Jonker
    Ralph Holz
    Sebastian Schinzel
 
DONE

scan_fold.mjs

// Instrument fold and written population boundary for programming:internet_scanning.
//
// WRITTEN BOUNDARY (fixed 2026-09-10, before the first figure was counted).
// The page is about ACTIVE INTERNET SCANNING: sending probe packets to an
// address set the researcher chose, over the public Internet, to learn which
// hosts answer and what they say.
//
//   SCAN    = the 930 papers tagged studyTypes 'network-scan-or-probe'. This is
//             the branch population design:automated_measurements routes with.
//             Every reporting-rate denominator on the page is a subset of it.
//   ACTIVE  = papers naming, as a tool they USED or PRODUCED, an instrument
//             whose folded kind is in MEMBERSHIP_KINDS below. This is the
//             page's own population.
//
// Membership is decided by the INSTRUMENT, not by tools[].category: a scanner
// filed under 'other' or 'infrastructure' is still a scanner (see
// tool-category-is-a-denominator). Concretely:
//   (a) any tools[] entry in ANY category whose name folds to a membership
//       family, plus
//   (b) any tools[] entry with category === 'network-scanner' whose name is
//       unmapped by (a) and whose HAND verdict below is a membership kind.
// Residue outside category 'network-scanner' is not hand-read — it would be
// the whole 16,737-name tool vocabulary — so ACTIVE is a floor and the report
// prints how many papers (a) adds beyond the category restriction.
//
// Reported beside ACTIVE but deliberately NOT in it, each with the page it is
// routed to:
//   dataset          someone else's scan (Censys, Shodan, FOFA, Rapid7 ...)
//   topology         traceroute / alias resolution / anycast census
//   dns              DNS-specific instruments            -> design:dns
//   vantage-platform RIPE Atlas, looking glasses         -> design:crawling_location
//   routing          BGP and AS-relationship inference
//   vuln-scanner     one host's application layer        -> security:web_vulnerabilities
//   passive          telescopes, sniffers, IDS
//   local-link       Wi-Fi / Bluetooth / cellular / LAN probes
//   perf, simulation, p2p-crawler, name-enumeration, packet-crafting, registry
//
// Every unmapped category==='network-scanner' string is a FAILURE, not a drop.
 
const F = (family, kind, ...patterns) => ({ family, kind, patterns, generic: false });
// A GENERIC family's regex is a *descriptor* ("a custom scanner", "ping"), not a
// coined product name, so it is only trusted inside tools[].category ===
// 'network-scanner'. Without that restriction /^custom\b/i alone puts 464
// papers in the population, almost all of them custom crawlers and scripts.
const G = (family, kind, ...patterns) => ({ family, kind, patterns, generic: true });
 
// Checked before the families. A name here is never folded, with its reason.
export const NEGATIVE = [
  [/^wizmap$/i, 'WizMap is an embedding-visualisation tool; /zmap/i matches it'],
];
 
// Ordered: first match wins. Specific products before the generic term they
// contain (ZGrab / ZDNS / ZMapv6 before ZMap; Paris traceroute before
// traceroute; Yarrp6 before Yarrp).
export const FAMILIES = [
  // --- address-space scanners and banner grabbers: the page's subject --------
  F('ZGrab', 'grabber', /zgrab/i),
  F('ZDNS', 'dns', /^zdns$/i),
  F('ZMap', 'active-scan', /z-?map/i),
  F('XMap', 'active-scan', /^x-?map\b/i),
  F('Masscan', 'active-scan', /masscan/i),
  F('nmap', 'active-scan', /n-?map/i),
  F('LZR', 'grabber', /^lzr$/i),
  F('Goscanner', 'grabber', /^goscanner$/i),
  F('TLS banner grabbers', 'grabber', /sslscan|sslyze|testssl|cipherscan|tls-?scanner|tls scanner|tls-crawler/i),
  F('JARM', 'grabber', /^jarm$/i),
  G('custom scanner', 'active-scan', /^custom\b|^(efficient|systematic) scanning (tool|software)$|^stateless scanner|^scanner application$|^scanning tool$/i),
 
  // --- IPv6 targeting: the only way a v6 "scan" has a target list ------------
  F('IPv6 Hitlist Service', 'ipv6-targeting', /hitlist|hit-list/i),
  F('6Gen', 'ipv6-targeting', /^6gen$/i),
  F('6Sense', 'ipv6-targeting', /^6sense$/i),
  F('6Graph', 'ipv6-targeting', /^6graph$/i),
  F('6Tree', 'ipv6-targeting', /^6tree$/i),
  F('Entropy/IP', 'ipv6-targeting', /entropy\/ip/i),
  F('addr6', 'ipv6-targeting', /^addr6$/i),
 
  // --- reachability sweeps ---------------------------------------------------
  G('ping', 'reachability', /^[hf]?ping\d?$/i, /^icmp( ping)?$/i, /^tcp pings?$/i, /^pingtools$/i),
  F('Trinocular', 'reachability', /^trinocular$/i),
 
  // --- remote censorship / reachability probing ------------------------------
  F('Censored Planet', 'censorship-probe', /censored planet/i),
  F('OONI', 'censorship-probe', /^ooni/i),
  F('Quack', 'censorship-probe', /^quack$/i),
  F('Hyperquack', 'censorship-probe', /^hyperquack$/i),
  F('Satellite', 'censorship-probe', /^satellite$/i),
  F('Augur', 'censorship-probe', /^augur$/i),
  F('Iris', 'censorship-probe', /^iris$/i),
  F('GeoInspector', 'censorship-probe', /^geoinspector$/i),
  F('Geneva', 'censorship-probe', /^geneva$/i),
 
  // --- scan datasets: someone else's scan ------------------------------------
  F('Censys', 'dataset', /censys/i),
  F('Shodan', 'dataset', /shodan/i),
  F('FOFA', 'dataset', /^fofa$/i),
  F('ZoomEye', 'dataset', /^zoomeye$/i),
  F('Rapid7 Open Data', 'dataset', /rapid7 (labs open data|sonar)/i),
  F('GreyNoise', 'dataset', /^greynoise$/i),
  F('LeakIX', 'dataset', /^leakix$/i),
  F('Open Resolver Project', 'dataset', /open resolver project/i),
 
  // --- topology --------------------------------------------------------------
  F('Yarrp', 'topology', /^(extended )?yarrp(6|box)?$/i),
  F('scamper', 'topology', /^scamper( python module)?$/i),
  F('Paris traceroute', 'topology', /paris traceroute/i),
  F('traIXroute', 'topology', /^traixroute$/i),
  F('mtr', 'topology', /^mtr$/i),
  F('MIDAR', 'topology', /^midar$/i),
  F('bdrmap', 'topology', /bdrmap/i),
  F('TNT', 'topology', /^p?y?tnt$/i),
  F('traceroute', 'topology', /traceroute/i),
 
  // --- routing ---------------------------------------------------------------
  F('RIPE RIS', 'routing', /ripe ris/i),
  F('RouteViews', 'routing', /^route ?views\b/i),
  F('BGPStream', 'routing', /bgpstream/i),
 
  // --- distributed vantage platforms ----------------------------------------
  F('RIPE Atlas', 'vantage-platform', /ripe atlas/i),
  F('perfSONAR', 'vantage-platform', /^perfsonar$/i),
  F('looking glass', 'vantage-platform', /looking glass|looking glasses|^periscope$/i),
 
  // --- DNS instruments -------------------------------------------------------
  F('massdns', 'dns', /^massdns$/i),
  F('fpdns', 'dns', /^fpdns$/i),
  F('OpenINTEL', 'dns', /openintel/i),
  F('dig/drill', 'dns', /^(dig|drill|nsupdate|unbound|dnspython|dnsviz)$/i),
 
  // --- passive ---------------------------------------------------------------
  F('Zeek/Bro', 'passive', /^(zeek|bro|bro ?-?ids?)$/i),
  F('Suricata', 'passive', /^suricata$/i),
  F('Snort', 'passive', /^snort$/i),
  F('p0f', 'passive', /^p0f\d?$/i),
  F('tshark/tcpdump', 'passive', /^(tshark|tcpdump|argus)$/i),
 
  // --- one host's application layer -----------------------------------------
  F('Nessus', 'vuln-scanner', /^nessus$/i),
  F('Acunetix', 'vuln-scanner', /acunetix/i),
  F('OpenVAS', 'vuln-scanner', /^openvas$/i),
  F('OWASP ZAP', 'vuln-scanner', /zed attack proxy|owasp zap|^zap$/i),
  F('Nikto', 'vuln-scanner', /^nikto$/i),
  F('sqlmap', 'vuln-scanner', /^sqlmap$/i),
];
 
export const KIND_OF = new Map(FAMILIES.map((f) => [f.family, f.kind]));
export const GENERIC_FAMILIES = new Set(FAMILIES.filter((f) => f.generic).map((f) => f.family));
 
// Kinds that put a paper in ACTIVE.
export const MEMBERSHIP_KINDS = new Set([
  'active-scan',
  'grabber',
  'reachability',
  'censorship-probe',
  'ipv6-targeting',
]);
 
export function foldScannerName(raw) {
  const v = String(raw).trim();
  for (const [re] of NEGATIVE) if (re.test(v)) return null;
  for (const { family, patterns } of FAMILIES) {
    for (const re of patterns) {
      if (re instanceof RegExp && re.test(v)) return family;
    }
  }
  return null;
}
 
// ---------------------------------------------------------------------------
// HAND verdicts for every category==='network-scanner' name the families above
// do not fold. One entry per distinct string, read off the tool's own
// `purpose` field and, where the name alone was ambiguous, the paper's title.
// The report throws on an unmapped string and on an entry nothing matches.
export const HAND = {
  '0trace': 'topology',
  '5GSniffer': 'local-link',
  'active DNS crawler': 'dns',
  'active scanning tool': 'dns',                 // "Query DKIM, SPF, DMARC records"
  'active-geolocator': 'topology',
  'AdaptOver': 'local-link',
  'Airmon-ng': 'local-link',
  'Ally': 'topology',
  'amass': 'name-enumeration',
  'AmpMap': 'active-scan',
  'Anaximander': 'topology',
  'Aquantone': 'name-enumeration',
  'Arachni': 'vuln-scanner',
  'arpscan': 'local-link',
  'Bitnodes': 'p2p-crawler',
  'BitTorrent crawler': 'p2p-crawler',
  'Black Widow': 'vuln-scanner',
  'Bluesniff': 'local-link',
  'BreadCrumb': 'censorship-probe',
  'Btsniffer': 'local-link',
  'BurpSuite Scanner': 'vuln-scanner',
  'CAIDA AS Relationships Dataset': 'routing',
  'Cenzic HailStorm Pro': 'vuln-scanner',
  'Chaos': 'simulation',
  'CheckHost': 'vantage-platform',
  'cloud_enum': 'name-enumeration',
  'CnCHunter': 'active-scan',
  'Commix': 'vuln-scanner',
  'COVScan': 'vuln-scanner',
  'Critical.IO': 'dataset',
  'ct-sans': 'dataset',
  'danglingcname': 'dns',
  'Diamond Miner': 'topology',
  'DMiner': 'dns',
  'dMAP (DPI Mapper)': 'active-scan',
  'tcp_probe': 'passive',
  'TMC': 'censorship-probe',
  'ThunderPing': 'vantage-platform',
  'DNS crawler': 'dns',
  'DNS Scanner': 'dns',
  'DNS X-Ray Tool': 'dns',
  'DoE-Hunter': 'dns',
  'DTLS-Fuzzer': 'lab-protocol-test',
  // HAND-AUDIT 2026-09-17: was 'active-scan'. USENIX 2026 "Analyzing the WebRTC
  // Ecosystem...": "an automated MitM testing framework (DTLS-MitM-Scanner (DMS))
  // to test the DTLS channel of a DTLS-SRTP connection", "requiring only MitM
  // capabilities", over 24 named service providers. No address set is enumerated;
  // the same paper's other instrument, TLS-Attacker, is already lab-protocol-test.
  'DTLS-MitM-Scanner (DMS)': 'lab-protocol-test',
  'EAST': 'active-scan',                          // STARTTLS test cases against live MTAs
  'Enemy of the State': 'vuln-scanner',
  'ERIS': 'active-scan',
  // Two spellings, two papers, two uses — and HAND is keyed by the exact string,
  // so they can and must differ. USENIX 2016 "Identifying and Characterizing
  // Sybils in the Tor Network" runs the scan: "We exposed malicious exit relays
  // using Winter et al.'s exitmap tool".
  'exitmap': 'active-scan',
  // HAND-AUDIT 2026-09-17: was 'active-scan'. WWW 2018 CrimeBB is a forum-scraping
  // paper; its studyTypes do not include network-scan-or-probe. It uses Exitmap
  // only for the node list: "We first use Exitmap [45] to get current available
  // exit nodes, from which we build Tor circuits and fetch the index page of the
  // forums." Enumerating an overlay's nodes is what p2p-crawler is for.
  'Exitmap': 'p2p-crawler',
  'FastMDA': 'topology',
  'FingerprinTLS': 'passive',
  'FlashRoute': 'topology',
  'forwarders-based scanner': 'active-scan',
  'gateway checker': 'active-scan',
  'Google Tsunami': 'vuln-scanner',
  'gQUIC-module': 'grabber',
  'h2spec': 'lab-protocol-test',
  'HP WebInspect': 'vuln-scanner',
  'HTTP/OneProbe': 'topology',
  'IBM Rational AppScan': 'vuln-scanner',
  'ibportstate': 'local-link',
  'ICMP Paris': 'topology',
  'ICMP_Sonar': 'active-scan',
  'iffinder': 'topology',
  'iGreedy': 'topology',
  'ike-scan': 'grabber',
  'Internet Census': 'dataset',
  'iperf': 'perf',
  'iPerf3': 'perf',
  'IPFS Crawler': 'p2p-crawler',
  'IRBlock': 'censorship-probe',
  'IRLscanner': 'active-scan',
  'IRTT': 'perf',
  'Iterative Network Tracing (INT)': 'topology',
  'James': 'topology',
  'kapar': 'topology',
  'Karma': 'active-scan',
  'l2ping': 'local-link',
  'LACeS': 'topology',
  'LanDscAPe': 'active-scan',
  // HAND-AUDIT 2026-09-17: was 'topology'. IMC 2023 "Illuminating Router Vendor
  // Diversity": LFP is "a novel lightweight, accurate, and more complete remote
  // vendor fingerprinting methodology and tool", benchmarked as "at least as good
  // as Nmap while sending orders of magnitude fewer packets". It probes hosts to
  // learn what they say; it infers no path and resolves no alias. That is a
  // grabber, and this correction puts the paper INTO ACTIVE.
  'LFP': 'grabber',
  'LTE PROBE': 'local-link',
  'Lumscan': 'censorship-probe',
  'MacStealer': 'local-link',
  'MAnycast anycast census': 'topology',
  'MAnycast2': 'topology',
  'Mao et al. AS-path inference algorithm': 'routing',
  'MAP-IT': 'routing',
  'MarcoPolo': 'routing',
  'McAfee SECURE': 'vuln-scanner',
  'Mercator': 'topology',
  'Mint': 'censorship-probe',
  'modified ntpdedup': 'grabber',
  'Monocle': 'censorship-probe',
  'mrinfo': 'topology',
  'mrinfo-rec': 'topology',
  'N-Stalker QA Edition': 'vuln-scanner',
  'Namehelp': 'vantage-platform',
  'nc': 'active-scan',
  'Netcat': 'active-scan',
  'ndisc6': 'local-link',
  'NetfilterQueue': 'packet-crafting',
  'netperf': 'perf',
  'netstat': 'host-tool',
  'Netstat': 'host-tool',
  'Network Analyzer': 'topology',
  'NextTrace': 'topology',
  'node probing program': 'active-scan',
  'NodeFinder': 'p2p-crawler',
  'NR-Scope': 'local-link',
  'ns-3': 'simulation',
  'Nuclei': 'vuln-scanner',
  'nuttcp': 'perf',
  'OneProbe': 'topology',
  'onesixtyone': 'grabber',
  'onion-grab': 'grabber',
  'OSAVRoute': 'active-scan',
  'PciCheckerLite': 'vuln-scanner',
  'Perf': 'host-tool',
  'PLCScan': 'grabber',
  'Prefixscan': 'topology',
  'Python script': 'active-scan',                 // "Scan IPv4 space with DNS queries"
  'QScanner': 'grabber',
  'QualysGuard PCI': 'vuln-scanner',
  'quicgrabber': 'grabber',
  'RAINBOW': 'p2p-crawler',
  'Rapid7 NeXpose': 'vuln-scanner',
  'RAT-Scan': 'active-scan',
  'RDScan': 'dns',
  'revtr 2.0': 'topology',
  'RMap': 'dns',
  'SAD DNS': 'dns',
  'SadDNS': 'dns',
  'Scapy': 'packet-crafting',
  'SecSpider': 'dataset',
  'Shadow': 'simulation',
  'ShaperProbe': 'perf',
  'Skipfish': 'vuln-scanner',
  'smbmap': 'grabber',
  // HAND-AUDIT 2026-09-17: was 'grabber'. IMC 2025 "Replication: Characterizing
  // MPLS Tunnels" does not send the SNMPv3 probes; it reads CAIDA's ITDK: "we
  // mapped each IP address to an inferred router using the MIDAR, iffinder, and
  // SNMPv3 alias resolution graph included in the March 2025 ITDK". MIDAR and
  // iffinder, named in that same sentence, are already topology. ('dataset' is
  // the arguable alternative; the sentence's own company decided it.) This
  // correction takes the paper OUT of ACTIVE.
  'SNMPv3': 'topology',
  'Speedtrap': 'topology',
  'Spoki': 'passive',
  'Spoofer project': 'vantage-platform',
  'SSFNet': 'simulation',
  'Swordqiu': 'routing',
  'Time Series Latency Probing (TSLP)': 'topology',
  'TLS-Attacker': 'lab-protocol-test',
  'Tor DNSEL': 'dataset',
  'TorFlow': 'p2p-crawler',
  'tracebox': 'topology',
  'tracefilter': 'topology',
  'tracenet': 'topology',
  'Trufflehunter': 'dns',
  'Tsunami': 'vuln-scanner',
  'Ubertooth One': 'local-link',
  // HAND-AUDIT 2026-09-17: 'Unicorn' was here as 'active-scan'. Its one tuple is
  // now suppressed in scripts/tool_use_overrides.mjs — IMC 2024 "Have you SYN
  // me?" never ran it, it fingerprinted its traffic at a telescope — so the name
  // no longer reaches this map and an entry here would fail the staleness guard.
  'University of Michigan Daily Full IPv4 HTTPS Handshakes': 'dataset',
  'URHunter': 'dns',
  'v6disc.sh': 'local-link',
  'Verfploeter': 'topology',
  'Verloc': 'topology',
  'VerLoc': 'topology',
  'W3af': 'vuln-scanner',
  'Wapiti': 'vuln-scanner',
  'WeHeY': 'perf',
  'WhatWeb': 'grabber',
  'whois': 'registry',
  'WiFi Inspector': 'local-link',
  'WMap': 'vuln-scanner',
  'WPScan': 'vuln-scanner',
  'wrk': 'perf',
  'Xray': 'vuln-scanner',
  'YoDNS': 'dns',
  'ZreFinder': 'dns',
};
 
// A hand verdict is a family of one: its display name is the string itself.
//
// `category` is the tools[] category the name was found under. Hand verdicts
// and GENERIC families are only trusted under 'network-scanner', because those
// strings ("Python script", "Unicorn", "custom scanner", "nc") are ordinary
// words elsewhere in the tool vocabulary. Pass category === null to fold a bare
// string, which is what the self-test does.
// ---------------------------------------------------------------------------
// COMPOUND TUPLE STRINGS. One tools[] entry that names more than one
// instrument. FAMILIES is ordered and first-match-wins, so a compound string
// lands in exactly one family and the other instrument silently disappears —
// which is how this page and design:dns came to disagree by one paper about
// the size of the ZMap family (the two folds test ZGrab and ZMap in opposite
// order, so the same string went to different families). Each key is
// hand-read against the paper's own evidence quote. The sweep in
// scripts/reconcile_zmap.mjs fails if any NEW compound name appears whose
// parts fold to a family the whole string does not.
// Key is the lower-cased, trimmed tools[] name; value is the component names,
// each folded through the normal path.
export const COMPOUND_SPLIT = new Map([
  // CCS 2016 "A Surfeit of SSH Cipher Suites". evidence.quote: "We ran two
  // IPv4-address-space-wide scans for SSH servers using ZGrab/ZMap [13]." —
  // matched verbatim in data/fulltext/2016/CCS/a-surfeit-of-ssh-cipher-suites/
  // paper.cols.txt on 2026-09-17. Both instruments were used.
  ['zgrab/zmap', ['ZGrab', 'ZMap']],
]);
 
// Every verdict a tools[] name yields, compound names included. Deduplicated
// by family, because a paper naming one instrument twice is still one paper.
export function kindsOfName(raw, category = null) {
  const k = String(raw).trim().toLowerCase();
  const parts = COMPOUND_SPLIT.has(k) ? COMPOUND_SPLIT.get(k) : [raw];
  const seen = new Map();
  for (const part of parts) {
    const v = kindOfName(part, category);
    if (v && !seen.has(v.family)) seen.set(v.family, v);
  }
  return [...seen.values()];
}
 
export function kindOfName(raw, category = null) {
  const fam = foldScannerName(raw);
  if (fam) {
    if (GENERIC_FAMILIES.has(fam) && category !== null && category !== 'network-scanner') return null;
    return { family: fam, kind: KIND_OF.get(fam), via: 'family' };
  }
  const v = String(raw).trim();
  if (Object.prototype.hasOwnProperty.call(HAND, v)) {
    if (category !== null && category !== 'network-scanner') return null;
    return { family: v, kind: HAND[v], via: 'hand' };
  }
  return null;
}
 
export const ROUTED_TO = {
  'dns': 'design:dns',
  'vantage-platform': 'design:crawling_location',
  'vuln-scanner': 'security:web_vulnerabilities',
  'dataset': 'design:existing_datasets',
  'topology': 'design:ip_classification',
  'routing': 'design:ip_classification',
};
 
function selfTest() {
  const cases = [
    ['ZMap', 'ZMap', 'active-scan'],
    ['zmap', 'ZMap', 'active-scan'],
    ['ZMapv6', 'ZMap', 'active-scan'],
    ['zmap6', 'ZMap', 'active-scan'],
    ['ZMap-based script', 'ZMap', 'active-scan'],
    ['TCP Options module for ZMap', 'ZMap', 'active-scan'],
    ['ZGrab/ZMap', 'ZGrab', 'grabber'],
    ['zgrab2', 'ZGrab', 'grabber'],
    ['ZGrab 2.0', 'ZGrab', 'grabber'],
    ['XMap', 'XMap', 'active-scan'],
    ['Xmap (extended)', 'XMap', 'active-scan'],
    ['Masscan', 'Masscan', 'active-scan'],
    ['Nmap Security Scanner', 'nmap', 'active-scan'],
    ['nmap plugin for Special DROWN', 'nmap', 'active-scan'],
    ['LZR', 'LZR', 'grabber'],
    ['Goscanner', 'Goscanner', 'grabber'],
    ['TLS-Crawler', 'TLS banner grabbers', 'grabber'],
    ['custom scanner', 'custom scanner', 'active-scan'],
    ['efficient scanning software', 'custom scanner', 'active-scan'],
    ['stateless scanner (custom)', 'custom scanner', 'active-scan'],
    ['systematic scanning tool', 'custom scanner', 'active-scan'],
    ['scanner application', 'custom scanner', 'active-scan'],
    ['ARGUS', 'tshark/tcpdump', 'passive'],
    ['Routeviews BGP table', 'RouteViews', 'routing'],
    ['Tnt', 'TNT', 'topology'],
    ['active scanning tool', 'active scanning tool', 'dns'],
    ['IPv6 Hitlist Service', 'IPv6 Hitlist Service', 'ipv6-targeting'],
    ['Hitlist-Downloader', 'IPv6 Hitlist Service', 'ipv6-targeting'],
    ['6Gen', '6Gen', 'ipv6-targeting'],
    ['6Sense', '6Sense', 'ipv6-targeting'],
    ['addr6', 'addr6', 'ipv6-targeting'],
    ['ping', 'ping', 'reachability'],
    ['hping3', 'ping', 'reachability'],
    ['fping', 'ping', 'reachability'],
    ['ICMP ping', 'ping', 'reachability'],
    ['TCP pings', 'ping', 'reachability'],
    ['Censys', 'Censys', 'dataset'],
    ['Censys Search API', 'Censys', 'dataset'],
    ['ZoomEye', 'ZoomEye', 'dataset'],
    ['Zoomeye', 'ZoomEye', 'dataset'],
    ['FOFA', 'FOFA', 'dataset'],
    ['Fofa', 'FOFA', 'dataset'],
    ['Rapid7 Sonar', 'Rapid7 Open Data', 'dataset'],
    ['Yarrp6', 'Yarrp', 'topology'],
    ['Yarrpbox', 'Yarrp', 'topology'],
    ['extended Yarrpbox', 'Yarrp', 'topology'],
    ['scamper Python module', 'scamper', 'topology'],
    ['Paris traceroute', 'Paris traceroute', 'topology'],
    ['tcptraceroute', 'traceroute', 'topology'],
    ['nmap traceroute', 'nmap', 'active-scan'],  // nmap wins: it is the instrument
    ['PyTNT', 'TNT', 'topology'],
    ['TNT', 'TNT', 'topology'],
    ['Route Views', 'RouteViews', 'routing'],
    ['RIPE Atlas', 'RIPE Atlas', 'vantage-platform'],
    ['public looking glasses', 'looking glass', 'vantage-platform'],
    ['Periscope', 'looking glass', 'vantage-platform'],
    ['dig', 'dig/drill', 'dns'],
    ['drill', 'dig/drill', 'dns'],
    ['DNSViz', 'dig/drill', 'dns'],
    ['Bro IDS', 'Zeek/Bro', 'passive'],
    ['p0f3', 'p0f', 'passive'],
    ['Tshark', 'tshark/tcpdump', 'passive'],
    ['Zed Attack Proxy (ZAP)', 'OWASP ZAP', 'vuln-scanner'],
    ['OWASP ZAP', 'OWASP ZAP', 'vuln-scanner'],
    ['Nikto', 'Nikto', 'vuln-scanner'],
    ['Quack', 'Quack', 'censorship-probe'],
    ['Satellite', 'Satellite', 'censorship-probe'],
    ['OONI Probe', 'OONI', 'censorship-probe'],
    // hand verdicts
    ['LanDscAPe', 'LanDscAPe', 'active-scan'],
    ['Scapy', 'Scapy', 'packet-crafting'],
    ['iperf', 'iperf', 'perf'],
    ['Ubertooth One', 'Ubertooth One', 'local-link'],
    ['whois', 'whois', 'registry'],
  ];
  for (const [raw, wantFam, wantKind] of cases) {
    const got = kindOfName(raw);
    if (!got || got.family !== wantFam || got.kind !== wantKind) {
      throw new Error(
        `scan_fold self-test: ${JSON.stringify(raw)} -> ${JSON.stringify(got)}, want ${wantFam}/${wantKind}`
      );
    }
  }
  // NEGATIVE guard: /z-?map/i matches an unrelated visualisation tool.
  if (foldScannerName('WizMap') !== null) throw new Error('scan_fold self-test: WizMap must not fold');
  // GENERIC and HAND names are category-restricted; distinctive names are not.
  if (kindOfName('custom scanner', 'other') !== null) throw new Error('scan_fold self-test: generic family must not fire outside network-scanner');
  if (kindOfName('custom scanner', 'network-scanner') === null) throw new Error('scan_fold self-test: generic family must fire inside network-scanner');
  if (kindOfName('Python script', 'other') !== null) throw new Error('scan_fold self-test: hand verdict must not fire outside network-scanner');
  if (kindOfName('ZMap', 'infrastructure') === null) throw new Error('scan_fold self-test: distinctive name must fire in any category');
  if (kindOfName('LEGO Mindstorms NXT') !== null) throw new Error('scan_fold self-test: unknown name must stay residue');
  for (const k of Object.values(HAND)) {
    if (typeof k !== 'string' || !k.length) throw new Error('scan_fold self-test: empty HAND kind');
  }
  if (MEMBERSHIP_KINDS.has('dataset')) throw new Error('scan_fold self-test: dataset must not be membership');
}
 
selfTest();

report_internet_scanning.mjs

#!/usr/bin/env node
// Every figure on programming:internet_scanning, with its denominator.
//
// Population boundary is written down in scripts/scan_fold.mjs and was fixed
// before the first count. Two denominators, never mixed:
//   SCAN   930 papers tagged studyTypes 'network-scan-or-probe'
//   ACTIVE 242 papers naming an active-scan / grabber / reachability /
//          censorship-probe / IPv6-targeting instrument they used or produced
//          (245 before the 2026-09-17 hand audit of the HAND verdicts and the
//          tuple suppression in tool_use_overrides.mjs)
//
// Flags: --wiki (DokuWiki tables)  --list <kind>  --quotes <regex>  --hits <regex>
//
// Exits 1 if a corpus contract, the fold coverage check, or a hand-map
// consistency check fails.
 
import fs from 'node:fs';
import path from 'node:path';
import { loadExtractions, pct, table, wikiTable, isSentinel, dataRoot, YEAR_BUCKETS } from './lib.mjs';
import { kindsOfName, MEMBERSHIP_KINDS, HAND, KIND_OF, ROUTED_TO, GENERIC_FAMILIES } from './scan_fold.mjs';
import { notRun, NOT_RUN, checkNotRun } from './tool_use_overrides.mjs';
 
const argv = process.argv.slice(2);
const WIKI = argv.includes('--wiki');
const opt = (n) => { const i = argv.indexOf(n); return i >= 0 ? argv[i + 1] : null; };
const LIST = opt('--list');
const QUOTES = opt('--quotes');
const HITS = opt('--hits');
 
const out = [];
const say = (s = '') => out.push(s);
const h = (s) => { say(''); say('='.repeat(78)); say(s); say('='.repeat(78)); };
const t = (headers, rows) => say(WIKI ? wikiTable(headers, rows) : table(headers, rows));
const fails = [];
const fail = (m) => { fails.push(m); say(`FAILURE: ${m}`); };
 
// ---------------------------------------------------------------- corpus ----
const rows = loadExtractions();
const DROVE = new Set(['used', 'produced']);
const key = (p) => `${p.venue}/${p.year}/${p.slug}`;
 
if (rows.length !== 5859) fail(`corpus ${rows.length} != 5859`);
 
// A NOT_RUN entry that matches nothing is a lie about what was read.
for (const m of checkNotRun(rows)) fail(m);
 
const SCAN = rows.filter((p) => p.studyTypes.includes('network-scan-or-probe'));
if (SCAN.length !== 930) fail(`network-scan-or-probe ${SCAN.length} != 930 (OVERVIEW.md)`);
 
// Every instrument a paper names, folded. Category is passed through so the
// generic families and the hand map stay inside category 'network-scanner'.
function instruments(p) {
  const seen = new Map(); // family -> {family, kind, via, names:Set, versions:Set}
  for (const tool of p.tools) {
    if (!DROVE.has(tool.usedOrMentioned)) continue;
    if (notRun(p, tool.name)) continue;
    for (const v of kindsOfName(tool.name, tool.category)) {
      if (!seen.has(v.family)) seen.set(v.family, { ...v, names: new Set(), versions: new Set(), cats: new Set() });
      const e = seen.get(v.family);
      e.names.add(String(tool.name).trim());
      e.cats.add(tool.category);
      if (!isSentinel(tool.version)) e.versions.add(String(tool.version));
    }
  }
  return [...seen.values()];
}
const memberOf = (p) => instruments(p).filter((v) => MEMBERSHIP_KINDS.has(v.kind));
const ACTIVE = rows.filter((p) => memberOf(p).length > 0);
const ACTIVE_KEYS = new Set(ACTIVE.map(key));
const SCAN_KEYS = new Set(SCAN.map(key));
 
// --------------------------------------------------------------- probes ----
// Two widths per practice. The loose one is the recall claim; the tight one
// requires the authors to be the subject of the sentence. Both are upper
// bounds, and section 8 measures the precision of the tight one by hand.
const PROBE = {
  rateLoose: /(rate[- ]?limit|packets? per second|\bpps\b|probing rate|scan(ning)? rate|bandwidth (of|cap|limit|is set)|--bandwidth|-B \d)/i,
  rateTight: /((we|our (scan|scanner|probe|measurement|experiment)s?)[^.]{0,120}(rate[- ]?limit|limit(ed)? (the |our )?(scan|probe|probing|sending|bandwidth|rate)|throttl)|(we|our)[^.]{0,80}(scan|prob|sent|send)[^.]{0,80}at (a )?(rate of |about |approximately )?\d[\d,.]*\s?(k|m|g)?(bps|b\/s|pps|packets)|(probing|scan(ning)?|sending) rate of \d|limit(ed)? (the |our )?(scan|probe|probing|bandwidth|rate)[^.]{0,60}\d)/i,
  blockLoose: /(black ?list|block ?list|exclusion (list|file)|excluded? (address|IP|prefix|network|range)|--blacklist-file|--blocklist-file)/i,
  blockTight: /((we|our)[^.]{0,140}(black ?list|block ?list|exclusion (list|file))|(black ?list|block ?list|exclusion list)[^.]{0,100}(we |our |scan)|--black(list)?-file|--block(list)?-file|honou?r(ed|ing)?[^.]{0,40}(black ?list|block ?list|exclusion|opt-?out|removal)|exclud(e|ed|ing)[^.]{0,80}(IP|address|prefix|network|range|subnet)[^.]{0,80}(scan|prob|request|our))/i,
  optLoose: /(opt(ed)?[- ]out|removal request|asked (us )?to be excluded)/i,
  optTight: /((we|our)[^.]{0,140}opt(ed)?[- ]out|opt[- ]out (request|list|form|page|mechanism)|(request(ed|s)?|asked)[^.]{0,60}(be )?(excluded|removed) (from|our)|removal request)/i,
  srcLoose: /(source (IP|address)|dedicated (IP|machine|server|host)|scanning (host|machine|server|IP)|reverse DNS|rDNS|PTR record|WHOIS)/i,
  srcTight: /((we|our)[^.]{0,120}(dedicated (IP|machine|server|host|address)|scanning (host|machine|server|IP|address)|(source|scanner) (IP|address)(es)?|our (scanner|scanning) (IP|host|machine))|(reverse DNS|rDNS|PTR record|WHOIS (record|entry|entries))[^.]{0,120}(our |we |explain|scan))/i,
  abuseLoose: /(abuse (complaint|report|email|e-mail|contact|team)|abuse@|complaints?)/i,
  abuseTight: /((we|our)[^.]{0,140}(abuse (complaint|report|email|e-mail|contact)|complaint)|abuse@|(received|got) [^.]{0,40}(complaint|abuse))/i,
  irb: /(\bIRB\b|institutional review board|ethics (board|committee|review|approval)|REB approval)/i,
  goodpractice: /(Durumeric|Partridge and Allman|Menlo Report|scanning best practice|good (Internet|internet) citizen)/i,
};
 
// Hand verdicts. 'own' = the matched sentence is about the authors' own scan.
// 'other' = the term is in the paper for a different reason (a target's rate
// limiting, a censor's blocklist, a defence recommendation, an analysis-time
// exclusion). Read on 2026-09-10 from paper.cols.txt with whitespace collapsed,
// first two matches per paper. Every entry names the deciding reading.
const RATE_VERDICT = {
  'CCS/2015/a-search-engine-backed-by-internet-wide-scanning': 'other',   // rate-limits its own *web interface*, not its scan
  'CCS/2025/poster-eris-evaluating-rov-via-icmpv6-rate-limiting-side-channels': 'other', // ICMPv6 rate limiting is the side channel under study
  'IEEE-SP/2023/pcspoof-compromising-the-safety-of-time-triggered-ethernet': 'other', // "sending ARP requests at 100 Mbps" is an attack rate on a TTEthernet lab testbed, not a scan rate. Read as 'own' on the first pass and flipped after the generic review pass disputed it on 2026-09-10
  'IMC/2016/on-the-free-bridge-across-the-digital-divide-assessing-the-quality-of-facebooks': 'other', // Free Basics traffic is throttled
  'IMC/2022/saving-brians-privacy-the-perils-of-privacy-exposure-through-reverse-dns': 'own', // "We rate-limit requests to authoritative name servers"
  'IMC/2025/chaos-in-the-chain-evaluate-deployment-and-construction-compliance-of-web-pki-ce': 'own', // "we limited the scanning rate to below 500KB/s"
  'NDSS/2020/measuring-the-deployment-of-network-censorship-filters-at-global-scale': 'own', // "we abide by best practices and limit the rate of sending requests"
  'NDSS/2024/proof-of-backhaul-trustfree-measurement-of-broadband-bandwidth': 'other', // tc-shaped test rig, not a scan rate
  'USENIX/2011/toward-secure-embedded-web-interfaces': 'other', // the embedded device throttles requests
  'USENIX/2021/accurately-measuring-global-risk-of-amplification-attacks-using-ampmap': 'other', // "block or rate-limit specific queries" is the defence it recommends
  'USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications': 'own', // "a low scanning rate of 10,000 packets per second"
  'USENIX/2025/exposing-and-circumventing-sni-based-quic-censorship-of-the-great-firewall-of-ch': 'own', // "at a rate of 100 packets per second"
};
const BLOCK_VERDICT = {
  'CCS/2018/practical-state-recovery-attacks-against-legacy-rng-implementations': 'own', // "respecting hosts who wished to be blacklisted"
  'IEEE-SP/2017/augur-internet-wide-detection-of-connectivity-disruptions': 'other', // block lists of *censored sites*, used as validation data
  'IMC/2011/detecting-validating-and-characterizing-computer-infections-in-the-wild': 'other', // blacklists are an input dataset
  'IMC/2016/measuring-the-security-harm-of-tls-crypto-shortcuts': 'own', // "followed the institutional blacklist"
  'IMC/2018/clusters-in-the-expanse-understanding-and-unbiasing-ipv6-hitlists': 'own', // "maintaining a blacklist and using dedicated servers"
  'IMC/2022/deep-dive-into-the-iot-backend-ecosystem': 'other', // FireHOL blocklist is the object of measurement
  'IMC/2023/ecn-with-quic-challenges-in-the-wild': 'own', // "how to be added to our blocklist to be excluded from further measurements"
  'NDSS/2020/decentralized-control-a-case-study-of-russia': 'other', // Roskomnadzor blocklists are the subject
  'NDSS/2025/revealing-the-black-box-of-device-search-engine-scanning-assets-strategies-and-ethical-consideration': 'other', // operators blocklisting the scanners it studies
  'USENIX/2014/an-internet-wide-view-of-internet-wide-scanning': 'own', // "only 208 organizations have requested that we exclude their networks from our scans" — 208, checked against paper.pdf with pypdf; a first read of a narrow context window misread it as 908
  'USENIX/2018/quack-scalable-remote-measurement-of-application-layer-censorship': 'own', // "respecting an IP blacklist shared with other scanning research"
  // DROPPED 2026-09-17, was 'other' ("analysis-time exclusion of known scan
  // projects"). The reading still stands, but the paper left ACTIVE when its
  // only membership tuple — a ZMap run over a closed veth loop — was
  // suppressed (see scripts/tool_use_overrides.mjs), so it is no longer a hit
  // and the pinned-sample guard rejected it. The sample is 11, not 12.
};
const OPT_VERDICT = {
  'CCS/2015/a-search-engine-backed-by-internet-wide-scanning': 'own', // "we treat opt-out requests for scanning as a request to be removed from the search index"
  'IEEE-SP/2020/meddling-middlemen-empirical-analysis-of-the-risks-of-data-saving-mobile-browser': 'other', // browser users opting out of data saving
  'IMC/2010/demystifying-service-discovery-implementing-an-internet-wide-scanner': 'own', // TXT records pointing at a page with opt-out instructions
  'IMC/2018/in-the-ip-of-the-beholder-strategies-for-active-ipv6-topology-discovery': 'own', // "we received two opt-out requests with which we immediately complied"
  'IMC/2022/deep-dive-into-the-iot-backend-ecosystem': 'own', // "We run a Web server with experiment and opt-out information"
  'IMC/2024/destination-reachable-what-icmpv6-error-messages-reveal-about-their-sources': 'own', // "We received a single request for opt-out and acted accordingly"
  'NDSS/2024/understanding-the-implementation-and-security-implications-of-protective-dns-services': 'own', // PTR records giving contacts for opt-out
  'PETS/2023/certainty-detecting-dns-manipulation-at-scale-using-tls-certificates': 'own', // "offer administrators the option to opt out of our scanning"
  'USENIX/2018/quack-scalable-remote-measurement-of-application-layer-censorship': 'own', // "simple ways for packet recipients to opt out of future probes"
  'USENIX/2024/landscape-exploring-ldap-weaknesses-and-data-leaks-at-internet-scale': 'own', // "a website gives project details and contact information for opt-out requests"
};
const SRC_VERDICT = {
  'CCS/2015/a-search-engine-backed-by-internet-wide-scanning': 'own', // "All of the scanning hosts have WHOIS records and reverse DNS entries"
  'IEEE-SP/2010/investigation-of-triangular-spamming-a-stealthy-and-efficient-spamming-technique': 'other', // source-IP rewriting is the attack under study
  'IMC/2010/demystifying-service-discovery-implementing-an-internet-wide-scanner': 'own', // describes its own source IPs and how many it used
  'IMC/2016/identifying-and-aggregating-homogeneous-ipv4-24-blocks-with-hobbit': 'other', // rDNS patterns of the *targets*
  'IMC/2019/an-end-to-end-large-scale-measurement-of-dns-over-encryption-how-far-have-we-com': 'own', // "setting a reverse DNS record for our scanning system and building a website"
  'IMC/2023/does-it-spin-on-the-adoption-and-use-of-quics-spin-bit': 'own', // rDNS entries, project name in every request, dedicated website
  'NDSS/2020/on-using-application-layer-middlebox-protocols-for-peeking-behind-nat-gateways': 'own', // "The reverse DNS record of the scanning host was set to indicate its use for research purposes"
  'NDSS/2026/osavroute-advancing-outbound-source-address-validation-deployment-detection-with-non-cooperative-measurement': 'other', // source-address validation is the subject
  'USENIX/2021/lzr-identifying-unexpected-internet-services': 'other', // discusses scanners' source IPs as a defence property
  'USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications': 'own', // "The webpage hosted on each scanning IP/domain includes a contact email"
};
 
// Every non-corpus number that appears on the page, so check_page_numbers.mjs
// can run whole-page. Re-checked by scripts/external_checks_internet_scanning.sh.
const EXTERNAL = [
  'ZMap v4.4.0, released 2026-05-29; repo zmap/zmap not archived, pushed 2026-08-28',
  'ZGrab2 v1.0.0, released 2025-12-04; repo zmap/zgrab2 pushed 2026-09-07',
  'ZDNS v2.1.1, released 2026-05-28; repo zmap/zdns pushed 2026-09-09',
  'masscan latest GitHub Release 1.3.2, 2021-01-31; repo pushed 2026-04-23, not archived',
  'XMap 2.0.6, 2026-03-10; repo idealeer/xmap pushed 2026-03-10',
  'LZR repo stanford-esrg/lzr pushed 2026-01-06, not archived',
  'nmap 7.991 is the newest tarball on nmap.org/dist',
  'ZMap v4.4.0 has ONE --blocklist-file flag (-b); there is no --blocklist-file-extra. The port flag is -p/--target-ports. ziterate, zblocklist and ztee ship in src/ alongside zmap',
  'ZGrab2 input is CSV "IP, DOMAIN, TAG, PORT"; the multiple module takes a .ini',
  'ZMap changed to random per-probe IP IDs in early 2024 (durumeric2024_years section 3)',
  'ZMap defaults (v4.4.0 and main agree): --rate 10000 pps, --probes 1, --cooldown-time 8, --retries 10, --batch 64',
  'ZMap conf/blocklist.conf ships 16 CIDR entries, all IANA special-purpose and multicast ranges, header dated 2013',
  'IPv6 Hitlist Service open files, snapshot 2026-09-05: responsive-addresses 29,050,771 lines; aliased-prefixes 24,476,266; non-aliased-prefixes 222,836,153',
  'Rapid7 Project Sonar: opendata.rapid7.com now redirects to sonardata.rapid7.com, which is commercial access only (8 datasets, 47,457 files, 60.7 TB)',
  'search.censys.io returns HTTP 403 to automated clients; censys.com returns 200',
  '2^32 = 4,294,967,296 IPv4 addresses; the shipped blocklist removes 592,708,864 (13.8%), leaving 3,702,258,432 — computed by pages/scan_budget.py from the live file, collapsing overlaps (255.255.255.255/32 is inside 240.0.0.0/4, so a naive sum gives one address more)',
  'pages/scan_budget.py --self-test: 9 checks (check 9 covers human_time() boundaries). Default run: 4.29 days at 10,000 pps, 10.3 h at 100,000 pps, 41.5 min at 1 GbE line rate (1,488,095 pps), 4.1 min at 10 GbE (14,880,952 pps), on 84 bytes of wire time per probe',
];
 
h('0. FOLD COVERAGE — every category=network-scanner name must be decided');
{
  const un = new Map();
  const usedHand = new Set();
  const usedFam = new Set();
  let tuples = 0;
  for (const p of rows) {
    for (const tool of p.tools) {
      if (!DROVE.has(tool.usedOrMentioned) || tool.category !== 'network-scanner') continue;
      if (notRun(p, tool.name)) continue;
      tuples += 1;
      const vs = kindsOfName(tool.name, tool.category);
      if (!vs.length) { un.set(String(tool.name).trim(), (un.get(String(tool.name).trim()) || 0) + 1); continue; }
      for (const v of vs) {
        if (v.via === 'hand') usedHand.add(String(tool.name).trim()); else usedFam.add(v.family);
      }
    }
  }
  say(`category='network-scanner' used/produced tuples: ${tuples}`);
  say(`unmapped strings: ${un.size}`);
  for (const [k, v] of [...un].sort()) say(`  UNMAPPED ${v}  ${JSON.stringify(k)}`);
  if (un.size > 0) fail(`${un.size} unmapped network-scanner strings — every one must get a family or a HAND verdict`);
  const stale = Object.keys(HAND).filter((k) => !usedHand.has(k));
  say(`HAND entries: ${Object.keys(HAND).length}, matched: ${usedHand.size}, stale: ${stale.length}`);
  for (const s of stale) say(`  STALE HAND ${JSON.stringify(s)}`);
  if (stale.length > 0) fail(`${stale.length} HAND entries match nothing — a shadowed verdict is a lie about what was read`);
}
 
h('0b. IS THE ZMAP TOOL PAPER IN THE CORPUS? (the page says no, twice)');
{
  const inExtract = rows.filter((p) => p.year === 2013 && p.venue === 'USENIX' && /zmap|internet-wide-scanning/i.test(p.slug));
  const ftDir = path.join(dataRoot(), 'fulltext/2013/USENIX');
  const ftHits = fs.existsSync(ftDir) ? fs.readdirSync(ftDir).filter((x) => /zmap|scann/i.test(x)) : [];
  const metaFile = path.join(dataRoot(), 'corpus2/.meta/USENIX-2013.json');
  const meta = JSON.parse(fs.readFileSync(metaFile, 'utf8'));
  const recs = Array.isArray(meta) ? meta : (meta.papers ?? meta.records ?? []);
  const metaHits = recs.filter((r) => /zmap/i.test(JSON.stringify(r)));
  say(`extraction records matching /zmap|internet-wide-scanning/i in USENIX 2013: ${inExtract.length}`);
  say(`data/fulltext/2013/USENIX directories matching /zmap|scann/i:            ${ftHits.length}`);
  say(`corpus2/.meta/USENIX-2013.json records matching /zmap/i:                 ${metaHits.length} of ${recs.length}`);
  say(`PUBLISHED_ZMAP2013_ABSENT ${inExtract.length === 0 && ftHits.length === 0 && metaHits.length === 0 ? 'yes' : 'NO — the page says it is absent'}`);
  say(`PUBLISHED_USENIX2013_INDEX ${recs.length}`);
  if (inExtract.length || ftHits.length || metaHits.length) fail('the ZMap tool paper is in the corpus after all; the page says it is not');
}
 
h('1. POPULATIONS');
{
  const activeInScan = ACTIVE.filter((p) => SCAN_KEYS.has(key(p)));
  const activeOutside = ACTIVE.filter((p) => !SCAN_KEYS.has(key(p)));
  // What the tools[].category restriction alone would give.
  const nsOnly = rows.filter((p) =>
    p.tools.some((tool) => {
      if (!DROVE.has(tool.usedOrMentioned) || tool.category !== 'network-scanner') return false;
      if (notRun(p, tool.name)) return false;
      return kindsOfName(tool.name, tool.category).some((v) => MEMBERSHIP_KINDS.has(v.kind));
    })
  );
  const namesNS = SCAN.filter((p) =>
    p.tools.some((tool) => DROVE.has(tool.usedOrMentioned) && tool.category === 'network-scanner'
      && !notRun(p, tool.name))
  );
  t(['Population', 'Papers', 'Share'], [
    ['corpus (7 venues, 2010–2026)', rows.length, '100%'],
    ["SCAN — studyTypes 'network-scan-or-probe'", SCAN.length, pct(SCAN.length, rows.length)],
    ['ACTIVE — names an active-scan instrument (page population)', ACTIVE.length, `${pct(ACTIVE.length, rows.length)} of corpus`],
    ['  … of those, inside SCAN', activeInScan.length, pct(activeInScan.length, SCAN.length) + ' of SCAN'],
    ['  … of those, outside SCAN', activeOutside.length, pct(activeOutside.length, ACTIVE.length) + ' of ACTIVE'],
    ["SCAN papers naming ANY category='network-scanner' tool", namesNS.length, pct(namesNS.length, SCAN.length)],
    ["SCAN papers naming NO category='network-scanner' tool", SCAN.length - namesNS.length, pct(SCAN.length - namesNS.length, SCAN.length)],
  ]);
  say(`PUBLISHED_CORPUS ${rows.length}`);
  say(`PUBLISHED_SCAN ${SCAN.length}`);
  say(`PUBLISHED_ACTIVE ${ACTIVE.length}`);
  say(`PUBLISHED_ACTIVE_IN_SCAN ${activeInScan.length}`);
  say(`PUBLISHED_ACTIVE_IN_SCAN_PCT ${pct(activeInScan.length, SCAN.length)}`);
  say(`PUBLISHED_ACTIVE_OUTSIDE ${activeOutside.length}`);
  say(`PUBLISHED_NAMES_NS ${namesNS.length}`);
  say(`PUBLISHED_NO_NS ${SCAN.length - namesNS.length} ${pct(SCAN.length - namesNS.length, SCAN.length)}`);
  say('');
  say(`category-restricted membership would give ${nsOnly.length}; the any-category rule adds ${ACTIVE.length - nsOnly.length}`);
  say(`PUBLISHED_NS_ONLY ${nsOnly.length}`);
  const added = ACTIVE.filter((p) => !nsOnly.some((q) => key(q) === key(p)));
  for (const p of added) {
    const fam = memberOf(p).map((v) => `${v.family}[${[...v.cats].join('/')}]`).join(', ');
    say(`  +${key(p)}  ${fam}`);
  }
}
 
h('2. KIND UNION over SCAN (a paper can name several kinds)');
{
  const kinds = new Map();
  for (const p of SCAN) for (const k of new Set(instruments(p).map((v) => v.kind))) kinds.set(k, (kinds.get(k) || 0) + 1);
  t(['kind', 'membership?', 'Papers', 'Share of 930', 'routed to'], [...kinds.entries()]
    .sort((a, b) => b[1] - a[1])
    .map(([k, v]) => [k, MEMBERSHIP_KINDS.has(k) ? 'YES' : '', v, pct(v, SCAN.length), ROUTED_TO[k] ?? '']));
  const none = SCAN.filter((p) => instruments(p).length === 0);
  say('');
  say(`SCAN papers whose tools[] name no instrument this fold recognises: ${none.length} (${pct(none.length, SCAN.length)})`);
  say(`PUBLISHED_SCAN_NO_INSTRUMENT ${none.length} ${pct(none.length, SCAN.length)}`);
}
 
h('3. THE INSTRUMENTS — membership families over ACTIVE');
{
  const fam = new Map();
  const spell = new Map();
  for (const p of ACTIVE) {
    for (const v of memberOf(p)) {
      if (!fam.has(v.family)) { fam.set(v.family, new Set()); spell.set(v.family, new Set()); }
      fam.get(v.family).add(key(p));
      for (const n of v.names) spell.get(v.family).add(n);
    }
  }
  const ordered = [...fam.entries()].sort((a, b) => b[1].size - a[1].size || a[0].localeCompare(b[0]));
  t(['Family', 'kind', 'Papers', 'Share of ACTIVE', 'distinct spellings'], ordered
    .filter(([, s]) => s.size >= 2)
    .map(([f, s]) => [f, KIND_OF.get(f) ?? HAND[f], s.size, pct(s.size, ACTIVE.length), spell.get(f).size]));
  const ones = ordered.filter(([, s]) => s.size === 1);
  say('');
  say(`families named by exactly one paper: ${ones.length} — ${ones.map(([f]) => f).join(', ')}`);
  say(`PUBLISHED_ONEOFF_FAMILIES ${ones.length}`);
  for (const [f, s] of ordered.slice(0, 8)) say(`PUBLISHED_FAM_${f.replace(/\W+/g, '_').toUpperCase()} ${s.size}`);
  say('');
  // The roadmap queued this page on "143 name ZMap/ZGrab/masscan/nmap". Re-derive it.
  {
    const core = new Set(['ZMap', 'ZGrab', 'Masscan', 'nmap']);
    const hasCore = (p) => instruments(p).some((v) => core.has(v.family));
    const all = rows.filter(hasCore);
    const inScan = all.filter((p) => SCAN_KEYS.has(key(p)));
    say('');
    say(`ZMap|ZGrab|Masscan|nmap, used or produced: ${all.length} corpus-wide, ${inScan.length} inside SCAN (the roadmap row said 143)`);
    say(`PUBLISHED_CORE_FOUR_ALL ${all.length}`);
    say(`PUBLISHED_CORE_FOUR_SCAN ${inScan.length}`);
  }
  say('ZMap spellings: ' + [...spell.get('ZMap')].sort().join(' | '));
  say(`PUBLISHED_ZMAP_SPELLINGS ${spell.get('ZMap').size}`);
  // Home-grown: a membership instrument whose family is 'custom scanner' or a
  // one-paper coined name found only in this paper.
  const homegrown = ACTIVE.filter((p) => memberOf(p).some((v) => v.family === 'custom scanner' || (v.via === 'hand' && fam.get(v.family).size === 1 && [...p.tools].some((tool) => tool.usedOrMentioned === 'produced' && String(tool.name).trim() === v.family))));
  const customOnly = ACTIVE.filter((p) => memberOf(p).some((v) => v.family === 'custom scanner'));
  const oneoffProduced = ACTIVE.filter((p) => memberOf(p).some((v) => v.via === 'hand' && fam.get(v.family).size === 1 && p.tools.some((tool) => tool.usedOrMentioned === 'produced' && String(tool.name).trim() === v.family)));
  say(`ACTIVE papers whose scanner is home-grown: ${homegrown.length} (${pct(homegrown.length, ACTIVE.length)})`);
  say(`  = generic 'custom scanner' string:                    ${customOnly.length}`);
  say(`  + a one-paper name the paper itself PRODUCED:         ${oneoffProduced.length}`);
  say(`  - overlap:                                            ${customOnly.length + oneoffProduced.length - homegrown.length}`);
  say(`  (the other ${ones.length - oneoffProduced.length} one-paper families are third-party tools only this corpus's one paper used)`);
  say(`PUBLISHED_HOMEGROWN ${homegrown.length} ${pct(homegrown.length, ACTIVE.length)}`);
  say(`PUBLISHED_HOMEGROWN_CUSTOM ${customOnly.length}`);
  say(`PUBLISHED_HOMEGROWN_PRODUCED ${oneoffProduced.length}`);
  say(`PUBLISHED_ONEOFF_THIRDPARTY ${ones.length - oneoffProduced.length}`);
  if (LIST === 'homegrown') for (const p of homegrown) say(`  ${key(p)}`);
}
 
h('4. CURRENCY — membership families by year bucket, share of that bucket\'s ACTIVE papers');
{
  const FAMS = ['ZMap', 'ZGrab', 'nmap', 'XMap', 'Masscan', 'custom scanner', 'ping'];
  const hdr = ['Window', 'ACTIVE papers', ...FAMS];
  const body = YEAR_BUCKETS.map(([label, inb]) => {
    const b = ACTIVE.filter((p) => inb(p.year));
    return [label, b.length, ...FAMS.map((f) => {
      const n = b.filter((p) => memberOf(p).some((v) => v.family === f)).length;
      return `${n} (${pct(n, b.length)})`;
    })];
  });
  t(hdr, body);
  say('');
  say('Per year, ZMap and the whole ACTIVE population (2026 is provisional):');
  const years = [...new Set(rows.map((p) => p.year))].sort();
  t(['Year', 'corpus', 'SCAN', 'ACTIVE', 'ZMap'], years.map((y) => [
    y,
    rows.filter((p) => p.year === y).length,
    SCAN.filter((p) => p.year === y).length,
    ACTIVE.filter((p) => p.year === y).length,
    ACTIVE.filter((p) => p.year === y && memberOf(p).some((v) => v.family === 'ZMap')).length,
  ]));
  const zFirst = Math.min(...ACTIVE.filter((p) => memberOf(p).some((v) => v.family === 'ZMap')).map((p) => p.year));
  say(`first ZMap year in ACTIVE: ${zFirst}`);
  say(`PUBLISHED_ZMAP_FIRST_YEAR ${zFirst}`);
}
 
h('5. WHERE SCANNING PUBLISHES — venue');
{
  const venues = [...new Set(rows.map((p) => p.venue))].sort();
  t(['Venue', 'Papers', 'SCAN', 'SCAN share of venue', 'ACTIVE', 'ACTIVE share of venue'], venues.map((v) => {
    const all = rows.filter((p) => p.venue === v);
    const s = all.filter((p) => SCAN_KEYS.has(key(p)));
    const a = all.filter((p) => ACTIVE_KEYS.has(key(p)));
    return [v, all.length, s.length, pct(s.length, all.length), a.length, pct(a.length, all.length)];
  }));
}
 
// ---------------------------------------------------- full-text machinery ---
const ftCache = new Map();
function fulltextFlat(p) {
  const k = key(p);
  if (!ftCache.has(k)) {
    const f = path.join(dataRoot(), 'fulltext', String(p.year), p.venue, p.slug, 'paper.cols.txt');
    ftCache.set(k, fs.existsSync(f) ? fs.readFileSync(f, 'latin1').replace(/\s+/g, ' ') : null);
  }
  return ftCache.get(k);
}
function probe(pop, re) {
  const hits = [];
  let missing = 0;
  for (const p of pop) {
    const txt = fulltextFlat(p);
    if (txt === null) { missing += 1; continue; }
    if (re.test(txt)) hits.push(p);
  }
  return { hits, missing };
}
function ctx(p, re, width = 300) {
  const txt = fulltextFlat(p);
  if (txt === null) return [];
  const g = new RegExp(re.source, re.flags.includes('g') ? re.flags : re.flags + 'g');
  const o = [];
  let m;
  while ((m = g.exec(txt)) !== null) {
    o.push(txt.slice(Math.max(0, m.index - width / 2), m.index + width / 2));
    if (o.length >= 6) break;
  }
  return o;
}
 
h('6. WHAT A SCAN PAPER REPORTS — schema fields, denominator ACTIVE and SCAN');
{
  const stated = {
    'vantage location': (p) => p.vantage.some((v) => (v.locations ?? []).some((l) => !isSentinel(l))),
    'vantage infrastructure': (p) => p.vantage.some((v) => !isSentinel(v.infrastructure)),
    'any used/produced tool version': (p) => p.tools.some((tool) => DROVE.has(tool.usedOrMentioned) && !isSentinel(tool.version)),
    'version of the SCANNER itself': (p) => memberOf(p).some((v) => v.versions.size > 0),
    // `artifacts` and `ethics` are objects that can legitimately be absent
    // (320 and 894 papers corpus-wide). An absent record is NOT a stated value
    // and NOT a sentinel: it is its own row, printed below the table.
    'own artifact URL': (p) => p.artifacts !== null && !isSentinel(p.artifacts.availability) && (p.artifacts.codeUrl !== null || p.artifacts.dataUrl !== null || p.artifacts.links.length > 0),
    'ethics review outcome': (p) => p.ethics !== null && !isSentinel(p.ethics.reviewOutcome),
    'notified affected parties': (p) => p.ethics !== null && !isSentinel(p.ethics.notifiedAffectedParties),
    'harm mitigation described': (p) => p.ethics !== null && !isSentinel(p.ethics.harmMitigation),
    "robots.txt": (p) => p.ethics !== null && !isSentinel(p.ethics.robotsTxt),
    'temporal span start': (p) => p.temporal.some((x) => !isSentinel(x.spanStart)),
  };
  t(['Field', `ACTIVE (${ACTIVE.length})`, 'share', `SCAN (${SCAN.length})`, 'share'], Object.entries(stated).map(([label, f]) => {
    const a = ACTIVE.filter(f).length;
    const s = SCAN.filter(f).length;
    return [label, a, pct(a, ACTIVE.length), s, pct(s, SCAN.length)];
  }));
  const noEthics = ACTIVE.filter((p) => p.ethics === null).length;
  const noArt = ACTIVE.filter((p) => p.artifacts === null).length;
  say('');
  say(`ACTIVE papers with NO ethics record at all: ${noEthics} (${pct(noEthics, ACTIVE.length)}) — not counted as stated, not counted as a sentinel`);
  say(`ACTIVE papers with NO artifacts record at all: ${noArt} (${pct(noArt, ACTIVE.length)})`);
  say(`PUBLISHED_NO_ETHICS_RECORD ${noEthics} ${pct(noEthics, ACTIVE.length)}`);
  say(`PUBLISHED_NO_ARTIFACT_RECORD ${noArt} ${pct(noArt, ACTIVE.length)}`);
  for (const [label, f] of Object.entries(stated)) {
    say(`PUBLISHED_REPORT_${label.replace(/\W+/g, '_').toUpperCase()} ${ACTIVE.filter(f).length} ${pct(ACTIVE.filter(f).length, ACTIVE.length)} | SCAN ${SCAN.filter(f).length} ${pct(SCAN.filter(f).length, SCAN.length)}`);
  }
}
 
h('7. FULL-TEXT PROBES over ACTIVE — a loose upper bound and a tight estimate');
{
  const PAIRS = [
    ['states its own probe rate or rate limit', PROBE.rateLoose, PROBE.rateTight],
    ['honours a blocklist / exclusion list', PROBE.blockLoose, PROBE.blockTight],
    ['offers an opt-out', PROBE.optLoose, PROBE.optTight],
    ['source-address hygiene (dedicated IPs, rDNS/PTR, WHOIS, web page)', PROBE.srcLoose, PROBE.srcTight],
    ['abuse complaints received or handled', PROBE.abuseLoose, PROBE.abuseTight],
  ];
  const body = [];
  for (const [label, loose, tight] of PAIRS) {
    const l = probe(ACTIVE, loose);
    const tt = probe(ACTIVE, tight);
    body.push([label, `${l.hits.length} (${pct(l.hits.length, ACTIVE.length)})`, `${tt.hits.length} (${pct(tt.hits.length, ACTIVE.length)})`]);
    say(`PUBLISHED_PROBE_${label.replace(/\W+/g, '_').toUpperCase()}_LOOSE ${l.hits.length} ${pct(l.hits.length, ACTIVE.length)}`);
    say(`PUBLISHED_PROBE_${label.replace(/\W+/g, '_').toUpperCase()}_TIGHT ${tt.hits.length} ${pct(tt.hits.length, ACTIVE.length)}`);
  }
  t(['Practice', 'loose probe (upper bound)', 'first-person probe'], body);
  say('');
  say('Neither column is a population. The loose regex fires on any sentence in');
  say('the paper, including one about somebody else\'s rate limit or somebody');
  say('else\'s blocklist; the first-person regex requires "we"/"our" near the term');
  say('and still admits sentences about the authors\' own *subject*. Section 8');
  say('hand-reads a deterministic sample of each first-person hit list and gives');
  say('the measured precision.');
  const also = probe(ACTIVE, PROBE.irb).hits;
  const cite = probe(ACTIVE, PROBE.goodpractice).hits;
  say('');
  say(`ACTIVE papers naming an IRB / ethics board: ${also.length} (${pct(also.length, ACTIVE.length)}) [loose, no precision measured]`);
  say(`ACTIVE papers citing Durumeric / Partridge-Allman / Menlo / "good Internet citizen": ${cite.length} (${pct(cite.length, ACTIVE.length)}) [loose]`);
  say(`PUBLISHED_IRB ${also.length} ${pct(also.length, ACTIVE.length)}`);
  say(`PUBLISHED_GOODPRACTICE ${cite.length} ${pct(cite.length, ACTIVE.length)}`);
}
 
h('8. HAND-READ PRECISION — a PINNED sample of each first-person hit list');
{
  // The sample used to be recomputed as every-k-th of the sorted hit list. That
  // made the sample a function of ACTIVE: the 2026-09-17 hand audit moved ACTIVE
  // by three papers and the stride reshuffled all four samples, failing 46 of
  // these checks on papers that were still perfectly good hits — the read was
  // invalidated by arithmetic rather than by evidence. The sample is now the
  // verdict map itself: a frozen list of papers, each with the sentence that
  // decided it. The guards below are what keeps that honest.
  const CHECKS = [
    ['own probe rate', PROBE.rateTight, RATE_VERDICT, 84],
    ['own blocklist / exclusion', PROBE.blockTight, BLOCK_VERDICT, 90],
    ['own opt-out offer', PROBE.optTight, OPT_VERDICT, 69],
    ['own source-address hygiene', PROBE.srcTight, SRC_VERDICT, 114],
  ];
  const summary = [];
  for (const [label, re, verdicts, drawnFrom] of CHECKS) {
    const hits = probe(ACTIVE, re).hits.slice().sort((a, b) => key(a).localeCompare(key(b)));
    const hitKeys = new Set(hits.map(key));
    const sampleKeys = Object.keys(verdicts).sort();
    say('');
    say(`--- ${label}: ${hits.length} first-person hits of ${ACTIVE.length} ACTIVE; pinned sample of ${sampleKeys.length}`);
    say(`    sample drawn on 2026-09-10 as every-k-th of a hit list that was then ${drawnFrom} long`);
    // GUARD 1. Every pinned paper must still be a hit. If it is not, either the
    // probe changed or the paper left ACTIVE, and the verdict is about a paper
    // this figure no longer counts.
    for (const k of sampleKeys) {
      const inHits = hitKeys.has(k);
      say(`    ${verdicts[k].padEnd(6)} ${inHits ? ' ' : '!'} ${k}`);
      if (!inHits) fail(`${label}: pinned sample paper is no longer a hit: ${k} — re-read it or drop it with a reason`);
    }
    // GUARD 2. A frozen sample silently stops being representative as the hit
    // list grows. Pin the length it was drawn from and fail on real drift.
    const drift = Math.abs(hits.length - drawnFrom) / drawnFrom;
    say(`    hit list ${hits.length} now vs ${drawnFrom} at draw time — drift ${(100 * drift).toFixed(1)}%`);
    if (drift > 0.15) fail(`${label}: hit list moved ${(100 * drift).toFixed(1)}% since the sample was drawn (${drawnFrom} -> ${hits.length}); re-draw and re-read`);
    // GUARD 3. A sample clustered in one corner of the sorted hit list is not a
    // sample. Report where the pinned papers sit, so clustering is visible.
    const posns = sampleKeys.filter((k) => hitKeys.has(k)).map((k) => hits.findIndex((p) => key(p) === k));
    const spread = posns.length > 1 ? Math.max(...posns) - Math.min(...posns) : 0;
    say(`    sample positions in the sorted hit list: ${posns.join(', ')} (spread ${spread} of ${hits.length})`);
    if (posns.length > 1 && spread < 0.5 * hits.length) fail(`${label}: pinned sample spans only ${spread} of ${hits.length} sorted hits — it is clustered, not a sample`);
    const own = Object.values(verdicts).filter((v) => v === 'own').length;
    const tot = sampleKeys.length;
    const est = Math.round((hits.length * own) / tot);
    say(`    precision ${own}/${tot}; ${hits.length} x ${own}/${tot} = about ${est} papers (${pct(est, ACTIVE.length)} of ACTIVE)`);
    summary.push([label, `${hits.length} (${pct(hits.length, ACTIVE.length)})`, `${own}/${tot}`, `~${est} (${pct(est, ACTIVE.length)})`]);
    say(`PUBLISHED_PRECISION_${label.replace(/\W+/g, '_').toUpperCase()} ${own}/${tot} -> ~${est} ${pct(est, ACTIVE.length)}`);
  }
  say('');
  t(['Practice', 'first-person probe', 'hand-read precision', 'point estimate'], summary);
  say('');
  say('The point estimate is probe count x sample precision. It is an estimate on a');
  say('sample of 10-12, so read it as "about a fifth" rather than as a percentage.');
  say('Replacing these four estimates with four numbers needs a per-paper verdict');
  say('over the whole hit list — about 400 reads. That work is NOT done.');
}
 
h('9. IPv6 — the address space is not enumerable');
{
  const v6 = probe(ACTIVE, /IPv6/).hits;
  const v6scan = probe(SCAN, /IPv6/).hits;
  const hitlistLoose = probe(SCAN, /hit ?list/i).hits;
  const hitlistV6 = probe(SCAN, /(IPv6[^.]{0,80}hit ?list|hit ?list[^.]{0,80}IPv6|IPv6 Hitlist)/i).hits;
  // A /Gasser/ probe returned 93 SCAN papers (10.0%) and was REJECTED: the
  // surname is not unique and the hits are reference-list entries, so it
  // counts citations, not use. Kept here as a rejected probe, not published.
  const gasser = probe(SCAN, /Gasser/).hits;
  const targetGen = probe(SCAN, /(target generation|address generation algorithm|\b6Gen\b|Entropy\/IP|\b6Graph\b|\b6Sense\b|\b6Tree\b|\b6Hit\b|\b6Forest\b)/i).hits;
  const aliased = probe(SCAN, /(aliased (prefix|region)|alias(ed)? detection|dealias)/i).hits;
  const v6instr = SCAN.filter((p) => instruments(p).some((v) => v.kind === 'ipv6-targeting'));
  const v6instrAll = rows.filter((p) => instruments(p).some((v) => v.kind === 'ipv6-targeting'));
  t(['Query', 'Population', 'Papers', 'Share'], [
    ['mentions IPv6 anywhere', `SCAN ${SCAN.length}`, v6scan.length, pct(v6scan.length, SCAN.length)],
    ['mentions IPv6 anywhere', `ACTIVE ${ACTIVE.length}`, v6.length, pct(v6.length, ACTIVE.length)],
    ["names an IPv6-targeting instrument in tools[]", 'SCAN 930', v6instr.length, pct(v6instr.length, SCAN.length)],
    ["names an IPv6-targeting instrument in tools[]", 'corpus 5,859', v6instrAll.length, pct(v6instrAll.length, rows.length)],
    ['full text says "hitlist"/"hit list" (loose)', 'SCAN 930', hitlistLoose.length, pct(hitlistLoose.length, SCAN.length)],
    ['"hitlist" within 80 chars of "IPv6"', 'SCAN 930', hitlistV6.length, pct(hitlistV6.length, SCAN.length)],
    ['names a target-generation algorithm', 'SCAN 930', targetGen.length, pct(targetGen.length, SCAN.length)],
    ['discusses aliased prefixes / dealiasing', 'SCAN 930', aliased.length, pct(aliased.length, SCAN.length)],
  ]);
  say(`PUBLISHED_V6_SCAN ${v6scan.length} ${pct(v6scan.length, SCAN.length)}`);
  say(`PUBLISHED_V6_ACTIVE ${v6.length} ${pct(v6.length, ACTIVE.length)}`);
  say(`PUBLISHED_V6_INSTR_SCAN ${v6instr.length}`);
  say(`PUBLISHED_V6_INSTR_ALL ${v6instrAll.length}`);
  say(`PUBLISHED_HITLIST_LOOSE ${hitlistLoose.length} ${pct(hitlistLoose.length, SCAN.length)}`);
  say(`PUBLISHED_HITLIST_V6 ${hitlistV6.length} ${pct(hitlistV6.length, SCAN.length)}`);
  say(`REJECTED_PROBE /Gasser/ over SCAN: ${gasser.length} (${pct(gasser.length, SCAN.length)}) — surname is not unique, hits are reference-list entries; not published`);
  say(`PUBLISHED_TARGETGEN ${targetGen.length} ${pct(targetGen.length, SCAN.length)}`);
  say(`PUBLISHED_ALIASED ${aliased.length} ${pct(aliased.length, SCAN.length)}`);
  const v6subset = new Set([...hitlistV6, ...v6instr].map(key));
  const aliasIn = aliased.filter((p) => v6subset.has(key(p)));
  say('');
  say(`IPv6-SCANNING SUBSET (hitlist-near-IPv6 UNION names an IPv6-targeting instrument): ${v6subset.size} papers`);
  say(`  of those, discuss aliased prefixes / dealiasing: ${aliasIn.length} (${pct(aliasIn.length, v6subset.size)})`);
  say(`  the same 10 papers against all of SCAN would be ${pct(aliased.length, SCAN.length)} — the wrong denominator, most of SCAN never touches IPv6`);
  say(`PUBLISHED_V6_SUBSET ${v6subset.size}`);
  say(`PUBLISHED_ALIAS_IN_SUBSET ${aliasIn.length} ${pct(aliasIn.length, v6subset.size)}`);
  say('');
  say('IPv6-targeting instruments named, corpus-wide:');
  const m = new Map();
  for (const p of v6instrAll) for (const v of instruments(p).filter((x) => x.kind === 'ipv6-targeting')) {
    if (!m.has(v.family)) m.set(v.family, new Set());
    m.get(v.family).add(key(p));
  }
  for (const [f, s] of [...m].sort((a, b) => b[1].size - a[1].size)) say(`  ${String(s.size).padStart(3)} ${f}  (${[...s].join(', ')})`);
  say('');
  say('Per-year IPv6 mention rate inside SCAN (2026 provisional):');
  const years = [...new Set(SCAN.map((p) => p.year))].sort();
  t(['Year', 'SCAN', 'mentions IPv6', 'share'], years.map((y) => {
    const b = SCAN.filter((p) => p.year === y);
    const n = probe(b, /IPv6/).hits.length;
    return [y, b.length, n, pct(n, b.length)];
  }));
}
 
h('10. THE JOIN TO design:ip_classification');
{
  const ipCls = (p) => p.classification.some((c) => c.target === 'ip-address');
  const a = ACTIVE.filter(ipCls);
  const s = SCAN.filter(ipCls);
  const c = rows.filter(ipCls);
  t(['Population', 'N', "classification[].target == 'ip-address'", 'share'], [
    ['ACTIVE', ACTIVE.length, a.length, pct(a.length, ACTIVE.length)],
    ['SCAN', SCAN.length, s.length, pct(s.length, SCAN.length)],
    ['corpus', rows.length, c.length, pct(c.length, rows.length)],
  ]);
  say(`PUBLISHED_IPCLS_ACTIVE ${a.length} ${pct(a.length, ACTIVE.length)}`);
  say(`PUBLISHED_IPCLS_SCAN ${s.length} ${pct(s.length, SCAN.length)}`);
  say(`PUBLISHED_IPCLS_CORPUS ${c.length} ${pct(c.length, rows.length)}`);
  // Which resources those papers reach for.
  const res = new Map();
  for (const p of a) for (const cl of p.classification) {
    if (cl.target !== 'ip-address') continue;
    for (const rn of cl.resourceName ? [cl.resourceName] : (cl.resources ?? [])) {
      if (isSentinel(rn)) continue;
      const kk = String(rn).trim();
      if (!res.has(kk)) res.set(kk, new Set());
      res.get(kk).add(key(p));
    }
  }
  say('');
  say(`distinct raw resourceName strings on ACTIVE ip-address classifications: ${res.size} (free text; folded on design:ip_classification, not here)`);
  for (const [k, v] of [...res].sort((x, y) => y[1].size - x[1].size).slice(0, 15)) say(`  ${String(v.size).padStart(3)} ${k}`);
}
 
h('11. POSTERS AND SHORT RECORDS');
{
  const poster = (p) => /^poster-/.test(p.slug) || /^Poster:/.test(p.title);
  const ap = ACTIVE.filter(poster);
  const sp = SCAN.filter(poster);
  const short = ACTIVE.filter((p) => p.pages !== null && p.pages <= 4);
  say(`ACTIVE posters: ${ap.length} (${pct(ap.length, ACTIVE.length)}); SCAN posters: ${sp.length} (${pct(sp.length, SCAN.length)})`);
  say(`ACTIVE records of <=4 pages: ${short.length} (${pct(short.length, ACTIVE.length)})`);
  const full = ACTIVE.filter((p) => !(p.pages !== null && p.pages <= 4));
  const vantageStated = (p) => p.vantage.some((v) => (v.locations ?? []).some((l) => !isSentinel(l)));
  say(`vantage location stated: all ACTIVE ${pct(ACTIVE.filter(vantageStated).length, ACTIVE.length)}, excluding <=4-page records ${pct(full.filter(vantageStated).length, full.length)}`);
  say(`PUBLISHED_ACTIVE_POSTERS ${ap.length}`);
  say(`PUBLISHED_VANTAGE_FULLONLY ${pct(full.filter(vantageStated).length, full.length)}`);
}
 
h('12. MEASURED RESULTS available for the page (detection[].prevalence on ACTIVE)');
{
  let n = 0;
  const withPrev = ACTIVE.filter((p) => p.detection.some((d) => !isSentinel(d.prevalence)));
  say(`ACTIVE papers with at least one detection[].prevalence: ${withPrev.length} (${pct(withPrev.length, ACTIVE.length)})`);
  say(`PUBLISHED_ACTIVE_PREVALENCE ${withPrev.length} ${pct(withPrev.length, ACTIVE.length)}`);
  if (QUOTES) {
    const re = new RegExp(QUOTES, 'i');
    for (const p of ACTIVE) for (const d of p.detection) {
      if (isSentinel(d.prevalence)) continue;
      const blob = `${d.phenomenon} ${d.technique} ${d.metric} ${d.prevalence}`;
      if (!re.test(blob)) continue;
      say(`--- ${key(p)}`);
      say(`    phenomenon: ${d.phenomenon}`);
      say(`    technique : ${d.technique}`);
      say(`    metric    : ${d.metric}`);
      say(`    prevalence: ${d.prevalence}`);
      say(`    quote     : ${(d.evidence?.quote ?? '').replace(/\s+/g, ' ')}`);
      if (++n > 40) break;
    }
  }
}
 
if (HITS) {
  h(`SENTENCES matching ${HITS} in ACTIVE`);
  const re = new RegExp(HITS, 'i');
  let n = 0;
  for (const p of ACTIVE) {
    const cs = ctx(p, re);
    if (!cs.length) continue;
    say(`--- ${key(p)}`);
    for (const c of cs.slice(0, 2)) say(`    …${c}…`);
    if (++n > 40) break;
  }
}
 
if (LIST === 'active') { h('ACTIVE paper list'); for (const p of ACTIVE) say(`${key(p)}  [${memberOf(p).map((v) => v.family).join(', ')}]`); }
if (LIST === 'scan') { h('SCAN paper list'); for (const p of SCAN) say(key(p)); }
 
h('Z. NON-CORPUS FIGURES ON THE PAGE (checked by external_checks_internet_scanning.sh)');
for (const line of EXTERNAL) say(line);
 
h('RESULT');
say(fails.length === 0 ? 'OK — all contracts hold' : `${fails.length} FAILURE(S)`);
console.log(out.join('\n'));
if (fails.length) process.exit(1);