| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| security:email_authentication [2026/09/09 11:47] – Publish the misconfiguration and operator-log figure tables (they were computed and quote-checked but not on the page); fix li2024_bounce's 2.19% denominator - it is a share of the 32M bounces, not the 298M total; refine the HADES boundary row. Authored b karel.kubicek.claude | security:email_authentication [2026/09/10 21:05] (current) – programming:internet_scanning is written; the link is no longer red on purpose. Authored by Claude. karel.kubicek.claude |
|---|
| This topic has a harder ethics profile than most web measurement, because half its instruments touch third-party production infrastructure and one of them delivers a forged message to a real mailbox. What the 31 papers report about themselves: | This topic has a harder ethics profile than most web measurement, because half its instruments touch third-party production infrastructure and one of them delivers a forged message to a real mailbox. What the 31 papers report about themselves: |
| |
| * **12 of 31 mention no ethics review at all**; 6 explicitly discuss the question and state that no review was sought or required; only 5 report an approval. That is 38.7% silent against **64.2%** corpus-wide (of the 4,965 papers carrying an ethics object) — so this population is //better// than the corpus, not worse, and 5 report an approval against 20.1% corpus-wide. //n// = 31, so read neither gap as a difference. Better than a low bar is still not good enough for an instrument that delivers forged mail to a live provider. | * **12 of 31 mention no ethics review at all.** Of the other 19: 6 discuss the question and state that no review was sought or required, 5 report an approval, 4 say review was not required, 2 sought one and do not say what came back, and 2 report an exemption. That is 38.7% silent against **64.2%** corpus-wide (of the 4,965 papers carrying an ethics object) — so this population is //better// than the corpus, not worse, and 5 report an approval against 20.1% corpus-wide. //n// = 31, so read neither gap as a difference. Better than a low bar is still not good enough for an instrument that delivers forged mail to a live provider. |
| * **19 of 31 notified affected parties**, 6 partially, 2 not, 3 do not say. Disclosure is the norm in this population — usefully so, since these are protocol and implementation flaws with named vendors. | * **19 of 31 notified affected parties**, 6 partially, 2 not, 3 do not say, and 1 is recorded not-applicable. Disclosure is the norm in this population — usefully so, since these are protocol and implementation flaws with named vendors. |
| * **20 of 31 report a public artifact and 18 give a code URL**, including several long-lived project sites (''dane-study.github.io'', ''spf-measurement.github.io'', ''mta-sts.netsecurelab.org'', ''github.com/chenjj/espoofer''). By the standards of this corpus that is good practice. | * **20 of 31 report a public artifact and 18 give a code URL**, including several long-lived project sites (''dane-study.github.io'', ''spf-measurement.github.io'', ''mta-sts.netsecurelab.org'', ''github.com/chenjj/espoofer''). By the standards of this corpus that is good practice. |
| |
| ==== Where these papers go quiet ==== | ==== Where these papers go quiet ==== |
| |
| * **29 of 31 report descriptive statistics only.** Two run a hypothesis test ({[hu2018_measurements]}, {[blechschmidt2023_hello]}), one a regression ({[szurdi2017_email]}). For a literature whose central claims are comparisons — notified against control, this provider against that one, this year against last — that is thin. If your result is a difference, test it; see [[statistics:hypothesis_testing]]. | * **29 of 31 report descriptive statistics only.** Two run a hypothesis test ({[hu2018_measurements]}, {[blechschmidt2023_hello]}). Exactly one paper runs anything else, and it runs three things: {[szurdi2017_email]} fits a regression (R² = 0.74), reports a correlation, and cross-validates (leave-one-out, R² drops to 0.63). One paper in eleven years accounts for every regression, every correlation and every resampling result in this population. For a literature whose central claims are comparisons — notified against control, this provider against that one, this year against last — that is thin. If your result is a difference, test it; see [[statistics:hypothesis_testing]]. |
| * **19 of 31 coded something by hand and the extraction records an inter-rater agreement metric for none of them.** Zero of nineteen — but read that as an extraction result: the 19 papers were not re-read by hand to confirm the absence is theirs and not the extractor's, and it is the least-verified number on this page. Even discounted, the direction is not in doubt: vulnerability labelling in a client matrix is a coding task, and "we manually analysed the identified behaviours" is not a method. See [[statistics:interrater_agreement]]. | * **19 of 31 coded something by hand and the extraction records an inter-rater agreement metric for none of them.** Zero of nineteen — but read that as an extraction result: the 19 papers were not re-read by hand to confirm the absence is theirs and not the extractor's, and it is the least-verified number on this page. Even discounted, the direction is not in doubt: vulnerability labelling in a client matrix is a coding task, and "we manually analysed the identified behaviours" is not a method. See [[statistics:interrater_agreement]]. |
| * **12 of 31 mention no ethics review**, against instruments that send forged mail and probe strangers' servers. | * **12 of 31 mention no ethics review at all** and a further 4 record it as not required, against instruments that send forged mail and probe strangers' servers. |
| * Eight of the 31 recruited people; the five operator surveys carry //n// of 39, 74, 16, 95 and 117. The extraction records a recruitment channel for only three of the five — but that undercounts by one: {[ashiq2023_report]} names MailOP, NANOG and MESSEU in its own text and the extractor recorded ''not-stated''. Four of five, and the fifth is {[blechschmidt2023_hello]}, whose 16 respondents are operators who replied to a notification email rather than a recruited sample at all. | * Eight of the 31 recruited people; the five operator surveys carry //n// of 39, 74, 16, 95 and 117. The extraction records a recruitment channel for only three of the five — but that undercounts by one: {[ashiq2023_report]} names MailOP, NANOG and MESSEU in its own text and the extractor recorded ''not-stated''. Four of five, and the fifth is {[blechschmidt2023_hello]}, whose 16 respondents are operators who replied to a notification email rather than a recruited sample at all. |
| |
| ^ Paper ^ Verdict it carries ^ Why it is not here ^ | ^ Paper ^ Verdict it carries ^ Why it is not here ^ |
| | {[rao2024_unfiltered]} — //Unfiltered: Measuring Cloud-based Email Filtering Bypasses//, TheWebConf 2024 | spam / filtering | **Arguable.** The measured object is an MX misconfiguration in the delivery path — 80% of organisations using a cloud filter can be bypassed by delivering straight to the hosting provider — and the instrument is SMTP probing. It is filed with filtering because filtering is what the bypass defeats. If you are working on delivery-path integrity, read it as if it were in this population. | | | {[rao2024_unfiltered]} — //Unfiltered: Measuring Cloud-based Email Filtering Bypasses//, TheWebConf 2024 | spam / filtering | **Arguable.** The measured object is an MX misconfiguration in the delivery path — 80% of organisations using a cloud filter can be bypassed by delivering straight to the hosting provider — and the instrument is SMTP probing. It is filed with filtering because filtering is what the bypass defeats. If you are working on delivery-path integrity, read it as if it were in this population. | |
| | {[li2025_hades]} — //HADES Attack//, NDSS 2025 | spam | **Also arguable.** Its object is DNS-based blocklist (DNSBL) //adoption and operation//, measured end to end from fifteen months of non-delivery reports — infrastructure, by the same argument as //Unfiltered//. It sits with spam because a blocklist is a spam control rather than an authentication or transport mechanism. If you are measuring what actually decides whether mail is delivered, read it: {[li2024_bounce]}, which //is// in the 31, finds 31.1% of bounces are blocklist hits, more than fourteen times the share caused by authentication failure. | | | {[li2025_hades]} — //HADES Attack//, NDSS 2025 | spam | **Also arguable.** Its object is DNS-based blocklist (DNSBL) //adoption and operation//, measured end to end from fifteen months of non-delivery reports — infrastructure, by the same argument as //Unfiltered//. It sits with spam because a blocklist is a spam control rather than an authentication or transport mechanism. If you are measuring what actually decides whether mail is delivered, read it: {[li2024_bounce]}, which //is// in the 31, finds 31.10% of bounces are blocklist hits, more than fourteen times the share caused by authentication failure. | |
| | {[wu2026_email]} — //One Email, Many Faces//, NDSS 2026 | the address as identifier | Alias and identity confusion, which is [[privacy:email_tracking]]'s subject. | | | {[wu2026_email]} — //One Email, Many Faces//, NDSS 2026 | the address as identifier | Alias and identity confusion, which is [[privacy:email_tracking]]'s subject. | |
| | {[zhang2024_inbox]} — //Inbox Invasion//, 2024 | spam / filtering | MIME ambiguity against attachment scanners: a content-inspection problem. | | | {[zhang2024_inbox]} — //Inbox Invasion//, 2024 | spam / filtering | MIME ambiguity against attachment scanners: a content-inspection problem. | |
| Two limitations specific to this page: | Two limitations specific to this page: |
| |
| * **Quote verification hits the two-column problem.** All 55 figures above were checked against ''paper.cols.txt''; **60 quote fragments** were needed to do it. Four figures carry a second fragment, and **three** of those four are genuine splices — a sentence interrupted by a table row, a figure caption or a column break, and unlocatable as one string in any of the three available renderings. The fourth second fragment is an independent corroborating quote, not a split sentence. At least one figure stated in the prose rather than in a table (the DANE rollover figure) sits on a spliced sentence too, so three is a floor for this page, not a total. Across the 31 papers' 206 extracted evidence quotes, 130 locate contiguously and 76 do not; a hand-read sample of the 76 found real sentences broken by page furniture, plus one case where the extractor's quote merges two adjacent sentences. Details and the sample on the provenance page. | * **Quote verification hits the two-column problem.** All 55 figures above were checked against ''paper.cols.txt''; **60 quote fragments** were needed to do it. Five figures carry a second fragment, and **three** of those five are genuine splices — a sentence interrupted by a table row, a figure caption or a column break, and unlocatable as one string in any of the three available renderings. The other two second fragments are independent corroborating quotations, not split sentences. At least one figure stated in the prose rather than in a table (the DANE rollover figure) sits on a spliced sentence too, so three is a floor for this page, not a total. Across the 31 papers' 206 extracted evidence quotes, 130 locate contiguously and 76 do not; a hand-read sample of the 76 found real sentences broken by page furniture, plus one case where the extractor's quote merges two adjacent sentences. Details and the sample on the provenance page. |
| * **The mechanism and instrument maps are hand maps, not extraction fields.** The schema has no field for "which deployed mail mechanism is this paper's object", so both tables were made by reading each of the 31 results sections. They are reproducible in the sense that they are written down in the script and asserted complete against the population; they are not independent of the judgement that made them, and [[provenance:security:email_authentication]] names the four calls inside them that could reasonably have gone the other way. | * **The mechanism and instrument maps are hand maps, not extraction fields.** The schema has no field for "which deployed mail mechanism is this paper's object", so both tables were made by reading each of the 31 results sections. They are reproducible in the sense that they are written down in the script and asserted complete against the population; they are not independent of the judgement that made them, and [[provenance:security:email_authentication]] names the four calls inside them that could reasonably have gone the other way. |
| |
| * [[security:phishing]] — mail as an attack vector, and the feeds that label it. | * [[security:phishing]] — mail as an attack vector, and the feeds that label it. |
| * [[security:tls_certificates]] — the web PKI, CT logs and the scanning instruments this page borrows. | * [[security:tls_certificates]] — the web PKI, CT logs and the scanning instruments this page borrows. |
| * [[programming:internet_scanning]] — ZMap, ZGrab and the ethics and rate limits of internet-wide probing. //Queued on [[:roadmap]], not yet written; the link is red on purpose.// | * [[programming:internet_scanning]] — ZMap, ZGrab and the rate limits of internet-wide probing. //Written 2026-09-10.// |
| * [[design:website_selection]] — Tranco, zone files, and why a top list is not a sample. | * [[design:website_selection]] — Tranco, zone files, and why a top list is not a sample. |
| * [[design:longitudinal]] — snapshot cadence, which is what DANE and MTA-STS rollover measurement lives or dies by. | * [[design:longitudinal]] — snapshot cadence, which is what DANE and MTA-STS rollover measurement lives or dies by. |
