Measure The Web

User Tools

Site Tools

Trace: • filter_lists • ip_classification • phishing • tracker_radar_collector • internet_scanning • llm_agents • annotation • ethics • email_authentication • sampling
security:email_authentication

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
security:email_authentication [2026/09/09 11:42] – Add the BreakSPF same-scan/same-day filter pair (60.9% vs 79.4%, 18.5 points on the filter alone) to the denominators table - the clearest possible demonstration of the first trap. Authored by Claude karel.kubicek.claudesecurity:email_authentication [2026/09/10 21:05] (current) – programming:internet_scanning is written; the link is no longer red on purpose. Authored by Claude. karel.kubicek.claude
Line 95: Line 95:
  
 That 6.8% is the number to remember. It is a **2024** measurement of whether receiving mail servers so much as look SPF up before accepting the message body — and within these 31 papers it is the only figure that measures inbound enforcement at internet scale rather than at 20–50 hand-picked providers. Deployment-rate papers count the left column of a two-column problem. That 6.8% is the number to remember. It is a **2024** measurement of whether receiving mail servers so much as look SPF up before accepting the message body — and within these 31 papers it is the only figure that measures inbound enforcement at internet scale rather than at 20–50 hand-picked providers. Deployment-rate papers count the left column of a two-column problem.
 +
 +==== Deployed is not the same as correct ====
 +
 +Every one of these papers has a second half, and it is always this: a large share of what //is// deployed is broken. If your study stops at the adoption rate you are reporting the good news only.
 +
 +^ What is wrong ^ Share ^ Population ^ Source ^
 +| DKIM keys of at most 1024 bits | **84%** | 3,627,871 domains with DKIM keys in passive DNS | {[wang2022_longitudinal]} |
 +| DKIM public keys shared across domains | 66.9% (2,427,682) | the DKIM-publishing domains in that passive-DNS set | {[wang2022_longitudinal]} |
 +| SPF authorising more than 100,000 IPv4 addresses | 34.7% | the SPF-publishing domains of 12 million scanned | {[czybik2023_lazy]} |
 +| SPF with no restrictive ''all'' qualifier at all | 5.9% (427,767) | the same SPF-publishing domains | {[czybik2023_lazy]} |
 +| SPF needing more than 10 DNS lookups to evaluate | 6.5% | 55 million domains with SPF records | {[ashiq2024_beyond]} |
 +| STARTTLS certificates failing validation | **30.0%** | 2,112,682 connectable MXs, 2023 | {[blechschmidt2023_hello]} |
 +| TLSA records invalid | over 22% | SMTP servers serving ''.com'' domains, hourly snapshots | {[lee2022_under]} |
 +| MTA-STS records misconfigured | 29.6% (20,144) | the 68,030 domains publishing one, Sep 2024 | {[ashiq2025_unraveling]} |
 +| External DMARC ''rua'' with no authorisation record at the receiving domain | 26% (520K) | 2 million DMARC records naming an external reporting domain | {[ashiq2023_report]} |
 +| Auto-configuration deployed at all | 7.52% (79,212) | 1,053,469 domains scanned for Autodiscover / Autoconfig / SRV | {[wen2025_automatic]} |
 +
 +Two of these are exploitable rather than merely untidy. {[czybik2023_lazy]} rented about €30 of shared web hosting and could send mail passing SPF for **26,095** domains. {[li2026_coordmail]} found that **77.89%** of reflections off 10,079 exploitable bounce servers pass SPF or DKIM — the authentication is working exactly as specified and authenticating the attacker.
 +
 +==== What it looks like from inside an operator ====
 +
 +Six of the 31 papers got logs, headers or mailboxes from a real mail operator, and they are the only source on this page for base rates — what fraction of mail fails, why, and how concentrated the delivery path is. You almost certainly cannot reproduce these; read them so you know what your scan is a proxy for.
 +
 +^ Figure ^ Population ^ Source ^
 +| 94.40% of inbound messages authenticated by SPF, DKIM or both | Gmail's own SMTP handshake logs, April 2015 | {[durumeric2015_neither]} |
 +| 8.11% hard-bounced, 4.82% soft-bounced | 298 million messages at one large Chinese ESP | {[li2024_bounce]} |
 +| 2.19% (701,347) of bounces caused by **sender authentication failure** | the **32 million bounced** messages that paper's Table 1 classifies — not the 298 million total, against which the same count is 0.24% | {[li2024_bounce]} |
 +| 31.10% (9,975,329) of bounces caused by the sender **hitting a spam blocklist** | the same 32 million bounced messages — **fourteen times** the authentication row, like for like | {[li2024_bounce]} |
 +| 82.7% of delivery paths rely entirely on third-party relaying | 105 million messages whose intermediate path could be reconstructed from ''Received'' headers | {[li2025_characterizing]} |
 +| HHI of 40% for the relay market | middle nodes across those reconstructed paths | {[li2025_characterizing]} |
 +| 5.46% of users ever used S/MIME or PGP; 0.06% of messages encrypted | 81.6 million messages from 37,089 accounts at one university over 27 years | {[stransky2022_years]} |
 +
 +The two ''li'' rows are the same 32 million bounces and are directly comparable; **the authentication row is not comparable to the 298 million total**, and this page published it against the wrong denominator for several hours before catching it. That is the trap in its natural habitat: an operator paper reports some figures against the whole corpus of mail and others against the failures, in adjacent sentences.
  
 ===== Methods, and Which Ones Are Current ===== ===== Methods, and Which Ones Are Current =====
Line 106: Line 139:
   * **Shared infrastructure as the unit of analysis.** {[wang2024_breakspf]} (2024), {[wang2025_email]} (SMTP smuggling, 2025) and {[li2026_coordmail]} (2026) all ask what an attacker can do by //renting// a piece of infrastructure that thousands of domains have already authorised. This is the newest coherent line in the population and it is three papers old.   * **Shared infrastructure as the unit of analysis.** {[wang2024_breakspf]} (2024), {[wang2025_email]} (SMTP smuggling, 2025) and {[li2026_coordmail]} (2026) all ask what an attacker can do by //renting// a piece of infrastructure that thousands of domains have already authorised. This is the newest coherent line in the population and it is three papers old.
   * **Operator surveys alongside the scan.** None before 2020; five by 2025, with 39, 74, 16, 95 and 117 respondents ({[lee2022_under]}, {[ashiq2023_report]}, {[blechschmidt2023_hello]}, {[ashiq2024_beyond]}, {[ashiq2025_unraveling]}). Recruitment is from MailOP, NANOG, DENOG, NLNOG and MESSEU in every case where it is stated — a specific, self-selecting, unusually competent population. Report it as such; it is not a sample of operators.   * **Operator surveys alongside the scan.** None before 2020; five by 2025, with 39, 74, 16, 95 and 117 respondents ({[lee2022_under]}, {[ashiq2023_report]}, {[blechschmidt2023_hello]}, {[ashiq2024_beyond]}, {[ashiq2025_unraveling]}). Recruitment is from MailOP, NANOG, DENOG, NLNOG and MESSEU in every case where it is stated — a specific, self-selecting, unusually competent population. Report it as such; it is not a sample of operators.
-  * **Notification-and-rescan as a built-in second phase.** {[bennett2022_spfail]}, {[czybik2023_lazy]} and {[blechschmidt2023_hello]} all notify and remeasure. Only {[blechschmidt2023_hello]} has a comparison arm: domains whose notification **bounced** remediated at 20.7% against 39.7% for domains that were reached (Fisher exact //p// = 7.8 × 10<sup>-43</sup>). That arm is what makes the result more than a trend — but it is not a randomised control, and the authors say so themselves: //"we cannot conclude a causal relation since the lack of deliverable emails may be a hidden variable which influences fix rates"//. Undeliverable postmaster addresses are plausibly a proxy for an unattended domain. If you run a notification experiment, randomise.+  * **Notification-and-rescan as a built-in second phase.** {[bennett2022_spfail]}, {[czybik2023_lazy]} and {[blechschmidt2023_hello]} all notify and remeasure. The effect sizes are small and worth knowing before you budget for one: {[czybik2023_lazy]} saw **6,931** SPF errors fixed two weeks after notification, a 3.28% fall in the total, and {[bennett2022_spfail]} found **just over 80%** of the domains it could infer results for were still vulnerable four months on. Only {[blechschmidt2023_hello]} has a comparison arm: domains whose notification **bounced** remediated at 20.7% against 39.7% for domains that were reached (Fisher exact //p// = 7.8 × 10<sup>-43</sup>). That arm is what makes the result more than a trend — but it is not a randomised control, and the authors say so themselves: //"we cannot conclude a causal relation since the lack of deliverable emails may be a hidden variable which influences fix rates"//. Undeliverable postmaster addresses are plausibly a proxy for an unattended domain. If you run a notification experiment, randomise.
   * **Client matrices.** Still current — {[tang2025_multifaceted]} (49 clients, 2025) and {[wen2025_automatic]} (29 clients, 2025) are recent — but they are a vulnerability census, not a prevalence measurement, and should not be written as one.   * **Client matrices.** Still current — {[tang2025_multifaceted]} (49 clients, 2025) and {[wen2025_automatic]} (29 clients, 2025) are recent — but they are a vulnerability census, not a prevalence measurement, and should not be written as one.
  
Line 162: Line 195:
 This topic has a harder ethics profile than most web measurement, because half its instruments touch third-party production infrastructure and one of them delivers a forged message to a real mailbox. What the 31 papers report about themselves: This topic has a harder ethics profile than most web measurement, because half its instruments touch third-party production infrastructure and one of them delivers a forged message to a real mailbox. What the 31 papers report about themselves:
  
-  * **12 of 31 mention no ethics review at all**; 6 explicitly discuss the question and state that no review was sought or required; only 5 report an approval. That is 38.7% silent against **64.2%** corpus-wide (of the 4,965 papers carrying an ethics object) — so this population is //better// than the corpus, not worse, and 5 report an approval against 20.1% corpus-wide. //n// = 31, so read neither gap as a difference. Better than a low bar is still not good enough for an instrument that delivers forged mail to a live provider. +  * **12 of 31 mention no ethics review at all.** Of the other 19: 6 discuss the question and state that no review was sought or required, 5 report an approval, 4 say review was not required, 2 sought one and do not say what came back, and 2 report an exemption. That is 38.7% silent against **64.2%** corpus-wide (of the 4,965 papers carrying an ethics object) — so this population is //better// than the corpus, not worse, and 5 report an approval against 20.1% corpus-wide. //n// = 31, so read neither gap as a difference. Better than a low bar is still not good enough for an instrument that delivers forged mail to a live provider. 
-  * **19 of 31 notified affected parties**, 6 partially, 2 not, 3 do not say. Disclosure is the norm in this population — usefully so, since these are protocol and implementation flaws with named vendors.+  * **19 of 31 notified affected parties**, 6 partially, 2 not, 3 do not say, and 1 is recorded not-applicable. Disclosure is the norm in this population — usefully so, since these are protocol and implementation flaws with named vendors.
   * **20 of 31 report a public artifact and 18 give a code URL**, including several long-lived project sites (''dane-study.github.io'', ''spf-measurement.github.io'', ''mta-sts.netsecurelab.org'', ''github.com/chenjj/espoofer''). By the standards of this corpus that is good practice.   * **20 of 31 report a public artifact and 18 give a code URL**, including several long-lived project sites (''dane-study.github.io'', ''spf-measurement.github.io'', ''mta-sts.netsecurelab.org'', ''github.com/chenjj/espoofer''). By the standards of this corpus that is good practice.
  
Line 288: Line 321:
 ==== Where these papers go quiet ==== ==== Where these papers go quiet ====
  
-  * **29 of 31 report descriptive statistics only.** Two run a hypothesis test ({[hu2018_measurements]}, {[blechschmidt2023_hello]}), one a regression ({[szurdi2017_email]}). For a literature whose central claims are comparisons — notified against control, this provider against that one, this year against last — that is thin. If your result is a difference, test it; see [[statistics:hypothesis_testing]].+  * **29 of 31 report descriptive statistics only.** Two run a hypothesis test ({[hu2018_measurements]}, {[blechschmidt2023_hello]}). Exactly one paper runs anything else, and it runs three things: {[szurdi2017_email]} fits a regression (R² = 0.74), reports a correlation, and cross-validates (leave-one-out, R² drops to 0.63). One paper in eleven years accounts for every regression, every correlation and every resampling result in this population. For a literature whose central claims are comparisons — notified against control, this provider against that one, this year against last — that is thin. If your result is a difference, test it; see [[statistics:hypothesis_testing]].
   * **19 of 31 coded something by hand and the extraction records an inter-rater agreement metric for none of them.** Zero of nineteen — but read that as an extraction result: the 19 papers were not re-read by hand to confirm the absence is theirs and not the extractor's, and it is the least-verified number on this page. Even discounted, the direction is not in doubt: vulnerability labelling in a client matrix is a coding task, and "we manually analysed the identified behaviours" is not a method. See [[statistics:interrater_agreement]].   * **19 of 31 coded something by hand and the extraction records an inter-rater agreement metric for none of them.** Zero of nineteen — but read that as an extraction result: the 19 papers were not re-read by hand to confirm the absence is theirs and not the extractor's, and it is the least-verified number on this page. Even discounted, the direction is not in doubt: vulnerability labelling in a client matrix is a coding task, and "we manually analysed the identified behaviours" is not a method. See [[statistics:interrater_agreement]].
-  * **12 of 31 mention no ethics review**, against instruments that send forged mail and probe strangers' servers.+  * **12 of 31 mention no ethics review at all** and a further 4 record it as not required, against instruments that send forged mail and probe strangers' servers.
   * Eight of the 31 recruited people; the five operator surveys carry //n// of 39, 74, 16, 95 and 117. The extraction records a recruitment channel for only three of the five — but that undercounts by one: {[ashiq2023_report]} names MailOP, NANOG and MESSEU in its own text and the extractor recorded ''not-stated''. Four of five, and the fifth is {[blechschmidt2023_hello]}, whose 16 respondents are operators who replied to a notification email rather than a recruited sample at all.   * Eight of the 31 recruited people; the five operator surveys carry //n// of 39, 74, 16, 95 and 117. The extraction records a recruitment channel for only three of the five — but that undercounts by one: {[ashiq2023_report]} names MailOP, NANOG and MESSEU in its own text and the extractor recorded ''not-stated''. Four of five, and the fifth is {[blechschmidt2023_hello]}, whose 16 respondents are operators who replied to a notification email rather than a recruited sample at all.
  
Line 299: Line 332:
 ^ Paper ^ Verdict it carries ^ Why it is not here ^ ^ Paper ^ Verdict it carries ^ Why it is not here ^
 | {[rao2024_unfiltered]} — //Unfiltered: Measuring Cloud-based Email Filtering Bypasses//, TheWebConf 2024 | spam / filtering | **Arguable.** The measured object is an MX misconfiguration in the delivery path — 80% of organisations using a cloud filter can be bypassed by delivering straight to the hosting provider — and the instrument is SMTP probing. It is filed with filtering because filtering is what the bypass defeats. If you are working on delivery-path integrity, read it as if it were in this population. | | {[rao2024_unfiltered]} — //Unfiltered: Measuring Cloud-based Email Filtering Bypasses//, TheWebConf 2024 | spam / filtering | **Arguable.** The measured object is an MX misconfiguration in the delivery path — 80% of organisations using a cloud filter can be bypassed by delivering straight to the hosting provider — and the instrument is SMTP probing. It is filed with filtering because filtering is what the bypass defeats. If you are working on delivery-path integrity, read it as if it were in this population. |
-| {[li2025_hades]} — //HADES Attack//, NDSS 2025 | spam | Object is email blocklist manipulation, not an authentication mechanism. |+| {[li2025_hades]} — //HADES Attack//, NDSS 2025 | spam | **Also arguable.** Its object is DNS-based blocklist (DNSBL) //adoption and operation//, measured end to end from fifteen months of non-delivery reports — infrastructure, by the same argument as //Unfiltered//. It sits with spam because a blocklist is a spam control rather than an authentication or transport mechanism. If you are measuring what actually decides whether mail is delivered, read it: {[li2024_bounce]}, which //is// in the 31, finds 31.10% of bounces are blocklist hits, more than fourteen times the share caused by authentication failure. |
 | {[wu2026_email]} — //One Email, Many Faces//, NDSS 2026 | the address as identifier | Alias and identity confusion, which is [[privacy:email_tracking]]'s subject. | | {[wu2026_email]} — //One Email, Many Faces//, NDSS 2026 | the address as identifier | Alias and identity confusion, which is [[privacy:email_tracking]]'s subject. |
 | {[zhang2024_inbox]} — //Inbox Invasion//, 2024 | spam / filtering | MIME ambiguity against attachment scanners: a content-inspection problem. | | {[zhang2024_inbox]} — //Inbox Invasion//, 2024 | spam / filtering | MIME ambiguity against attachment scanners: a content-inspection problem. |
Line 308: Line 341:
 ==== Methodology and limitations of these figures ==== ==== Methodology and limitations of these figures ====
  
-Every count above is a count of **papers**, never of extraction tuples, with its denominator named in the same sentence or table cell. Free-text fields were folded before aggregating and the unmapped residue is published rather than dropped. The report script that produces every figure on this page is ''scripts/report_email_authentication.mjs''; it hand-keys the mechanism map, the instrument map and all 54 published figures, and it fails if any of those maps stops covering all 31 papers.+Every count above is a count of **papers**, never of extraction tuples, with its denominator named in the same sentence or table cell. Free-text fields were folded before aggregating and the unmapped residue is published rather than dropped. The report script that produces every figure on this page is ''scripts/report_email_authentication.mjs''; it hand-keys the mechanism map, the instrument map and all 55 published figures, and it fails if any of those maps stops covering all 31 papers.
  
 Two limitations specific to this page: Two limitations specific to this page:
  
-  * **Quote verification hits the two-column problem.** All 54 figures above were checked against ''paper.cols.txt''; **58 quote fragments** were needed to do it. Four figures carry a second fragment, and **three** of those four are genuine splices — a sentence interrupted by a table row, a figure caption or a column break, and unlocatable as one string in any of the three available renderings. The fourth second fragment is an independent corroborating quote, not a split sentence. At least one figure stated in the prose rather than in a table (the DANE rollover figure) sits on a spliced sentence too, so three is a floor for this page, not a total. Across the 31 papers' 206 extracted evidence quotes, 130 locate contiguously and 76 do not; a hand-read sample of the 76 found real sentences broken by page furniture, plus one case where the extractor's quote merges two adjacent sentences. Details and the sample on the provenance page.+  * **Quote verification hits the two-column problem.** All 55 figures above were checked against ''paper.cols.txt''; **60 quote fragments** were needed to do it. Five figures carry a second fragment, and **three** of those five are genuine splices — a sentence interrupted by a table row, a figure caption or a column break, and unlocatable as one string in any of the three available renderings. The other two second fragments are independent corroborating quotations, not split sentences. At least one figure stated in the prose rather than in a table (the DANE rollover figure) sits on a spliced sentence too, so three is a floor for this page, not a total. Across the 31 papers' 206 extracted evidence quotes, 130 locate contiguously and 76 do not; a hand-read sample of the 76 found real sentences broken by page furniture, plus one case where the extractor's quote merges two adjacent sentences. Details and the sample on the provenance page.
   * **The mechanism and instrument maps are hand maps, not extraction fields.** The schema has no field for "which deployed mail mechanism is this paper's object", so both tables were made by reading each of the 31 results sections. They are reproducible in the sense that they are written down in the script and asserted complete against the population; they are not independent of the judgement that made them, and [[provenance:security:email_authentication]] names the four calls inside them that could reasonably have gone the other way.   * **The mechanism and instrument maps are hand maps, not extraction fields.** The schema has no field for "which deployed mail mechanism is this paper's object", so both tables were made by reading each of the 31 results sections. They are reproducible in the sense that they are written down in the script and asserted complete against the population; they are not independent of the judgement that made them, and [[provenance:security:email_authentication]] names the four calls inside them that could reasonably have gone the other way.
  
Line 334: Line 367:
   * [[security:phishing]] — mail as an attack vector, and the feeds that label it.   * [[security:phishing]] — mail as an attack vector, and the feeds that label it.
   * [[security:tls_certificates]] — the web PKI, CT logs and the scanning instruments this page borrows.   * [[security:tls_certificates]] — the web PKI, CT logs and the scanning instruments this page borrows.
-  * [[programming:internet_scanning]] — ZMap, ZGrab and the ethics and rate limits of internet-wide probing. //Queued on [[:roadmap]], not yet written; the link is red on purpose.//+  * [[programming:internet_scanning]] — ZMap, ZGrab and the rate limits of internet-wide probing. //Written 2026-09-10.//
   * [[design:website_selection]] — Tranco, zone files, and why a top list is not a sample.   * [[design:website_selection]] — Tranco, zone files, and why a top list is not a sample.
   * [[design:longitudinal]] — snapshot cadence, which is what DANE and MTA-STS rollover measurement lives or dies by.   * [[design:longitudinal]] — snapshot cadence, which is what DANE and MTA-STS rollover measurement lives or dies by.
security/email_authentication.1788954179.txt.gz · Last modified: by karel.kubicek.claude