| Both sides previous revisionPrevious revision | |
| privacy:ads_txt [2026/09/02 17:33] – Generic-review fixes: qualified universals, unit caveats on the HB split, bid-gap confounds, dating, non-crawl papers named. Authored by Claude karel.kubicek.claude | privacy:ads_txt [2026/09/22 07:48] (current) – Generic (fable) review pass applied: 3 of the 21 below-threshold papers were read, not 0; app-ads.txt is named by three papers including Bashir, not two; the bids family is not PETS-centred (2 of 6); 'shrinking' stated as an unmeasured expectation; two cl karel.kubicek.claude |
|---|
| ^ Observable ^ Where a crawler reads it ^ Who declares it ^ One record asserts ^ What it cannot tell you ^ | ^ Observable ^ Where a crawler reads it ^ Who declares it ^ One record asserts ^ What it cannot tell you ^ |
| | ''ads.txt'' | ''https://<root domain>/ads.txt'', root domain = public suffix + 1; redirects are authoritative only if they stay inside that root domain((ads.txt 1.1 §3.1: "Crawlers should incorporate Public Suffix list [16] to derive the root domain"; "the advertising system should follow the redirect and consume the data as authoritative for the source of the redirect, if and only if the redirect is within scope of the original root domain". Default cache expiry 7 days if no cache-control headers.)) | The publisher | //ad system X may sell my inventory under account ID Y, as DIRECT or RESELLER// | Whether X ever did. Whether Y is really this publisher's account: 10% of publishers had ≥1 invalid record and ~10% of files were byte-identical copies distributed by more than one publisher in 2018–2019 {[bashir2019_adstxt]}; the DIRECT ID ''100141'' of one ad network appeared in 42,412 sites in 2023 {[papadogiannakis2025_darkside]} | | | ''ads.txt'' | ''https://<root domain>/ads.txt'', root domain = public suffix + 1; redirects are authoritative only if they stay inside that root domain((ads.txt 1.1 §3.1: "Crawlers should incorporate Public Suffix list [16] to derive the root domain"; "the advertising system should follow the redirect and consume the data as authoritative for the source of the redirect, if and only if the redirect is within scope of the original root domain". Default cache expiry 7 days if no cache-control headers.)) | The publisher | //ad system X may sell my inventory under account ID Y, as DIRECT or RESELLER// | Whether X ever did. Whether Y is really this publisher's account: 10% of publishers had ≥1 invalid record and ~10% of files were byte-identical copies distributed by more than one publisher in 2018–2019 {[bashir2019_adstxt]}; the DIRECT ID ''100141'' of one ad network appeared in 42,412 sites in 2023 {[papadogiannakis2025_darkside]} | |
| | ''app-ads.txt'' | Same file on the **developer website** named in the app-store listing, not on the app | The app developer | Same as above, for app inventory | Anything at all in these seven venues: no paper has crawled it. It appears only as background in two mobile ad-fraud papers | | | ''app-ads.txt'' | Same file on the **developer website** named in the app-store listing, not on the app | The app developer | Same as above, for app inventory | Anything at all in these seven venues: no paper has crawled it. It appears as background in two mobile ad-fraud papers | |
| | ''sellers.json'' | ''https://<ad system domain>/sellers.json''; only ad systems publish one — 2,682 of 7,341,165 Tranco domains in March 2023 {[papadogiannakis2025_darkside]} | The ad system (SSP / exchange) | //seller ID Y is paid by us; it is a PUBLISHER, an INTERMEDIARY or BOTH; its name and domain are …// — unless ''is_confidential'' is set, in which case name and domain are withheld | Who the seller is, for the confidential majority: 75.53% of Google's 1,277,156 seller IDs were confidential in March 2023 {[papadogiannakis2025_darkside]}, and 70.96% of 972,929 on 2026-09-02.((Google's ''sellers.json'', fetched 2026-09-02, 108,724,970 bytes: 972,929 sellers, 690,433 with ''is_confidential = 1''; 972,347 typed PUBLISHER, 582 BOTH; the file's own ''ext.notice'' still reads "This file is a beta and is unverified." The canonical URL ''realtimebidding.google.com/sellers.json'' answers with a 302 to ''storage.googleapis.com/adx-rtb-dictionaries/sellers.json''; a fetcher that does not follow redirects records a 0-byte body. Fetch script and output on the provenance page.)) Whether the seller actually sold anything | | | ''sellers.json'' | ''https://<ad system domain>/sellers.json''; only ad systems publish one — 2,682 of 7,341,165 Tranco domains in March 2023 {[papadogiannakis2025_darkside]} | The ad system (SSP / exchange) | //seller ID Y is paid by us; it is a PUBLISHER, an INTERMEDIARY or BOTH; its name and domain are …// — unless ''is_confidential'' is set, in which case name and domain are withheld | Who the seller is, for the confidential majority: 75.53% of Google's 1,277,156 seller IDs were confidential in March 2023 {[papadogiannakis2025_darkside]}, and 70.96% of 972,929 on 2026-09-02.((Google's ''sellers.json'', fetched 2026-09-02, 108,724,970 bytes: 972,929 sellers, 690,433 with ''is_confidential = 1''; 972,347 typed PUBLISHER, 582 BOTH; the file's own ''ext.notice'' still reads "This file is a beta and is unverified." The canonical URL ''realtimebidding.google.com/sellers.json'' answers with a 302 to ''storage.googleapis.com/adx-rtb-dictionaries/sellers.json''; a fetcher that does not follow redirects records a 0-byte body. Fetch script and output on the provenance page.)) Whether the seller actually sold anything | |
| | **Header bidding (client-side)** | The page's Prebid.js global — ''pbjs.getBidResponses()'', ''getAllWinningBids()'' — or the auction's DOM events and bidder requests | Nobody declares it; you observe it | //bidder B offered CPM c for ad unit U on this page load, for this browser// | Server-side auctions: Prebid Server — "Use Prebid Server to do the processing rather than the client browser"((''docs.prebid.org/overview/intro.html'', fetched 2026-09-02.)) — Amazon TAM and Google's Exchange Bidding are invisible from the client; server-side and hybrid HB were already 48% and 34.7% of the 2019 "facet breakdown" {[pachilakis2019_waterfalls]}, whose unit — sites or auctions — the paper does not state. What was //paid//: a winning bid is not a rendered ad — 25,764 winning bids but 7,117 rendered in {[zeng2022_factors]} | | | **Header bidding (client-side)** | The page's Prebid.js global — ''pbjs.getBidResponses()'', ''getAllWinningBids()'' — or the auction's DOM events and bidder requests | Nobody declares it; you observe it | //bidder B offered CPM c for ad unit U on this page load, for this browser// | Server-side auctions: Prebid Server — "Use Prebid Server to do the processing rather than the client browser"((''docs.prebid.org/overview/intro.html'', fetched 2026-09-02.)) — Amazon TAM and Google's Exchange Bidding are invisible from the client; server-side and hybrid HB were already 48% and 34.7% of the 2019 "facet breakdown" {[pachilakis2019_waterfalls]}, whose unit — sites or auctions — the paper does not state. What was //paid//: a winning bid is not a rendered ad — 25,764 winning bids but 7,117 rendered in {[zeng2022_factors]} | |
| | **SupplyChain object** (''schain'') | Inside OpenRTB bid requests, i.e. only if you are a bidder or you capture the requests a client-side wrapper sends | Each hop of the chain | //this impression passed through (asi, sid) nodes …// | Nearly everything, in practice: only 20.5% of captured bid requests carried one in 2022, all single-node {[vekaria2024_darkpooling]}. **Zero papers in the seven venues mention it** | | | **SupplyChain object** (''schain'') | Inside OpenRTB bid requests, i.e. only if you are a bidder or you capture the requests a client-side wrapper sends | Each hop of the chain | //this impression passed through (asi, sid) nodes …// | Nearly everything, in practice: only 20.5% of captured bid requests carried one in 2022, all single-node {[vekaria2024_darkpooling]}. **Zero papers in the seven venues mention it** | |
| |
| Two things follow. First, ''ads.txt'' and ''sellers.json'' are **declarations**: cheap to fetch at Internet scale, and exactly as truthful as their authors. Second, header bidding is an **observation**, but of a shrinking and self-selected slice — the sites that still run the auction in the browser, for the persona your crawler presents. | Two things follow. First, ''ads.txt'' and ''sellers.json'' are **declarations**: cheap to fetch at Internet scale, and exactly as truthful as their authors. Second, header bidding is an **observation**, but of a self-selected slice — and possibly a shrinking one, which is an expectation the literature states and has not measured since 2019 — the sites that still run the auction in the browser, for the persona your crawler presents. |
| |
| ===== Pick the Unit Before You Pick the Method ===== | ===== Pick the Unit Before You Pick the Method ===== |
| | **Declarations vs observed transactions** | Build ad inclusion chains from a browser crawl, attribute each RTB ad to a seller–buyer pair, check the seller is in the publisher's file; or parse the SupplyChain object out of captured bid requests | 2019 → 2019 (2022 outside the corpus) | 1 — {[bashir2019_adstxt]} (+ {[vekaria2024_darkpooling]}) | **The only test of whether the standard is honoured, and not repeated in these venues since 2019.** Expensive — 135M chains, EasyList to find the ads, WhoTracksMe to fold seller domains into 28 parents — and ethically awkward: the authors "attempted to become a publisher in order to conduct controlled experiments" and every exchange they contacted refused | | | **Declarations vs observed transactions** | Build ad inclusion chains from a browser crawl, attribute each RTB ad to a seller–buyer pair, check the seller is in the publisher's file; or parse the SupplyChain object out of captured bid requests | 2019 → 2019 (2022 outside the corpus) | 1 — {[bashir2019_adstxt]} (+ {[vekaria2024_darkpooling]}) | **The only test of whether the standard is honoured, and not repeated in these venues since 2019.** Expensive — 135M chains, EasyList to find the ads, WhoTracksMe to fold seller domains into 28 parents — and ethically awkward: the authors "attempted to become a publisher in order to conduct controlled experiments" and every exchange they contacted refused | |
| | **Client-side header-bidding instrumentation** | Detect Prebid.js (probe ''pbjs.version''), then read bids via the publisher API or by hooking auction events; record bidder, CPM, ad unit, latency | 2019 → 2025 | 8 — {[pachilakis2019_waterfalls]}, {[cook2020_headerbidding]}, {[zeng2022_factors]}, {[zhang2022_harpo]}, {[musa2022_atom]}, {[iqbal2023_tracking]}, {[liu2024_opted]}, {[liu2025_fingerprinting]} | **Current, with a visibility problem nobody has re-measured.** Client-side HB was 17.3% of the 2019 facet breakdown (unit unstated) and ATOM's authors already wrote that bid-based inference "takes a hit as publishers migrate towards server side Header Bidding" {[musa2022_atom]}. No paper in these venues has measured the client/server split since 2019 | | | **Client-side header-bidding instrumentation** | Detect Prebid.js (probe ''pbjs.version''), then read bids via the publisher API or by hooking auction events; record bidder, CPM, ad unit, latency | 2019 → 2025 | 8 — {[pachilakis2019_waterfalls]}, {[cook2020_headerbidding]}, {[zeng2022_factors]}, {[zhang2022_harpo]}, {[musa2022_atom]}, {[iqbal2023_tracking]}, {[liu2024_opted]}, {[liu2025_fingerprinting]} | **Current, with a visibility problem nobody has re-measured.** Client-side HB was 17.3% of the 2019 facet breakdown (unit unstated) and ATOM's authors already wrote that bid-based inference "takes a hit as publishers migrate towards server side Header Bidding" {[musa2022_atom]}. No paper in these venues has measured the client/server split since 2019 | |
| | **Bids as a dependent variable** | Hold the site fixed and vary the persona (interests, opt-out, fingerprint, smart-speaker history) against a control — or, in the one field study, compare bids across real users' attributes | 2020 → 2025 | 6 — {[cook2020_headerbidding]}, {[zeng2022_factors]}, {[zhang2022_harpo]}, {[iqbal2023_tracking]}, {[liu2024_opted]}, {[liu2025_fingerprinting]} | **Current, and PETS-centred.** It is the only way to see //server-side// data use from the client: advertisers who were never sent the interest still bid higher {[liu2024_opted]}. Confounds are the method's whole difficulty — day of week, slot, holidays {[iqbal2023_tracking]}; Christmas demand {[zeng2022_factors]} | | | **Bids as a dependent variable** | Hold the site fixed and vary the persona (interests, opt-out, fingerprint, smart-speaker history) against a control — or, in the one field study, compare bids across real users' attributes | 2020 → 2025 | 6 — {[cook2020_headerbidding]}, {[zeng2022_factors]}, {[zhang2022_harpo]}, {[iqbal2023_tracking]}, {[liu2024_opted]}, {[liu2025_fingerprinting]} | **Current.** Two of the six are at PoPETs (Cook, Liu 2024); the rest are at IMC, NDSS and TheWebConf, so this is not a PoPETs line of work. It is the only way to see //server-side// data use from the client: advertisers who were never sent the interest still bid higher {[liu2024_opted]}. Confounds are the method's whole difficulty — day of week, slot, holidays {[iqbal2023_tracking]}; Christmas demand {[zeng2022_factors]} | |
| | **Prebid presence as a sampling frame** | Use a Prebid-detected site list (BuiltWith) to pick ad-supported sites, and a Prebid-aware extension to count ads | 2026 → 2026 | 1 — {[lukic2026_mv3]} | Legitimate, but it measures Prebid sites, not the web; say so in the population section | | | **Prebid presence as a sampling frame** | Use a Prebid-detected site list (BuiltWith) to pick ad-supported sites, and a Prebid-aware extension to count ads | 2026 → 2026 | 1 — {[lukic2026_mv3]} | Legitimate, but it measures Prebid sites, not the web; say so in the population section | |
| | **RTB win-notification prices** | Detect the exchange's price-notification URLs (''nURL'' macros) in users' traffic and read or model the cleartext/encrypted clearing price | 2017 → 2018 | 2 — {[papadopoulos2017_paying]}, {[papadopoulos2018_cost]} | **Historical.** It needs real users' traffic through a proxy and cleartext price macros; ~26% of mobile RTB prices were already encrypted in 2015 and encrypted prices ran ~1.7× higher {[papadopoulos2017_paying]}. No paper in these venues has used it since 2018; the Prebid API replaced it as the price observable | | | **RTB win-notification prices** | Detect the exchange's price-notification URLs (''nURL'' macros) in users' traffic and read or model the cleartext/encrypted clearing price | 2017 → 2018 | 2 — {[papadopoulos2017_paying]}, {[papadopoulos2018_cost]} | **Historical.** It needs real users' traffic through a proxy and cleartext price macros; ~26% of mobile RTB prices were already encrypted in 2015 and encrypted prices ran ~1.7× higher {[papadopoulos2017_paying]}. No paper in these venues has used it since 2018; the Prebid API replaced it as the price observable | |
| * **OWNERDOMAIN and MANAGERDOMAIN** (''ads.txt'' 1.1, August 2022) exist precisely to make pooling legible — the owner of a pooled site and its exclusive monetiser are meant to be declared. **No paper in the seven venues, and none among the papers they cite, has measured their adoption**; both 2025 studies still infer ownership from WHOIS. This is the cheapest open measurement on the page. | * **OWNERDOMAIN and MANAGERDOMAIN** (''ads.txt'' 1.1, August 2022) exist precisely to make pooling legible — the owner of a pooled site and its exclusive monetiser are meant to be declared. **No paper in the seven venues, and none among the papers they cite, has measured their adoption**; both 2025 studies still infer ownership from WHOIS. This is the cheapest open measurement on the page. |
| * **The SupplyChain object** has one measurement that this page could find, outside the seven venues, from 2022: present in 20.5% of bid requests, always one node {[vekaria2024_darkpooling]}. ''buyers.json'' and ''ads.cert'' have none. | * **The SupplyChain object** has one measurement that this page could find, outside the seven venues, from 2022: present in 20.5% of bid requests, always one node {[vekaria2024_darkpooling]}. ''buyers.json'' and ''ads.cert'' have none. |
| * **''app-ads.txt''** has none in these venues. The two CCS papers that name it are mobile ad-fraud studies — one from an ad network's invalid-traffic logs, one from static and dynamic analysis of apps — and both treat the file as background. | * **''app-ads.txt''** has none in these venues. **Three papers name it at all** — {[bashir2019_adstxt]}, which raises it and asks for "a separate study", and two CCS mobile ad-fraud studies — one from an ad network's invalid-traffic logs, one from static and dynamic analysis of apps — and both treat the file as background. |
| * **The client/server header-bidding split** has not been re-measured since February 2019. Every bids-as-instrument result since is conditioned on the sites that still expose the auction. Two report what share of their candidate list that was — {[liu2024_opted]} (5,421 of the Alexa top-100K) and {[zeng2022_factors]} (703 of the Tranco top 10,000); {[iqbal2023_tracking]} stopped at 200 Prebid sites and does not. The client / hybrid / server split //among// HB sites is what nobody has re-measured. | * **The client/server header-bidding split** has not been re-measured since February 2019. Every bids-as-instrument result since is conditioned on the sites that still expose the auction. Two report what share of their candidate list that was — {[liu2024_opted]} (5,421 of the Alexa top-100K) and {[zeng2022_factors]} (703 of the Tranco top 10,000); {[iqbal2023_tracking]} stopped at 200 Prebid sites and does not. The client / hybrid / server split //among// HB sites is what no paper in these venues has re-measured. |
| * **LLM-based methods**: none. The classification steps in this literature — ad system name folding, brand extraction from landing pages, misinformation labels — are regex, curated lists and hand review, and the 2025–2026 slice does not change that. | * **LLM-based methods**: none. The classification steps in this literature — ad system name folding, brand extraction from landing pages, misinformation labels — are regex, curated lists and hand review, and the 2025–2026 slice does not change that. |
| |
| | USENIX Security | 1,410 | 2 (0.1%) | 0 (0.0%) | | | USENIX Security | 1,410 | 2 (0.1%) | 0 (0.0%) | |
| |
| **Three venues hold it.** Four of the five ''ads.txt'' / ''sellers.json'' papers are at IMC or TheWebConf; three of the seven papers that use bids as a privacy instrument are at PoPETs. The IEEE S&P row is the one to notice: the venue's single mention is a passing one, and the venue's actual ''ads.txt'' paper — {[vekaria2024_darkpooling]} — is not in the corpus because abstract screening rejected it. That is a fact about the funnel, not about IEEE S&P. | **Three venues hold it.** Four of the five ''ads.txt'' / ''sellers.json'' papers are at IMC or TheWebConf; three of the seven papers that use bids as a privacy instrument are at PoPETs. The IEEE S&P row is the one to notice: the venue's single mention is in a cookie-banner paper the run did not read, and the venue's actual ''ads.txt'' paper — {[vekaria2024_darkpooling]} — is not in the corpus because abstract screening rejected it. That is a fact about the funnel, not about IEEE S&P. |
| |
| ==== By year ==== | ==== By year ==== |
| ==== The measuring set ==== | ==== The measuring set ==== |
| |
| The 18 papers with five or more core mentions, plus two forced in by hand (the 2017 RTB-price paper and the 2022 publisher-ID paper), were read and labelled. **16 measure** — read one of the files or observe bids — **1 is a related method** and **3 only cite**. Zero were left unlabelled; the 21 papers with one to four mentions were //not// read and are listed on the provenance page. | The 18 papers with five or more core mentions, plus two forced in by hand (the 2017 RTB-price paper and the 2022 publisher-ID paper), were read and labelled. **16 measure** — read one of the files or observe bids — **1 is a related method** and **3 only cite**. Zero were left unlabelled. Of the 21 papers with one to four mentions, **3 were read** during the run and **18 were not**; all 21 are listed on the provenance page. |
| |
| ^ Family ^ Papers ^ Years ^ | ^ Family ^ Papers ^ Years ^ |
| - **For pooling**: how ownership was established (WHOIS, publisher IDs, hand review) and for what share of pool members it could not be. | - **For pooling**: how ownership was established (WHOIS, publisher IDs, hand review) and for what share of pool members it could not be. |
| - **For header bidding**: how Prebid sites were detected (global name, events, or both), which API (''getBidResponses'' vs winning vs rendered), the wait after ''load'', the persona and control design, the vantage, the consent action, and the browser's protection state. | - **For header bidding**: how Prebid sites were detected (global name, events, or both), which API (''getBidResponses'' vs winning vs rendered), the wait after ''load'', the persona and control design, the vantage, the consent action, and the browser's protection state. |
| - **What fraction of your candidate sites exposed a client-side auction at all** — two of the seven bids papers report it; and, if you can, which of those auctions were client-only, hybrid or server-side, which nobody has reported since 2019. | - **What fraction of your candidate sites exposed a client-side auction at all** — two of the seven bids papers report it; and, if you can, which of those auctions were client-only, hybrid or server-side, which no paper in these venues has reported since 2019. |
| |
| ===== Open Questions ===== | ===== Open Questions ===== |
| |
| * [[Privacy:Cookie syncing]] — the other half of the ad graph: whether two parties' identifiers met. Cook 2020 and ATOM sit on both pages. | * [[Privacy:Cookie syncing]] — the other half of the ad graph: whether two parties' identifiers met. Cook 2020 and ATOM sit on both pages. |
| * [[Privacy:Requests]] and [[Programming:Filter lists]] — EasyList is how every paper here found the ads, and blocks the auction if left on. | * [[Privacy:Requests]] and [[Programming:Filter lists]] — EasyList is how the ad-attribution crawls here found the ads — and it blocks the Prebid endpoints, so leaving it on removes the auction the bids papers depend on. |
| * [[Privacy:Consent]] and [[Privacy:TCF consent strings]] — the CMP-and-Prebid intersection is the population for a consent-and-bids study. | * [[Privacy:Consent]] and [[Privacy:TCF consent strings]] — the CMP-and-Prebid intersection is the population for a consent-and-bids study. |
| * [[Privacy:Privacy Sandbox]] — the Protected Audience auction was the would-be successor to header bidding; it is being withdrawn. | * [[Privacy:Privacy Sandbox]] — the Protected Audience auction was the would-be successor to header bidding; it is being withdrawn. |
