Measure The Web

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Measuring the Ad Supply Chain: ads.txt, sellers.json and Header Bidding ======

Three things about the advertising supply chain can be read from the outside without being a buyer or a seller. A publisher declares who may sell its inventory in ''ads.txt'' (for apps, ''app-ads.txt'' on the developer's site). An ad system declares whom it pays in ''sellers.json''. And where the auction runs in the browser — client-side header bidding, in practice Prebid.js — the bids themselves are visible to a script on the page. Together they are the **observable ad graph**: which parties are //authorised// to sell a site's ad space, which parties //say// they represent it, and which parties //actually bid// on it when your crawler shows up.

**This page is about measuring that graph, not about the ad-tech business it describes.** It is the mechanism behind a family of 2019–2026 papers that ask who funds misinformation, whether "authorised seller" lists are honoured, whether privacy signals change what advertisers pay, and how low-quality publishers launder inventory by pooling it under someone else's seller ID. It is **not** about identifier exchange between parties — that is [[Privacy:Cookie syncing]], and two papers ({[cook2020_headerbidding]}, {[musa2022_atom]}) appear on both pages for different reasons: there because bids are an //ad-semantics// way to infer that data was shared, here because bids are the observable.

<WRAP important>
**The load-bearing decision on this page is what you count as a "relationship".** The same pair (publisher, ad system) can be: **declared** — a line in the publisher's ''ads.txt''; **confirmed** — that line's seller ID also appears in the ad system's ''sellers.json'', and the ''sellers.json'' domain points back at the publisher; **observed** — a bid, or an ad inclusion chain, that actually passed through that ad system on a page your crawler loaded; or **paid** — money moved, which no external measurement sees. Papers in these venues report all of the first three and are not consistent about which. Bashir et al. {[bashir2019_adstxt]} needed 135M observed ad inclusion chains to test whether declarations were honoured; since 2022 ({[vekaria2024_darkpooling]}, then {[papadogiannakis2023_whofunds]} in these venues) every ''ads.txt'' paper — four in the corpus, three of them from one group — has reported the //confirmed// relationship, the ''ads.txt'' ∩ ''sellers.json'' intersection, because "each individual file of the above two files may provide fake information, their intersection provides the truth" {[papadogiannakis2025_before]}. State which of the four you measured, in the abstract.
</WRAP>

===== What to Read First =====

  * **A Longitudinal Analysis of the ads.txt Standard** {[bashir2019_adstxt]}, IMC 2019 — the reference measurement of the standard itself: 26 crawls of the Alexa Top-100K over 15 months, the record-error taxonomy, and the only compliance test in these venues that matched declarations against //observed// ad traffic (135M RTB inclusion chains). Read Section 3 for the crawl, Section 5 for why compliance cannot be tested without becoming a publisher, and Section 8 for the caveats you will be quoting.
  * **The Inventory is Dark and Full of Misinformation** {[vekaria2024_darkpooling]}, IEEE S&P 2024 — the paper that turned ''ads.txt'' and ''sellers.json'' from an adoption question into a fraud-detection instrument. It names the misrepresentation classes, defines **dark pooling** (unrelated publishers sharing one seller ID), and is the only paper cited on this page that parsed the OpenRTB SupplyChain object from captured bid requests. **It is not in the corpus behind this site** — the screening model labelled it neither a security nor a privacy measurement — so nothing in [[#Use in Publications]] counts it. Read it from the arXiv version.
  * **Welcome to the Dark Side** {[papadogiannakis2025_darkside]}, TheWebConf 2025 — the scale study: ''ads.txt'' from 456,971 of ~7M Tranco domains, ''sellers.json'' from 2,682 ad systems, 185,535 seller-ID pools, ~15,000 of them dark. Released crawlers and a public monitoring service. Read it for what a 2023-era pipeline looks like end to end, and for the ''sellers.json'' confidentiality figures.
  * **No More Chasing Waterfalls** {[pachilakis2019_waterfalls]}, IMC 2019 — the header-bidding reference: how to detect an auction from DOM events and requests, the client/server/hybrid split (server-side was already 48% in 2019, in a breakdown whose unit the paper does not state), and latency. Its tool link is dead (see [[#Tools, datasets and services that already exist]]).
  * **Inferring Tracker-Advertiser Relationships … using Header Bidding** {[cook2020_headerbidding]}, PoPETs 2020 — the paper that made bids a //dependent variable//: call ''getBidResponses()'', change one thing about the persona, watch who bids more. Every paper in these venues since that uses bids as evidence of data use ({[musa2022_atom]}, {[iqbal2023_tracking]}, {[liu2024_opted]}, {[liu2025_fingerprinting]}) is a descendant.

===== The Three Observables, and What Each Cannot See =====

The minimum a measurement needs to know about each file is where it lives, who wrote it, and what one record asserts. Everything else is in the IAB Tech Lab specifications((ads.txt Specification Version 1.1, IAB Tech Lab, "Released August 2022", PDF linked from ''iabtechlab.com/standards/ads-txt/''. sellers.json 1.0, "July 2019", linked from ''iabtechlab.com/sellers-json/''. SupplyChain object 1.0, ''github.com/InteractiveAdvertisingBureau/openrtb/blob/master/supplychainobject.md''. All fetched 2026-09-02.)) and does not belong here.

^ Observable ^ Where a crawler reads it ^ Who declares it ^ One record asserts ^ What it cannot tell you ^
| ''ads.txt'' | ''https://<root domain>/ads.txt'', root domain = public suffix + 1; redirects are authoritative only if they stay inside that root domain((ads.txt 1.1 §3.1: "Crawlers should incorporate Public Suffix list [16] to derive the root domain"; "the advertising system should follow the redirect and consume the data as authoritative for the source of the redirect, if and only if the redirect is within scope of the original root domain". Default cache expiry 7 days if no cache-control headers.)) | The publisher | //ad system X may sell my inventory under account ID Y, as DIRECT or RESELLER// | Whether X ever did. Whether Y is really this publisher's account: 10% of publishers had ≥1 invalid record and ~10% of files were byte-identical copies distributed by more than one publisher in 2018–2019 {[bashir2019_adstxt]}; the DIRECT ID ''100141'' of one ad network appeared in 42,412 sites in 2023 {[papadogiannakis2025_darkside]} |
| ''app-ads.txt'' | Same file on the **developer website** named in the app-store listing, not on the app | The app developer | Same as above, for app inventory | Anything at all in these seven venues: no paper has crawled it. It appears as background in two mobile ad-fraud papers |
| ''sellers.json'' | ''https://<ad system domain>/sellers.json''; only ad systems publish one — 2,682 of 7,341,165 Tranco domains in March 2023 {[papadogiannakis2025_darkside]} | The ad system (SSP / exchange) | //seller ID Y is paid by us; it is a PUBLISHER, an INTERMEDIARY or BOTH; its name and domain are …// — unless ''is_confidential'' is set, in which case name and domain are withheld | Who the seller is, for the confidential majority: 75.53% of Google's 1,277,156 seller IDs were confidential in March 2023 {[papadogiannakis2025_darkside]}, and 70.96% of 972,929 on 2026-09-02.((Google's ''sellers.json'', fetched 2026-09-02, 108,724,970 bytes: 972,929 sellers, 690,433 with ''is_confidential = 1''; 972,347 typed PUBLISHER, 582 BOTH; the file's own ''ext.notice'' still reads "This file is a beta and is unverified." The canonical URL ''realtimebidding.google.com/sellers.json'' answers with a 302 to ''storage.googleapis.com/adx-rtb-dictionaries/sellers.json''; a fetcher that does not follow redirects records a 0-byte body. Fetch script and output on the provenance page.)) Whether the seller actually sold anything |
| **Header bidding (client-side)** | The page's Prebid.js global — ''pbjs.getBidResponses()'', ''getAllWinningBids()'' — or the auction's DOM events and bidder requests | Nobody declares it; you observe it | //bidder B offered CPM c for ad unit U on this page load, for this browser// | Server-side auctions: Prebid Server — "Use Prebid Server to do the processing rather than the client browser"((''docs.prebid.org/overview/intro.html'', fetched 2026-09-02.)) — Amazon TAM and Google's Exchange Bidding are invisible from the client; server-side and hybrid HB were already 48% and 34.7% of the 2019 "facet breakdown" {[pachilakis2019_waterfalls]}, whose unit — sites or auctions — the paper does not state. What was //paid//: a winning bid is not a rendered ad — 25,764 winning bids but 7,117 rendered in {[zeng2022_factors]} |
| **SupplyChain object** (''schain'') | Inside OpenRTB bid requests, i.e. only if you are a bidder or you capture the requests a client-side wrapper sends | Each hop of the chain | //this impression passed through (asi, sid) nodes …// | Nearly everything, in practice: only 20.5% of captured bid requests carried one in 2022, all single-node {[vekaria2024_darkpooling]}. **Zero papers in the seven venues mention it** |

Two things follow. First, ''ads.txt'' and ''sellers.json'' are **declarations**: cheap to fetch at Internet scale, and exactly as truthful as their authors. Second, header bidding is an **observation**, but of a self-selected slice — and possibly a shrinking one, which is an expectation the literature states and has not measured since 2019 — the sites that still run the auction in the browser, for the persona your crawler presents.

===== Pick the Unit Before You Pick the Method =====

Published figures in this area are not comparable across rows. Each is right for its own unit and denominator; none is "the adoption rate of ads.txt".

^ Unit ^ A published figure in that unit ^ Its denominator, stated ^
| **Sites serving a valid ''ads.txt''** | 12.7% → 19.7% between January 2018 and April 2019 {[bashir2019_adstxt]} | Alexa Top-100K (the January-2018 set; a re-fetched 100K gives the same curve); 26 crawls. The paper's full crawl was the union of every list, 240K sites |
| **Sites serving one, conditioned on showing RTB ads** | 46.6% → 62.3% over the same 15 months {[bashir2019_adstxt]} | The subset of those sites on which the paper's own browser crawl saw RTB advertisements |
| **Sites serving one, at the scale of the whole list** | 456,971 domains {[papadogiannakis2025_darkside]} | ~7M Tranco domains, February–March 2023, fetched with the IAB reference crawler |
| **Sites in one vertical** | 198 fake-news and 627 real-news sites (13 Dec 2021); 262 fake-news sites on re-crawl (31 Jan 2023) {[papadogiannakis2023_whofunds]} | 1,044 fake-news and 1,368 real-news sites from MediaBias/FactCheck, CJR and two academic lists |
| | 35 sites (9.1%) {[sheaib2025_unmasking]} | 383 illegal movie-streaming sites, August 2024 |
| **Ad systems declared per site** | median 17 sellers; top 20% list ≥ 42 {[bashir2019_adstxt]} | Publishers with a valid file, April 2019 |
| | fake-news sites average 27 DIRECT systems, real-news 41 {[papadogiannakis2023_whofunds]} | Sites with a valid file in each list |
| **Share of a vertical with a DIRECT line to one exchange** | 80.8% Google, 52.5% AppNexus, 49% Index Exchange {[papadogiannakis2023_whofunds]} | 198 fake-news sites with ''ads.txt'' |
| | 77.1% Google {[sheaib2025_unmasking]} | 35 streaming sites with ''ads.txt'' |
| **Confirmed relationships** (''ads.txt'' ∩ ''sellers.json'') | 985 (87%) misinformation sites with ≥ 1 substantiated relationship {[papadogiannakis2025_before]} | 1,132 misinformation sites with a file, January 2024 |
| **Seller-ID pools** | 79K (seller ID, exchange) pairs shared by > 1 domain {[vekaria2024_darkpooling]} | Tranco top-100K sites with ''ads.txt'', February 2022 |
| | 185,535 pools, ~15,000 of them //dark// (different WHOIS owners) {[papadogiannakis2025_darkside]} | 591,546 distinct DIRECT IDs from 456,971 files; WHOIS owner known for 3,981 sites |
| **Pools containing a low-quality site that are dark** | 59.6% of pools with a misinformation site; 82% with a piracy site {[papadogiannakis2025_darkside]} | Pools intersecting 1,163 MBFC and 1,395 NextDNS-listed sites |
| **''sellers.json'' entries that are problematic** | confidential sellers 46.1% vs 0.1%; non-unique seller IDs 95.3% vs 62.6% {[vekaria2024_darkpooling]} | Entries of exchanges that list ≥ 1 misinformation site vs exchanges that list none |
| **Sites running header bidding** | 14.28% (~5,000); 20–23% of the top 5K {[pachilakis2019_waterfalls]} | 35,000 top Alexa sites, February 2019, stateless Chrome |
| | 5,421 sites with the default ''pbjs'' global; 352 with a CMP //and// Prebid.js {[liu2024_opted]} | Alexa top-100K; the 352 is the paper's audit set |
| | 703 sites with Prebid.js {[zeng2022_factors]} | Tranco top 10,000, late 2021 |
| **Demand partners** | 84 unique; DFP on > 80% of HB publishers {[pachilakis2019_waterfalls]} | ~5,000 HB sites crawled daily for 34 days |
| **Winning bid** | mean $5.47, median $4.16 CPM {[zeng2022_factors]} | 7,117 //rendered// winning bids from 286 real US users on 10 fixed sites, 10–21 December 2021 |
| | 300×250 median 0.031 CPM {[pachilakis2019_waterfalls]} | Bids received by a clean stateless crawler, February 2019 — a baseline persona, not a user |
| **Bid ratio under a treatment** | up to 30× the control mean {[iqbal2023_tracking]}; 6.28× {[zhang2022_harpo]}; personas bid //higher// than control after opting out {[liu2024_opted]} | 200, 3 and 352 Prebid sites respectively; the treatment differs in each |

**Three consequences.** The 2019 "20%" and the 2023 "456,971" are not two points on one adoption curve; one is a share of a top list, the other a count over a seven-million-domain tail. A stateless crawler's bids are a price for //nobody//: 0.031 CPM median for a 300×250 slot in 2019, against $4.16 median //winning// bid from real users on ten US news sites in December 2021. The gap is two orders of magnitude, and slot mix, winning-versus-all bids, site selection and season are all inside it; no paper has run the two personas side by side. And every ''sellers.json'' figure has a hidden denominator: whatever share of that exchange's sellers is confidential.

===== Methods, and Which Ones Are Current =====

<WRAP important>
A ranking of what the literature //did// is a fact about the literature, not advice about what to do now. Statuses are as of **2026-09-02**. The corpus behind the counts is thin here — 16 measuring papers across seven venues, 2017–2026 — so each //current// or //historical// label rests on the named papers plus the vendor and specification checks logged on [[provenance:privacy:ads_txt]], not on a trend. The 2025–2026 venue-years are provisional; read the labels as arguments.
</WRAP>

Sixteen papers in the corpus read one of the three observables or observe bids (see [[#Use in Publications]] for how the set was built). One more measures ownership by a sibling method without touching ''ads.txt'', and is the last row.

^ Family ^ What it does ^ First / most recent in corpus ^ Papers ^ Status in 2026 ^
| **Fetch-and-parse ''ads.txt'' at list scale** | ''GET /ads.txt'' on every root domain of a top list; parse the four fields; count sites, records, systems, DIRECT vs RESELLER, errors | 2019 → 2025 | 5 — {[bashir2019_adstxt]}, {[papadogiannakis2023_whofunds]}, {[papadogiannakis2025_darkside]}, {[papadogiannakis2025_before]}, {[sheaib2025_unmasking]} (+ {[vekaria2024_darkpooling]} outside the corpus) | **Current and cheap.** Bashir's 100K sites took "2–3 hours" on a 16-node cluster; the 2025 study used the IAB reference crawler with only the User-Agent changed. Re-crawl: every 15–30 days {[bashir2019_adstxt]}, or one request per domain per month {[papadogiannakis2025_darkside]}. Alone it yields //declared// relationships only |
| **''ads.txt'' ∩ ''sellers.json'' intersection** | For each DIRECT record, look the seller ID up in the named ad system's ''sellers.json''; keep the relationship only if the exchange confirms it and its ''domain'' matches | 2023 → 2025 | 4 — {[papadogiannakis2023_whofunds]}, {[papadogiannakis2025_darkside]}, {[papadogiannakis2025_before]}, {[sheaib2025_unmasking]} (+ {[vekaria2024_darkpooling]}) | **Current default for a //confirmed// relationship.** Recursive: an INTERMEDIARY's domain should host its own ''sellers.json'', so crawl the graph, not a list {[vekaria2024_darkpooling]}. Bounded by confidentiality — on piracy sites only 5 of 188 DIRECT seller entries could be confirmed; among the 587 further unique DIRECT entries the paper counts separately, 169 IDs were absent from ''sellers.json'' and 418 had a domain mismatch, and the paper does not reconcile the two totals {[sheaib2025_unmasking]} |
| **Pooling: shared seller IDs → ownership check** | Group sites by (ad system, seller ID); call a pool //dark// when its members have different owners, established from WHOIS, page-level Google publisher IDs, or by hand | 2025 → 2025 (2022 outside the corpus) | 1 — {[papadogiannakis2025_darkside]} (+ {[vekaria2024_darkpooling]}); ownership method from {[papadogiannakis2022_publisherids]} | **Current, and where the 2022–2026 findings come from.** The ownership step is the weak link: WHOIS was redacted for most of 185,535 pools' members (owner found for 3,981 sites), and page-level publisher IDs are Google-only |
| **Declarations vs observed transactions** | Build ad inclusion chains from a browser crawl, attribute each RTB ad to a seller–buyer pair, check the seller is in the publisher's file; or parse the SupplyChain object out of captured bid requests | 2019 → 2019 (2022 outside the corpus) | 1 — {[bashir2019_adstxt]} (+ {[vekaria2024_darkpooling]}) | **The only test of whether the standard is honoured, and not repeated in these venues since 2019.** Expensive — 135M chains, EasyList to find the ads, WhoTracksMe to fold seller domains into 28 parents — and ethically awkward: the authors "attempted to become a publisher in order to conduct controlled experiments" and every exchange they contacted refused |
| **Client-side header-bidding instrumentation** | Detect Prebid.js (probe ''pbjs.version''), then read bids via the publisher API or by hooking auction events; record bidder, CPM, ad unit, latency | 2019 → 2025 | 8 — {[pachilakis2019_waterfalls]}, {[cook2020_headerbidding]}, {[zeng2022_factors]}, {[zhang2022_harpo]}, {[musa2022_atom]}, {[iqbal2023_tracking]}, {[liu2024_opted]}, {[liu2025_fingerprinting]} | **Current, with a visibility problem nobody has re-measured.** Client-side HB was 17.3% of the 2019 facet breakdown (unit unstated) and ATOM's authors already wrote that bid-based inference "takes a hit as publishers migrate towards server side Header Bidding" {[musa2022_atom]}. No paper in these venues has measured the client/server split since 2019 |
| **Bids as a dependent variable** | Hold the site fixed and vary the persona (interests, opt-out, fingerprint, smart-speaker history) against a control — or, in the one field study, compare bids across real users' attributes | 2020 → 2025 | 6 — {[cook2020_headerbidding]}, {[zeng2022_factors]}, {[zhang2022_harpo]}, {[iqbal2023_tracking]}, {[liu2024_opted]}, {[liu2025_fingerprinting]} | **Current.** Two of the six are at PoPETs (Cook, Liu 2024); the rest are at IMC, NDSS and TheWebConf, so this is not a PoPETs line of work. It is the only way to see //server-side// data use from the client: advertisers who were never sent the interest still bid higher {[liu2024_opted]}. Confounds are the method's whole difficulty — day of week, slot, holidays {[iqbal2023_tracking]}; Christmas demand {[zeng2022_factors]} |
| **Prebid presence as a sampling frame** | Use a Prebid-detected site list (BuiltWith) to pick ad-supported sites, and a Prebid-aware extension to count ads | 2026 → 2026 | 1 — {[lukic2026_mv3]} | Legitimate, but it measures Prebid sites, not the web; say so in the population section |
| **RTB win-notification prices** | Detect the exchange's price-notification URLs (''nURL'' macros) in users' traffic and read or model the cleartext/encrypted clearing price | 2017 → 2018 | 2 — {[papadopoulos2017_paying]}, {[papadopoulos2018_cost]} | **Historical.** It needs real users' traffic through a proxy and cleartext price macros; ~26% of mobile RTB prices were already encrypted in 2015 and encrypted prices ran ~1.7× higher {[papadopoulos2017_paying]}. No paper in these venues has used it since 2018; the Prebid API replaced it as the price observable |
| **(Publisher-ID ownership graph)** | Extract Google ''pub-'', ''UA-''/''G-'' and ''GTM-'' identifiers from HTML, requests and cookies; link sites that share one | 2022 → 2022 | 1 — {[papadogiannakis2022_publisherids]} | Not an ''ads.txt'' method, and listed because the pooling row depends on it. ~10% of 962K Tranco sites carried a Publisher ID in April 2021 |

==== What is superseded, and what has never been measured ====

  * **''ads.txt'' alone as evidence of a relationship.** Bashir et al. could only test declarations against traffic. Since {[vekaria2024_darkpooling]} (crawl of February 2022) and {[papadogiannakis2023_whofunds]} the intersection with ''sellers.json'' is what every ''ads.txt'' paper reports — four in the corpus, three from one group — and the 2025 papers treat a DIRECT line the exchange does not confirm as unverified, not as a relationship. If you can only fetch one file, you have measured //declarations//, and the title of your figure should say so.
  * **Alexa as the sampling frame.** 8 of the 16 measuring papers drew from Alexa; the list no longer exists, and the two 2019 reference studies are therefore unrepeatable as designed. See [[Design:Website selection]].
  * **Assuming the Prebid global is called ''pbjs''.** It is only the default: the build option ''globalVarName'' renames it and ''defineGlobal: false'' removes it.((''github.com/prebid/Prebid.js'' README, build customisation table: ''globalVarName'' — "Prebid global variable name", default ''pbjs''; ''defineGlobal'' — "If false, do not set a global variable". Fetched 2026-09-02.)) {[liu2024_opted]} says explicitly that it did "not consider personalized custom API labels (i.e., other than pbjs)", so its 5,421 is a lower bound. Detect the auction from its events and bidder requests as well {[pachilakis2019_waterfalls]}, or accept and state the undercount.
  * **OWNERDOMAIN and MANAGERDOMAIN** (''ads.txt'' 1.1, August 2022) exist precisely to make pooling legible — the owner of a pooled site and its exclusive monetiser are meant to be declared. **No paper in the seven venues, and none among the papers they cite, has measured their adoption**; both 2025 studies still infer ownership from WHOIS. This is the cheapest open measurement on the page.
  * **The SupplyChain object** has one measurement that this page could find, outside the seven venues, from 2022: present in 20.5% of bid requests, always one node {[vekaria2024_darkpooling]}. ''buyers.json'' and ''ads.cert'' have none.
  * **''app-ads.txt''** has none in these venues. **Three papers name it at all** — {[bashir2019_adstxt]}, which raises it and asks for "a separate study", and two CCS mobile ad-fraud studies — one from an ad network's invalid-traffic logs, one from static and dynamic analysis of apps — and both treat the file as background.
  * **The client/server header-bidding split** has not been re-measured since February 2019. Every bids-as-instrument result since is conditioned on the sites that still expose the auction. Two report what share of their candidate list that was — {[liu2024_opted]} (5,421 of the Alexa top-100K) and {[zeng2022_factors]} (703 of the Tranco top 10,000); {[iqbal2023_tracking]} stopped at 200 Prebid sites and does not. The client / hybrid / server split //among// HB sites is what no paper in these venues has re-measured.
  * **LLM-based methods**: none. The classification steps in this literature — ad system name folding, brand extraction from landing pages, misinformation labels — are regex, curated lists and hand review, and the 2025–2026 slice does not change that.

===== Crawl Configuration That Decides What You See =====

==== Fetching ads.txt and sellers.json ====

  * **Root domain, not hostname.** The file is authoritative for the public-suffix-plus-one domain; a crawl keyed on ''www.'' hostnames or on subdomains double-counts and misses. Follow redirects only within that root domain — the specification makes an off-domain redirect non-authoritative, and the IAB reference crawler implements that rule.((''github.com/InteractiveAdvertisingBureau/adstxtcrawler'' — "A reference implementation in python of a simple crawler for Ads.txt"; created 2017-05-26, last pushed 2024-06-06, no licence file. Fetched via the GitHub API 2026-09-02.))
  * **Validate before you count.** 56.4% of the 2,381 seller domains ever named in Alexa Top-100K files failed WHOIS/DNS validation {[bashir2019_adstxt]}; 10% of publishers had at least one syntactically invalid record. Report records //that follow the specification// and say how many you dropped.
  * **A User-Agent that looks like a browser.** The 2025 scale crawl kept the IAB crawler "as-is and only change the user-agent header so that we are not blocked by websites" {[papadogiannakis2025_darkside]}. And follow redirects: Google's canonical ''sellers.json'' URL is a 302 to a Cloud Storage object, and a fetcher that stops at the first response records a 0-byte file.
  * **Fold the ad-system names before you rank them.** ''google.com'', ''doubleclick.net'' and friends are one seller; Bashir et al. mapped 101 domains to 28 parents with WhoTracksMe and call the clustering "not perfect". Print the residue. Use the entity maps on [[Design:Existing datasets]] rather than writing your own alias list.
  * **''sellers.json'' is a graph crawl.** Start from the ad-system domains your ''ads.txt'' corpus names, fetch each ''/sellers.json'', then recurse into every INTERMEDIARY / BOTH domain listed {[vekaria2024_darkpooling]}, {[papadogiannakis2025_darkside]}. Expect non-standard locations for the biggest systems — Google's is not at ''google.com/sellers.json'' — and expect copies: 28 domains served a copy of Google's file {[papadogiannakis2025_darkside]}.
  * **Time-stamp everything and keep the raw files.** Relationships churn: 39 of 470 exchanges de-listed at least one misinformation site between October 2021 and February 2022, and one dropped ~87% of its domains in a month {[vekaria2024_darkpooling]}; Google, Lijit and Amazon dropped 51%, 43% and 37% of their misinformation-site relationships between December 2021 and January 2024 {[papadogiannakis2025_before]}. A single snapshot is a date, not a fact about the ecosystem. Rate: one request per domain per month was judged enough {[papadogiannakis2025_darkside]}.
  * **Ownership evidence is the bottleneck of every pooling claim.** WHOIS is redacted for most domains (owner recovered for 3,981 sites out of 185,535 pools' members {[papadogiannakis2025_darkside]}); Google publisher IDs cover only Google-monetised sites {[papadogiannakis2022_publisherids]}; hand review does not scale. Report the share of pools for which ownership could //not// be established, as {[vekaria2024_darkpooling]}'s Table 6 does.

==== Observing header bidding ====

  * **Detect, then read, then wait.** The pattern in {[cook2020_headerbidding]}, {[iqbal2023_tracking]} and {[liu2024_opted]}: probe ''pbjs.version'' (or hook the auction events) to find Prebid sites, then call ''pbjs.getBidResponses()'' — and ''requestBids()'' yourself if no auction ran {[liu2024_opted]}. The documented response object carries ''bidder'', ''cpm'', ''currency'', ''adUnitCode'', ''timeToRespond'' and ''mediaType''.((''docs.prebid.org/dev-docs/publisher-api-reference/getBidResponses.html'', fetched 2026-09-02.)) Auctions are slow: median 600 ms, 4% of sites over 5 s {[pachilakis2019_waterfalls]}; a crawler that leaves at ''load'' has no bids.
  * **A bid is not a rendered ad, and a rendered ad is not a paid one.** {[zeng2022_factors]} distinguishes ''getBidResponses()'' (all bids), ''getAllWinningBids()'' (rendered) and ''getAllPrebidWinningBids()'' (won the auction, then lost to the ad server's waterfall) and analyses only the 7,117 rendered winners of 25,764. Choose one and name it.
  * **Your persona is the price.** A stateless clean browser is a baseline nobody targets: 0.031 CPM median bid in 2019 {[pachilakis2019_waterfalls]} against $4.16 median winning bid from real users in 2021 {[zeng2022_factors]} — not the same unit, but the same direction as every treatment study below. Every crawler-based treatment design in the table above ({[cook2020_headerbidding]}, {[zhang2022_harpo]}, {[iqbal2023_tracking]}, {[liu2024_opted]}, {[liu2025_fingerprinting]}) runs a control persona alongside, from the same IP range, on the same day; {[liu2024_opted]} ran 17 personas per jurisdiction from Frankfurt and Northern California and visited each site nine times. Fewer repetitions than that and day-of-week and slot effects swamp the treatment {[iqbal2023_tracking]}.
  * **Zero bids are data.** 22% of all bids in {[cook2020_headerbidding]} were zero, and the paper cannot say whether that is misconfiguration or intent. Do not drop them silently.
  * **Consent and vantage change who bids.** A European vantage brings a banner into the path, and 352 of the Alexa top-100K had both a CMP and Prebid — that intersection, not the 5,421 Prebid sites, is the auditable population for a consent study {[liu2024_opted]}. See [[Privacy:Consent]] and [[Design:Crawling location]]. Of the 16 measuring papers, 11 state a vantage location; 6 of those are the United States.
  * **Ad blockers and filter lists.** Prebid endpoints are on EasyList, so a crawler with an EasyList-based blocker sees no auction; whether Firefox's tracking protection does the same has not been measured in these venues. {[musa2022_atom]} disabled tracking protection in OpenWPM and says so; {[cook2020_headerbidding]} only notes browsers' default protections as a caveat — see [[Privacy:Browser protection]].
  * **Real users are the only way to price a person rather than a persona.** The one field study {[zeng2022_factors]} — a browser extension in 286 Prolific participants' own browsers — is the only bid dataset in these venues that is about people. Its cost is ten fixed sites and an eleven-day window before Christmas.

===== Tools, datasets and services that already exist =====

Checked on 2026-09-02; the check is logged on [[provenance:privacy:ads_txt]]. A dead link in a 2019 paper is not a dead method, but you should not cite the link.

^ What ^ From ^ State on 2026-09-02 ^ Use it for ^
| IAB reference ''ads.txt'' crawler (Python) | IAB Tech Lab | Public; last commit June 2024; no licence file | The fetch-and-parse step. Used as-is (UA changed) by {[papadogiannakis2025_darkside]} |
| 26 ''ads.txt'' snapshots of Alexa Top-100K, Jan 2018–Apr 2019, cleaned seller list, parent-organisation clusters | {[bashir2019_adstxt]} | Live, single archive at ''personalization.ccs.neu.edu/Projects/Adstxt/'' | A historical baseline nobody can re-crawl (Alexa is gone) |
| Crawlers for ''ads.txt'' and ''sellers.json'', misinformation and piracy site lists, WHOIS-redaction keyword list | {[papadogiannakis2025_darkside]} | Live, ''gitlab.com/papamano/welcome-to-the-dark-side'' | The 2023 pipeline; the keyword list saves you a week |
| AdSparency monitoring service | {[papadogiannakis2025_darkside]} | Live, ''adsparency.ics.forth.gr'' | Looking a domain up before you crawl it; the authors call its output "an indication of misuse", not evidence |
| Dark-pooling measurement code | {[vekaria2024_darkpooling]} | Live, MIT, ''github.com/Yash-Vekaria/ad-inventory-fraud-measurement'', last pushed May 2024 | The misrepresentation checks (Tables 2 and 3 of the paper) as code |
| ''sellers.json'' of the largest exchange | Google | Live; about 109 MB; 972,929 sellers, 70.96% confidential. ''realtimebidding.google.com/sellers.json'' redirects (302) to ''storage.googleapis.com/adx-rtb-dictionaries/sellers.json'' — follow it | Stream it; do not ''json.load'' it on a laptop without checking memory first |
| Crawler that stores ''ads.txt'' alongside HTML, cookies and traffic ("Scrape Titan") | {[papadogiannakis2023_whofunds]} | Live, ''gitlab.com/papamano/scrape-titan'' | Collecting the file in the same visit as the page — the only way to join declarations to what the page actually loaded |
| HBDetector Chrome extension | {[pachilakis2019_waterfalls]} | **Dead** — ''github.com/mipach/HBDetector'' returns 404 | Read the paper's Section 3 for the event list and reimplement; the maintained industry equivalent is Prebid's Professor Prebid extension (its older ''header-bidder-expert'' has not been touched since 2022) |
| HARPO source | {[zhang2022_harpo]} | **Dead** — ''github.com/bitzj2015/Harpo-NDSS22'' returns 404 | — |
| Inclusion-chain crawler ("DeepCrawling") | {[bashir2019_adstxt]} | Moved to ''github.com/sajjadium/Crawlium'' | Declarations-vs-transactions designs |
| Alexa Echo ad-targeting code and data | {[iqbal2023_tracking]} | Live, ''privsec-research.github.io/alexaechos'' | A worked bids-as-treatment pipeline on OpenWPM |
| Illegal-streaming crawl, raw data and scripts | {[sheaib2025_unmasking]} | Live, ''doi.org/10.17617/3.STVMDI'' (Edmond) | A small, complete ''ads.txt'' + ''sellers.json'' verification example |

===== Use in Publications =====

All figures below come from the [[Literature:Corpus|publication corpus]] — seven venues, 2010–2026, 5,869 papers with readable full text. The full query log, the report script and its unedited output are on [[provenance:privacy:ads_txt]].

==== How many papers touch it at all ====

None of the three observables is a field in the extraction schema, so reach is measured from the text, whitespace-collapsed. Denominator: **5,869 papers with full text**.

^ Term ^ Papers, ≥ 1 mention ^ Papers, ≥ 5 mentions ^
| ''ads.txt'' | 12 | 8 |
| ''app-ads.txt'' | 3 | 1 |
| ''sellers.json'' | 5 | 4 |
| header bidding | 24 | 7 |
| Prebid | 11 | 6 |
| OpenRTB | 13 | 2 |
| SupplyChain object / ''schain'' | 0 | 0 |
| ''buyers.json'' | 0 | 0 |
| ''ads.cert'' | 2 | 0 |
| **any of the seven core terms, at least once** | **41 (0.7%)** | |
| **core terms summed, five or more times** | **18 (0.3%)** | |
| //context:// real-time bidding / RTB | 73 | 21 |
| //context:// dark pool(ing) | 8 | 4 |

This is a small literature in these venues: 41 papers ever name one of the observables and 18 name one five or more times. The first mention of any of them is in 2017; there is nothing in 2010–2016 because ''ads.txt'' did not exist until June 2017 and nobody in these venues wrote about header bidding or OpenRTB before it did.

==== By venue ====

Denominator: that venue's own papers with full text.

^ Venue ^ Papers ^ ≥ 1 core mention ^ ≥ 5 core mentions ^
| PoPETs | 510 | 12 (2.4%) | 5 (1.0%) |
| TheWebConf | 843 | 13 (1.5%) | 5 (0.6%) |
| IMC | 637 | 9 (1.4%) | 5 (0.8%) |
| CCS | 989 | 3 (0.3%) | 2 (0.2%) |
| NDSS | 701 | 1 (0.1%) | 1 (0.1%) |
| IEEE S&P | 779 | 1 (0.1%) | 0 (0.0%) |
| USENIX Security | 1,410 | 2 (0.1%) | 0 (0.0%) |

**Three venues hold it.** Four of the five ''ads.txt'' / ''sellers.json'' papers are at IMC or TheWebConf; three of the seven papers that use bids as a privacy instrument are at PoPETs. The IEEE S&P row is the one to notice: the venue's single mention is in a cookie-banner paper the run did not read, and the venue's actual ''ads.txt'' paper — {[vekaria2024_darkpooling]} — is not in the corpus because abstract screening rejected it. That is a fact about the funnel, not about IEEE S&P.

==== By year ====

Papers mentioning any core term, per year, against that year's corpus size. **This measures attention, not prevalence.** 2025 is thin at the edges and 2026 is provisional — CCS 2026 and IMC 2026 have not been held and IEEE S&P 2026 and TheWebConf 2026 are incompletely indexed — so the last two rows are not evidence of anything.

^ Year ^ Papers ^ ≥ 1 core mention ^ ≥ 5 ^ ''ads.txt'' ≥ 1 ^ ''sellers.json'' ≥ 1 ^ header bidding ≥ 1 ^
| 2010–2016 | 1,047 | 0 | 0 | 0 | 0 | 0 |
| 2017 | 232 | 1 | 0 | 0 | 0 | 0 |
| 2018 | 254 | 2 | 1 | 0 | 0 | 0 |
| 2019 | 402 | 4 | 2 | 1 | 1 | 2 |
| 2020 | 402 | 5 | 1 | 0 | 0 | 4 |
| 2021 | 380 | 3 | 2 | 2 | 0 | 1 |
| 2022 | 546 | 8 | 3 | 2 | 0 | 6 |
| 2023 | 720 | 4 | 2 | 1 | 1 | 3 |
| 2024 | 701 | 6 | 2 | 2 | 0 | 3 |
| 2025* | 770 | 7 | 4 | 4 | 3 | 4 |
| 2026* | 415 | 1 | 1 | 0 | 0 | 1 |

The ''sellers.json'' column is the one methodological signal in this table: one paper in 2019 (Bashir, naming it as forthcoming), one in 2023, three in 2025. The intersection method is a 2023–2025 development in these venues.

==== The measuring set ====

The 18 papers with five or more core mentions, plus two forced in by hand (the 2017 RTB-price paper and the 2022 publisher-ID paper), were read and labelled. **16 measure** — read one of the files or observe bids — **1 is a related method** and **3 only cite**. Zero were left unlabelled. Of the 21 papers with one to four mentions, **3 were read** during the run and **18 were not**; all 21 are listed on the provenance page.

^ Family ^ Papers ^ Years ^
| Bids as a dependent variable or signal (''hb-instrument'') | 7 | 2020–2025 |
| ''ads.txt'' / ''sellers.json'' read to answer another question (''adstxt-instrument'') | 3 | 2023–2025 |
| ''ads.txt'' / ''sellers.json'' as the measured object (''adstxt-object'') | 2 | 2019, 2025 |
| RTB win-notification prices (''rtb-price'') | 2 | 2017–2018 |
| Header bidding as the measured object (''hb-object'') | 1 | 2019 |
| Prebid presence as a sampling frame (''hb-sampling'') | 1 | 2026 |

Of the 16, **13 have a crawl-configuration record**; the other three are two passive-proxy studies of real users' traffic ({[papadopoulos2017_paying]}, {[papadopoulos2018_cost]}) and one browser extension in real users' browsers ({[zeng2022_factors]}). Among the 13: statefulness **6 stateless, 5 stateful, 1 both, 1 not stated**; consent action **5 no-interaction, 2 accept-all, 1 accept-and-reject, 1 CMP-specific, 1 not applicable, 3 not stated**; interaction depth **4 landing page only, 4 landing plus subpages, 3 single target page, 2 not stated**; headless mode **not stated by 10 of 13**. Population sources named, paper-counted after a short alias fold: **Alexa 8, Tranco 5, MediaBias/FactCheck 3, NextDNS piracy list 2, Prolific 1, BuiltWith 1**, with 30 unmapped source strings printed on the provenance page. **10 of 16 release a public artifact**, 4 mention none, 2 promised one.

==== What the corpus cannot tell you ====

The paper this whole literature cites for dark pooling is not in it, and neither is anything from EuroS&P, where its 2026 follow-up on notifying the ecosystem appeared,((Vekaria, Nithyanand, Shafiq, "Towards Multi-Stakeholder Vulnerability Notifications in the Ad-Tech Supply Chain", IEEE EuroS&P 2026 (Lisbon, 6–10 July 2026), listed on ''eurosp2026.ieee-security.org/program.html''; DOI ''10.1109/EuroSP68448.2026.00061'' (resolves to IEEE Xplore document 11624241); preprint ''arXiv:2406.06958'' (v2, 2 April 2026). Fetched 2026-09-02.)) nor the advertising-economics literature that measures Prebid configurations at panel scale.((Johnson & Neumann, "The advent of privacy-centric digital advertising: Tracing privacy-enhancing technology adoption", working paper dated 21 March 2024, measures Privacy Sandbox adoption from sites' Prebid settings on a panel of roughly sixty thousand sites drawn from a Tranco top-100K list. Working paper, not peer-reviewed; cited here only as an existence proof that the Prebid configuration is used as an observable outside computer science.)) Industry measurement of ''ads.txt'' adoption exists but publishes headline percentages without a stated denominator or method, and none of it is cited on this page. Any count above is a count over **CCS, IMC, NDSS, PoPETs, USENIX Security, TheWebConf and IEEE S&P**, 2010–2026. See [[Literature:Corpus]].

===== What to Report =====

For a reviewer to accept an ad-supply-chain figure, the methodology section needs, in this order:

  - **Which relationship**: declared, confirmed, observed or paid — and for //confirmed//, whether a ''sellers.json'' domain mismatch counted as unconfirmed.
  - **The file population**: how many root domains were requested, how many returned a file, how many files parsed under the specification, how many records were dropped as invalid, and the dates of every snapshot.
  - **The ad-system fold**: the alias list that turned seller domains into organisations, and its residue.
  - **For ''sellers.json''**: which ad systems' files you fetched, whether you recursed into intermediaries, and the share of the relevant entries that were confidential — that share is the ceiling on what you could confirm.
  - **For pooling**: how ownership was established (WHOIS, publisher IDs, hand review) and for what share of pool members it could not be.
  - **For header bidding**: how Prebid sites were detected (global name, events, or both), which API (''getBidResponses'' vs winning vs rendered), the wait after ''load'', the persona and control design, the vantage, the consent action, and the browser's protection state.
  - **What fraction of your candidate sites exposed a client-side auction at all** — two of the seven bids papers report it; and, if you can, which of those auctions were client-only, hybrid or server-side, which no paper in these venues has reported since 2019.

===== Open Questions =====

  * **Has the client-side share of header bidding fallen since 2019?** 17.3% client-only, 34.7% hybrid, 48% server-side is the only measurement, and the paper never says whether its unit is sites or auctions. Every bids-as-instrument paper depends on the answer.
  * **How widely are OWNERDOMAIN and MANAGERDOMAIN declared, and do they match ''sellers.json''?** Two fields, four years old, no measurement. A weekend with the IAB crawler would settle it.
  * **What does ''app-ads.txt'' look like?** Bashir et al. asked for "a separate study" in 2019; none has appeared in these venues.
  * **Do confirmed relationships predict served ads?** {[papadogiannakis2025_before]} ran both an intersection crawl and an ad-clicking crawl on the same sites; no paper in these venues has published the join.
  * **Is dark pooling declining after notification?** The 2026 EuroS&P notification study says notifications to ad networks work; a re-measurement of the 2023 pool census would show whether the ecosystem moved.

===== Methodology and limitations of these figures =====

  * **The regex is not the population.** The candidate set is "five or more mentions of ''ads.txt'', ''app-ads.txt'', ''sellers.json'', header bidding, Prebid, OpenRTB or the SupplyChain object, summed", plus two papers forced in by hand. Three papers with 1–4 mentions happened to be read during the run and turned out to mention the terms only in passing; the other 18 were not read. A paper that measured bids without ever writing "header bidding" or "Prebid" would be missed.
  * **Counts are of papers, never tuples**, and sentinels are reported as sentinels. Every crawl-configuration figure above says how many of the 13 crawling papers did not state the value.
  * **Sixteen is a small set.** The family table's "first / most recent" years are facts about 16 papers, and the 2025–2026 venue-years are provisional. No percentage on this page is a prevalence of anything in the world; the prevalence figures are the papers' own, with the papers' own denominators.
  * **The most-cited paper on the topic is missing from the corpus** for a reason unrelated to relevance (abstract screening), and this page cites it from an arXiv version dated October 2023, not from the IEEE camera-ready.
  * **Quotes** behind the figures were checked against the papers' own full text; the list, the report script ''report_ads_txt.mjs'' and its unedited output, the alias folds and their residue, the external checks and their rejections, and the reviewers' findings are on **[[provenance:privacy:ads_txt]]**.

===== Related Pages =====

  * [[Privacy:Cookie syncing]] — the other half of the ad graph: whether two parties' identifiers met. Cook 2020 and ATOM sit on both pages.
  * [[Privacy:Requests]] and [[Programming:Filter lists]] — EasyList is how the ad-attribution crawls here found the ads — and it blocks the Prebid endpoints, so leaving it on removes the auction the bids papers depend on.
  * [[Privacy:Consent]] and [[Privacy:TCF consent strings]] — the CMP-and-Prebid intersection is the population for a consent-and-bids study.
  * [[Privacy:Privacy Sandbox]] — the Protected Audience auction was the would-be successor to header bidding; it is being withdrawn.
  * [[Design:Website selection]] — Alexa's retirement is why the 2019 reference studies cannot be repeated as designed.
  * [[Design:Existing datasets]] — entity maps (WhoTracksMe, Tracker Radar) for folding seller domains into organisations.
  * [[Practices:Notifying websites]] — the 2026 notification study of dark pooling belongs to that literature.

====== References ======

<bibtex bibliography></bibtex>

~~DISCUSSION~~