A web archive lets you measure a page you cannot visit — because the date has passed, because the site is gone, or because you want everyone else to be able to measure exactly the same bytes. That is a genuinely different instrument from a crawler, and it fails in genuinely different ways. This page is about what an archive can and cannot answer, not about how archiving works.

Researchers have used archives for longitudinal analyses of privacy [3Lerner, Ada; Simpson, Anna Kornfeld; Kohno, Tadayoshi; Roesner, Franziska (2016): "Internet Jones and the Raiders of the lost trackers: An archaeological study of web tracking from 1996 to 2016", in: 25th USENIX Security Symposium (USENIX Security 16). (Link), 9Jha, Nikhil; Trevisan, Martino; Mellia, Marco; Fernandez, Daniel; Irarrazaval, Rodrigo (2024): "Privacy Policies and Consent Management Platforms: Growth and Users' Interactions over Time", arXiv preprint arXiv:2402.18321 ., 10Amos, Ryan; Acar, Gunes; Lucherini, Eli; Kshirsagar, Mihir; Narayanan, Arvind; Mayer, Jonathan (2021): "Privacy Policies over Time: Curation and Analysis of a Million-Document Dataset", in: Proceedings of the Web Conference 2021, pp. 2165–2176. Association for Computing Machinery, New York, NY, USA. (DOI) (Link), 11Dimova, Yana; Acar, Gunes; Olejnik, Lukasz; Joosen, Wouter; Van Goethem, Tom (2021): "The CNAME of the game: Large-scale analysis of DNS-based tracking evasion", Proceedings on Privacy Enhancing Technologies 2021:394–412. (DOI) (Link), 4Singh, Sachin Kumar; Mahmud, Faisal; Ricci, Robert; Siby, Sandra (2026): "The Empire Strikes Back (at Your Privacy): An Archaeology of Tracking on Government Websites", Proceedings on Privacy Enhancing Technologies 2026(2):108-126. (DOI)] and of security features [12Pletinckx, Stijn; Borgolte, Kevin; Fiebig, Tobias (2021): "Out of sight, out of mind: Detecting orphaned web pages at internet-scale", in: Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pp. 21–35. Association for Computing Machinery, New York, NY, USA. (DOI) (Link), 13Roth, Sebastian; Barron, Timothy; Calzavara, Stefano; Nikiforakis, Nick; Stock, Ben (2020): "Complex security policy? A longitudinal analysis of deployed content security policies", in: Proceedings of the 27th Network and Distributed System Security Symposium (NDSS)., 14Stock, Ben; Johns, Martin; Steffens, Marius; Backes, Michael (2017): "How the Web Tangled Itself: Uncovering the History of Client-Side Web (In) Security", in: 26th USENIX Security Symposium (USENIX Security 17), pp. 971-987. USENIX Association, Vancouver, BC. (Link)]. The attraction is reproducibility: a snapshot is static, so a later reader can fetch the same one, and nobody has to write another crawler. The cost is that a snapshot is a partial recording made by someone else's crawler, from their vantage point, at a moment they chose, and almost every pitfall below follows from that one sentence.

Two numbers frame everything that follows. In our corpus of seven security and privacy venues (see Use in Publications), 115 of 5,118 empirical papers (2.2%) name a web archive at all, against 1,236 (24.2%) that ran a live crawl. Of the papers that do, only about a quarter collected nothing of their own — 58.2% ran a live crawl as well. The archive is normally a supplement, not the instrument.

Four papers, in the order that gets a newcomer productive fastest:

1. [2Hantke, Florian; Calzavara, Stefano; Wilhelm, Moritz; Rabitti, Alvise; Stock, Ben (2023): "You Call This Archaeology? Evaluating Web Archives for Reproducible Web Security Measurements", in: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 3168–3182. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] — the systematic study. Compares seven archives, quantifies what goes wrong in the Internet Archive, and ends with a best-practice list. If you read one, read this.
2. [3Lerner, Ada; Simpson, Anna Kornfeld; Kohno, Tadayoshi; Roesner, Franziska (2016): "Internet Jones and the Raiders of the lost trackers: An archaeological study of web tracking from 1996 to 2016", in: 25th USENIX Security Symposium (USENIX Security 16). (Link)] — the founding archive-based measurement in this literature (tracking, 1996–2016), and still the only paper that measured how often replay escapes to the live web.
3. [1Zhu, Jingyuan; Sun, Huanchen; Madhyastha, Harsha V. (2025): "Toward Better Efficiency vs. Fidelity Tradeoffs in Web Archives", in: Proceedings of the ACM Internet Measurement Conference, pp. 1025-1031. (DOI)] — why a static crawl misses half the page, measured in 2025, from the archive operators' side of the problem.
4. [4Singh, Sachin Kumar; Mahmud, Faisal; Ricci, Robert; Siby, Sandra (2026): "The Empire Strikes Back (at Your Privacy): An Archaeology of Tracking on Government Websites", Proceedings on Privacy Enhancing Technologies 2026(2):108-126. (DOI)] — the model recent application: a 2026 longitudinal study that states its snapshot rule, its rate limiting and its handling of classifier anachronism, all of which you can copy.

For the alternative to using an archive at all, [5Hantke, Florian; Snyder, Peter; Haddadi, Hamed; Stock, Ben (2025): "Web Execution Bundles: Reproducible, Accurate, and Archivable Web Measurements", in: Proceedings of the USENIX Security Symposium. (Link)].

Decide this before anything else on this page matters. Archives are good at questions of the form “was this present in the HTML on this date”; they are weak at “what did a user's browser actually do”, because the runtime half of the page was mostly not recorded.

If your phenomenon is in the right-hand column, an archive is either the wrong instrument or a very expensive one — read Recording Your Own Archive instead.

Web archives have historically used static crawlers: fetch HTML, parse it for links to images, stylesheets and scripts, fetch those, recurse. Nothing is executed. On the modern web many resource URLs only exist after a script runs, so they are never requested and never stored.

Zhu et al. [1Zhu, Jingyuan; Sun, Huanchen; Madhyastha, Harsha V. (2025): "Toward Better Efficiency vs. Fidelity Tradeoffs in Web Archives", in: Proceedings of the ACM Internet Measurement Conference, pp. 1025-1031. (DOI)] quantified the gap by crawling the same 10,000 pages both ways and replaying the static copies with pywb. For 54.2% of pages the two loads were equivalent; for the remaining 45.8% the statically crawled copy was missing rendered content, or an interaction that worked on the dynamic copy did not work on the static one. Their explanation for why archives live with this is economic rather than technical — browser-based crawling throughput is about 3× lower than browserless, and archives are run by non-profits.

The consequence for a measurement is not “the page looks a bit worse”. It is that the thing you are counting may be systematically invisible. Hantke et al. [2Hantke, Florian; Calzavara, Stefano; Wilhelm, Moritz; Rabitti, Alvise; Stock, Ben (2023): "You Call This Archaeology? Evaluating Web Archives for Reproducible Web Security Measurements", in: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 3168–3182. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] ran both a static parse of archived HTML and a Playwright-driven replay of the same archived pages, on the 2,026 sites they could compare:

A static reading of the archive understates tracker prevalence by about twenty percentage points in both years, and understates the average tracker count by a factor of 3.5–4. Their static analysis also produced a plausible-looking trend — the share of sites with no tracker rising to about 40% by 2022 — that they attribute to the static bias rather than to the web. That is the failure mode worth fearing: not an obviously broken number, but a clean, publishable, wrong one.

Reading the archive dynamically fixes the bias and costs you the scale. The same paper reports going from one request per site to more than 100, hitting HTTP 429 from the Internet Archive even with a 5-second sleep after each page, and being able to compare only 2,026 sites as a result.

URL rewriting inside a replayed page is not perfect, chiefly because it cannot rewrite URLs that JavaScript builds at runtime. When it fails, the replayed page issues a request to the live web. Lerner et al. measured this on archived snapshots of the Alexa top 500 and found that 16.1% of all requests attempted to escape the archive — 17.2% of unique URLs and 11.5% of domains [3Lerner, Ada; Simpson, Anna Kornfeld; Kohno, Tadayoshi; Roesner, Franziska (2016): "Internet Jones and the Raiders of the lost trackers: An archaeological study of web tracking from 1996 to 2016", in: 25th USENIX Security Symposium (USENIX Security 16). (Link)]. That measurement is from 2016, on 2015-era replay, and nobody has repeated it since; treat it as evidence that escapes matter, not as today's rate.

To put that in proportion: in the same table, robots.txt exclusion accounted for 2.0% of requests and the resource simply not being archived for 1.4%. Escapes were by far the largest single failure mode — eight times robots exclusion. Their crawler blocked those requests, which is the right default: an escaped request measures the present under a past timestamp. If your tooling does not block them, your “2014 measurement” silently contains 2026 responses. Blocking is one rule in whatever drives your browser — abort any request whose URL is not under web.archive.org — and count what you blocked, because that count is a fidelity statistic worth reporting. The same mechanism is a security problem, not only a measurement one — Lerner et al. later showed that an attacker who controls an escape destination can rewrite what a client sees in the archive, and that 73% of Top 500 sites and 80% of Top Million domains had snapshots vulnerable to at least one such attack [6Lerner, Ada; Kohno, Tadayoshi; Roesner, Franziska (2017): "Rewriting History: Changing the Archived Web from the Present", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, pp. 1741-1755. (DOI)].

Coverage is good at the head of a popularity list and degrades fast outside it.

• Hantke et al. served fresh data (within ±6 weeks of the requested date) for about 55% of a stratified random sample of 5,000 Tranco domains — see Which Archive for how the alternatives compare [2Hantke, Florian; Calzavara, Stefano; Wilhelm, Moritz; Rabitti, Alvise; Stock, Ben (2023): "You Call This Archaeology? Evaluating Web Archives for Reproducible Web Security Measurements", in: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 3168–3182. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)].
• On a population that is not a popularity list at all — 758,192 URLs from web censorship lists — Weinberg et al. found snapshots for 423,265 (55.8%), with a median of only 3 snapshots per URL and large gaps, which was too sparse to compute per-topic survival curves [7Weinberg, Zachary; Sharif, Mahmood; Szurdi, Janos; Christin, Nicolas (2017): "Topics of Controversy: An Empirical Analysis of Web Censorship Lists", in: Proceedings on Privacy Enhancing Technologies. (DOI)].
• On 18,900 government websites from 61 countries, Singh et al. found 16,500 had at least one snapshot — 87% of the total, or an average of 88% when averaged per country [4Singh, Sachin Kumar; Mahmud, Faisal; Ricci, Robert; Siby, Sandra (2026): "The Empire Strikes Back (at Your Privacy): An Archaeology of Tracking on Government Websites", Proceedings on Privacy Enhancing Technologies 2026(2):108-126. (DOI)].

So “is it archived” is a property of your specific population, and it is a denominator you have to report. Check it before you design the study, not after.

The Internet Archive crawls predominantly from the US, and it aggregates content from many contributors, so its “view” corresponds to no single real vantage point. Comparing archived responses against live responses collected from Germany, California and Australia, Hantke et al. found that overall 8.0–8.7% of sites showed a syntactic difference in security headers and 7.0–7.7% a semantic one, with the US comparison the closest — as expected [2Hantke, Florian; Calzavara, Stefano; Wilhelm, Moritz; Rabitti, Alvise; Stock, Ben (2023): "You Call This Archaeology? Evaluating Web Archives for Reproducible Web Security Measurements", in: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 3168–3182. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)]. Mature headers were fine (X-Frame-Options 3.6%, CSP 6.7%, HSTS 6.5% from the US vantage); newer ones were not, with COOP differing on about 70% of the sites deploying it, almost entirely because Google properties serve those headers selectively.

Part of that gap is deliberate: sites treat the archive's crawler differently. Of 212 otherwise-unexplained differences, 93 sites were doing User-Agent sniffing and simply returned no security headers to archive.org_bot. After accounting for redirects, stale snapshots and sniffing, 1.26% of sites (81 of 6,412) differed for no reason the authors could establish. That residue is roughly the baseline for the live web too: a purely live study found 127 of 8,174 sites (1.55%) returning inconsistent security headers between repeated identical requests [8Roth, Sebastian; Calzavara, Stefano; Wilhelm, Moritz; Rabitti, Alvise; Stock, Ben (2022): "The Security Lottery: Measuring Client-Side Web Security Inconsistencies", in: Proceedings of the USENIX Security Symposium, pp. 2047-2064. (Link)]. If you plan an archive-based measurement, budget for roughly this residue and test your own population for sniffing by requesting it live with the archive's User-Agent.

The Internet Archive stores responses from many contributors (identified by the x-archive-src header, resolvable through the Metadata API), with a range of status codes, and under a requested URL that may have been reached by different redirect chains at different times. Hantke et al. show that when two temporally adjacent snapshots disagree about a site's security posture, the contributor is the single strongest explanatory feature, ahead of the status code; the NULL contributor (24% of responses) disagrees with the others most often. Between 1.7% (HSTS) and 34.6% (COEP) of their snapshot neighbourhoods contained observations with different security implications, and the majority of those differences came from the header simply being absent in some snapshots.

The practices below are Hantke et al.'s best-practice list [2Hantke, Florian; Calzavara, Stefano; Wilhelm, Moritz; Rabitti, Alvise; Stock, Ben (2023): "You Call This Archaeology? Evaluating Web Archives for Reproducible Web Security Measurements", in: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 3168–3182. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] plus what the newer archive-based papers actually did. They are cheap; skipping them is the difference between a measurement and an artefact.

1. Use neighbourhoods, not single snapshots. For each URL and date, fetch the snapshots within a few days and aggregate them, rather than trusting the temporally closest one. Most spurious differences are “the header was missing in one capture”. Report bounds if the aggregation matters: Hantke et al. detect 8% fewer third-party script hosts using the closest snapshot instead of the union, and 12% fewer using the intersection instead of the closest.
2. Filter the contributor. Dropping the NULL contributor removes the noisiest source and costs less than 3% of neighbourhoods.
3. Restrict to 2XX, unless error pages are your subject.
4. Fix the final origin across the whole time series. The archive files responses for different post-redirect origins under the same requested URL, and comparing them across years is an apples-to-oranges comparison.
5. Wait for stabilisation. The archive keeps ingesting data for a requested date for about four days — of the 11,364 domains for which they found data within ±1 day, 1,332 got a closer snapshot on day 1, 1,720 on day 2 and 1,760 on day 3 — after which content is effectively stable. Measure after that, and share the end URLs (with resolved timestamps), not the request URLs, so a reader lands on the same bytes. Note that this is not a permanence guarantee: exclusions can be applied retroactively at a site owner's request, and Lerner et al. measured robots-based blocking at 2.0% of replay requests already. If a result is load-bearing, keep your own WARCs of the snapshots it rests on.
6. De-rewrite URLs before matching. Replayed HTML contains https://web.archive.org/web/<timestamp>/https://a.com/script.js. Every filter-list, eTLD+1 or third-party test must run against the recovered original URL. Singh et al. name this as an explicit step in their pipeline [4Singh, Sachin Kumar; Mahmud, Faisal; Ricci, Robert; Siby, Sandra (2026): "The Empire Strikes Back (at Your Privacy): An Archaeology of Tracking on Government Websites", Proceedings on Privacy Enhancing Technologies 2026(2):108-126. (DOI)].
7. Be a polite client. There is no published read-side rate limit (the widely cited “15 per minute” is the limit on Save Page Now submissions, not on reads¹⁾), but 429s are real. Singh et al. used 3–5 seconds between sequential downloads and no concurrency [4Singh, Sachin Kumar; Mahmud, Faisal; Ricci, Robert; Siby, Sandra (2026): "The Empire Strikes Back (at Your Privacy): An Archaeology of Tracking on Government Websites", Proceedings on Privacy Enhancing Technologies 2026(2):108-126. (DOI)]; the maintained wayback Python library defaults to 0.4 CDX calls per second and 8 Memento calls per second²⁾. Budget 2–5× the wall-clock of a live crawl [2Hantke, Florian; Calzavara, Stefano; Wilhelm, Moritz; Rabitti, Alvise; Stock, Ben (2023): "You Call This Archaeology? Evaluating Web Archives for Reproducible Web Security Measurements", in: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 3168–3182. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)].

The archive-specific mistake that no live crawl can make: applying today's classifier to yesterday's page. A filter list, a tracker database, a vendor list or an LLM prompt written in 2026 encodes 2026's ecosystem. Run it over a 2006 snapshot and you are measuring the intersection of the past web with the present taxonomy.

There is no clean fix, but there is a defensible practice, and Singh et al. [4Singh, Sachin Kumar; Mahmud, Faisal; Ricci, Robert; Siby, Sandra (2026): "The Empire Strikes Back (at Your Privacy): An Archaeology of Tracking on Government Websites", Proceedings on Privacy Enhancing Technologies 2026(2):108-126. (DOI)] show it. They label with one fixed list version (EasyList and EasyPrivacy, June 2025) for all years — so that list drift cannot masquerade as a trend — and then validate that choice against historical list versions recovered from GitHub and from the archive itself, at three anchor years. The validation is the interesting part: on a random sample of 10,000 2006 URLs, the 2006 EasyList flagged 67 URLs and the June 2025 list flagged 50, with only 6 in common. Both lists were then manually adjudicated. Whatever you choose, hold the classifier fixed across time, say which version, and show what the alternative would have done.

Hantke et al. [2Hantke, Florian; Calzavara, Stefano; Wilhelm, Moritz; Rabitti, Alvise; Stock, Ben (2023): "You Call This Archaeology? Evaluating Web Archives for Reproducible Web Security Measurements", in: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 3168–3182. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] remain the systematic comparison: 64 API endpoints from 38 Memento-supporting archives, plus Common Crawl. Their results, which are what you should assume until someone repeats the study:

• They collected 64 API endpoints from 38 archives; only 13 endpoints produced even one hit for the Tranco top 5,000, and the four weakest of those (library and museum collections) covered at most 8% of it. The paper attributes the failures to endpoints that are simply outdated.
• The Internet Archive dominated on every axis: most domains covered, freshest data, and the only archive that stays useful outside the top 5,000. On a stratified random sample of Tranco 1M it served fresh data for ~55% of domains against 7% for the best alternative.
• Combining the Internet Archive with all the others added a negligible number of fresh hits. Using it as a single source is a defensible choice, and the multi-archive fallback that older digital-library work recommends does not pay for itself in this setting.
• Several archives had stopped crawling fresh data by 2022 (the Library of Congress endpoint returned zero fresh hits), so an archive's usefulness is a function of your target period, not just its name.

What to check before committing to one:

• Crawl location. Both the Wayback Machine and HTTP Archive crawl from the US. If your question is regional — EU consent banners, geo-blocking, localised advertising — an archive is answering it from the wrong place. See crawling_location.
• Coverage of your population, measured, not assumed (see above). A stratified probe of a few thousand URLs costs an afternoon.
• Temporal granularity. Do snapshots exist near the dates you need, and how many per URL? A median of 3 snapshots over ten years supports a “did it ever” claim and not much else.
• Whether replay is browser-driven. For anything runtime-dependent, this decides whether the archive can answer at all.

The Wayback Machine passed one trillion archived web pages in October 2025⁴⁾ and covers 1996 onwards. Snapshot frequency is driven by the site: popular domains are captured close to daily, tail domains sporadically or never. Anyone can submit a page through Save Page Now, which means part of the archive is user-nominated rather than crawler-driven.

Three access paths, in increasing order of usefulness for research:

• Availability API — https://archive.org/wayback/available?url=<url>&timestamp=<YYYYMMDDhhmmss> returns the closest snapshot as JSON. Fine for “is there anything near this date”, useless for enumeration.
• CDX API — https://web.archive.org/cdx/search/cdx?url=<url>&output=json lists every capture with timestamp, MIME type, status code and content digest. This is the one to build on: collapse=digest suppresses unchanged captures, from/to bound the range, and the digest column lets you detect that nothing changed without refetching. Expect intermittent 504s under load — retry with backoff rather than treating them as absence (one of our three probes on 2026-08-13 returned a 504 that succeeded on retry).
• Replay — https://web.archive.org/web/<timestamp>/<url>, optionally with the id_ modifier for the unrewritten original response. Point a browser here for dynamic analysis, accepting the request amplification and 429 risk described above.

Do not write your own client for the first two. ''wayback'' (Python, maintained by EDGI) wraps the CDX and Memento endpoints, applies conservative rate limits by default, and raises a typed error on 429 instead of silently returning nothing.

It is also possible to request a snapshot of a single resource rather than a whole page — a given fingerprint.js build, say — though a subresource typically has far fewer captures than the pages that include it.

HTTP Archive is a different instrument that is often confused with a web archive: it does not let you replay a page, it publishes the HAR files of its own monthly browser-driven crawl into public BigQuery tables. Because the crawl is browser-driven, it does not have the static-capture blindness described above — requests, response headers and third-party resources are all recorded. What it does not give you is an arbitrary date: you get its monthly grid, from its US vantage point, on its URL list.

Pages are loaded with WebPageTest driving Chrome, and the full HAR is stored; the wptid column links each row back to its WebPageTest result. The URL list has been based on the Chrome User Experience Report (CrUX) since 1 July 2018. Data before that exists back to 2010 but covers far fewer sites, so comparisons across that boundary are not like-for-like. The current sample size per month is published at state-of-the-web § Sample Size — check it rather than quoting a number from a paper.

Access:

Create a Google Cloud project, then in the BigQuery Explorer sidebar use + Add → Star a project by name and star httparchive. The dataset is public and needs no further permission; har.fyi's getting-started guide has the click-through if you need it.

The httparchive.crawl dataset is partitioned by date and split into pages (one row per crawled page) and requests (one row per HTTP request, with headers and payload):

SELECT *
FROM `httparchive.crawl.requests`
WHERE DATE = '2025-02-01'
  AND client = 'desktop'

Two properties make it worth the cost. It crawls subpages as well as landing pages — filter with is_root_page — which is unusual and directly addresses the landing-page bias discussed in website_selection. And it carries the CrUX popularity rank magnitude, which is a bucket and not a rank: it takes the values 1,000 / 5,000 / 10,000 / 50,000 / 100,000 and up, and the buckets are cumulative but exclusive of each other. rank = 1000 selects exactly the top 1,000 origins, but rank = 10000 selects ranks 1,001–10,000 — for the top ten thousand you need rank ⇐ 10000⁵⁾. The query below is therefore right for the top 1,000 and would be a trap at any other value:

SELECT *
FROM `httparchive.crawl.pages`
WHERE DATE = '2025-02-01'
  AND client = 'desktop'
  AND is_root_page
  AND rank = 1000

For a fixed set of sites tracked over time, downloading the HAR files from Google Cloud Storage using the wptid from the pages table is free and avoids the query cost entirely: see the download guide. Working with the resulting files is covered in traffic_files.

Common Crawl publishes a monthly crawl as WARC/WAT/WET files with a queryable CDX index (https://index.commoncrawl.org/collinfo.json lists the available crawls; the July 2026 index is CC-MAIN-2026-30). It is not a replay archive — there is no efficient way to ask “what did this URL look like on that date” — but it is enormous, free, and served from S3, so it is the practical choice for breadth: as a sampling frame beyond the reach of popularity lists, for link-graph work, or for corpus construction. Hantke et al. used it as a second archival source and found it added little to the Internet Archive for date-targeted measurement; Roth et al. found its CSP data largely matched the Internet Archive's [2Hantke, Florian; Calzavara, Stefano; Wilhelm, Moritz; Rabitti, Alvise; Stock, Ben (2023): "You Call This Archaeology? Evaluating Web Archives for Reproducible Web Security Measurements", in: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 3168–3182. Association for Computing Machinery, New York, NY, USA. (DOI) (Link), 13Roth, Sebastian; Barron, Timothy; Calzavara, Stefano; Nikiforakis, Nick; Stock, Ben (2020): "Complex security policy? A longitudinal analysis of deployed content security policies", in: Proceedings of the 27th Network and Distributed System Security Symposium (NDSS).]. Note that in the recent literature Common Crawl also appears as an LLM training corpus (CC-News and derivatives), which is a completely different use — do not conflate the two when reading related work.

Worth knowing about, mostly for jurisdiction-specific work:

• Arquivo.pt, the Portuguese Web Archive, operated by FCCN within Portugal's FCT — not a public library service. It holds 23,048 million files across 73 million websites (1.6 PB compressed) as of January 2026⁶⁾ and offers Memento, CDX and full-text search APIs — its TimeMap endpoint still answers, which is how the Memento protocol survives its aggregator. In Hantke et al.'s probe it placed fourth on raw hits (3,412 against the Internet Archive's 4,061) but far behind on freshness (338 fresh hits against 3,713).
• Archive-It, the Internet Archive's subscription service for institutional collections. Narrower than the main archive, but Hantke et al. found high freshness for what it does cover — useful if your population overlaps a curated collection.
• National and institutional archives — the Icelandic, Croatian and Czech web archives, the Library of Congress, several university libraries. Now that Time Travel is gone, the two surviving inventories are MemGator's ''archives.json'' — 20 endpoints of which 7 are flagged inactive, several of them named “(DEFUNCT)” or “(MOVED TO A-IT)” in the file itself — and Wikipedia's list, which is broader and unverified. In Hantke et al.'s probe, most of these archives returned few or no fresh hits and several appear to have stopped crawling; treat any of them as a possible source for a national population and verify coverage yourself before designing around it.

If your goal is reproducibility rather than history, the better move is often to archive your own crawl rather than mine somebody else's. This gives you a browser-driven capture with no coverage gaps for the pages you crawled, no anachronism between crawler and replayer, and escapes you can actually block because you control the replayer — at the cost of not being able to reach backwards in time.

• browsertrix-crawler (Webrecorder) drives a real browser and writes WARC/WACZ. Actively developed (v1.14.2, August 2026); the hosted Browsertrix service is the same engine.
• pywb replays WARC collections locally, including the “fuzzy matching” of URLs that archives rely on when a CDN varies resource URLs between loads. This is what Zhu et al. replay through [1Zhu, Jingyuan; Sun, Huanchen; Madhyastha, Harsha V. (2025): "Toward Better Efficiency vs. Fidelity Tradeoffs in Web Archives", in: Proceedings of the ACM Internet Measurement Conference, pp. 1025-1031. (DOI)]. Latest release v2.9.1 (October 2025).
• WARC itself is ISO 28500; the current specification is WARC 1.1, maintained by the IIPC at iipc.github.io/warc-specifications. warcio is the usual Python read/write library.
• WebREC and the .web format [5Hantke, Florian; Snyder, Peter; Haddadi, Hamed; Stock, Ben (2025): "Web Execution Bundles: Reproducible, Accurate, and Archivable Web Measurements", in: Proceedings of the USENIX Security Symposium. (Link)] are the research-specific version of this idea: a measurement tool plus an archival format designed so that a measurement (not just a page) can be re-run later without recrawling. The authors report that 70% of the papers surveyed in a 2024 crawling SoK could have been run with WebREC unmodified, and 48% could have been run against existing .web archives with no new crawling at all. Code: cispa/webrec.

The trade-off is stated plainly: an archive you recorded yourself answers “can someone else reproduce my result”, and a public archive answers “what was there before I started looking”. Only the second is time travel, and it is the one with all the caveats.

Figures below are from a structured extraction over 5,859 papers from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026. Every claim is a claim about those seven venues.

Of 5,342 papers for which the extraction recorded any data-provenance tuple, the modes break down as follows. A paper can use several, so the column does not sum to 100%, and not-stated is a value rather than an answer:

That enum undercounts, because a paper whose whole dataset comes from the Wayback Machine is often coded as existing-dataset instead — Singh et al. [4Singh, Sachin Kumar; Mahmud, Faisal; Ricci, Robert; Siby, Sandra (2026): "The Empire Strikes Back (at Your Privacy): An Archaeology of Tracking on Government Websites", Proceedings on Privacy Enhancing Technologies 2026(2):108-126. (DOI)] is exactly that case. Sweeping five free-text fields for archive names and folding the spellings (see Methodology and limitations of these figures) gives a fuller count: 122 papers name a web archive, 1.8× the enum, of which 85 name a replay archive — one you can point a client at (Internet Archive, Arquivo.pt, Archive-It, Memento, a national archive, or an unnamed “web archive”) — and the rest only Common Crawl or HTTP Archive.

Seven of those 122 are not empirical papers (they are mostly LLM work using Common Crawl as a training corpus), so the share of empirical work is 115 of 5,118, or 2.2%, against 24.2% that ran a live crawl. And the archive is almost never the whole study:

The last two rows differ by a factor of four for the same reason the headline count does: a paper built entirely on the Wayback Machine is often coded existing-dataset rather than web-archive. 31 is the number to use — 8 is the enum's floor, and 23 papers sit in the gap, including [6Lerner, Ada; Kohno, Tadayoshi; Roesner, Franziska (2017): "Rewriting History: Changing the Archived Web from the Present", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, pp. 1741-1755. (DOI)] and [4Singh, Sachin Kumar; Mahmud, Faisal; Ricci, Robert; Siby, Sandra (2026): "The Empire Strikes Back (at Your Privacy): An Archaeology of Tracking on Government Websites", Proceedings on Privacy Enhancing Technologies 2026(2):108-126. (DOI)].

Archive use rose once, in the mid-2010s, and has been flat since. Denominators are empirical papers in the same bucket:

The last bucket is provisional by construction: CCS 2026 and IMC 2026 have not been held, and IEEE S&P 2026 and WWW 2026 are only partly indexed. So the honest reading is: the method arrived with Lerner et al.'s 2016 archaeology of tracking and Stock et al.'s 2017 history of client-side security, settled at two to three percent of empirical work, and has neither been superseded nor spread. What has changed since 2023 is that the field now knows how badly a naive archive measurement can go wrong: the newest papers in this corpus state their mitigations explicitly where earlier ones did not ([4Singh, Sachin Kumar; Mahmud, Faisal; Ricci, Robert; Siby, Sandra (2026): "The Empire Strikes Back (at Your Privacy): An Archaeology of Tracking on Government Websites", Proceedings on Privacy Enhancing Technologies 2026(2):108-126. (DOI)] is the clearest example). We have not measured that as a trend — the reporting figures below are not time-sliced — so read it as an impression from reading the papers, not a result.

By venue, over empirical papers: PETS 3.6%, IMC 3.4%, NDSS 2.7%, WWW 2.4%, USENIX 1.9%, CCS 1.7%, IEEE S&P 0.9%.

Paper counts, folded across 27 spellings of the Internet Archive alone, of the 122 papers naming an archive:

Two more families are counted by the report script and deliberately excluded from the 122, so they have no share of it: self-hosted archiving software (Heritrix, pywb, Webrecorder, warcprox — 6 papers, of which 2 also name a public archive), and CC-News used as an NLP training corpus (3 papers, of which 2 also name a public archive). A paper replaying its own WARCs is not using a public archive, and a paper fine-tuning on CC-News is not doing archival measurement at all.

The concentration is the finding, and it matches Hantke et al.'s experimental result: for date-targeted work, there is effectively one archive. It also means the field's archive-based results share one vantage point, one crawler and one set of coverage biases — a systemic dependency nobody has stress-tested.

Of the 94 papers whose archive use produced a datable provenance tuple:

Archive papers date their data better than live-crawl papers do, which makes sense: the date is the point. But two thirds do not say how many snapshots they took, and nearly two thirds do not say how they chose one when several existed — and after everything above, snapshot selection is precisely the decision that determines the result. This is the reporting gap specific to archive work.

• How they were produced. One structured record per paper was extracted from full text; each tuple carries a verbatim evidence quote and its section, so any figure here traces to a sentence. temporal.mode is an enum and is stable; the archive names are free text and are not.
• Folding. Archive names agree run-to-run on roughly a fifth of exact strings, so they were folded into families before counting — the Internet Archive alone appears under 27 distinct spellings, including its CDX, Availability and Metadata APIs and three client libraries. The fold rules, the excluded names and the full residue are on the provenance page. Counts are of papers, never tuples.
• The enum undercounts and the name sweep overcounts. temporal.mode = web-archive misses papers whose archive use was coded as existing-dataset; the name sweep counts a paper that merely mentions the Wayback Machine alongside one that is built on it. The truth is between 68 and 122; both are given rather than one being presented as the answer.
• Silence is not absence. “Does not state how many snapshots” means the paper did not say so. These are reporting figures.
• Venue coverage. Seven venues only. Digital-library and web-science venues (JCDL, TPDL, WWW's archiving tracks elsewhere) carry much of the archiving literature and are not in this corpus, so the numbers describe how security and privacy measurement uses archives, not the state of web archiving research.
• 2025 and 2026 are incomplete for reasons of calendar and indexing; the provisional bucket is labelled wherever it appears.
• Every query behind this section, its report script and its unedited output are on archives; corpus-level caveats are on corpus.

An archive-based methods paragraph a reader can act on states, in this order:

1. Which archive, by name and access path (CDX, Memento, replay, BigQuery), not “the web archive”.
2. Coverage of your population: how many of the URLs you wanted had any snapshot, and how many had one near each date you needed. This is a denominator, and it is usually not the same as your sample size.
3. Snapshot selection rule: closest-to-date, first-of-year, random-within-year, neighbourhood union or intersection — and how many snapshots per URL you ended up with.
4. Static or dynamic: did you parse stored HTML, or replay in a browser? Everything a reader can conclude about third parties, cookies and scripts hinges on this one word.
5. Escape handling: were live-web requests blocked, and how many were there?
6. Filtering: contributor, status code, final origin, freshness window.
7. Classifier versioning: which filter-list/database version was applied to which years, and why (see The anachronism trap).
8. The resolved snapshot URLs, as an artifact. They are short, they are stable after the stabilisation window, and they make everything above checkable.

• website_selection and sampling — an archive is a sampling frame with its own coverage bias; what is in it is not what you asked for.
• crawling_location — both major archives crawl from the US, which is the wrong vantage point for anything regional.
• crawler — the live-crawl alternative, and the tooling you would replay an archive with.
• traffic_files — HAR and WARC handling, which is what an archive hands you.
• crux — the URL list behind HTTP Archive's crawl.
• stateful_stateless — archives are stateless by construction, which rules out a class of measurement.

[1]

Zhu, Jingyuan; Sun, Huanchen; Madhyastha, Harsha V. (2025): "Toward Better Efficiency vs. Fidelity Tradeoffs in Web Archives", in: Proceedings of the ACM Internet Measurement Conference, pp. 1025-1031. (DOI)

[2]

Hantke, Florian; Calzavara, Stefano; Wilhelm, Moritz; Rabitti, Alvise; Stock, Ben (2023): "You Call This Archaeology? Evaluating Web Archives for Reproducible Web Security Measurements", in: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 3168–3182. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)

[3]

Lerner, Ada; Simpson, Anna Kornfeld; Kohno, Tadayoshi; Roesner, Franziska (2016): "Internet Jones and the Raiders of the lost trackers: An archaeological study of web tracking from 1996 to 2016", in: 25th USENIX Security Symposium (USENIX Security 16). (Link)

[4]

Singh, Sachin Kumar; Mahmud, Faisal; Ricci, Robert; Siby, Sandra (2026): "The Empire Strikes Back (at Your Privacy): An Archaeology of Tracking on Government Websites", Proceedings on Privacy Enhancing Technologies 2026(2):108-126. (DOI)

[5]

Hantke, Florian; Snyder, Peter; Haddadi, Hamed; Stock, Ben (2025): "Web Execution Bundles: Reproducible, Accurate, and Archivable Web Measurements", in: Proceedings of the USENIX Security Symposium. (Link)

[6]

Lerner, Ada; Kohno, Tadayoshi; Roesner, Franziska (2017): "Rewriting History: Changing the Archived Web from the Present", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, pp. 1741-1755. (DOI)

[7]

Weinberg, Zachary; Sharif, Mahmood; Szurdi, Janos; Christin, Nicolas (2017): "Topics of Controversy: An Empirical Analysis of Web Censorship Lists", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[8]

Roth, Sebastian; Calzavara, Stefano; Wilhelm, Moritz; Rabitti, Alvise; Stock, Ben (2022): "The Security Lottery: Measuring Client-Side Web Security Inconsistencies", in: Proceedings of the USENIX Security Symposium, pp. 2047-2064. (Link)

[9]

Jha, Nikhil; Trevisan, Martino; Mellia, Marco; Fernandez, Daniel; Irarrazaval, Rodrigo (2024): "Privacy Policies and Consent Management Platforms: Growth and Users' Interactions over Time", arXiv preprint arXiv:2402.18321 .

[10]

Amos, Ryan; Acar, Gunes; Lucherini, Eli; Kshirsagar, Mihir; Narayanan, Arvind; Mayer, Jonathan (2021): "Privacy Policies over Time: Curation and Analysis of a Million-Document Dataset", in: Proceedings of the Web Conference 2021, pp. 2165–2176. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)

[11]

Dimova, Yana; Acar, Gunes; Olejnik, Lukasz; Joosen, Wouter; Van Goethem, Tom (2021): "The CNAME of the game: Large-scale analysis of DNS-based tracking evasion", Proceedings on Privacy Enhancing Technologies 2021:394–412. (DOI) (Link)

[12]

Pletinckx, Stijn; Borgolte, Kevin; Fiebig, Tobias (2021): "Out of sight, out of mind: Detecting orphaned web pages at internet-scale", in: Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pp. 21–35. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)

[13]

Roth, Sebastian; Barron, Timothy; Calzavara, Stefano; Nikiforakis, Nick; Stock, Ben (2020): "Complex security policy? A longitudinal analysis of deployed content security policies", in: Proceedings of the 27th Network and Distributed System Security Symposium (NDSS).

[14]

Stock, Ben; Johns, Martin; Steffens, Marius; Backes, Michael (2017): "How the Web Tangled Itself: Uncovering the History of Client-Side Web (In) Security", in: 26th USENIX Security Symposium (USENIX Security 17), pp. 971-987. USENIX Association, Vancouver, BC. (Link)