| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| design:archives [2025/03/18 14:07] – archives | design:archives [2026/08/21 14:50] (current) – Boxes: <wrap> renders a span, use uppercase <WRAP>; one box per list, not per bullet karel.kubicek.claude |
|---|
| ====== Use of web archives for research ====== | ====== Use of web archives for research ====== |
| |
| | A web archive lets you measure a page you cannot visit — because the date has passed, because the site is gone, or because you want everyone else to be able to measure exactly the same bytes. That is a genuinely different instrument from a crawler, and it fails in genuinely different ways. This page is about **what an archive can and cannot answer**, not about how archiving works. |
| |
| <wrap todo>This page contains only notes</wrap> | Researchers have used archives for longitudinal analyses of privacy {[lerner2016internet,jha2024privacy,amos2021privacy,dimova2021cname,singh2026_empire]} and of security features {[pletinckx2021out,roth2020complex,stock2017web]}. The attraction is reproducibility: a snapshot is static, so a later reader can fetch the same one, and nobody has to write another crawler. The cost is that a snapshot is a **partial recording made by someone else's crawler, from their vantage point, at a moment they chose**, and almost every pitfall below follows from that one sentence. |
| |
| * https://en.wikipedia.org/wiki/List_of_Web_archiving_initiatives | Two numbers frame everything that follows. In our corpus of seven security and privacy venues (see [[#Use in Publications]]), **115 of 5,118 empirical papers (2.2%) name a web archive at all**, against 1,236 (24.2%) that ran a live crawl. Of the papers that do, **only about a quarter collected nothing of their own** — 58.2% ran a live crawl as well. The archive is normally a supplement, not the instrument. |
| * [[https://publications.cispa.de/articles/conference_contribution/You_Call_This_Archaeology_Evaluating_Web_Archives_for_Reproducible_Web_Security_Measurements/24614574?file=43248819|You Call This Archaeology? Evaluating Web Archives for Reproducible Web Security Measurements]] - evaluation of archives | |
| |
| | <WRAP important> |
| | **The one thing to take away.** The Wayback Machine mostly does not run a browser. It stores what a **static** crawler fetched: the HTML, and the subresources whose URLs appear in that HTML. Resources requested only after JavaScript executes are frequently absent, and those are exactly the third-party requests, trackers and dynamically injected scripts that privacy measurement is about. Zhu et al. estimate that **only around 5% of the archived pages available on the Wayback Machine were crawled with a browser** {[zhu2025_toward]}, and Hantke et al. found that a static read of archived pages sees trackers on **73.9% of sites where a browser-driven replay of the same archived pages sees 95.3%** {[hantke2023you]}. An archive is a good instrument for headers, HTML and policy text; it is a weak one for anything that only exists at runtime. |
| | </WRAP> |
| |
| Papers using archives | ===== What to Read First ===== |
| |
| * Internet Archive: https://adalerner.com/internetjones-usenix2016.pdf | Four papers, in the order that gets a newcomer productive fastest: |
| * Internet Archive: https://privacypolicies.cs.princeton.edu/ | |
| * HTTPArchive: https://arxiv.org/pdf/2402.18321 | |
| |
| | - {[hantke2023you]} — the systematic study. Compares seven archives, quantifies what goes wrong in the Internet Archive, and ends with a best-practice list. If you read one, read this. |
| | - {[lerner2016internet]} — the founding archive-based measurement in this literature (tracking, 1996–2016), and still the only paper that measured how often replay escapes to the live web. |
| | - {[zhu2025_toward]} — why a static crawl misses half the page, measured in 2025, from the archive operators' side of the problem. |
| | - {[singh2026_empire]} — the model recent application: a 2026 longitudinal study that states its snapshot rule, its rate limiting and its handling of classifier anachronism, all of which you can copy. |
| |
| | For the alternative to using an archive at all, {[hantke2025web]}. |
| |
| Various researchers have utilized web archives for longitudinal analyses of privacy {[lerner2016internet,jha2024privacy,amos2021privacy,dimova2021cname]} and security features {[pletinckx2021out,roth2020complex,stock2017web]}. | ===== Can an Archive Answer Your Question? ===== |
| Using archives for research have several advantages. Their use helps towards solving the issue of reproducibility of research, given that a static snapshot is used for each page visit. It also eliminates the need to write project-specific crawlers. However, there are a number of pitfalls associated with the use of archives. Additionally, there are numerous archives out there, and there are a number of things to consider before selecting one. | |
| |
| | Decide this before anything else on this page matters. Archives are good at questions of the form "was this present in the HTML on this date"; they are weak at "what did a user's browser actually do", because the runtime half of the page was mostly not recorded. |
| |
| ====== Limitations ====== | ^ Usually answerable from an archive ^ Usually not, or only with browser-driven replay at small scale ^ |
| There are some general limitations related to the use of web archives, studied by prior work {[hantke2023you,hantke2025web,lerner2016internet]}. For instance, Hantke et al. {[hantke2023you]} find that replayed web pages can introduce errors because of missing HTTP headers. The reason for this type of inconsistencies is that the behavior of the web page at the time of archiving might differ from the behavior of the same page at replaying time. \\ | | HTTP response headers (CSP, HSTS, XFO, Referrer-Policy) | Cookies set by JavaScript | |
| A number of things can go wrong during the archiving process because of the dynamic nature of the web. For example, URLs might not be properly archived, which leads to missing resources when replaying the web page {[lerner2016internet]}. This problem then cascades to additional resources which are dependent on the missing resource. | | HTML structure, forms, meta tags | Dynamically injected third parties and trackers | |
| Typically, a modern browser is used to replay an archived web page. However, this decision will have an influence on the results, as the behavior of the web page will differ when using an older version of the browser. \\ | | Privacy-policy, terms and consent text | Fingerprinting API calls | |
| Additionally, at the time of archiving, crawlers typically read and store web resources but do not execute them. This means that any dynamic resources might not be archived, making it difficult to interact with archived web pages (e.g., to fill in a form, interact with a cookie banner...). \\ | | Script //inclusion// URLs present in the HTML | Consent-banner interaction and post-consent behaviour | |
| | | Link graphs, redirects, page existence over time | Request timing, ordering and performance | |
| | | Whether a site or page existed at all | Anything requiring login or state | |
| |
| All of these limitations impact our results and should be kept in mind when opting for using archives instead of crawling live websites. In the future, the community might evolve towards better standards for archiving websites which can mitigate at least some of the mentioned limitations {[hantke2025web]}. | If your phenomenon is in the right-hand column, an archive is either the wrong instrument or a very expensive one — read [[#Recording Your Own Archive]] instead. |
| |
| | ===== What an Archive Preserves, and What It Does Not ===== |
| |
| ====== What to consider when selecting an archive: ====== | ==== Static capture: the missing half of the page ==== |
| * crawl location: Both the Wayback Machine and HTTPArchive run their crawls with an IP address from the US. This has an impact on results if you want to examine properties that are related to a certain region or area (e.g. legislation in the EU). | |
| * the completeness of the data: Some archives crawl popular websites regularly and are more suited for large-scale measurements, such as the Wayback machine {[hantke2023you]}. | |
| * availability: Depending on which websites you want to visit and how granular/coarse you need your analysis to be, you might need to check whether any snapshots are available around a specific date or time period. | |
| |
| | Web archives have historically used //static// crawlers: fetch HTML, parse it for links to images, stylesheets and scripts, fetch those, recurse. Nothing is executed. On the modern web many resource URLs only exist after a script runs, so they are never requested and never stored. |
| |
| | Zhu et al. {[zhu2025_toward]} quantified the gap by crawling the same 10,000 pages both ways and replaying the static copies with ''pywb''. For **54.2%** of pages the two loads were equivalent; for the remaining **45.8%** the statically crawled copy was missing rendered content, or an interaction that worked on the dynamic copy did not work on the static one. Their explanation for why archives live with this is economic rather than technical — browser-based crawling throughput is about **3× lower** than browserless, and archives are run by non-profits. |
| |
| ====== So which archive should I choose? ====== | The consequence for a measurement is not "the page looks a bit worse". It is that **the thing you are counting may be systematically invisible**. Hantke et al. {[hantke2023you]} ran both a static parse of archived HTML and a Playwright-driven replay of the same archived pages, on the 2,026 sites they could compare: |
| The most widely known and used archives for research are The Web Archive's Wayback Machine and HTTP Archive. Hantke et. al.{[hantke2023you]} performed a comparative study on 7 archives and their suitability for web security measurement studies. One of their conclusions is that the Wayback machine is the best option in terms of completeness and freshness of the data regarding popular websites. There are other options for smaller archives hat might be less complete. For instance, Portugal's public library service [[https://arquivo.pt/|(Arquivo.pt)]] includes archives for a number of websites in the EU. Also consider combining multiple archives for better completeness and/or availability. | |
| |
| | ^ Internet Archive snapshots, 2,026 sites ^ 2016 static ^ 2016 dynamic ^ 2022 static ^ 2022 dynamic ^ |
| | | Avg. number of trackers | 1.95 | 6.80 | 1.71 | 7.23 | |
| | | Avg. number of remote inclusions | 3.41 | 6.63 | 3.51 | 7.04 | |
| | | Unique trackers | 465 | 851 | 405 | 899 | |
| | | Websites with trackers | 73.89% | 95.31% | 73.10% | 91.56% | |
| |
| ===== HTTPArchive ===== | A static reading of the archive understates tracker prevalence by about twenty percentage points in both years, and understates the average tracker count by a factor of 3.5–4. Their static analysis also produced a plausible-looking //trend// — the share of sites with no tracker rising to about 40% by 2022 — that they attribute to the static bias rather than to the web. That is the failure mode worth fearing: not an obviously broken number, but a clean, publishable, wrong one. |
| [[https://httparchive.org/|HTTPArchive]] is an open-source project that keeps track of how the web is built. The dataset includes monthly crawls for millions of websites for both mobile and web (Chrome) clients. The crawls are based on the [[https://developer.chrome.com/docs/crux|Chrome User Experience Report (CrUX)]] since the 1st of July 2018. | |
| Older data is also available starting from 2010. However, given that they did not crawl the full Crux back then, significantly less websites were crawled between 2010 and July 2018, which makes it hard to compare across different data points before July 2018. | |
| For example, in February 2025, almost 13M desktop websites are included in the crawl. An overview of the sample size per date is available here https://httparchive.org/reports/state-of-the-web#numUrls. \\ | |
| |
| WebPageTest is used to crawl the webpages, which records all requests and responses. In the end, HAR files are saved for each website visit. These HAR files are then stored in BigQuery tables. | Reading the archive dynamically fixes the bias and costs you the scale. The same paper reports going from **one request per site to more than 100**, hitting HTTP 429 from the Internet Archive even with a 5-second sleep after each page, and being able to compare only 2,026 sites as a result. |
| |
| In order to access the data: | ==== Escapes: when an archived page measures today ==== |
| 1. Go to https://developer.chrome.com/docs/crux and log in with Google account | |
| 2. Click "Select a project" and then "New Project" | |
| 3. Give your project a name and click the "Create" button | |
| 4. Navigate to the BigQuery console https://console.cloud.google.com/bigquery | |
| 5. In order to add the HTTP Archive tables to your project, click on the "+ Add" button on top of the Explorer sidebar and choose the Star a project by name option from the side menu. | |
| 6. Type in "httparchive" and click "STAR". | |
| 7. You can now browse the dataset, it is publicly accessible | |
| |
| Full tutorial https://har.fyi/guides/getting-started/. | URL rewriting inside a replayed page is not perfect, chiefly because it cannot rewrite URLs that JavaScript builds at runtime. When it fails, the replayed page issues a request to the **live** web. Lerner et al. measured this on archived snapshots of the Alexa top 500 and found that **16.1% of all requests attempted to escape the archive** — 17.2% of unique URLs and 11.5% of domains {[lerner2016internet]}. That measurement is from 2016, on 2015-era replay, and **nobody has repeated it since**; treat it as evidence that escapes matter, not as today's rate. |
| |
| | To put that in proportion: in the same table, robots.txt exclusion accounted for 2.0% of requests and the resource simply not being archived for 1.4%. **Escapes were by far the largest single failure mode — eight times robots exclusion.** Their crawler blocked those requests, which is the right default: an escaped request measures the present under a past timestamp. If your tooling does not block them, your "2014 measurement" silently contains 2026 responses. Blocking is one rule in whatever drives your browser — abort any request whose URL is not under ''web.archive.org'' — and **count what you blocked**, because that count is a fidelity statistic worth reporting. The same mechanism is a security problem, not only a measurement one — Lerner et al. later showed that an attacker who controls an escape destination can rewrite what a client sees in the archive, and that **73% of Top 500 sites and 80% of Top Million domains** had snapshots vulnerable to at least one such attack {[lerner2017_rewriting]}. |
| |
| The 'crawls' table contains all of the HTTP requests and responses. \\ | ==== Coverage: which pages are in there at all ==== |
| It is subdivided in two main tables: | |
| * pages - contains information about the crawled webpages | |
| * requests - contains headers and payload for requests and responses | |
| The tables are partitioned by date. As an example, the following query will return all requests on desktop pages for February 2025: | |
| |
| <code> | Coverage is good at the head of a popularity list and degrades fast outside it. |
| SELECT | |
| * | |
| FROM | |
| `httparchive.crawl.requests` | |
| WHERE | |
| date = '2025-02-01' | |
| AND client = 'desktop' | |
| </code> | |
| |
| This query costs 200TB in credits. It is therefore likely that the free credits will not be sufficient when you want to query the actual data. | * Hantke et al. served fresh data (within ±6 weeks of the requested date) for about **55%** of a stratified random sample of 5,000 Tranco domains — see [[#Which Archive]] for how the alternatives compare {[hantke2023you]}. |
| There is also a dataset that can be used for exploratory purposes: 'sample_data'. \\ | * On a population that is not a popularity list at all — 758,192 URLs from web censorship lists — Weinberg et al. found snapshots for **423,265 (55.8%)**, with a **median of only 3 snapshots per URL** and large gaps, which was too sparse to compute per-topic survival curves {[weinberg2017_topics]}. |
| | * On 18,900 government websites from 61 countries, Singh et al. found **16,500** had at least one snapshot — 87% of the total, or an average of 88% when averaged per country {[singh2026_empire]}. |
| |
| One advantage to using HTTP Archive is that they crawl the root page of each website as well as a number of subpages. Subpages can be distinguished in the dataset with the 'is_root_page' attribute. For example, we can rewrite the query above to only give us the landing pages of each website: | So "is it archived" is a property of your specific population, and it is a denominator you have to report. Check it before you design the study, not after. |
| |
| <code> | ==== Vantage point, bots and header differences ==== |
| SELECT | |
| * | The Internet Archive crawls predominantly from the US, and it aggregates content from many contributors, so its "view" corresponds to no single real vantage point. Comparing archived responses against live responses collected from Germany, California and Australia, Hantke et al. found that overall **8.0–8.7% of sites showed a syntactic difference in security headers and 7.0–7.7% a semantic one**, with the US comparison the closest — as expected {[hantke2023you]}. Mature headers were fine (X-Frame-Options 3.6%, CSP 6.7%, HSTS 6.5% from the US vantage); newer ones were not, with **COOP differing on about 70%** of the sites deploying it, almost entirely because Google properties serve those headers selectively. |
| FROM | |
| `httparchive.crawl.requests` | Part of that gap is deliberate: sites treat the archive's crawler differently. Of 212 otherwise-unexplained differences, **93 sites were doing User-Agent sniffing** and simply returned no security headers to ''archive.org_bot''. After accounting for redirects, stale snapshots and sniffing, **1.26% of sites (81 of 6,412) differed for no reason the authors could establish**. That residue is roughly the baseline for the live web too: a purely live study found **127 of 8,174 sites (1.55%)** returning inconsistent security headers between repeated identical requests {[roth2022_security]}. If you plan an archive-based measurement, budget for roughly this residue and test your own population for sniffing by requesting it live with the archive's User-Agent. |
| WHERE | |
| date = '2025-02-01' | ==== Contributors, status codes and the final origin ==== |
| | |
| | The Internet Archive stores responses from many contributors (identified by the ''x-archive-src'' header, resolvable through the Metadata API), with a range of status codes, and under a requested URL that may have been reached by different redirect chains at different times. Hantke et al. show that when two temporally adjacent snapshots disagree about a site's security posture, **the contributor is the single strongest explanatory feature**, ahead of the status code; the ''NULL'' contributor (24% of responses) disagrees with the others most often. Between **1.7% (HSTS) and 34.6% (COEP)** of their snapshot neighbourhoods contained observations with different security implications, and the majority of those differences came from the header simply being absent in some snapshots. |
| | |
| | ===== Doing It Right ===== |
| | |
| | The practices below are Hantke et al.'s best-practice list {[hantke2023you]} plus what the newer archive-based papers actually did. They are cheap; skipping them is the difference between a measurement and an artefact. |
| | |
| | - **Use neighbourhoods, not single snapshots.** For each URL and date, fetch the snapshots within a few days and aggregate them, rather than trusting the temporally closest one. Most spurious differences are "the header was missing in one capture". Report bounds if the aggregation matters: Hantke et al. detect **8% fewer** third-party script hosts using the closest snapshot instead of the union, and **12% fewer** using the intersection instead of the closest. |
| | - **Filter the contributor.** Dropping the ''NULL'' contributor removes the noisiest source and costs less than 3% of neighbourhoods. |
| | - **Restrict to 2XX**, unless error pages are your subject. |
| | - **Fix the final origin** across the whole time series. The archive files responses for different post-redirect origins under the same requested URL, and comparing them across years is an apples-to-oranges comparison. |
| | - **Wait for stabilisation.** The archive keeps ingesting data for a requested date for about **four days** — of the 11,364 domains for which they found data within ±1 day, 1,332 got a closer snapshot on day 1, 1,720 on day 2 and 1,760 on day 3 — after which content is effectively stable. Measure after that, and share the **end** URLs (with resolved timestamps), not the request URLs, so a reader lands on the same bytes. Note that this is not a permanence guarantee: exclusions can be applied retroactively at a site owner's request, and Lerner et al. measured robots-based blocking at 2.0% of replay requests already. If a result is load-bearing, keep your own WARCs of the snapshots it rests on. |
| | - **De-rewrite URLs before matching.** Replayed HTML contains ''%%https://web.archive.org/web/<timestamp>/https://a.com/script.js%%''. Every filter-list, eTLD+1 or third-party test must run against the recovered original URL. Singh et al. name this as an explicit step in their pipeline {[singh2026_empire]}. |
| | - **Be a polite client.** There is no published read-side rate limit (the widely cited "15 per minute" is the limit on **Save Page Now submissions**, not on reads((Internet Archive, //Too Many Requests//, 2019-11-10: "We are limiting the number of URLs you can submit to be Archived to the Wayback Machine, using the Save Page Now features, to no more than 15 per minute." — a submission limit, routinely mis-cited as a read limit.))), but 429s are real. Singh et al. used **3–5 seconds between sequential downloads and no concurrency** {[singh2026_empire]}; the maintained ''wayback'' Python library defaults to **0.4 CDX calls per second and 8 Memento calls per second**((https://wayback.readthedocs.io/en/latest/usage.html — ''WaybackSession'' defaults, checked 2026-08-13. Its documentation also states the Internet Archive's actual limits are not published.)). Budget 2–5× the wall-clock of a live crawl {[hantke2023you]}. |
| | |
| | ==== The anachronism trap ==== |
| | |
| | The archive-specific mistake that no live crawl can make: **applying today's classifier to yesterday's page**. A filter list, a tracker database, a vendor list or an LLM prompt written in 2026 encodes 2026's ecosystem. Run it over a 2006 snapshot and you are measuring the intersection of the past web with the present taxonomy. |
| | |
| | There is no clean fix, but there is a defensible practice, and Singh et al. {[singh2026_empire]} show it. They label with one fixed list version (EasyList and EasyPrivacy, June 2025) for all years — so that list drift cannot masquerade as a trend — and then //validate that choice// against historical list versions recovered from GitHub and from the archive itself, at three anchor years. The validation is the interesting part: on a random sample of 10,000 2006 URLs, **the 2006 EasyList flagged 67 URLs and the June 2025 list flagged 50, with only 6 in common**. Both lists were then manually adjudicated. Whatever you choose, hold the classifier fixed across time, say which version, and show what the alternative would have done. |
| | |
| | ===== Which Archive ===== |
| | |
| | <WRAP important> |
| | **The Memento Time Travel aggregator is gone.** ''timetravel.mementoweb.org'' — the cross-archive search service and the ''archivelist.xml'' endpoint that prior work used to //enumerate// public web archives — was operated by Los Alamos National Laboratory from 2015 until it was decommissioned in 2025; ''mementoweb.org'' is now a static, volunteer-run GitHub Pages site.((https://mementoweb.org/about/, checked 2026-08-13: "As a result of a managerial decision, the site was taken down towards the end of 2025 … The Time Travel service was a Memento aggregator that allowed searching across a wide range of public web archives … It operated between 2015 and 2025". Both ''timetravel.mementoweb.org'' and ''labs.mementoweb.org'' no longer resolve in DNS.)) The Memento **protocol** (RFC 7089, Informational) is unaffected and individual archives still speak it — Arquivo.pt's TimeMap endpoint answers normally — but the discovery step in Hantke et al.'s method {[hantke2023you]} cannot be reproduced as published. |
| | |
| | What remains is **[[https://github.com/oduwsdl/MemGator|MemGator]]**, an open-source Memento aggregator from the Old Dominion University group that co-authored the protocol, with a public instance at ''memgator.cs.odu.edu''. It works: a TimeMap request returns aggregated, correctly formed ''rel="memento"'' links, and the repository is actively maintained (last push April 2026). It is a smaller thing than Time Travel was — **13 live upstream archives** rather than the 30-plus Time Travel reached, still versioned ''1.0-rc8'', and its own ''/about'' page reports consecutive-failure counts in the tens of thousands for two of them. That failure count is itself a measurement of how many national archives have gone quiet. Treat MemGator as the surviving instrument, not as a like-for-like replacement. |
| | </WRAP> |
| | |
| | Hantke et al. {[hantke2023you]} remain the systematic comparison: 64 API endpoints from 38 Memento-supporting archives, plus Common Crawl. Their results, which are what you should assume until someone repeats the study: |
| | |
| | * They collected 64 API endpoints from 38 archives; **only 13 endpoints produced even one hit** for the Tranco top 5,000, and the four weakest of those (library and museum collections) covered at most 8% of it. The paper attributes the failures to endpoints that are simply outdated. |
| | * The Internet Archive dominated on every axis: most domains covered, freshest data, and the only archive that stays useful outside the top 5,000. On a stratified random sample of Tranco 1M it served fresh data for ~55% of domains against 7% for the best alternative. |
| | * **Combining the Internet Archive with all the others added a negligible number of fresh hits.** Using it as a single source is a defensible choice, and the multi-archive fallback that older digital-library work recommends does not pay for itself in this setting. |
| | * Several archives had **stopped crawling fresh data** by 2022 (the Library of Congress endpoint returned zero fresh hits), so an archive's usefulness is a function of your target //period//, not just its name. |
| | |
| | What to check before committing to one: |
| | |
| | * **Crawl location.** Both the Wayback Machine and HTTP Archive crawl from the US. If your question is regional — EU consent banners, geo-blocking, localised advertising — an archive is answering it from the wrong place. See [[design:crawling_location]]. |
| | * **Coverage of //your// population**, measured, not assumed (see above). A stratified probe of a few thousand URLs costs an afternoon. |
| | * **Temporal granularity.** Do snapshots exist near the dates you need, and how many per URL? A median of 3 snapshots over ten years supports a "did it ever" claim and not much else. |
| | * **Whether replay is browser-driven.** For anything runtime-dependent, this decides whether the archive can answer at all. |
| | |
| | ==== Internet Archive / Wayback Machine ==== |
| | |
| | The Wayback Machine passed **one trillion archived web pages in October 2025**((https://blog.archive.org/2025/10/31/one-trillion-web-pages-archived-internet-archive-celebrates-a-civilization-scale-milestone/, published 2025-10-31: "In October, it surpassed the threshold of preserving one trillion web pages.")) and covers 1996 onwards. Snapshot frequency is driven by the site: popular domains are captured close to daily, tail domains sporadically or never. Anyone can submit a page through Save Page Now, which means part of the archive is user-nominated rather than crawler-driven. |
| | |
| | Three access paths, in increasing order of usefulness for research: |
| | |
| | * **Availability API** — ''%%https://archive.org/wayback/available?url=<url>×tamp=<YYYYMMDDhhmmss>%%'' returns the closest snapshot as JSON. Fine for "is there anything near this date", useless for enumeration. |
| | * **CDX API** — ''%%https://web.archive.org/cdx/search/cdx?url=<url>&output=json%%'' lists every capture with timestamp, MIME type, status code and content digest. This is the one to build on: ''collapse=digest'' suppresses unchanged captures, ''from''/''to'' bound the range, and the digest column lets you detect that nothing changed without refetching. Expect intermittent ''504''s under load — retry with backoff rather than treating them as absence (one of our three probes on 2026-08-13 returned a 504 that succeeded on retry). |
| | * **Replay** — ''%%https://web.archive.org/web/<timestamp>/<url>%%'', optionally with the ''id_'' modifier for the unrewritten original response. Point a browser here for dynamic analysis, accepting the request amplification and 429 risk described above. |
| | |
| | Do not write your own client for the first two. [[https://github.com/edgi-govdata-archiving/wayback|''wayback'']] (Python, maintained by EDGI) wraps the CDX and Memento endpoints, applies conservative rate limits by default, and raises a typed error on 429 instead of silently returning nothing. |
| | |
| | It is also possible to request a snapshot of a single **resource** rather than a whole page — a given ''fingerprint.js'' build, say — though a subresource typically has far fewer captures than the pages that include it. |
| | |
| | ==== HTTP Archive ==== |
| | |
| | [[https://httparchive.org/|HTTP Archive]] is a different instrument that is often confused with a web archive: it does not let you replay a page, it publishes the **HAR files of its own monthly browser-driven crawl** into public BigQuery tables. Because the crawl is browser-driven, it does not have the static-capture blindness described above — requests, response headers and third-party resources are all recorded. What it does not give you is an arbitrary date: you get its monthly grid, from its US vantage point, on its URL list. |
| | |
| | Pages are loaded with [[https://www.webpagetest.org/|WebPageTest]] driving Chrome, and the full HAR is stored; the ''wptid'' column links each row back to its WebPageTest result. The URL list has been based on the [[programming:crux|Chrome User Experience Report (CrUX)]] since 1 July 2018. Data before that exists back to 2010 but covers far fewer sites, so comparisons across that boundary are not like-for-like. The current sample size per month is published at [[https://httparchive.org/reports/state-of-the-web#numUrls|state-of-the-web § Sample Size]] — check it rather than quoting a number from a paper. |
| | |
| | Access: |
| | |
| | Create a Google Cloud project, then in the BigQuery Explorer sidebar use **+ Add → Star a project by name** and star ''httparchive''. The dataset is public and needs no further permission; [[https://har.fyi/guides/getting-started/|har.fyi's getting-started guide]] has the click-through if you need it. |
| | |
| | The ''httparchive.crawl'' dataset is partitioned by date and split into ''pages'' (one row per crawled page) and ''requests'' (one row per HTTP request, with headers and payload): |
| | |
| | <code sql> |
| | SELECT * |
| | FROM `httparchive.crawl.requests` |
| | WHERE date = '2025-02-01' |
| AND client = 'desktop' | AND client = 'desktop' |
| AND is_root_page | |
| </code> | </code> |
| |
| Another advantage is that CRUX includes ranking of website popularity per buckets (e.g. top 1000, top 10000...) which is available in HTTPArchive. | <WRAP important> |
| For instance, let's take the 1000 most popular websites of February 2025 with the following SQL query: | That query scans on the order of 200 TB and will exhaust the free tier immediately. Always filter on ''date'' and ''client'' (they are the partition keys), select only the columns you need, and develop against the ''sample_data'' dataset or with ''LIMIT''. See [[https://har.fyi/guides/minimizing-costs/|har.fyi's cost guide]] before running anything at full scale. |
| | </WRAP> |
| |
| <code> | Two properties make it worth the cost. It crawls **subpages as well as landing pages** — filter with ''is_root_page'' — which is unusual and directly addresses the landing-page bias discussed in [[design:website_selection]]. And it carries the CrUX popularity **rank magnitude**, which is a bucket and not a rank: it takes the values 1,000 / 5,000 / 10,000 / 50,000 / 100,000 and up, and the buckets are **cumulative but exclusive of each other**. ''rank = 1000'' selects exactly the top 1,000 origins, but ''rank = 10000'' selects ranks 1,001–10,000 — for the top ten thousand you need ''rank <= 10000''((https://developer.chrome.com/blog/crux-rank-magnitude, checked 2026-08-13: "Row 2 may look surprising, indicating that there are only 9k origins in the top 10k set; this is because the origins in row 1 are also part of the top 10k set. To select the top 10k origins, one needs to specify ''experimental.popularity.rank <= 10000'' when querying." Half-steps (5k, 50k, 500k …) were added in the October 2022 dataset.)). The query below is therefore right for the top 1,000 and would be a trap at any other value: |
| SELECT | |
| * | <code sql> |
| FROM | SELECT * |
| `httparchive.crawl.pages` | FROM `httparchive.crawl.pages` |
| WHERE | WHERE date = '2025-02-01' |
| date = '2025-02-01' | |
| AND client = 'desktop' | AND client = 'desktop' |
| AND is_root_page | AND is_root_page |
| </code> | </code> |
| |
| If you only need a specific subset of websites e.g. you want to a longitudinal analysis on the same set of websites, than another option is to [[https://discuss.httparchive.org/t/how-to-download-the-http-archive-data/679|save the HAR files and them locally]]. There is no cost associated with downloading files from Google storage. You need to use the WebpageTest id (wptid from the pages table) in order to search for specific urls. | For a fixed set of sites tracked over time, downloading the HAR files from Google Cloud Storage using the ''wptid'' from the ''pages'' table is free and avoids the query cost entirely: see [[https://discuss.httparchive.org/t/how-to-download-the-http-archive-data/679|the download guide]]. Working with the resulting files is covered in [[programming:traffic_files]]. |
| |
| | ==== Common Crawl ==== |
| |
| ===== Wayback machine ===== | [[https://commoncrawl.org/|Common Crawl]] publishes a monthly crawl as WARC/WAT/WET files with a queryable CDX index (''%%https://index.commoncrawl.org/collinfo.json%%'' lists the available crawls; the July 2026 index is ''CC-MAIN-2026-30''). It is not a replay archive — there is no efficient way to ask "what did //this// URL look like on //that// date" — but it is enormous, free, and served from S3, so it is the practical choice for **breadth**: as a sampling frame beyond the reach of popularity lists, for link-graph work, or for corpus construction. Hantke et al. used it as a second archival source and found it added little to the Internet Archive for date-targeted measurement; Roth et al. found its CSP data largely matched the Internet Archive's {[hantke2023you,roth2020complex]}. Note that in the recent literature Common Crawl also appears as an **LLM training corpus** (CC-News and derivatives), which is a completely different use — do not conflate the two when reading related work. |
| The Internet Archive's Wayback machine is also an open source project that aims to build a digital library. The archive includes 800+ archived web pages in WARC format. Anyone can store a snapshot of their webpage on the Wayback Machine. Along with that, the organization also crawls the web sporadically. | |
| Unlike HTTP Archive, they don't schedule monthly crawls, it is highly dependent on the website whether it has been captured at a specific time. Therefore, if you are doing research on less popular websites, you cannot be certain about their availability in the archive. You can verify if a snapshot is available for a web page or a resource with the [[https://archive.org/help/wayback_api.php|Wayback Availablity API]]. | |
| |
| Another limitation is that Wayback Machine applies [[https://archive.org/details/toomanyrequests_20191110|rate limiting]] to their API to no more than 15 requests per minute. For this reason, consider reducing the traffic, for example by excluding image- and font-type resources. | ==== Other archives ==== |
| |
| On the other hand, it is more straightforward to work with than HTTP Archive since no Google account, SQL querying etc. is needed. Additionally, it is possible to request a snapshot for a specific resource e.g. for a fingprint.js script. However, depending on the resource, there might be less available snapshots for the resource than the whole webpage. | Worth knowing about, mostly for jurisdiction-specific work: |
| | |
| | * **[[https://arquivo.pt/|Arquivo.pt]]**, the Portuguese Web Archive, operated by FCCN within Portugal's FCT — not a public library service. It holds **23,048 million files across 73 million websites (1.6 PB compressed) as of January 2026**((https://sobre.arquivo.pt/en/about/press/the-portuguese-web-archive-in-numbers/, checked 2026-08-13.)) and offers Memento, CDX and full-text search APIs — its TimeMap endpoint still answers, which is how the Memento protocol survives its aggregator. In Hantke et al.'s probe it placed fourth on raw hits (3,412 against the Internet Archive's 4,061) but far behind on freshness (**338 fresh hits against 3,713**). |
| | * **[[https://archive-it.org/|Archive-It]]**, the Internet Archive's subscription service for institutional collections. Narrower than the main archive, but Hantke et al. found high //freshness// for what it does cover — useful if your population overlaps a curated collection. |
| | * **National and institutional archives** — the Icelandic, Croatian and Czech web archives, the Library of Congress, several university libraries. Now that Time Travel is gone, the two surviving inventories are MemGator's [[https://oduwsdl.github.io/MemGator/archives.json|''archives.json'']] — **20 endpoints of which 7 are flagged inactive**, several of them named "(DEFUNCT)" or "(MOVED TO A-IT)" in the file itself — and [[https://en.wikipedia.org/wiki/List_of_web_archiving_initiatives|Wikipedia's list]], which is broader and unverified. In Hantke et al.'s probe, most of these archives returned few or no fresh hits and several appear to have stopped crawling; treat any of them as a possible source for a **national** population and verify coverage yourself before designing around it. |
| | |
| | ===== Recording Your Own Archive ===== |
| | |
| | If your goal is reproducibility rather than history, the better move is often to archive **your own** crawl rather than mine somebody else's. This gives you a browser-driven capture with no coverage gaps for the pages you crawled, no anachronism between crawler and replayer, and escapes you can actually block because you control the replayer — at the cost of not being able to reach backwards in time. |
| | |
| | * **[[https://github.com/webrecorder/browsertrix-crawler|browsertrix-crawler]]** (Webrecorder) drives a real browser and writes WARC/WACZ. Actively developed (v1.14.2, August 2026); the hosted **Browsertrix** service is the same engine. |
| | * **[[https://github.com/webrecorder/pywb|pywb]]** replays WARC collections locally, including the "fuzzy matching" of URLs that archives rely on when a CDN varies resource URLs between loads. This is what Zhu et al. replay through {[zhu2025_toward]}. Latest release v2.9.1 (October 2025). |
| | * **WARC** itself is ISO 28500; the current specification is **WARC 1.1**, maintained by the IIPC at [[https://iipc.github.io/warc-specifications/|iipc.github.io/warc-specifications]]. ''warcio'' is the usual Python read/write library. |
| | * **WebREC and the ''.web'' format** {[hantke2025web]} are the research-specific version of this idea: a measurement tool plus an archival format designed so that a //measurement// (not just a page) can be re-run later without recrawling. The authors report that 70% of the papers surveyed in a 2024 crawling SoK could have been run with WebREC unmodified, and 48% could have been run against existing ''.web'' archives with no new crawling at all. Code: [[https://github.com/cispa/webrec|cispa/webrec]]. |
| | |
| | The trade-off is stated plainly: an archive you recorded yourself answers "can someone else reproduce my result", and a public archive answers "what was there before I started looking". Only the second is time travel, and it is the one with all the caveats. |
| | |
| | ===== Use in Publications ===== |
| | |
| | Figures below are from a structured extraction over 5,859 papers from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026. Every claim is a claim about those seven venues. |
| | |
| | ==== Archives are rare, and they are a supplement ==== |
| | |
| | Of **5,342 papers for which the extraction recorded any data-provenance tuple**, the modes break down as follows. A paper can use several, so the column does not sum to 100%, and ''not-stated'' is a value rather than an answer: |
| | |
| | ^ Data provenance ^ Papers ^ Share of 5,342 ^ |
| | | existing-dataset | 2,534 | 47.4% | |
| | | active-probing | 1,780 | 33.3% | |
| | | live-crawl | 1,261 | 23.6% | |
| | | passive-collection | 1,012 | 18.9% | |
| | | not-stated | 353 | 6.6% | |
| | | **web-archive** | **68** | **1.3%** | |
| | |
| | That enum undercounts, because a paper whose whole dataset comes from the Wayback Machine is often coded as ''existing-dataset'' instead — Singh et al. {[singh2026_empire]} is exactly that case. Sweeping five free-text fields for archive names and folding the spellings (see [[#Methodology and limitations of these figures]]) gives a fuller count: **122 papers name a web archive, 1.8× the enum**, of which **85** name a //replay// archive — one you can point a client at (Internet Archive, Arquivo.pt, Archive-It, Memento, a national archive, or an unnamed "web archive") — and the rest only Common Crawl or HTTP Archive. |
| | |
| | Seven of those 122 are not empirical papers (they are mostly LLM work using Common Crawl as a training corpus), so the share of empirical work is **115 of 5,118, or 2.2%**, against 24.2% that ran a live crawl. And the archive is almost never the whole study: |
| | |
| | ^ Of the 122 papers naming an archive ^ Papers ^ Share ^ |
| | | also ran a live crawl (''temporal.mode'') | 71 | 58.2% | |
| | | are in the ''crawled'' population | 82 | 67.2% | |
| | | collected **nothing of their own** — no crawl, probe or passive tuple | 31 | 25.4% | |
| | | //strict reading//: every provenance tuple is ''web-archive'' | //8// | //6.6%// | |
| | |
| | The last two rows differ by a factor of four for the same reason the headline count does: a paper built entirely on the Wayback Machine is often coded ''existing-dataset'' rather than ''web-archive''. **31 is the number to use** — 8 is the enum's floor, and 23 papers sit in the gap, including {[lerner2017_rewriting]} and {[singh2026_empire]}. |
| | |
| | ==== It is not growing ==== |
| | |
| | Archive use rose once, in the mid-2010s, and has been flat since. Denominators are empirical papers in the same bucket: |
| | |
| | ^ Years ^ Empirical papers naming an archive ^ Empirical papers ^ Share ^ |
| | | 2010–2013 | 1 | 460 | 0.2% | |
| | | 2014–2017 | 14 | 718 | 1.9% | |
| | | 2018–2021 | 39 | 1,272 | 3.1% | |
| | | 2022–2024 | 36 | 1,649 | 2.2% | |
| | | 2025–2026 //(provisional)// | 25 | 1,019 | 2.5% | |
| | |
| | //The last bucket is provisional by construction: CCS 2026 and IMC 2026 have not been held, and IEEE S&P 2026 and WWW 2026 are only partly indexed.// So the honest reading is: the method arrived with Lerner et al.'s 2016 archaeology of tracking and Stock et al.'s 2017 history of client-side security, settled at two to three percent of empirical work, and has neither been superseded nor spread. What //has// changed since 2023 is that the field now knows how badly a naive archive measurement can go wrong: the newest papers in this corpus state their mitigations explicitly where earlier ones did not ({[singh2026_empire]} is the clearest example). We have not measured that as a trend — the reporting figures below are not time-sliced — so read it as an impression from reading the papers, not a result. |
| | |
| | By venue, over empirical papers: PETS 3.6%, IMC 3.4%, NDSS 2.7%, WWW 2.4%, USENIX 1.9%, CCS 1.7%, IEEE S&P 0.9%. |
| | |
| | ==== Which archives ==== |
| | |
| | Paper counts, folded across 27 spellings of the Internet Archive alone, of the 122 papers naming an archive: |
| | |
| | ^ Archive ^ Papers ^ Share of 122 ^ Raw spellings folded ^ |
| | | Internet Archive / Wayback Machine | 82 | 67.2% | 27 | |
| | | Common Crawl | 28 | 23.0% | 12 | |
| | | HTTP Archive | 16 | 13.1% | 5 | |
| | | Unspecified "web archive" | 4 | 3.3% | 6 | |
| | | Memento / Time Travel | 1 | 0.8% | 1 | |
| | |
| | Two more families are counted by the report script and deliberately **excluded from the 122**, so they have no share of it: self-hosted archiving software (Heritrix, pywb, Webrecorder, warcprox — 6 papers, of which 2 also name a public archive), and CC-News used as an NLP training corpus (3 papers, of which 2 also name a public archive). A paper replaying its own WARCs is not using a public archive, and a paper fine-tuning on CC-News is not doing archival measurement at all. |
| | |
| | The concentration is the finding, and it matches Hantke et al.'s experimental result: for date-targeted work, there is effectively one archive. It also means the field's archive-based results share one vantage point, one crawler and one set of coverage biases — a systemic dependency nobody has stress-tested. |
| | |
| | ==== What archive papers report, and what they omit ==== |
| | |
| | Of the 94 papers whose archive use produced a datable provenance tuple: |
| | |
| | ^ Stated on the archive tuple ^ Papers ^ Share of 94 ^ //(live-crawl papers, n=1,261)// ^ |
| | | First archival date | 71 | 75.5% | //63.1%// | |
| | | Last archival date | 71 | 75.5% | //62.7%// | |
| | | How many snapshots were fetched | 30 | 31.9% | //30.6%// | |
| | | Cadence — how often, or how a snapshot was chosen | 33 | 35.1% | //25.1%// | |
| | |
| | Archive papers date their data **better** than live-crawl papers do, which makes sense: the date is the point. But two thirds do not say **how many snapshots they took**, and nearly two thirds do not say **how they chose one when several existed** — and after everything above, snapshot selection is precisely the decision that determines the result. This is the reporting gap specific to archive work. |
| | |
| | ==== Methodology and limitations of these figures ==== |
| | |
| | * **How they were produced.** One structured record per paper was extracted from full text; each tuple carries a verbatim evidence quote and its section, so any figure here traces to a sentence. ''temporal.mode'' is an enum and is stable; the archive //names// are free text and are not. |
| | * **Folding.** Archive names agree run-to-run on roughly a fifth of exact strings, so they were folded into families before counting — the Internet Archive alone appears under 27 distinct spellings, including its CDX, Availability and Metadata APIs and three client libraries. The fold rules, the excluded names and the full residue are on the [[provenance:design:archives|provenance page]]. Counts are of papers, never tuples. |
| | * **The enum undercounts and the name sweep overcounts.** ''temporal.mode = web-archive'' misses papers whose archive use was coded as ''existing-dataset''; the name sweep counts a paper that merely //mentions// the Wayback Machine alongside one that is built on it. The truth is between 68 and 122; both are given rather than one being presented as the answer. |
| | * **Silence is not absence.** "Does not state how many snapshots" means the paper did not say so. These are reporting figures. |
| | * **Venue coverage.** Seven venues only. Digital-library and web-science venues (JCDL, TPDL, WWW's archiving tracks elsewhere) carry much of the archiving literature and are **not** in this corpus, so the numbers describe how security and privacy measurement uses archives, not the state of web archiving research. |
| | * **2025 and 2026 are incomplete** for reasons of calendar and indexing; the provisional bucket is labelled wherever it appears. |
| | * **Every query behind this section, its report script and its unedited output** are on [[provenance:design:archives]]; corpus-level caveats are on [[literature:corpus]]. |
| | |
| | ===== What to Report ===== |
| | |
| | An archive-based methods paragraph a reader can act on states, in this order: |
| | |
| | - **Which archive**, by name and access path (CDX, Memento, replay, BigQuery), not "the web archive". |
| | - **Coverage of your population**: how many of the URLs you wanted had any snapshot, and how many had one near each date you needed. This is a denominator, and it is usually not the same as your sample size. |
| | - **Snapshot selection rule**: closest-to-date, first-of-year, random-within-year, neighbourhood union or intersection — and how many snapshots per URL you ended up with. |
| | - **Static or dynamic**: did you parse stored HTML, or replay in a browser? Everything a reader can conclude about third parties, cookies and scripts hinges on this one word. |
| | - **Escape handling**: were live-web requests blocked, and how many were there? |
| | - **Filtering**: contributor, status code, final origin, freshness window. |
| | - **Classifier versioning**: which filter-list/database version was applied to which years, and why (see [[#The anachronism trap]]). |
| | - **The resolved snapshot URLs**, as an artifact. They are short, they are stable after the stabilisation window, and they make everything above checkable. |
| | |
| | ===== Open Questions ===== |
| | |
| | <WRAP todo> |
| | * **The archive comparison needs redoing, and the instrument for redoing it shrank.** Memento Time Travel was decommissioned in 2025; MemGator survives but reaches 13 archives rather than 30-plus, so Hantke et al.'s enumeration {[hantke2023you]} cannot be repeated as written and its 2022 coverage numbers are now four years old. Nobody has published a current census of which public web archives still crawl. |
| | * **Nobody has re-measured static-vs-dynamic archive bias at scale.** The 73.9%/95.3% tracker gap {[hantke2023you]} rests on 2,026 sites, limited by rate-limiting. Zhu et al.'s {[zhu2025_toward]} 45.8% fidelity gap is measured on their own crawls, not on the Internet Archive's holdings. A large-scale measurement of what the Internet Archive itself is missing, per resource type, does not exist. |
| | * **Monoculture risk is unquantified.** Roughly two thirds of archive-using papers depend on one archive, one crawler and one vantage point. What that shared bias does to the field's longitudinal results has not been studied — and Common Crawl, the only comparison Hantke et al. could make, was rejected on coverage rather than validated on agreement. |
| | * **Archive-based measurement of regional phenomena has no instrument.** Both major sources crawl from the US. For GDPR-era consent, geo-blocking or regional advertising, there is no archive with an EU vantage point at usable coverage — the national archives fail the freshness test. |
| | </WRAP> |
| | |
| | ===== Related Pages ===== |
| | |
| | * [[design:website_selection]] and [[design:sampling]] — an archive is a sampling frame with its own coverage bias; what is //in// it is not what you asked for. |
| | * [[design:crawling_location]] — both major archives crawl from the US, which is the wrong vantage point for anything regional. |
| | * [[programming:crawler]] — the live-crawl alternative, and the tooling you would replay an archive with. |
| | * [[programming:traffic_files]] — HAR and WARC handling, which is what an archive hands you. |
| | * [[programming:crux]] — the URL list behind HTTP Archive's crawl. |
| | * [[programming:stateful_stateless]] — archives are stateless by construction, which rules out a class of measurement. |
| |
| ====== References ====== | ====== References ====== |
| /* This enables discussion under this article. */ | /* This enables discussion under this article. */ |
| ~~DISCUSSION~~ | ~~DISCUSSION~~ |
| | |
