This is an old revision of the document!
Table of Contents
Artifacts
This is the Artifacts namespace. There are no child pages: what belongs here is the measurement-specific release problem, not a general lecture on open science. A PhD student who has just finished a crawl lands here from start and from Website selection, and needs to know what to put in the artefact, where to put it so a 2026 Available badge will accept it, and what to do when the data has people in it.
A GitHub URL is not an archival artifact. USENIX Security 2026's Artifacts Available badge — mandatory for every accepted paper — refuses GitHub, GitLab and personal pages. The archived copy has to be reachable via a long-term stable reference or DOI. IEEE S&P 2026 and TheWebConf 2026 say the same for Available. Zenodo is the host those calls name first.1)
Of 5,859 papers in this corpus (seven venues, 2010–2026), 3,321 (56.7%) release at least one authors'-own artifact URL. That rate rose 23.7% (2010–2013) → 65.0% (2022–2024); 2024 itself is 457 of 690 (66.2%). Among those 3,321, GitHub still hosts 1,896 (57.1%) and a named archival host or a DOI resolver hosts 622 (18.7%) — Zenodo, OSF, Figshare, Dryad, Dataverse, 4TU, Software Heritage, plus 34 other-DOI URLs whose prefix was not one of those (a resolver, not a verified archive). Zenodo among own-link papers was 42 of 457 in 2024 (9.2%) and 228 of 592 in 2025 (38.5%) — the jump is USENIX's 2025–2026 open-science policy, under which 219 of 293 USENIX papers in 2025–2026 have a Zenodo link (74.7%). 2025–2026 are provisional venue-years; do not read the last column as a completed year.
What this page is for
- Here: what a measurement artefact has to contain; which hosts the 2026 calls accept; the corpus rates; what to withhold and how to say so.
- Study preregistration: the thing a badge is not. 3,321 papers release a URL; 15 preregistered. They answer different questions.
- Ethics: IRB, residential proxies, crawler identification. This page only covers the release decision once the data exists.
- Website selection / Sampling / Longitudinal: pinning the list. Publishing the pinned list is the artefact step those pages point at.
What to read first
- Demir et al., TheWebConf 2022 [1Demir, Nurullah; Große-Kampmann, Matteo; Urban, Tobias; Wressnegger, Christian; Holz, Thorsten; Pohlmann, Norbert (2022): "Reproducibility and Replicability of Web Measurement Studies", in: Proceedings of the ACM Web Conference. (DOI)] — 117 web-measurement papers coded against 18 reproducibility criteria, plus a 4.5-million-page experiment that varies the setup on purpose. It is the checklist this page does not need to reinvent, and the evidence that the configuration you did not write down moves the number.
- Your venue's live artifact-evaluation page, not last year's. The 2026 calls disagree with each other (table below) and with 2024. The corpus cannot tell you which badge is current; the call can.
- The USENIX 2026 badges page if you are submitting there. Available is no longer optional, and GitHub is no longer enough.
What to put in it
A measurement artefact is not “the code”. The papers that cannot be re-run are missing the sample, the pins, and the per-site outcomes. Demir et al. [1Demir, Nurullah; Große-Kampmann, Matteo; Urban, Tobias; Wressnegger, Christian; Holz, Thorsten; Pohlmann, Norbert (2022): "Reproducibility and Replicability of Web Measurement Studies", in: Proceedings of the ACM Web Conference. (DOI)] coded 18 criteria; configuration details fail often (crawler adjustments, location, browser), and so does the repository: C9 “crawler publicly available” was omitted by 67% of their 117 papers, and only 24% made results openly available.
For a crawl, the minimum that lets someone else regenerate your headline number — or re-analyse it with a clustered standard error, see Hypothesis testing — is:
| Piece | Why it is not optional |
|---|---|
| The URL list, as fetched. Rank, domain, fetch date. A Tranco id if you used one (Tranco); the exact rows if you filtered. “The top 10k” is not a sample. | Website selection, Sampling |
| The crawler pin. Library version, browser build (not “Chrome”), headless/headful, statefulness, interaction depth, consent action. | Crawler, Stateful stateless, Consent |
| The vantage point. Provider, city, ASN if you know it. | Crawling location |
| The classifier, with its version. EasyList commit, Disconnect snapshot, your model weights, the threshold. | Requests, Website classification |
| Per-site outcomes, not only aggregates. One row per site: loaded or not, the labels, the counts. This is what makes a cluster-aware re-analysis possible. | Hypothesis testing, Regression |
| What you deleted, and why. Raw cookies, request bodies, identifiers. A README that says “we stripped PII” is not a schema. | this page, Ethics |
| A licence, and a DOI. GitHub's default branch has neither. | venue table below |
A system paper that evaluated a defence on a testbed has a different minimum (the implementation, the configs, the test vectors). This page is for the student who crawled.
Use in publications
Figures below are from scripts/report_artifacts.mjs over the 5,859-paper extraction (2010–2026, CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P). “Own-link” means at least one artifacts.links[] entry with belongsToAuthors === true. That is the same definition as Corpus's “Releases an artifact link” column.
| Population | N | Authors-own link | Share |
|---|---|---|---|
| all papers | 5859 | 3321 | 56.7% |
| empirical | 5118 | 2872 | 56.1% |
| crawled | 1120 | 684 | 61.1% |
Dropping the 251 posters and ≤4-page records moves the all-papers rate 56.7% → 58.4% (3,277 / 5,608). The gap this page is about is not a poster artefact.
Availability is a different question from a URL. Among empirical papers, 2,890 of 5,118 (56.5%) state an availability other than none-mentioned / no record — OVERVIEW.md's artifacts.availability row. 2,845 papers in the whole corpus are labelled public; 54 of those have no authors'-own URL (they released “as part of scamper”, a WINE dataset id, a citation). 530 papers have an authors'-own URL and are not labelled public (promised, restricted, on-request, withheld, or the extractor stored the URL under none-mentioned). The headline uses the URL.
| availability | Papers (of 5859) | Share | Of which crawled |
|---|---|---|---|
| public | 2845 | 48.6% | — |
| promised-not-yet-available | 289 | 4.9% | 72 |
| on-request | 91 | 1.6% | 32 |
| restricted | 79 | 1.3% | 24 |
| explicitly-withheld | 52 | 0.9% | 10 |
| none-mentioned | 2183 | 37.3% | — |
| no artifacts record | 320 | 5.5% | — |
What they release
Paper-counted kinds, of the 3,321 with an authors'-own URL. A paper with source-code and a dataset is in both rows; the kinds sum to 4,199.
| kind | Papers | Share of 3321 |
|---|---|---|
| code-and-data | 1307 | 39.4% |
| source-code | 1175 | 35.4% |
| project-page | 508 | 15.3% |
| dataset | 448 | 13.5% |
| web-demo-or-service | 296 | 8.9% |
| extended-version-or-appendix | 236 | 7.1% |
| other | 94 | 2.8% |
| survey-instrument | 88 | 2.6% |
| browser-extension | 23 | 0.7% |
| mobile-app | 12 | 0.4% |
| preregistration | 12 | 0.4% |
The 12 preregistration-kind links are the schema signal. Study preregistration's full-text sweep finds 15 papers that preregistered; do not publish 12 as the preregistration count. A venue badge is not a preregistration.
Where they host it
Hostnames folded into families (scripts/host_fold.mjs; residue printed on the provenance page). A paper with GitHub and Zenodo is in both rows. Shares are of 3,321.
| Family | Permanence | Papers | Share of 3321 |
|---|---|---|---|
| GitHub | mutable-repo | 1896 | 57.1% |
| Zenodo | archival | 436 | 13.1% |
| Google Sites (project page) | project-page | 157 | 4.7% |
| OSF | archival | 112 | 3.4% |
| arXiv | publisher-page | 97 | 2.9% |
| USENIX paper page | publisher-page | 91 | 2.7% |
| YouTube / Vimeo (demo) | other | 65 | 2.0% |
| URL shortener | shortener | 59 | 1.8% |
| Google Drive / Docs | mutable-repo | 41 | 1.2% |
| anonymous.4open.science (double-blind GitHub) | anonymized | 36 | 1.1% |
| other DOI repository | archival | 34 | 1.0% |
| Figshare | archival | 28 | 0.8% |
| GitLab | mutable-repo | 25 | 0.8% |
Union of archival families: 622 / 3,321 = 18.7%. 59 papers (1.8%) give a shortener as the artifact URL (bit.ly, tinyurl, goo.gl). That URL is not an artifact; it is a redirect the owner can retarget. 667 distinct hosts did not match a family — almost all lab and university project pages. GitHub remains the default; the archival hosts are the minority, and they are new.
Over time
Own-link share of all papers that year, then GitHub / Zenodo share of the own-link papers. 2025–2026 starred: CCS/IMC 2026 unheld; IEEE S&P/WWW 2026 incompletely selected.
| Year | Papers | Own-link | Own-link share | GitHub (of own-link) | GitHub share | Zenodo (of own-link) | Zenodo share |
|---|---|---|---|---|---|---|---|
| 2010 | 119 | 29 | 24.4% | 0 | 0.0% | 0 | 0.0% |
| 2011 | 116 | 28 | 24.1% | 1 | 3.6% | 0 | 0.0% |
| 2012 | 151 | 37 | 24.5% | 2 | 5.4% | 0 | 0.0% |
| 2013 | 125 | 27 | 21.6% | 6 | 22.2% | 0 | 0.0% |
| 2014 | 166 | 56 | 33.7% | 9 | 16.1% | 0 | 0.0% |
| 2015 | 190 | 71 | 37.4% | 17 | 23.9% | 0 | 0.0% |
| 2016 | 182 | 72 | 39.6% | 13 | 18.1% | 0 | 0.0% |
| 2017 | 231 | 96 | 41.6% | 35 | 36.5% | 0 | 0.0% |
| 2018 | 254 | 118 | 46.5% | 57 | 48.3% | 1 | 0.8% |
| 2019 | 402 | 176 | 43.8% | 99 | 56.3% | 2 | 1.1% |
| 2020 | 404 | 206 | 51.0% | 129 | 62.6% | 1 | 0.5% |
| 2021 | 379 | 228 | 60.2% | 146 | 64.0% | 4 | 1.8% |
| 2022 | 546 | 338 | 61.9% | 234 | 69.2% | 5 | 1.5% |
| 2023 | 719 | 475 | 66.1% | 321 | 67.6% | 21 | 4.4% |
| 2024 | 690 | 457 | 66.2% | 307 | 67.2% | 42 | 9.2% |
| 2025* | 770 | 592 | 76.9% | 318 | 53.7% | 228 | 38.5% |
| 2026* | 415 | 315 | 75.9% | 202 | 64.1% | 132 | 41.9% |
| Window | Papers | Own-link | Share |
|---|---|---|---|
| 2010–2013 | 511 | 121 | 23.7% |
| 2014–2017 | 769 | 295 | 38.4% |
| 2018–2021 | 1439 | 728 | 50.6% |
| 2022–2024 | 1955 | 1270 | 65.0% |
| 2025–2026* | 1185 | 907 | 76.5% |
GitHub's share of own-link papers peaked in 2022 (69.2%) and dropped in 2025 (53.7%) as Zenodo rose. That is a venue-policy effect, not a GitHub decline in absolute terms: 2025 still has 318 GitHub papers, more than 2022's 234.
By venue
| Venue | Papers | Own-link | Share | GitHub of own-link | Zenodo of own-link |
|---|---|---|---|---|---|
| CCS | 990 | 467 | 47.2% | 252 | 25 |
| IEEE-SP | 767 | 423 | 55.1% | 270 | 12 |
| IMC | 638 | 313 | 49.1% | 163 | 12 |
| NDSS | 701 | 443 | 63.2% | 310 | 102 |
| PETS | 510 | 258 | 50.6% | 158 | 5 |
| USENIX | 1410 | 980 | 69.5% | 460 | 233 |
| WWW | 843 | 437 | 51.8% | 283 | 47 |
USENIX is the venue that currently requires an open-science appendix, and it is the venue with the highest own-link rate (69.5%) and more than half of the Zenodo mass (233 of 436). PETS, which has run optional artifact evaluation for years, has 5 Zenodo papers. Optional AE does not produce archival deposits; a mandatory Available badge does.
Venue artifact evaluation, today
Checked 2026-08-27 against each call. Training data is stale here by construction. scripts/external_checks_artifacts.sh re-fetches these pages.
| Venue | Live rule (read 2026-08-27) | Badges | GitHub enough for Available? |
|---|---|---|---|
| USENIX Security | Open-science appendix mandatory at submission; Available re-verified after acceptance. Functional / Reproduced optional, later. | Available (mandatory), Functional, Reproduced | No. Long-term stable reference or DOI required (GitHub/GitLab/personal pages refused). |
| IEEE S&P | Optional, post-acceptance. | Available, Functional, Reproduced | No for Available. DOI-backed (Zenodo / FigShare / Dryad). GitHub allowed as an extra pointer. |
| CCS | Optional AE after acceptance, ACM badges. Artifact Appendix encouraged after evaluation, not a mandatory submission appendix. | Available, Evaluated (Functional or Reusable), Results Reproduced — ACM Artifact Review and Badging v1.1 | No for Available. ACM's Available badge is an archival deposit. |
| NDSS | Optional, post-acceptance. 2026 cycle closed: 114 artifacts evaluated, 112 Available, 97 Functional, 70 Reproduced, 3 Distinguished Artifact Awards. | Available, Functional, Reproduced | Check the live call. NDSS 2026 used the same three-badge vocabulary. |
| TheWebConf | Optional, camera-ready. Light review: the artifact exists, is downloadable, has instructions. | Available only | No. DOI required. GitHub may sit next to the DOI in the resource-availability statement. |
| IMC | Artifact-availability declaration at submission (full / partial / none). Accepted papers shepherded to deliver what they promised. Community Contribution Award requires public data or code by camera-ready. Separate Replicability Track. | No ACM-style badge set on the 2026 CFP. The declaration is the mechanism. | Not a badge question. A GitHub repo can satisfy “publicly available” for the award; pin a commit and also deposit. |
| PETS / PoPETs | Optional. Available / Functional / Reproduced. Live page (read 2026-08-27) is titled PoPETs 2027 AE; PETS 2026 Artifact Award already announced. | Available, Functional, Reproduced | Optional means you can skip it. If you want Available, deposit. |
The schema's artifacts.badge field is non-null for 15 of 5,859 papers (0.3%). That is not a badge-award rate. Papers almost never write the badge name in a sentence the extractor can attach. A full-text sweep for “artifact evaluation” hits 59 papers — still an upper bound, still not an award rate. Do not use this corpus to say how often a venue awards badges. Use the venue's own AE results page, as NDSS 2026 publishes.
GitHub vs a DOI
GitHub is the right place to develop. It is the wrong place to cite. The default branch moves; a force-push rewrites history; a renamed account 404s the camera-ready URL. The 2026 Available calls noticed.
The pattern that satisfies every row of the table above:
- Develop on GitHub.
- At camera-ready, make a release (a tag, not “latest”).
- Push that release to Zenodo (the GitHub–Zenodo integration mints a DOI per release). You get a version DOI (this snapshot) and a concept DOI (the record that will always resolve to the newest version). Cite the version DOI in the paper.
- Leave the GitHub URL in the README so people can file issues. It is a pointer, not the archive.
OSF, Figshare, Dryad, institutional Dataverse, 4TU.ResearchData and Software Heritage also issue DOIs. Zenodo is the one the security AE pages name first, and it is the one 436 papers in this corpus already use.
anonymous.4open.science (36 papers) is a double-blind stand-in for GitHub during review. It is not the camera-ready archive. Swap it for a DOI before the Available check.
When the crawl contains personal data
USENIX 2026's open-science appendix requires you to explain a missing artifact. “We do not release the data” without a reason is no longer a complete sentence at that venue. The reasons in this corpus are real, and they cluster.
People in the data. The reasons cluster. These are the sentences, not a typology we imposed.
| Paper | Why the artifact is not public |
|---|---|
| Dey et al., IMC 2013, Profiling high-school students with Facebook [2Dey, Ratan; Ding, Yuan; Ross, Keith W. (2013): "Profiling high-school students with facebook: how online privacy laws can actually increase minors' risk", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] | PII of minors. “we will not be making our data sets public and we will not explicitly identify the high schools involved” |
| Lyons et al., USENIX Security 2023, Log: It's Big, It's Heavy, It's Filled with Personal Data! [3Lyons, Allan; Gamba, Julien; Shawaga, Austin; Reardon, Joel; Tapiador, Juan; Egelman, Serge; Vallina-Rodriguez, Narseo (2023): "Log: It's Big, It's Heavy, It's Filled with Personal Data! Measuring the Logging of Sensitive Information in the Android Ecosystem", in: Proceedings of the USENIX Security Symposium. (Link)] | Re-identification. “unique combinations of app names in conjunction with data from other sources could still identify participants, which is why we are not releasing our data.” |
| Bertram et al., CCS 2019, Five Years of the Right to be Forgotten [4Bertram, Theo; Bursztein, Elie; Caro, Stephanie; Chao, Hubert; Feman, Rutledge Chin; Fleischer, Peter; Gustafsson, Albin; Hemerly, Jess; Hibbert, Chris; Invernizzi, Luca; Donnelly, Lanah Kammourieh; Ketover, Jason; Laefer, Jay; Nicholas, Paul; Niu, Yuan; Obhi, Harjinder; Price, David; Strait, Andrew; Thomas, Kurt; Verney, Al (2019): "Five Years of the Right to be Forgotten", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)] | The URL list is the sensitive object. “We cannot directly reveal a sample mapping” |
| Kim et al., IEEE S&P 2019, Touching the Untouchables [5Kim, Hongil; Lee, Jiho; Lee, Eunkyu; Kim, Yongdae (2019): "Touching the Untouchables: Dynamic Security Analysis of the LTE Control Plane", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)] | Dual-use. “A public release is not planned as LTEFuzz can be used for malicious purposes” |
| Pastrana et al., TheWebConf 2018, CrimeBB [6Pastrana, Sergio; Thomas, Daniel R.; Hutchings, Alice; Clayton, Richard (2018): "CrimeBB: Enabling Cybercrime Research on Underground Forums at Scale", in: Proceedings of the ACM Web Conference. (DOI)] | Restricted-access release, not a public URL. Available to academic researchers from the Cambridge Cybercrime Centre. |
| Beverly et al., IMC 2018, In the IP of the Beholder [7Beverly, Robert; Durairajan, Ramakrishnan; Plonka, David; Rohrer, Justin P. (2018): "In the IP of the Beholder: Strategies for Active IPv6 Topology Discovery", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] | Split release: public seeds and targets; complete traces under restricted distribution. |
| Thomas et al., IEEE S&P 2015, Ad Injection at Scale [8Thomas, Kurt; Bursztein, Elie; Grier, Chris; Ho, Grant; Jagpal, Nav; Kapravelos, Alexandros; McCoy, Damon; Nappa, Antonio; Paxson, Vern; Pearce, Paul; Provos, Niels; Rajab, Moheeb Abu (2015): "Ad Injection at Scale: Assessing Deceptive Advertisement Modifications", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)] | Operator data. Aggregates after a short lifetime; raw data never left Google. |
Google's Right to be Forgotten paper cannot publish the URL list it classified, because the list is the sensitive object [4Bertram, Theo; Bursztein, Elie; Caro, Stephanie; Chao, Hubert; Feman, Rutledge Chin; Fleischer, Peter; Gustafsson, Albin; Hemerly, Jess; Hibbert, Chris; Invernizzi, Luca; Donnelly, Lanah Kammourieh; Ketover, Jason; Laefer, Jay; Nicholas, Paul; Niu, Yuan; Obhi, Harjinder; Price, David; Strait, Andrew; Thomas, Kurt; Verney, Al (2019): "Five Years of the Right to be Forgotten", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)]. Lyons et al. logged identifiers off Android devices [3Lyons, Allan; Gamba, Julien; Shawaga, Austin; Reardon, Joel; Tapiador, Juan; Egelman, Serge; Vallina-Rodriguez, Narseo (2023): "Log: It's Big, It's Heavy, It's Filled with Personal Data! Measuring the Logging of Sensitive Information in the Android Ecosystem", in: Proceedings of the USENIX Security Symposium. (Link)]. Dey, Ding and Ross crawled Facebook profiles of high-school students [2Dey, Ratan; Ding, Yuan; Ross, Keith W. (2013): "Profiling high-school students with facebook: how online privacy laws can actually increase minors' risk", in: Proceedings of the ACM Internet Measurement Conference. (DOI)].
Operator NDAs and campus traffic. IMC is full of this: COVID-era mobile-operator traces, campus NetFlow, MNO handover logs, all under agreements that forbid re-sharing even for research. The paper can still release the analysis code, the schema, and synthetic or aggregated tables.
Dual-use. Kim et al. built LTEFuzz and wrote “A public release is not planned as LTEFuzz can be used for malicious purposes”; they planned a private release to carriers [5Kim, Hongil; Lee, Jiho; Lee, Eunkyu; Kim, Yongdae (2019): "Touching the Untouchables: Dynamic Security Analysis of the LTE Control Plane", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)]. That is a different withholding from PII. The harm is the capability, not the dataset. See Ethics on dual-use.
Restricted access is a release. Pastrana et al.'s CrimeBB is available to academic researchers from the Cambridge Cybercrime Centre, not on a public URL [6Pastrana, Sergio; Thomas, Daniel R.; Hutchings, Alice; Clayton, Richard (2018): "CrimeBB: Enabling Cybercrime Research on Underground Forums at Scale", in: Proceedings of the ACM Web Conference. (DOI)]. Beverly et al. published seeds and targets and held the complete IPv6 traces under restricted distribution [7Beverly, Robert; Durairajan, Ramakrishnan; Plonka, David; Rohrer, Justin P. (2018): "In the IP of the Beholder: Strategies for Active IPv6 Topology Discovery", in: Proceedings of the ACM Internet Measurement Conference. (DOI)]. Both count as an artifact in the schema (restricted / a dataset URL). A reviewer can still get the data. A bit.ly link to a Google Drive folder that you might delete is worse than a named restricted archive.
What you can always release, even when the raw crawl cannot go out:
- The URL list (or the Tranco id plus the filter), unless the URLs themselves are the sensitive object (delisting requests, victim sites, the high-school Facebook study).
- The crawler, the pins, the classifier, the analysis scripts.
- Aggregates and per-bin counts that do not identify a site or a person.
- A data dictionary of what you deleted.
Thomas et al. crawled ad injection from inside Google and wrote “never sharing raw data outside of Google” [8Thomas, Kurt; Bursztein, Elie; Grier, Chris; Ho, Grant; Jagpal, Nav; Kapravelos, Alexandros; McCoy, Damon; Nappa, Antonio; Paxson, Vern; Pearce, Paul; Provos, Niels; Rajab, Moheeb Abu (2015): "Ad Injection at Scale: Assessing Deceptive Advertisement Modifications", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)]. Only aggregates could be retained. That is a complete answer; “the data is sensitive” is not.
If you withhold, name the reason in the paper in one sentence that a shepherd can check: PII / NDA / dual-use / the URLs are the finding. Then say what you did release. USENIX 2026's appendix is a template for this even if you are not submitting there.
A check you can run
pin_artifact.py classifies a URL the way the 2026 Available calls do, and for a Zenodo record prints the version DOI, the concept DOI, the files and the licence. --require-doi exits 1 unless every URL is on a named archival host (Zenodo, OSF, Figshare, Dryad, Dataverse, 4TU, Software Heritage). A publisher DOI is not enough; a GitHub path that happens to contain 10.5281 is not Zenodo.
- pin_artifact.py
#!/usr/bin/env python3 """Is this artifact URL archival, or just a pointer at a mutable repo? USENIX Security 2026's Artifacts Available badge refuses GitHub, GitLab and personal pages: the archived copy has to be reachable via a long-term stable reference or DOI. IEEE S&P 2026 and TheWebConf 2026 say the same for Available. This script classifies a URL the way those calls do, and for a Zenodo record prints the version DOI, the concept DOI, the files and the licence — so a camera-ready statement can cite the record that was actually deposited, not the living GitHub default branch. A publisher DOI (ACM, IEEE, …) is not an archival artifact. --require-doi accepts only named archives (Zenodo, OSF, Figshare, Dryad, Dataverse, 4TU, Software Heritage) and their known DOI prefixes, not doi.org in general. uv run python pin_artifact.py https://doi.org/10.5281/zenodo.1421702 uv run python pin_artifact.py --require-doi https://github.com/mozilla/OpenWPM uv run python pin_artifact.py --file urls.txt Exit status is 1 if any URL is a shortener, is unparseable, is a Zenodo URL without a record id, or (with --require-doi) is not on a named archival host. No API key needed for Zenodo. """ from __future__ import annotations import argparse import json import re import sys import urllib.error import urllib.parse import urllib.request UA = "measuretheweb-pin-artifact/1.0 (https://measuretheweb.org/artifacts)" # Ordered: first match wins. Keep in sync with scripts/host_fold.mjs. # (family, permanence, host-or-path test) DOI_ARCHIVAL = { "10.5281": "Zenodo", "10.17605": "OSF", "10.6084": "Figshare", "10.5061": "Dryad", "10.7910": "Harvard Dataverse", "10.4121": "4TU.ResearchData", } ARCHIVAL_HOSTS = { "zenodo.org": "Zenodo", "osf.io": "OSF", "figshare.com": "Figshare", "datadryad.org": "Dryad", "dryad.org": "Dryad", "dataverse.harvard.edu": "Harvard Dataverse", "data.4tu.nl": "4TU.ResearchData", "archive.softwareheritage.org": "Software Heritage", } MUTABLE_HOSTS = { "github.com": "GitHub", "gist.github.com": "GitHub", "githubusercontent.com": "GitHub", "gitlab.com": "GitLab", "bitbucket.org": "Bitbucket", "huggingface.co": "Hugging Face", "drive.google.com": "Google Drive / Docs", "docs.google.com": "Google Drive / Docs", "dropbox.com": "Dropbox", } SHORTENERS = { "bit.ly", "tinyurl.com", "goo.gl", "t.ly", "ow.ly", "is.gd", "cutt.ly", "rebrand.ly", "shorturl.at", } DOI_HOSTS = {"doi.org", "dx.doi.org"} ZENODO_ID_RE = re.compile(r"(?:zenodo\.org/(?:records|record)/|10\.5281/zenodo\.)(\d+)", re.I) def parse(url: str) -> tuple[str, str, str]: raw = url.strip() if raw == "": raise ValueError("empty URL") m = re.match(r"^doi:\s*(10\.\S+)", raw, re.I) if m: return "doi.org", "/" + m.group(1), raw with_scheme = raw if re.match(r"^[a-z][a-z0-9+.-]*://", raw, re.I) else "https://" + raw parts = urllib.parse.urlparse(with_scheme.replace(" ", "")) host = parts.hostname if host is None or host == "": raise ValueError(f"no host in {raw!r}") if host.startswith("www."): host = host[4:] return host.lower(), parts.path, raw def doi_prefix(path: str) -> str | None: m = re.search(r"(10\.\d+)", path) if m is None: return None return m.group(1) def classify(url: str) -> tuple[str, str]: """Return (family, permanence). DOI-prefix matching applies only to doi.org / dx.doi.org / doi: URLs. A GitHub path that happens to contain 10.5281 is not Zenodo. A generic publisher DOI is not archival. """ host, path, raw = parse(url) if host in SHORTENERS: return "URL shortener", "shortener" is_doi = host in DOI_HOSTS if is_doi: pref = doi_prefix(path) or doi_prefix(raw) if pref is not None and pref in DOI_ARCHIVAL: return DOI_ARCHIVAL[pref], "archival" if pref == "10.48550": return "arXiv", "publisher-page" return "DOI resolver (unknown archive)", "unknown" if host in ARCHIVAL_HOSTS: family = ARCHIVAL_HOSTS[host] if family == "Zenodo" and zenodo_id(raw) is None: return "Zenodo (no record id)", "unknown" return family, "archival" if host.endswith(".osf.io"): return "OSF", "archival" if host.endswith(".figshare.com"): return "Figshare", "archival" if host.endswith(".github.io") or host.endswith(".githubusercontent.com"): return "GitHub", "mutable-repo" if host in MUTABLE_HOSTS: return MUTABLE_HOSTS[host], "mutable-repo" if host.endswith(".dropbox.com"): return "Dropbox", "mutable-repo" if host == "sites.google.com": return "Google Sites (project page)", "project-page" if host == "anonymous.4open.science": return "anonymous.4open.science (double-blind GitHub)", "anonymized" if host == "arxiv.org": return "arXiv", "publisher-page" if host == "usenix.org" or host.endswith(".usenix.org"): return "USENIX paper page", "publisher-page" return f"unmapped host {host}", "unknown" def get_json(url: str) -> dict: req = urllib.request.Request(url, headers={"Accept": "application/json", "User-Agent": UA}) with urllib.request.urlopen(req, timeout=30) as r: return json.load(r) def zenodo_id(url: str) -> str | None: m = ZENODO_ID_RE.search(url) if m is None: return None return m.group(1) def describe_zenodo(url: str) -> str: rec_id = zenodo_id(url) if rec_id is None: return "could not parse a Zenodo record id" try: rec = get_json(f"https://zenodo.org/api/records/{rec_id}") except urllib.error.HTTPError as e: return f"Zenodo API HTTP {e.code} for record {rec_id}" md = rec["metadata"] files = rec["files"] version_doi = md["doi"] concept = md["relations"]["version"][0]["parent"]["pid_value"] created = rec["created"][:10] license_id = md["license"]["id"] titles = ", ".join(f["key"] + f" ({f['size']} bytes)" for f in files[:8]) more = "" if len(files) <= 8 else f" … +{len(files) - 8} files" return ( f"record {rec_id}; version DOI 10.5281/zenodo.{rec_id} ({version_doi}); " f"concept DOI 10.5281/zenodo.{concept}; created {created}; " f"licence {license_id}; {len(files)} files: {titles}{more}" ) def main() -> int: ap = argparse.ArgumentParser() ap.add_argument("urls", nargs="*") ap.add_argument("--file", help="one URL per line") ap.add_argument( "--require-doi", action="store_true", help="exit 1 unless every URL is on a named archival host (Zenodo/OSF/Figshare/Dryad/Dataverse/4TU/Software Heritage). A publisher DOI is not enough.", ) args = ap.parse_args() urls = list(args.urls) if args.file: urls += [line.strip() for line in open(args.file) if line.strip() != ""] if len(urls) == 0: ap.error("give at least one URL") bad = 0 for url in urls: try: family, permanence = classify(url) except ValueError as e: print(f"{'UNPARSEABLE':<12} {url} ({e})") bad += 1 continue extra = "" if family == "Zenodo": desc = describe_zenodo(url) extra = " | " + desc if desc.startswith("could not parse") or desc.startswith("Zenodo API HTTP"): bad += 1 print(f"{permanence:<12} {family:<44} {url}{extra}") if permanence == "shortener": bad += 1 if permanence == "unknown" and family.startswith("Zenodo"): bad += 1 if args.require_doi and permanence != "archival": bad += 1 return 1 if bad else 0 if __name__ == "__main__": raise SystemExit(main())
Live run, 2026-08-27. The Zenodo record is an arbitrary deposited snapshot, used here to show the API fields, not as a measurement artefact to imitate. OpenWPM's GitHub URL is the counterexample: it is the right development URL and it fails an Available check.
uv run python pin_artifact.py https://github.com/mozilla/OpenWPM https://doi.org/10.5281/zenodo.1421702
mutable-repo GitHub https://github.com/mozilla/OpenWPM archival Zenodo https://doi.org/10.5281/zenodo.1421702 | record 1421702; version DOI 10.5281/zenodo.1421702 (10.5281/zenodo.1421702); concept DOI 10.5281/zenodo.1421701; created 2018-09-19; licence other-open; 1 files: Eichhoernchen/monero-digging-paper-v1.zip (20788 bytes)
uv run python pin_artifact.py –require-doi https://github.com/mozilla/OpenWPM (exits 1)
mutable-repo GitHub https://github.com/mozilla/OpenWPM
Cite OpenWPM itself as [9Englehardt, Steven; Narayanan, Arvind (2016): "Online Tracking: A 1-million-site Measurement and Analysis", in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1388–1401. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)]; the GitHub URL above is the project, not a 2016 snapshot.
A reporting checklist
- A DOI, in the paper, next to the GitHub URL. The GitHub URL is for issues. The DOI is the citation.
- The version DOI, not the concept DOI, if you want the reader to get this snapshot.
- The list, the pins, the per-site table, not only the crawler.
- The licence on the deposit. Zenodo will not let you mint a DOI without one; GitHub will.
- If you withheld: the reason, and what you released instead.
- No shortener. 59 papers in this corpus did that.
- No Google Drive / Dropbox as the archival copy. 41 + 13 papers. Fine as a working folder; not a camera-ready artifact.
- anonymous.4open.science is for review. Replace it.
Related pages
- Study preregistration — a badge does not pin your analysis plan.
- Ethics — the review and the harm analysis that decide whether you can release.
- Website selection — publish the list you actually fetched.
- Sampling / Longitudinal — the pins a later wave needs.
- Crawler — the browser binary belongs in the artefact.
- Hypothesis testing / Pvalue corrections — per-site outcomes are what make a re-analysis possible.
- Corpus — seven venues, 2010–2026, 5,859 papers.
Methodology and limitations of these figures
Every corpus figure above comes from scripts/report_artifacts.mjs over data/extract/run1 (5,859 papers). Limits that carry:
- Seven venues only. CHI, SOUPS, ACSAC, EuroS&P, RAID, AsiaCCS are absent. Artifact-evaluation culture at those venues is not in these rates.
- “Own-link” is a URL the extractor stored as the authors'. A paper that says “code available on request” with no URL is not in the 3,321. A paper that prints someone else's GitHub is not either (
belongsToAuthors === false). - Hosts are folded. GitHub includes gist, raw.githubusercontent.com and
*.github.io. Zenodo includesdoi.org/10.5281and the unschemeddoi:10.5281/zenodo.N. The residue (667 hosts) is on the provenance page. Exact-string “github.com” undercounts. artifacts.badgeis not a badge-award rate. 15 papers. Use the venue AE results.- 2025–2026 are provisional. Own-link 76.9% / 75.9% in those years will move when the missing venue-years land.
data/extract/artifact_links.jsonlis the old 4,322-paper corpus. This page does not use it. OVERVIEW.md still points at it; it is stale.
The complete query log, the report with its unedited output, the host fold and its residue, every quote with its verification verdict, and every external source with how it was verified are on artifacts. Corpus-wide caveats are on Corpus.
References
- [1]
- Demir, Nurullah; Große-Kampmann, Matteo; Urban, Tobias; Wressnegger, Christian; Holz, Thorsten; Pohlmann, Norbert (2022): "Reproducibility and Replicability of Web Measurement Studies", in: Proceedings of the ACM Web Conference. (DOI)
- [2]
- Dey, Ratan; Ding, Yuan; Ross, Keith W. (2013): "Profiling high-school students with facebook: how online privacy laws can actually increase minors' risk", in: Proceedings of the ACM Internet Measurement Conference. (DOI)
- [3]
- Lyons, Allan; Gamba, Julien; Shawaga, Austin; Reardon, Joel; Tapiador, Juan; Egelman, Serge; Vallina-Rodriguez, Narseo (2023): "Log: It's Big, It's Heavy, It's Filled with Personal Data! Measuring the Logging of Sensitive Information in the Android Ecosystem", in: Proceedings of the USENIX Security Symposium. (Link)
- [4]
- Bertram, Theo; Bursztein, Elie; Caro, Stephanie; Chao, Hubert; Feman, Rutledge Chin; Fleischer, Peter; Gustafsson, Albin; Hemerly, Jess; Hibbert, Chris; Invernizzi, Luca; Donnelly, Lanah Kammourieh; Ketover, Jason; Laefer, Jay; Nicholas, Paul; Niu, Yuan; Obhi, Harjinder; Price, David; Strait, Andrew; Thomas, Kurt; Verney, Al (2019): "Five Years of the Right to be Forgotten", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)
- [5]
- Kim, Hongil; Lee, Jiho; Lee, Eunkyu; Kim, Yongdae (2019): "Touching the Untouchables: Dynamic Security Analysis of the LTE Control Plane", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)
- [6]
- Pastrana, Sergio; Thomas, Daniel R.; Hutchings, Alice; Clayton, Richard (2018): "CrimeBB: Enabling Cybercrime Research on Underground Forums at Scale", in: Proceedings of the ACM Web Conference. (DOI)
- [7]
- Beverly, Robert; Durairajan, Ramakrishnan; Plonka, David; Rohrer, Justin P. (2018): "In the IP of the Beholder: Strategies for Active IPv6 Topology Discovery", in: Proceedings of the ACM Internet Measurement Conference. (DOI)
- [8]
- Thomas, Kurt; Bursztein, Elie; Grier, Chris; Ho, Grant; Jagpal, Nav; Kapravelos, Alexandros; McCoy, Damon; Nappa, Antonio; Paxson, Vern; Pearce, Paul; Provos, Niels; Rajab, Moheeb Abu (2015): "Ad Injection at Scale: Assessing Deceptive Advertisement Modifications", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)
- [9]
- Englehardt, Steven; Narayanan, Arvind (2016): "Online Tracking: A 1-million-site Measurement and Analysis", in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1388–1401. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)
