Measure The Web

User Tools

Site Tools

Trace: • contributing
artifacts

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
artifacts [2026/08/27 13:19] – Create Artifacts: 3321/5859 own-link (56.7%), GitHub vs Zenodo, 2026 AE table, pin_artifact.py. Authored by Claude. karel.kubicek.claudeartifacts [2026/08/27 13:37] (current) – Generic-review fixes: host_fold DOI-prefix, pin_artifact record paths, regenerated Zenodo 431/617, causal wording. Authored by Claude. karel.kubicek.claude
Line 6: Line 6:
 **A GitHub URL is not an archival artifact.** USENIX Security 2026's Artifacts Available badge — mandatory for every accepted paper — refuses GitHub, GitLab and personal pages. The archived copy has to be reachable via a long-term stable reference or DOI. IEEE S&P 2026 and TheWebConf 2026 say the same for Available. Zenodo is the host those calls name first.((USENIX Security 2026 badges, [[https://secartifacts.github.io/usenixsec2026/badges|secartifacts.github.io/usenixsec2026/badges]], read 2026-08-27: //"software development repositories such as GitHub, GitLab, or personal web pages are not acceptable for this badge"//; Zenodo recommended, FigShare, Dryad and Software Heritage named. IEEE S&P 2026 [[https://sp2026.ieee-security.org/cfartifacts.html|cfartifacts.html]], same day: Available requires a DOI-backed deposit. TheWebConf 2026 [[https://www2026.thewebconf.org/calls/artifact-badging.html|artifact-badging.html]], same day: DOI required; GitHub is allowed //alongside// the DOI, not instead of it.)) **A GitHub URL is not an archival artifact.** USENIX Security 2026's Artifacts Available badge — mandatory for every accepted paper — refuses GitHub, GitLab and personal pages. The archived copy has to be reachable via a long-term stable reference or DOI. IEEE S&P 2026 and TheWebConf 2026 say the same for Available. Zenodo is the host those calls name first.((USENIX Security 2026 badges, [[https://secartifacts.github.io/usenixsec2026/badges|secartifacts.github.io/usenixsec2026/badges]], read 2026-08-27: //"software development repositories such as GitHub, GitLab, or personal web pages are not acceptable for this badge"//; Zenodo recommended, FigShare, Dryad and Software Heritage named. IEEE S&P 2026 [[https://sp2026.ieee-security.org/cfartifacts.html|cfartifacts.html]], same day: Available requires a DOI-backed deposit. TheWebConf 2026 [[https://www2026.thewebconf.org/calls/artifact-badging.html|artifact-badging.html]], same day: DOI required; GitHub is allowed //alongside// the DOI, not instead of it.))
  
-Of 5,859 papers in this corpus (seven venues, 2010–2026), **3,321 (56.7%) release at least one authors'-own artifact URL.** That rate rose **23.7% (2010–2013) → 65.0% (2022–2024)**; 2024 itself is 457 of 690 (66.2%). Among those 3,321, **GitHub still hosts 1,896 (57.1%)** and **an archival host (Zenodo, OSF, Figshare, Dryad, Dataverse, 4TU, Software Heritage, or another DOI) hosts 622 (18.7%)**. Zenodo among own-link papers was 42 of 457 in 2024 (9.2%) and **228 of 592 in 2025 (38.5%)** — the jump is USENIX's 2025–2026 open-science policy, under which 219 of 293 USENIX papers that year have a Zenodo link (74.7%). 2025–2026 are provisional venue-years; do not read the last column as a completed year.+Of 5,859 papers in this corpus (seven venues, 2010–2026), **3,321 (56.7%) release at least one authors'-own artifact URL.** That rate rose **23.7% (2010–2013) → 65.0% (2022–2024)**; 2024 itself is 457 of 690 (66.2%). Among those 3,321, **GitHub still hosts 1,896 (57.1%)** and **a named archival host or a DOI resolver hosts 617 (18.6%)** — Zenodo, OSF, Figshare, Dryad, Dataverse, 4TU, Software Heritage, plus 34 other-DOI URLs whose prefix was not one of those (a resolver, not a verified archive). Zenodo among own-link papers was 41 of 457 in 2024 (9.0%) and **224 of 592 in 2025 (37.8%)** — that jump **coincides with** USENIX's 2025–2026 open-science policy, under which 217 of 293 USENIX papers in 2025–2026 have a Zenodo link (74.1%). 2025–2026 are provisional venue-years; do not read the last column as a completed year.
 </WRAP> </WRAP>
  
Line 25: Line 25:
 ===== What to put in it ===== ===== What to put in it =====
  
-A measurement artefact is not "the code". The papers that cannot be re-run are missing the **sample**, the **pins**, and the **per-site outcomes**. Demir et al. {[demir2022_reproducibility]} coded 18 criteria; the ones that fail in this literature are configuration details, not the existence of a repository.+A measurement artefact is not "the code". The papers that cannot be re-run are missing the **sample**, the **pins**, and the **per-site outcomes**. Demir et al. {[demir2022_reproducibility]} coded 18 criteria; configuration details fail often (crawler adjustments, location, browser), and so does the repository: C9 "crawler publicly available" was omitted by 67% of their 117 papers, and only 24% made results openly available.
  
 For a crawl, the minimum that lets someone else regenerate your headline number — or re-analyse it with a clustered standard error, see [[Statistics:Hypothesis testing]] — is: For a crawl, the minimum that lets someone else regenerate your headline number — or re-analyse it with a clustered standard error, see [[Statistics:Hypothesis testing]] — is:
  
 ^ Piece ^ Why it is not optional ^ ^ Piece ^ Why it is not optional ^
-| **The URL list, as fetched.** Rank, domain, fetch date. A Tranco id if you used one ([[Programming:Tranco]]); the exact rows if you filtered. "The top 10k" is not a sample. | [[Design:Website selection]], [[Design:Sampling]] | +| **The URL list, as fetched.** Rank, domain, fetch date. A Tranco id if you used one ([[Programming:Tranco]]); the exact rows if you filtered. "The top 10k" is not a sample. | Without it nobody can reconstruct the population. | 
-| **The crawler pin.** Library version, browser build (not "Chrome"), headless/headful, statefulness, interaction depth, consent action. | [[Programming:Crawler]], [[Programming:Stateful stateless]], [[Privacy:Consent]] | +| **The crawler pin.** Library version, browser build (not "Chrome"), headless/headful, statefulness, interaction depth, consent action. | The number moves when these move ([[Programming:Crawler]], [[Programming:Stateful stateless]], [[Privacy:Consent]]). | 
-| **The vantage point.** Provider, city, ASN if you know it. | [[Design:Crawling location]] | +| **The vantage point.** Provider, city, ASN if you know it. | Geo and CDN splits are vantage-dependent ([[Design:Crawling location]]). | 
-| **The classifier, with its version.** EasyList commit, Disconnect snapshot, your model weights, the threshold. | [[Privacy:Requests]], [[Design:Website classification]] | +| **The classifier, with its version.** EasyList commit, Disconnect snapshot, your model weights, the threshold. | A later EasyList is a different experiment ([[Privacy:Requests]], [[Design:Website classification]]). | 
-| **Per-site outcomes**, not only aggregates. One row per site: loaded or not, the labels, the counts. This is what makes a cluster-aware re-analysis possible. | [[Statistics:Hypothesis testing]], [[Statistics:Regression]] | +| **Per-site outcomes**, not only aggregates. One row per site: loaded or not, the labels, the counts. | Makes a cluster-aware re-analysis possible ([[Statistics:Hypothesis testing]], [[Statistics:Regression]]). Does not by itself reproduce the paper's aggregation. | 
-| **What you deleted, and why.** Raw cookies, request bodies, identifiers. A README that says "we stripped PII" is not a schema. | this page, [[Practices:Ethics]] | +| **The analysis scripts and the environment pin.** The command that produced the table; a lockfile or image digest. | Per-site outcomes without the reducer are a dataset, not a regenerated headline ([[Programming:Docker]]). | 
-| **A licence, and a DOI.** GitHub's default branch has neither. | venue table below |+| **What you deleted, and why.** Raw cookies, request bodies, identifiers. A README that says "we stripped PII" is not a schema. | Otherwise a re-run cannot tell missingness from redaction ([[Practices:Ethics]]). | 
 +| **A licence, and a DOI.** GitHub's default branch is not a frozen snapshot and has no DOI. A LICENSE file on the repo is not an archive. | Venue table below |
  
 A system paper that evaluated a defence on a testbed has a different minimum (the implementation, the configs, the test vectors). This page is for the student who crawled. A system paper that evaluated a defence on a testbed has a different minimum (the implementation, the configs, the test vectors). This page is for the student who crawled.
Line 87: Line 88:
 ^ Family ^ Permanence ^ Papers ^ Share of 3321 ^ ^ Family ^ Permanence ^ Papers ^ Share of 3321 ^
 | GitHub | mutable-repo | 1896 | 57.1% | | GitHub | mutable-repo | 1896 | 57.1% |
-| Zenodo | archival | 436 | 13.1% |+| Zenodo | archival | 431 | 13.0% |
 | Google Sites (project page) | project-page | 157 | 4.7% | | Google Sites (project page) | project-page | 157 | 4.7% |
 | OSF | archival | 112 | 3.4% | | OSF | archival | 112 | 3.4% |
Line 100: Line 101:
 | GitLab | mutable-repo | 25 | 0.8% | | GitLab | mutable-repo | 25 | 0.8% |
  
-Union of archival families: **622 / 3,321 = 18.7%**. **59 papers (1.8%) give a shortener as the artifact URL** (bit.ly, tinyurl, goo.gl). That URL is not an artifact; it is a redirect the owner can retarget. 667 distinct hosts did not match a family — almost all lab and university project pages. GitHub remains the default; the archival hosts are the minority, and they are new.+Union of archival families: **617 / 3,321 = 18.6%**. **59 papers (1.8%) give a shortener as the artifact URL** (bit.ly, tinyurl, goo.gl). That URL is not an artifact; it is a redirect the owner can retarget. 669 distinct hosts did not match a family. High-count residue hosts are mostly lab and university project pages; the tail also includes pastebins, form hosts, and registries (full list on the provenance page). GitHub remains the default; the archival hosts are the minority, and they are new.
  
 ==== Over time ==== ==== Over time ====
Line 121: Line 122:
 | 2022 | 546 | 338 | 61.9% | 234 | 69.2% | 5 | 1.5% | | 2022 | 546 | 338 | 61.9% | 234 | 69.2% | 5 | 1.5% |
 | 2023 | 719 | 475 | 66.1% | 321 | 67.6% | 21 | 4.4% | | 2023 | 719 | 475 | 66.1% | 321 | 67.6% | 21 | 4.4% |
-| 2024 | 690 | 457 | 66.2% | 307 | 67.2% | 42 | 9.2% | +| 2024 | 690 | 457 | 66.2% | 307 | 67.2% | 41 | 9.0% | 
-| 2025* | 770 | 592 | 76.9% | 318 | 53.7% | 228 | 38.5% |+| 2025* | 770 | 592 | 76.9% | 318 | 53.7% | 224 | 37.8% |
 | 2026* | 415 | 315 | 75.9% | 202 | 64.1% | 132 | 41.9% | | 2026* | 415 | 315 | 75.9% | 202 | 64.1% | 132 | 41.9% |
  
Line 132: Line 133:
 | 2025–2026* | 1185 | 907 | 76.5% | | 2025–2026* | 1185 | 907 | 76.5% |
  
-GitHub's share of own-link papers peaked in 2022 (69.2%) and dropped in 2025 (53.7%) as Zenodo rose. That is a venue-policy effect, not a GitHub decline in absolute terms: 2025 still has 318 GitHub papers, more than 2022's 234.+GitHub's share of own-link papers peaked in 2022 (69.2%) and dropped in 2025 (53.7%) as Zenodo rose. That **coincides with** the USENIX 2025–2026 policy change; it is not a GitHub decline in absolute terms: 2025 still has 318 GitHub papers, more than 2022's 234.
  
 ==== By venue ==== ==== By venue ====
Line 138: Line 139:
 ^ Venue ^ Papers ^ Own-link ^ Share ^ GitHub of own-link ^ Zenodo of own-link ^ ^ Venue ^ Papers ^ Own-link ^ Share ^ GitHub of own-link ^ Zenodo of own-link ^
 | CCS | 990 | 467 | 47.2% | 252 | 25 | | CCS | 990 | 467 | 47.2% | 252 | 25 |
-| IEEE-SP | 767 | 423 | 55.1% | 270 | 12 |+| IEEE-SP | 767 | 423 | 55.1% | 270 | 11 |
 | IMC | 638 | 313 | 49.1% | 163 | 12 | | IMC | 638 | 313 | 49.1% | 163 | 12 |
-| NDSS | 701 | 443 | 63.2% | 310 | 102 |+| NDSS | 701 | 443 | 63.2% | 310 | 100 |
 | PETS | 510 | 258 | 50.6% | 158 | 5 | | PETS | 510 | 258 | 50.6% | 158 | 5 |
-| USENIX | 1410 | 980 | 69.5% | 460 | 233 |+| USENIX | 1410 | 980 | 69.5% | 460 | 231 |
 | WWW | 843 | 437 | 51.8% | 283 | 47 | | WWW | 843 | 437 | 51.8% | 283 | 47 |
  
-USENIX is the venue that currently **requires** an open-science appendix, and it is the venue with the highest own-link rate (69.5%) and almost all of the Zenodo mass (233 of 436). PETS, which has run optional artifact evaluation for years, has **5** Zenodo papers. Optional AE does not produce archival deposits; a mandatory Available badge does.+USENIX is the venue that currently **requires** an open-science appendix, and it is the venue with the highest own-link rate (69.5%) and more than half of the Zenodo mass (231 of 431). PETS, which has run optional artifact evaluation for years, has **5** Zenodo papers. Optional AE is not the same mechanism as a mandatory Available badge; the rates should not be read as "optional AE does not work."
  
 ===== Venue artifact evaluation, today ===== ===== Venue artifact evaluation, today =====
Line 151: Line 152:
 Checked 2026-08-27 against each call. Training data is stale here by construction. ''scripts/external_checks_artifacts.sh'' re-fetches these pages. Checked 2026-08-27 against each call. Training data is stale here by construction. ''scripts/external_checks_artifacts.sh'' re-fetches these pages.
  
-^ Venue ^ 2026 rule ^ Badges ^ GitHub enough for Available? ^ +^ Venue ^ Live rule (read 2026-08-27) ^ Badges ^ GitHub enough for Available? ^ 
-| **USENIX Security** | Open-science appendix mandatory at submission; Available re-verified after acceptance. Functional / Reproduced optional, later. | Available (mandatory), Functional, Reproduced | **No.** DOI-backed archive required. |+| **USENIX Security** | Open-science appendix mandatory at submission; Available re-verified after acceptance. Functional / Reproduced optional, later. | Available (mandatory), Functional, Reproduced | **No.** Long-term stable reference or DOI required (GitHub/GitLab/personal pages refused). |
 | **IEEE S&P** | Optional, post-acceptance. | Available, Functional, Reproduced | **No** for Available. DOI-backed (Zenodo / FigShare / Dryad). GitHub allowed as an extra pointer. | | **IEEE S&P** | Optional, post-acceptance. | Available, Functional, Reproduced | **No** for Available. DOI-backed (Zenodo / FigShare / Dryad). GitHub allowed as an extra pointer. |
-| **CCS** | Open-science appendix mandatory. Optional AE after acceptance, ACM badges. | Available, Evaluated (Functional or Reusable), Results Reproduced — ACM Artifact Review and Badging v1.1 | **No** for Available. ACM's Available badge is an archival deposit. |+| **CCS** | Optional AE after acceptance, ACM badges. Artifact Appendix encouraged after evaluation, not a mandatory submission appendix. | Available, Evaluated (Functional or Reusable), Results Reproduced — ACM Artifact Review and Badging v1.1 | **No** for Available. ACM's Available badge is an archival deposit. |
 | **NDSS** | Optional, post-acceptance. 2026 cycle closed: **114** artifacts evaluated, **112** Available, **97** Functional, **70** Reproduced, 3 Distinguished Artifact Awards. | Available, Functional, Reproduced | Check the live call. NDSS 2026 used the same three-badge vocabulary. | | **NDSS** | Optional, post-acceptance. 2026 cycle closed: **114** artifacts evaluated, **112** Available, **97** Functional, **70** Reproduced, 3 Distinguished Artifact Awards. | Available, Functional, Reproduced | Check the live call. NDSS 2026 used the same three-badge vocabulary. |
 | **TheWebConf** | Optional, camera-ready. Light review: the artifact exists, is downloadable, has instructions. | **Available only** | **No.** DOI required. GitHub may sit next to the DOI in the resource-availability statement. | | **TheWebConf** | Optional, camera-ready. Light review: the artifact exists, is downloadable, has instructions. | **Available only** | **No.** DOI required. GitHub may sit next to the DOI in the resource-availability statement. |
 | **IMC** | Artifact-availability **declaration** at submission (full / partial / none). Accepted papers shepherded to deliver what they promised. Community Contribution Award requires public data or code by camera-ready. Separate Replicability Track. | No ACM-style badge set on the 2026 CFP. The declaration is the mechanism. | Not a badge question. A GitHub repo can satisfy "publicly available" for the award; pin a commit and also deposit. | | **IMC** | Artifact-availability **declaration** at submission (full / partial / none). Accepted papers shepherded to deliver what they promised. Community Contribution Award requires public data or code by camera-ready. Separate Replicability Track. | No ACM-style badge set on the 2026 CFP. The declaration is the mechanism. | Not a badge question. A GitHub repo can satisfy "publicly available" for the award; pin a commit and also deposit. |
-| **PETS / PoPETs** | Optional. Available / Functional / Reproduced. PETS 2026 Artifact Award already announced. | Available, Functional, Reproduced | Optional means you can skip it. If you want Available, deposit. |+| **PETS / PoPETs** | Optional. Available / Functional / Reproduced. Live page (read 2026-08-27) is titled PoPETs **2027** AE; PETS 2026 Artifact Award already announced. | Available, Functional, Reproduced | Optional means you can skip it. If you want Available, deposit. |
  
 The schema's ''artifacts.badge'' field is non-null for **15 of 5,859 papers (0.3%)**. That is not a badge-award rate. Papers almost never write the badge name in a sentence the extractor can attach. A full-text sweep for "artifact evaluation" hits 59 papers — still an upper bound, still not an award rate. **Do not use this corpus to say how often a venue awards badges.** Use the venue's own AE results page, as NDSS 2026 publishes. The schema's ''artifacts.badge'' field is non-null for **15 of 5,859 papers (0.3%)**. That is not a badge-award rate. Papers almost never write the badge name in a sentence the extractor can attach. A full-text sweep for "artifact evaluation" hits 59 papers — still an upper bound, still not an award rate. **Do not use this corpus to say how often a venue awards badges.** Use the venue's own AE results page, as NDSS 2026 publishes.
Line 164: Line 165:
 ===== GitHub vs a DOI ===== ===== GitHub vs a DOI =====
  
-GitHub is the right place to **develop**. It is the wrong place to **cite**. The default branch moves; a force-push rewrites history; a renamed account 404s the camera-ready URL. The 2026 Available calls noticed.+GitHub is the right place to **develop**. It is the wrong place to **satisfy an Available badge**. The default branch moves; a force-push rewrites history; a renamed account 404s the camera-ready URL. A commit or tag permalink is a citable pointer at source; it is still not the archival deposit the 2026 Available calls ask for.
  
 The pattern that satisfies every row of the table above: The pattern that satisfies every row of the table above:
Line 173: Line 174:
   - Leave the GitHub URL in the README so people can file issues. It is a pointer, not the archive.   - Leave the GitHub URL in the README so people can file issues. It is a pointer, not the archive.
  
-OSF, Figshare, Dryad, institutional Dataverse, 4TU.ResearchData and Software Heritage also issue DOIs. Zenodo is the one the security AE pages name first, and it is the one 436 papers in this corpus already use.+OSF, Figshare, Dryad, institutional Dataverse, 4TU.ResearchData and Software Heritage also issue DOIs. Zenodo is the one the security AE pages name first, and it is the one 431 papers in this corpus already use.
  
 ''anonymous.4open.science'' (36 papers) is a double-blind stand-in for GitHub during review. It is not the camera-ready archive. Swap it for a DOI before the Available check. ''anonymous.4open.science'' (36 papers) is a double-blind stand-in for GitHub during review. It is not the camera-ready archive. Swap it for a DOI before the Available check.
Line 185: Line 186:
 ^ Paper ^ Why the artifact is not public ^ ^ Paper ^ Why the artifact is not public ^
 | Dey et al., IMC 2013, //Profiling high-school students with Facebook// {[dey2013_profiling]} | PII of minors. //"we will not be making our data sets public and we will not explicitly identify the high schools involved"// | | Dey et al., IMC 2013, //Profiling high-school students with Facebook// {[dey2013_profiling]} | PII of minors. //"we will not be making our data sets public and we will not explicitly identify the high schools involved"// |
-| Lyons et al., USENIX Security 2023, //Log: It's Big, It's Heavy, It's Filled with Personal Data!// {[lyons2023_heavy]} | Device identifiers. //"We are not releasing our data."// | +| Lyons et al., USENIX Security 2023, //Log: It's Big, It's Heavy, It's Filled with Personal Data!// {[lyons2023_heavy]} | Re-identification. //"unique combinations of app names in conjunction with data from other sources could still identify participants, which is why we are not releasing our data."// | 
-| Bertram et al., CCS 2019, //Five Years of the Right to be Forgotten// {[bertram2019_five]} | The URL list is the sensitive object. //"We cannot directly reveal a sample mapping between URLs and our annotations"// |+| Bertram et al., CCS 2019, //Five Years of the Right to be Forgotten// {[bertram2019_five]} | The URL list is the sensitive object. //"We cannot directly reveal a sample mapping"// |
 | Kim et al., IEEE S&P 2019, //Touching the Untouchables// {[kim2019_touching]} | Dual-use. //"A public release is not planned as LTEFuzz can be used for malicious purposes"// | | Kim et al., IEEE S&P 2019, //Touching the Untouchables// {[kim2019_touching]} | Dual-use. //"A public release is not planned as LTEFuzz can be used for malicious purposes"// |
 | Pastrana et al., TheWebConf 2018, //CrimeBB// {[pastrana2018_crimebb]} | Restricted-access release, not a public URL. Available to academic researchers from the Cambridge Cybercrime Centre. | | Pastrana et al., TheWebConf 2018, //CrimeBB// {[pastrana2018_crimebb]} | Restricted-access release, not a public URL. Available to academic researchers from the Cambridge Cybercrime Centre. |
Line 200: Line 201:
 **Restricted access is a release.** Pastrana et al.'s CrimeBB is available to academic researchers from the Cambridge Cybercrime Centre, not on a public URL {[pastrana2018_crimebb]}. Beverly et al. published seeds and targets and held the complete IPv6 traces under restricted distribution {[beverly2018_beholder]}. Both count as an artifact in the schema (''restricted'' / a dataset URL). A reviewer can still get the data. A bit.ly link to a Google Drive folder that you might delete is worse than a named restricted archive. **Restricted access is a release.** Pastrana et al.'s CrimeBB is available to academic researchers from the Cambridge Cybercrime Centre, not on a public URL {[pastrana2018_crimebb]}. Beverly et al. published seeds and targets and held the complete IPv6 traces under restricted distribution {[beverly2018_beholder]}. Both count as an artifact in the schema (''restricted'' / a dataset URL). A reviewer can still get the data. A bit.ly link to a Google Drive folder that you might delete is worse than a named restricted archive.
  
-**What you can always release**, even when the raw crawl cannot go out:+**What you can often still release**, even when the raw crawl cannot go out. Dual-use, an NDA, or copyright can still block the crawler or the classifier — Kim et al. withheld LTEFuzz itself {[kim2019_touching]}. For each withheld piece, say so.
  
   * The URL list (or the Tranco id plus the filter), unless the URLs themselves are the sensitive object (delisting requests, victim sites, the high-school Facebook study).   * The URL list (or the Tranco id plus the filter), unless the URLs themselves are the sensitive object (delisting requests, victim sites, the high-school Facebook study).
-  * The crawler, the pins, the classifier, the analysis scripts.+  * The crawler, the pins, the classifier, the analysis scripts — unless the capability is the harm.
   * Aggregates and per-bin counts that do not identify a site or a person.   * Aggregates and per-bin counts that do not identify a site or a person.
   * A data dictionary of what you deleted.   * A data dictionary of what you deleted.
  
-Thomas et al. crawled ad injection from inside Google and wrote //"never sharing raw data outside of Google; and setting a short lifetime on the data collected after which only aggregates could be stored"// {[thomas2015_injection]}. The aggregates were the artifact. That is a complete answer; "the data is sensitive" is not.+Thomas et al. crawled ad injection from inside Google and wrote //"never sharing raw data outside of Google"// {[thomas2015_injection]}. Only aggregates could be retained. That is a complete answer; "the data is sensitive" is not.
  
 <WRAP todo> <WRAP todo>
Line 215: Line 216:
 ===== A check you can run ===== ===== A check you can run =====
  
-''pin_artifact.py'' classifies a URL the way the 2026 Available calls do, and for a Zenodo record prints the version DOI, the concept DOI, the files and the licence. ''%%--require-doi%%'' exits 1 unless every URL is archival — gate a camera-ready checklist on it.+''pin_artifact.py'' classifies a URL the way the 2026 Available calls do, and for a Zenodo record prints the version DOI, the concept DOI, the files and the licence. ''%%--require-doi%%'' exits 1 unless every URL is on a **named archival host with a record path** (Zenodo, OSF, Figshare, Dryad, Dataverse, 4TU, Software Heritage). A publisher DOI is not enough; ''osf.io/'' with no record id is not enough; a GitHub path that happens to contain ''10.5281'' is not Zenodo.
  
 <file python pin_artifact.py> <file python pin_artifact.py>
Line 229: Line 230:
 the living GitHub default branch. the living GitHub default branch.
  
-    python3 pin_artifact.py https://doi.org/10.5281/zenodo.1421702 +A publisher DOI (ACM, IEEE, …) is not an archival artifact. --require-doi 
-    python3 pin_artifact.py https://github.com/mozilla/OpenWPM --require-doi +accepts only named archives (Zenodo, OSF, Figshare, Dryad, Dataverse, 4TU, 
-    python3 pin_artifact.py --file urls.txt+Software Heritage) and their known DOI prefixes, not doi.org in general.
  
-Exit status is 1 if any URL is a shortener, is unparseable, or (with +    uv run python pin_artifact.py https://doi.org/10.5281/zenodo.1421702 
---require-doi) is not on an archival host. No API key needed for Zenodo.+    uv run python pin_artifact.py --require-doi https://github.com/mozilla/OpenWPM 
 +    uv run python pin_artifact.py --file urls.txt 
 + 
 +Exit status is 1 if any URL is a shortener, is unparseable, is a Zenodo URL 
 +without a record id, or (with --require-doi) is not on a named archival host. 
 +No API key needed for Zenodo.
 """ """
 from __future__ import annotations from __future__ import annotations
Line 282: Line 288:
 } }
  
-SHORTENERS = {"bit.ly", "tinyurl.com", "goo.gl", "t.ly", "ow.ly", "is.gd", "cutt.ly"}+SHORTENERS = { 
 +    "bit.ly", 
 +    "tinyurl.com", 
 +    "goo.gl", 
 +    "t.ly", 
 +    "ow.ly", 
 +    "is.gd", 
 +    "cutt.ly", 
 +    "rebrand.ly", 
 +    "shorturl.at", 
 +} 
 +DOI_HOSTS = {"doi.org", "dx.doi.org"}
  
 ZENODO_ID_RE = re.compile(r"(?:zenodo\.org/(?:records|record)/|10\.5281/zenodo\.)(\d+)", re.I) ZENODO_ID_RE = re.compile(r"(?:zenodo\.org/(?:records|record)/|10\.5281/zenodo\.)(\d+)", re.I)
Line 294: Line 311:
     if m:     if m:
         return "doi.org", "/" + m.group(1), raw         return "doi.org", "/" + m.group(1), raw
 +    if ' ' in raw or '\t' in raw:
 +        raise ValueError(f"whitespace in URL: {raw!r}")
     with_scheme = raw if re.match(r"^[a-z][a-z0-9+.-]*://", raw, re.I) else "https://" + raw     with_scheme = raw if re.match(r"^[a-z][a-z0-9+.-]*://", raw, re.I) else "https://" + raw
-    parts = urllib.parse.urlparse(with_scheme.replace(" ", ""))+    parts = urllib.parse.urlparse(with_scheme)
     host = parts.hostname     host = parts.hostname
     if host is None or host == "":     if host is None or host == "":
Line 312: Line 331:
  
 def classify(url: str) -> tuple[str, str]: def classify(url: str) -> tuple[str, str]:
-    """Return (family, permanence)."""+    """Return (family, permanence). 
 + 
 +    DOI-prefix matching applies only to doi.org / dx.doi.org / doi: URLs. 
 +    A GitHub path that happens to contain 10.5281 is not Zenodo. 
 +    A generic publisher DOI is not archival. 
 +    """
     host, path, raw = parse(url)     host, path, raw = parse(url)
     if host in SHORTENERS:     if host in SHORTENERS:
         return "URL shortener", "shortener"         return "URL shortener", "shortener"
-    pref = doi_prefix(path) or doi_prefix(raw) +    is_doi = host in DOI_HOSTS 
-    if pref is not None and pref in DOI_ARCHIVAL: +    if is_doi: 
-        return DOI_ARCHIVAL[pref], "archival"+        pref = doi_prefix(path) or doi_prefix(raw) 
 +        if pref is not None and pref in DOI_ARCHIVAL: 
 +            return DOI_ARCHIVAL[pref], "archival" 
 +        if pref == "10.48550": 
 +            return "arXiv", "publisher-page" 
 +        return "DOI resolver (unknown archive)", "unknown"
     if host in ARCHIVAL_HOSTS:     if host in ARCHIVAL_HOSTS:
-        return ARCHIVAL_HOSTS[host], "archival"+        family = ARCHIVAL_HOSTS[host] 
 +        if family == "Zenodo" and zenodo_id(raw) is None: 
 +            return "Zenodo (no record id)", "unknown" 
 +        path_l = path.lower().rstrip("/") or "/" 
 +        if path_l in ("/", "/account", "/login", "/signin") or path.lower().startswith("/account"): 
 +            return f"{family} (no record path)", "unknown" 
 +        return family, "archival"
     if host.endswith(".osf.io"):     if host.endswith(".osf.io"):
         return "OSF", "archival"         return "OSF", "archival"
     if host.endswith(".figshare.com"):     if host.endswith(".figshare.com"):
         return "Figshare", "archival"         return "Figshare", "archival"
-    if host in ("doi.org", "dx.doi.org"): 
-        return "DOI repository", "archival" 
     if host.endswith(".github.io") or host.endswith(".githubusercontent.com"):     if host.endswith(".github.io") or host.endswith(".githubusercontent.com"):
         return "GitHub", "mutable-repo"         return "GitHub", "mutable-repo"
Line 337: Line 370:
     if host == "anonymous.4open.science":     if host == "anonymous.4open.science":
         return "anonymous.4open.science (double-blind GitHub)", "anonymized"         return "anonymous.4open.science (double-blind GitHub)", "anonymized"
-    if host == "arxiv.org" or pref == "10.48550":+    if host == "arxiv.org":
         return "arXiv", "publisher-page"         return "arXiv", "publisher-page"
     if host == "usenix.org" or host.endswith(".usenix.org"):     if host == "usenix.org" or host.endswith(".usenix.org"):
Line 387: Line 420:
         "--require-doi",         "--require-doi",
         action="store_true",         action="store_true",
-        help="exit 1 unless every URL is on an archival host (Zenodo/OSF/Figshare/Dryad/Dataverse/4TU/Software Heritage)",+        help="exit 1 unless every URL is on a named archival host (Zenodo/OSF/Figshare/Dryad/Dataverse/4TU/Software Heritage). A publisher DOI is not enough.",
     )     )
     args = ap.parse_args()     args = ap.parse_args()
Line 406: Line 439:
         extra = ""         extra = ""
         if family == "Zenodo":         if family == "Zenodo":
-            extra = "  |  " + describe_zenodo(url)+            desc = describe_zenodo(url) 
 +            extra = "  |  " + desc 
 +            if desc.startswith("could not parse") or desc.startswith("Zenodo API HTTP"): 
 +                bad += 1
         print(f"{permanence:<12} {family:<44} {url}{extra}")         print(f"{permanence:<12} {family:<44} {url}{extra}")
         if permanence == "shortener":         if permanence == "shortener":
 +            bad += 1
 +        if permanence == "unknown" and family.startswith("Zenodo"):
             bad += 1             bad += 1
         if args.require_doi and permanence != "archival":         if args.require_doi and permanence != "archival":
Line 463: Line 501:
   * **Seven venues only.** CHI, SOUPS, ACSAC, EuroS&P, RAID, AsiaCCS are absent. Artifact-evaluation culture at those venues is not in these rates.   * **Seven venues only.** CHI, SOUPS, ACSAC, EuroS&P, RAID, AsiaCCS are absent. Artifact-evaluation culture at those venues is not in these rates.
   * **"Own-link" is a URL the extractor stored as the authors'.** A paper that says "code available on request" with no URL is not in the 3,321. A paper that prints someone else's GitHub is not either (''belongsToAuthors === false'').   * **"Own-link" is a URL the extractor stored as the authors'.** A paper that says "code available on request" with no URL is not in the 3,321. A paper that prints someone else's GitHub is not either (''belongsToAuthors === false'').
-  * **Hosts are folded.** GitHub includes gist, raw.githubusercontent.com and ''%%*.github.io%%''. Zenodo includes ''doi.org/10.5281'' and the unschemed ''doi:10.5281/zenodo.N''. The residue (667 hosts) is on the provenance page. Exact-string "github.com" undercounts.+  * **Hosts are folded.** GitHub includes gist, raw.githubusercontent.com and ''%%*.github.io%%''. Zenodo includes ''doi.org/10.5281'' and the unschemed ''doi:10.5281/zenodo.N'', not a GitHub path that happens to contain those digits. The residue (669 hosts) is on the provenance page. Exact-string "github.com" undercounts.
   * **''artifacts.badge'' is not a badge-award rate.** 15 papers. Use the venue AE results.   * **''artifacts.badge'' is not a badge-award rate.** 15 papers. Use the venue AE results.
   * **2025–2026 are provisional.** Own-link 76.9% / 75.9% in those years will move when the missing venue-years land.   * **2025–2026 are provisional.** Own-link 76.9% / 75.9% in those years will move when the missing venue-years land.
artifacts.1787836784.txt.gz · Last modified: by karel.kubicek.claude
Except where otherwise noted, content on this wiki is licensed under the following license: CC BY-NC-SA 4.0
CC BY-NC-SA 4.0 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki