artifacts
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revision | |||
| artifacts [2026/08/27 13:30] – Apply focused-review fixes: pin_artifact DOI/shortener, Demir/CCS/USENIX/PETS wording, verbatim quotes. Authored by Claude. karel.kubicek.claude | artifacts [2026/08/27 13:37] (current) – Generic-review fixes: host_fold DOI-prefix, pin_artifact record paths, regenerated Zenodo 431/617, causal wording. Authored by Claude. karel.kubicek.claude | ||
|---|---|---|---|
| Line 6: | Line 6: | ||
| **A GitHub URL is not an archival artifact.** USENIX Security 2026's Artifacts Available badge — mandatory for every accepted paper — refuses GitHub, GitLab and personal pages. The archived copy has to be reachable via a long-term stable reference or DOI. IEEE S&P 2026 and TheWebConf 2026 say the same for Available. Zenodo is the host those calls name first.((USENIX Security 2026 badges, [[https:// | **A GitHub URL is not an archival artifact.** USENIX Security 2026's Artifacts Available badge — mandatory for every accepted paper — refuses GitHub, GitLab and personal pages. The archived copy has to be reachable via a long-term stable reference or DOI. IEEE S&P 2026 and TheWebConf 2026 say the same for Available. Zenodo is the host those calls name first.((USENIX Security 2026 badges, [[https:// | ||
| - | Of 5,859 papers in this corpus (seven venues, 2010–2026), | + | Of 5,859 papers in this corpus (seven venues, 2010–2026), |
| </ | </ | ||
| Line 30: | Line 30: | ||
| ^ Piece ^ Why it is not optional ^ | ^ Piece ^ Why it is not optional ^ | ||
| - | | **The URL list, as fetched.** Rank, domain, fetch date. A Tranco id if you used one ([[Programming: | + | | **The URL list, as fetched.** Rank, domain, fetch date. A Tranco id if you used one ([[Programming: |
| - | | **The crawler pin.** Library version, browser build (not " | + | | **The crawler pin.** Library version, browser build (not " |
| - | | **The vantage point.** Provider, city, ASN if you know it. | [[Design: | + | | **The vantage point.** Provider, city, ASN if you know it. | Geo and CDN splits are vantage-dependent ([[Design: |
| - | | **The classifier, with its version.** EasyList commit, Disconnect snapshot, your model weights, the threshold. | [[Privacy: | + | | **The classifier, with its version.** EasyList commit, Disconnect snapshot, your model weights, the threshold. | A later EasyList is a different experiment ([[Privacy: |
| - | | **Per-site outcomes**, not only aggregates. One row per site: loaded or not, the labels, the counts. | + | | **Per-site outcomes**, not only aggregates. One row per site: loaded or not, the labels, the counts. |
| - | | **What you deleted, and why.** Raw cookies, request bodies, identifiers. A README that says "we stripped PII" is not a schema. | this page, [[Practices: | + | | **The analysis scripts and the environment pin.** The command that produced the table; a lockfile or image digest. | Per-site outcomes without the reducer are a dataset, not a regenerated headline ([[Programming: |
| - | | **A licence, and a DOI.** GitHub' | + | | **What you deleted, and why.** Raw cookies, request bodies, identifiers. A README that says "we stripped PII" is not a schema. | Otherwise a re-run cannot tell missingness from redaction ([[Practices: |
| + | | **A licence, and a DOI.** GitHub' | ||
| A system paper that evaluated a defence on a testbed has a different minimum (the implementation, | A system paper that evaluated a defence on a testbed has a different minimum (the implementation, | ||
| Line 87: | Line 88: | ||
| ^ Family ^ Permanence ^ Papers ^ Share of 3321 ^ | ^ Family ^ Permanence ^ Papers ^ Share of 3321 ^ | ||
| | GitHub | mutable-repo | 1896 | 57.1% | | | GitHub | mutable-repo | 1896 | 57.1% | | ||
| - | | Zenodo | archival | 436 | 13.1% | | + | | Zenodo | archival | 431 | 13.0% | |
| | Google Sites (project page) | project-page | 157 | 4.7% | | | Google Sites (project page) | project-page | 157 | 4.7% | | ||
| | OSF | archival | 112 | 3.4% | | | OSF | archival | 112 | 3.4% | | ||
| Line 100: | Line 101: | ||
| | GitLab | mutable-repo | 25 | 0.8% | | | GitLab | mutable-repo | 25 | 0.8% | | ||
| - | Union of archival families: **622 / 3,321 = 18.7%**. **59 papers (1.8%) give a shortener as the artifact URL** (bit.ly, tinyurl, goo.gl). That URL is not an artifact; it is a redirect the owner can retarget. | + | Union of archival families: **617 / 3,321 = 18.6%**. **59 papers (1.8%) give a shortener as the artifact URL** (bit.ly, tinyurl, goo.gl). That URL is not an artifact; it is a redirect the owner can retarget. |
| ==== Over time ==== | ==== Over time ==== | ||
| Line 121: | Line 122: | ||
| | 2022 | 546 | 338 | 61.9% | 234 | 69.2% | 5 | 1.5% | | | 2022 | 546 | 338 | 61.9% | 234 | 69.2% | 5 | 1.5% | | ||
| | 2023 | 719 | 475 | 66.1% | 321 | 67.6% | 21 | 4.4% | | | 2023 | 719 | 475 | 66.1% | 321 | 67.6% | 21 | 4.4% | | ||
| - | | 2024 | 690 | 457 | 66.2% | 307 | 67.2% | 42 | 9.2% | | + | | 2024 | 690 | 457 | 66.2% | 307 | 67.2% | 41 | 9.0% | |
| - | | 2025* | 770 | 592 | 76.9% | 318 | 53.7% | 228 | 38.5% | | + | | 2025* | 770 | 592 | 76.9% | 318 | 53.7% | 224 | 37.8% | |
| | 2026* | 415 | 315 | 75.9% | 202 | 64.1% | 132 | 41.9% | | | 2026* | 415 | 315 | 75.9% | 202 | 64.1% | 132 | 41.9% | | ||
| Line 132: | Line 133: | ||
| | 2025–2026* | 1185 | 907 | 76.5% | | | 2025–2026* | 1185 | 907 | 76.5% | | ||
| - | GitHub' | + | GitHub' |
| ==== By venue ==== | ==== By venue ==== | ||
| Line 138: | Line 139: | ||
| ^ Venue ^ Papers ^ Own-link ^ Share ^ GitHub of own-link ^ Zenodo of own-link ^ | ^ Venue ^ Papers ^ Own-link ^ Share ^ GitHub of own-link ^ Zenodo of own-link ^ | ||
| | CCS | 990 | 467 | 47.2% | 252 | 25 | | | CCS | 990 | 467 | 47.2% | 252 | 25 | | ||
| - | | IEEE-SP | 767 | 423 | 55.1% | 270 | 12 | | + | | IEEE-SP | 767 | 423 | 55.1% | 270 | 11 | |
| | IMC | 638 | 313 | 49.1% | 163 | 12 | | | IMC | 638 | 313 | 49.1% | 163 | 12 | | ||
| - | | NDSS | 701 | 443 | 63.2% | 310 | 102 | | + | | NDSS | 701 | 443 | 63.2% | 310 | 100 | |
| | PETS | 510 | 258 | 50.6% | 158 | 5 | | | PETS | 510 | 258 | 50.6% | 158 | 5 | | ||
| - | | USENIX | 1410 | 980 | 69.5% | 460 | 233 | | + | | USENIX | 1410 | 980 | 69.5% | 460 | 231 | |
| | WWW | 843 | 437 | 51.8% | 283 | 47 | | | WWW | 843 | 437 | 51.8% | 283 | 47 | | ||
| - | USENIX is the venue that currently **requires** an open-science appendix, and it is the venue with the highest own-link rate (69.5%) and more than half of the Zenodo mass (233 of 436). PETS, which has run optional artifact evaluation for years, has **5** Zenodo papers. Optional AE does not produce archival deposits; | + | USENIX is the venue that currently **requires** an open-science appendix, and it is the venue with the highest own-link rate (69.5%) and more than half of the Zenodo mass (231 of 431). PETS, which has run optional artifact evaluation for years, has **5** Zenodo papers. Optional AE is not the same mechanism as a mandatory Available badge; the rates should not be read as " |
| ===== Venue artifact evaluation, today ===== | ===== Venue artifact evaluation, today ===== | ||
| Line 164: | Line 165: | ||
| ===== GitHub vs a DOI ===== | ===== GitHub vs a DOI ===== | ||
| - | GitHub is the right place to **develop**. It is the wrong place to **cite**. The default branch moves; a force-push rewrites history; a renamed account 404s the camera-ready URL. The 2026 Available calls noticed. | + | GitHub is the right place to **develop**. It is the wrong place to **satisfy an Available badge**. The default branch moves; a force-push rewrites history; a renamed account 404s the camera-ready URL. A commit or tag permalink is a citable pointer at source; it is still not the archival deposit the 2026 Available calls ask for. |
| The pattern that satisfies every row of the table above: | The pattern that satisfies every row of the table above: | ||
| Line 173: | Line 174: | ||
| - Leave the GitHub URL in the README so people can file issues. It is a pointer, not the archive. | - Leave the GitHub URL in the README so people can file issues. It is a pointer, not the archive. | ||
| - | OSF, Figshare, Dryad, institutional Dataverse, 4TU.ResearchData and Software Heritage also issue DOIs. Zenodo is the one the security AE pages name first, and it is the one 436 papers in this corpus already use. | + | OSF, Figshare, Dryad, institutional Dataverse, 4TU.ResearchData and Software Heritage also issue DOIs. Zenodo is the one the security AE pages name first, and it is the one 431 papers in this corpus already use. |
| '' | '' | ||
| Line 200: | Line 201: | ||
| **Restricted access is a release.** Pastrana et al.'s CrimeBB is available to academic researchers from the Cambridge Cybercrime Centre, not on a public URL {[pastrana2018_crimebb]}. Beverly et al. published seeds and targets and held the complete IPv6 traces under restricted distribution {[beverly2018_beholder]}. Both count as an artifact in the schema ('' | **Restricted access is a release.** Pastrana et al.'s CrimeBB is available to academic researchers from the Cambridge Cybercrime Centre, not on a public URL {[pastrana2018_crimebb]}. Beverly et al. published seeds and targets and held the complete IPv6 traces under restricted distribution {[beverly2018_beholder]}. Both count as an artifact in the schema ('' | ||
| - | **What you can always | + | **What you can often still release**, even when the raw crawl cannot go out. Dual-use, an NDA, or copyright can still block the crawler or the classifier — Kim et al. withheld LTEFuzz itself {[kim2019_touching]}. For each withheld piece, say so. |
| * The URL list (or the Tranco id plus the filter), unless the URLs themselves are the sensitive object (delisting requests, victim sites, the high-school Facebook study). | * The URL list (or the Tranco id plus the filter), unless the URLs themselves are the sensitive object (delisting requests, victim sites, the high-school Facebook study). | ||
| - | * The crawler, the pins, the classifier, the analysis scripts. | + | * The crawler, the pins, the classifier, the analysis scripts |
| * Aggregates and per-bin counts that do not identify a site or a person. | * Aggregates and per-bin counts that do not identify a site or a person. | ||
| * A data dictionary of what you deleted. | * A data dictionary of what you deleted. | ||
| Line 215: | Line 216: | ||
| ===== A check you can run ===== | ===== A check you can run ===== | ||
| - | '' | + | '' |
| <file python pin_artifact.py> | <file python pin_artifact.py> | ||
| Line 310: | Line 311: | ||
| if m: | if m: | ||
| return " | return " | ||
| + | if ' ' in raw or ' | ||
| + | raise ValueError(f" | ||
| with_scheme = raw if re.match(r" | with_scheme = raw if re.match(r" | ||
| - | parts = urllib.parse.urlparse(with_scheme.replace(" | + | parts = urllib.parse.urlparse(with_scheme) |
| host = parts.hostname | host = parts.hostname | ||
| if host is None or host == "": | if host is None or host == "": | ||
| Line 349: | Line 352: | ||
| if family == " | if family == " | ||
| return " | return " | ||
| + | path_l = path.lower().rstrip("/" | ||
| + | if path_l in ("/", | ||
| + | return f" | ||
| return family, " | return family, " | ||
| if host.endswith(" | if host.endswith(" | ||
| Line 495: | Line 501: | ||
| * **Seven venues only.** CHI, SOUPS, ACSAC, EuroS& | * **Seven venues only.** CHI, SOUPS, ACSAC, EuroS& | ||
| * **" | * **" | ||
| - | * **Hosts are folded.** GitHub includes gist, raw.githubusercontent.com and '' | + | * **Hosts are folded.** GitHub includes gist, raw.githubusercontent.com and '' |
| * **'' | * **'' | ||
| * **2025–2026 are provisional.** Own-link 76.9% / 75.9% in those years will move when the missing venue-years land. | * **2025–2026 are provisional.** Own-link 76.9% / 75.9% in those years will move when the missing venue-years land. | ||
artifacts.1787837457.txt.gz · Last modified: by karel.kubicek.claude