| Both sides previous revisionPrevious revision | |
| statistics:study_preregistration [2026/08/21 08:30] – [Open Questions] karelkubicek | statistics:study_preregistration [2026/08/21 14:50] (current) – Boxes: <wrap> renders a span, use uppercase <WRAP>; one box per list, not per bullet karel.kubicek.claude |
|---|
| Note what the registration also does that the paper's text does not: it states four hypotheses, in advance, with the outcome variable and the model named. Since the registration is public, anyone can diff it against the published analysis themselves. That is the entire mechanism, and it costs you a URL in an appendix. | Note what the registration also does that the paper's text does not: it states four hypotheses, in advance, with the outcome variable and the model named. Since the registration is public, anyone can diff it against the published analysis themselves. That is the entire mechanism, and it costs you a URL in an appendix. |
| |
| <wrap todo> | <WRAP todo> |
| The paper does not tell the reader whether it followed its plan. Neither do the other six "silent" papers among the fifteen. **If you preregister, say in the paper whether you deviated** — otherwise the registration is a link nobody follows and the preregistration bought your reader nothing. | The paper does not tell the reader whether it followed its plan. Neither do the other six "silent" papers among the fifteen. **If you preregister, say in the paper whether you deviated** — otherwise the registration is a link nobody follows and the preregistration bought your reader nothing. |
| </wrap> | </WRAP> |
| |
| ===== Mechanics: OSF and AsPredicted ===== | ===== Mechanics: OSF and AsPredicted ===== |
| The nearest thing to a Registered Report anywhere adjacent to this literature is the **International Fuzzing Workshop (FUZZING)**, which has run one since its first edition: phase 1 reviews //"a full-length preregistered report"// on //"(i) the significance and novelty of the hypotheses or techniques, and (ii) the soundness and reproducibility of the methodology specified to validate the claims or hypotheses — but explicitly not based on the strength of the (preliminary) results"//, and accepted reports get in-principle acceptance for a journal version (an SE special issue in 2022, ACM TOSEM Registered Papers from 2023). It shows up in this corpus only as a citation: **five papers** — all of them fuzzing papers, at NDSS 2023, CCS 2024, NDSS 2024, IEEE S&P 2026 and NDSS 2026 — cite a FUZZING paper that carries "Registered Report" in its title, and none of the seven venues has copied the mechanism. Software engineering has adopted the mechanism where security has not: **MSR** and **ESEM** run conference Registered Reports tracks, and the **EMSE** journal has taken their stage-2 submissions //"since 2019"//. **CHI has not**, as of 2026: its 2026 contribution-types page lists neither registered reports nor preregistration.((MSR 2026 (''2026.msrconf.org/track/msr-2026-registered-reports''), ESEM 2024 (''conf.researchr.org/track/esem-2024/esem-2024-registered-reports''), EMSE (''emsejournal.github.io/registered_reports/'') and CHI 2026 (''chi2026.acm.org/contributions-to-chi/'') were all fetched on 2026-08-13 and return HTTP 200. **ACM TOSEM's own Registered Papers page could not be verified**: ''dl.acm.org'', ''tosem.acm.org'' and the static PDF route all return HTTP 403 to every non-browser client tried. The claim that FUZZING's accepted reports become TOSEM Registered Papers rests on the FUZZING track page, not on ACM's.)) | The nearest thing to a Registered Report anywhere adjacent to this literature is the **International Fuzzing Workshop (FUZZING)**, which has run one since its first edition: phase 1 reviews //"a full-length preregistered report"// on //"(i) the significance and novelty of the hypotheses or techniques, and (ii) the soundness and reproducibility of the methodology specified to validate the claims or hypotheses — but explicitly not based on the strength of the (preliminary) results"//, and accepted reports get in-principle acceptance for a journal version (an SE special issue in 2022, ACM TOSEM Registered Papers from 2023). It shows up in this corpus only as a citation: **five papers** — all of them fuzzing papers, at NDSS 2023, CCS 2024, NDSS 2024, IEEE S&P 2026 and NDSS 2026 — cite a FUZZING paper that carries "Registered Report" in its title, and none of the seven venues has copied the mechanism. Software engineering has adopted the mechanism where security has not: **MSR** and **ESEM** run conference Registered Reports tracks, and the **EMSE** journal has taken their stage-2 submissions //"since 2019"//. **CHI has not**, as of 2026: its 2026 contribution-types page lists neither registered reports nor preregistration.((MSR 2026 (''2026.msrconf.org/track/msr-2026-registered-reports''), ESEM 2024 (''conf.researchr.org/track/esem-2024/esem-2024-registered-reports''), EMSE (''emsejournal.github.io/registered_reports/'') and CHI 2026 (''chi2026.acm.org/contributions-to-chi/'') were all fetched on 2026-08-13 and return HTTP 200. **ACM TOSEM's own Registered Papers page could not be verified**: ''dl.acm.org'', ''tosem.acm.org'' and the static PDF route all return HTTP 403 to every non-browser client tried. The claim that FUZZING's accepted reports become TOSEM Registered Papers rests on the FUZZING track page, not on ACM's.)) |
| |
| <wrap todo> | <WRAP todo> |
| **This is the open lever.** Klemmer et al. {[klemmer2025_transparency]} found that //"More than half of our participants felt that preregistration and registered reports might enable null result publication"// in usable privacy and security — a community that believes it has a publication-bias problem and has no venue mechanism to fix it, sitting next to a workshop in an adjacent field that has run the mechanism successfully since 2022. Whoever pitches a Registered Reports track to a PETS or IMC steering committee has the FUZZING precedent to point at. | **This is the open lever.** Klemmer et al. {[klemmer2025_transparency]} found that //"More than half of our participants felt that preregistration and registered reports might enable null result publication"// in usable privacy and security — a community that believes it has a publication-bias problem and has no venue mechanism to fix it, sitting next to a workshop in an adjacent field that has run the mechanism successfully since 2022. Whoever pitches a Registered Reports track to a PETS or IMC steering committee has the FUZZING precedent to point at. |
| </wrap> | </WRAP> |
| |
| ===== Use in Publications ===== | ===== Use in Publications ===== |
