Measure The Web

User Tools

Site Tools

Trace: • hypothesis_testing • interaction • foxhound • ip_classification
practices:ethics

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
practices:ethics [2026/08/18 23:09] – Rewrite from notes stub into a full page: measured reporting rates (the participants-vs-crawls split), whether a crawl needs review and what to do when no board exists, what the seven venues require in 2026 including the generative-AI policies, harm from karel.kubicek.claudepractices:ethics [2026/08/21 14:50] (current) – Boxes: <wrap> renders a span, use uppercase <WRAP>; one box per list, not per bullet karel.kubicek.claude
Line 55: Line 55:
 | 2025–2026* | 1,019 | 467 | 45.8% | 303 | 263 | 86.8% | 166 | 50 | 30.1% | | 2025–2026* | 1,019 | 467 | 45.8% | 303 | 263 | 86.8% | 166 | 50 | 30.1% |
  
-<wrap todo>* 2025–2026 is provisional: CCS 2026 and IMC 2026 have not been held, and IEEE S&P 2026 and TheWebConf 2026 abstracts are not in OpenAlex, so those venue-years are under-represented by construction. See [[Literature:Corpus]].</wrap>+<WRAP todo>* 2025–2026 is provisional: CCS 2026 and IMC 2026 have not been held, and IEEE S&P 2026 and TheWebConf 2026 abstracts are not in OpenAlex, so those venue-years are under-represented by construction. See [[Literature:Corpus]].</WRAP>
  
 In the last complete window, 85.2% of participant studies and 28.8% of participant-free crawls state an outcome. Put the two series on the same axis and the lag is measurable: **participant-free crawls reach in 2025–2026 (30.1%) a reporting rate participant studies had already passed before 2014** (28.6% in 2010–2013, 58.7% in 2014–2017). If you are writing a crawl paper, you are in the second column, and the second column is where the reviewers' patience is currently running out. In the last complete window, 85.2% of participant studies and 28.8% of participant-free crawls state an outcome. Put the two series on the same axis and the lag is measurable: **participant-free crawls reach in 2025–2026 (30.1%) a reporting rate participant studies had already passed before 2014** (28.6% in 2010–2013, 58.7% in 2014–2017). If you are writing a crawl paper, you are in the second column, and the second column is where the reviewers' patience is currently running out.
Line 148: Line 148:
 | TheWebConf '26* | Encouraged | Via ACM's human-participants policy | Not named | Not required | | TheWebConf '26* | Encouraged | Via ACM's human-participants policy | Not named | Not required |
  
-<wrap todo>* Six of the seven rows are the currently open cycle. TheWebConf is not: ''www2027.thewebconf.org'' does not resolve and ''thewebconf.org'' lists WWW 2027 (Dublin) with no call for papers yet, so that row is the **last published** cycle, WWW 2026, which was held in April 2026.</wrap>+<WRAP todo>* Six of the seven rows are the currently open cycle. TheWebConf is not: ''www2027.thewebconf.org'' does not resolve and ''thewebconf.org'' lists WWW 2027 (Dublin) with no call for papers yet, so that row is the **last published** cycle, WWW 2026, which was held in April 2026.</WRAP>
  
 Four things to take from it. Four things to take from it.
Line 194: Line 194:
 Two things this means for a measurement paper specifically. If you used an LLM **as an instrument** — to classify cookies, label banners, read privacy policies, decide whether a site is in scope — that is a methods decision and a disclosure, not a writing aid, and IEEE S&P's phrasing ("as an object of study") is the only one that clearly covers it. And if the model touched anything you collected, the ethics section and the AI section now interact: sending crawled content, participant text or vulnerability details to a hosted model is a disclosure of that data to a third party, and none of these policies says so for you. Two things this means for a measurement paper specifically. If you used an LLM **as an instrument** — to classify cookies, label banners, read privacy policies, decide whether a site is in scope — that is a methods decision and a disclosure, not a writing aid, and IEEE S&P's phrasing ("as an object of study") is the only one that clearly covers it. And if the model touched anything you collected, the ethics section and the AI section now interact: sending crawled content, participant text or vulnerability details to a hosted model is a disclosure of that data to a third party, and none of these policies says so for you.
  
-<wrap todo>The corpus cannot yet say how often LLM-based measurement pipelines disclose this: ''tools[]'' has an ''llm'' category, but there is no schema field for "was AI use disclosed", and the venue requirements are newer than most of the corpus. A full-text sweep would be the way in.</wrap>+<WRAP todo>The corpus cannot yet say how often LLM-based measurement pipelines disclose this: ''tools[]'' has an ''llm'' category, but there is no schema field for "was AI use disclosed", and the venue requirements are newer than most of the corpus. A full-text sweep would be the way in.</WRAP>
  
 ===== Harm from crawling, and what the field does about it ===== ===== Harm from crawling, and what the field does about it =====
Line 395: Line 395:
   * [[Design:Website selection]] — a robots.txt exclusion is a change to your sampling frame.   * [[Design:Website selection]] — a robots.txt exclusion is a change to your sampling frame.
   * [[:Artifacts]] — what you can release once the data has people in it.   * [[:Artifacts]] — what you can release once the data has people in it.
 +  * [[Statistics:Interrater agreement]] — hired annotators and crowdworkers are participants; 5.1% of the corpus's hand-coding papers use them.
  
 ===== Methodology and limitations of these figures ===== ===== Methodology and limitations of these figures =====
practices/ethics.1787094540.txt.gz · Last modified: by karel.kubicek.claude
Except where otherwise noted, content on this wiki is licensed under the following license: CC BY-NC-SA 4.0
CC BY-NC-SA 4.0 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki