| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| practices:ethics [2026/08/13 19:50] – Cross-link the new practices:notifying_websites page from the ethics notes. Authored by Claude karel.kubicek.claude | practices:ethics [2026/08/21 14:50] (current) – Boxes: <wrap> renders a span, use uppercase <WRAP>; one box per list, not per bullet karel.kubicek.claude |
|---|
| ====== Ethics in Web Measurements ====== | ====== Ethics in Web Measurements ====== |
| |
| <wrap todo>This page only contains notes</wrap> | You are about to point a browser at a hundred thousand websites you do not own. Nobody consented, and most of the operators will never know you were there. Your institution's review board deals with interviews and blood samples, and when you ask them they will probably tell you this is not human-subjects research and to go away. That answer is usually correct, it takes ten minutes to obtain, and **it settles almost none of the questions your reviewers will actually ask you.** |
| |
| ===== Harm from crawling ===== | This page is about the questions that are left. Who can be harmed by an automated measurement and how; what the field actually does about it, measured rather than asserted; where your review board's remit ends and your own judgement starts; and what has to be in the paper so that a reviewer does not send it back. It is deliberately not a summary of the Menlo Report or of 45 CFR 46 — those are short, and you should read them yourself; the links are below. What is not written down anywhere is the operational part: that the ethics apparatus of computer science was built for studies with participants, that a crawl has none, and that crawling papers are more than a decade behind participant studies in saying what they did instead — a gap this page measures rather than asserts. |
| |
| * Excessive use of resources | <WRAP important> |
| * Collecting unlicensed data | **"Not human subjects" is a statement about your paperwork, not about your study.** |
| * Actions causing financial harm (ordering something, wasting time of website owners, etc.) | |
| * Security risks | A US Institutional Review Board applies the Common Rule, whose remit is *human subjects* — living individuals about whom an investigator obtains data through intervention, interaction, or identifiable private information. An automated crawl of public pages usually satisfies none of those, so the board rules it outside their purview. **That is a jurisdictional finding.** It says the board will not review you. It does not say your study is harmless, and it is not a sentence you can put in a paper in place of an ethics section. |
| | |
| | Two things follow, and the corpus shows both going wrong. |
| | |
| | * **The determination can be wrong, and only the board can tell you.** Snyder et al. {[snyder2017_fifteen]} set out to analyse publicly posted doxes: //"When beginning this project, we believed that this analysis of publicly available information was not human subjects research. Upon later discussion with our institution's IRB, we discovered that this collection and analysis of public data did qualify as human subject[s research]"//. Public does not mean out of scope. Ask before you collect, not after. |
| | * **Being out of scope of the board does not make you out of scope of the reviewers.** The venues that carry this literature moved hard between 2022 and 2026. IEEE S&P, NDSS and IMC now run standing ethics bodies; **six of the seven venues behind this site say a paper can be rejected on ethics grounds** (TheWebConf only through ACM's human-participants policy, which does not reach a participant-free crawl); and three say in the call for papers that an IRB determination is not by itself sufficient. A "not human subjects" line and nothing else is now a reviewable defect. See [[#What the venues require, as of August 2026]]. |
| | </WRAP> |
| | |
| | ===== How much of the field does this, measured ===== |
| | |
| | The figures below come from the publication corpus behind this site: **5,859 papers, seven venues** (CCS, IMC, NDSS, PoPETs, USENIX Security, TheWebConf, IEEE S&P), 2010–2026, with a structured extraction over each one. Every claim here is a claim about *those seven venues*: EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS are absent, and so, importantly, is the entire ethics and STS literature that writes about this from the outside. |
| | |
| | The population for this section is the **5,118 empirical papers** — the ones that measured the world as it is. A proofs-only paper has nothing to review, and including it would manufacture silence. |
| | |
| | ^ ''ethics.reviewOutcome'' ^ Papers ^ Share of 5,118 ^ |
| | | ''approved'' | 994 | 19.4% | |
| | | ''exempt'' | 157 | 3.1% | |
| | | ''not-required'' | 179 | 3.5% | |
| | | ''sought-outcome-unstated'' | 80 | 1.6% | |
| | | ''explicitly-discussed-no-review'' | 318 | 6.2% | |
| | | ''none-mentioned'' //(sentinel)// | 2,744 | 53.6% | |
| | | no ''ethics'' record extracted //(sentinel)// | 646 | 12.6% | |
| | |
| | **One empirical paper in three (1,728, 33.8%) says anything checkable about ethics review at all.** Of those 1,728, two thirds (1,151) report an approval or an exemption; the rest say a review was not required, was sought with no outcome given, or was explicitly discussed and not obtained. |
| | |
| | ==== The split that matters: participants versus no participants ==== |
| | |
| | The headline above is an average over two populations that behave completely differently, and the difference is the single most useful thing on this page. |
| | |
| | ^ Population ^ Papers ^ States a review outcome ^ Approved or exempt ^ Silent ^ |
| | | Recruited participants | 1,357 | 1,022 (75.3%) | 896 (66.0%) | 335 (24.7%) | |
| | | Ran a crawl (any) | 1,120 | 310 (27.7%) | 188 (16.8%) | 810 (72.3%) | |
| | | …of which also recruited participants | 149 | 103 (69.1%) | 95 (63.8%) | 46 (30.9%) | |
| | | …**crawled and recruited nobody** | 971 | 207 (21.3%) | 93 (9.6%) | 764 (78.7%) | |
| | | All empirical papers | 5,118 | 1,728 (33.8%) | 1,151 (22.5%) | 3,390 (66.2%) | |
| | |
| | "Silent" is ''none-mentioned'' plus papers where the extraction found no ethics record at all. |
| | |
| | **Four out of five papers that ran a crawl and recruited nobody say nothing about review.** Recruit a single participant and the rate inverts: 69.1% of the crawls that also ran a study state an outcome. The norm did not arrive for web measurement; it arrived for human subjects and stopped at the boundary of the Common Rule. And it is not a fossil of the early corpus — the gap is roughly stable while both series rise: |
| | |
| | ^ Window ^ Empirical ^ states ^ ^ Recruited participants ^ states ^ ^ Crawl, no participants ^ states ^ ^ |
| | | 2010–2013 | 460 | 47 | 10.2% | 84 | 24 | 28.6% | 89 | 6 | 6.7% | |
| | | 2014–2017 | 718 | 155 | 21.6% | 155 | 91 | 58.7% | 155 | 23 | 14.8% | |
| | | 2018–2021 | 1,272 | 378 | 29.7% | 329 | 230 | 69.9% | 262 | 42 | 16.0% | |
| | | 2022–2024 | 1,649 | 681 | 41.3% | 486 | 414 | 85.2% | 299 | 86 | 28.8% | |
| | | 2025–2026* | 1,019 | 467 | 45.8% | 303 | 263 | 86.8% | 166 | 50 | 30.1% | |
| | |
| | <WRAP todo>* 2025–2026 is provisional: CCS 2026 and IMC 2026 have not been held, and IEEE S&P 2026 and TheWebConf 2026 abstracts are not in OpenAlex, so those venue-years are under-represented by construction. See [[Literature:Corpus]].</WRAP> |
| | |
| | In the last complete window, 85.2% of participant studies and 28.8% of participant-free crawls state an outcome. Put the two series on the same axis and the lag is measurable: **participant-free crawls reach in 2025–2026 (30.1%) a reporting rate participant studies had already passed before 2014** (28.6% in 2010–2013, 58.7% in 2014–2017). If you are writing a crawl paper, you are in the second column, and the second column is where the reviewers' patience is currently running out. |
| | |
| | ==== Where the reviewers who have read one before are ==== |
| | |
| | ^ Venue ^ Empirical papers ^ States a review outcome ^ Share ^ Cites the Menlo Report ^ Share ^ |
| | | PoPETs | 477 | 259 | 54.3% | 14 | 2.9% | |
| | | USENIX Security | 1,226 | 506 | 41.3% | 76 | 6.2% | |
| | | IEEE S&P | 646 | 246 | 38.1% | 26 | 4.0% | |
| | | IMC | 625 | 208 | 33.3% | 46 | 7.4% | |
| | | NDSS | 586 | 189 | 32.3% | 29 | 4.9% | |
| | | CCS | 818 | 197 | 24.1% | 24 | 2.9% | |
| | | TheWebConf | 740 | 123 | 16.6% | 10 | 1.4% | |
| | |
| | PoPETs states an outcome more than three times as often as TheWebConf. Part of that is subject matter and part is venue policy, but the practical reading is: the further your target venue is from the privacy community, the more of the ethics argument you have to make from scratch in the paper, because the reviewer has not seen the template. |
| | |
| | ==== It is not that nobody thinks about it ==== |
| | |
| | Reporting a *review* is not the same as reporting *care*, and the corpus separates them. Of the 1,120 papers that ran a crawl, **697 (62.2%) describe at least one concrete step they took to limit harm**. On the like-for-like population — the 971 that crawled and recruited nobody — it is **585 (60.2%) describing a mitigation against 207 (21.3%) stating a review outcome**, so almost three times as many papers say what they did as say who checked it. The field is not careless; it is silent about the institutional half and reasonably articulate about the practical half. What is missing is almost always the sentence "we asked X and they said Y". |
| | |
| | ==== An independent count says the same thing ==== |
| | |
| | This is not an artefact of one extraction. Ramulu et al. {[ramulu2025_ethics]} hand-read **all 1,154 papers published at CCS, IEEE S&P, NDSS and USENIX Security in 2024** and found **839 with no discussion of ethics at all**, 315 with some discussion, and 270 with a dedicated section. Of the 116 papers that involved human subjects, 102 discussed ethics and //"all but 17"// named a review board — while only **52 of the ~1,038 papers without human subjects** mentioned one. (Their sentence does not restate its base, so "all but 17" is 99 of 116 or 85 of 102; either way the contrast with 52 holds.) Different corpus, different method, same shape: the apparatus tracks participants, not harm. |
| | |
| | Their interview study with 24 researchers (several of them ethics-committee members and program chairs) is the best single account of //why//: ethical reasoning in this field happens in private conversations inside research groups and inside program committees, and almost none of it reaches the written record that a new PhD student can learn from. If you have read nothing else on this topic, read that paper. |
| | |
| | ===== Does your crawl need an ethics review? ===== |
| | |
| | ==== The US answer, and what it does not cover ==== |
| | |
| | Under the revised Common Rule (45 CFR 46), a *human subject* is a living individual about whom an investigator obtains information through **intervention or interaction** with the individual, or obtains **identifiable private information**.((45 CFR 46.102(e)(1), as revised effective 2018. Read from eCFR on 2026-08-18: [[https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-A/part-46|ecfr.gov/current/title-45/part-46]].)) A crawl of public pages typically involves neither, which is why the sentence "our IRB determined this is not human subjects research" appears so often — the literal phrasing shows up in a full-text sweep of **79 of the 5,118 empirical papers**, and ''IRB'' appears anywhere in 987 (19.3%). |
| | |
| | Read the good examples and notice what they do beyond the determination. ICLab {[niaki2020_iclab]} states the determination //and// the reason it holds — no personally identifiable information about the volunteers — and then keeps going. Censored Planet {[raman2020_censored]} states that boards //"typically consider network measurement studies to be outside of their purview when they do not involve human subjects or their personally identifiable data"// and then devotes the section to the harm the measurement could do to people in the measured countries, which no board was ever going to assess. That is the correct shape: the determination is the first sentence of the ethics section, not the whole of it. |
| | |
| | What the determination explicitly does **not** cover, and what you therefore have to argue yourself: |
| | |
| | * Harm to **site operators**: load, cost, corrupted analytics, on-call pages, wasted staff time. |
| | * Harm to **third parties who did not consent to be in your vantage point** — the people whose devices your residential proxy resolves through. |
| | * Harm to **users of the sites you touch**, if your interaction has any side effect at all. |
| | * Harm from **publication**: naming a vulnerable site, a country, a small operator, or a demographically identifiable group. |
| | * **Legal exposure** — yours and your institution's. A board that has declared you out of scope has also declared it is not advising you on this. |
| | |
| | ==== Ask anyway, and ask before you collect ==== |
| | |
| | The determination costs you an email and gives you a citable sentence. It also occasionally comes back the other way, as it did for Snyder et al. {[snyder2017_fifteen]} — and it comes back the other way most often for exactly the studies that feel safest, because "public data" is a description of access, not of identifiability. Ask early enough that the answer can still change the design. |
| | |
| | Where the board declines, the corpus shows three substitutes that reviewers have accepted: |
| | |
| | * **An ad-hoc committee of colleagues.** Bennett et al. {[bennett2022_spfail]}: //"While our institution has an IRB, our research did not involve human subjects, so it did not fall under the purview the IRB. In lieu of formal IRB approval, we consulted an ad-hoc ethics committee consisting of subject-matter experts within our same department. They approved our experiment plan as described herein."// This is the cheapest credible answer and the one most likely to survive review. Note the last sentence: they did not just consult, they got a decision and reported it. |
| | * **A named external board.** The [[https://research.torproject.org/safetyboard/|Tor Research Safety Board]] gives //"feedback to researchers who use our guidelines to assess the safety of their planned research"// on the Tor network (read 2026-08-18), and is named in 7 papers in the corpus. It is the only standing domain-specific ethics board this literature names — and PoPETs' call for papers names it explicitly as an acceptable external panel alongside an IRB. There is no equivalent for the open web; the nearest proposals are the pre-registration board in {[hantke2024_redlines]} and the "federated ethics boards" of Dirksen et al.((Alexandra Dirksen, Sebastian Giessler, Hendrik Erz, Martin Johns and Tobias Fiebig, //Don't Patch the Researcher, Patch the Game: A Systematic Approach for Responsible Research via Federated Ethics Boards//, New Security Paradigms Workshop 2024, [[https://dl.acm.org/doi/10.1145/3703465.3703475|doi:10.1145/3703465.3703475]]. NSPW is outside the seven venues behind this site.)) Neither exists yet. |
| | * **Legal counsel or the data protection officer**, where the risk is legal rather than to participants — 25 papers name a legal department or general counsel, 28 name a data protection officer or office. |
| | |
| | ==== If your institution has no board at all ==== |
| | |
| | Twenty papers in the corpus say so outright: //"Our university located in central Europe unfortunately does not have an ethics board but has a set of guidelines that we followed in our research"// {[krombholz2017_noidea]}. The sentence recurs from 2013 to 2026, and **16 of the 20 recruited participants** — so this is not a crawling-specific gap at all; it is a gap in the institutions, and it bites hardest on exactly the studies a board would normally review. US-style IRB review is an obligation attached to US federal research funding; most of the world has no equivalent standing body for computer science, and a good number of departments that do have one exclude CS from its remit. |
| | |
| | It is not a reason to write nothing. What those twenty papers put in place instead is exactly the list a reviewer will accept, and each of these is a real sentence from the corpus: |
| | |
| | * **A named alternative body.** //"we sought authorization and supervision from our network ma[nagement department]"//; //"the study protocols were cleared with the projects ethics officer"//; //"a dedicated team whose focus is privacy and human protection"//. |
| | * **A named external standard, followed and cited.** //"Our institution does not have an IRB, but we refer to and strictly follow the Menlo Report and the best practices of netw[ork measurement]"//. **Half of the twenty** name the Menlo Report, and it is the strongest of the substitutes because it is checkable. |
| | * **A documented internal process.** //"we had extensive ethics-related discussions within our interdisci[plinary team]"// — weakest of the three, but still better than silence if you say who was in the room and what they objected to. |
| | * **The applicable law, where the study is in the EU.** //"Our institution is in the EU and has no formal IRB process. However, we followed the strict rules of our national and European privacy regulations."// |
| | |
| | ==== What the corpus calls the body it asked ==== |
| | |
| | 1,347 empirical papers name a review body, in **595 distinct strings**. Folded by kind of institution: |
| | |
| | ^ Kind of body ^ Papers ^ Share of 1,347 ^ |
| | | A US-style Institutional Review Board (IRB) | 986 | 73.2% | |
| | | An ethics committee or ethical review board (non-IRB wording) | 241 | 17.9% | |
| | | The institution, named only as such | 47 | 3.5% | |
| | | Data protection officer or data protection office | 28 | 2.1% | |
| | | Legal department, general counsel or compliance office | 25 | 1.9% | |
| | | A regulator or government body | 7 | 0.5% | |
| | | An internal company or partner review | 5 | 0.4% | |
| | | An external or community safety board | 4 | 0.3% | |
| | |
| | The vocabulary of this literature is overwhelmingly American, and that matters when you read it: "we obtained IRB approval" from a European group usually means a departmental ethics committee with a different remit and no federal backing. Say which yours is. |
| | |
| | ===== What the venues require, as of August 2026 ===== |
| | |
| | This changes every cycle, and it changed a lot between 2024 and 2026. Every row below was read from the venue's own call for papers on **2026-08-18**; check yours before you submit, because two of these moved within the last twelve months. |
| | |
| | ^ Venue (cycle) ^ Ethics section ^ Rejectable on ethics alone ^ Standing ethics body ^ Open-science appendix ^ |
| | | USENIX Security '27 | Strongly encouraged (was **mandatory** in '26) | Yes, at co-chairs' and PC discretion | Ethics Committee chair on the organising committee, but the CFP describes no review role for it | **Mandatory** | |
| | | IEEE S&P '27 | **Mandatory**, as a HotCRP field at registration; added to the manuscript at camera-ready | Yes | **Research Ethics Committee** | Encouraged | |
| | | NDSS '27 | Optional | Yes | **Ethics Review Board** | Encouraged | |
| | | ACM CCS '26 | Required only for papers "with potential ethical concerns" | Yes | Not named | **Mandatory, always** | |
| | | PoPETs '27 | **Mandatory**, enforced by the template | Yes | Not named; Tor Research Safety Board named as an acceptable external panel | **Mandatory** (a justified "no" is fine) | |
| | | ACM IMC '26 | **Mandatory** | Yes | **Ethics Committee** | Not at submission; enforced at shepherding | |
| | | TheWebConf '26* | Encouraged | Via ACM's human-participants policy | Not named | Not required | |
| | |
| | <WRAP todo>* Six of the seven rows are the currently open cycle. TheWebConf is not: ''www2027.thewebconf.org'' does not resolve and ''thewebconf.org'' lists WWW 2027 (Dublin) with no call for papers yet, so that row is the **last published** cycle, WWW 2026, which was held in April 2026.</WRAP> |
| | |
| | Four things to take from it. |
| | |
| | **The requirement is not monotonic.** USENIX Security '26 required //"a dedicated appendix after the main body of the paper (but before any references) called 'Ethical Considerations'"//; '27 lists among its main changes //"Ethics appendix is no longer mandatory (but is strongly encouraged)"//.((USENIX Security '26 and '27 calls for papers, [[https://www.usenix.org/conference/usenixsecurity26/call-for-papers|usenix.org/…/usenixsecurity26/call-for-papers]] and [[https://www.usenix.org/conference/usenixsecurity27/call-for-papers|…/usenixsecurity27/call-for-papers]], both read 2026-08-18.)) Do not learn the rule from a paper published two years ago. |
| | |
| | **"My IRB said no review needed" is explicitly not enough, at three venues.** CCS: //"institutional (IRB/ERB) approval is neither strictly necessary nor always sufficient to demonstrate ethical conduct"//.((ACM CCS 2026 call for papers, [[https://www.sigsac.org/ccs/CCS2026/call-for/call-for-papers.html|sigsac.org/ccs/CCS2026]], read 2026-08-18.)) NDSS: //"IRB exemptions may not be sufficient grounds for proper mitigation of ethical concerns"//.((NDSS 2027 call for papers, [[https://www.ndss-symposium.org/ndss2027/submissions/call-for-papers/|ndss-symposium.org/ndss2027]], read 2026-08-18.)) USENIX Security asks the question and answers it in one word — //"Is IRB approval a sufficient alternative? No! IRBs are not expected to understand computer security research well or to know about best practices and community norms in our field, and so IRB approval does not absolve researchers from considering ethical aspects of their work."// |
| | |
| | **Two venues tell you what to do if you have no board.** IMC: //"If the submission describes research involving human subjects and none of the authors are at an institution with an IRB (or equivalent), the authors are nonetheless expected to follow a research protocol that adheres to ethical principles, as stated in the ACM policy on research involving human subjects"//.((ACM IMC 2026 submission instructions, [[https://conferences.sigcomm.org/imc/2026/submission-instructions/|conferences.sigcomm.org/imc/2026]], read 2026-08-18.)) USENIX Security: //"If authors do not have access to an IRB … please clearly state that you do not have access to an IRB."// Say it; do not leave the reader to infer it. |
| | |
| | **The ACM baseline applies at CCS, IMC and TheWebConf whatever the CFP says.** ACM's //Publications Policy on Research Involving Human Participants and Subjects// (approved 15 August 2021, grounded explicitly in the Belmont Report) covers every ACM venue and answers the no-local-law case directly: //"Where such research is conducted in countries where no such local governing laws and regulations related to human participant and subject research exist, Authors must at a bare minimum be prepared to show compliance with the above detailed principles."//((ACM, //Publications Policy on Research Involving Human Participants and Subjects//, approved 2021-08-15, [[https://www.acm.org/publications/policies/research-involving-human-participants-and-subjects|acm.org/publications/policies]]. acm.org returns HTTP 403 to scripted clients; text read from a Wayback capture dated 2025-12-02 on 2026-08-18.)) |
| | |
| | ==== The template the field is converging on: a stakeholder analysis ==== |
| | |
| | USENIX Security's ethics guidelines are the most explicit statement anywhere of what an ethics section should contain, and they are worth following whatever venue you submit to. They ask for a **stakeholder-based ethics analysis** in five parts:((USENIX Security '26 call for papers, "Ethics Guidelines" section, read 2026-08-18. The '25 cycle published the same guidance as a standalone page, [[https://www.usenix.org/conference/usenixsecurity25/ethics-guidelines|usenix.org/…/usenixsecurity25/ethics-guidelines]], still live on 2026-08-18.)) |
| | |
| | - **Stakeholders** — //"all possible stakeholders (people, including the research team and society at large, and entities including companies) that may be impacted"//, and how each is affected both by the procedure and by the publication. |
| | - **Ethical principles** — which you considered, per stakeholder. The Menlo Report's four are the suggested starting point. |
| | - **Harms** — both //"tangible harms"// and //"violations of human rights even if there are no directly tangible harms"//. |
| | - **Mitigations** — //"both mitigated and unmitigated (potential) harms"//, and the steps taken. Note that they want the *unmitigated* ones written down too. |
| | - **Decision** — why you proceeded, and separately why you published. |
| | |
| | For a crawl, that structure fills itself in: operators, end users, third parties in your vantage point, other researchers who will be blocked after you, and your own team. For the moral-reasoning frameworks behind it, the guidelines point at {[kohno2023_trolley]}, which builds security-specific trolley problems to show where consequentialist and deontological reasoning give different answers — useful precisely when your co-authors disagree and neither can say why. |
| | |
| | ==== The standard everyone cites, and its status ==== |
| | |
| | The **Menlo Report** (//Ethical Principles Guiding Information and Communication Technology Research//, August 2012, funded by the US Department of Homeland Security) adapts the Belmont Report's three principles — Respect for Persons, Beneficence, Justice — and adds a fourth, **Respect for Law and Public Interest**. Its 2013 companion, //Applying Ethical Principles to Information and Communication Technology Research//, is the one with the worked examples, and is the more useful of the two for a measurement study. |
| | |
| | **The string //Menlo Report// appears in 225 of the 5,118 empirical papers (4.4%), and the rate is climbing sharply** — 0% in 2010–2013, 5.1% in 2022–2024, 10.0% in the provisional 2025–2026 slice. (That is a full-text sweep, so it is an upper bound on real use and a lower bound on nothing: a paper can follow the report without naming it.) It is named in the calls for papers of IEEE S&P, USENIX Security and NDSS. |
| | |
| | Two caveats a student should know before leaning on it. It is **archived**: DHS marks both pages //"Archived Content — In an effort to keep DHS.gov current, the archive contains outdated information that may not reflect current policy or programs"//, last updated 2023-01-12, and no successor or revision exists.((DHS publication pages [[https://www.dhs.gov/publication/st-menlo-report|dhs.gov/publication/st-menlo-report]] and [[https://www.dhs.gov/publication/st-menlo-companion|/st-menlo-companion]], checked 2026-08-18. Both now 301-redirect into an ''/archive/science-and-technology/'' path, and dhs.gov blocks scripted HTTP clients with HTTP 403 either way; the pages resolve normally in a browser. Script-friendly mirrors that returned HTTP 200 on 2026-08-18: [[https://catalog.caida.org/paper/2012_menlo_report_actual_formatted|CAIDA catalog, report]] and [[https://catalog.caida.org/paper/2013_menlo_report_companion_actual_formatted|CAIDA catalog, companion]].)) And it is not a manual: Hantke et al. {[hantke2024_redlines]} report that their interviewees //"highlighted the absence of practical ethical guidelines"//, summarising the Menlo Report as //"largely theoretical"// and lacking //"practical examples"// — the authors' characterisation of what they heard, not a quotation from an interviewee. Which is why the stakeholder template above, and not the report itself, is what you actually write from. |
| | |
| | ==== AI in your pipeline is now a disclosure requirement ==== |
| | |
| | Distinct from the ethics section, and newer: **every one of the seven venues now has a policy on generative-AI use by authors**, and four of them require a disclosure. This is where the security and privacy venues diverge sharply from the AI venues, and it catches people who have moved between the two. |
| | |
| | * **PoPETs '27** makes AI use one of three **mandatory template sections** — //"NEW FOR 2027: Updated LaTeX 2027 PoPETs template with mandatory sections, mandatory use or desk reject"// — and treats //"Hallucinations, fabrication, omissions, plagiarism, and falsification"// as research misconduct, with //"Hallucinated references … grounds for desk rejection"//. |
| | * **IEEE S&P '27** requires a //"Generative AI usage considerations"// field on HotCRP, adopts IEEE's organisation-wide policy plus a local one, and — unlike the others — covers generative AI **as an object of study** as well as a writing aid. The LLM policy and the ethics-considerations section were both //newly added// in the 2026 cycle. |
| | * **USENIX Security '27** attaches a specific desk-reject trigger: a violation //"which results in fabrications or hallucinations, including non-existing references or incorrect authors, invented claims, and falsified results is considered academic misconduct and might lead to the paper being desk rejected or other sanctions"//. |
| | * **NDSS '27** permits it under a three-part test — no plagiarism or falsification, the work is //"not primarily the result of the tools' generative capabilities"//, and the authors accept responsibility for //"the veracity and correctness of all material"// — and requires disclosure at submission. |
| | * **CCS '26, IMC '26 and TheWebConf** defer to the **ACM Policy on Authorship**: AI cannot be an author, and CCS requires a dedicated //"Generative AI Usage"// section if AI generated or substantially rewrote substantive content. CCS and IMC also bar **reviewers** from uploading submissions to public AI tools; at CCS that is referable to the ACM Publications Board as misconduct. |
| | |
| | All read 2026-08-18 from the calls for papers already cited above. |
| | |
| | Two things this means for a measurement paper specifically. If you used an LLM **as an instrument** — to classify cookies, label banners, read privacy policies, decide whether a site is in scope — that is a methods decision and a disclosure, not a writing aid, and IEEE S&P's phrasing ("as an object of study") is the only one that clearly covers it. And if the model touched anything you collected, the ethics section and the AI section now interact: sending crawled content, participant text or vulnerability details to a hosted model is a disclosure of that data to a third party, and none of these policies says so for you. |
| | |
| | <WRAP todo>The corpus cannot yet say how often LLM-based measurement pipelines disclose this: ''tools[]'' has an ''llm'' category, but there is no schema field for "was AI use disclosed", and the venue requirements are newer than most of the corpus. A full-text sweep would be the way in.</WRAP> |
| | |
| | ===== Harm from crawling, and what the field does about it ===== |
| | |
| | ==== What papers actually do, ranked ==== |
| | |
| | Of the 1,120 crawling papers, 697 describe a concrete mitigation. Folded into families — the fold is **multi-label**, because one sentence routinely names three steps, so the shares do not sum to 100%: |
| | |
| | ^ Mitigation family ^ Papers ^ Share of 697 ^ |
| | | Data minimisation — collecting less than you could | 180 | 25.8% | |
| | | Rate limiting, pacing and load control | 164 | 23.5% | |
| | | Disclosure and notification of affected parties | 123 | 17.6% | |
| | | Anonymisation, pseudonymisation and hashing | 108 | 15.5% | |
| | | Staying non-intrusive: no exploitation, no tampering, no logins | 104 | 14.9% | |
| | | Test accounts, synthetic identities and the researchers' own assets | 92 | 13.2% | |
| | | Isolation: sandboxes, VMs, containers, lab-only deployment | 59 | 8.5% | |
| | | Access control, encryption and secure storage | 53 | 7.6% | |
| | | Deletion and retention limits | 50 | 7.2% | |
| | | Avoiding financial harm to operators and advertisers | 33 | 4.7% | |
| | | Consent, opt-in and opt-out mechanisms | 28 | 4.0% | |
| | | Screening and handling of harmful or illegal content | 25 | 3.6% | |
| | | Respecting robots.txt, terms of service and acceptable-use policies | 17 | 2.4% | |
| | | Formal review, legal advice and named legal regimes | 17 | 2.4% | |
| | | Identifying the crawler and offering an operator an opt-out //(hand-decided)// | 10 | 1.4% | |
| | |
| | Read this as a menu, not as a ranking of importance. The top four are what the field says most often; the bottom four are where the gap between what the field says and what a 2026 reviewer expects is widest. |
| | |
| | ==== Load, and the number that makes it concrete ==== |
| | |
| | Rate limiting is the second most common step and the one most often stated as a bare adjective ("we crawled politely"). It is worth a number. Five real ones from the corpus, each verified in the paper: |
| | |
| | ^ What was reported ^ Paper ^ |
| | | //"We crawl at a rate of 2.7 URLs per second to limit the load on the site"// | Zhang et al., NDSS 2014 {[zhang2014_dspin]} | |
| | | //"We hard-coded a 1 minute delay between subsequent page loads to avoid overloading any servers"// | Bashir et al., USENIX Security 2016 {[bashir2016tracing]} | |
| | | //"We made sure to not make more than ten queries per second, and we believe such a load would have a negligible impact"// | Almansoori et al., PoPETs 2022 {[almansoori2022_dualuse]} | |
| | | //"visited by our crawlers no more than 4 times per day"// | Zeng et al., IMC 2021 {[zeng2021_polls]} | |
| | | //"we disabled image downloads and limited the number of parallel crawl workers to 256. We slowed down our crawls by pausing the workers when they received HTTP 429 … or HTTP 503"// | Amos et al., TheWebConf 2021 {[amos2021privacy]} | |
| | |
| | Any of these is publishable; //"we rate-limited our crawler"// is not, because it does not let a reader tell your crawl apart from a scraper that took a site down. Note what the last one adds: a **reactive** rule. A fixed rate is a guess about the smallest server in your sample; backing off on 429 and 503 is a rule that adapts to it. |
| | |
| | One framing worth knowing, because it comes from the affected parties rather than from researchers: the operators interviewed by Hantke et al. {[hantke2024_redlines]} said their logs already show scans of this kind continuously, and that they did not expect additional harm from a researcher's — //"if researchers run non-invasive and well-pre-tested scans likely to get lost in the Internet's noise, the operators in our study might be willing to accept them"//. That is a defensible argument in a paper, and it has two conditions attached: non-invasive, and pre-tested. |
| |
| ==== robots.txt ==== | ==== robots.txt ==== |
| |
| * Is it meant for us? | ^ ''ethics.robotsTxt'' ^ Papers ^ Share of 1,120 crawling papers ^ |
| * Parsing | | ''respected'' | 15 | 1.3% | |
| | | ''ignored'' | 9 | 0.8% | |
| | | ''discussed'' | 29 | 2.6% | |
| | | ''not-stated'' | 939 | 83.8% | |
| | | no ethics record | 128 | 11.4% | |
| |
| ===== Legal aspects of crawling ===== | **53 of 1,120 crawling papers (4.7%) say anything at all about robots.txt**, and a full-text sweep for the literal string finds only 55 (4.9%) — so this is not an extraction artefact. It is the **lowest reporting rate of any field in the extraction behind this site**, below the next four: |
| |
| * Fraud | ^ Field ^ Population ^ Stated ^ Share ^ |
| * Trespass | | ''ethics.robotsTxt'' | 1,120 crawling papers | 53 | 4.7% | |
| * Breach of contract (terms of service) | | ''crawlConfig.headless'' | 1,120 crawling papers | 140 | 12.5% | |
| | | ''humanAnnotation.agreementMetric'' | 3,318 papers that coded data by hand | 512 | 15.4% | |
| | | ''crawlConfig.statefulness'' | 1,120 crawling papers | 219 | 19.6% | |
| | | ''crawlConfig.consentAction'' | 1,120 crawling papers | 349 | 31.2% | |
| |
| ===== Ethics approval ===== | The reason it is unresolved rather than merely unreported is that the file does not answer the question a researcher has. `robots.txt` is a directive to *automated indexers*; a research crawl that loads a page once, in the way a browser would, and never republishes the content is not obviously the addressee, and a strict reading excludes exactly the pages a measurement is about (a `Disallow: /` on a site's tracking-heavy subpages removes the finding). There is no consensus in this literature, and inventing one here would be dishonest. What is not defensible is silence: **decide, do the same thing throughout the crawl, and write the sentence.** If you honour it, say what share of your sample it removed, because that is a bias in your denominator, not a footnote. If you do not, say why and what you did instead — most commonly a rate limit strict enough that the file's purpose (protecting the server) is served by other means. |
| |
| | ==== Terms of service and acceptable-use policies ==== |
| |
| ===== Disclosing what you found ===== | Two different documents, both routinely skipped. |
| |
| Notifying the affected operators is the post-measurement half of research ethics, and it has its own methodology: how to obtain a contact at scale, what response rate to expect, how long to embargo, and what reviewers now require. See [[Practices:Notifying websites]]. | * **The site's terms of service** may prohibit automated access. Whether breaching them is unlawful is jurisdiction-dependent and has moved — see [[#What the law says, and does not]] for what //Van Buren// and //hiQ// did and did not settle. The short version is that in the US the computer-crime exposure narrowed and the **contract** exposure did not. The corpus contains 129 crawling papers (11.5%) that mention terms of service at all, which is the more telling number: the document that most directly governs the legality of a crawl is discussed in one paper in nine. |
| | * **Your own network's acceptable-use policy** is the one that will actually stop you, and it is the one nobody reads. **Five crawling papers in the corpus mention an acceptable-use policy at all.** What those policies say varies enormously — one national research network requires a day's notice before you scan, another has no scanning clause whatever (both quoted just below) — so the answer is genuinely local. Talk to your network operations team before the crawl, not after the abuse ticket. Seven papers in the corpus name a network operations, network services or network management department among the bodies that reviewed the study — which is a perfectly good thing to be able to write when no ethics board will. |
| |
| ===== Scientific sources for this page ===== | ==== Your network's acceptable-use policy is a real constraint ==== |
| |
| | Only five crawling papers in the corpus mention one, and the policies are not uniform. Three worth knowing about, all read on 2026-08-18: |
| |
| * https://publications.cispa.de/articles/conference_contribution/Where_Are_the_Red_Lines_Towards_Ethical_Server-Side_Scans_in_Security_and_Privacy_Research/25826794?file=46339201 | * **SURF (NL)** has the only genuinely scan-specific national-research-network policy found, and it reads like the Red Lines checklist: the scanning host must have //"a PTR record (reverse DNS) with a meaningful name … indicating this is an intentional scan host and who is responsible for it"//, and //"A webserver runs on the host … which serves a web page that makes explicit what the goals of the scans are and who can be contacted."//((SURF, //Richtlijnen voor scans//, [[https://servicedesk.surf.nl/wiki/spaces/WIKI/pages/195871200/Richtlijnen+voor+scans|servicedesk.surf.nl]], read 2026-08-18.)) If your NREN has a policy at all, this is what a good one looks like. |
| | * **Jisc/JANET (UK)** requires notice: //"Security Contacts must notify Jisc if undertaking penetration testing or scanning on the Janet Network from outside of the Janet Network at least 1 working day in advance."//((Jisc, //Janet Security Policy// MF-POL-007 Issue 5.1, now at [[https://www.jisc.ac.uk/janet/policies/security|jisc.ac.uk/janet/policies/security]]; the older ''community.jisc.ac.uk/library/janet-policies/security-policy'' path 301-redirects there. jisc.ac.uk returns HTTP 403 to scripted clients, so the text was read from a Wayback capture of 2026-08-14, on 2026-08-18.)) |
| | * **Internet2 (US)** has, in full: //"The Internet2 Network can be used for any legal purpose, so long as it does not interfere with or adversely affect the operation of the Internet2 Network or any network user, as may be determined by Internet2."//((Internet2 Network Acceptable Use Policy, [[https://internet2.edu/community/about-us/policies/internet2-network-acceptable-use-policy-aup/|internet2.edu]], read 2026-08-18.)) No scanning clause at all. Silence is not permission — it means the decision sits with your institution. |
| |
| /* | Two more that catch people out. **WHOIS and RDAP are contractually rate-limited**: Verisign's terms prohibit //"electronic processes that are high-volume and automated except as reasonably necessary to register domain names or modify existing registrations"//, which describes most bulk WHOIS enrichment in this literature.((Verisign WHOIS Terms of Use and RDAP Terms of Service, [[https://www.verisign.com/legal-center/whois-terms/|verisign.com/legal-center/whois-terms]], read 2026-08-18.)) And **cloud providers differ**: Azure dropped pre-approval for penetration testing of your own resources in June 2017 and lists port scanning of your own endpoints as permitted, while AWS permits testing of listed services without approval and prohibits DoS simulation and request flooding under that policy (it runs a separate opt-in programme for DDoS simulation) — in both cases only against resources //you// own, which is not what a measurement crawl does.((AWS, //Penetration Testing//, [[https://aws.amazon.com/security/penetration-testing/|aws.amazon.com/security/penetration-testing]]; Microsoft, //Penetration testing// on Azure, [[https://learn.microsoft.com/en-us/azure/security/fundamentals/pen-testing|learn.microsoft.com]], which names the //Microsoft Cloud Unified Penetration Testing Rules of Engagement// as authoritative — that document returns HTTP 403 to scripted clients and was not read directly. Both read 2026-08-18.)) |
| This is a comment not visible on the page. It outlines the syntax (for more, go to https://measuretheweb.org/wiki/syntax), especially that related to bibliography. Remove it once you created the page. If you use any citations (documented at bottom), keep the References section. | |
| |
| ===== Other sources ===== | ==== Actions with side effects ==== |
| |
| * Menlo report https://www.dhs.gov/sites/default/files/publications/CSD-MenloPrinciplesCORE-20120803_1.pdf | A read-only crawl is the easy case. As soon as you click, submit, register or purchase, the calculus changes, and the corpus's own mitigations show what the field considers the boundary: |
| * Ethics guidelines of all major conferences: | |
| * https://www.usenix.org/conference/usenixsecurity26/call-for-papers#ethics | |
| * https://sp2027.ieee-security.org/cfpapers.html#:~:text=Research%20Ethics%20Considerations | |
| * ... | |
| * AI guidelines of conferences (note huge difference between guidelines in sec/privacy vs AI) | |
| * https://sp2027.ieee-security.org/cfpapers.html#:~:text=Generative%20AI%20Policy,-As | |
| * https://petsymposium.org/authors-2027.php#submission-guidelines:~:text=PoPETs%202027%20AI%20Policy%20for%20Authors | |
| |
| ===== Header Level 2 ===== | * **Ad clicks cost someone money.** The corpus's answers are: never click, click each ad exactly once, or click and then **estimate and publish the cost you imposed**. Two worked estimates to copy the form of — Datta et al. {[datta2015_automated]} avoided per-click fees entirely and priced their per-impression footprint at //"about $400"// across the whole study; Subramani et al. {[subramani2020_push]} priced theirs from the standard CPM at //"USD 1.12"// for the worst-hit landing domain and USD 0.04 on average. Publishing the estimate is now the expected form, and it is a two-line calculation. |
| ==== Header Level 3 ==== | * **Registrations, purchases and orders** are where "measurement" becomes "transaction". The reported practice is fictitious identities, no real payment instruments, closing the accounts afterwards, and — where a purchase is unavoidable — the smallest possible number, disclosed. |
| | * **Form submission** on a live site can create records a human then has to clean up. The corpus's answer is syntactically valid but obviously invalid data, or not submitting at all. |
| | * **Consent banners** are an interaction, and clicking "accept" on 100,000 sites is a state change on 100,000 sites. See [[Privacy:Consent]]; the relevant point here is that the choice is a design decision with an ethics component, not a default. |
| |
| === Links === | ==== Identify your crawler and offer an opt-out ==== |
| |
| External links are recognized automatically: www.google.com, but if you want a link text: [[http://www.google.com|This Link points to google]]. | **Ten of the 697 papers that state a mitigation describe doing this** — the rarest thing on the table, and the cheapest credible thing on it. (Ten and not seventeen: a bare "we offered an opt-out" in a paper is as often an opt-out for a study participant or for the recipient of a notification email as it is for a site operator, so this row is decided by reading the paper, not by matching the string. The twenty-one candidates and the verdict on each are on the provenance page.) The version this literature has converged on, set out most fully by Hantke et al. {[hantke2024_redlines]} for server-side scanning and directly transferable to crawling: |
| |
| Internal links are created by using square brackets. You can either just give a [[pagename]] or use an additional [[pagename|link text]]. | - Put up a **study web page** explaining who you are, what you are doing, over what period, and how to make you stop. |
| | - Link it from a **custom HTTP header** on every request, and from your **User-Agent**. |
| | - Make the **reverse DNS** of your source addresses resolve to that page. |
| | - Keep the **source addresses fixed** for the whole study, so an operator can allow-list or block you. |
| | - Provide an **explicit opt-out** — ideally a form, not an email address you will stop reading. |
| |
| === Lists === | This trades some measurement realism for a large amount of goodwill, and it interacts with the vantage-point choices on [[Design:Crawling location]]: an identifiable crawler is a crawler that bot-management systems can treat differently, so say what you did and treat unexplained blocks as a possible artefact. |
| |
| Lists and their levels are decided by indentation (2 spaces = 1 level) | It is worth knowing that the industrial scanners are not a model here. Wu et al. {[wu2025_revealing]} traced 1,407 scanner IP addresses belonging to commercial device search engines and watched them against 28 honeypots for a year; they report that the engines //"often fail to provide transparency and do not allow users to opt out of scans"//, send malformed requests, and publish personally identifiable information and screenshots in their results. The bar this section describes is one academic measurement can clear and commercial scanning does not, which is a useful thing to be able to say in an ethics section. |
| |
| - this is 1. item | ===== What the law says, and does not ===== |
| - 2. item | |
| - nested a. item | |
| * bullet-point item | |
| |
| === Code === | * **US.** 45 CFR 46.102(e) defines a human subject; 46.104(d)(4)(i) is the separate exemption for //"secondary research uses of identifiable private information … [where] the identifiable private information … [is] publicly available"//. These are two different questions — the first is whether the Common Rule applies at all, the second is a carve-out if it does — and papers routinely conflate them. Note also that **Part 46 nowhere defines "publicly available"**, so an argument that rests on that exemption rests on an undefined term.((45 CFR 46, read from the eCFR renderer API on 2026-08-18. Absence of a definition confirmed by reading §§ 46.102 and 46.104 in full.)) |
| | * **EU.** There is no GDPR requirement for an ethics board. **Article 89** requires //"appropriate safeguards"// for research processing — naming pseudonymisation and data minimisation — and permits member states to derogate from data-subject rights //"in so far as such rights are likely to render impossible or seriously impair the achievement of the specific purposes"//. That derogation is a national-law question, not a self-certification.((Regulation (EU) 2016/679, Article 89(1) and 89(2), consolidated text on EUR-Lex (CELEX 02016R0679), read 2026-08-18.)) Horizon Europe separately makes an ethics self-assessment part of the grant agreement — //"The ethics self-assessment will become part of your Grant Agreement … and may thus give rise to binding obligations"// — so EU-funded projects carry an obligation their national law may not impose.((European Commission, //EU Grants: How to complete your ethics self-assessment//, version 2.0, 13 July 2021, [[https://ec.europa.eu/info/funding-tenders/opportunities/docs/2021-2027/common/guidance/how-to-complete-your-ethics-self-assessment_en.pdf|ec.europa.eu, PDF]], read 2026-08-18.)) |
| | * **National research-ethics statutes are mostly biomedical.** The Swiss Human Research Act is scoped to //"research concerning human diseases and concerning the structure and function of the human body"//; the Dutch WMO covers medical-scientific research and explicitly excludes file research. Neither reaches web measurement.((Swiss Federal Act on Research involving Human Beings (HRA, SR 810.30), Art. 2, official English translation, read 2026-08-18. Netherlands: CCMO (the statutory Central Committee on Research Involving Human Subjects), //Your research: is it subject to the WMO or not?//, [[https://english.ccmo.nl/investigators/legal-framework-for-medical-scientific-research/your-research-is-it-subject-to-the-wmo-or-not|english.ccmo.nl]], read 2026-08-18. Only these two national statutes were checked; a German counterpart was not.)) If your supervisor tells you "there's a law about this", check whether the law is a health-research law — it usually is. |
| | * **The EU AI Act does not reach research-only systems.** Art. 2(6) excludes systems //"specifically developed and put into service for the sole purpose of scientific research and development"//, and Art. 2(8) excludes pre-market R&D — but explicitly **not** //"testing in real world conditions"//. If you deploy a classifier against live sites and act on its output, read Art. 2(8) carefully.((Regulation (EU) 2024/1689 (AI Act), Art. 2(6) and 2(8), and Recital 25, on EUR-Lex, read 2026-08-18.)) |
| | * **The three older theories the field worries about are rarely the ones that bite.** //Trespass to chattels// requires actual impairment of the system, which a rate-limited crawl does not cause and which is exactly what your load number is evidence about. //Fraud// theories attach to what you say, not to what you fetch — creating an account with a false identity, or misrepresenting who you are to a support agent, is where a measurement crosses into that territory, and several papers in the corpus do both. //Breach of contract// is the live one, and it is the one below. This page is not legal advice, and none of these has a settled answer for research crawling; the point of naming them is that they are three different questions with three different mitigations, and "we crawled public pages" answers only the first. |
| | * **Who owns what you collected.** Copyright and database rights do not disappear because a page was served to you, and a licence question that is invisible while the data sits on your disk becomes the reviewers' question the moment you release a dataset or train on it. The corpus's own practice, where papers say anything, is to release derived measurements and withhold the crawled content; where a paper reuses somebody else's crawl it names the licence it is operating under — Common Crawl's Terms of Use is the one that recurs. Note the asymmetry with the rest of this page: the ethics of //collecting// are discussed far more often than the ethics of //redistributing//. If you did not collect the data yourself, the provenance question is sharper still, and {[thomas2017_ethical]} is the framework to reach for. |
| | * **US scraping law moved, and is narrower than it is often quoted as being.** //Van Buren v. United States// (2021) held that the CFAA's "exceeds authorized access" clause reaches only information a person's access does not extend to at all, not misuse of access they legitimately have.((//Van Buren v. United States//, 593 U.S. 374 (2021), slip opinion [[https://www.supremecourt.gov/opinions/20pdf/19-783_k53l.pdf|supremecourt.gov/opinions/20pdf/19-783_k53l.pdf]], read 2026-08-18.)) //hiQ Labs v. LinkedIn// (9th Cir., 18 April 2022) is cited far more loosely than it should be: on remand the panel **affirmed a preliminary injunction**, concluding that hiQ had //"raised a serious question as to whether the CFAA 'without authorization' [clause]"// reaches scraping of public profiles.((//hiQ Labs, Inc. v. LinkedIn Corp.//, No. 17-16783 (9th Cir. 18 April 2022), [[https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/17-16783.pdf|cdn.ca9.uscourts.gov/…/17-16783.pdf]], read 2026-08-18.)) That is a preliminary-injunction standard on one claim in one circuit, not a merits holding that scraping is lawful. **And the case did not end there.** Eight months later, on 8 December 2022, the district court entered a **consent judgment and permanent injunction**: judgment of //"$500,000 USD … against hiQ and in favor of LinkedIn"//, and hiQ permanently enjoined from //"using automated means to access and/or copy data from the LinkedIn platform, whether logged in to a LinkedIn account or not, without express written permission"// and from //"creating or using accounts with fake identities"//. The case was terminated the next day.((//hiQ Labs, Inc. v. LinkedIn Corp.//, No. 3:17-cv-03301-EMC (N.D. Cal.), docket entry 406, Consent Judgment and Permanent Injunction, filed 8 December 2022; docket terminated 9 December 2022. Read from the CourtListener/RECAP copy of the filed document, [[https://www.courtlistener.com/docket/6071320/hiq-labs-inc-v-linkedin-corporation/|courtlistener.com/docket/6071320]], on 2026-08-18.)) The CFAA claim narrowed; **the contract claim is what the final judgment rests on**. That is the sentence to remember, and note that "creating or using accounts with fake identities" is a technique this page describes several papers using. |
| |
| For a short inline monospace, use ''double quote''. For proper code (but in a separate paragraph), use ''<code LANG>'': | Taking a //finding// to a regulator is [[Practices:Legal enforcement]]'s territory. The legality of the //crawl itself// is this page's, and the point is that "it's public, so it's fine" was never a complete answer, and that the part of it that survived is the part nobody reads: the contract. |
| |
| <code python> | ===== Server-side scanning: the one place this field has a real checklist ===== |
| string = "World" | |
| print(f'Hello {string}') | |
| </code> | |
| |
| For large code, use ''<file LANG filename>'', it will make code downloadable. For instance: | If your measurement sends anything a server might act on — probing for a vulnerability, testing access control, fuzzing a parameter — you are in the one corner of this topic where the community has done the empirical work and produced concrete guidance. |
| |
| <file php example.php> | Hantke et al. {[hantke2024_redlines]} conducted 23 interviews — 9 legal experts, 10 website and server operators, 5 members of conference research-ethics committees, //"based in Germany, the UK, the US, and Switzerland"// — and surveyed 119 operators, about five concrete scanning scenarios. (German law is the paper's case study for the legal analysis; the interviewees are not all German.) Three findings a student should carry: |
| <?php echo "hello world!"; ?> | |
| </file> | |
| |
| === Figures === | * **There is no agreed red line, and that is the finding.** One interviewee named //"fuzzing random stuff"//, intentionally damaging someone's business, and extracting personal information; the paper's next sentence reports another saying //"the red lines are not clear"//. What the operators were closer to agreeing on is the remedy: most would not consider legal action, treating it as a last resort after direct communication fails. |
| | * **Asking every operator for consent is not practical** at measurement scale, and the paper says so plainly: it would introduce observer effects and select for security-aware operators. This is a rare, citable statement that a *desirable* ethical step is infeasible, which is exactly what you need when a reviewer asks why you did not obtain consent. |
| | * **Ethics review at these venues is a post-mortem.** A conference committee reviews the paper after the experiment ran; if it went wrong, the harm is done. Their proposal is a pre-registration board — a trusted third party that reviews scanning proposals ex-ante, assigns time slots, and publishes a list of ongoing scans with contact points. It does not exist yet. Until it does, the ex-ante review is you. |
| |
| To use floats, you have to use the ''<WRAP>'' tag. For instance, the following will create figure on the right side 50% large: | Their operational best practices — laboratory pre-study first, data minimisation, minimal data manipulation, resource minimisation, **monitoring that the scanned sites are still up and pausing if one is not**, transparency via a study page and matching reverse DNS, fixed source addresses, explicit opt-out — are the closest thing this field has to a checklist, and they generalise to any measurement that touches live infrastructure. |
| <WRAP right 50% box> | |
| {{PATH_TO_FILE|ALT_TEXT}} | |
| <div>CAPTION</div> | |
| </WRAP> | |
| |
| */ | For a worked example of the same reasoning applied end to end, read El Hajj Chehade et al. {[chehade2025_forbidden]}, who test broken access control on 100 live web applications. Their ethics section is a **stakeholder analysis** in the Menlo sense — end users, operators, the research team — with the risks and benefits enumerated per stakeholder and each design decision traced to one of them (two accounts per site, both created by the researchers, swapping identifiers between them, so no third party's data is ever accessible). Copy the structure. |
| |
| ====== References ====== | ===== Where your traffic comes from is an ethics question ===== |
| |
| /* | ==== Residential proxies ==== |
| To insert citations, follow these steps: | |
| |
| - Verify the BibTeX entry exists in https://measuretheweb.org/literature/bibliography. If not, add it there. | A residential proxy resolves your requests through someone else's home connection. Whether that person knowingly agreed is the question, and the academic answer is not reassuring: Mi et al. {[mi2019_resident]} characterised residential IP proxy networks as a "dark service" with hosts enrolled without meaningful consent, and the follow-up work on mobile proxy networks {[mi2021_your]} found the same pattern on phones. **This is a human-subjects problem wearing a procurement invoice.** Twenty-five papers in the corpus mention residential proxies at all, and the best known of them — including both of the papers just cited — are studies //of// these networks rather than uses of them, which is a fair summary of where the literature's attention has gone. |
| - Use {[CitationKey]} where needed in the text; it will render as a numbered reference. | |
| - Keep this section unchanged to display the bibliography. | |
| |
| If any step fails, a purple warning will appear on the preview page. | [[Design:Crawling location]] carries the current state of the market and the July 2026 enforcement action against one major provider. The ethics-side rules of thumb: |
| */ | |
| | * Put the provider **in the paper by name**, with what it claims about consent and where that claim is published. |
| | * Take the consent claim to your ethics board **specifically** — not the study in general, the provider. If the board has declined jurisdiction over the study, this is still the question to put in front of them, because it is the one part that genuinely involves people. |
| | * Ask whether you need one. Singh et al. {[singh2025_where]}, measuring web tracking across 23 countries in 2025, rejected residential proxies explicitly — partly because the business is "gray" and may run on compromised IoT devices, partly for measurement reasons — and recruited volunteers instead. |
| | * Note the practical hierarchy: **datacentre IPs from a proxy vendor are not residential IPs from the same vendor**, and the ethics question is mostly about the second. |
| | |
| | Two things to know before you take a vendor's consent claim to a board. **No provider publishes an independently auditable consent framework.** The closest thing on the market is accreditation by the Ethical Web Data Collection Initiative, whose members include Oxylabs and Decodo — but the accrediting body and the auditor are the same trade association, the audited companies are its paying members, and no audit report, sampling methodology or enrolment evidence is published. Bright Data, NetNut/Alarum and IPRoyal are not members. Vendors' own claims are strong and unverifiable in the same way: Bright Data states that //"Residential IP addresses are linked to 100% real people who opted in to the Bright Data network"// and, for zones created after 7 July 2026, gates residential access behind human KYC review; Oxylabs describes a self-defined tier system in which //"a majority of Oxylabs' proxy network is composed of Tier A+ model proxies"// — its top tier, consented and paid — with no published breakdown behind "a majority".((Bright Data proxy-network documentation, [[https://docs.brightdata.com/proxy-networks/residential/network-access|docs.brightdata.com]], read 2026-08-18. Oxylabs' general sourcing language is on its [[https://oxylabs.io/legal/ethics-code|ethics code]] page, but the tier framework and the "a majority" claim are on a separate company post, [[https://dev.to/oxylabs-io/oxylabs-residential-proxy-pool-handbook|Oxylabs Residential Proxy Pool Handbook]], both read 2026-08-18. EWDCI members and audit mechanics: [[https://ethicalwebdata.com/ewdci-our-members/|ethicalwebdata.com/ewdci-our-members]] and [[https://ethicalwebdata.com/ewdci-our-principles/|/ewdci-our-principles]]; both return HTTP 403 to scripted clients and were read in a browser.)) Read those as claims to be assessed, not as findings. |
| | |
| | **And no published checklist exists for how a board should assess such a claim.** A full-text search of the USENIX Security, IEEE S&P and PoPETs ethics guidance on 2026-08-18 returns **zero** occurrences of "proxy", "residential" or "VPN" — the three venues whose guidance is detailed enough to have contained one — and nothing was found in the regulatory literature either. That absence is itself the finding, and it matches what researchers report doing: an interview study of sixteen EU internet-measurement researchers found that vendor due diligence is largely delegated to the vendor, and that //"researchers reported that formal ethical frameworks and oversight bodies play only a limited role in their decision-making. Many felt that IRBs lacked the technical expertise to identify hidden risks."//((Amin, Athar, Feldmann, Dao and Kaur, //Navigating the Ethics of Internet Measurement: Researchers' Perspectives from a Case Study in the EU//, arXiv:2511.10408, [[https://arxiv.org/abs/2511.10408|arxiv.org/abs/2511.10408]], v2 of 19 January 2026, read 2026-08-18. A preprint, not peer-reviewed at the time of reading.)) If you cannot state a defensible basis for the consent claim yourself, nobody downstream of you will. |
| | |
| | ==== Volunteer devices and data donation ==== |
| | |
| | The moment you ship code to a volunteer's machine, you are back inside the Common Rule and inside the GDPR, and the "not human subjects" shortcut is gone. That is the cost. The benefit is realism no crawl can buy: real profiles, real geography, real network conditions {[zeber2020representativeness]}. |
| | |
| | Singh et al. {[singh2025_where]} is the current worked example, and its ethics section is a good template: volunteers' machines are used **only as vantage points** with no access to pre-existing data on them; browser instances are isolated from the volunteer's own profile and history; a consent document explains what is recorded and how it is stored; participation is voluntary with withdrawal at any time; volunteers may **opt out of individual websites** in the target list; the only identifier collected is the IP address, and it is anonymised after analysis. Twenty-two volunteers covered 23 countries. |
| | |
| | Note two things a reviewer will ask. First, **your country coverage is your volunteers' countries** — a volunteer panel is a convenience sample, and the paper says so. Second, running the measurement on someone's own machine means their bandwidth, their electricity and their risk: if the target list contains anything that could get a person in trouble in their jurisdiction, the per-site opt-out is not a nicety. |
| | |
| | For infrastructure rather than a bespoke panel, ICLab {[niaki2020_iclab]} is the long-running example of a volunteer-operated measurement platform designed so that the operator's identity is not collected; the same design question — what can the volunteer be blamed for? — governs any deployment of this kind. See also [[Design:Crawling location]] for the platforms available and [[Design:User studies]] for recruitment, compensation and consent mechanics. |
| | |
| | **Do not plan around a platform that no longer exists.** Checked on 2026-08-18: **Mozilla Rally is dead** — its GitHub organisation was //"archived by an administrator on Jul 29, 2024"// and `rally.mozilla.org` now redirects to `mozilla.org`, and we found no shutdown announcement on any Mozilla channel. The Markup's **Citizen Browser** panel has published nothing since October 2022. AlgorithmWatch's **DataSkop** shows no activity after its 2022–2023 TikTok campaign and has not been formally closed. **Tracking Exposed** ended at the start of 2023 and became [[https://aiforensics.org/|AI Forensics]], which is alive and running algorithmic audits. Neither **Prolific** nor **Mechanical Turk** has a policy that squarely addresses installing research software on a participant's own device — Prolific requires you to disclose //"anything unusual the participant might have to do, such as downloading software"//, and MTurk's acceptable-use policy bans only software //"that contains harmful content (e.g., spyware, keyloggers, rootkits, or similar malware)"//. That is a gap in both directions: nobody has said yes, and nobody has said no. **And MTurk itself is closing to new work**: its own site now carries the banner //"Amazon Mechanical Turk is no longer accepting new customers. We recommend existing customers migrate to a third-party solution."//((Mozilla Rally: [[https://github.com/mozilla-rally|github.com/mozilla-rally]] ("archived by an administrator on Jul 29, 2024") and [[https://rally.mozilla.org|rally.mozilla.org]], which 301s to mozilla.org. The Markup: [[https://themarkup.org/series/citizen-browser|themarkup.org/series/citizen-browser]]. DataSkop: [[https://algorithmwatch.org/en/dataskop/|algorithmwatch.org/en/dataskop]]. Tracking Exposed / AI Forensics: [[https://tracking.exposed/|tracking.exposed]] and [[https://aiforensics.org/about|aiforensics.org/about]]. Prolific: [[https://researcher-help.prolific.com/en/articles/445126-study-details|researcher-help.prolific.com]]. MTurk: [[https://www.mturk.com/acceptable-use-policy|mturk.com/acceptable-use-policy]], where the banner is on the live page. All read 2026-08-18.)) |
| | |
| | The one route that got **better** is regulatory. The EU's **DSA Article 40 researcher data access** delegated act was adopted on 2 July 2025 and the Data Access Portal is operating: by 19 May 2026 the national Digital Services Coordinators had received **49 applications**, mostly for social-media platform data.((European Commission, //Commission adopts delegated act on data access under the Digital Services Act// and the May 2026 roundtable readout, [[https://digital-strategy.ec.europa.eu/en/news/commission-adopts-delegated-act-data-access-under-digital-services-act|digital-strategy.ec.europa.eu]], read 2026-08-18.)) If your target is a very large platform rather than the open web, that is now a real alternative to building a panel — and one with no volunteer to put at risk. Two constraints before you plan around it: the route is only open to **vetted researchers**, accredited through the Digital Services Coordinator of an EU member state, and it reaches only designated very large platforms and search engines, not the web at large. |
| | |
| | ===== Disclosing what you found ===== |
| | |
| | Notifying the affected operators is the post-measurement half of research ethics, and it has its own methodology: how to obtain a contact at scale, what response rate to expect, how long to embargo, and what reviewers now require. See [[Practices:Notifying websites]]. |
| | |
| | Two numbers to place it. Of the 4,472 empirical papers carrying an ethics record, **1,636 (36.6%) report notifying affected parties** and a further 524 (11.7%) did so partially — so nearly half the field reports something here, against a third that reports a review outcome. Ramulu et al. {[ramulu2025_ethics]} found the same ordering by hand: vulnerability disclosure is the ethical topic security papers discuss most, named in 257 of their 1,154 papers. Contacting a **regulator**, by contrast, is reported by 147 (3.3%); that route is [[Practices:Legal enforcement]]. |
| | |
| | ===== A reporting checklist ===== |
| | |
| | What to put in the paper. Every item here is something the corpus shows a large share of papers omitting. |
| | |
| | - **Which body you asked, its name and kind, and what it said** — including "it declined jurisdiction" and including "we have none, so we did X instead". Not just "IRB approved". |
| | - **Why the determination is right**, in one sentence: what data you collect, from whom, and why it is not identifiable private information. |
| | - **Who can be harmed**, by group: operators, end users, third parties in your infrastructure, people named in your results. A stakeholder analysis in the Menlo sense — {[chehade2025_forbidden]} is the model. |
| | - **Your load, as a number.** Requests per second, per site, per day; total requests; the pause rule on errors. |
| | - **Your robots.txt decision** and what it cost your sample. |
| | - **Your terms-of-service and acceptable-use position**, and whether your network operations team knew. |
| | - **How you identified the crawler**, and how an operator could have made you stop. |
| | - **Any interaction with a side effect**: clicks, registrations, submissions, purchases, and their estimated cost. |
| | - **Your vantage point's provenance** — and for a residential proxy, the provider, its consent claim, and your board's view of that claim. |
| | - **Data handling**: what you collected, what you deleted, when, and who could read it. |
| | - **Disclosure**: to whom, when, what came back, and whether you embargoed. |
| | - **Where the data came from, if you did not collect it.** Provenance, licence, and — for anything of illicit origin — the reasoning in {[thomas2017_ethical]}. |
| | - **What you are releasing, and what that enables.** Not only "is there personal data in it": a scanner, an evasion technique or an attack tool is a dual-use release, and the harm question is about the capability, not the dataset. |
| | - **Any generative AI in the pipeline**, separately from any in the writing. Which model, what it decided, whether your data left your machine to reach it. |
| | - **What went wrong.** An incident reported in the paper — a site you took down, a scan you had to pause — is a stronger ethics section than a clean one, and Hantke et al. {[hantke2024_redlines]} recommend it explicitly. |
| | |
| | ===== Papers to read first ===== |
| | |
| | If you read three: {[ramulu2025_ethics]} for what the field actually does and why so little of it is written down; {[hantke2024_redlines]} for the only concrete, empirically grounded checklist this literature has, and for the operators' and ethics committees' own words; and {[chehade2025_forbidden]} for a worked stakeholder analysis you can copy the structure of. |
| | |
| | Then {[thomas2017_ethical]} for the question of working with data you did not collect and might not lawfully hold — the framework there is the one to reach for whenever the dataset came from a leak, a breach or a scrape someone else did; {[snyder2017_fifteen]} for the two-sentence cautionary tale about "public" data; {[singh2025_where]} for a current volunteer-panel design done carefully; {[bowser2015_supporting]} for an early proposal for a web-specific ethics review, which is worth reading mainly to see how long this gap has been visible; and {[wu2025_revealing]} for what large-scale scanning looks like when nobody applies these norms — a year of honeypot observation of commercial device search engines. |
| | |
| | ===== Related pages ===== |
| | |
| | * [[Practices:Notifying websites]] — the operator-facing channel: contact discovery, response rates, embargoes, control groups. |
| | * [[Practices:Legal enforcement]] — taking a finding to a regulator, and the regulator opening a file on you. |
| | * [[Design:Crawling location]] — vantage points, residential proxies and volunteer devices from the measurement side. |
| | * [[Design:User studies]] — once there are participants, the apparatus on this page is the least of it. |
| | * [[Programming:Crawler]] — where the rate limit and the User-Agent actually get set. |
| | * [[Privacy:Consent]] — interacting with a banner is an interaction. |
| | * [[Design:Website selection]] — a robots.txt exclusion is a change to your sampling frame. |
| | * [[:Artifacts]] — what you can release once the data has people in it. |
| | * [[Statistics:Interrater agreement]] — hired annotators and crowdworkers are participants; 5.1% of the corpus's hand-coding papers use them. |
| | |
| | ===== Methodology and limitations of these figures ===== |
| | |
| | Every corpus figure above comes from ''scripts/report_ethics.mjs'' over ''data/extract/run1'' (5,859 papers, 2010–2026). Five limits carry directly: |
| | |
| | * **Seven venues only.** EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS are absent. CHI and SOUPS in particular are where the human-subjects methodology literature lives, so the picture here understates the field's ethics discourse and reports only what the security and measurement venues put in print. |
| | * **"States a review outcome" is a low bar and a proxy.** It counts a paper that says a review was not required exactly as it counts one that reports approval. It is a measure of *disclosure*, not of *conduct* — a silent paper may have done everything right, and this page never counts silence as misconduct. |
| | * **Sentinels are values, not answers.** ''none-mentioned'' and an absent ethics record are reported as their own rows everywhere and are never merged into a stated outcome. Merging them is the single easiest way to turn "how often the field reports this" into a much rosier number. |
| | * **Posters and short records bias the silence figures upward**, because a four-page record has its methodology compressed out. Dropping the 198 posters and ≤4-page records moves the empirical reporting rate from 33.8% to 34.8% and the crawl-only rate from 21.3% to 21.6%. The gap this page is about survives the check. |
| | * **Free-text fields are folded, not counted verbatim.** ''ethics.reviewBody'' and ''ethics.harmMitigation'' are unstable strings; the folds, their rules and their full unmapped residue are on the provenance page. The harm-mitigation fold is multi-label by construction. |
| | |
| | The complete query log, the report script with its unedited output, both folds with their residue printed in full, every quote with its verification verdict, and every external source with how it was verified are on **[[provenance:practices:ethics]]**. Corpus-wide caveats are on [[Literature:Corpus]]. |
| | |
| | ====== References ====== |
| |
| <bibtex bibliography></bibtex> | <bibtex bibliography></bibtex> |
| /* This enables discussion under this article. */ | /* This enables discussion under this article. */ |
| ~~DISCUSSION~~ | ~~DISCUSSION~~ |
| | |
