| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| research_journey [2025/03/03 16:22] – conferences karelkubicek | research_journey [2026/09/07 07:03] (current) – Rewrite: re-scoped from a WiP outline to the decisions that are expensive to revisit, every step linking one page. Authored by Claude karel.kubicek.claude |
|---|
| ====== Research Journey ====== | ====== Research Journey ====== |
| |
| At the moment, this page is WiP and only relevant to automated measurements. Maybe we have to create the same page for manual studies (I am not covering them at the wiki at all now). The plan is that each point links to some page. | The other pages on this site are organised by **what a thing is**. This one is organised by **when you have to decide it**, and it only lists decisions that are expensive or impossible to revisit later. A choice you can change on a Tuesday afternoon is not here; a choice that costs you a re-crawl, a re-annotation, or a paragraph in //Limitations// is. |
| |
| Note that some aspects can repeat in multiple steps of the research journey | This page is about automated measurements. If your study recruits people rather than crawling sites, [[Design:User studies]] is the entry point, and the ordering below mostly does not apply — but the first three items still do. |
| |
| - Pilot study and topic exploration | ===== Before you collect anything ===== |
| * Manual exploration (existing inspection tools) | |
| * Literature review | |
| * Study preregistration | |
| * [[practices:ethics|Ethics approval]] | |
| - Development | |
| * Crawling technologies | |
| * OpenWPM | |
| * Tracker Radar Collector | |
| * Custom crawler | |
| * [[programming:interaction|Interacting with websites]] | |
| * Stateful and stateless crawling | |
| * Techniques for automated interaction | |
| * [[privacy:consent|Granting consent to websites]] | |
| * Registration and login to websites | |
| * Data collection for privacy and security tasks | |
| * Instrumentation of requests, cookies, JavaScript calls | |
| * Collection of emails and other messages | |
| * [[programming:multilingual_support|Multilingual support]] | |
| * Supporting specific platforms (Facebook, TikTok, Amazon, etc.) | |
| - Execution of experiment, deployment | |
| * [[design:website_selection|Websites selection]] | |
| * Crawling location (VPNs, proxies) | |
| * Crawling using archival data | |
| * Deployment issues | |
| * Executing long experiments | |
| * Monitoring experiments | |
| - Processing results | |
| * Classification of privacy and security issues | |
| * Classifying [[privacy:requests|requests]] | |
| * Classifying [[privacy:cookies|cookies]] | |
| * Classifying fingerprinting | |
| * Classifying JavaScript code | |
| * Classifying Dark patterns | |
| * Classifying emails and other messages | |
| * [[design:website_classification|Website classification]] | |
| * Statistics | |
| - Submission | |
| * Conferences | |
| * Writing tips, literature review | |
| * Artifacts | |
| * Rebuttals (external recommendations by [[https://medium.com/@stub_AS/how-to-write-a-rebuttal-31e93c4f4b3f|Ankit Singla]] and [[https://matt.might.net/articles/peer-review-rebuttals/|Matt Might]]) | |
| - After publication | |
| * Notifying websites (disclosure) | |
| * Enforcement: Working with regulatory authorities | |
| * Public relations | |
| * Conference (external recommendations by [[https://homes.cs.washington.edu/~mernst/advice/conference-attendance.html|Michael Ernst]] | |
| - Other research practices | |
| * [[https://github.com/pliang279/awesome-phd-advice|PhD and research advice]] | |
| |
| | Everything here is either impossible to add retroactively, or only credible if it predates the data. |
| |
| /* | - **[[Practices:Ethics|Ethics approval]]** — the review that has to happen before the crawl, not before the submission. Some venues now ask what your board said. |
| ====== References ====== | - **[[Statistics:Study preregistration|Preregistration]]** — the only way a confirmatory claim stays confirmatory. A crawl generates hypotheses as fast as it generates rows; a plan deposited afterwards cannot tell the two apart. |
| | - **[[Writing:Literature review|Literature review]]** — done now it tells you what to measure; done at writing time it tells you what you should have measured. Watch the keyword-derived denominator: the set of papers your search returns is not the set of papers that exist. |
| | - **[[Artifacts|What you will be able to release]]** — decide before collection, because consent, withholding and the shape of the deposit all constrain what you may record. Retrofitting a releasable dataset onto a finished crawl is the common way a badge is lost. |
| |
| To insert citations, follow these steps: | ===== Fixing the frame ===== |
| |
| - Verify the BibTeX entry exists in https://measuretheweb.org/literature/bibliography. If not, add it there. | These four decisions define the population every later number is about. Changing any of them invalidates the data you already have. |
| - Use {[CitationKey]} where needed in the text; it will render as a numbered reference. | |
| - Keep this section unchanged to display the bibliography. | |
| |
| If any step fails, a purple warning will appear on the preview page. | - **[[Design:Website selection|Which list stands in for the web]]** — and which //version// of it. Record the list id on the day you drew it; [[Programming:Tranco|Tranco]] ids are the reason this is possible at all. |
| */ | - **[[Design:Sampling|How you draw from that list]]**, and **[[Statistics:How many sites|how many]]**. Top-//n// and stratified answer different questions, and the size you need depends on the effect, not on what fits in the week. |
| | - **[[Design:Crawling location|Where you appear to be]]** — country, and datacenter versus residential. A single-vantage result is a result about one path, and the wrong vantage silently hands you the wrong legal regime. |
| | - **[[Design:Automated measurements|Crawl, scan or unpack]]** — the instrument //is// the design decision. Then [[Programming:Crawler|which crawler]], remembering that the instrument's own defaults are choices you are making. |
| |
| <bibtex bibliography></bibtex> | ===== The configuration you will be asked to justify ===== |
| |
| | Each of these is cheap to set and expensive to have set wrongly, because the answer is baked into every row you collect. Reviewers ask about all four. |
| | |
| | - **[[Programming:Stateful Stateless|Stateful or stateless]]** — whether the profile survives between page loads. No paper in this corpus runs a controlled comparison of the two, so you cannot cite your way out of stating what you did. |
| | - **[[Privacy:Consent|What the crawler did with the consent banner]]** — accepted, rejected, ignored, or never saw one. On the audited corpus only **4.9%** of crawling papers actually state this (55 of 1,120; see [[provenance:privacy:consent]] section 7.3, where the unaudited figure was 31.2%); it changes what the crawl observed more than most instrument choices do. |
| | - **[[Programming:Interaction|How deep you go, and what you touch]]** — landing page only, //n// subpages, or scripted interaction. Depth is a population decision disguised as an implementation detail. |
| | - **[[Programming:Traffic files|What you record]]** — requests, cookies, storage, script calls, a HAR. You can always analyse less than you recorded and never more; [[Privacy:Browser storage]] is the usual thing people wish they had captured. |
| | |
| | ===== If it repeats, or runs long ===== |
| | |
| | - **[[Design:Longitudinal|Pinning for wave two]]** — list, browser, vantage and classifier. All four have to be pinned //and recorded// in wave one, or wave two measures your toolchain instead of the web. |
| | - **[[Programming:Deployment|Surviving the run]]** — checkpoints, retries, monitoring, and what you will report when the machine dies on day four. Decide the failure accounting before the failures, so a partial crawl stays analysable. |
| | |
| | ===== Once you have the data ===== |
| | |
| | Still not fully reversible: re-annotating is a month, and a validation set has to be drawn honestly rather than chosen. |
| | |
| | - **[[Design:Website classification|Labelling]]**, then **validating the labels** — with humans, a classifier, or an LLM. Corpus-wide, **42.4%** of the 4,439 papers that classified something report no validation at all (see [[provenance:design:website_classification]]); [[Statistics:Interrater agreement|inter-rater agreement]] is what makes hand-coded ground truth a measurement. |
| | - **[[Statistics:Hypothesis testing|Your unit of analysis]]** — sites, requests, or third parties. Getting it wrong is the most common statistical error in this literature, and it is not fixable at revision time. |
| | - **[[Statistics:Biases|What the frame already did to the number]]** — top-//n//, one vantage, a missing denominator. Better named in your own //Limitations// than in a review. |
| | |
| | ===== After the paper exists ===== |
| | |
| | - **[[Practices:Notifying websites|Disclosure]]** — to site operators or vendors, and on what timeline. |
| | - **[[Practices:Legal enforcement|Enforcement]]** — taking a finding to a data protection authority, which is a different document from a paper. |
| | - **[[Practices:Public relations|Public relations]]** — the sentence a journalist will quote is one you can choose in advance. |
| | - **[[Writing:Conferences|Where to submit]]**, and the current CFP dates. External advice on rebuttals: [[https://medium.com/@stub_AS/how-to-write-a-rebuttal-31e93c4f4b3f|Ankit Singla]] (computer systems, closest to the security and privacy style), [[http://www.pl-enthusiast.net/2014/09/17/advice-writing-author-response/|Michael Hicks]] (programming languages), [[https://matt.might.net/articles/peer-review-rebuttals/|Matt Might]] (bioinformatics, long rebuttals); on attending the conference, [[https://homes.cs.washington.edu/~mernst/advice/conference-attendance.html|Michael Ernst]]. |
| | |
| | ===== What this page deliberately is not ===== |
| | |
| | It is not a table of contents — [[start]] and the seven namespace pages carry those, and a step here links one page rather than every page that touches the step. Several of these decisions recur (you will choose a vantage point again in wave two); they are listed at the point where getting them wrong costs the most. |
| |
| /* This enables discussion under this article. */ | /* This enables discussion under this article. */ |
| ~~DISCUSSION~~ | ~~DISCUSSION~~ |
| | |
