Differences

This shows you the differences between two versions of the page.

--- programming:interaction [2026/08/27 12:22] – Reviewer 2 (citations/quotes): replace the Musch and Johns paraphrase-in-quotation-marks with the paper's own sentence, and add its real denominator (206,676 sites with an indicator, 30,494 subpage-only) plus the per-technique spread the +17% aggregate hi karel.kubicek.claude
+++ programming:interaction [2026/08/27 12:39] (current) – Reviewer 4 (generic): add 'Interacting is not free' — click fraud, form side effects, load multiplied by 3x, and the visibility trade-off — and a Monday-morning default design; replace the unaudited 'none of them justified anywhere' with the measured just karel.kubicek.claude
@@ Line 8: / Line 8: @@
 <WRAP important>
-Three things a fresh measurement gets wrong most often.
+Three things this page exists to stop you getting wrong.
   - **"We crawled 10,000 websites" is not a measurement unit.** A website is not a page. Of the **857 papers in [[literature:corpus|this corpus]] that ran an automated web crawl of the web**, **417** give a value on the site-depth axis, and **155 of those 417 (37.2%)** stayed on the landing page. The landing page is a systematically atypical page — see [[#Deeper is not simply more]].
   - **Ten subpages is the folk default and it is not where the curve flattens.** The median stated ''subpagesPerSite'' in this corpus is **10**. The one paper that actually measured where the increase stops found cookies stabilising **after 100 subpages**, not ten {[urban2020beyond]}. See [[#Where the increase stops]].
-  - **Navigating is not interacting.** Almost nothing on a modern page loads at ''load''. In the corpus, **69 of 857 web crawls (8.1%)** say anything about scrolling and **28 (3.3%)** about filling a form. The measured ladder in [[#What a strategy can and cannot see|the probe below]] shows a beacon that only a scroll reveals and another that only a click reveals — on a fixture built to have them, but built that way because real pages are.
+  - **Navigating is not interacting.** A great deal of what a privacy measurement counts — lazily-loaded ad slots, content behind a tab or an accordion, anything a form submission triggers — does not exist at ''load''. In the corpus, **69 of 857 web crawls (8.1%)** describe scrolling in their own methods and **28 (3.3%)** describe filling a form. (Loosen the pattern to any mention anywhere in the paper and it is 120 and 59 — still small.) The measured ladder in [[#What a strategy can and cannot see|the probe below]] shows a beacon that only a scroll reveals and another that only a click reveals — on a fixture built to have them, but built that way because real pages are.
 </WRAP>
@@ Line 47: / Line 47: @@
 | ''not-stated'' | — | 136 |
-''single-target-page'' is the largest single value, and treating it as a depth answer is the easiest mistake to make with this data. It fires on website-fingerprinting trace collection, phishing-blocklist monitors, search-result scrapes and dashboard pulls. Those crawls have a page list, not a site list, so "how deep did you go" does not apply to them. **Every depth percentage on this page has a denominator of 417, not 857 and not 5,859.**
+''single-target-page'' is the largest single value, and treating it as a depth answer is the easiest mistake to make with this data. It fires on website-fingerprinting trace collection, phishing-blocklist monitors, search-result scrapes and dashboard pulls. Those crawls have a page list, not a site list, so "how deep did you go" does not apply to them. **Every percentage of the depth //distribution// on this page has a denominator of 417, not 857 and not 5,859.** (Reporting rates and on-page-action shares are of 857; each table says which.)
 There is also a terminology trap that a keyword search will not survive: in the phishing, malvertising and spam literature **"landing page" means the destination of a redirect or ad click**, not the root of a site. The same two words name the thing this page is about and the thing that page is about.
@@ Line 53: / Line 53: @@
 ===== How Deep the Field Actually Goes =====
-Depth is, by a wide margin, the best-reported crawl-configuration field in this corpus — better than browser, statefulness or headless mode. That is worth knowing before you decide it is not worth a sentence.
+Depth is the **best-reported** crawl-configuration field in this corpus, though only just: of the 1,120 crawling papers, 75.1% state a depth against 69.6% for authentication — and then the field falls off a cliff.
+^ ''crawlConfig'' field ^ Papers stating it ^ Share of 1,120 crawling papers ^
+| ''interactionDepth'' | 841 | 75.1% |
+| ''authentication'' | 779 | 69.6% |
+| ''browsers'' | 529 | 47.2% |
+| ''consentAction'' | 349 | 31.2% |
+| ''statefulness'' | 219 | 19.6% |
+| ''headless'' | 140 | 12.5% |
+That is worth knowing before you decide depth is not worth a sentence: it is the one crawl-configuration choice the field already has a habit of writing down, so omitting it stands out.
 ^ Bucket ^ Web crawls ^ States a depth ^ On the site-depth axis ^ Of those, went beyond the landing page ^ Gives a subpage count ^
@@ Line 64: / Line 74: @@
 Two readings, and they point in different directions.
-The good news: **about four in five web crawls say something about depth**, and that has been true since 2010. Compare it with statefulness, which 19.6% of crawling papers state ([[Programming:Stateful Stateless#Reporting has not improved in sixteen years|the flattest field in the schema]]).
+The good news: **about four in five web crawls say something about depth**, and that has been roughly true since 2010 — though the direction is very slightly down, from 83.8% in the first bucket to 79.1% in the last, which [[Programming:Stateful Stateless|the statefulness page]] reads as the one crawl-configuration field whose reporting rate has //declined//. Compare it with statefulness, which 19.6% of crawling papers state ([[Programming:Stateful Stateless#Reporting has not improved in sixteen years|the flattest field in the schema]]).
 The bad news: the second and third numbers barely move. Sixteen years after the field started worrying about this, **the share that goes beyond the landing page is not rising**, and **only 104 of 857 web crawls (12.1%) put a number on how many subpages** — 102 of the 417 on the depth axis, 24.5%. "We also crawled subpages" is not a reproducible statement, and it is the commonest form the claim takes.
@@ Line 85: / Line 95: @@
 | 100 or more | 17 | 16.3% |
-Median 10. The five commonest values are 5 (14 papers), 3 (12), 10 (12), 1 (11) and 20 (8) — all small, all round, and none of them justified anywhere. The tail above 100 is not "more of the same": those are deep crawls of a single application or forum (100, 200, 300, 500, 1,000 and 2,000 pages per target all appear), where the population is one site and the pages are the sample.
+Median 10, and it does not settle: per four-year bucket the median runs 5, 5, 10, 15, 5 on 8, 11, 37, 33 and 15 papers, which is small-sample noise rather than a trend. The five commonest values overall are 5 (14 papers), 3 (12), 10 (12), 1 (11) and 20 (8) — all small and all round. Whether they are //justified// is harder to establish than it looks, and a full-text sweep over the 104 puts an upper bound on it: **13 (12.5%)** cite Urban et al. anywhere in the paper, **12 (11.5%)** cite Aqeel et al. or Hispar, **18 (17.3%)** use the words //pilot// or //pre-study//, and **10 (9.6%)** use //saturation//, //diminishing returns// or //plateau// anywhere at all. Every one of those is an upper bound on justifying //this// choice, because a citation in related work is not a justification of a number in the methods section. The tail above 100 is not "more of the same": those are deep crawls of a single application or forum (100, 200, 300, 500, 1,000 and 2,000 pages per target all appear), where the population is one site and the pages are the sample.
-The honest summary is that **the field converged on "a handful of subpages" without measuring whether a handful is enough**, and the one paper that did measure says it is not.
+The honest summary is that **the field converged on "a handful of subpages" without measuring whether a handful is enough**, and the one paper in this corpus that did measure says it is not.
 ===== Deeper is Not Simply More =====
@@ Line 120: / Line 130: @@
 <WRAP important>
-**Per-page density and union coverage move in opposite directions.** The landing page is the most heavily instrumented //single// page on a typical site. The set of internal pages reaches //more distinct// third parties. Which of those your number is a count of decides whether crawling deeper raises it or lowers it — and it is the sentence almost no methods section contains. A per-page mean is not a prevalence, and a prevalence over sites is not a per-page mean.
+**Per-page density and union coverage move in opposite directions.** The landing page is the most heavily instrumented //single// page on a typical site. The set of internal pages reaches //more distinct// third parties. Which of those your number is a count of decides whether crawling deeper raises it or lowers it — and it is not something any of the reporting checklists asks for, so it is on you to say it. A per-page mean is not a prevalence, and a prevalence over sites is not a per-page mean.
 </WRAP>
@@ Line 199: / Line 209: @@
 | mentions bot or crawler detection | 61 | 7.1% |
-The only two rows with a clear trend across the sixteen years are **scrolling** — 2.5% of web crawls in 2010–2013 against 11.1% in 2025–2026 — and **bot detection**, which no paper in the 2010–2013 bucket raises and 11.8% of the 2025–2026 bucket does. Clicking is flat and has been since 2010. Mouse movement and hovering are, and have always been, rounding errors.
+The only two rows with a clear trend across the sixteen years are **scrolling** — 2.5% of web crawls in 2010–2013 against 11.1% in 2025–2026 — and **bot detection**, which not one paper in the 2010–2013 bucket raises //in a first-person sentence// against 11.8% of the 2025–2026 bucket. (Three of those 80 papers do mention it somewhere in the text — a probe's zero is never the same as nobody.) Clicking has **not** risen: it sits between 29.9% and 43.8% in every bucket, and the //highest// value is the earliest one. Mouse movement and hovering are, and have always been, under 4%.
 ==== What a strategy can and cannot see ====
@@ Line 229: / Line 239: @@
 Read off the last row. Each additional capability found exactly one thing the previous one could not, and no amount of extra depth substitutes for the scroll, nor extra scrolling for the click. **These are not degrees of the same knob.** A design that goes three levels deep without scrolling still misses everything below the fold.
+==== Interacting is not free, and it is not only your cost ====
+Everything above says //do more//. Before you scale it to ten thousand sites, four consequences that a landing-page crawl does not have:
+  * **A click on an ad is an ad impression somebody pays for.** Ad clicks are billed. A crawl that clicks indiscriminately at scale generates click fraud, and the money comes out of an advertiser's budget. Reitinger et al. designed around exactly this: rather than fetch an ad's destination link — which "could charge advertisers as a higher-cost //click//" — they parse the destination out of the URL instead {[reitinger2024_creepy]}. If you must resolve where an ad goes, resolve it without clicking it.
+  * **A form submission has a side effect on a real system.** Filling and submitting forms at scale sends real newsletter signups, real support tickets, real account registrations and real search queries against production databases. This is why EvoCrawl's evaluation runs against **ten self-hostable web applications** — WordPress, HotCRP, Kanboard, ImpressCMS, phpBB, Humhub, GitLab and others, each shipped as a Dockerfile in the repository's ''experiments/'' folder — rather than against the live web {[guo2025_evocrawl]}. The interaction depth it achieves is only safe on a system you own. Read the [[Practices:Ethics|ethics page]] before you point a form-filler at a site list.
+  * **Interaction multiplies your load by more than you expect.** The Khaleesi comparison above is 3× the requests from the same list. Rate limits, politeness delays and any ''robots.txt'' policy you have should be set for the interactive figure, not the landing-page one.
+  * **You are more visible, and being detected is not neutral.** A crawler that clicks and scrolls looks less like a bot to a fraud model and more like a user — which is the point, and is also why the resulting impressions and clicks are harder for the site to filter out. "We made the crawler look human" and "we made our traffic indistinguishable from the traffic the site bills for" are the same sentence.
+None of this argues for staying on the landing page. It argues for saying, in the paper, which interactions you performed, on how many sites, and what you did to keep them from being charged for.
 ==== The code ====
@@ Line 243: / Line 264: @@
 // Third parties are served from a SECOND origin (127.0.0.2) so the crawler
 // counts them as third-party the way a real measurement would.
+//
+// PORTABILITY: Linux gives you the whole 127.0.0.0/8 loopback range for free.
+// macOS does not — binding 127.0.0.2 fails with EADDRNOTAVAIL until you add the
+// alias:  sudo ifconfig lo0 alias 127.0.0.2 up
+// (undo with `sudo ifconfig lo0 -alias 127.0.0.2`). On Windows, use WSL.
 import http from 'node:http';
@@ Line 465: / Line 491: @@
 ===== Forms, Logins and the Half of the Web You Are Not Measuring =====
-Of the 857 web crawls in this corpus, **553 (64.5%)** explicitly did not authenticate, **204 (23.8%)** do not say, and **81 (9.5%)** got past a login of any kind — 42 by registering an account, 24 by logging in manually and 15 with automated login. Not one of the 857 used SSO.((The corpus has exactly one paper with ''authentication: sso'' — an Android subscription-app study — and it is not a web crawl, so it is outside this page's population.)) Whatever your topic, **a crawl of the logged-out web is the design nine papers in ten use**, and the reason is usually the one Lauinger et al. give: login walls stop deeper traversal.
+Of the 857 web crawls in this corpus, **553 (64.5%)** are recorded as not authenticating, **204 (23.8%)** do not say, and **at most 81 (9.5%)** got past a login of any kind — 42 by registering an account, 24 by logging in manually and 15 with automated login. Not one of the 857 used SSO.((The corpus has exactly one paper with ''authentication: sso'' — an Android subscription-app study — and it is not a web crawl, so it is outside this page's population.)) Whatever your topic, **at most one crawl in ten gets past a login**, and the reason is usually the one Lauinger et al. give: login walls stop deeper traversal. Treat the 553 with care: ''authentication'' shares the same single ''crawlConfig'' quote as the depth, and no discriminant was run for it, so an unknown share of those ''none'' values are the extractor's default reading of a paper that simply never mentions logging in.
 Two things worth knowing before you decide the effort is not worth it.
@@ Line 477: / Line 503: @@
 A form-filler that assigns to ''input.value'' fires no ''keydown'', no ''keyup'' and no ''input'' event unless the framework listens for programmatic changes, so the very scripts you are trying to catch never run. If your measurement is about what happens //while// a user types, you must actually type. Playwright's ''locator.pressSequentially()'' and Puppeteer's ''ElementHandle.type()'' with a ''delay'' both do this; ''locator.fill()'' does not.
-The rest of the login problem — Shepherd-style automated login, SSO, and automated registration — is [[Programming:Registration|Automating Login and Registration to Websites]].
+The rest of the login problem — Shepherd-style automated login, SSO, and automated registration — is [[Programming:Registration|Automating Login and Registration to Websites]], **currently a stub**, so expect to go to the papers directly for now.
 ===== Your Crawler Will Be Noticed =====
-Interaction is the axis on which a crawler most visibly stops resembling a user, and the share of papers that say so is rising: **no paper in the 2010–2013 bucket mentions bot or crawler detection, against 11.8% of the 2025–2026 bucket.**
+Interaction is the axis on which a crawler most visibly stops resembling a user, and the share of papers that say so is rising: **not one of the 80 web crawls in the 2010–2013 bucket describes bot or crawler detection in a first-person sentence — three mention it in passing — against 11.8% of the 2025–2026 bucket.**
 The concrete failure modes to plan for:
@@ Line 497: / Line 523: @@
 ^ Method ^ Status ^ Evidence ^
 | **Landing-page-only crawls of a ranked list** | **Still the single commonest design, and still defensible for the right question** — but no longer defensible //silently//. Since 2020 there is a citable reason to justify it or drop it | 155 of 417 depth-stating web crawls; {[aqeel2020_landing]}, {[urban2020beyond]} |
-| **"Plus a handful of random subpages"** | **Current practice.** Median 10, unchanged across all five buckets, and not derived from any measurement | 104 papers give a count; median 10 |
+| **"Plus a handful of random subpages"** | **Current practice**, and not derived from any measurement. The overall median is 10; per four-year bucket it is 5, 5, 10, 15 and 5, on 8, 11, 37, 33 and 15 papers — too few to call a trend either way | 104 papers give a count |
 | **A saturation study to choose the count** | **Done once, in 2020, and not replicated.** If you do it for your own metric you will be the second | {[urban2020beyond]} |
 | **A purpose-built internal-page top list** | **Superseded by circumstance.** Hispar is offline and its bootstrap list is retired. The HTTP Archive's one-secondary-page-per-site is what remains, with a known selection rule | {[aqeel2020_landing]}; checks of 2026-08-27 |
@@ Line 525: / Line 551: @@
   - **The failures.** How many target sites yielded zero pages, how many yielded fewer subpages than you asked for, and how many hit a challenge you could not pass. All three are non-random.
   - **The comparison, if you make one.** If you compare against a published prevalence, check the other paper's depth first. A 3× difference in request counts is available from the depth choice alone {[iqbal2022_khaleesi]}.
+===== What a Complete Statement Looks Like =====
+Two sentences from Aziz and Wilson are, as far as this corpus goes, the model {[aziz2024_johnny]}:
+> For each domain, we programmed the crawler to load the domain's homepage, scroll to the bottom of the page, then sleep for 25 seconds. Further, we programmed our crawler to select nine internal hyperlinks at random from the homepage and crawl them using the same load, scroll, and sleep approach.
+Five of the seven items on the checklist above are in those two sentences: the unit (domain), the depth (homepage plus nine), the selection rule (at random, from the homepage), the on-page action (scroll to the bottom) and the wait (25 seconds, stated). Plus the detail that most often silently differs and is on none of the checklists: the subpages get the **same** load-scroll-sleep treatment as the landing page, which is what makes a landing-versus-subpage comparison a comparison rather than two different instruments. The two items it does not cover are the consent treatment — which is elsewhere in that paper, whose whole design is four experimental conditions — and the failure counts. Splitting them across the methods section is the usual and fine arrangement; leaving them out is not.
+Five of seven, in two sentences. That is the whole cost.
+===== If You Just Need a Default =====
+Absent a better reason — and the sections above are the better reasons — this is a defensible starting design for a privacy or security crawl over a ranked site list, and every element of it is something you can point a reviewer at:
+  - **Harvest the frontier once.** Visit each landing page, collect its same-site links, store them with the date. Freeze that list and replay it in every arm of the experiment.
+  - **Pick subpages at random from it, with a published seed**, and publish the exclusion list if you use one. Do not take them in DOM order unless you want the navigation chrome.
+  - **Start at ten and run a saturation pilot for //your// metric** on a few hundred sites before committing to the number. Ten is the field's median and 100 is where the one published curve flattens; you will not know which is right for a metric nobody has plotted.
+  - **On every page, including the landing page: scroll to the bottom, then wait a fixed, stated time.** Same treatment on the subpages as on the landing page, or the comparison between them is not a comparison.
+  - **Click nothing by default.** Add clicking only where your question needs it, and when you do, exclude ad elements — see [[#Interacting is not free, and it is not only your cost|above]].
+  - **Crawl logged out**, and say so. Getting past a login is a project, not a setting.
+  - **Log the failures** — sites yielding zero pages, sites yielding fewer subpages than asked, challenges you could not pass — and report all three.
+The point of the default is not that it is optimal. It is that every line of it is a sentence you can write in the methods section, which is more than most published crawls manage.
 ===== Methodology and Limitations of These Figures =====
@@ Line 533: / Line 583: @@
   * **The denominator is 857, not 5,859.** Only 1,120 papers in the corpus ran a crawl at all, and 857 of those crawled the web. The depth ratios use a further-narrowed denominator of 417 for the reason given [[#Depth has more than three positions|above]].
-  * **''crawlConfig'' carries one evidence quote for the whole object.** The quote sitting behind an ''interactionDepth'' value very often evidences the browser or the statefulness instead, so the site's usual "read the evidence quote" check cannot validate a depth label. In place of it, the provenance page reports a **discriminant**: whether each paper's own text contains a first-person sentence naming the site root, or something below it, or both, or neither. The labels do track the text — ''landing-page-only'' papers name a deeper unit and nothing else in **4 of 155** cases, while the ''not-stated'' control has neither phrase in **83.8%** of cases — but **42 of 155 (27.1%)** ''landing-page-only'' papers contain neither phrase, and a hand-read sample of those found the label is a defensible inference from "we visited each website" rather than a stated depth. The 81.9% "states a depth" figure should therefore be read as an upper bound.
+  * **''crawlConfig'' carries one evidence quote for the whole object** — shared by the browser, the statefulness, the consent action, the depth and four other keys. Measured on a reproducible sample of 12 papers that state both a depth and a subpage count: **6 of the 12 shared quotes evidence the depth, 5 evidence a different key entirely** (statefulness, vantage, repeat visits) and 1 is about visit structure rather than site depth. So the site's usual "read the evidence quote" check cannot validate a depth label about half the time. In place of it, the provenance page reports a **discriminant**: whether each paper's own text contains a first-person sentence naming the site root, or something below it, or both, or neither. The labels do track the text — ''landing-page-only'' papers name a deeper unit and nothing else in **4 of 155** cases, while the ''not-stated'' control has neither phrase in **83.8%** of cases — but **42 of 155 (27.1%)** ''landing-page-only'' papers contain neither phrase, and a hand-read sample of those found the label is a defensible inference from "we visited each website" rather than a stated depth. The 81.9% "states a depth" figure should therefore be read as an upper bound.
   * **The on-page-action figures are upper bounds by construction.** They count papers whose text matches a pattern in a first-person sentence, not papers that did the thing.
@@ Line 541: / Line 591: @@
 <WRAP todo>
+//"Nobody" below means "no paper in this corpus's seven venues, 2010–2026". EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS are not in it, and neither is the web-performance literature outside IMC.//
   * **Nobody has repeated the saturation study.** Urban et al.'s curve is one metric (cookies), one rank band (top 1,000), one year (2019–2020). A 2026 replication for distinct third-party domains, for ad slots, and for the long tail of the ranking would settle the number the whole field is guessing at.
   * **Selection rule versus count, held apart.** No paper in this corpus varies //which// subpages it picks while holding //how many// fixed. Ten nav links, ten random links and ten search-popular pages from the same sites would answer whether the rule or the count dominates.