Differences

This shows you the differences between two versions of the page.

--- programming:interaction [2026/08/27 12:09] – Replace the 2.4 KB stub with a full page on crawl interaction: the four axes (depth, subpage selection, on-page action, identity), corpus figures with denominators (857 web crawls, 417 on the depth axis), the Urban/Aqeel disagreement, the 100-subpage satu karel.kubicek.claude
+++ programming:interaction [2026/10/08 20:49] (current) – Finish PII leakage collateral: back-links and provenance; Authored by Claude karel.kubicek.claude
@@ Line 8: / Line 8: @@
 <WRAP important>
-Three things a fresh measurement gets wrong most often.
+Three things this page exists to stop you getting wrong.
   - **"We crawled 10,000 websites" is not a measurement unit.** A website is not a page. Of the **857 papers in [[literature:corpus|this corpus]] that ran an automated web crawl of the web**, **417** give a value on the site-depth axis, and **155 of those 417 (37.2%)** stayed on the landing page. The landing page is a systematically atypical page — see [[#Deeper is not simply more]].
-  - **Ten subpages is the folk default and it is not where the curve flattens.** The median stated ''subpagesPerSite'' in this corpus is **10**. The one paper that actually measured where the increase stops found cookies stabilising **after 100 subpages**, not ten {[urban2020beyond]}. See [[#Where the increase stops]].
+  - **Ten subpages is the folk default and it is not where the curve flattens.** The median stated ''subpagesPerSite'' in this corpus is **10**. The one paper in the corpus that actually measured where the increase stops found cookies stabilising **after 100 subpages**, not ten {[urban2020beyond]}, and a 2026 study puts the fixed depth at which the //median// site stops gaining anything at **23 to 43 pages**, depending on the metric {[ramakrishnan2026_crawling]}. See [[#Where the increase stops]].
-  - **Navigating is not interacting.** Almost nothing on a modern page loads at ''load''. In the corpus, **69 of 857 web crawls (8.1%)** say anything about scrolling and **28 (3.3%)** about filling a form. The measured ladder in [[#What a strategy can and cannot see|the probe below]] shows a beacon that only a scroll reveals and another that only a click reveals — on a fixture built to have them, but built that way because real pages are.
+  - **Navigating is not interacting.** A great deal of what a privacy measurement counts — lazily-loaded ad slots, content behind a tab or an accordion, anything a form submission triggers — does not exist at ''load''. In the corpus, **69 of 857 web crawls (8.1%)** describe scrolling in their own methods and **28 (3.3%)** describe filling a form. (Loosen the pattern to any mention anywhere in the paper and it is 120 and 59 — still small.) The measured ladder in [[#What a strategy can and cannot see|the probe below]] shows a beacon that only a scroll reveals and another that only a click reveals — on a fixture built to have them, but built that way because real pages are.
 </WRAP>
@@ Line 18: / Line 18: @@
   * **On Landing and Internal Web Pages: The Strange Case of Jekyll and Hyde in Web Performance Measurement** {[aqeel2020_landing]}, IMC 2020 — the paper that made "we used a top list, so we crawled landing pages" a stated limitation rather than an unexamined default. It reviews 119 web-performance papers and concludes that about two-thirds would need revising to apply to internal pages. Read it even if your topic is privacy, not performance: the argument is about top lists, and every privacy crawl uses one.
-  * **Beyond the Front Page: Measuring Third Party Dynamics in the Field** {[urban2020beyond]}, TheWebConf 2020 — the privacy-side counterpart, published the same year, and the only paper here that measures **how many** subpages you need. Read it for the pre-study in §4, not only for the headline.
+  * **Beyond the Front Page: Measuring Third Party Dynamics in the Field** {[urban2020beyond]}, TheWebConf 2020 — the privacy-side counterpart, published the same year, and the only paper in the corpus that measures **how many** subpages you need. Read it for the pre-study in §4, not only for the headline.
+  * **Crawling in the Deep: Evaluating Web Crawling Configurations for Web Privacy Measurements** {[ramakrishnan2026_crawling]}, IMC 2026 — the controlled experiment the rest of this page has to piece together from several papers: traversal order, statefulness, interaction, depth and per-page wait varied on the //same// pages of 9,673 CrUX sites, measured on tracking domains, fingerprinting APIs and tracking cookies. It is newer than [[literature:corpus|the corpus]] (IMC 2026 had not been held when it was built), so it is in none of the corpus counts below. Read §4.3 for the stopping rule and §5.1 for the recommendations.
   * **Apophanies or Epiphanies? How Crawlers Impact Our Understanding of the Web** {[ahmad2020_apophanies]}, TheWebConf 2020 — what changes when you change the instrument. Read it before you argue that your crawler choice does not matter.
   * **Towards Realistic and Reproducible Web Crawl Measurements** {[jueckstock2021_realistic]}, TheWebConf 2021 — realism as an axis you can rank designs on, rather than a limitation-section apology.
@@ Line 47: / Line 48: @@
 | ''not-stated'' | — | 136 |
-''single-target-page'' is the largest single value, and treating it as a depth answer is the easiest mistake to make with this data. It fires on website-fingerprinting trace collection, phishing-blocklist monitors, search-result scrapes and dashboard pulls. Those crawls have a page list, not a site list, so "how deep did you go" does not apply to them. **Every depth percentage on this page has a denominator of 417, not 857 and not 5,859.**
+''single-target-page'' is the largest single value, and treating it as a depth answer is the easiest mistake to make with this data. It fires on website-fingerprinting trace collection, phishing-blocklist monitors, search-result scrapes and dashboard pulls. Those crawls have a page list, not a site list, so "how deep did you go" does not apply to them. **Every percentage of the depth //distribution// on this page has a denominator of 417, not 857 and not 5,859.** (Reporting rates and on-page-action shares are of 857; each table says which.)
 There is also a terminology trap that a keyword search will not survive: in the phishing, malvertising and spam literature **"landing page" means the destination of a redirect or ad click**, not the root of a site. The same two words name the thing this page is about and the thing that page is about.
@@ Line 53: / Line 54: @@
 ===== How Deep the Field Actually Goes =====
-Depth is, by a wide margin, the best-reported crawl-configuration field in this corpus — better than browser, statefulness or headless mode. That is worth knowing before you decide it is not worth a sentence.
+Depth is the **best-reported** crawl-configuration field in this corpus, though only just: of the 1,120 crawling papers, 75.1% state a depth against 69.6% for authentication — and then the field falls off a cliff.
+^ ''crawlConfig'' field ^ Papers stating it ^ Share of 1,120 crawling papers ^
+| ''interactionDepth'' | 841 | 75.1% |
+| ''authentication'' | 779 | 69.6% |
+| ''browsers'' | 529 | 47.2% |
+| ''consentAction'' (field populated; schema, not an audited paper claim) | 349 | 31.2% |
+| ''statefulness'' | 219 | 19.6% |
+| ''headless'' | 140 | 12.5% |
+The consent row is raw schema population, not a supported paper-claim count. The full audit found **55 of 1,120 crawling papers (4.9%)** stating an action and **279 of 313 (89.1%)** unsupported ''no-interaction'' labels.
+That is worth knowing before you decide depth is not worth a sentence: it is the one crawl-configuration choice the field already has a habit of writing down, so omitting it stands out.
 ^ Bucket ^ Web crawls ^ States a depth ^ On the site-depth axis ^ Of those, went beyond the landing page ^ Gives a subpage count ^
@@ Line 64: / Line 77: @@
 Two readings, and they point in different directions.
-The good news: **about four in five web crawls say something about depth**, and that has been true since 2010. Compare it with statefulness, which 19.6% of crawling papers state ([[Programming:Stateful Stateless#Reporting has not improved in sixteen years|the flattest field in the schema]]).
+The good news: **about four in five web crawls say something about depth**, and that has been roughly true since 2010 — though the direction is very slightly down, from 83.8% in the first bucket to 79.1% in the last, which [[Programming:Stateful Stateless|the statefulness page]] reads as the one crawl-configuration field whose reporting rate has //declined//. Compare it with statefulness, which 19.6% of crawling papers state ([[Programming:Stateful Stateless#Reporting has not improved in sixteen years|the flattest field in the schema]]).
 The bad news: the second and third numbers barely move. Sixteen years after the field started worrying about this, **the share that goes beyond the landing page is not rising**, and **only 104 of 857 web crawls (12.1%) put a number on how many subpages** — 102 of the 417 on the depth axis, 24.5%. "We also crawled subpages" is not a reproducible statement, and it is the commonest form the claim takes.
@@ Line 85: / Line 98: @@
 | 100 or more | 17 | 16.3% |
-Median 10. The five commonest values are 5 (14 papers), 3 (12), 10 (12), 1 (11) and 20 (8) — all small, all round, and none of them justified anywhere. The tail above 100 is not "more of the same": those are deep crawls of a single application or forum (100, 200, 300, 500, 1,000 and 2,000 pages per target all appear), where the population is one site and the pages are the sample.
+Median 10, and it does not settle: per four-year bucket the median runs 5, 5, 10, 15, 5 on 8, 11, 37, 33 and 15 papers, which is small-sample noise rather than a trend. The five commonest values overall are 5 (14 papers), 3 (12), 10 (12), 1 (11) and 20 (8) — all small and all round. Whether they are //justified// is harder to establish than it looks, and a full-text sweep over the 104 puts an upper bound on it: **13 (12.5%)** cite Urban et al. anywhere in the paper, **12 (11.5%)** cite Aqeel et al. or Hispar, **18 (17.3%)** use the words //pilot// or //pre-study//, and **10 (9.6%)** use //saturation//, //diminishing returns// or //plateau// anywhere at all. Every one of those is an upper bound on justifying //this// choice, because a citation in related work is not a justification of a number in the methods section. The tail above 100 is not "more of the same": those are deep crawls of a single application or forum (100, 200, 300, 500, 1,000 and 2,000 pages per target all appear), where the population is one site and the pages are the sample.
-The honest summary is that **the field converged on "a handful of subpages" without measuring whether a handful is enough**, and the one paper that did measure says it is not.
+The honest summary is that **the field converged on "a handful of subpages" without measuring whether a handful is enough**, and the one paper in this corpus that did measure says it is not.
 ===== Deeper is Not Simply More =====
@@ Line 120: / Line 133: @@
 <WRAP important>
-**Per-page density and union coverage move in opposite directions.** The landing page is the most heavily instrumented //single// page on a typical site. The set of internal pages reaches //more distinct// third parties. Which of those your number is a count of decides whether crawling deeper raises it or lowers it — and it is the sentence almost no methods section contains. A per-page mean is not a prevalence, and a prevalence over sites is not a per-page mean.
+**Per-page density and union coverage move in opposite directions.** The landing page is the most heavily instrumented //single// page on a typical site. The set of internal pages reaches //more distinct// third parties. Which of those your number is a count of decides whether crawling deeper raises it or lowers it — and it is not something any of the reporting checklists asks for, so it is on you to say it. A per-page mean is not a prevalence, and a prevalence over sites is not a per-page mean.
 </WRAP>
@@ Line 137: / Line 150: @@
 Two smaller replications in the same direction:
-  * Musch and Johns, sweeping for JavaScript anti-debugging over front pages plus **three random internal subpages**: "Visiting subpages increased prevalence by about 17%" {[musch2021_debug]}.
+  * Musch and Johns swept for JavaScript anti-debugging over front pages plus **three random internal subpages** {[musch2021_debug]}: "we can see that visiting subpages did indeed significantly increase the prevalence by about 17% compared to only crawling the front pages". The denominator is 206,676 sites with an indicator, of which **30,494 showed it only on a subpage**. And the aggregate hides the variance: per technique the subpage-only increase runs from **+18%** and **+19%** for the two commonest indicators up to **+80%** for triggered breakpoints — so how much depth matters depends on which behaviour you are counting, not just on the depth.
   * Lauinger et al. state the other side plainly, and give the reason a landing-only design is often the right one: "We only crawl the homepage of each visited site due to the presence of many sites that thwart deeper traversal by requiring log-ins." {[lauinger2017_thou]}
@@ Line 154: / Line 167: @@
 A note this page has to make, because the previous version of it carried the opposite claim: the earlier text here said the increase becomes small after about **ten** subpages. That number is not in the paper. The pre-study's smallest step is 25, and its stated stabilisation point is 100.((Checked against ''data/fulltext/2020/WWW/beyond-the-front-page-.../paper.cols.txt'' on 2026-08-27; see [[provenance:programming:interaction]].))
+==== A second saturation study (2026): per metric, and a stopping rule instead of a number ====
+Ramakrishnan et al. crawled the CrUX top 10,000 (October 2024 list; 9,673 sites reachable) with OpenWPM, up to **50 live pages** per site out of at most 100 requested, and asked the same question for three metric families at once. The depth analysis is reported for their best configuration — breadth-first, state kept within the site, simulated interaction, 30 s per page — and the authors say the takeaways hold for the others {[ramakrishnan2026_crawling]}. Their unit is strict: the depth at which the **median site's error reaches 0%**, where error is the share of what a full 50-page crawl found on that site that a shorter crawl missed. The population is the sites where 50 live pages were found — about four in five under breadth-first search.
+^ Metric ^ Fixed depth for 0% median error ^ Adaptive rule: stop after //k// pages with nothing new ^ Median stopping depth ^ Pages crawled, adaptive vs fixed ^
+| tracking domains (EasyList, EasyPrivacy, Fanboy) | 33 | //k// = 11 | 20 | 178.7K vs 234.1K (23.7% fewer) |
+| all requested domains | 39 | //k// = 11 | 22 | 202.9K vs 276.6K (26.6% fewer) |
+| fingerprinting APIs (OpenWPM's list) | 23 | //k// = 12 | 18 | 100.7K vs 110.7K (9.0% fewer) |
+| all Web APIs | 24 | //k// = 13 | 20 | 110.9K vs 115.5K (4.0% fewer) |
+| tracking cookies (CookieBlock classifier) | 37 | //k// = 8 | 16 | 95.7K vs 146.3K (34.6% fewer) |
+| all cookies | 43 | //k// = 8 | 22 | 117.8K vs 179.7K (34.4% fewer) |
+(The paper states the fixed-depth page counts as "//x//% more" than adaptive; the "fewer" column is the same pair of numbers read the other way.)
+Three things to take from it:
+  * **Ten is too few on every metric measured so far.** The fixed depths run from 23 to 43, and Urban et al.'s cookie curve flattens at 100. The two studies use different units — a mean cookie count against a per-site 0% median error — but neither puts the knee anywhere near the field's median.
+  * **Stop per site, not per study.** Sites saturate at very different depths, so any single fixed number is inefficient on many sites and too shallow for some. A "stop after //k// consecutive pages with nothing new" rule needs only a per-site set of what has been seen, and //k// = 13 reached 0% median error on all six metrics (//k// = 8 was enough for cookies). It is not free everywhere: for tracking domains at //k// = 11, the 95th-percentile site still runs to the 50-page cap.
+  * **The advantage depends on how much error you tolerate.** At a looser target for tracking domains — three sites in four under 10% error — a fixed 28 pages does the job, and the adaptive rule (//k// = 13) saves only about 5% of pages. Adaptive stopping pays most when you want nothing missed.
+Its limits are the ones to state if you cite it: a 50-page ceiling, so "0% error" means nothing missed //relative to page 50// and says nothing about page 51; popular sites only; one US academic vantage point; OpenWPM on Firefox 130 with no bot mitigation; and no consent-banner interaction, so everything is the pre-consent state. A re-run on 1,000 of the sites five months later, on freshly collected pages, gave the same adaptive thresholds — identical for domains and APIs, within one page for cookies — though breadth-first now reached 50 live pages on 72% of sites rather than 79%.
 ===== Choosing Which Subpages: a Sampling Decision in Disguise =====
@@ Line 173: / Line 208: @@
   * **Same-origin is not the same as first-party.** A link to ''shop.example.com'' from ''example.com'' is same-site and cross-origin; a link to ''example.co.uk'' is neither. Say which test you used, and say whether you used the Public Suffix List.
-  * **The frontier is state-dependent.** Which links exist on the landing page depends on your consent choice, your geolocation, your logged-in state and the hour. Two arms of an experiment that each harvest their own links are not visiting the same pages. Harvest the link set **once**, freeze it, and replay it in every arm — that is what Urban et al. do (collect first-party hyperlinks one day before the first measurement, then reuse the same list across regions), and what Lin et al. do by replaying the same CSS selectors across all three crawl groups.
+  * **The frontier is state-dependent.** Which links exist on the landing page depends on your consent choice, your geolocation, your logged-in state and the hour. Two arms of an experiment that each harvest their own links are not visiting the same pages. Harvest the link set **once**, freeze it, and replay it in every arm — that is what Urban et al. do (collect first-party hyperlinks one day before the first measurement, then reuse the same list across regions), and what Lin et al. do by replaying the same CSS selectors across all three crawl groups. Ramakrishnan et al. build their whole design on it: one page-collection crawl per traversal strategy, then every configuration replays that ordered page list, "whereas if each crawl visited different pages, the variations in privacy metrics could be due to those pages rather than the crawling parameters" {[ramakrishnan2026_crawling]}.
+==== Traversal order: go breadth-first ====
+Once you go past one hop, the rule above has a second half: in what order do you expand the frontier? Ramakrishnan et al. compared breadth-first, depth-first with backtracking, and a breadth-limited hybrid (breadth-first but following at most three links per page) on the same 9,673 sites {[ramakrishnan2026_crawling]}:
+  * **Breadth-first finds live pages; depth-first runs out.** Breadth-first reached 50 live pages on about **79%** of sites, the hybrid on about **67%**, and depth-first fell short on nearly **half** within the 100-request budget. The authors' reading, which they do not test further, is that links deep in a site are less likely to still be live.
+  * **And the pages it finds carry more.** Breadth-first yielded **1–14% more** tracking domains and **1–26% more** tracking cookies in total than the other strategies at the same nominal depth, across all sites — which includes the sites where depth-first found fewer live pages. Restricted to sites where all three strategies found 50 live pages, breadth-first still had the higher totals (the paper gives no percentage for that comparison), so it is not only a page-count effect.
+  * **The gap is in the upper tail, not the median.** The median site shows about the same number of tracking domains under every configuration; the differences are at the 75th percentile and above. A per-site median will hide the choice, an aggregate total will show it.
+The security-scanning side points somewhere slightly different. Stafeev and Pellegrino, measuring code and link coverage rather than privacy metrics, found "randomized algorithms, in particular randomized BFS, offer better performance than standard algorithms, providing a general increase of average coverage, from +8% in LoCs up to +21% for JavaScript source code" {[stafeev2024_state]} — but in their data //plain// breadth-first and depth-first are within about 1% of each other, and depth-first is marginally ahead on JavaScript source coverage. Their result is an argument for randomising the frontier — and all three of Ramakrishnan et al.'s strategies pick links at random in Stafeev's sense, so the two studies do not conflict; the breadth-first-over-depth-first argument comes from Ramakrishnan et al.'s privacy metrics, and from the practical point that depth-first runs out of live pages.
 ==== The one page list that included internal pages is gone ====
@@ Line 179: / Line 224: @@
 Aqeel et al. shipped **Hispar**: a top list of complete URLs rather than domains — around 2,000 sites, one landing page and up to 49 internal pages each, 100,000 URLs, refreshed weekly, with the internal pages discovered through Google ''site:'' queries {[aqeel2020_landing]}.
-**Do not plan a study around it.** Checked on 2026-08-27: ''hispar.cs.duke.edu'' no longer resolves at all; its last Internet Archive capture is 2024-11-16; the generator at ''github.com/waqaraqeel/hispar'' was last pushed on 2020-10-27 and ''lit_grabber'' on 2020-05-26. Its bootstrap list, the Alexa Top 1M, was itself retired in 2022, so even the generator cannot be run as written. The idea outlived the artefact.
+**Do not plan a study around it.** Checked on 2026-08-27: ''hispar.cs.duke.edu'' no longer resolves at all; the last Internet Archive capture that returned any content is 2024-11-16, and the last capture of any kind is a 403 on 2024-12-03; the generator at ''github.com/waqaraqeel/hispar'' was last pushed on 2020-10-27 and ''lit_grabber'' on 2020-05-26. Its bootstrap list, the Alexa Top 1M, was itself retired in 2022, so even the generator cannot be run as written. The idea outlived the artefact.
 **What exists instead, and what it gives you.** Since April 2022 the HTTP Archive crawls **one secondary page per site** alongside the root, and both are in the public BigQuery dataset with ''is_root_page'' and ''root_page'' columns to tell them apart.((HTTP Archive BigQuery reference, //Pages table//, ''har.fyi/reference/tables/pages/'': "Pages are tested on a monthly basis and as of April 2022, both the root page and one secondary page are tested." Checked 2026-08-27. Note that ''httparchive.org/faq'' still says the opposite — "The HTTP Archive examines each URL in the list, but does not crawl the website's other pages" — and is stale.)) Read the selection rule before you use it as an internal-page sample: in the crawl controller, ''MAX_DEPTH = 1'' and ''MAX_BREADTH = 1'', and the child job is the **first** link on the page whose hostname equals the parent's, whose extension is not an image, PDF or archive, and which has not already been visited.((''github.com/HTTPArchive/crawl'', files ''crawl.py'' and ''crawl_pubsub.py'' at commit ''4b83ac0'' (2025-08-20). Checked 2026-08-27.)) That is the //first link in DOM order// rule from the table above, with all of its bias: on most sites it is a navigation item. It is a real internal page and it is not a random one.
@@ Line 199: / Line 244: @@
 | mentions bot or crawler detection | 61 | 7.1% |
-The only two rows with a clear trend across the sixteen years are **scrolling** — 2.5% of web crawls in 2010–2013 against 11.1% in 2025–2026 — and **bot detection**, which no paper in the 2010–2013 bucket raises and 11.8% of the 2025–2026 bucket does. Clicking is flat and has been since 2010. Mouse movement and hovering are, and have always been, rounding errors.
+The only two rows with a clear trend across the sixteen years are **scrolling** — 2.5% of web crawls in 2010–2013 against 11.1% in 2025–2026 — and **bot detection**, which not one paper in the 2010–2013 bucket raises //in a first-person sentence// against 11.8% of the 2025–2026 bucket. (Three of those 80 papers do mention it somewhere in the text — a probe's zero is never the same as nobody.) Clicking has **not** risen: it sits between 29.9% and 43.8% in every bucket, and the //highest// value is the earliest one. Mouse movement and hovering are, and have always been, under 4%.
+==== How long to stay on the page ====
+The wait is the on-page choice with a measured answer. Ramakrishnan et al. kept each page open for 30 s — the upper end of what most prior crawls used — timestamped every event, and asked what a shorter wait would have missed, relative to 30 s {[ramakrishnan2026_crawling]}:
+^ Metric ^ Median site misses nothing by ^ Sites missing nothing at 15 s ^ at 20 s ^
+| tracking domains | 10 s | 77.3% (5-page crawl) | not stated |
+| fingerprinting APIs | about half of sites miss nothing at 5 s | 83.8% | over 87% |
+| tracking cookies | 10 s | 65.4% | 70.9% |
+Counted per (site, tracking domain) pair instead of per site, **93–95%** are first requested within 15 s of page load, and nearly all by 20 s. The paper's recommendation is **15 s** for a crawl short on time. The paper applies it to all metrics, and its five-month re-run found three sites in four gained nothing past 15 s on any metric. Our own reading of the cookie row adds a caution: about a third of sites still set a new tracking cookie between 15 s and 30 s, so if cookies are the metric, a longer wait costs little and buys something. Two conditions travel with the number: the clock starts at page load, and the paper does not say which of its twelve configurations the timings come from — in the interactive ones the page is scrolled to the bottom every 5 s, so a shorter wait also means fewer scrolls.
+The same paper also puts a size on what light interaction adds, alongside the earlier setup comparisons of Demir et al. {[demir2022_reproducibility]}, {[demir2023_similarity]}. Their interaction is **Tab keystrokes at short random intervals plus a scroll to the bottom every 5 s**, with no clicking or other element interaction. The configuration with interaction //and// kept browser state gave **0.8–11.6% more** tracking domains in total than the other configurations of the same traversal strategy, and its per-site distributions differed significantly from all others under paired Wilcoxon tests with Holm correction {[ramakrishnan2026_crawling]}. For fingerprinting APIs the margin shrinks to 0.5–1.2%, because the set of APIs is finite. It is a real effect and a modest one, and it is for state, keystrokes and scrolling together — the paper never separates them — so it is not a figure for scrolling alone.
 ==== What a strategy can and cannot see ====
@@ Line 214: / Line 272: @@
 landing + all links, scroll, and click every button                      4          ✓          ✓          ✓          ✓          ✗
 depth 2: landing, its links, and their links, with scroll and click      5          ✓          ✓          ✓          ✓          ✓
+t-landing  on the landing page
+t-article  on one subpage, at load
+t-scroll   on that subpage, only after scrolling to the bottom
+t-click    on another subpage, only after a button click
+t-deep     on a page linked only from a subpage, never from the landing page
+Playwright 1.62.1, Chromium 151.0.7922.34, linux. Needs no network.
 </code>
-Playwright 1.62.1, Chromium 151.0.7922.34, Linux. Reproduce with [[#The code|the two files below]]; it needs no network.
+Reproduce with [[#The code|the two files below]]: the version line, the platform and the legend are printed by the script itself.
 Read off the second row. Following the **first** same-origin link found nothing that the landing page had not already shown, because the first link in DOM order is the //About// page and the //About// page is empty of third parties. That is not a quirk of the fixture — it is what the HTTP Archive's secondary-page rule does on any site whose navigation starts with an institutional page, and it is why "we crawled a secondary page" and "we sampled an internal page" are different claims.
 Read off the last row. Each additional capability found exactly one thing the previous one could not, and no amount of extra depth substitutes for the scroll, nor extra scrolling for the click. **These are not degrees of the same knob.** A design that goes three levels deep without scrolling still misses everything below the fold.
+==== Interacting is not free, and it is not only your cost ====
+Everything above says //do more//. Before you scale it to ten thousand sites, four consequences that a landing-page crawl does not have:
+  * **A click on an ad is an ad impression somebody pays for.** Ad clicks are billed. A crawl that clicks indiscriminately at scale generates click fraud, and the money comes out of an advertiser's budget. Reitinger et al. designed around exactly this: rather than fetch an ad's destination link — which "could charge advertisers as a higher-cost //click//" — they parse the destination out of the URL instead {[reitinger2024_creepy]}. If you must resolve where an ad goes, resolve it without clicking it.
+  * **A form submission has a side effect on a real system.** Filling and submitting forms at scale sends real newsletter signups, real support tickets, real account registrations and real search queries against production databases. This is why EvoCrawl's evaluation runs against **ten self-hostable web applications** — WordPress, HotCRP, Kanboard, ImpressCMS, phpBB, Humhub, GitLab and others, each shipped as a Dockerfile in the repository's ''experiments/'' folder — rather than against the live web {[guo2025_evocrawl]}. The interaction depth it achieves is only safe on a system you own. Read the [[Practices:Ethics|ethics page]] before you point a form-filler at a site list.
+  * **Interaction multiplies your load by more than you expect.** The Khaleesi comparison above is 3× the requests from the same list. Rate limits, politeness delays and any ''robots.txt'' policy you have should be set for the interactive figure, not the landing-page one.
+  * **You are more visible, and being detected is not neutral.** A crawler that clicks and scrolls looks less like a bot to a fraud model and more like a user — which is the point, and is also why the resulting impressions and clicks are harder for the site to filter out. "We made the crawler look human" and "we made our traffic indistinguishable from the traffic the site bills for" are the same sentence.
+None of this argues for staying on the landing page. It argues for saying, in the paper, which interactions you performed, on how many sites, and what you did to keep them from being charged for.
 ==== The code ====
@@ Line 235: / Line 312: @@
 // Third parties are served from a SECOND origin (127.0.0.2) so the crawler
 // counts them as third-party the way a real measurement would.
+//
+// PORTABILITY: Linux gives you the whole 127.0.0.0/8 loopback range for free.
+// macOS does not — binding 127.0.0.2 fails with EADDRNOTAVAIL until you add the
+// alias:  sudo ifconfig lo0 alias 127.0.0.2 up
+// (undo with `sudo ifconfig lo0 -alias 127.0.0.2`). On Windows, use WSL.
 import http from 'node:http';
@@ Line 338: / Line 420: @@
 // Collect the third-party hostnames+paths seen during one strategy.
+let chromiumVersion = null;
 async function run(name, fn) {
   const seen = new Set();
   const browser = await chromium.launch();
+  chromiumVersion = browser.version();
   const context = await browser.newContext();
   context.on('request', (r) => {
@@ Line 356: / Line 441: @@
 // gets them in unless it deliberately shuffles.
 const links = (page) =>
-  page.$$eval('a[href]', (as) =>
+  page.locator('a[href]').evaluateAll((as) =>
     as.map((a) => a.href).filter((h) => new URL(h).hostname === location.hostname));
@@ Line 397: / Line 482: @@
     await page.goto(l);
     await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
-    for (const b of await page.$$('button')) await b.click().catch(() => {});
+    for (const b of await page.locator('button').all()) await b.click().catch(() => {});
     await page.waitForTimeout(200);
   }
@@ Line 414: / Line 499: @@
       await page.goto(url);
       await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
-      for (const b of await page.$$('button')) await b.click().catch(() => {});
+      for (const b of await page.locator('button').all()) await b.click().catch(() => {});
       await page.waitForTimeout(200);
       next.push(...(await links(page)));
@@ Line 441: / Line 526: @@
   );
 }
+const pwVersion = (await import('playwright/package.json', { with: { type: 'json' } })).default.version;
+console.log(`
+t-landing  on the landing page
+t-article  on one subpage, at load
+t-scroll   on that subpage, only after scrolling to the bottom
+t-click    on another subpage, only after a button click
+t-deep     on a page linked only from a subpage, never from the landing page
+Playwright ${pwVersion}, Chromium ${chromiumVersion}, ${process.platform}. Needs no network.`);
 </file>
 ===== Forms, Logins and the Half of the Web You Are Not Measuring =====
-Of the 857 web crawls in this corpus, **553 (64.5%)** explicitly did not authenticate, **204 (23.8%)** do not say, and **81 (9.5%)** got past a login of any kind — 42 by registering an account, 24 by logging in manually, 15 with automated login and one via SSO. Whatever your topic, **a crawl of the logged-out web is the design nine papers in ten use**, and the reason is usually the one Lauinger et al. give: login walls stop deeper traversal.
+Of the 857 web crawls in this corpus, **553 (64.5%)** are recorded as not authenticating, **204 (23.8%)** do not say, and **at most 81 (9.5%)** got past a login of any kind — 42 by registering an account, 24 by logging in manually and 15 with automated login. The remaining **19 (2.2%)** have no ''crawlConfig'' object at all; their crawl is known only from ''studyTypes'', so they have no authentication label to report, and 553 + 204 + 81 + 19 = 857. Not one of the 857 used SSO.((The corpus has exactly one paper with ''authentication: sso'' — an Android subscription-app study — and it is not a web crawl, so it is outside this page's population.)) Whatever your topic, **at most one crawl in ten gets past a login**, and the reason is usually the one Lauinger et al. give: login walls stop deeper traversal. Treat the 553 with care: ''authentication'' shares the same single ''crawlConfig'' quote as the depth, so the label cannot be checked against its own evidence, and a large share of those ''none'' values are the extractor's default reading of a paper that simply never mentions logging in. **That share has now been measured — but on a different population.** A hand audit of the **145** ''none'' labels among the **260 platform-subject papers with a crawl configuration** (not these 857 web crawls) found **119 (82.1%)** where the paper states nothing either way, only **12 (8.3%)** where it states the collection was unauthenticated, and **6 (4.1%)** contradicted by the paper's own text — including one whose ''crawlConfig'' evidence quote itself says the authors scraped Instagram "using an alias account" ([[provenance:design:platforms]]). The two populations overlap and are not the same set, so do not carry 82.1% across to the 553; read it as evidence that ''none'' on this field is mostly inferred from silence, and that the 553 is an upper bound.
 Two things worth knowing before you decide the effort is not worth it.
@@ Line 457: / Line 551: @@
 A form-filler that assigns to ''input.value'' fires no ''keydown'', no ''keyup'' and no ''input'' event unless the framework listens for programmatic changes, so the very scripts you are trying to catch never run. If your measurement is about what happens //while// a user types, you must actually type. Playwright's ''locator.pressSequentially()'' and Puppeteer's ''ElementHandle.type()'' with a ''delay'' both do this; ''locator.fill()'' does not.
-The rest of the login problem — Shepherd-style automated login, SSO, and automated registration — is [[Programming:Registration|Automating Login and Registration to Websites]].
+The rest of the login problem — Shepherd-style automated login, SSO, and automated registration — is [[Programming:Registration|Automating Login and Registration to Websites]], **currently a stub**, so expect to go to the papers directly for now.
 ===== Your Crawler Will Be Noticed =====
-Interaction is the axis on which a crawler most visibly stops resembling a user, and the share of papers that say so is rising: **no paper in the 2010–2013 bucket mentions bot or crawler detection, against 11.8% of the 2025–2026 bucket.**
+Interaction is the axis on which a crawler most visibly stops resembling a user, and the share of papers that say so is rising: **not one of the 80 web crawls in the 2010–2013 bucket describes bot or crawler detection in a first-person sentence — three mention it in passing — against 11.8% of the 2025–2026 bucket.**
 The concrete failure modes to plan for:
@@ Line 477: / Line 571: @@
 ^ Method ^ Status ^ Evidence ^
 | **Landing-page-only crawls of a ranked list** | **Still the single commonest design, and still defensible for the right question** — but no longer defensible //silently//. Since 2020 there is a citable reason to justify it or drop it | 155 of 417 depth-stating web crawls; {[aqeel2020_landing]}, {[urban2020beyond]} |
-| **"Plus a handful of random subpages"** | **Current practice.** Median 10, unchanged across all five buckets, and not derived from any measurement | 104 papers give a count; median 10 |
+| **"Plus a handful of random subpages"** | **Current practice**, and not derived from any measurement. The overall median is 10; per four-year bucket it is 5, 5, 10, 15 and 5, on 8, 11, 37, 33 and 15 papers — too few to call a trend either way | 104 papers give a count |
-| **A saturation study to choose the count** | **Done once, in 2020, and not replicated.** If you do it for your own metric you will be the second | {[urban2020beyond]} |
+| **A saturation study to choose the count** | **Done twice**: cookies on the top 1,000 in 2020, and three metric families on the CrUX top 10,000 in 2026. Both put the knee well above ten; a study for your own metric is still worth running | {[urban2020beyond]}, {[ramakrishnan2026_crawling]} |
+| **Adaptive per-site stopping** (stop after //k// pages with nothing new) | **New in 2026, one study, not yet adopted.** Same 0% median error as a fixed depth with 4–35% fewer pages | {[ramakrishnan2026_crawling]} |
+| **Breadth-first traversal** | **Current recommendation** for privacy metrics, from one study; randomising the frontier is the separate recommendation from security scanning | {[ramakrishnan2026_crawling]}, {[stafeev2024_state]} |
 | **A purpose-built internal-page top list** | **Superseded by circumstance.** Hispar is offline and its bootstrap list is retired. The HTTP Archive's one-secondary-page-per-site is what remains, with a known selection rule | {[aqeel2020_landing]}; checks of 2026-08-27 |
 | **Scrolling as a routine step** | **Rising and not yet standard** — 2.5% of 2010–2013 web crawls, 11.1% of 2025–2026 | full-text sweep |
@@ Line 499: / Line 595: @@
   - **The unit.** Pages or sites? If a site contributed several pages, is your prevalence over sites or over pages? Say which, and say how you aggregated (any-page, majority, mean).
-  - **The depth.** Landing page only, landing plus //N//, or traverse to depth //d// with budget //b//. If you stayed on the landing page, say so — it is a legitimate choice with a citable justification.
+  - **The depth.** Landing page only, landing plus //N//, or traverse to depth //d// with budget //b//. If you stayed on the landing page, say so — it is a legitimate choice with a citable justification. If you stopped adaptively, give the metric the rule watched, //k// and the cap.
-  - **How you chose the pages.** The link-selection rule verbatim: same-origin or same-site, DOM order or random, seed if random, exclusions, and whether the frontier was harvested once and frozen or re-harvested per arm.
+  - **How you chose the pages.** The link-selection rule verbatim: same-origin or same-site, DOM order or random, seed if random, exclusions, breadth- or depth-first beyond one hop, and whether the frontier was harvested once and frozen or re-harvested per arm.
   - **What happened on the page.** Scroll? Click? Type? Wait how long, and for what event — ''load'', ''networkidle'', a fixed timer? A fixed timer is fine; an unstated one is not.
   - **What you did about the banner and the login**, with a pointer to [[Privacy:Consent]] and [[Programming:Registration]].
   - **The failures.** How many target sites yielded zero pages, how many yielded fewer subpages than you asked for, and how many hit a challenge you could not pass. All three are non-random.
   - **The comparison, if you make one.** If you compare against a published prevalence, check the other paper's depth first. A 3× difference in request counts is available from the depth choice alone {[iqbal2022_khaleesi]}.
+===== What a Complete Statement Looks Like =====
+Two sentences from Aziz and Wilson are, as far as this corpus goes, the model {[aziz2024_johnny]}:
+> For each domain, we programmed the crawler to load the domain's homepage, scroll to the bottom of the page, then sleep for 25 seconds. Further, we programmed our crawler to select nine internal hyperlinks at random from the homepage and crawl them using the same load, scroll, and sleep approach.
+Five of the seven items on the checklist above are in those two sentences: the unit (domain), the depth (homepage plus nine), the selection rule (at random, from the homepage), the on-page action (scroll to the bottom) and the wait (25 seconds, stated). Plus the detail that most often silently differs and is on none of the checklists: the subpages get the **same** load-scroll-sleep treatment as the landing page, which is what makes a landing-versus-subpage comparison a comparison rather than two different instruments. The two items it does not cover are the consent treatment — which is elsewhere in that paper, whose whole design is four experimental conditions — and the failure counts. Splitting them across the methods section is the usual and fine arrangement; leaving them out is not.
+Five of seven, in two sentences. That is the whole cost.
+===== If You Just Need a Default =====
+Absent a better reason — and the sections above are the better reasons — this is a defensible starting design for a privacy or security crawl over a ranked site list, and every element of it is something you can point a reviewer at:
+  - **Harvest the frontier once.** Visit each landing page, collect its same-site links, store them with the date. Freeze that list and replay it in every arm of the experiment.
+  - **Pick subpages at random from it, with a published seed**, and publish the exclusion list if you use one. Do not take them in DOM order unless you want the navigation chrome.
+  - **Go breadth-first, cap at 50 pages, and stop each site adaptively**: after 13 consecutive pages that add nothing new to your metric (8 if the metric is cookies) {[ramakrishnan2026_crawling]}. If you need a fixed number instead, ten is too few on every metric measured so far — the published saturation points run from 23 to 100 pages — so run a saturation pilot for //your// metric on a few hundred sites before committing.
+  - **On every page, including the landing page: scroll to the bottom, then wait a fixed, stated time.** 15 s is the measured cost-effective wait {[ramakrishnan2026_crawling]}; for a cookie study, consider longer (see [[#How long to stay on the page]]). Same treatment on the subpages as on the landing page, or the comparison between them is not a comparison.
+  - **Click nothing by default.** Add clicking only where your question needs it, and when you do, exclude ad elements — see [[#Interacting is not free, and it is not only your cost|above]].
+  - **Crawl logged out**, and say so. Getting past a login is a project, not a setting.
+  - **Log the failures** — sites yielding zero pages, sites yielding fewer subpages than asked, challenges you could not pass — and report all three.
+The point of the default is not that it is optimal. It is that every line of it is a sentence you can write in the methods section, which is more than most published crawls manage.
 ===== Methodology and Limitations of These Figures =====
@@ Line 513: / Line 633: @@
   * **The denominator is 857, not 5,859.** Only 1,120 papers in the corpus ran a crawl at all, and 857 of those crawled the web. The depth ratios use a further-narrowed denominator of 417 for the reason given [[#Depth has more than three positions|above]].
-  * **''crawlConfig'' carries one evidence quote for the whole object.** The quote sitting behind an ''interactionDepth'' value very often evidences the browser or the statefulness instead, so the site's usual "read the evidence quote" check cannot validate a depth label. In place of it, the provenance page reports a **discriminant**: whether each paper's own text contains a first-person sentence naming the site root, or something below it, or both, or neither. The labels do track the text — ''landing-page-only'' papers name a deeper unit and nothing else in **4 of 155** cases, while the ''not-stated'' control has neither phrase in **83.8%** of cases — but **42 of 155 (27.1%)** ''landing-page-only'' papers contain neither phrase, and a hand-read sample of those found the label is a defensible inference from "we visited each website" rather than a stated depth. The 81.9% "states a depth" figure should therefore be read as an upper bound.
+  * **''crawlConfig'' carries one evidence quote for the whole object** — shared by the browser, the statefulness, the consent action, the depth and four other keys. Measured on a reproducible sample of 12 papers that state both a depth and a subpage count: **6 of the 12 shared quotes evidence the depth, 5 evidence a different key entirely** (statefulness, vantage, repeat visits) and 1 is about visit structure rather than site depth. So the site's usual "read the evidence quote" check cannot validate a depth label about half the time. In place of it, the provenance page reports a **discriminant**: whether each paper's own text contains a first-person sentence naming the site root, or something below it, or both, or neither. The labels do track the text — ''landing-page-only'' papers name a deeper unit and nothing else in **4 of 155** cases, while the ''not-stated'' control has neither phrase in **83.8%** of cases — but **42 of 155 (27.1%)** ''landing-page-only'' papers contain neither phrase, and a hand-read sample of those found the label is a defensible inference from "we visited each website" rather than a stated depth. The 81.9% "states a depth" figure should therefore be read as an upper bound.
   * **The on-page-action figures are upper bounds by construction.** They count papers whose text matches a pattern in a first-person sentence, not papers that did the thing.
@@ Line 521: / Line 641: @@
 <WRAP todo>
-  * **Nobody has repeated the saturation study.** Urban et al.'s curve is one metric (cookies), one rank band (top 1,000), one year (2019–2020). A 2026 replication for distinct third-party domains, for ad slots, and for the long tail of the ranking would settle the number the whole field is guessing at.
+//"Nobody" below means "no paper in this corpus's seven venues, 2010–2026". EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS are not in it, and neither is the web-performance literature outside IMC.//
-  * **Selection rule versus count, held apart.** No paper in this corpus varies //which// subpages it picks while holding //how many// fixed. Ten nav links, ten random links and ten search-popular pages from the same sites would answer whether the rule or the count dominates.
-  * **What does a scroll actually add, at scale?** The fixture on this page shows it can matter. Nobody has published the distribution of "third parties visible only after scrolling" over a real site list.
+  * **The saturation study has been repeated once, and both runs stop at their own ceiling.** Urban et al.'s curve is one metric (cookies), one rank band (top 1,000), one year (2019–2020). Ramakrishnan et al. add tracking domains and fingerprinting APIs on the CrUX top 10,000 in 2024–2025, but measure error relative to a 50-page crawl, so they cannot say what lies past page 50 {[ramakrishnan2026_crawling]}. Ad slots as a metric, the long tail of the ranking, and the post-consent state are still unmeasured.
+  * **Selection rule versus count, held apart.** No paper in this corpus varies //which// subpages it picks while holding //how many// fixed. Ramakrishnan et al. do it for traversal order — breadth-first ahead of depth-first even on sites where both found 50 live pages {[ramakrishnan2026_crawling]} — but not for the link-choice rule. Ten nav links, ten random links and ten search-popular pages from the same sites would answer whether the rule or the count dominates.
+  * **What does a scroll actually add, at scale?** The fixture on this page shows it can matter, and Ramakrishnan et al. measure a 0.8–11.6% gain in tracking domains for keystrokes and scrolling combined with kept state {[ramakrishnan2026_crawling]}. Nobody has published the scroll-only effect, or the distribution of "third parties visible only after scrolling", over a real site list.
   * **A replacement for Hispar.** The HTTP Archive's one-first-link-per-site is a start and is not a sample. A maintained, versioned, reproducible internal-page list is a genuine gap and a small enough artefact for one group to own.
   * **How much does an LLM agent actually change the measurement, and at what cost per site?** The five papers using one report task success, not measurement deltas. The comparison that matters — same site list, scripted crawl versus agent crawl, same metric — has not been published.
@@ Line 533: / Line 655: @@
   * [[Programming:Stateful Stateless|Stateful and Stateless Crawling]] — whether the interaction you performed is still in effect on the next page.
   * [[Programming:Registration|Automating Login and Registration to Websites]] — the logged-in web this page's 9.5% figure is about.
+  * [[Privacy:PII leakage]] — form filling, login and site search as the treatment that puts a personal value on the page, and why submitting and not submitting are different treatments.
   * [[Programming:Crawler|Comparison of Crawling Libraries]] — which of these actions each library can actually perform.
   * [[Design:Website selection]] and [[Design:Sampling|Representative Sampling Methods]] — the site list, and why subpage selection is a sampling problem with the same failure modes.