Differences

This shows you the differences between two versions of the page.

--- programming:interaction [2026/08/27 12:34] – Self-review: replace an unmeasured 'almost no methods section contains' with a claim that can be checked (no reporting checklist asks for it). Authored by Claude karel.kubicek.claude
+++ programming:interaction [2026/08/27 12:39] (current) – Reviewer 4 (generic): add 'Interacting is not free' — click fraud, form side effects, load multiplied by 3x, and the visibility trade-off — and a Monday-morning default design; replace the unaudited 'none of them justified anywhere' with the measured just karel.kubicek.claude
@@ Line 8: / Line 8: @@
 <WRAP important>
-Three things a fresh measurement gets wrong most often.
+Three things this page exists to stop you getting wrong.
   - **"We crawled 10,000 websites" is not a measurement unit.** A website is not a page. Of the **857 papers in [[literature:corpus|this corpus]] that ran an automated web crawl of the web**, **417** give a value on the site-depth axis, and **155 of those 417 (37.2%)** stayed on the landing page. The landing page is a systematically atypical page — see [[#Deeper is not simply more]].
   - **Ten subpages is the folk default and it is not where the curve flattens.** The median stated ''subpagesPerSite'' in this corpus is **10**. The one paper that actually measured where the increase stops found cookies stabilising **after 100 subpages**, not ten {[urban2020beyond]}. See [[#Where the increase stops]].
-  - **Navigating is not interacting.** A great deal of what a privacy measurement counts — lazily-loaded ad slots, content behind a tab or an accordion, anything a form submission triggers — does not exist at ''load''. In the corpus, **69 of 857 web crawls (8.1%)** say anything about scrolling and **28 (3.3%)** about filling a form. The measured ladder in [[#What a strategy can and cannot see|the probe below]] shows a beacon that only a scroll reveals and another that only a click reveals — on a fixture built to have them, but built that way because real pages are.
+  - **Navigating is not interacting.** A great deal of what a privacy measurement counts — lazily-loaded ad slots, content behind a tab or an accordion, anything a form submission triggers — does not exist at ''load''. In the corpus, **69 of 857 web crawls (8.1%)** describe scrolling in their own methods and **28 (3.3%)** describe filling a form. (Loosen the pattern to any mention anywhere in the paper and it is 120 and 59 — still small.) The measured ladder in [[#What a strategy can and cannot see|the probe below]] shows a beacon that only a scroll reveals and another that only a click reveals — on a fixture built to have them, but built that way because real pages are.
 </WRAP>
@@ Line 47: / Line 47: @@
 | ''not-stated'' | — | 136 |
-''single-target-page'' is the largest single value, and treating it as a depth answer is the easiest mistake to make with this data. It fires on website-fingerprinting trace collection, phishing-blocklist monitors, search-result scrapes and dashboard pulls. Those crawls have a page list, not a site list, so "how deep did you go" does not apply to them. **Every depth percentage on this page has a denominator of 417, not 857 and not 5,859.**
+''single-target-page'' is the largest single value, and treating it as a depth answer is the easiest mistake to make with this data. It fires on website-fingerprinting trace collection, phishing-blocklist monitors, search-result scrapes and dashboard pulls. Those crawls have a page list, not a site list, so "how deep did you go" does not apply to them. **Every percentage of the depth //distribution// on this page has a denominator of 417, not 857 and not 5,859.** (Reporting rates and on-page-action shares are of 857; each table says which.)
 There is also a terminology trap that a keyword search will not survive: in the phishing, malvertising and spam literature **"landing page" means the destination of a redirect or ad click**, not the root of a site. The same two words name the thing this page is about and the thing that page is about.
@@ Line 74: / Line 74: @@
 Two readings, and they point in different directions.
-The good news: **about four in five web crawls say something about depth**, and that has been true since 2010. Compare it with statefulness, which 19.6% of crawling papers state ([[Programming:Stateful Stateless#Reporting has not improved in sixteen years|the flattest field in the schema]]).
+The good news: **about four in five web crawls say something about depth**, and that has been roughly true since 2010 — though the direction is very slightly down, from 83.8% in the first bucket to 79.1% in the last, which [[Programming:Stateful Stateless|the statefulness page]] reads as the one crawl-configuration field whose reporting rate has //declined//. Compare it with statefulness, which 19.6% of crawling papers state ([[Programming:Stateful Stateless#Reporting has not improved in sixteen years|the flattest field in the schema]]).
 The bad news: the second and third numbers barely move. Sixteen years after the field started worrying about this, **the share that goes beyond the landing page is not rising**, and **only 104 of 857 web crawls (12.1%) put a number on how many subpages** — 102 of the 417 on the depth axis, 24.5%. "We also crawled subpages" is not a reproducible statement, and it is the commonest form the claim takes.
@@ Line 95: / Line 95: @@
 | 100 or more | 17 | 16.3% |
-Median 10, and it does not settle: per four-year bucket the median runs 5, 5, 10, 15, 5 on 8, 11, 37, 33 and 15 papers, which is small-sample noise rather than a trend. The five commonest values overall are 5 (14 papers), 3 (12), 10 (12), 1 (11) and 20 (8) — all small, all round, and none of them justified anywhere. The tail above 100 is not "more of the same": those are deep crawls of a single application or forum (100, 200, 300, 500, 1,000 and 2,000 pages per target all appear), where the population is one site and the pages are the sample.
+Median 10, and it does not settle: per four-year bucket the median runs 5, 5, 10, 15, 5 on 8, 11, 37, 33 and 15 papers, which is small-sample noise rather than a trend. The five commonest values overall are 5 (14 papers), 3 (12), 10 (12), 1 (11) and 20 (8) — all small and all round. Whether they are //justified// is harder to establish than it looks, and a full-text sweep over the 104 puts an upper bound on it: **13 (12.5%)** cite Urban et al. anywhere in the paper, **12 (11.5%)** cite Aqeel et al. or Hispar, **18 (17.3%)** use the words //pilot// or //pre-study//, and **10 (9.6%)** use //saturation//, //diminishing returns// or //plateau// anywhere at all. Every one of those is an upper bound on justifying //this// choice, because a citation in related work is not a justification of a number in the methods section. The tail above 100 is not "more of the same": those are deep crawls of a single application or forum (100, 200, 300, 500, 1,000 and 2,000 pages per target all appear), where the population is one site and the pages are the sample.
 The honest summary is that **the field converged on "a handful of subpages" without measuring whether a handful is enough**, and the one paper in this corpus that did measure says it is not.
@@ Line 239: / Line 239: @@
 Read off the last row. Each additional capability found exactly one thing the previous one could not, and no amount of extra depth substitutes for the scroll, nor extra scrolling for the click. **These are not degrees of the same knob.** A design that goes three levels deep without scrolling still misses everything below the fold.
+==== Interacting is not free, and it is not only your cost ====
+Everything above says //do more//. Before you scale it to ten thousand sites, four consequences that a landing-page crawl does not have:
+  * **A click on an ad is an ad impression somebody pays for.** Ad clicks are billed. A crawl that clicks indiscriminately at scale generates click fraud, and the money comes out of an advertiser's budget. Reitinger et al. designed around exactly this: rather than fetch an ad's destination link — which "could charge advertisers as a higher-cost //click//" — they parse the destination out of the URL instead {[reitinger2024_creepy]}. If you must resolve where an ad goes, resolve it without clicking it.
+  * **A form submission has a side effect on a real system.** Filling and submitting forms at scale sends real newsletter signups, real support tickets, real account registrations and real search queries against production databases. This is why EvoCrawl's evaluation runs against **ten self-hostable web applications** — WordPress, HotCRP, Kanboard, ImpressCMS, phpBB, Humhub, GitLab and others, each shipped as a Dockerfile in the repository's ''experiments/'' folder — rather than against the live web {[guo2025_evocrawl]}. The interaction depth it achieves is only safe on a system you own. Read the [[Practices:Ethics|ethics page]] before you point a form-filler at a site list.
+  * **Interaction multiplies your load by more than you expect.** The Khaleesi comparison above is 3× the requests from the same list. Rate limits, politeness delays and any ''robots.txt'' policy you have should be set for the interactive figure, not the landing-page one.
+  * **You are more visible, and being detected is not neutral.** A crawler that clicks and scrolls looks less like a bot to a fraud model and more like a user — which is the point, and is also why the resulting impressions and clicks are harder for the site to filter out. "We made the crawler look human" and "we made our traffic indistinguishable from the traffic the site bills for" are the same sentence.
+None of this argues for staying on the landing page. It argues for saying, in the paper, which interactions you performed, on how many sites, and what you did to keep them from being charged for.
 ==== The code ====
@@ Line 253: / Line 264: @@
 // Third parties are served from a SECOND origin (127.0.0.2) so the crawler
 // counts them as third-party the way a real measurement would.
+//
+// PORTABILITY: Linux gives you the whole 127.0.0.0/8 loopback range for free.
+// macOS does not — binding 127.0.0.2 fails with EADDRNOTAVAIL until you add the
+// alias:  sudo ifconfig lo0 alias 127.0.0.2 up
+// (undo with `sudo ifconfig lo0 -alias 127.0.0.2`). On Windows, use WSL.
 import http from 'node:http';
@@ Line 475: / Line 491: @@
 ===== Forms, Logins and the Half of the Web You Are Not Measuring =====
-Of the 857 web crawls in this corpus, **553 (64.5%)** explicitly did not authenticate, **204 (23.8%)** do not say, and **81 (9.5%)** got past a login of any kind — 42 by registering an account, 24 by logging in manually and 15 with automated login. Not one of the 857 used SSO.((The corpus has exactly one paper with ''authentication: sso'' — an Android subscription-app study — and it is not a web crawl, so it is outside this page's population.)) Whatever your topic, **a crawl of the logged-out web is the design nine papers in ten use**, and the reason is usually the one Lauinger et al. give: login walls stop deeper traversal.
+Of the 857 web crawls in this corpus, **553 (64.5%)** are recorded as not authenticating, **204 (23.8%)** do not say, and **at most 81 (9.5%)** got past a login of any kind — 42 by registering an account, 24 by logging in manually and 15 with automated login. Not one of the 857 used SSO.((The corpus has exactly one paper with ''authentication: sso'' — an Android subscription-app study — and it is not a web crawl, so it is outside this page's population.)) Whatever your topic, **at most one crawl in ten gets past a login**, and the reason is usually the one Lauinger et al. give: login walls stop deeper traversal. Treat the 553 with care: ''authentication'' shares the same single ''crawlConfig'' quote as the depth, and no discriminant was run for it, so an unknown share of those ''none'' values are the extractor's default reading of a paper that simply never mentions logging in.
 Two things worth knowing before you decide the effort is not worth it.
@@ Line 487: / Line 503: @@
 A form-filler that assigns to ''input.value'' fires no ''keydown'', no ''keyup'' and no ''input'' event unless the framework listens for programmatic changes, so the very scripts you are trying to catch never run. If your measurement is about what happens //while// a user types, you must actually type. Playwright's ''locator.pressSequentially()'' and Puppeteer's ''ElementHandle.type()'' with a ''delay'' both do this; ''locator.fill()'' does not.
-The rest of the login problem — Shepherd-style automated login, SSO, and automated registration — is [[Programming:Registration|Automating Login and Registration to Websites]].
+The rest of the login problem — Shepherd-style automated login, SSO, and automated registration — is [[Programming:Registration|Automating Login and Registration to Websites]], **currently a stub**, so expect to go to the papers directly for now.
 ===== Your Crawler Will Be Noticed =====
@@ Line 545: / Line 561: @@
 Five of seven, in two sentences. That is the whole cost.
+===== If You Just Need a Default =====
+Absent a better reason — and the sections above are the better reasons — this is a defensible starting design for a privacy or security crawl over a ranked site list, and every element of it is something you can point a reviewer at:
+  - **Harvest the frontier once.** Visit each landing page, collect its same-site links, store them with the date. Freeze that list and replay it in every arm of the experiment.
+  - **Pick subpages at random from it, with a published seed**, and publish the exclusion list if you use one. Do not take them in DOM order unless you want the navigation chrome.
+  - **Start at ten and run a saturation pilot for //your// metric** on a few hundred sites before committing to the number. Ten is the field's median and 100 is where the one published curve flattens; you will not know which is right for a metric nobody has plotted.
+  - **On every page, including the landing page: scroll to the bottom, then wait a fixed, stated time.** Same treatment on the subpages as on the landing page, or the comparison between them is not a comparison.
+  - **Click nothing by default.** Add clicking only where your question needs it, and when you do, exclude ad elements — see [[#Interacting is not free, and it is not only your cost|above]].
+  - **Crawl logged out**, and say so. Getting past a login is a project, not a setting.
+  - **Log the failures** — sites yielding zero pages, sites yielding fewer subpages than asked, challenges you could not pass — and report all three.
+The point of the default is not that it is optimal. It is that every line of it is a sentence you can write in the methods section, which is more than most published crawls manage.
 ===== Methodology and Limitations of These Figures =====