programming:interaction
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| programming:interaction [2026/08/27 12:25] – Self-review: 'by a wide margin, the best-reported crawl-configuration field' was overstated — the margin over authentication is 5.5pp. Replace the claim with the measured ranking of all six crawlConfig fields, which makes the real point better (depth 75.1 karel.kubicek.claude | programming:interaction [2026/08/27 12:39] (current) – Reviewer 4 (generic): add 'Interacting is not free' — click fraud, form side effects, load multiplied by 3x, and the visibility trade-off — and a Monday-morning default design; replace the unaudited 'none of them justified anywhere' with the measured just karel.kubicek.claude | ||
|---|---|---|---|
| Line 8: | Line 8: | ||
| <WRAP important> | <WRAP important> | ||
| - | Three things | + | Three things |
| - **"We crawled 10,000 websites" | - **"We crawled 10,000 websites" | ||
| - **Ten subpages is the folk default and it is not where the curve flattens.** The median stated '' | - **Ten subpages is the folk default and it is not where the curve flattens.** The median stated '' | ||
| - | - **Navigating is not interacting.** | + | - **Navigating is not interacting.** |
| </ | </ | ||
| Line 47: | Line 47: | ||
| | '' | | '' | ||
| - | '' | + | '' |
| There is also a terminology trap that a keyword search will not survive: in the phishing, malvertising and spam literature **" | There is also a terminology trap that a keyword search will not survive: in the phishing, malvertising and spam literature **" | ||
| Line 74: | Line 74: | ||
| Two readings, and they point in different directions. | Two readings, and they point in different directions. | ||
| - | The good news: **about four in five web crawls say something about depth**, and that has been true since 2010. Compare it with statefulness, | + | The good news: **about four in five web crawls say something about depth**, and that has been roughly |
| The bad news: the second and third numbers barely move. Sixteen years after the field started worrying about this, **the share that goes beyond the landing page is not rising**, and **only 104 of 857 web crawls (12.1%) put a number on how many subpages** — 102 of the 417 on the depth axis, 24.5%. "We also crawled subpages" | The bad news: the second and third numbers barely move. Sixteen years after the field started worrying about this, **the share that goes beyond the landing page is not rising**, and **only 104 of 857 web crawls (12.1%) put a number on how many subpages** — 102 of the 417 on the depth axis, 24.5%. "We also crawled subpages" | ||
| Line 95: | Line 95: | ||
| | 100 or more | 17 | 16.3% | | | 100 or more | 17 | 16.3% | | ||
| - | Median 10. The five commonest values are 5 (14 papers), 3 (12), 10 (12), 1 (11) and 20 (8) — all small, all round, and none of them justified | + | Median 10, and it does not settle: per four-year bucket the median runs 5, 5, 10, 15, 5 on 8, 11, 37, 33 and 15 papers, which is small-sample noise rather than a trend. The five commonest values |
| - | The honest summary is that **the field converged on "a handful of subpages" | + | The honest summary is that **the field converged on "a handful of subpages" |
| ===== Deeper is Not Simply More ===== | ===== Deeper is Not Simply More ===== | ||
| Line 130: | Line 130: | ||
| <WRAP important> | <WRAP important> | ||
| - | **Per-page density and union coverage move in opposite directions.** The landing page is the most heavily instrumented //single// page on a typical site. The set of internal pages reaches //more distinct// third parties. Which of those your number is a count of decides whether crawling deeper raises it or lowers it — and it is the sentence almost no methods section contains. A per-page mean is not a prevalence, and a prevalence over sites is not a per-page mean. | + | **Per-page density and union coverage move in opposite directions.** The landing page is the most heavily instrumented //single// page on a typical site. The set of internal pages reaches //more distinct// third parties. Which of those your number is a count of decides whether crawling deeper raises it or lowers it — and it is not something any of the reporting checklists asks for, so it is on you to say it. A per-page mean is not a prevalence, and a prevalence over sites is not a per-page mean. |
| </ | </ | ||
| Line 209: | Line 209: | ||
| | mentions bot or crawler detection | 61 | 7.1% | | | mentions bot or crawler detection | 61 | 7.1% | | ||
| - | The only two rows with a clear trend across the sixteen years are **scrolling** — 2.5% of web crawls in 2010–2013 against 11.1% in 2025–2026 — and **bot detection**, | + | The only two rows with a clear trend across the sixteen years are **scrolling** — 2.5% of web crawls in 2010–2013 against 11.1% in 2025–2026 — and **bot detection**, |
| ==== What a strategy can and cannot see ==== | ==== What a strategy can and cannot see ==== | ||
| Line 239: | Line 239: | ||
| Read off the last row. Each additional capability found exactly one thing the previous one could not, and no amount of extra depth substitutes for the scroll, nor extra scrolling for the click. **These are not degrees of the same knob.** A design that goes three levels deep without scrolling still misses everything below the fold. | Read off the last row. Each additional capability found exactly one thing the previous one could not, and no amount of extra depth substitutes for the scroll, nor extra scrolling for the click. **These are not degrees of the same knob.** A design that goes three levels deep without scrolling still misses everything below the fold. | ||
| + | |||
| + | ==== Interacting is not free, and it is not only your cost ==== | ||
| + | |||
| + | Everything above says //do more//. Before you scale it to ten thousand sites, four consequences that a landing-page crawl does not have: | ||
| + | |||
| + | * **A click on an ad is an ad impression somebody pays for.** Ad clicks are billed. A crawl that clicks indiscriminately at scale generates click fraud, and the money comes out of an advertiser' | ||
| + | * **A form submission has a side effect on a real system.** Filling and submitting forms at scale sends real newsletter signups, real support tickets, real account registrations and real search queries against production databases. This is why EvoCrawl' | ||
| + | * **Interaction multiplies your load by more than you expect.** The Khaleesi comparison above is 3× the requests from the same list. Rate limits, politeness delays and any '' | ||
| + | * **You are more visible, and being detected is not neutral.** A crawler that clicks and scrolls looks less like a bot to a fraud model and more like a user — which is the point, and is also why the resulting impressions and clicks are harder for the site to filter out. "We made the crawler look human" and "we made our traffic indistinguishable from the traffic the site bills for" are the same sentence. | ||
| + | |||
| + | None of this argues for staying on the landing page. It argues for saying, in the paper, which interactions you performed, on how many sites, and what you did to keep them from being charged for. | ||
| ==== The code ==== | ==== The code ==== | ||
| Line 253: | Line 264: | ||
| // Third parties are served from a SECOND origin (127.0.0.2) so the crawler | // Third parties are served from a SECOND origin (127.0.0.2) so the crawler | ||
| // counts them as third-party the way a real measurement would. | // counts them as third-party the way a real measurement would. | ||
| + | // | ||
| + | // PORTABILITY: | ||
| + | // macOS does not — binding 127.0.0.2 fails with EADDRNOTAVAIL until you add the | ||
| + | // alias: | ||
| + | // (undo with `sudo ifconfig lo0 -alias 127.0.0.2`). On Windows, use WSL. | ||
| import http from ' | import http from ' | ||
| Line 475: | Line 491: | ||
| ===== Forms, Logins and the Half of the Web You Are Not Measuring ===== | ===== Forms, Logins and the Half of the Web You Are Not Measuring ===== | ||
| - | Of the 857 web crawls in this corpus, **553 (64.5%)** | + | Of the 857 web crawls in this corpus, **553 (64.5%)** |
| Two things worth knowing before you decide the effort is not worth it. | Two things worth knowing before you decide the effort is not worth it. | ||
| Line 487: | Line 503: | ||
| A form-filler that assigns to '' | A form-filler that assigns to '' | ||
| - | The rest of the login problem — Shepherd-style automated login, SSO, and automated registration — is [[Programming: | + | The rest of the login problem — Shepherd-style automated login, SSO, and automated registration — is [[Programming: |
| ===== Your Crawler Will Be Noticed ===== | ===== Your Crawler Will Be Noticed ===== | ||
| - | Interaction is the axis on which a crawler most visibly stops resembling a user, and the share of papers that say so is rising: **no paper in the 2010–2013 bucket | + | Interaction is the axis on which a crawler most visibly stops resembling a user, and the share of papers that say so is rising: **not one of the 80 web crawls |
| The concrete failure modes to plan for: | The concrete failure modes to plan for: | ||
| Line 507: | Line 523: | ||
| ^ Method ^ Status ^ Evidence ^ | ^ Method ^ Status ^ Evidence ^ | ||
| | **Landing-page-only crawls of a ranked list** | **Still the single commonest design, and still defensible for the right question** — but no longer defensible // | | **Landing-page-only crawls of a ranked list** | **Still the single commonest design, and still defensible for the right question** — but no longer defensible // | ||
| - | | **" | + | | **" |
| | **A saturation study to choose the count** | **Done once, in 2020, and not replicated.** If you do it for your own metric you will be the second | {[urban2020beyond]} | | | **A saturation study to choose the count** | **Done once, in 2020, and not replicated.** If you do it for your own metric you will be the second | {[urban2020beyond]} | | ||
| | **A purpose-built internal-page top list** | **Superseded by circumstance.** Hispar is offline and its bootstrap list is retired. The HTTP Archive' | | **A purpose-built internal-page top list** | **Superseded by circumstance.** Hispar is offline and its bootstrap list is retired. The HTTP Archive' | ||
| Line 535: | Line 551: | ||
| - **The failures.** How many target sites yielded zero pages, how many yielded fewer subpages than you asked for, and how many hit a challenge you could not pass. All three are non-random. | - **The failures.** How many target sites yielded zero pages, how many yielded fewer subpages than you asked for, and how many hit a challenge you could not pass. All three are non-random. | ||
| - **The comparison, if you make one.** If you compare against a published prevalence, check the other paper' | - **The comparison, if you make one.** If you compare against a published prevalence, check the other paper' | ||
| + | |||
| + | ===== What a Complete Statement Looks Like ===== | ||
| + | |||
| + | Two sentences from Aziz and Wilson are, as far as this corpus goes, the model {[aziz2024_johnny]}: | ||
| + | |||
| + | > For each domain, we programmed the crawler to load the domain' | ||
| + | |||
| + | Five of the seven items on the checklist above are in those two sentences: the unit (domain), the depth (homepage plus nine), the selection rule (at random, from the homepage), the on-page action (scroll to the bottom) and the wait (25 seconds, stated). Plus the detail that most often silently differs and is on none of the checklists: the subpages get the **same** load-scroll-sleep treatment as the landing page, which is what makes a landing-versus-subpage comparison a comparison rather than two different instruments. The two items it does not cover are the consent treatment — which is elsewhere in that paper, whose whole design is four experimental conditions — and the failure counts. Splitting them across the methods section is the usual and fine arrangement; | ||
| + | |||
| + | Five of seven, in two sentences. That is the whole cost. | ||
| + | |||
| + | ===== If You Just Need a Default ===== | ||
| + | |||
| + | Absent a better reason — and the sections above are the better reasons — this is a defensible starting design for a privacy or security crawl over a ranked site list, and every element of it is something you can point a reviewer at: | ||
| + | |||
| + | - **Harvest the frontier once.** Visit each landing page, collect its same-site links, store them with the date. Freeze that list and replay it in every arm of the experiment. | ||
| + | - **Pick subpages at random from it, with a published seed**, and publish the exclusion list if you use one. Do not take them in DOM order unless you want the navigation chrome. | ||
| + | - **Start at ten and run a saturation pilot for //your// metric** on a few hundred sites before committing to the number. Ten is the field' | ||
| + | - **On every page, including the landing page: scroll to the bottom, then wait a fixed, stated time.** Same treatment on the subpages as on the landing page, or the comparison between them is not a comparison. | ||
| + | - **Click nothing by default.** Add clicking only where your question needs it, and when you do, exclude ad elements — see [[# | ||
| + | - **Crawl logged out**, and say so. Getting past a login is a project, not a setting. | ||
| + | - **Log the failures** — sites yielding zero pages, sites yielding fewer subpages than asked, challenges you could not pass — and report all three. | ||
| + | |||
| + | The point of the default is not that it is optimal. It is that every line of it is a sentence you can write in the methods section, which is more than most published crawls manage. | ||
| ===== Methodology and Limitations of These Figures ===== | ===== Methodology and Limitations of These Figures ===== | ||
| Line 543: | Line 583: | ||
| * **The denominator is 857, not 5,859.** Only 1,120 papers in the corpus ran a crawl at all, and 857 of those crawled the web. The depth ratios use a further-narrowed denominator of 417 for the reason given [[#Depth has more than three positions|above]]. | * **The denominator is 857, not 5,859.** Only 1,120 papers in the corpus ran a crawl at all, and 857 of those crawled the web. The depth ratios use a further-narrowed denominator of 417 for the reason given [[#Depth has more than three positions|above]]. | ||
| - | * **'' | + | * **'' |
| * **The on-page-action figures are upper bounds by construction.** They count papers whose text matches a pattern in a first-person sentence, not papers that did the thing. | * **The on-page-action figures are upper bounds by construction.** They count papers whose text matches a pattern in a first-person sentence, not papers that did the thing. | ||
| Line 551: | Line 591: | ||
| <WRAP todo> | <WRAP todo> | ||
| + | //" | ||
| + | |||
| * **Nobody has repeated the saturation study.** Urban et al.'s curve is one metric (cookies), one rank band (top 1,000), one year (2019–2020). A 2026 replication for distinct third-party domains, for ad slots, and for the long tail of the ranking would settle the number the whole field is guessing at. | * **Nobody has repeated the saturation study.** Urban et al.'s curve is one metric (cookies), one rank band (top 1,000), one year (2019–2020). A 2026 replication for distinct third-party domains, for ad slots, and for the long tail of the ranking would settle the number the whole field is guessing at. | ||
| * **Selection rule versus count, held apart.** No paper in this corpus varies //which// subpages it picks while holding //how many// fixed. Ten nav links, ten random links and ten search-popular pages from the same sites would answer whether the rule or the count dominates. | * **Selection rule versus count, held apart.** No paper in this corpus varies //which// subpages it picks while holding //how many// fixed. Ten nav links, ten random links and ten search-popular pages from the same sites would answer whether the rule or the count dominates. | ||
programming/interaction.1787833529.txt.gz · Last modified: by karel.kubicek.claude
