| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| programming:interaction [2026/09/22 01:32] – Add the 19 papers with no crawlConfig object so the 857 authentication buckets reconcile (figures review finding: 553+204+81=838). Authored by Claude karel.kubicek.claude | programming:interaction [2026/10/08 20:49] (current) – Finish PII leakage collateral: back-links and provenance; Authored by Claude karel.kubicek.claude |
|---|
| |
| - **"We crawled 10,000 websites" is not a measurement unit.** A website is not a page. Of the **857 papers in [[literature:corpus|this corpus]] that ran an automated web crawl of the web**, **417** give a value on the site-depth axis, and **155 of those 417 (37.2%)** stayed on the landing page. The landing page is a systematically atypical page — see [[#Deeper is not simply more]]. | - **"We crawled 10,000 websites" is not a measurement unit.** A website is not a page. Of the **857 papers in [[literature:corpus|this corpus]] that ran an automated web crawl of the web**, **417** give a value on the site-depth axis, and **155 of those 417 (37.2%)** stayed on the landing page. The landing page is a systematically atypical page — see [[#Deeper is not simply more]]. |
| - **Ten subpages is the folk default and it is not where the curve flattens.** The median stated ''subpagesPerSite'' in this corpus is **10**. The one paper that actually measured where the increase stops found cookies stabilising **after 100 subpages**, not ten {[urban2020beyond]}. See [[#Where the increase stops]]. | - **Ten subpages is the folk default and it is not where the curve flattens.** The median stated ''subpagesPerSite'' in this corpus is **10**. The one paper in the corpus that actually measured where the increase stops found cookies stabilising **after 100 subpages**, not ten {[urban2020beyond]}, and a 2026 study puts the fixed depth at which the //median// site stops gaining anything at **23 to 43 pages**, depending on the metric {[ramakrishnan2026_crawling]}. See [[#Where the increase stops]]. |
| - **Navigating is not interacting.** A great deal of what a privacy measurement counts — lazily-loaded ad slots, content behind a tab or an accordion, anything a form submission triggers — does not exist at ''load''. In the corpus, **69 of 857 web crawls (8.1%)** describe scrolling in their own methods and **28 (3.3%)** describe filling a form. (Loosen the pattern to any mention anywhere in the paper and it is 120 and 59 — still small.) The measured ladder in [[#What a strategy can and cannot see|the probe below]] shows a beacon that only a scroll reveals and another that only a click reveals — on a fixture built to have them, but built that way because real pages are. | - **Navigating is not interacting.** A great deal of what a privacy measurement counts — lazily-loaded ad slots, content behind a tab or an accordion, anything a form submission triggers — does not exist at ''load''. In the corpus, **69 of 857 web crawls (8.1%)** describe scrolling in their own methods and **28 (3.3%)** describe filling a form. (Loosen the pattern to any mention anywhere in the paper and it is 120 and 59 — still small.) The measured ladder in [[#What a strategy can and cannot see|the probe below]] shows a beacon that only a scroll reveals and another that only a click reveals — on a fixture built to have them, but built that way because real pages are. |
| </WRAP> | </WRAP> |
| |
| * **On Landing and Internal Web Pages: The Strange Case of Jekyll and Hyde in Web Performance Measurement** {[aqeel2020_landing]}, IMC 2020 — the paper that made "we used a top list, so we crawled landing pages" a stated limitation rather than an unexamined default. It reviews 119 web-performance papers and concludes that about two-thirds would need revising to apply to internal pages. Read it even if your topic is privacy, not performance: the argument is about top lists, and every privacy crawl uses one. | * **On Landing and Internal Web Pages: The Strange Case of Jekyll and Hyde in Web Performance Measurement** {[aqeel2020_landing]}, IMC 2020 — the paper that made "we used a top list, so we crawled landing pages" a stated limitation rather than an unexamined default. It reviews 119 web-performance papers and concludes that about two-thirds would need revising to apply to internal pages. Read it even if your topic is privacy, not performance: the argument is about top lists, and every privacy crawl uses one. |
| * **Beyond the Front Page: Measuring Third Party Dynamics in the Field** {[urban2020beyond]}, TheWebConf 2020 — the privacy-side counterpart, published the same year, and the only paper here that measures **how many** subpages you need. Read it for the pre-study in §4, not only for the headline. | * **Beyond the Front Page: Measuring Third Party Dynamics in the Field** {[urban2020beyond]}, TheWebConf 2020 — the privacy-side counterpart, published the same year, and the only paper in the corpus that measures **how many** subpages you need. Read it for the pre-study in §4, not only for the headline. |
| | * **Crawling in the Deep: Evaluating Web Crawling Configurations for Web Privacy Measurements** {[ramakrishnan2026_crawling]}, IMC 2026 — the controlled experiment the rest of this page has to piece together from several papers: traversal order, statefulness, interaction, depth and per-page wait varied on the //same// pages of 9,673 CrUX sites, measured on tracking domains, fingerprinting APIs and tracking cookies. It is newer than [[literature:corpus|the corpus]] (IMC 2026 had not been held when it was built), so it is in none of the corpus counts below. Read §4.3 for the stopping rule and §5.1 for the recommendations. |
| * **Apophanies or Epiphanies? How Crawlers Impact Our Understanding of the Web** {[ahmad2020_apophanies]}, TheWebConf 2020 — what changes when you change the instrument. Read it before you argue that your crawler choice does not matter. | * **Apophanies or Epiphanies? How Crawlers Impact Our Understanding of the Web** {[ahmad2020_apophanies]}, TheWebConf 2020 — what changes when you change the instrument. Read it before you argue that your crawler choice does not matter. |
| * **Towards Realistic and Reproducible Web Crawl Measurements** {[jueckstock2021_realistic]}, TheWebConf 2021 — realism as an axis you can rank designs on, rather than a limitation-section apology. | * **Towards Realistic and Reproducible Web Crawl Measurements** {[jueckstock2021_realistic]}, TheWebConf 2021 — realism as an axis you can rank designs on, rather than a limitation-section apology. |
| |
| A note this page has to make, because the previous version of it carried the opposite claim: the earlier text here said the increase becomes small after about **ten** subpages. That number is not in the paper. The pre-study's smallest step is 25, and its stated stabilisation point is 100.((Checked against ''data/fulltext/2020/WWW/beyond-the-front-page-.../paper.cols.txt'' on 2026-08-27; see [[provenance:programming:interaction]].)) | A note this page has to make, because the previous version of it carried the opposite claim: the earlier text here said the increase becomes small after about **ten** subpages. That number is not in the paper. The pre-study's smallest step is 25, and its stated stabilisation point is 100.((Checked against ''data/fulltext/2020/WWW/beyond-the-front-page-.../paper.cols.txt'' on 2026-08-27; see [[provenance:programming:interaction]].)) |
| | |
| | ==== A second saturation study (2026): per metric, and a stopping rule instead of a number ==== |
| | |
| | Ramakrishnan et al. crawled the CrUX top 10,000 (October 2024 list; 9,673 sites reachable) with OpenWPM, up to **50 live pages** per site out of at most 100 requested, and asked the same question for three metric families at once. The depth analysis is reported for their best configuration — breadth-first, state kept within the site, simulated interaction, 30 s per page — and the authors say the takeaways hold for the others {[ramakrishnan2026_crawling]}. Their unit is strict: the depth at which the **median site's error reaches 0%**, where error is the share of what a full 50-page crawl found on that site that a shorter crawl missed. The population is the sites where 50 live pages were found — about four in five under breadth-first search. |
| | |
| | ^ Metric ^ Fixed depth for 0% median error ^ Adaptive rule: stop after //k// pages with nothing new ^ Median stopping depth ^ Pages crawled, adaptive vs fixed ^ |
| | | tracking domains (EasyList, EasyPrivacy, Fanboy) | 33 | //k// = 11 | 20 | 178.7K vs 234.1K (23.7% fewer) | |
| | | all requested domains | 39 | //k// = 11 | 22 | 202.9K vs 276.6K (26.6% fewer) | |
| | | fingerprinting APIs (OpenWPM's list) | 23 | //k// = 12 | 18 | 100.7K vs 110.7K (9.0% fewer) | |
| | | all Web APIs | 24 | //k// = 13 | 20 | 110.9K vs 115.5K (4.0% fewer) | |
| | | tracking cookies (CookieBlock classifier) | 37 | //k// = 8 | 16 | 95.7K vs 146.3K (34.6% fewer) | |
| | | all cookies | 43 | //k// = 8 | 22 | 117.8K vs 179.7K (34.4% fewer) | |
| | |
| | (The paper states the fixed-depth page counts as "//x//% more" than adaptive; the "fewer" column is the same pair of numbers read the other way.) |
| | |
| | Three things to take from it: |
| | |
| | * **Ten is too few on every metric measured so far.** The fixed depths run from 23 to 43, and Urban et al.'s cookie curve flattens at 100. The two studies use different units — a mean cookie count against a per-site 0% median error — but neither puts the knee anywhere near the field's median. |
| | * **Stop per site, not per study.** Sites saturate at very different depths, so any single fixed number is inefficient on many sites and too shallow for some. A "stop after //k// consecutive pages with nothing new" rule needs only a per-site set of what has been seen, and //k// = 13 reached 0% median error on all six metrics (//k// = 8 was enough for cookies). It is not free everywhere: for tracking domains at //k// = 11, the 95th-percentile site still runs to the 50-page cap. |
| | * **The advantage depends on how much error you tolerate.** At a looser target for tracking domains — three sites in four under 10% error — a fixed 28 pages does the job, and the adaptive rule (//k// = 13) saves only about 5% of pages. Adaptive stopping pays most when you want nothing missed. |
| | |
| | Its limits are the ones to state if you cite it: a 50-page ceiling, so "0% error" means nothing missed //relative to page 50// and says nothing about page 51; popular sites only; one US academic vantage point; OpenWPM on Firefox 130 with no bot mitigation; and no consent-banner interaction, so everything is the pre-consent state. A re-run on 1,000 of the sites five months later, on freshly collected pages, gave the same adaptive thresholds — identical for domains and APIs, within one page for cookies — though breadth-first now reached 50 live pages on 72% of sites rather than 79%. |
| |
| ===== Choosing Which Subpages: a Sampling Decision in Disguise ===== | ===== Choosing Which Subpages: a Sampling Decision in Disguise ===== |
| |
| * **Same-origin is not the same as first-party.** A link to ''shop.example.com'' from ''example.com'' is same-site and cross-origin; a link to ''example.co.uk'' is neither. Say which test you used, and say whether you used the Public Suffix List. | * **Same-origin is not the same as first-party.** A link to ''shop.example.com'' from ''example.com'' is same-site and cross-origin; a link to ''example.co.uk'' is neither. Say which test you used, and say whether you used the Public Suffix List. |
| * **The frontier is state-dependent.** Which links exist on the landing page depends on your consent choice, your geolocation, your logged-in state and the hour. Two arms of an experiment that each harvest their own links are not visiting the same pages. Harvest the link set **once**, freeze it, and replay it in every arm — that is what Urban et al. do (collect first-party hyperlinks one day before the first measurement, then reuse the same list across regions), and what Lin et al. do by replaying the same CSS selectors across all three crawl groups. | * **The frontier is state-dependent.** Which links exist on the landing page depends on your consent choice, your geolocation, your logged-in state and the hour. Two arms of an experiment that each harvest their own links are not visiting the same pages. Harvest the link set **once**, freeze it, and replay it in every arm — that is what Urban et al. do (collect first-party hyperlinks one day before the first measurement, then reuse the same list across regions), and what Lin et al. do by replaying the same CSS selectors across all three crawl groups. Ramakrishnan et al. build their whole design on it: one page-collection crawl per traversal strategy, then every configuration replays that ordered page list, "whereas if each crawl visited different pages, the variations in privacy metrics could be due to those pages rather than the crawling parameters" {[ramakrishnan2026_crawling]}. |
| | |
| | ==== Traversal order: go breadth-first ==== |
| | |
| | Once you go past one hop, the rule above has a second half: in what order do you expand the frontier? Ramakrishnan et al. compared breadth-first, depth-first with backtracking, and a breadth-limited hybrid (breadth-first but following at most three links per page) on the same 9,673 sites {[ramakrishnan2026_crawling]}: |
| | |
| | * **Breadth-first finds live pages; depth-first runs out.** Breadth-first reached 50 live pages on about **79%** of sites, the hybrid on about **67%**, and depth-first fell short on nearly **half** within the 100-request budget. The authors' reading, which they do not test further, is that links deep in a site are less likely to still be live. |
| | * **And the pages it finds carry more.** Breadth-first yielded **1–14% more** tracking domains and **1–26% more** tracking cookies in total than the other strategies at the same nominal depth, across all sites — which includes the sites where depth-first found fewer live pages. Restricted to sites where all three strategies found 50 live pages, breadth-first still had the higher totals (the paper gives no percentage for that comparison), so it is not only a page-count effect. |
| | * **The gap is in the upper tail, not the median.** The median site shows about the same number of tracking domains under every configuration; the differences are at the 75th percentile and above. A per-site median will hide the choice, an aggregate total will show it. |
| | |
| | The security-scanning side points somewhere slightly different. Stafeev and Pellegrino, measuring code and link coverage rather than privacy metrics, found "randomized algorithms, in particular randomized BFS, offer better performance than standard algorithms, providing a general increase of average coverage, from +8% in LoCs up to +21% for JavaScript source code" {[stafeev2024_state]} — but in their data //plain// breadth-first and depth-first are within about 1% of each other, and depth-first is marginally ahead on JavaScript source coverage. Their result is an argument for randomising the frontier — and all three of Ramakrishnan et al.'s strategies pick links at random in Stafeev's sense, so the two studies do not conflict; the breadth-first-over-depth-first argument comes from Ramakrishnan et al.'s privacy metrics, and from the practical point that depth-first runs out of live pages. |
| |
| ==== The one page list that included internal pages is gone ==== | ==== The one page list that included internal pages is gone ==== |
| |
| The only two rows with a clear trend across the sixteen years are **scrolling** — 2.5% of web crawls in 2010–2013 against 11.1% in 2025–2026 — and **bot detection**, which not one paper in the 2010–2013 bucket raises //in a first-person sentence// against 11.8% of the 2025–2026 bucket. (Three of those 80 papers do mention it somewhere in the text — a probe's zero is never the same as nobody.) Clicking has **not** risen: it sits between 29.9% and 43.8% in every bucket, and the //highest// value is the earliest one. Mouse movement and hovering are, and have always been, under 4%. | The only two rows with a clear trend across the sixteen years are **scrolling** — 2.5% of web crawls in 2010–2013 against 11.1% in 2025–2026 — and **bot detection**, which not one paper in the 2010–2013 bucket raises //in a first-person sentence// against 11.8% of the 2025–2026 bucket. (Three of those 80 papers do mention it somewhere in the text — a probe's zero is never the same as nobody.) Clicking has **not** risen: it sits between 29.9% and 43.8% in every bucket, and the //highest// value is the earliest one. Mouse movement and hovering are, and have always been, under 4%. |
| | |
| | ==== How long to stay on the page ==== |
| | |
| | The wait is the on-page choice with a measured answer. Ramakrishnan et al. kept each page open for 30 s — the upper end of what most prior crawls used — timestamped every event, and asked what a shorter wait would have missed, relative to 30 s {[ramakrishnan2026_crawling]}: |
| | |
| | ^ Metric ^ Median site misses nothing by ^ Sites missing nothing at 15 s ^ at 20 s ^ |
| | | tracking domains | 10 s | 77.3% (5-page crawl) | not stated | |
| | | fingerprinting APIs | about half of sites miss nothing at 5 s | 83.8% | over 87% | |
| | | tracking cookies | 10 s | 65.4% | 70.9% | |
| | |
| | Counted per (site, tracking domain) pair instead of per site, **93–95%** are first requested within 15 s of page load, and nearly all by 20 s. The paper's recommendation is **15 s** for a crawl short on time. The paper applies it to all metrics, and its five-month re-run found three sites in four gained nothing past 15 s on any metric. Our own reading of the cookie row adds a caution: about a third of sites still set a new tracking cookie between 15 s and 30 s, so if cookies are the metric, a longer wait costs little and buys something. Two conditions travel with the number: the clock starts at page load, and the paper does not say which of its twelve configurations the timings come from — in the interactive ones the page is scrolled to the bottom every 5 s, so a shorter wait also means fewer scrolls. |
| | |
| | The same paper also puts a size on what light interaction adds, alongside the earlier setup comparisons of Demir et al. {[demir2022_reproducibility]}, {[demir2023_similarity]}. Their interaction is **Tab keystrokes at short random intervals plus a scroll to the bottom every 5 s**, with no clicking or other element interaction. The configuration with interaction //and// kept browser state gave **0.8–11.6% more** tracking domains in total than the other configurations of the same traversal strategy, and its per-site distributions differed significantly from all others under paired Wilcoxon tests with Holm correction {[ramakrishnan2026_crawling]}. For fingerprinting APIs the margin shrinks to 0.5–1.2%, because the set of APIs is finite. It is a real effect and a modest one, and it is for state, keystrokes and scrolling together — the paper never separates them — so it is not a figure for scrolling alone. |
| |
| ==== What a strategy can and cannot see ==== | ==== What a strategy can and cannot see ==== |
| | **Landing-page-only crawls of a ranked list** | **Still the single commonest design, and still defensible for the right question** — but no longer defensible //silently//. Since 2020 there is a citable reason to justify it or drop it | 155 of 417 depth-stating web crawls; {[aqeel2020_landing]}, {[urban2020beyond]} | | | **Landing-page-only crawls of a ranked list** | **Still the single commonest design, and still defensible for the right question** — but no longer defensible //silently//. Since 2020 there is a citable reason to justify it or drop it | 155 of 417 depth-stating web crawls; {[aqeel2020_landing]}, {[urban2020beyond]} | |
| | **"Plus a handful of random subpages"** | **Current practice**, and not derived from any measurement. The overall median is 10; per four-year bucket it is 5, 5, 10, 15 and 5, on 8, 11, 37, 33 and 15 papers — too few to call a trend either way | 104 papers give a count | | | **"Plus a handful of random subpages"** | **Current practice**, and not derived from any measurement. The overall median is 10; per four-year bucket it is 5, 5, 10, 15 and 5, on 8, 11, 37, 33 and 15 papers — too few to call a trend either way | 104 papers give a count | |
| | **A saturation study to choose the count** | **Done once, in 2020, and not replicated.** If you do it for your own metric you will be the second | {[urban2020beyond]} | | | **A saturation study to choose the count** | **Done twice**: cookies on the top 1,000 in 2020, and three metric families on the CrUX top 10,000 in 2026. Both put the knee well above ten; a study for your own metric is still worth running | {[urban2020beyond]}, {[ramakrishnan2026_crawling]} | |
| | | **Adaptive per-site stopping** (stop after //k// pages with nothing new) | **New in 2026, one study, not yet adopted.** Same 0% median error as a fixed depth with 4–35% fewer pages | {[ramakrishnan2026_crawling]} | |
| | | **Breadth-first traversal** | **Current recommendation** for privacy metrics, from one study; randomising the frontier is the separate recommendation from security scanning | {[ramakrishnan2026_crawling]}, {[stafeev2024_state]} | |
| | **A purpose-built internal-page top list** | **Superseded by circumstance.** Hispar is offline and its bootstrap list is retired. The HTTP Archive's one-secondary-page-per-site is what remains, with a known selection rule | {[aqeel2020_landing]}; checks of 2026-08-27 | | | **A purpose-built internal-page top list** | **Superseded by circumstance.** Hispar is offline and its bootstrap list is retired. The HTTP Archive's one-secondary-page-per-site is what remains, with a known selection rule | {[aqeel2020_landing]}; checks of 2026-08-27 | |
| | **Scrolling as a routine step** | **Rising and not yet standard** — 2.5% of 2010–2013 web crawls, 11.1% of 2025–2026 | full-text sweep | | | **Scrolling as a routine step** | **Rising and not yet standard** — 2.5% of 2010–2013 web crawls, 11.1% of 2025–2026 | full-text sweep | |
| |
| - **The unit.** Pages or sites? If a site contributed several pages, is your prevalence over sites or over pages? Say which, and say how you aggregated (any-page, majority, mean). | - **The unit.** Pages or sites? If a site contributed several pages, is your prevalence over sites or over pages? Say which, and say how you aggregated (any-page, majority, mean). |
| - **The depth.** Landing page only, landing plus //N//, or traverse to depth //d// with budget //b//. If you stayed on the landing page, say so — it is a legitimate choice with a citable justification. | - **The depth.** Landing page only, landing plus //N//, or traverse to depth //d// with budget //b//. If you stayed on the landing page, say so — it is a legitimate choice with a citable justification. If you stopped adaptively, give the metric the rule watched, //k// and the cap. |
| - **How you chose the pages.** The link-selection rule verbatim: same-origin or same-site, DOM order or random, seed if random, exclusions, and whether the frontier was harvested once and frozen or re-harvested per arm. | - **How you chose the pages.** The link-selection rule verbatim: same-origin or same-site, DOM order or random, seed if random, exclusions, breadth- or depth-first beyond one hop, and whether the frontier was harvested once and frozen or re-harvested per arm. |
| - **What happened on the page.** Scroll? Click? Type? Wait how long, and for what event — ''load'', ''networkidle'', a fixed timer? A fixed timer is fine; an unstated one is not. | - **What happened on the page.** Scroll? Click? Type? Wait how long, and for what event — ''load'', ''networkidle'', a fixed timer? A fixed timer is fine; an unstated one is not. |
| - **What you did about the banner and the login**, with a pointer to [[Privacy:Consent]] and [[Programming:Registration]]. | - **What you did about the banner and the login**, with a pointer to [[Privacy:Consent]] and [[Programming:Registration]]. |
| - **Harvest the frontier once.** Visit each landing page, collect its same-site links, store them with the date. Freeze that list and replay it in every arm of the experiment. | - **Harvest the frontier once.** Visit each landing page, collect its same-site links, store them with the date. Freeze that list and replay it in every arm of the experiment. |
| - **Pick subpages at random from it, with a published seed**, and publish the exclusion list if you use one. Do not take them in DOM order unless you want the navigation chrome. | - **Pick subpages at random from it, with a published seed**, and publish the exclusion list if you use one. Do not take them in DOM order unless you want the navigation chrome. |
| - **Start at ten and run a saturation pilot for //your// metric** on a few hundred sites before committing to the number. Ten is the field's median and 100 is where the one published curve flattens; you will not know which is right for a metric nobody has plotted. | - **Go breadth-first, cap at 50 pages, and stop each site adaptively**: after 13 consecutive pages that add nothing new to your metric (8 if the metric is cookies) {[ramakrishnan2026_crawling]}. If you need a fixed number instead, ten is too few on every metric measured so far — the published saturation points run from 23 to 100 pages — so run a saturation pilot for //your// metric on a few hundred sites before committing. |
| - **On every page, including the landing page: scroll to the bottom, then wait a fixed, stated time.** Same treatment on the subpages as on the landing page, or the comparison between them is not a comparison. | - **On every page, including the landing page: scroll to the bottom, then wait a fixed, stated time.** 15 s is the measured cost-effective wait {[ramakrishnan2026_crawling]}; for a cookie study, consider longer (see [[#How long to stay on the page]]). Same treatment on the subpages as on the landing page, or the comparison between them is not a comparison. |
| - **Click nothing by default.** Add clicking only where your question needs it, and when you do, exclude ad elements — see [[#Interacting is not free, and it is not only your cost|above]]. | - **Click nothing by default.** Add clicking only where your question needs it, and when you do, exclude ad elements — see [[#Interacting is not free, and it is not only your cost|above]]. |
| - **Crawl logged out**, and say so. Getting past a login is a project, not a setting. | - **Crawl logged out**, and say so. Getting past a login is a project, not a setting. |
| //"Nobody" below means "no paper in this corpus's seven venues, 2010–2026". EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS are not in it, and neither is the web-performance literature outside IMC.// | //"Nobody" below means "no paper in this corpus's seven venues, 2010–2026". EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS are not in it, and neither is the web-performance literature outside IMC.// |
| |
| * **Nobody has repeated the saturation study.** Urban et al.'s curve is one metric (cookies), one rank band (top 1,000), one year (2019–2020). A 2026 replication for distinct third-party domains, for ad slots, and for the long tail of the ranking would settle the number the whole field is guessing at. | * **The saturation study has been repeated once, and both runs stop at their own ceiling.** Urban et al.'s curve is one metric (cookies), one rank band (top 1,000), one year (2019–2020). Ramakrishnan et al. add tracking domains and fingerprinting APIs on the CrUX top 10,000 in 2024–2025, but measure error relative to a 50-page crawl, so they cannot say what lies past page 50 {[ramakrishnan2026_crawling]}. Ad slots as a metric, the long tail of the ranking, and the post-consent state are still unmeasured. |
| * **Selection rule versus count, held apart.** No paper in this corpus varies //which// subpages it picks while holding //how many// fixed. Ten nav links, ten random links and ten search-popular pages from the same sites would answer whether the rule or the count dominates. | * **Selection rule versus count, held apart.** No paper in this corpus varies //which// subpages it picks while holding //how many// fixed. Ramakrishnan et al. do it for traversal order — breadth-first ahead of depth-first even on sites where both found 50 live pages {[ramakrishnan2026_crawling]} — but not for the link-choice rule. Ten nav links, ten random links and ten search-popular pages from the same sites would answer whether the rule or the count dominates. |
| * **What does a scroll actually add, at scale?** The fixture on this page shows it can matter. Nobody has published the distribution of "third parties visible only after scrolling" over a real site list. | * **What does a scroll actually add, at scale?** The fixture on this page shows it can matter, and Ramakrishnan et al. measure a 0.8–11.6% gain in tracking domains for keystrokes and scrolling combined with kept state {[ramakrishnan2026_crawling]}. Nobody has published the scroll-only effect, or the distribution of "third parties visible only after scrolling", over a real site list. |
| * **A replacement for Hispar.** The HTTP Archive's one-first-link-per-site is a start and is not a sample. A maintained, versioned, reproducible internal-page list is a genuine gap and a small enough artefact for one group to own. | * **A replacement for Hispar.** The HTTP Archive's one-first-link-per-site is a start and is not a sample. A maintained, versioned, reproducible internal-page list is a genuine gap and a small enough artefact for one group to own. |
| * **How much does an LLM agent actually change the measurement, and at what cost per site?** The five papers using one report task success, not measurement deltas. The comparison that matters — same site list, scripted crawl versus agent crawl, same metric — has not been published. | * **How much does an LLM agent actually change the measurement, and at what cost per site?** The five papers using one report task success, not measurement deltas. The comparison that matters — same site list, scripted crawl versus agent crawl, same metric — has not been published. |
| * [[Programming:Stateful Stateless|Stateful and Stateless Crawling]] — whether the interaction you performed is still in effect on the next page. | * [[Programming:Stateful Stateless|Stateful and Stateless Crawling]] — whether the interaction you performed is still in effect on the next page. |
| * [[Programming:Registration|Automating Login and Registration to Websites]] — the logged-in web this page's 9.5% figure is about. | * [[Programming:Registration|Automating Login and Registration to Websites]] — the logged-in web this page's 9.5% figure is about. |
| | * [[Privacy:PII leakage]] — form filling, login and site search as the treatment that puts a personal value on the page, and why submitting and not submitting are different treatments. |
| * [[Programming:Crawler|Comparison of Crawling Libraries]] — which of these actions each library can actually perform. | * [[Programming:Crawler|Comparison of Crawling Libraries]] — which of these actions each library can actually perform. |
| * [[Design:Website selection]] and [[Design:Sampling|Representative Sampling Methods]] — the site list, and why subpage selection is a sampling problem with the same failure modes. | * [[Design:Website selection]] and [[Design:Sampling|Representative Sampling Methods]] — the site list, and why subpage selection is a sampling problem with the same failure modes. |
