Measure The Web

User Tools

Site Tools

Trace: • crawler_detection • filter_lists • crawler • pagegraph • tranco • webxray
programming:stateful_stateless

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
programming:stateful_stateless [2026/09/17 07:22] – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claudeprogramming:stateful_stateless [2026/09/29 11:39] (current) – Add Ramakrishnan et al. IMC 2026 within-site stateful/stateless comparison and cookie inversion; update open question. Authored by Claude karel.kubicek.claude
Line 144: Line 144:
   * **A trained profile is treated differently.** Agarwal et al.: "having an established persona from a particular demographic … results in up to 15% more cookies stored than for a baseline with no set persona" {[agarwal2020_stop]}.   * **A trained profile is treated differently.** Agarwal et al.: "having an established persona from a particular demographic … results in up to 15% more cookies stored than for a baseline with no set persona" {[agarwal2020_stop]}.
   * **State accumulates within a site, not only across sites.** Urban et al., under a seeded-stateless design with state kept across a site's own subpages: "subsites set considerably more (36 %) cookies than the respective landing pages. On average, 55 cookies were set when loading a landing page while 78 were set when a subsite was accessed" {[urban2020beyond]}. See [[Programming:Interaction]].   * **State accumulates within a site, not only across sites.** Urban et al., under a seeded-stateless design with state kept across a site's own subpages: "subsites set considerably more (36 %) cookies than the respective landing pages. On average, 55 cookies were set when loading a landing page while 78 were set when a subsite was accessed" {[urban2020beyond]}. See [[Programming:Interaction]].
 +  * **Within a site, stateful crawls see more — except on raw cookie totals.** Ramakrishnan et al. collected one ordered page list per traversal strategy for 9,673 CrUX sites, then replayed it from one vantage point under twelve configurations: three strategies, state kept across a site's pages or cleared after every page, with and without simulated interaction {[ramakrishnan2026_crawling]}. Stateful and interactive crawls generally saw more tracking domains and fingerprinting-API use, and breadth-first + stateful + interaction had the highest totals of the twelve; the paper reports that combination against the rest, not state on its own, and the fingerprinting-API margin is only 0.5–1.2%. Tracking cookies went the other way in aggregate: the //stateless// breadth-first crawls produced the most unique tracking cookies (with interaction, 9.5% more than the stateless crawl without it at 50 pages), because a few sites set many new, uniquely named cookies on every fresh visit — cookies the authors judge "likely are not meaningfully different". The median site's count was about the same under every configuration, while the stateful distributions had the higher 75th percentiles. So a stateless crawl's unique-cookie total can be driven by a handful of cookie-renaming sites; look at the per-site distribution before trusting the larger number. This is state kept //within// a site only; carrying state from one site to the next is a separate axis the authors did not test.
  
 And one that cuts the other way: **cookie respawning with browser fingerprinting** was measured on 30,000 Alexa sites with a stateless harness, by comparing paired visits rather than by accumulating a profile — "1, 150 (3.83%) of the Alexa top 30, 000 websites use cookie respawning with browser fingerprinting" {[fouad2022_cookie]}. Note the qualifier: that figure counts respawning //combined with// fingerprinting, not respawning in general. A stateful //phenomenon// does not always require a stateful //crawl//; sometimes it requires two controlled visits. The same is true of cookie syncing at first contact. What accumulation buys is the //aged// identity, not the mechanism. And one that cuts the other way: **cookie respawning with browser fingerprinting** was measured on 30,000 Alexa sites with a stateless harness, by comparing paired visits rather than by accumulating a profile — "1, 150 (3.83%) of the Alexa top 30, 000 websites use cookie respawning with browser fingerprinting" {[fouad2022_cookie]}. Note the qualifier: that figure counts respawning //combined with// fingerprinting, not respawning in general. A stateful //phenomenon// does not always require a stateful //crawl//; sometimes it requires two controlled visits. The same is true of cookie syncing at first contact. What accumulation buys is the //aged// identity, not the mechanism.
Line 437: Line 438:
 | Authentication | 779 | 69.6% | | Authentication | 779 | 69.6% |
 | At least one browser named | 529 | 47.2% | | At least one browser named | 529 | 47.2% |
-| Consent action | 349 | 31.2% |+| Consent action field populated (schema; not an audited paper claim) | 349 | 31.2% |
 | **Stateful or stateless** | **219** | **19.6%** | | **Stateful or stateless** | **219** | **19.6%** |
 | Headless or headful | 140 | 12.5% | | Headless or headful | 140 | 12.5% |
Line 471: Line 472:
 | ''authentication'' | 59.8% | 70.1% | 69.2% | 72.8% | 69.2% | 12.9 pp | +9.4 pp | | ''authentication'' | 59.8% | 70.1% | 69.2% | 72.8% | 69.2% | 12.9 pp | +9.4 pp |
 | ''browsers'' (≥1 named) | 32.4% | 48.5% | 49.7% | 47.0% | 50.5% | 18.2 pp | **+18.2 pp** | | ''browsers'' (≥1 named) | 32.4% | 48.5% | 49.7% | 47.0% | 50.5% | 18.2 pp | **+18.2 pp** |
-| ''consentAction'' | 24.5% | 29.9% | 32.1% | 33.6% | 29.8% | 9.1 pp | +5.3 pp |+| ''consentAction'' (schema field populated; not a paper-claim series) | 24.5% | 29.9% | 32.1% | 33.6% | 29.8% | 9.1 pp | +5.3 pp |
 | **''statefulness''** | 15.7% | 21.0% | 19.8% | 20.6% | 18.2% | **5.3 pp** | **+2.5 pp** | | **''statefulness''** | 15.7% | 21.0% | 19.8% | 20.6% | 18.2% | **5.3 pp** | **+2.5 pp** |
 | ''headless'' | 1.0% | 15.6% | 13.3% | 14.5% | 11.1% | 14.6 pp | +10.1 pp | | ''headless'' | 1.0% | 15.6% | 13.3% | 14.5% | 11.1% | 14.6 pp | +10.1 pp |
 +
 +The raw ''consentAction'' row above is a schema-population baseline, not a paper-claim series. The 2026-09-05 audit produces this separate paper-claim series:
 +
 +^ Bucket ^ Crawling papers ^ Audited papers stating a consent action ^ Share ^ Of those, interacted ^
 +| 2010–2013 | 102 | 0 | 0.0% | 0 |
 +| 2014–2017 | 167 | 0 | 0.0% | 0 |
 +| 2018–2021 | 308 | 11 | 3.6% | 3 |
 +| 2022–2024 | 345 | 25 | 7.2% | 16 |
 +| 2025–2026* | 198 | 19 | 9.6% | 13 |
 +
 +The old raw-series adjacent tests are all non-significant (p = 0.4008, 0.679, 0.7387 and 0.3911). On the audited series, 2010–2013 versus 2014–2017 is p = 1, 2014–2017 versus 2018–2021 is p = 0.009883, 2018–2021 versus 2022–2024 is p = 0.05751, and 2022–2024 versus provisional 2025–2026 is p = 0.3324.
  
 Denominators are the crawling papers in each bucket, from the table above (102 / 167 / 308 / 345 / 198). Naming the browser gained 18 points and headless mode gained 10 from a near-zero base; statefulness gained 2.5 and has the narrowest **range** of the six (5.3 pp). Interaction depth is the only field whose range is nearly as narrow (6.2 pp), and it got that way by //declining// from 78.4% to 72.2% rather than by standing still. It is not that the field decided the axis does not matter — Demir et al. made it a named criterion in 2022, and Zeber et al. and Jueckstock et al. had made it a measured concern in 2020 and 2021. It is that nothing turned the concern into a reporting norm: no venue asks for it on a checklist, and no widely used tool writes it into a config file that ends up in an artifact. Denominators are the crawling papers in each bucket, from the table above (102 / 167 / 308 / 345 / 198). Naming the browser gained 18 points and headless mode gained 10 from a near-zero base; statefulness gained 2.5 and has the narrowest **range** of the six (5.3 pp). Interaction depth is the only field whose range is nearly as narrow (6.2 pp), and it got that way by //declining// from 78.4% to 72.2% rather than by standing still. It is not that the field decided the axis does not matter — Demir et al. made it a named criterion in 2022, and Zeber et al. and Jueckstock et al. had made it a measured concern in 2020 and 2021. It is that nothing turned the concern into a reporting norm: no venue asks for it on a checklist, and no widely used tool writes it into a config file that ends up in an artifact.
Line 509: Line 521:
  
 ^ Subset of crawling papers ^ N ^ State statefulness ^ Share stating ^ stateless ^ stateful ^ both ^ ^ Subset of crawling papers ^ N ^ State statefulness ^ Share stating ^ stateless ^ stateful ^ both ^
-| Crawls that acted on a consent notice | 36 | 22 | **61.1%** | 11 | 4 | 7 | +| Crawls that an audit found acted on a consent notice | 32 | 20 | **62.5%** | 10 | 4 | 6 | 
-| …the 28 of those whose interaction was verified by hand | 28 | 16 | **57.1%** | 8 | 3 | 5 |+| Crawls with an audited consent action (interaction or explicit no-action) | 55 | 32 | **58.2%** | 18 | 8 | 6 |
 | Crawls that logged in | 90 | 54 | **60.0%** | 7 | 39 | 8 | | Crawls that logged in | 90 | 54 | **60.0%** | 7 | 39 | 8 |
 | Repeat-visit designs (2 or more visits per target) | 199 | 85 | 42.7% | 49 | 21 | 15 | | Repeat-visit designs (2 or more visits per target) | 199 | 85 | 42.7% | 49 | 21 | 15 |
Line 517: Line 529:
 | //all crawling papers// | 1,120 | 219 | 19.6% | 113 | 77 | 29 | | //all crawling papers// | 1,120 | 219 | 19.6% | 113 | 77 | 29 |
  
-The consent row needs a caveat that [[privacy:consent]] supplies: the same 36 papers were hand-audited there and **7 (19.4%) turned out to be extraction false positives** — they never touched a banner. The second row recomputes the rate on the 28 the audit **fully** supported (a 29th is supported but with an overstated enum value), and it barely moves, so the finding is robust to the error.+The consent rows use the 2026-09-05 full-text audit. Of the **313** ''no-interaction'' labels, **279 (89.1%)** were unsupported defaults; on the interacting side, **7 of 36 (19.4%)** labels were false positives. Of the **32** audited-interacting papers, **20** state statefulness (62.5%); of the **55** audited-action papers, **32** state statefulness (58.2%). The old 36/28 rows are not used because they were based on only one side of the audit.
  
 The good news first: where the design makes the axis unavoidable, reporting roughly triples. A login is state, and 39 of the 54 login crawls that say anything say stateful. The bad news is the repeat-visit row: **199 papers visit the same target two or more times and 114 of them (57.3%) never say whether state carried between the visits** — which is the one thing that determines whether the repeat visit is a replication or a second step in a sequence. If you take one reporting rule from this page, take that one. The good news first: where the design makes the axis unavoidable, reporting roughly triples. A login is state, and 39 of the 54 login crawls that say anything say stateful. The bad news is the repeat-visit row: **199 papers visit the same target two or more times and 114 of them (57.3%) never say whether state carried between the visits** — which is the one thing that determines whether the repeat visit is a replication or a second step in a sequence. If you take one reporting rule from this page, take that one.
Line 608: Line 620:
   - **{[demir2022_reproducibility]} — what to write down.** Criterion C11 and practice P9 are the reporting standard; the paper also measures how badly the field met it.   - **{[demir2022_reproducibility]} — what to write down.** Criterion C11 and practice P9 are the reporting standard; the paper also measures how badly the field met it.
   - **{[rasaii2025_crumbs]} — the modern stateful design, done well.** Split the list, accept on the first half, measure the second half, randomise the order. The clearest recent example of a result that a stateless crawl cannot produce.   - **{[rasaii2025_crumbs]} — the modern stateful design, done well.** Split the list, accept on the first half, measure the second half, randomise the order. The clearest recent example of a result that a stateless crawl cannot produce.
 +  - **{[ramakrishnan2026_crawling]} — the closest to a clean within-site comparison.** Same page lists, same vantage, stateful and stateless arms among twelve configurations, on tracking domains, fingerprinting APIs and cookies. IMC 2026, newer than the corpus. Read §4.2 for the cookie inversion.
   - **{[urban2020beyond]} — seeded stateless, and why.** One sentence in §4.3.2 explains the whole third design position.   - **{[urban2020beyond]} — seeded stateless, and why.** One sentence in §4.3.2 explains the whole third design position.
   - **{[acar2014_never]} — why a clean profile is hard.** Respawning and syncing across a deliberate state clear.   - **{[acar2014_never]} — why a clean profile is hard.** Respawning and syncing across a deliberate state clear.
Line 617: Line 630:
  
 <WRAP todo> <WRAP todo>
-  * **Nobody has run the clean experiment.** Demir et al. announce one and do not deliver it: §2.2 says "Since the effects of C5 and C11 are not yet adequately discussed by previous work, we analyze them in Section 4", and §4 then runs "four exemplarily case studies focusing on C4, C5, C10, and C12" — repetition, crawler technology, interaction and geolocation. C11, the crawling strategy, is the one criterion they flagged and did not vary; their own runs are described in Appendix C as "stateless coordinated crawls" {[demir2022_reproducibility]}. Sixteen papers in this corpus run both arms, every one of them incidentally to another question. A same-sites, same-time, same-vantage crawl differing //only// in statefulness, reporting the effect on third-party counts, tracker counts and filter-list hit rates, would be a short and highly citable paper.+  * **Nobody has run the clean experiment.** Demir et al. announce one and do not deliver it: §2.2 says "Since the effects of C5 and C11 are not yet adequately discussed by previous work, we analyze them in Section 4", and §4 then runs "four exemplarily case studies focusing on C4, C5, C10, and C12" — repetition, crawler technology, interaction and geolocation. C11, the crawling strategy, is the one criterion they flagged and did not vary; their own runs are described in Appendix C as "stateless coordinated crawls" {[demir2022_reproducibility]}. Sixteen papers in this corpus run both arms, every one of them incidentally to another question. A same-sites, same-time, same-vantage crawl differing //only// in statefulness, reporting the effect on third-party counts, tracker counts and filter-list hit rates, would be a short and highly citable paper. **Update, 2026-09-29:** the closest so far is //within a site// — Ramakrishnan et al. replay the same page lists with state kept and with state cleared after every page, from one vantage point, alongside interaction and traversal-order arms {[ramakrishnan2026_crawling]} (results under [[#What a stateful design buys you, quantified]]). They report the best combination against the rest rather than an isolated state effect, and they leave state carried //across// sites to future work, so this question still stands — fully for the cross-site case.
   * **How much does visit order actually change a stateful result?** The confound is universally acknowledged and never quantified.   * **How much does visit order actually change a stateful result?** The confound is universally acknowledged and never quantified.
   * **Does the seed profile artefact bite?** Englehardt and Narayanan predicted that cloning one seed into ''N'' browsers inflates cookie-sync counts. Nobody has measured the size of the inflation, and it is the design large stateful crawls have used since.   * **Does the seed profile artefact bite?** Englehardt and Narayanan predicted that cloning one seed into ''N'' browsers inflates cookie-sync counts. Nobody has measured the size of the inflation, and it is the design large stateful crawls have used since.
programming/stateful_stateless.1789629776.txt.gz · Last modified: by karel.kubicek.claude