| Both sides previous revisionPrevious revision | |
| design:sampling [2026/09/17 07:22] – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude | design:sampling [2026/09/21 14:55] (current) – Crawl depth: scope the 15.0%/12.1% and 22.8%/37.2% figures against programming:interaction (680 vs 417/857 denominators, same 155 landing-only count, sets not identical); refresh the stale Hispar box and Open Questions bullet with HTTP Archive's one-secon karel.kubicek.claude |
|---|
| If you do want internal pages, you cannot enumerate them either, and exhaustively crawling a site is both expensive and, as Aqeel et al. note, arguably unethical: it "may introduce fake page visits or ad impressions, distort the statistics that the web site collects, increase the load on the web server, and cost the web site money". Their solution is the practical one — query a search engine for ''site:example.com'' and take the top //N// results, on the argument that search results are "biased towards what people search for and click on" and so approximate pages real users reach. Their released list, Hispar, was one landing plus up to 49 internal pages for about 2,000 sites, refreshed weekly. | If you do want internal pages, you cannot enumerate them either, and exhaustively crawling a site is both expensive and, as Aqeel et al. note, arguably unethical: it "may introduce fake page visits or ad impressions, distort the statistics that the web site collects, increase the load on the web server, and cost the web site money". Their solution is the practical one — query a search engine for ''site:example.com'' and take the top //N// results, on the argument that search results are "biased towards what people search for and click on" and so approximate pages real users reach. Their released list, Hispar, was one landing plus up to 49 internal pages for about 2,000 sites, refreshed weekly. |
| |
| <WRAP todo>**Hispar is gone.** ''hispar.cs.duke.edu'' does not resolve as of 2026-08-12 (NXDOMAIN). The method reproduces easily, but a search API is now a paid dependency, and no maintained equivalent list was found. If you know of one, please add it.</WRAP> | <WRAP todo>**Hispar is gone, and its nearest replacement is not a sample.** ''hispar.cs.duke.edu'' does not resolve as of 2026-08-12 (NXDOMAIN) and the method needs a search API, which is now a paid dependency. **There is one maintained source of internal pages**, though it is a by-product rather than a list: since April 2022 the HTTP Archive crawls **one secondary page per site** alongside the root, and both sit in the public BigQuery dataset with ''is_root_page'' and ''root_page'' columns to separate them. Read the selection rule before treating it as an internal-page sample — the crawl controller runs ''MAX_DEPTH = 1'' and ''MAX_BREADTH = 1'' and takes the **first** same-origin link on the page, which on most sites is a navigation item such as //About// or //Contact//. It is a real internal page and it is not a random one, and a measured demonstration of what that rule misses is on [[programming:interaction#The one page list that included internal pages is gone|Interaction and crawl depth]]. If you know of a maintained internal-page //list//, please add it.</WRAP> |
| |
| Only **15.0% of the crawling papers in this population state how many subpages per site** they visited; among those that do, the median is 10. | Only **15.0% of the crawling papers in this population state how many subpages per site** they visited — 102 of 680 — and among those that do, the median is 10. [[programming:interaction#How Deep the Field Actually Goes|Interaction and crawl depth]] reports the same field over its own population and gets **12.1%**, 104 of the 857 papers in the corpus that ran a web crawl; see //[[#One page per site is the norm|One page per site is the norm]]// for why the two denominators differ. |
| |
| ==== A crawl is not a sample of what people load ==== | ==== A crawl is not a sample of what people load ==== |
| |
| **Only 32.4% visit more than one page per site**, and only **15.0% state how many subpages per site** (median 10, where stated). Among the 492 papers whose unit is explicitly ''websites'' and which crawled, **24.8% visited the landing page only** — which is precisely the population Aqeel et al. {[aqeel2020_landing]} showed would need its claims revised. | **Only 32.4% visit more than one page per site**, and only **15.0% state how many subpages per site** (median 10, where stated). Among the 492 papers whose unit is explicitly ''websites'' and which crawled, **24.8% visited the landing page only** — which is precisely the population Aqeel et al. {[aqeel2020_landing]} showed would need its claims revised. |
| | |
| | **These shares are not comparable with the depth figures on [[programming:interaction#Depth has more than three positions|Interaction and crawl depth]], and neither page is wrong.** That page counts the **857** corpus papers that ran a //web// crawl and then reports depth over the **417** of them that put a value on the site-depth axis at all, excluding ''single-target-page'' and ''not-stated''; this page counts the **680** papers with a web-unit population that also crawled, and keeps every value in the denominator — the 222 single-target-page, the 69 not-stated and the 14 with no crawl-configuration record are all inside the 680 above. The two arrive at the same raw count of landing-page-only papers, **155**, and divide it differently: **22.8% of 680 here, 37.2% of 417 there**. Read the 37.2% as //among papers that say//, and the 22.8% as //among papers that crawled//. The 155-paper sets are also not identical — the two populations are built from different fields — so the matching count is not an invitation to merge them. |
| |
| ==== Methodology and limitations of these figures ==== | ==== Methodology and limitations of these figures ==== |
| <WRAP todo> | <WRAP todo> |
| * **How much do results actually move between rank strata?** Several papers stratify, but we found none that reports the same measurement per stratum as its contribution. That table — prevalence of //X// at ranks 1–1k, 1k–10k, 10k–100k, 100k–1M — would tell the field how much its top-//n// habit costs, and it is a cheap by-product of any stratified crawl. | * **How much do results actually move between rank strata?** Several papers stratify, but we found none that reports the same measurement per stratum as its contribution. That table — prevalence of //X// at ranks 1–1k, 1k–10k, 10k–100k, 100k–1M — would tell the field how much its top-//n// habit costs, and it is a cheap by-product of any stratified crawl. |
| * **No maintained internal-page list.** Hispar is offline (see above), and a search API is now a paid dependency. The 2020 result that landing pages misrepresent sites therefore stands unaddressed, with no tooling to address it. | * **No maintained internal-page list.** Hispar is offline (see above) and a search API is now a paid dependency. The HTTP Archive's one-secondary-page-per-site is the only maintained source of internal pages, and its rule — the first same-origin link in DOM order — makes it a by-product, not a sample ([[programming:interaction#The one page list that included internal pages is gone|what that rule misses, measured]]). A maintained, versioned, reproducible internal-page //list// remains a genuine gap, and the 2020 result that landing pages misrepresent sites still stands largely unaddressed. |
| * **Attrition is not in any structured record.** The corpus cannot say how many papers report both the drawn and the analysed denominator, because the extraction schema has no field for it. A targeted full-text study would be worth doing; our impression from reading is that it is a minority. | * **Attrition is not in any structured record.** The corpus cannot say how many papers report both the drawn and the analysed denominator, because the extraction schema has no field for it. A targeted full-text study would be worth doing; our impression from reading is that it is a minority. |
| * **Is rank weighting used outside third-party analysis?** Prominence {[englehardt2016online]} is the one rank-weighted estimator we found in this corpus, and it is specific to ranking third parties. We did not find a paper that reports a weighted //prevalence// — "//x//% of sites, weighted by rank" — alongside the unweighted one, nor one that reweights a rank-stratified sample by stratum size to recover a frame-wide figure. Both are routine in survey statistics. If you know of an example, please add it. | * **Is rank weighting used outside third-party analysis?** Prominence {[englehardt2016online]} is the one rank-weighted estimator we found in this corpus, and it is specific to ranking third parties. We did not find a paper that reports a weighted //prevalence// — "//x//% of sites, weighted by rank" — alongside the unweighted one, nor one that reweights a rank-stratified sample by stratum size to recover a frame-wide figure. Both are routine in survey statistics. If you know of an example, please add it. |
