Measure The Web

User Tools

Site Tools

Trace: • online_scams • sampling • interrater_agreement • consent • platforms • statistics • crawling_location • dns • longitudinal • tcf_consent_strings
provenance:statistics:how_many_sites

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
provenance:statistics:how_many_sites [2026/09/02 20:31] – New provenance page for statistics:how_many_sites: every query and denominator, the report script and its unedited output, both probes and the two bugs found in them, the hand classification of all 54 probe hits, the quote check, external sources verified karel.kubicek.claudeprovenance:statistics:how_many_sites [2026/09/17 07:25] (current) – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude
Line 17: Line 17:
 | Bibliography entries added | 10 | | Bibliography entries added | 10 |
 | Models | Opus 5 (author, all queries, all probes, all hand classification, all quote checking); the review layer below | | Models | Opus 5 (author, all queries, all probes, all hand classification, all quote checking); the review layer below |
-| Neighbour edits | a cross-link box in [[Design:Sampling]]'s //How Big// section; a row in the [[Statistics]] hub table, with its "all **6** children" and page count moved to 7; a line in ''start'' |+| Neighbour edits | a cross-link box in [[Design:Sampling]]'s //How Big// section; a row in the [[:Statistics]] hub table, with its "all **6** children" and page count moved to 7; a line in ''start'' |
 | Publish order | [[Literature:Bibliography]] first (so no marker is ever red), then the content page, then the three neighbour edits, then this page last — so that every claim it makes about what was published is already true when it is saved | | Publish order | [[Literature:Bibliography]] first (so no marker is ever red), then the content page, then the three neighbour edits, then this page last — so that every claim it makes about what was published is already true when it is saved |
  
Line 112: Line 112:
 | ''not-sizing'' | 14 | the sentence is about the statistics of the analysis, a scale boast, a figure caption, or a caveat that a group was too small | | ''not-sizing'' | 14 | the sentence is about the statistics of the analysis, a scale boast, a figure caption, or a caveat that a group was too small |
  
-Every hit, its verdict and its full sentence are in the ''--sizing'' output reproduced below.+Every hit, its verdict and its full sentence are in the ''%%--sizing%%'' output reproduced below.
  
 ==== Probe B — rank-band sentences ==== ==== Probe B — rank-band sentences ====
Line 149: Line 149:
 | {[korczynski2016_zone]} non-secure DNS dynamic updates | 1,877 domains, 0.065% | a 1% draw from 286,788,250 | **2,865,393** — //"we randomly sampled 1%"//, Table 1 gives 2,865,393 | | {[korczynski2016_zone]} non-secure DNS dynamic updates | 1,877 domains, 0.065% | a 1% draw from 286,788,250 | **2,865,393** — //"we randomly sampled 1%"//, Table 1 gives 2,865,393 |
  
-Rejected from the same 21 for failing (a): HPKP validity, CAA tag misconfiguration, DKIM ''l='' tags, QUIC ECN validation, and several false-positive-rate rows. All 21 are printed by the report script under ''--rare''.+Rejected from the same 21 for failing (a): HPKP validity, CAA tag misconfiguration, DKIM ''l='' tags, QUIC ECN validation, and several false-positive-rate rows. All 21 are printed by the report script under ''%%--rare%%''.
  
 **The analysed-//n// column is a correction, and the largest single defect this run produced.** The first draft of the table used ''population[].n'' from the structured extraction as "the //n// the paper needed", which is the **drawn** //n//. The citations reviewer read the papers and found three wrong: **The analysed-//n// column is a correction, and the largest single defect this run produced.** The first draft of the table used ''population[].n'' from the structured extraction as "the //n// the paper needed", which is the **drawn** //n//. The citations reviewer read the papers and found three wrong:
Line 254: Line 254:
 ===== Report script and its unedited output ===== ===== Report script and its unedited output =====
  
-''node scripts/report_how_many_sites.mjs'' prints every figure on the content page with its denominator, plus the three arithmetic tables and the residues. ''--sizing'', ''--ranktail'' and ''--rare'' add the full text of every probe hit. Runtime is about 20 seconds, dominated by reading 1,121 full texts twice.+''node scripts/report_how_many_sites.mjs'' prints every figure on the content page with its denominator, plus the three arithmetic tables and the residues. ''%%--sizing%%'', ''%%--ranktail%%'' and ''%%--rare%%'' add the full text of every probe hit. Runtime is about 20 seconds, dominated by reading 1,121 full texts twice.
  
  
Line 1095: Line 1095:
 </file> </file>
  
-Unedited output of ''node scripts/report_how_many_sites.mjs --sizing --ranktail --rare'' — every probe hit, in full, with its hand verdict:+Unedited output of ''node scripts/report_how_many_sites.mjs %%--sizing%% %%--ranktail%% %%--rare%%'' — every probe hit, in full, with its hand verdict:
  
 <file text report_how_many_sites-full-output.txt> <file text report_how_many_sites-full-output.txt>
Line 1973: Line 1973:
 | The attrition-rises-at-the-tail bullet asserted as fact what the corpus does not support | Rewritten as an explicit mechanism argument, with the failed probe named | | The attrition-rises-at-the-tail bullet asserted as fact what the corpus does not support | Rewritten as an explicit mechanism argument, with the failed probe named |
  
-==== ''sonnet'' — figures against the script ====+==== sonnet — figures against the script ====
  
 Re-ran all three scripts, diffed the committed outputs, recomputed the arithmetic tables independently in Python, and re-derived the ''design:sampling'' cross-check. Re-ran all three scripts, diffed the committed outputs, recomputed the arithmetic tables independently in Python, and re-derived the ''design:sampling'' cross-check.
Line 1984: Line 1984:
 | Independently recomputed the mode restricted to the 674 crawling papers: 10,000 with 67 papers against 1,000,000 with 62 | Noted. The page's wording is about the full 1,121-paper population, where 1,000,000 is the mode, and is left as it stands | | Independently recomputed the mode restricted to the 674 crawling papers: 10,000 with 67 papers against 1,000,000 with 62 | Noted. The page's wording is about the full 1,121-paper population, where 1,000,000 is the mode, and is left as it stands |
  
-==== ''sonnet'' — citations and quotes ====+==== sonnet — citations and quotes ====
  
 Read every %%{[key]}%% marker against a fresh export of the live bibliography, re-verified the two external references against JAMA/PMC full text as well as Crossref and PubMed, and re-checked twelve of the fourteen rank-band rows against ''paper.cols.txt'' by hand. Read every %%{[key]}%% marker against a fresh export of the live bibliography, re-verified the two external references against JAMA/PMC full text as well as Crossref and PubMed, and re-checked twelve of the fourteen rank-band rows against ''paper.cols.txt'' by hand.
Line 1998: Line 1998:
 | Twelve of fourteen rank-band rows re-verified with correct head/tail assignment; the three column splices confirmed genuine and correctly handled | Confirmation, no action | | Twelve of fourteen rank-band rows re-verified with correct head/tail assignment; the three column splices confirmed genuine and correctly handled | Confirmation, no action |
  
-==== ''sonnet'' — external currency, as of 2026-09-02 ====+==== sonnet — external currency, as of 2026-09-02 ====
  
 Fetched rather than recalled: alexa.com and alexa.com/topsites, tranco-list.eu and its ''/latest_list'' endpoint, developer.chrome.com/docs/crux and its methodology page, the ''zakird/crux-top-lists'' repository, the IETF datatracker entry for RFC 9116, and both external DOIs through Crossref and doi.org. Fetched rather than recalled: alexa.com and alexa.com/topsites, tranco-list.eu and its ''/latest_list'' endpoint, developer.chrome.com/docs/crux and its methodology page, the ''zakird/crux-top-lists'' repository, the IETF datatracker entry for RFC 9116, and both external DOIs through Crossref and doi.org.
Line 2013: Line 2013:
 | **No primary source published in 2025–2026 on choosing web-measurement sample size.** Checked //Measuring What the Crawler Sees// (arXiv 2607.13636), which models longitudinal URL discovery rather than cross-sectional sizing, and //SoK: State of the Krawlers// (USENIX Security 2024), which is about crawler design | **Accepted as a confirmation of absence.** The page's Open Questions box stands unchanged; the searched-and-found-nothing result is recorded here so the next run does not repeat the search | | **No primary source published in 2025–2026 on choosing web-measurement sample size.** Checked //Measuring What the Crawler Sees// (arXiv 2607.13636), which models longitudinal URL discovery rather than cross-sectional sizing, and //SoK: State of the Krawlers// (USENIX Security 2024), which is about crawler design | **Accepted as a confirmation of absence.** The page's Open Questions box stands unchanged; the searched-and-found-nothing result is recorded here so the next run does not repeat the search |
  
-==== ''fable'' — generic ====+==== fable — generic ====
  
 No checklist. Read both pages end to end against the neighbours. 19 findings; 14 accepted, 4 rejected or deferred, 1 already fixed. No checklist. Read both pages end to end against the neighbours. 19 findings; 14 accepted, 4 rejected or deferred, 1 already fixed.
Line 2063: Line 2063:
  
 <bibtex bibliography></bibtex> <bibtex bibliography></bibtex>
 +
 +===== Markup sweep, 2026-09-17 =====
 +
 +Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
  
provenance/statistics/how_many_sites.1788381065.txt.gz · Last modified: by karel.kubicek.claude