Differences

This shows you the differences between two versions of the page.

--- provenance:programming:crawler:webxray:residue_pass [2026/09/11 22:40] – Refresh the code appendix: report_tail_pass.py now prints both metrics in B and E2 and three sensitivity runs in E3; check_tail_figures.py parses them and asserts all three; mutation harness extended to 23. Authored by Claude karel.kubicek.claude
+++ provenance:programming:crawler:webxray:residue_pass [2026/09/17 07:24] (current) – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude
@@ Line 19: / Line 19: @@
 the scripts, J–R the outputs.
-**Why this is a third page.** These blocks are 390 kB and the
+**Why this is a third page.** These blocks are 392 kB and the
-random-sample appendix's are already 286 kB; putting them together would
+random-sample appendix's are already 289 kB; putting them together would
-make one 676 kB page carrying two probe logs. Both figures are computed
+make one 681 kB page carrying two probe logs. Both figures are computed
 when this page is generated, from the same file lists the two manifests use, so
 neither can go stale while the other moves. That page's own "why this is a
@@ Line 46: / Line 46: @@
 | E | ''scripts/owner_tail_corrections.py'' | 8,654 | ''4f30dc28ccd516bf'' | no |
 | F | ''scripts/report_tail_pass.py'' | 20,276 | ''91ef54dfa24bf452'' | no |
-| G | ''scripts/check_tail_figures.py'' | 16,607 | ''30640e5cf99280fa'' | no |
+| G | ''scripts/check_tail_figures.py'' | 17,939 | ''401e5956f79c89e9'' | no |
 | H | ''scripts/check_tail_merge_mutations.sh'' | 2,704 | ''6a77cccbcb73f91b'' | no |
-| I | ''scripts/check_tail_figures_mutations.sh'' | 4,558 | ''4a89023003a14d50'' | no |
+| I | ''scripts/check_tail_figures_mutations.sh'' | 4,975 | ''6664ce97cdbfb5a0'' | no |
 | J | ''out/check_tail_merge_mutations-output.txt'' | 1,142 | ''ebb83f0c555d0527'' | no |
 | K | ''out/tail_probe.txt'' | 177,214 | ''5f487708ad8b7ba9'' | **yes** — one lone CR normalised to LF |
-| L | ''out/owner_tail_merge-output.txt'' | 1,042 | ''b1fe634a350c43bc'' | no |
+| L | ''out/owner_tail_merge-output.txt'' | 1,042 | ''d6c7dcdf4535daa7'' | no |
 | M | ''out/tail_verify-output.txt'' | 2,512 | ''f772acd997a598d8'' | no |
-| N | ''out/owner_tail_corrections-output.txt'' | 6,659 | ''df43d5b6edba64dd'' | no |
+| N | ''out/owner_tail_corrections-output.txt'' | 6,669 | ''808c2387aea5a34c'' | no |
 | O | ''out/owner_random_sample-tail-output.txt'' | 52,594 | ''f46dd7c6f37fb231'' | no |
 | P | ''out/owner_random_sample-tail.diff'' | 63,634 | ''07ae04745a451e50'' | no |
 | Q | ''out/report_tail_pass-output.txt'' | 13,529 | ''c5e2c644f7b4c35c'' | no |
-| R | ''out/check_tail_figures_mutations-output.txt'' | 1,258 | ''6d9e16aa449db68a'' | no |
+| R | ''out/check_tail_figures_mutations-output.txt'' | 1,329 | ''9c28a7c4448ac143'' | no |
 ===== A. scripts/whois43.py — a WHOIS client, because this container has none =====
@@ Line 1189: / Line 1189: @@
     ap.add_argument("--page", required=True)
     ap.add_argument("--output", required=True)
+    ap.add_argument("--corrections", default=None,
+                    help="out/owner_tail_corrections-output.txt, for the "
+                         "hand-changed verdict count the report cannot produce")
     args = ap.parse_args()
     page = open(args.page, encoding="utf8").read()
@@ Line 1425: / Line 1428: @@
             if b_ != a_:
                 sup[f"{b_}%"] = f"{a_}%"
+    # Figures the REPORT does not produce, but the page carries and this sitting
+    # moved. Section G can only sweep what the report prints, and the total
+    # number of hand-changed verdicts is not one of those: it is the first
+    # sitting's 29 plus whatever this one re-scored. It went stale in exactly
+    # the way this check exists to stop -- the page carried it TWICE, the round
+    # four patch fixed one copy, and the site-wide sweep found the other. So it
+    # is derived here from the corrections script's own output rather than
+    # typed, and both directions are asserted.
+    if args.corrections:
+        ctext = open(args.corrections, encoding="utf8").read()
+        mres = re.search(r"^actions: .*rescore=(\d+)", ctext, re.M)
+        if not mres:
+            sys.exit("FATAL: could not read the rescore count from "
+                     f"{args.corrections}")
+        prior = 29          # the 2026-09-05 sitting's own hand changes
+        total = prior + int(mres.group(1))
+        need("hand-changed verdicts, total", f"{total} of the 525")
+        sup[f"{prior} of the 525"] = f"{total} of the 525"
     # **p-values are deliberately NOT swept for absence.** They are written at
     # two or three decimals interchangeably, and a two-decimal form is a prefix
@@ Line 1545: / Line 1567: @@
 #   bash scripts/check_tail_figures_mutations.sh
 set -u
-PAGE=${1:-pages/own_tail6.txt}
+PAGE=${1:-pages/own_tail7.txt}
 OUT=out/report_tail_pass-output.txt
+CORR=out/owner_tail_corrections-output.txt
 TMP=$(mktemp -d)
 pass=0; fail=0
@@ Line 1557: / Line 1580: @@
     fail=$((fail+1)); return
   fi
-  if python3 scripts/check_tail_figures.py --page "$TMP/p.txt" --output "$OUT" >/dev/null 2>&1; then
+  if python3 scripts/check_tail_figures.py --page "$TMP/p.txt" --output "$OUT" \
+       --corrections "$CORR" >/dev/null 2>&1; then
     echo "SURVIVED $desc"
     fail=$((fail+1))
@@ Line 1602: / Line 1626: @@
 mutate "reintroduce the superseded Tracker Radar rate 73.9% -> 73.7%" 's|73\.9% | 73.7% |'
 mutate "reintroduce the superseded webXray encounter rate 93.8% -> 93.9%" 's|\*\*93\.8%\*\*|93.9%|'
+# This one is a single copy of a figure the page carries TWICE. It is the drift
+# the site-wide sweep found after round four had fixed the other copy, and the
+# guard could not see it because the report script does not print the quantity.
+mutate "revert ONE copy of the hand-changed count 30 -> 29 of the 525" 's|30 of the 525|29 of the 525|'
 echo
@@ Line 4108: / Line 4136: @@
 ===== L. Unedited output: scripts/owner_tail_merge.py =====
-Run as ''python3 scripts/owner_tail_merge.py --sample out/owner_sample.json --base out/adj_rows_scored.json --tail out/tail40.json --dir out/tail_adj --out out/adj_rows_tail.json''.
+Run as ''python3 scripts/owner_tail_merge.py %%--sample%% out/owner_sample.json %%--base%% out/adj_rows_scored.json %%--tail%% out/tail40.json %%--dir%% out/tail_adj %%--out%% out/adj_rows_tail.json''.
 <code>
@@ Line 4132: / Line 4160: @@
   archived-legal           2
-wrote /tmp/discard_rows.json (175 rows)
+wrote out/adj_rows_tail.json (175 rows)
 </code>
 ===== M. Unedited output: scripts/owner_verify_sources.py on the second pass's rows =====
-Run as ''python3 scripts/owner_verify_sources.py --rows out/tail_rows_only.json --cache out/tail_verify_cache --out out/tail_verify.json''.
+Run as ''python3 scripts/owner_verify_sources.py %%--rows%% out/tail_rows_only.json %%--cache%% out/tail_verify_cache %%--out%% out/tail_verify.json''.
 <code>
@@ Line 4173: / Line 4201: @@
 ===== N. Unedited output: scripts/owner_tail_corrections.py =====
-Run as ''python3 scripts/owner_tail_corrections.py --rows out/adj_rows_tail.json --out out/adj_rows_tail_final.json''.
+Run as ''python3 scripts/owner_tail_corrections.py %%--rows%% out/adj_rows_tail.json %%--out%% out/adj_rows_tail_final.json''.
 <code>
@@ Line 4229: / Line 4257: @@
 actions: keep=2, rescore=1, resource=3
 rows whose VERDICT changed: 1 (i.ua)
-wrote /tmp/discard2.json (175 rows)
+wrote out/adj_rows_tail_final.json (175 rows)
 </code>
 ===== O. Unedited output: scripts/owner_random_sample.py re-run after the second pass =====
-Run as ''python3 scripts/owner_random_sample.py --sample out/owner_sample.json --rows out/adj_rows_tail_final.json --table --wiki''.
+Run as ''python3 scripts/owner_random_sample.py %%--sample%% out/owner_sample.json %%--rows%% out/adj_rows_tail_final.json %%--table%% %%--wiki%%''.
 <code>
@@ Line 4966: / Line 4994: @@
 ===== Q. Unedited output: scripts/report_tail_pass.py =====
-Run as ''python3 scripts/report_tail_pass.py --sample out/owner_sample.json --before out/adj_rows_scored.json --after out/adj_rows_tail_final.json --before-output out/owner_random_sample-output.txt --after-output out/owner_random_sample-tail-output.txt''.
+Run as ''python3 scripts/report_tail_pass.py %%--sample%% out/owner_sample.json %%--before%% out/adj_rows_scored.json %%--after%% out/adj_rows_tail_final.json %%--before-output%% out/owner_random_sample-output.txt %%--after-output%% out/owner_random_sample-tail-output.txt''.
 <code>
@@ Line 5204: / Line 5232: @@
 caught   reintroduce the superseded Tracker Radar rate 73.9% -> 73.7%
 caught   reintroduce the superseded webXray encounter rate 93.8% -> 93.9%
+caught   revert ONE copy of the hand-changed count 30 -> 29 of the 525
-mutations caught: 23; survived or broken: 0
+mutations caught: 24; survived or broken: 0
 </code>
@@ Line 5215: / Line 5244: @@
   * [[provenance:programming:crawler:webxray:random_sample]] — the first sitting's code: the draw, the estimator, the inter-rater pass.
   * [[literature:corpus]] — corpus scope, the selection funnel, extraction stability.
+===== Markup sweep, 2026-09-17 =====
+Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.