Differences

This shows you the differences between two versions of the page.

--- provenance:programming:crawler:webxray:random_sample [2026/09/11 19:20] – Point to the new residue_pass appendix; refresh the owner_verify_sources.py block after its whois:// / charset / rate-limit fixes. Authored by Claude karel.kubicek.claude
+++ provenance:programming:crawler:webxray:random_sample [2026/09/17 07:24] (current) – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude
@@ Line 26: / Line 26: @@
 AF–AK their outputs.
-**Why this is a separate page.** These blocks are 285 kB, and the parent
+**Why this is a separate page.** These blocks are 290 kB, and the parent
 provenance page is already large without them. Raw size alone would not decide
 it — ''literature:bibliography'' is 388 kB and renders fine — but ~60 kB of this
@@ Line 57: / Line 57: @@
 | C | ''out/adj/INSTRUCTIONS.md'' | 4,965 | ''1985026abac3d888'' | no |
 | D | ''scripts/owner_merge_rows.py'' | 4,572 | ''84ee130f2e8c2310'' | no |
-| E | ''scripts/owner_verify_sources.py'' | 12,417 | ''6c3778961aa5672c'' | no |
+| E | ''scripts/owner_verify_sources.py'' | 12,941 | ''4e07cc2df4e06b37'' | no |
 | F | ''scripts/owner_verify_rendered.mjs'' | 2,923 | ''0bf6b4615724e84d'' | no |
-| G | ''scripts/owner_corrections.py'' | 6,847 | ''f145c0908d4bf9a6'' | no |
+| G | ''scripts/owner_corrections.py'' | 9,595 | ''a9e92c9f427e408c'' | no |
 | H | ''scripts/owner_verdict_fixes.py'' | 7,894 | ''2b1bc378b8cb6750'' | no |
 | I | ''scripts/owner_random_sample.py'' | 17,780 | ''9ff3ddf1dab339c5'' | no |
@@ Line 70: / Line 70: @@
 | P | ''scripts/wayback_webxray.sh'' | 2,625 | ''80c3b9a6378508b0'' | no |
 | Q | ''out/owner_sample-output.txt'' | 1,657 | ''90f34e73b14c797b'' | no |
-| R | ''out/owner_corrections-output.txt'' | 4,830 | ''961e16e7549cd98c'' | no |
+| R | ''out/owner_corrections-output.txt'' | 6,399 | ''c0a0a60fb92b6e67'' | no |
 | S | ''out/owner_verdict_fixes-output.txt'' | 4,006 | ''0322ffec6b5dbdbf'' | no |
 | T | ''out/owner_citation_audit-output.txt'' | 1,252 | ''b40a2f1db1afce1d'' | no |
@@ Line 681: / Line 681: @@
         # burst limit, and both stubs in the first run answered correctly 25 s
         # later. Retrying forever would turn a registrar's block into a hang.
+        err = None
         for attempt in range(2):
             if attempt:
@@ Line 687: / Line 688: @@
                 out = whois43.ask(server, dom).encode("utf8")
             except Exception as e:
-                return None, f"whois: {e}"
+                # A raised exception is a transport failure too -- a reset or a
+                # timeout is exactly what a rate limiter does -- so it must take
+                # the retry as well. It did not until 2026-09-11, when a review
+                # pass read the comment above ("two attempts, 20 s apart") and
+                # then read the code, which returned on the first exception.
+                err, out = e, b""
+                continue
+            err = None
             if len(out) >= 400 and dom.encode().lower() in out.lower():
                 break
+        if err is not None:
+            return None, f"whois: {err}"
         # Registrar WHOIS servers rate-limit. MarkMonitor answers the fourth
         # rapid query with a stub that has no registrant block, and caching that
@@ Line 862: / Line 872: @@
 </file>
-===== G. scripts/owner_corrections.py — the ten citations fixed by hand =====
+===== G. scripts/owner_corrections.py — the 11 citations fixed by hand =====
 Every row a machine flagged and a human then re-checked, with the action taken and why: ''requote'' (the claim held, the quote was a paraphrase), ''resource'' (wrong page for a right claim), ''demote'' (no primary source on a second look; the row leaves every rate).
@@ Line 880: / Line 890: @@
     quotation marks" failure, and it happened in 4 of 175 rows despite an
     instruction that said verbatim -- which is the argument for verifying every
-    citation by machine rather than trusting the adjudicator.
+    citation by machine rather than trusting the adjudicator. A fifth requote,
+    ``compass-fit.jp``, is a transcription slip rather than a paraphrase: one
+    dropped character. It is counted with them because the remedy is the same,
+    not because it is the same defect. It was
+    found on 2026-09-11 and applied on 2026-09-17, by re-running a *fixed*
+    checker over rows the broken one had passed: ``norm()`` used to delete every non-Latin character,
+    so ``compass-fit.jp``'s Japanese quote had been checked on the six Latin
+    letters ``COMPASS-FIT`` it happens to contain. A checker that cannot see a
+    script cannot police it, and every row it passed in that script is
+    unverified until it is re-run.
   * ``resource`` -- the cited URL was the wrong page (a 404 template, a
     JS-only route, a blocked host). A different primary source for the same
@@ Line 950: / Line 969: @@
         "registrant law firm, which is a textbook WHOIS trap, but calling it an "
         "error needs a source naming the real owner and there is none."),
+    "compass-fit.jp": ("requote",
+        "https://www.microad.co.jp/news/detail/1185/",
+        "株式会社マイクロアド（本社：東京都渋谷区 代表取締役社長：渡辺健太郎　以下、マイクロアド）の台湾における現地法人MicroAd Taiwan,Ltd.（本社：台北市 董事長：丸木勇人　以下、マイクロアド台湾）は、ログリー株式会社（本社：東京都渋谷区、代表取締役社長：吉永浩和、証券コード：6579、以下、ログリー社）と業務提携し、台湾国内での、ネイティブ型広告配信サービス「COMPASS-FIT」の提供を開始致しました。",
+        "found 2026-09-11, applied 2026-09-17. The norm() character-class fix "
+        "of 2026-09-11 was re-run that day over all 175 rows of the first "
+        "sitting -- rows the broken checker had passed, this one on the six "
+        "Latin letters COMPASS-FIT its Japanese quote happens to contain. The "
+        "mismatch is one character: the body reads "
+        "台湾国内での、ネイティブ型広告配信サービス... and the old quote drops the "
+        "full-width comma after 台湾国内での. (The comma-less form is the "
+        "page's headline, which stops at 開始; the 開始致しました。 ending is the "
+        "body's. The old quote is one and then the other, so it is on "
+        "neither.) Replaced by the body sentence in full, 株式会社マイクロアド ... "
+        "開始致しました。. This is NOT the shortest span on the page naming a "
+        "MicroAd entity and the service -- the page title is 71 normalised "
+        "characters and names both -- it is the one that names the parent and "
+        "the operating subsidiary together, which is what owner_today = "
+        "MicroAd, Inc. rests on where the old quote named only the service. "
+        "Verified OK against a live fetch of 2026-09-17 (39,110 bytes). A "
+        "transcription slip, not a paraphrase: the claim never rested on the "
+        "missing character, so no published rate moves."),
 }
@@ Line 1002: / Line 1042: @@
 </file>
-===== H. scripts/owner_verdict_fixes.py — retiring the ''self-named'' verdict =====
+===== H. scripts/owner_verdict_fixes.py — retiring the self-named verdict =====
 The methodological correction made during the run, kept rather than smoothed away. Each of the 26 re-scores carries the entity string, the adjudicated owner and the reason.
@@ Line 2233: / Line 2273: @@
 ===== R. Unedited output: scripts/owner_corrections.py =====
-Run as ''python3 scripts/owner_corrections.py --rows out/adj_rows.json --out out/adj_rows_final.json''.
+Run as ''python3 scripts/owner_corrections.py %%--rows%% out/adj_rows.json %%--out%% out/adj_rows_final.json''.
 <code>
- corrections applied to 175 rows, after machine verification of every cited source: demote=1, requote=4, resource=5
+ corrections applied to 175 rows, after machine verification of every cited source: demote=1, requote=5, resource=5
 twiago.com             requote   quote was re-ordered ('HRB 81984 (Amtsgericht Köln)'); the page reads 'Handelsregisternummer : Amtsgericht Köln HRB 81984'. Claim unchanged.
@@ Line 2287: / Line 2327: @@
                        was quote : Condé Nast Entertainment
                        now source: None
+compass-fit.jp         requote   found 2026-09-11, applied 2026-09-17. The norm() character-class fix of 2026-09-11 was re-run that day over all 175 rows of the first sitting -- rows the broken checker had passed, this one on the six Latin letters COMPASS-FIT its Japanese quote happens to contain. The mismatch is one character: the body reads 台湾国内での、ネイティブ型広告配信サービス... and the old quote drops the full-width comma after 台湾国内での. (The comma-less form is the page's headline, which stops at 開始; the 開始致しました。 ending is the body's. The old quote is one and then the other, so it is on neither.) Replaced by the body sentence in full, 株式会社マイクロアド ... 開始致しました。. This is NOT the shortest span on the page naming a MicroAd entity and the service -- the page title is 71 normalised characters and names both -- it is the one that names the parent and the operating subsidiary together, which is what owner_today = MicroAd, Inc. rests on where the old quote named only the service. Verified OK against a live fetch of 2026-09-17 (39,110 bytes). A transcription slip, not a paraphrase: the claim never rested on the missing character, so no published rate moves.
+                       was source: https://www.microad.co.jp/news/detail/1185/
+                       was quote : 台湾国内でのネイティブ型広告配信サービス「COMPASS-FIT」の提供を開始致しました。
+                       now source: https://www.microad.co.jp/news/detail/1185/
 wrote out/adj_rows_final.json
@@ Line 2293: / Line 2338: @@
 ===== S. Unedited output: scripts/owner_verdict_fixes.py =====
-Run as ''python3 scripts/owner_verdict_fixes.py --rows out/adj_rows_final.json --out out/adj_rows_scored.json''.
+Run as ''python3 scripts/owner_verdict_fixes.py %%--rows%% out/adj_rows_final.json %%--out%% out/adj_rows_scored.json''.
 <code>
@@ Line 2394: / Line 2439: @@
 ===== V. Unedited output: scripts/owner_random_sample.py — every figure, and the 175-row table =====
-Run as ''python3 scripts/owner_random_sample.py --sample out/owner_sample.json --rows out/adj_rows_scored.json --table --wiki''.
+Run as ''python3 scripts/owner_random_sample.py %%--sample%% out/owner_sample.json %%--rows%% out/adj_rows_scored.json %%--table%% %%--wiki%%''.
 <code>
@@ Line 5442: / Line 5487: @@
 ===== AF. Unedited output: scripts/owner_irr_sample.py =====
-Run as ''python3 scripts/owner_irr_sample.py --sample out/owner_sample.json --batches out/adj --out out/irr --n 35''.
+Run as ''python3 scripts/owner_irr_sample.py %%--sample%% out/owner_sample.json %%--batches%% out/adj %%--out%% out/irr %%--n%% 35''.
 <code>
@@ Line 5466: / Line 5511: @@
 ===== AG. Unedited output: scripts/owner_merge_rows.py on the second rater's rows =====
-Run as ''python3 scripts/owner_merge_rows.py --sample out/irr/irr_sample_for_merge.json --dir out/irr/adj2 --out out/irr/adj2_rows.json''.
+Run as ''python3 scripts/owner_merge_rows.py %%--sample%% out/irr/irr_sample_for_merge.json %%--dir%% out/irr/adj2 %%--out%% out/irr/adj2_rows.json''.
 <code>
@@ Line 5478: / Line 5523: @@
 ===== AH. Unedited output: scripts/owner_verify_sources.py on the second rater's rows =====
-Run as ''python3 scripts/owner_verify_sources.py --rows out/irr/adj2_rows.json --cache out/irr/verify_cache --out out/irr/verify_irr.json''.
+Run as ''python3 scripts/owner_verify_sources.py %%--rows%% out/irr/adj2_rows.json %%--cache%% out/irr/verify_cache %%--out%% out/irr/verify_irr.json''.
 <code>
@@ Line 5518: / Line 5563: @@
 ===== AI. Unedited output: scripts/owner_irr_corrections.py =====
-Run as ''python3 scripts/owner_irr_corrections.py --rows out/irr/adj2_rows.json --out out/irr/adj2_rows_final.json''.
+Run as ''python3 scripts/owner_irr_corrections.py %%--rows%% out/irr/adj2_rows.json %%--out%% out/irr/adj2_rows_final.json''.
 <code>
@@ Line 5534: / Line 5579: @@
 ===== AJ. Unedited output: scripts/owner_irr_kappa.py — every agreement figure and every disagreement =====
-Run as ''python3 scripts/owner_irr_kappa.py --r1 out/adj_rows.json --r1-scored out/adj_rows_scored.json --r2 out/irr/adj2_rows.json --subsample out/irr/irr_subsample.json --json-out out/owner_irr_kappa.json''.
+Run as ''python3 scripts/owner_irr_kappa.py %%--r1%% out/adj_rows.json %%--r1-scored%% out/adj_rows_scored.json %%--r2%% out/irr/adj2_rows.json %%--subsample%% out/irr/irr_subsample.json %%--json-out%% out/owner_irr_kappa.json''.
 <code>
@@ Line 5762: / Line 5807: @@
   * [[provenance:programming:crawler:webxray:residue_pass]] — the same treatment for the 2026-09-11 second pass over the 40 rows this sample could not settle.
   * [[literature:corpus]] — corpus scope, the selection funnel, extraction stability.
+===== Markup sweep, 2026-09-17 =====
+Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.