provenance:programming:crawler:webxray:random_sample
Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
| provenance:programming:crawler:webxray:random_sample [2026/09/05 19:35] – Random-sample ownership accuracy appendix: all 16 scripts, 10 unedited outputs and the adjudication brief, with a per-file hash manifest. Authored by Claude karel.kubicek.claude | provenance:programming:crawler:webxray:random_sample [2026/09/17 07:24] (current) – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude | ||
|---|---|---|---|
| Line 4: | Line 4: | ||
| measurement of 2026-09-05** — the one that produced the per-list current / | measurement of 2026-09-05** — the one that produced the per-list current / | ||
| stale / granularity / error rates on | stale / granularity / error rates on | ||
| - | [[programming: | + | [[programming: |
| + | **inter-rater re-adjudication of 2026-09-11** (sections AA–AK), which measured | ||
| + | how far a second model reading the same brief reaches the same verdicts. | ||
| + | |||
| + | The **second pass over the residue**, run later the same day — which | ||
| + | re-adjudicated the 40 domains this sample could not settle, moved 28 of them | ||
| + | into the rates, and changed the content page's headline table and its Fisher | ||
| + | table — has its own appendix, for exactly the reason given below: | ||
| + | [[provenance: | ||
| + | 343 kB, most of it one probe log, and adding them here would have made this page | ||
| + | 628 kB. That page computes both figures when it is generated, from the same file | ||
| + | lists both manifests use. | ||
| Read [[provenance: | Read [[provenance: | ||
| Line 11: | Line 22: | ||
| Corpus-level caveats are on [[literature: | Corpus-level caveats are on [[literature: | ||
| - | **Why this is a separate page.** These blocks are 218 kB, and the parent | + | **Sections are in letter order, A–Z then AA–AK, and that is also chronological: |
| + | A–P are the 2026-09-05 scripts, Q–Z its outputs, AA–AE the 2026-09-11 scripts and | ||
| + | AF–AK their outputs. | ||
| + | |||
| + | **Why this is a separate page.** These blocks are 290 kB, and the parent | ||
| provenance page is already large without them. Raw size alone would not decide | provenance page is already large without them. Raw size alone would not decide | ||
| it — '' | it — '' | ||
| Line 29: | Line 44: | ||
| * **trailing newlines are stripped**, because DokuWiki discards them inside a block anyway; | * **trailing newlines are stripped**, because DokuWiki discards them inside a block anyway; | ||
| - | * **line endings are normalised to LF.** The last column of the table below says, per file, whether that changed a byte. It changes one byte in one file ('' | + | * **line endings are normalised to LF.** The last column of the table below says, per file, whether that changed a byte. It changes one byte in one file ('' |
| The **bytes and SHA-256 in the table are of the file on disk**, not of the | The **bytes and SHA-256 in the table are of the file on disk**, not of the | ||
| Line 42: | Line 57: | ||
| | C | '' | | C | '' | ||
| | D | '' | | D | '' | ||
| - | | E | '' | + | | E | '' |
| | F | '' | | F | '' | ||
| - | | G | '' | + | | G | '' |
| | H | '' | | H | '' | ||
| | I | '' | | I | '' | ||
| Line 55: | Line 70: | ||
| | P | '' | | P | '' | ||
| | Q | '' | | Q | '' | ||
| - | | R | '' | + | | R | '' |
| | S | '' | | S | '' | ||
| | T | '' | | T | '' | ||
| Line 64: | Line 79: | ||
| | Y | '' | | Y | '' | ||
| | Z | '' | | Z | '' | ||
| + | | AA | '' | ||
| + | | AB | '' | ||
| + | | AC | '' | ||
| + | | AD | '' | ||
| + | | AE | '' | ||
| + | | AF | '' | ||
| + | | AG | '' | ||
| + | | AH | '' | ||
| + | | AI | '' | ||
| + | | AJ | '' | ||
| + | | AK | '' | ||
| ===== A. scripts/ | ===== A. scripts/ | ||
| Line 314: | Line 340: | ||
| ===== C. adjudication brief, as given to the sub-agents ===== | ===== C. adjudication brief, as given to the sub-agents ===== | ||
| - | Twenty-one | + | Nineteen |
| < | < | ||
| Line 535: | Line 561: | ||
| OK the quote (or, for a long quote, a 60-character window of it) | OK the quote (or, for a long quote, a 60-character window of it) | ||
| appears in the fetched bytes after normalisation | appears in the fetched bytes after normalisation | ||
| + | OK-MARKUP the quote is not in the page's *text* but is in its markup -- a | ||
| + | citation to a `<script src=...> | ||
| + | text normaliser strips tags, so such a quote can never be found in | ||
| + | the haystack it builds; checking the raw bytes as well is what | ||
| + | distinguishes "the evidence is not there" from "this checker cannot | ||
| + | see that kind of evidence" | ||
| + | the browser pass, both returned NOTFOUND for a `<script src>` line | ||
| + | that is verbatim in kameleoon.com/ | ||
| + | disqualification that had already been acted on. | ||
| NOTFOUND | NOTFOUND | ||
| FETCHFAIL the URL could not be fetched from this host <- needs a human look, | FETCHFAIL the URL could not be fetched from this host <- needs a human look, | ||
| Line 546: | Line 581: | ||
| - | def norm(s): | + | def norm(s, strip_tags=True): |
| s = s.lower() | s = s.lower() | ||
| # Modern sites ship their page text inside an escaped JSON payload, so the | # Modern sites ship their page text inside an escaped JSON payload, so the | ||
| Line 558: | Line 593: | ||
| s = s.replace("& | s = s.replace("& | ||
| s = s.replace("& | s = s.replace("& | ||
| - | s = re.sub(r"< | + | |
| + | | ||
| # Every remaining entity becomes a space on BOTH sides, or "& | # Every remaining entity becomes a space on BOTH sides, or "& | ||
| # page and " | # page and " | ||
| # citation is reported as a fabrication. Caught on medallia.com.au. | # citation is reported as a fabrication. Caught on medallia.com.au. | ||
| s = re.sub(r"& | s = re.sub(r"& | ||
| - | s = re.sub(r" | + | |
| + | # and CJK character, so a quote in one of those scripts normalises to | ||
| + | # whatever Latin substring it happens to contain -- or, if it contains none, | ||
| + | # to the empty string, which this function then reports as NOQUOTE rather | ||
| + | # than as a citation it could not check. It was doing exactly that: | ||
| + | # `tns-counter.ru`' | ||
| + | # " | ||
| + | # both were reported OK. Found on 2026-09-11, when the second rater cited a | ||
| + | # wholly Chinese sentence for `qbox.me` and the result was NOQUOTE. | ||
| + | # The character class is now the one `owner_verify_rendered.mjs` has always | ||
| + | # used, so the bytes pass and the browser pass normalise alike. | ||
| + | | ||
| return re.sub(r" | return re.sub(r" | ||
| + | |||
| + | |||
| + | def decode(body): | ||
| + | """ | ||
| + | |||
| + | Decoding as UTF-8 with ``errors=" | ||
| + | windows-1251 or shift_jis page into U+FFFD, so a Cyrillic or Japanese quote | ||
| + | can never match and the row is reported NOTFOUND -- a fabrication verdict on | ||
| + | a correct citation. The normaliser was already fixed once for this class of | ||
| + | bug (it used to delete non-Latin letters); this is the same bug one layer | ||
| + | earlier, found on 2026-09-11 when ``help.i.ua/ | ||
| + | page declares ``charset=windows-1251`` and every Ukrainian word in it | ||
| + | decoded to replacement characters. | ||
| + | |||
| + | Sniffed from the bytes rather than from the HTTP header, because the header | ||
| + | is not cached -- the cache holds the body alone. If nothing is declared, or | ||
| + | the declared codec is unknown, the old behaviour is the fallback. | ||
| + | """ | ||
| + | m = re.search(rb""" | ||
| + | if m: | ||
| + | enc = m.group(1).decode(" | ||
| + | if enc.lower().replace(" | ||
| + | try: | ||
| + | return body.decode(enc) | ||
| + | except (LookupError, | ||
| + | pass | ||
| + | return body.decode(" | ||
| Line 585: | Line 659: | ||
| except Exception as e: | except Exception as e: | ||
| return None, f"tls: {e}" | return None, f"tls: {e}" | ||
| + | os.makedirs(cache, | ||
| + | open(path, " | ||
| + | return out, None | ||
| + | if url.startswith(" | ||
| + | # The 2026-09-11 second pass over the residue cites registry and | ||
| + | # registrar WHOIS records, which are not reachable over HTTP: the | ||
| + | # protocol is a port-43 socket. Without this branch every such row | ||
| + | # FETCHFAILs, which reads as an unverifiable citation when in fact the | ||
| + | # checker had no client. The URL is `whois://< | ||
| + | # query is re-issued against the *named server*, not re-resolved | ||
| + | # through IANA, so the row is checked against the record it cites. | ||
| + | rest = url[len(" | ||
| + | server, _, dom = rest.partition("/" | ||
| + | if not dom: | ||
| + | return None, " | ||
| + | sys.path.insert(0, | ||
| + | import time | ||
| + | import whois43 | ||
| + | out = b"" | ||
| + | # Two attempts, 20 s apart. One retry is enough: the observed limit is a | ||
| + | # burst limit, and both stubs in the first run answered correctly 25 s | ||
| + | # later. Retrying forever would turn a registrar' | ||
| + | err = None | ||
| + | for attempt in range(2): | ||
| + | if attempt: | ||
| + | time.sleep(20) | ||
| + | try: | ||
| + | out = whois43.ask(server, | ||
| + | except Exception as e: | ||
| + | # A raised exception is a transport failure too -- a reset or a | ||
| + | # timeout is exactly what a rate limiter does -- so it must take | ||
| + | # the retry as well. It did not until 2026-09-11, when a review | ||
| + | # pass read the comment above ("two attempts, 20 s apart" | ||
| + | # then read the code, which returned on the first exception. | ||
| + | err, out = e, b"" | ||
| + | continue | ||
| + | err = None | ||
| + | if len(out) >= 400 and dom.encode().lower() in out.lower(): | ||
| + | break | ||
| + | if err is not None: | ||
| + | return None, f" | ||
| + | # Registrar WHOIS servers rate-limit. MarkMonitor answers the fourth | ||
| + | # rapid query with a stub that has no registrant block, and caching that | ||
| + | # stub turns a correct citation into a permanent NOTFOUND: it happened | ||
| + | # to ``sa-as.com`` and ``blogblog.com`` in this script' | ||
| + | # 2026-09-11, and both verified on a retry 25 seconds later. A short or | ||
| + | # domain-less answer is a transport failure, not evidence, so it is | ||
| + | # neither returned nor cached. | ||
| + | if len(out) < 400 or dom.encode().lower() not in out.lower(): | ||
| + | return None, (f" | ||
| + | f" | ||
| os.makedirs(cache, | os.makedirs(cache, | ||
| open(path, " | open(path, " | ||
| Line 636: | Line 761: | ||
| " | " | ||
| continue | continue | ||
| - | hay = norm(body.decode(" | + | hay = norm(decode(body)) |
| nq = norm(quote) | nq = norm(quote) | ||
| # A quote the adjudicator elided ("A ... B") is checked fragment by | # A quote the adjudicator elided ("A ... B") is checked fragment by | ||
| Line 643: | Line 768: | ||
| frags = [norm(f) for f in re.split(r" | frags = [norm(f) for f in re.split(r" | ||
| frags = [f for f in frags if len(f) >= 12] | frags = [f for f in frags if len(f) >= 12] | ||
| + | # The same normalisation WITHOUT the tag strip, so a quote that is | ||
| + | # itself markup can be found. Kept separate rather than merged into | ||
| + | # `hay`: a prose quote must match the page's text, and dropping the tag | ||
| + | # strip for every row would let a quote match an HTML comment or an | ||
| + | # attribute it was never on. | ||
| + | raw_hay = norm(decode(body), | ||
| if not nq: | if not nq: | ||
| status = " | status = " | ||
| Line 649: | Line 780: | ||
| elif len(frags) > 1 and all(f in hay for f in frags): | elif len(frags) > 1 and all(f in hay for f in frags): | ||
| status = " | status = " | ||
| + | elif nq in raw_hay: | ||
| + | status = " | ||
| else: | else: | ||
| # long quotes get spliced by boilerplate; | # long quotes get spliced by boilerplate; | ||
| Line 739: | Line 872: | ||
| </ | </ | ||
| - | ===== G. scripts/ | + | ===== G. scripts/ |
| Every row a machine flagged and a human then re-checked, with the action taken and why: '' | Every row a machine flagged and a human then re-checked, with the action taken and why: '' | ||
| Line 757: | Line 890: | ||
| quotation marks" failure, and it happened in 4 of 175 rows despite an | quotation marks" failure, and it happened in 4 of 175 rows despite an | ||
| instruction that said verbatim -- which is the argument for verifying every | instruction that said verbatim -- which is the argument for verifying every | ||
| - | citation by machine rather than trusting the adjudicator. | + | citation by machine rather than trusting the adjudicator. A fifth requote, |
| + | ``compass-fit.jp``, | ||
| + | dropped character. It is counted with them because the remedy is the same, | ||
| + | not because it is the same defect. It was | ||
| + | found on 2026-09-11 and applied on 2026-09-17, by re-running a *fixed* | ||
| + | checker over rows the broken one had passed: ``norm()`` used to delete every non-Latin character, | ||
| + | so ``compass-fit.jp``' | ||
| + | letters ``COMPASS-FIT`` it happens to contain. A checker that cannot see a | ||
| + | script cannot police it, and every row it passed in that script is | ||
| + | unverified until it is re-run. | ||
| * ``resource`` -- the cited URL was the wrong page (a 404 template, a | * ``resource`` -- the cited URL was the wrong page (a 404 template, a | ||
| JS-only route, a blocked host). A different primary source for the same | JS-only route, a blocked host). A different primary source for the same | ||
| Line 827: | Line 969: | ||
| " | " | ||
| "error needs a source naming the real owner and there is none." | "error needs a source naming the real owner and there is none." | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | "found 2026-09-11, applied 2026-09-17. The norm() character-class fix " | ||
| + | "of 2026-09-11 was re-run that day over all 175 rows of the first " | ||
| + | " | ||
| + | "Latin letters COMPASS-FIT its Japanese quote happens to contain. The " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | "the operating subsidiary together, which is what owner_today = " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| } | } | ||
| Line 879: | Line 1042: | ||
| </ | </ | ||
| - | ===== H. scripts/ | + | ===== H. scripts/ |
| The methodological correction made during the run, kept rather than smoothed away. Each of the 26 re-scores carries the entity string, the adjudicated owner and the reason. | The methodological correction made during the run, kept rather than smoothed away. Each of the 26 re-scores carries the entity string, the adjudicated owner and the reason. | ||
| Line 2110: | Line 2273: | ||
| ===== R. Unedited output: scripts/ | ===== R. Unedited output: scripts/ | ||
| - | Run as '' | + | Run as '' |
| < | < | ||
| - | 10 corrections applied to 175 rows, after machine verification of every cited source: demote=1, requote=4, resource=5 | + | 11 corrections applied to 175 rows, after machine verification of every cited source: demote=1, requote=5, resource=5 |
| twiago.com | twiago.com | ||
| Line 2164: | Line 2327: | ||
| was quote : Condé Nast Entertainment | was quote : Condé Nast Entertainment | ||
| now source: None | now source: None | ||
| + | |||
| + | compass-fit.jp | ||
| + | was source: https:// | ||
| + | was quote : 台湾国内でのネイティブ型広告配信サービス「COMPASS-FIT」の提供を開始致しました。 | ||
| + | now source: https:// | ||
| wrote out/ | wrote out/ | ||
| Line 2170: | Line 2338: | ||
| ===== S. Unedited output: scripts/ | ===== S. Unedited output: scripts/ | ||
| - | Run as '' | + | Run as '' |
| < | < | ||
| Line 2271: | Line 2439: | ||
| ===== V. Unedited output: scripts/ | ===== V. Unedited output: scripts/ | ||
| - | Run as '' | + | Run as '' |
| < | < | ||
| Line 4271: | Line 4439: | ||
| TITLE: Live chat software | TITLE: Live chat software | ||
| COPYRIGHT: © 2026 Zendesk, Inc| | COPYRIGHT: © 2026 Zendesk, Inc| | ||
| + | </ | ||
| + | |||
| + | ===== AA. scripts/ | ||
| + | |||
| + | Draws the 20% subsample re-adjudicated on 2026-09-11 under its own seed (20260911, deliberately not the parent draw's 20260905) and rebuilds the sub-agent batch files for it by **copying the first rater' | ||
| + | |||
| + | <file python owner_irr_sample.py> | ||
| + | # | ||
| + | """ | ||
| + | rebuild the sub-agent batch files for it **byte-identically** to the ones the | ||
| + | first rater saw. | ||
| + | |||
| + | Why this exists | ||
| + | --------------- | ||
| + | The 175 domains of ``owner_sample.py`` were adjudicated once, by one Sonnet | ||
| + | sub-agent per batch, against ``out/ | ||
| + | provenance page recorded that as the study' | ||
| + | named the fix: " | ||
| + | agreement" | ||
| + | |||
| + | Design, and why each choice was made: | ||
| + | |||
| + | * **Simple random sample of the 175 distinct domains, not a re-stratification.** | ||
| + | The subsample is not estimating anything about the web; it is estimating | ||
| + | agreement between two readings of the same brief. | ||
| + | prevalence would make the subsample non-representative *of the sample*, which | ||
| + | is the population that matters here. 35 = round(0.2 * 175). | ||
| + | |||
| + | * **A separate seed, recorded.** | ||
| + | would make the subsample a deterministic function of the draw it is auditing | ||
| + | -- harmless in fact, but indefensible in principle. | ||
| + | this sitting' | ||
| + | |||
| + | * **Per-domain blocks copied verbatim out of ``out/ | ||
| + | rater saw the three lists' claims plus a mechanical probe (TLS / RDAP / HTTP) | ||
| + | captured on 2026-09-05. | ||
| + | second rater a *different* stimulus, and any disagreement would then be partly | ||
| + | the world moving rather than the raters differing. | ||
| + | reused exactly, and the six-day gap between the frozen probe and the second | ||
| + | rater' | ||
| + | |||
| + | python3 owner_irr_sample.py --sample out/ | ||
| + | --batches out/adj --out out/irr --n 35 | ||
| + | """ | ||
| + | |||
| + | import argparse | ||
| + | import hashlib | ||
| + | import json | ||
| + | import os | ||
| + | import random | ||
| + | import re | ||
| + | import sys | ||
| + | |||
| + | SEED = 20260911 | ||
| + | SEP = " | ||
| + | |||
| + | |||
| + | def read_blocks(batchdir): | ||
| + | """ | ||
| + | blocks = {} | ||
| + | for fn in sorted(os.listdir(batchdir)): | ||
| + | if not re.fullmatch(r" | ||
| + | continue | ||
| + | text = open(os.path.join(batchdir, | ||
| + | for chunk in text.split(SEP): | ||
| + | chunk = chunk.strip(" | ||
| + | if not chunk.strip(): | ||
| + | continue | ||
| + | m = re.search(r" | ||
| + | if m is None: | ||
| + | raise SystemExit(f" | ||
| + | d = m.group(1) | ||
| + | # forter.com legitimately appears twice: batch04' | ||
| + | # of its 9 rows, and the dropped domain was re-issued as the | ||
| + | # one-domain batch20. The two blocks are byte-identical, | ||
| + | # duplicate is harmless -- but a duplicate that DIFFERED would mean | ||
| + | # the two raters saw different stimuli, so it is checked. | ||
| + | if d in blocks and blocks[d] != chunk: | ||
| + | raise SystemExit(f" | ||
| + | blocks[d] = chunk | ||
| + | return blocks | ||
| + | |||
| + | |||
| + | def main(): | ||
| + | ap = argparse.ArgumentParser() | ||
| + | ap.add_argument(" | ||
| + | ap.add_argument(" | ||
| + | ap.add_argument(" | ||
| + | ap.add_argument(" | ||
| + | ap.add_argument(" | ||
| + | args = ap.parse_args() | ||
| + | |||
| + | S = json.load(open(args.sample)) | ||
| + | union = sorted(r[" | ||
| + | claims = {r[" | ||
| + | blocks = read_blocks(args.batches) | ||
| + | |||
| + | missing = sorted(set(union) - set(blocks)) | ||
| + | extra = sorted(set(blocks) - set(union)) | ||
| + | if missing or extra: | ||
| + | raise SystemExit(f" | ||
| + | | ||
| + | |||
| + | rng = random.Random(SEED) | ||
| + | drawn = sorted(rng.sample(union, | ||
| + | |||
| + | print(f" | ||
| + | print(f" | ||
| + | f" | ||
| + | print() | ||
| + | |||
| + | # How the subsample lands across the three per-list draws and the strata -- | ||
| + | # reported, not controlled for. A simple random sample of the 175 is not | ||
| + | # balanced by construction and pretending otherwise would be the error. | ||
| + | for lab in (" | ||
| + | insample = [d for d in drawn if lab in claims[d][" | ||
| + | named = [d for d in drawn if claims[d][lab] is not None] | ||
| + | print(f" | ||
| + | f" | ||
| + | strat = {} | ||
| + | for d in drawn: | ||
| + | for lab in S[" | ||
| + | for r in S[" | ||
| + | if r[" | ||
| + | strat[r[" | ||
| + | break | ||
| + | print(f" | ||
| + | + ", " | ||
| + | print() | ||
| + | |||
| + | os.makedirs(args.out, | ||
| + | adjdir = os.path.join(args.out, | ||
| + | os.makedirs(adjdir, | ||
| + | nb = 0 | ||
| + | for i in range(0, len(drawn), args.per_batch): | ||
| + | part = drawn[i:i + args.per_batch] | ||
| + | body = " | ||
| + | path = os.path.join(adjdir, | ||
| + | open(path, " | ||
| + | print(f" | ||
| + | nb += 1 | ||
| + | print() | ||
| + | |||
| + | out = { | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | } | ||
| + | # A sample-shaped file restricted to the subsample, so that | ||
| + | # `owner_merge_rows.py` -- the same gate the first rater' | ||
| + | # through, unmodified -- can check the second rater' | ||
| + | # list files without believing 140 domains are missing. | ||
| + | sub = {k: v for k, v in S.items() if k != " | ||
| + | sub[" | ||
| + | sub[" | ||
| + | " | ||
| + | if r[" | ||
| + | for lab in S[" | ||
| + | mergefile = os.path.join(args.out, | ||
| + | json.dump(sub, | ||
| + | print(f" | ||
| + | |||
| + | manifest = os.path.join(args.out, | ||
| + | json.dump(out, | ||
| + | digest = hashlib.sha256(open(manifest, | ||
| + | print(f" | ||
| + | # The subsample must be reproducible from the seed alone; print the list so | ||
| + | # the page can carry it without the reader needing the file. | ||
| + | print(" | ||
| + | return 0 | ||
| + | |||
| + | |||
| + | if __name__ == " | ||
| + | sys.exit(main()) | ||
| + | </ | ||
| + | |||
| + | ===== AB. scripts/ | ||
| + | |||
| + | Cohen kappa per list over five populations, | ||
| + | |||
| + | <file python owner_irr_kappa.py> | ||
| + | # | ||
| + | """ | ||
| + | |||
| + | python3 owner_irr_kappa.py --r1 out/ | ||
| + | --r1-scored out/ | ||
| + | --subsample out/ | ||
| + | |||
| + | Rater 1 21 Sonnet sub-agents, 2026-09-05, one batch each (`out/ | ||
| + | Rater 2 5 Fable sub-agents, 2026-09-11, the same brief byte-for-byte and the | ||
| + | same per-domain blocks, on a 35-domain random subsample (`out/ | ||
| + | |||
| + | What is and is not measured | ||
| + | --------------------------- | ||
| + | Both rater' | ||
| + | to `absent` when the list file has no entry for the domain. | ||
| + | about the file, not a judgement, so agreement on those cells is a property of | ||
| + | the merge script and would inflate every figure: 96 of webXray' | ||
| + | verdicts are forced `absent`. | ||
| + | **second** is the headline: | ||
| + | |||
| + | ALL all 35 rows, raw vocabulary. | ||
| + | | ||
| + | NAMED rows where the list file names an owner. | ||
| + | from the list file by `owner_merge_rows.py` rather than judged, so | ||
| + | the two raters' | ||
| + | | ||
| + | the equivalence rather than assuming it. If the list files move | ||
| + | | ||
| + | move between 2026-08-17 and 2026-09-05 -- the sets could diverge, | ||
| + | and an `or` selection would then silently pair a real verdict | ||
| + | | ||
| + | ADJUDICABLE | ||
| + | | ||
| + | not source is a difference in what was found, not in how it was | ||
| + | read; that axis is measured separately as ADJUDICABILITY. | ||
| + | |||
| + | and two collapses of ADJUDICABLE, | ||
| + | share and nothing else: | ||
| + | |||
| + | CURRENT-vs-REST | ||
| + | ADJUDICABILITY | ||
| + | |||
| + | `self-named` was retired after the first sitting (`owner_verdict_fixes.py`), | ||
| + | it is in the brief both raters were given, so it is kept as its own category in | ||
| + | the raw comparison and its per-rater usage is reported: how often two raters | ||
| + | independently reach for a verdict is evidence about the verdict. | ||
| + | |||
| + | Kappa | ||
| + | ----- | ||
| + | Cohen' | ||
| + | |||
| + | kappa = (po - pe) / (1 - pe), pe = sum_c p1(c) * p2(c) | ||
| + | |||
| + | Intervals are bootstrap percentiles over **domains** for the per-list rows and | ||
| + | over **cells** for the pooled rows -- a pooled row has already flattened the | ||
| + | three lists' verdicts into one sequence, so its resample can draw two cells of | ||
| + | one domain independently and its interval is narrower than it should be. That is | ||
| + | stated on the page as well. 10,000 resamples, percentile rather than the | ||
| + | normal approximation: | ||
| + | variance is not usable. | ||
| + | row one category, so 1 - pe = 0) is dropped and the count of drops is printed -- | ||
| + | silently dropping them would narrow the interval. | ||
| + | |||
| + | Kappa is not read against the usual verbal bands here. They are a convention, | ||
| + | not a threshold, and at these marginals -- one category carrying most of the | ||
| + | mass -- kappa is driven as much by the marginals as by the disagreements. | ||
| + | observed agreement is printed beside every kappa for that reason. | ||
| + | """ | ||
| + | |||
| + | import argparse, json, random, sys | ||
| + | from collections import Counter | ||
| + | |||
| + | LISTS = [(" | ||
| + | | ||
| + | OWNERSHIP = {" | ||
| + | NBOOT = 10000 | ||
| + | SEED = 20260911 | ||
| + | |||
| + | |||
| + | def kappa(pairs): | ||
| + | """ | ||
| + | n = len(pairs) | ||
| + | if n == 0: | ||
| + | return None | ||
| + | po = sum(1 for a, b in pairs if a == b) / n | ||
| + | m1, m2 = Counter(a for a, _ in pairs), Counter(b for _, b in pairs) | ||
| + | # sorted(), not set(): Python' | ||
| + | # so the float addition order varied between runs and the JSON artefact' | ||
| + | # 15th-16th decimal moved. No published 3-decimal figure ever changed, but a | ||
| + | # " | ||
| + | pe = sum(m1[c] / n * m2[c] / n for c in sorted(set(m1) | set(m2))) | ||
| + | if abs(1 - pe) < 1e-12: | ||
| + | return {" | ||
| + | return {" | ||
| + | |||
| + | |||
| + | def boot(pairs, seed): | ||
| + | rng = random.Random(seed) | ||
| + | n = len(pairs) | ||
| + | ks, degenerate = [], 0 | ||
| + | if n < 2: | ||
| + | return None, None, 0 | ||
| + | for _ in range(NBOOT): | ||
| + | r = [pairs[rng.randrange(n)] for _ in range(n)] | ||
| + | k = kappa(r) | ||
| + | if k is None or k[" | ||
| + | degenerate += 1 | ||
| + | continue | ||
| + | ks.append(k[" | ||
| + | if len(ks) < NBOOT // 10: | ||
| + | return None, None, degenerate | ||
| + | ks.sort() | ||
| + | return ks[int(0.025 * len(ks))], ks[int(0.975 * len(ks))], degenerate | ||
| + | |||
| + | |||
| + | def fmt(k, lo, hi, deg): | ||
| + | if k is None: | ||
| + | return " | ||
| + | if k[" | ||
| + | return f" | ||
| + | s = f" | ||
| + | if lo is not None: | ||
| + | s += f" | ||
| + | else: | ||
| + | s += " | ||
| + | if deg: | ||
| + | s += f" | ||
| + | return s | ||
| + | |||
| + | |||
| + | def assert_absent_agrees(doms, | ||
| + | """ | ||
| + | exactly the same cells. Every population below is selected with `or`, which | ||
| + | is only equivalent to the docstring' | ||
| + | while that holds. Checked, not assumed.""" | ||
| + | bad = [(d, lab, R1[d][key], R2[d][key]) | ||
| + | for lab, key in LISTS for d in doms | ||
| + | if (R1[d][key] == " | ||
| + | if bad: | ||
| + | lines = " | ||
| + | raise SystemExit( | ||
| + | f" | ||
| + | "from the list file, not judged, so this means the two raters were " | ||
| + | " | ||
| + | "would be wrong.\n" | ||
| + | print(f" | ||
| + | "list file, not a judgement) -- the NAMED/ | ||
| + | |||
| + | |||
| + | def main(): | ||
| + | ap = argparse.ArgumentParser() | ||
| + | ap.add_argument(" | ||
| + | ap.add_argument(" | ||
| + | ap.add_argument(" | ||
| + | ap.add_argument(" | ||
| + | ap.add_argument(" | ||
| + | args = ap.parse_args() | ||
| + | |||
| + | sub = json.load(open(args.subsample)) | ||
| + | doms = sorted(sub[" | ||
| + | R1 = {r[" | ||
| + | R1S = {r[" | ||
| + | R2 = {r[" | ||
| + | |||
| + | missing = [d for d in doms if d not in R1 or d not in R2] | ||
| + | if missing: | ||
| + | raise SystemExit(f" | ||
| + | assert_absent_agrees(doms, | ||
| + | |||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | print(f" | ||
| + | f" | ||
| + | print(f" | ||
| + | print(f" | ||
| + | print(f" | ||
| + | print() | ||
| + | |||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | for lab, key in LISTS: | ||
| + | c1 = Counter(R1[d][key] for d in doms) | ||
| + | c2 = Counter(R2[d][key] for d in doms) | ||
| + | cats = sorted(set(c1) | set(c2)) | ||
| + | print(f" | ||
| + | for c in cats: | ||
| + | print(f" | ||
| + | both = Counter() | ||
| + | for lab, key in LISTS: | ||
| + | for d in doms: | ||
| + | if R1[d][key] == " | ||
| + | both[" | ||
| + | if R2[d][key] == " | ||
| + | both[" | ||
| + | if R1[d][key] == " | ||
| + | both[" | ||
| + | print(f" | ||
| + | f" | ||
| + | print() | ||
| + | |||
| + | results = {} | ||
| + | for title, pick in ( | ||
| + | ("1. ALL rows, raw vocabulary | ||
| + | | ||
| + | ): | ||
| + | print(" | ||
| + | for lab, key in LISTS: | ||
| + | pairs = [pick(d, key) for d in doms] | ||
| + | k = kappa(pairs); | ||
| + | print(f" | ||
| + | results[(" | ||
| + | print() | ||
| + | |||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | for lab, key in LISTS: | ||
| + | sel = [d for d in doms if R1[d][key] != " | ||
| + | pairs = [(R1[d][key], | ||
| + | k = kappa(pairs); | ||
| + | print(f" | ||
| + | results[(" | ||
| + | allp = [(R1[d][key], | ||
| + | if R1[d][key] != " | ||
| + | k = kappa(allp); | ||
| + | print(f" | ||
| + | f"the three lists' verdicts on one domain are not independent)" | ||
| + | results[(" | ||
| + | print() | ||
| + | |||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | sourced, read_, tot = 0, 0, 0 | ||
| + | for lab, key in LISTS: | ||
| + | a = b = n = 0 | ||
| + | for d in doms: | ||
| + | v1, v2 = R1[d][key], R2[d][key] | ||
| + | if v1 == " | ||
| + | continue | ||
| + | n += 1 | ||
| + | if v1 == v2: | ||
| + | continue | ||
| + | if " | ||
| + | a += 1 | ||
| + | else: | ||
| + | b += 1 | ||
| + | print(f" | ||
| + | f" | ||
| + | sourced += a; read_ += b; tot += n | ||
| + | dis = sourced + read_ | ||
| + | print(f" | ||
| + | f" | ||
| + | if dis: | ||
| + | print(f" | ||
| + | f" | ||
| + | results[(" | ||
| + | print() | ||
| + | |||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | pooled = [] | ||
| + | for lab, key in LISTS: | ||
| + | sel = [d for d in doms if R1[d][key] in OWNERSHIP and R2[d][key] in OWNERSHIP] | ||
| + | pairs = [(R1[d][key], | ||
| + | pooled += pairs | ||
| + | k = kappa(pairs); | ||
| + | print(f" | ||
| + | results[(" | ||
| + | k = kappa(pooled); | ||
| + | print(f" | ||
| + | results[(" | ||
| + | print() | ||
| + | |||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | pooled = [] | ||
| + | for lab, key in LISTS: | ||
| + | sel = [d for d in doms if R1[d][key] in OWNERSHIP and R2[d][key] in OWNERSHIP] | ||
| + | pairs = [(R1[d][key] == " | ||
| + | pooled += pairs | ||
| + | k = kappa(pairs); | ||
| + | print(f" | ||
| + | results[(" | ||
| + | k = kappa(pooled); | ||
| + | print(f" | ||
| + | results[(" | ||
| + | print() | ||
| + | |||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | for lab, key in LISTS: | ||
| + | sel = [d for d in doms if R1[d][key] != " | ||
| + | pairs = [(R1[d][key] in OWNERSHIP, R2[d][key] in OWNERSHIP) for d in sel] | ||
| + | k = kappa(pairs); | ||
| + | n1 = sum(1 for a, _ in pairs if a); n2 = sum(1 for _, b in pairs if b) | ||
| + | print(f" | ||
| + | results[(" | ||
| + | r1u = sum(1 for d in doms if R1[d][" | ||
| + | r2u = sum(1 for d in doms if R2[d][" | ||
| + | agree = sum(1 for d in doms | ||
| + | if (R1[d][" | ||
| + | print(f" | ||
| + | f" | ||
| + | print() | ||
| + | |||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | for lab, key in LISTS: | ||
| + | s1 = [d for d in doms if R1[d][key] in OWNERSHIP] | ||
| + | s2 = [d for d in doms if R2[d][key] in OWNERSHIP] | ||
| + | c1 = sum(1 for d in s1 if R1[d][key] == " | ||
| + | c2 = sum(1 for d in s2 if R2[d][key] == " | ||
| + | f1 = f" | ||
| + | f2 = f" | ||
| + | print(f" | ||
| + | print() | ||
| + | |||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | ndis = 0 | ||
| + | for d in doms: | ||
| + | diffs = [(lab, R1[d][key], R2[d][key]) for lab, key in LISTS | ||
| + | if (R1[d][key] != " | ||
| + | and R1[d][key] != R2[d][key]] | ||
| + | if not diffs: | ||
| + | continue | ||
| + | ndis += 1 | ||
| + | print(f" | ||
| + | print(f" | ||
| + | print(f" | ||
| + | for lab, a, b in diffs: | ||
| + | print(f" | ||
| + | print(f" | ||
| + | print() | ||
| + | |||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | print(" | ||
| + | for lab, key in LISTS: | ||
| + | sel = [d for d in doms if R1S[d][key] != " | ||
| + | pairs = [(R1S[d][key], | ||
| + | k = kappa(pairs); | ||
| + | print(f" | ||
| + | results[(" | ||
| + | changed = sum(1 for d in doms for _, key in LISTS if R1[d][key] != R1S[d][key]) | ||
| + | print(f" | ||
| + | print() | ||
| + | |||
| + | if args.json_out: | ||
| + | out = {f" | ||
| + | {" | ||
| + | " | ||
| + | for (scope, lab), (k, lo, hi) in results.items()} | ||
| + | json.dump(out, | ||
| + | print(f" | ||
| + | return 0 | ||
| + | |||
| + | |||
| + | if __name__ == " | ||
| + | sys.exit(main()) | ||
| + | </ | ||
| + | |||
| + | ===== AC. scripts/ | ||
| + | |||
| + | The three rows that failed '' | ||
| + | |||
| + | <file python owner_irr_corrections.py> | ||
| + | # | ||
| + | """ | ||
| + | rater' | ||
| + | NOTFOUND in the bytes **and** NOTFOUND in a real browser is a failed citation, | ||
| + | and its row is demoted to ``unresolved`` rather than published. | ||
| + | |||
| + | python3 owner_irr_corrections.py --rows out/ | ||
| + | --out out/ | ||
| + | |||
| + | Three rows failed ``owner_verify_sources.py`` on 2026-09-11 and each is recorded | ||
| + | here with what a hand check found, because two of the three are **not** bad | ||
| + | citations and demoting them would have been the wrong call: | ||
| + | |||
| + | adgrx.com | ||
| + | | ||
| + | been renewed, so ``curl -sL`` and Chromium both refuse the | ||
| + | | ||
| + | | ||
| + | limit of the verifier: an expired certificate makes a correct | ||
| + | | ||
| + | cedscdn.it | ||
| + | | ||
| + | pass nor the browser pass can open -- the brief offers exactly one | ||
| + | | ||
| + | A port-43 query to ``whois.nic.it`` returns | ||
| + | | ||
| + | | ||
| + | the gap is the brief' | ||
| + | kameleoon.io NOTFOUND in bytes and NOTFOUND in the browser -- and the citation | ||
| + | is sound. **This row was demoted and the demotion was wrong**, and | ||
| + | it is recorded here rather than quietly reversed. The quote is a | ||
| + | | ||
| + | to markup: ``owner_verify_sources.norm()`` strips ``< | ||
| + | | ||
| + | | ||
| + | | ||
| + | | ||
| + | the page instead of trusting the checker. The verifier now has an | ||
| + | | ||
| + | |||
| + | **No row is demoted.** All three machine failures turned out to be limits of the | ||
| + | instruments rather than bad citations, which is itself the finding: two of the | ||
| + | three would have been acted on without a hand check, and one of them was. The | ||
| + | inter-rater figures are therefore identical before and after this script, and | ||
| + | that equality is asserted below rather than assumed. | ||
| + | """ | ||
| + | import argparse, json, sys | ||
| + | |||
| + | # domain -> (action, note) | ||
| + | CORRECTIONS = { | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | "A port-43 query to whois.nic.it returns the quoted line verbatim. Row " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | "a `<script src>` line, and both the bytes pass (which strips tags) and " | ||
| + | "the browser pass (which reads innerText) are blind to markup by " | ||
| + | " | ||
| + | "the verifier now reports OK-MARKUP instead." | ||
| + | } | ||
| + | |||
| + | KEYS = (" | ||
| + | |||
| + | |||
| + | def main(): | ||
| + | ap = argparse.ArgumentParser() | ||
| + | ap.add_argument(" | ||
| + | ap.add_argument(" | ||
| + | args = ap.parse_args() | ||
| + | rows = json.load(open(args.rows)) | ||
| + | by = {r[" | ||
| + | |||
| + | unlisted = sorted(set(CORRECTIONS) - set(by)) | ||
| + | if unlisted: | ||
| + | raise SystemExit(f" | ||
| + | |||
| + | changed = 0 | ||
| + | for dom, (action, note) in CORRECTIONS.items(): | ||
| + | r = by[dom] | ||
| + | print(f" | ||
| + | print(f" | ||
| + | if action == " | ||
| + | continue | ||
| + | before = {k: r[k] for k in KEYS} | ||
| + | r[" | ||
| + | r[" | ||
| + | r[" | ||
| + | for k in KEYS: | ||
| + | # `absent` is a fact about the list file and survives a demotion; | ||
| + | # every other verdict becomes unknown, exactly as the brief says an | ||
| + | # unresolved row must. | ||
| + | if r[k] != " | ||
| + | r[k] = " | ||
| + | after = {k: r[k] for k in KEYS} | ||
| + | for k in KEYS: | ||
| + | if before[k] != after[k]: | ||
| + | print(f" | ||
| + | changed += 1 | ||
| + | r[" | ||
| + | print() | ||
| + | print(f" | ||
| + | # The page states that the corrected and uncorrected inter-rater figures are | ||
| + | # the same. That is only true while nothing is demoted, so it is checked | ||
| + | # here rather than left as a sentence somebody has to remember. | ||
| + | if changed: | ||
| + | print(" | ||
| + | "from the uncorrected ones and both must be published." | ||
| + | json.dump(rows, | ||
| + | print(f" | ||
| + | return 0 | ||
| + | |||
| + | |||
| + | if __name__ == " | ||
| + | sys.exit(main()) | ||
| + | </ | ||
| + | |||
| + | ===== AD. scripts/ | ||
| + | |||
| + | Asserts that every inter-rater figure on each page matches the estimator' | ||
| + | |||
| + | <file python check_irr_figures.py> | ||
| + | # | ||
| + | """ | ||
| + | ``owner_irr_kappa.py``' | ||
| + | |||
| + | python3 check_irr_figures.py --page prov=pages/ | ||
| + | --page content=pages/ | ||
| + | --json-corrected out/ | ||
| + | --output out/ | ||
| + | |||
| + | **Each page is checked against its own claims, not against the union.** The first | ||
| + | version joined both pages and asked whether a figure was on *either*; its own | ||
| + | mutation harness then showed two corruptions surviving, because the same kappa | ||
| + | appears on both pages and breaking one copy left the other to satisfy the check. | ||
| + | A guard over a union of documents cannot tell you which document is wrong. | ||
| + | |||
| + | Same contract, and same limit, as ``check_owner_sample_figures.py``: | ||
| + | the page to the tool, so a bug **inside** ``owner_irr_kappa.py`` passes it, and | ||
| + | so does a correct figure attached to the wrong sentence. It catches one thing -- | ||
| + | a page figure drifting from the script that produced it. | ||
| + | |||
| + | It fails when a figure is **missing** as well as when it disagrees, because a | ||
| + | guard that passes because its needle was deleted asserts nothing. ``REQUIRED`` | ||
| + | names, per figure, which of its fields the pages actually publish: a guard that | ||
| + | demanded every field of every scope would have to be weakened the first time a | ||
| + | page chose not to quote one, and a weakened guard is how the published-figure | ||
| + | defects on these pages got through before. | ||
| + | |||
| + | The last check is the one that catches the opposite error: a three-decimal value | ||
| + | sitting in kappa context on the page that neither JSON produces. That is how a | ||
| + | hand-typed figure gets in. | ||
| + | """ | ||
| + | import argparse, json, re, sys | ||
| + | |||
| + | # role -> {scope|list -> fields THAT page publishes}: k=kappa, p=raw agreement, | ||
| + | # n=denominator (" | ||
| + | PROV_REQUIRED = { | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | } | ||
| + | |||
| + | # The content page quotes a subset, and writes its denominators in prose | ||
| + | # ("on 12 rows") rather than as " | ||
| + | CONTENT_REQUIRED = { | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | " | ||
| + | } | ||
| + | |||
| + | REQUIRED = {" | ||
| + | |||
| + | # Figures taken from the post-demotion recomputation. Only the provenance page | ||
| + | # carries these. | ||
| + | CORRECTED = {" | ||
| + | " | ||
| + | |||
| + | SIGNED = lambda x: f" | ||
| + | BARE = lambda x: f" | ||
| + | |||
| + | |||
| + | def check(K, want, joined, tag, fails, counter): | ||
| + | for key, fields in want.items(): | ||
| + | if key not in K: | ||
| + | sys.exit(f" | ||
| + | " | ||
| + | v = K[key] | ||
| + | if v is None: | ||
| + | fails.append(f" | ||
| + | " | ||
| + | continue | ||
| + | probes = [] | ||
| + | if " | ||
| + | probes.append((" | ||
| + | if " | ||
| + | probes.append((" | ||
| + | if " | ||
| + | probes.append((" | ||
| + | for label, val in probes: | ||
| + | counter[0] += 1 | ||
| + | if val not in joined: | ||
| + | fails.append(f" | ||
| + | if " | ||
| + | counter[0] += 1 | ||
| + | lo, hi = SIGNED(v[" | ||
| + | pat = re.escape(lo) + r" | ||
| + | if not re.search(pat, | ||
| + | fails.append(f" | ||
| + | |||
| + | |||
| + | def main(): | ||
| + | ap = argparse.ArgumentParser() | ||
| + | ap.add_argument(" | ||
| + | ap.add_argument(" | ||
| + | ap.add_argument(" | ||
| + | ap.add_argument(" | ||
| + | ap.add_argument(" | ||
| + | help=" | ||
| + | " | ||
| + | " | ||
| + | args = ap.parse_args() | ||
| + | |||
| + | pages = {} | ||
| + | for spec in args.page: | ||
| + | if " | ||
| + | sys.exit(f" | ||
| + | | ||
| + | role, path = spec.split(" | ||
| + | if role not in REQUIRED: | ||
| + | sys.exit(f" | ||
| + | pages[role] = open(path, encoding=" | ||
| + | missing_roles = sorted(set(REQUIRED) - set(pages)) | ||
| + | if missing_roles: | ||
| + | sys.exit(f" | ||
| + | " | ||
| + | |||
| + | K = json.load(open(args.json)) | ||
| + | Kc = json.load(open(args.json_corrected)) | ||
| + | out = open(args.output, | ||
| + | |||
| + | fails, counter = [], [0] | ||
| + | for role, text in sorted(pages.items()): | ||
| + | check(K, REQUIRED[role], | ||
| + | check(Kc, CORRECTED[role], | ||
| + | joined = " | ||
| + | |||
| + | # The committed output must still carry the sections the figures come from, | ||
| + | # or the JSON above is from a different run than the published log. | ||
| + | for needle in ("2. NAMED rows only", "2b. What the NAMED disagreements are made of", | ||
| + | " | ||
| + | counter[0] += 1 | ||
| + | if needle not in out: | ||
| + | fails.append(f" | ||
| + | " | ||
| + | |||
| + | # Figures that live only in the scripts' | ||
| + | # A reviewer mutated each of these on the page and this guard survived every | ||
| + | # one, because it only ever read the JSON. They are parsed out of the two | ||
| + | # committed outputs so they are checked against the tool, not retyped here. | ||
| + | kout = out | ||
| + | vout = open(args.verify_output, | ||
| + | |||
| + | def need(label, value, where): | ||
| + | counter[0] += 1 | ||
| + | if value not in where: | ||
| + | fails.append(f" | ||
| + | |||
| + | # section 0: how many cells each rater scored `self-named` | ||
| + | m = re.search(r" | ||
| + | r"both on the same cell (\d+)", | ||
| + | if not m: | ||
| + | fails.append(" | ||
| + | else: | ||
| + | r1, r2, both = m.groups() | ||
| + | # Bare digits are useless needles -- " | ||
| + | # are deliberately coupled to the sentence' | ||
| + | # sentence breaks the guard instead of silently unchecking the figure. | ||
| + | # Checked wherever the sentence SHAPE occurs, on every page, rather | ||
| + | # than as a needle in the union: both pages carry this sentence, so | ||
| + | # corrupting one copy left the other to satisfy a union check. | ||
| + | counter[0] += 1 | ||
| + | seen = 0 | ||
| + | for role, text in sorted(pages.items()): | ||
| + | for mm in re.finditer(r" | ||
| + | seen += 1 | ||
| + | if mm.group(1) != r2: | ||
| + | fails.append(f" | ||
| + | | ||
| + | if not seen: | ||
| + | fails.append(" | ||
| + | need(" | ||
| + | need(" | ||
| + | |||
| + | # section 2b: the sourcing-vs-reading split, per list and pooled | ||
| + | for line in kout.splitlines(): | ||
| + | m = re.match(r" | ||
| + | | ||
| + | | ||
| + | if m: | ||
| + | lab, n, ag, so, rd = m.groups() | ||
| + | # Whole-row match, not four bare digits: " | ||
| + | # over these pages, so per-cell needles passed a mutated table. | ||
| + | name = " | ||
| + | row = (r" | ||
| + | | ||
| + | | ||
| + | counter[0] += 1 | ||
| + | if not re.search(row, | ||
| + | fails.append(f" | ||
| + | | ||
| + | m = re.search(r" | ||
| + | if not m: | ||
| + | fails.append(" | ||
| + | else: | ||
| + | need(" | ||
| + | |||
| + | # section 5: row-level resolution counts | ||
| + | m = re.search(r" | ||
| + | r" | ||
| + | if not m: | ||
| + | fails.append(" | ||
| + | else: | ||
| + | a, n, b, same = m.groups() | ||
| + | need(" | ||
| + | need(" | ||
| + | need(" | ||
| + | |||
| + | # section 6: the unweighted current-share table, both raters, all three lists | ||
| + | for line in kout.splitlines(): | ||
| + | m = re.match(r" | ||
| + | | ||
| + | if m: | ||
| + | lab, a, b = m.groups() | ||
| + | need(f" | ||
| + | need(f" | ||
| + | |||
| + | # the citation tally, from owner_verify_sources.py' | ||
| + | m = re.search(r" | ||
| + | if not m: | ||
| + | fails.append(" | ||
| + | else: | ||
| + | for part in m.group(1).split(", | ||
| + | k_, v_ = part.split(" | ||
| + | need(f" | ||
| + | |||
| + | # A three-decimal value in kappa context that neither JSON produces is a | ||
| + | # figure nobody can source. | ||
| + | produced = set() | ||
| + | for src in (K, Kc): | ||
| + | for v in src.values(): | ||
| + | if not v: | ||
| + | continue | ||
| + | produced.add(BARE(v[" | ||
| + | if v[" | ||
| + | produced |= {BARE(v[" | ||
| + | for m in re.finditer(r" | ||
| + | for num in re.findall(r" | ||
| + | counter[0] += 1 | ||
| + | if num not in produced: | ||
| + | fails.append(f" | ||
| + | " | ||
| + | |||
| + | for f in fails: | ||
| + | print(" | ||
| + | print(f" | ||
| + | f"over {len(pages)} page(s), each against its own claims; " | ||
| + | f" | ||
| + | return 1 if fails else 0 | ||
| + | |||
| + | |||
| + | if __name__ == " | ||
| + | sys.exit(main()) | ||
| + | </ | ||
| + | |||
| + | ===== AE. scripts/ | ||
| + | |||
| + | Twelve corruptions, | ||
| + | |||
| + | <file bash check_irr_figures_mutations.sh> | ||
| + | #!/bin/bash | ||
| + | # Mutation test for check_irr_figures.py. | ||
| + | # | ||
| + | # A guard that passes tells you nothing until you have seen it fail. This | ||
| + | # corrupts one inter-rater figure at a time -- on either page -- and asserts the | ||
| + | # guard rejects every corruption. If any mutation survives, the guard is not | ||
| + | # checking what its name claims and the exit code says so. | ||
| + | # | ||
| + | # bash scripts/ | ||
| + | set -u | ||
| + | PROV=pages/ | ||
| + | PAGE=pages/ | ||
| + | J=out/ | ||
| + | JC=out/ | ||
| + | OUT=out/ | ||
| + | VOUT=out/ | ||
| + | TMP=$(mktemp -d) | ||
| + | fails=0 | ||
| + | |||
| + | mutate () { # $1 = label, $2 = which page, $3 = sed expression | ||
| + | local src tgt other | ||
| + | if [ " | ||
| + | sed " | ||
| + | if cmp -s " | ||
| + | echo " | ||
| + | fails=$((fails+1)); | ||
| + | fi | ||
| + | # The mutated file keeps its own role, so the guard is asked whether THAT | ||
| + | # page still matches its own claims. Joining the two pages and asking whether | ||
| + | # a figure is on either is what let two of these mutations survive the first | ||
| + | # version of this harness. | ||
| + | if [ " | ||
| + | else roles=(--page " | ||
| + | if python3 scripts/ | ||
| + | | ||
| + | | ||
| + | echo " | ||
| + | fails=$((fails+1)) | ||
| + | else | ||
| + | echo " | ||
| + | fi | ||
| + | } | ||
| + | |||
| + | mutate "prov: pooled NAMED kappa +0.400 -> +0.500" | ||
| + | mutate "prov: webXray NAMED kappa +0.027 -> +0.270" | ||
| + | mutate "prov: Tracker Radar NAMED kappa +0.478 deleted" | ||
| + | mutate "prov: NAMED interval upper +0.551 -> +0.552" | ||
| + | mutate "prov: webXray NAMED n=12 -> n=13" | ||
| + | mutate "prov: pooled raw agreement 58.5% -> 59.5%" | ||
| + | mutate "prov: ADJUDICABLE pooled +0.680 -> +0.690" | ||
| + | mutate "prov: current-vs-rest pooled +0.708 -> +0.718" | ||
| + | # The corrected-variant figures are currently EQUAL to the uncorrected ones, | ||
| + | # because owner_irr_corrections.py demotes nothing. Mutating a value that is no | ||
| + | # longer on the page is a no-op, and this harness reports a no-op mutation as a | ||
| + | # survivor rather than as a pass -- which is how this line was caught after the | ||
| + | # kameleoon.io demotion was reversed. These two target figures that are on the | ||
| + | # page today and are not covered by the kappa mutations above. | ||
| + | mutate "prov: adjudicable-pooled agreement 84.6% -> 85.6%" prov ' | ||
| + | mutate "prov: current-vs-rest agreement 87.2% -> 88.2%" | ||
| + | mutate "prov: invent an unsourced kappa of 0.842" | ||
| + | # Figures that live only in the scripts' | ||
| + | # one of these and the guard survived, because it only read the kappa JSON. | ||
| + | mutate "prov: self-named rater2 6 -> 9 cells" | ||
| + | mutate "prov: split table sourcing 17 -> 20" | ||
| + | mutate "prov: sourcing headline 63.0% -> 70.0%" | ||
| + | mutate "prov: rater-1 unresolved 10 of 35 -> 13 of 35" | ||
| + | mutate "prov: webXray current-share 70.0% (7\/ | ||
| + | mutate "prov: citation tally OK 30 -> OK 32" | ||
| + | mutate "page: pooled kappa +0.400 -> +0.410" | ||
| + | mutate "page: Disconnect NAMED kappa +0.460 -> +0.461" | ||
| + | |||
| + | rm -rf " | ||
| + | echo | ||
| + | if [ " | ||
| + | echo " | ||
| + | exit 1 | ||
| + | fi | ||
| + | echo "all mutations caught." | ||
| + | </ | ||
| + | |||
| + | ===== AF. Unedited output: scripts/ | ||
| + | |||
| + | Run as '' | ||
| + | |||
| + | < | ||
| + | Universe: 175 distinct domains adjudicated on 2026-09-05. | ||
| + | Seed 20260911 (separate from the 20260905 draw); simple random sample, n=35 (20.0%). | ||
| + | |||
| + | webXray | ||
| + | Tracker Radar : 12 of the subsample are in this list's own 60-domain draw; 30 of 35 have an entry in the file | ||
| + | Disconnect | ||
| + | strata of origin (a domain drawn for two lists counts twice): Q1=11, Q2=6, Q3=14, Q4=5 | ||
| + | |||
| + | batch00.txt: | ||
| + | batch01.txt: | ||
| + | batch02.txt: | ||
| + | batch03.txt: | ||
| + | batch04.txt: | ||
| + | |||
| + | Wrote out/ | ||
| + | Wrote out/ | ||
| + | Subsample: accessibe.com adgrx.com amperwave.net awltovhc.com azure.com britishairways.com cedexis.com cedscdn.it cnn.com company-target.com eyeota.net forter.com glomex.com histats.com hitc.com iqzone.com juicer.cc kameleoon.io lijit.com makaira.io mapixl.com mediaset.es mountain.com myaccountaccess.com owneriq.net pages02.net qbox.me sa-as.com sharethis.com sparkasse.de stripst.com tqlkg.com tvtime.com wishabi.com wrbm.com | ||
| + | </ | ||
| + | |||
| + | ===== AG. Unedited output: scripts/ | ||
| + | |||
| + | Run as '' | ||
| + | |||
| + | < | ||
| + | batches merged: 5, rows: 35 | ||
| + | sampled domains: 35; missing rows: 0; rows for unsampled domains: 0 | ||
| + | absent-verdicts corrected against the list files: 0 | ||
| + | |||
| + | wrote out/ | ||
| + | </ | ||
| + | |||
| + | ===== AH. Unedited output: scripts/ | ||
| + | |||
| + | Run as '' | ||
| + | |||
| + | < | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK-MARKUP | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | OK | ||
| + | |||
| + | Verification of cited primary sources: FETCHFAIL=2, | ||
| + | </ | ||
| + | |||
| + | ===== AI. Unedited output: scripts/ | ||
| + | |||
| + | Run as '' | ||
| + | |||
| + | < | ||
| + | KEEP adgrx.com | ||
| + | FETCHFAIL is an expired certificate on samsungads.ca (Let's Encrypt, notAfter 2026-03-12), | ||
| + | KEEP cedscdn.it | ||
| + | FETCHFAIL is an unsupported URL scheme: `whois:// | ||
| + | KEEP kameleoon.io | ||
| + | NOTFOUND in both instruments was a FALSE disqualification: | ||
| + | |||
| + | rows examined: 3; verdict cells changed: 0 | ||
| + | wrote out/ | ||
| + | </ | ||
| + | |||
| + | ===== AJ. Unedited output: scripts/ | ||
| + | |||
| + | Run as '' | ||
| + | |||
| + | < | ||
| + | `absent` agrees on all 105 cells (it is a fact about the list file, not a judgement) -- the NAMED/ | ||
| + | ============================================================================== | ||
| + | INTER-RATER AGREEMENT, OWNERSHIP ADJUDICATION | ||
| + | ============================================================================== | ||
| + | Subsample: 35 of 175 domains, seed 20260911 (parent draw seed 20260905). | ||
| + | Rater 1: 2026-09-05, Sonnet sub-agents (out/ | ||
| + | Rater 2: 2026-09-11, Fable sub-agents (out/ | ||
| + | Bootstrap: 10,000 resamples over domains, seed 20260911. | ||
| + | |||
| + | ------------------------------------------------------------------------------ | ||
| + | 0. Vocabulary each rater used on these 35 domains (105 cells per rater) | ||
| + | ------------------------------------------------------------------------------ | ||
| + | webXray | ||
| + | absent | ||
| + | current | ||
| + | error rater1= 1 rater2= 1 | ||
| + | granularity | ||
| + | self-named | ||
| + | stale rater1= 2 rater2= 2 | ||
| + | unknown | ||
| + | Tracker Radar | ||
| + | absent | ||
| + | current | ||
| + | error rater1= 0 rater2= 1 | ||
| + | granularity | ||
| + | stale rater1= 7 rater2= 9 | ||
| + | unknown | ||
| + | Disconnect | ||
| + | absent | ||
| + | current | ||
| + | granularity | ||
| + | self-named | ||
| + | unknown | ||
| + | `self-named` (retired after sitting 1, still in the brief): rater1 4 cells, rater2 6 cells, both on the same cell 3 | ||
| + | |||
| + | ------------------------------------------------------------------------------ | ||
| + | 1. ALL rows, raw vocabulary | ||
| + | ------------------------------------------------------------------------------ | ||
| + | webXray | ||
| + | Tracker Radar | ||
| + | Disconnect | ||
| + | |||
| + | ------------------------------------------------------------------------------ | ||
| + | 2. NAMED rows only (the list file names an owner: the rows the brief | ||
| + | | ||
| + | ------------------------------------------------------------------------------ | ||
| + | webXray | ||
| + | Tracker Radar | ||
| + | Disconnect | ||
| + | POOLED | ||
| + | |||
| + | ------------------------------------------------------------------------------ | ||
| + | 2b. What the NAMED disagreements are made of | ||
| + | An `unknown` against a verdict is a difference in what was SOURCED. | ||
| + | A verdict against a different verdict is a difference in what was READ. | ||
| + | ------------------------------------------------------------------------------ | ||
| + | webXray | ||
| + | Tracker Radar n=30 agreed=21 | ||
| + | Disconnect | ||
| + | POOLED | ||
| + | -> 63.0% of the 27 disagreeing cells are one rater reaching `unknown` where the other reached a verdict. | ||
| + | |||
| + | ------------------------------------------------------------------------------ | ||
| + | 3. ADJUDICABLE rows (both raters reached an ownership verdict) | ||
| + | ------------------------------------------------------------------------------ | ||
| + | webXray | ||
| + | Tracker Radar | ||
| + | Disconnect | ||
| + | POOLED | ||
| + | |||
| + | ------------------------------------------------------------------------------ | ||
| + | 4. CURRENT vs REST, on ADJUDICABLE rows (the distinction every | ||
| + | | ||
| + | ------------------------------------------------------------------------------ | ||
| + | webXray | ||
| + | Tracker Radar | ||
| + | Disconnect | ||
| + | POOLED | ||
| + | |||
| + | ------------------------------------------------------------------------------ | ||
| + | 5. ADJUDICABILITY, | ||
| + | ------------------------------------------------------------------------------ | ||
| + | webXray | ||
| + | Tracker Radar | ||
| + | Disconnect | ||
| + | row-level `resolution`: | ||
| + | |||
| + | ------------------------------------------------------------------------------ | ||
| + | 6. Unweighted `current` share on the subsample, per rater | ||
| + | NOT the estimator: 35 rows with no stratum weights. It answers one | ||
| + | | ||
| + | ------------------------------------------------------------------------------ | ||
| + | webXray | ||
| + | Tracker Radar rater1 66.7% (16/ | ||
| + | Disconnect | ||
| + | |||
| + | ------------------------------------------------------------------------------ | ||
| + | 7. Every disagreement, | ||
| + | ------------------------------------------------------------------------------ | ||
| + | adgrx.com | ||
| + | | ||
| + | r2: AdGear Technologies, | ||
| + | | ||
| + | | ||
| + | | ||
| + | awltovhc.com | ||
| + | | ||
| + | r2: Commission Junction LLC (CJ Affiliate), part of Publicis Groupe | ||
| + | | ||
| + | | ||
| + | | ||
| + | cedexis.com | ||
| + | | ||
| + | r2: Cloud Software Group, Inc. (Citrix brand; formerly Cedexis, Inc.) | ||
| + | | ||
| + | cedscdn.it | ||
| + | | ||
| + | r2: CED Digital & Servizi S.r.l. | ||
| + | | ||
| + | company-target.com | ||
| + | | ||
| + | r2: Demandbase, Inc. (Demandbase) | ||
| + | | ||
| + | | ||
| + | | ||
| + | eyeota.net | ||
| + | | ||
| + | r2: Eyeota Pte Ltd (Eyeota), a subsidiary of Dun & Bradstreet Holdings, Inc. | ||
| + | | ||
| + | forter.com | ||
| + | | ||
| + | r2: Forter Ltd. (Israel) with US affiliate Forter, Inc. (Forter) | ||
| + | | ||
| + | histats.com | ||
| + | | ||
| + | r2: Wisecode S.r.l. unipersonale (Histats) | ||
| + | | ||
| + | iqzone.com | ||
| + | | ||
| + | r2: IQzone Inc. | ||
| + | | ||
| + | kameleoon.io | ||
| + | | ||
| + | r2: Kameleoon SAS (Kameleoon) | ||
| + | | ||
| + | owneriq.net | ||
| + | | ||
| + | r2: Inmar-OIQ, LLC (ownerIQ), a subsidiary of Inmar, Inc. | ||
| + | | ||
| + | pages02.net | ||
| + | | ||
| + | r2: Acoustic, L.P. (Acoustic Campaign, formerly IBM Watson Campaign Automation / Silverpop) | ||
| + | | ||
| + | | ||
| + | sa-as.com | ||
| + | | ||
| + | r2: FoundryCo, Inc. (Foundry, formerly IDG Communications, | ||
| + | | ||
| + | sharethis.com | ||
| + | | ||
| + | r2: Predactiv, Inc. (ShareThis, a Predactiv Company; contact entity ShareThis, Inc.) | ||
| + | | ||
| + | | ||
| + | tqlkg.com | ||
| + | | ||
| + | r2: Commission Junction LLC (CJ Affiliate), part of Publicis Groupe | ||
| + | | ||
| + | | ||
| + | wishabi.com | ||
| + | | ||
| + | r2: Flipp Corp. (Flipp; formerly Wishabi Inc.) | ||
| + | | ||
| + | | ||
| + | | ||
| + | -> 16 of 35 domains carry at least one disagreement. | ||
| + | |||
| + | ------------------------------------------------------------------------------ | ||
| + | 8. Rater 2 against the PUBLISHED verdicts (rater 1 after the hand | ||
| + | | ||
| + | | ||
| + | this is a sensitivity figure, not an inter-rater figure. | ||
| + | ------------------------------------------------------------------------------ | ||
| + | webXray | ||
| + | Tracker Radar | ||
| + | Disconnect | ||
| + | (the correction pass moved 5 of 105 cells on this subsample) | ||
| + | |||
| + | wrote out/ | ||
| + | </ | ||
| + | |||
| + | ===== AK. Unedited output: scripts/ | ||
| + | |||
| + | Run as '' | ||
| + | |||
| + | < | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | caught | ||
| + | |||
| + | all mutations caught. | ||
| </ | </ | ||
| Line 4278: | Line 5805: | ||
| * [[provenance: | * [[provenance: | ||
| * [[programming: | * [[programming: | ||
| + | * [[provenance: | ||
| * [[literature: | * [[literature: | ||
| + | |||
| + | ===== Markup sweep, 2026-09-17 ===== | ||
| + | |||
| + | Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with '' | ||
provenance/programming/crawler/webxray/random_sample.1788636914.txt.gz · Last modified: by karel.kubicek.claude