| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| provenance:roadmap [2026/09/08 11:35] – Record the 2026-09-08 decisions: design:platforms:ad_archives written and dequeued, and its queuing count corrected from 69 to 11 (the probe term was a homonym). Authored by Claude karel.kubicek.claude | provenance:roadmap [2026/10/08 20:49] (current) – Finish PII leakage collateral: back-links and provenance; Authored by Claude karel.kubicek.claude |
|---|
| * **They are not comparable across families.** ''cdn_edge'' (48) and ''payments_checkout'' (48) are wide regular expressions that match incidental mentions; ''anti_adblock'' (3) is narrow and specific. A larger number is not a stronger case, which is why every disposition on [[roadmap]] names the reason rather than the rank. | * **They are not comparable across families.** ''cdn_edge'' (48) and ''payments_checkout'' (48) are wide regular expressions that match incidental mentions; ''anti_adblock'' (3) is narrow and specific. A larger number is not a stronger case, which is why every disposition on [[roadmap]] names the reason rather than the rank. |
| * **The year split is not a trend.** 2025 is thin and 2026 is provisional — see [[literature:corpus]]. The split is here because a family that is entirely pre-2020 is a different proposition from one that is entirely post-2024, not because the slope means anything. | * **The year split is not a trend.** 2025 is thin and 2026 is provisional — see [[literature:corpus]]. The split is here because a family that is entirely pre-2020 is a different proposition from one that is entirely post-2024, not because the slope means anything. |
| | * **2026-09-15, writing [[:Privacy:Age assurance]] found a defect in this probe.** The ''age_assurance'' pattern is ''age (verification|assurance|estimation|gate)'' with **no word boundary before ''age''**, so //F-BLEAU: Fast Black-Box Leakage Estimation// (IEEE S&P 2019) matched on //Leak**age Estimation**//. Corrected, that row is **10**, not 11. The same class of defect bit three probes written for that page (''voltage signal'' → ''age signal'', ''improve their agency'' → ''prove their age'', ''webpage screenshot'' → ''age screen''), so **check every ''age''-prefixed alternation in this file for a leading ''\b''** before reusing a count from it. Only the ''age_assurance'' row uses that construction; the other fourteen families were not re-checked. Detail: [[:provenance:privacy:age_assurance]]. |
| * Reading the titles changed two dispositions, and this is the reason the probe prints them. ''age_assurance'' looks like 11 papers on age gates and is mostly COPPA and children's-privacy work; ''ctv_streaming'' looks like 16 papers with only one on the web platform and contains a coherent six-paper smart-TV tracking spine. Both rows on [[roadmap]] say so. | * Reading the titles changed two dispositions, and this is the reason the probe prints them. ''age_assurance'' looks like 11 papers on age gates and is mostly COPPA and children's-privacy work; ''ctv_streaming'' looks like 16 papers with only one on the web platform and contains a coherent six-paper smart-TV tracking spine. Both rows on [[roadmap]] say so. |
| |
| |
| * **Ten queued pages**, with ids fixed so that later work lands in the right place. Four ids were open questions in the backlog and were decided here: ''privacy:policies'' rather than ''privacy:privacy_policies'' (which sorts next to this site's own privacy_policy page); ''programming:internet_scanning'' rather than ''design:scanning'' (an instrument, matching [[Programming:Crawler]]); ''statistics:annotation'' rather than ''statistics:classifier_validation'' or ''design:validation'', because the page has to cover LLM annotation as well as validation; ''privacy:data_subject_rights'' to avoid colliding with [[Privacy:Requests]], which is //web// requests. | * **Ten queued pages**, with ids fixed so that later work lands in the right place. Four ids were open questions in the backlog and were decided here: ''privacy:policies'' rather than ''privacy:privacy_policies'' (which sorts next to this site's own privacy_policy page); ''programming:internet_scanning'' rather than ''design:scanning'' (an instrument, matching [[Programming:Crawler]]); ''statistics:annotation'' rather than ''statistics:classifier_validation'' or ''design:validation'', because the page has to cover LLM annotation as well as validation; ''privacy:data_subject_rights'' to avoid colliding with [[Privacy:Requests]], which is //web// requests. |
| * **''statistics:annotation'' absorbs two backlog items** — hoisting the label-validation section out of [[Design:Website classification]], and the LLM-as-annotator gap. They are one page: 30 candidate papers use an LLM to annotate, **27 of them in 2024–2026**, and the corpus-wide 42.4% report-no-validation figure cannot live on a topic page without misstating that page's denominator. | * **''statistics:annotation'' absorbs two backlog items** — hoisting the label-validation section out of [[Design:Website classification]], and the LLM-as-annotator gap. They are one page: 30 candidate papers use an LLM to annotate, **27 of them in 2024–2026**, and the corpus-wide report-no-validation figure cannot live on a topic page without misstating that page's denominator. **Both counts moved when the page was written on 2026-09-11.** The 30 is a candidate set with 70.0% precision and 13.1% recall — 21 of the 30 really annotate, and the ''classification.method == "llm"'' enum finds **175** papers the probe catches only 23 of. And the 42.4% is the share of papers with **at least one** unvalidated classifier; the share that validate nothing is **29.9%** (1,328 of 4,439). See [[provenance:statistics:annotation]]. |
| * **[[Design:Connected TV]] and [[Privacy:Age assurance]] are queued despite being thin**, as an explicit editorial decision: a page that states a method and a small, dated literature is more useful to a student than silence, and the thinness is itself reportable. The line was drawn at WebRTC leaks (6 candidates) — below it sit WebAssembly and every other thinly-published web technology, and admitting one admits all. | * **[[Design:Connected TV]] and [[Privacy:Age assurance]] were queued despite being thin**, as an explicit editorial decision: a page that states a method and a small, dated literature is more useful to a student than silence, and the thinness is itself reportable. The line was drawn at WebRTC leaks (6 candidates) — below it sit WebAssembly and every other thinly-published web technology, and admitting one admits all. |
| * **Six proposals were assessed and not written**, with counts, on [[roadmap]]. Three become sections on existing pages rather than pages. | * **Six proposals were assessed and not written**, with counts, on [[roadmap]]. Three become sections on existing pages rather than pages. |
| |
| * **This is a warning about every count in the Queued table.** [[roadmap]] already says the counts are candidate sets and floors, not populations. The ad-archives row shows the other failure direction: a candidate set can be a large **over**-count, by a factor of six here, when the probe term is a homonym. A queued row's count justifies //looking//, and nothing more. | * **This is a warning about every count in the Queued table.** [[roadmap]] already says the counts are candidate sets and floors, not populations. The ad-archives row shows the other failure direction: a candidate set can be a large **over**-count, by a factor of six here, when the probe term is a homonym. A queued row's count justifies //looking//, and nothing more. |
| * The queued page still deserved writing on the corrected number. Eleven papers is thin, but they are coherent, five of them measure the archives' own error rates, and the regulatory driver is live — which is what the row claimed. | * The queued page still deserved writing on the corrected number. Eleven papers is thin, but they are coherent, five of them measure the archives' own error rates, and the regulatory driver is live — which is what the row claimed. |
| | |
| | ===== 3c. Decisions taken on 2026-09-11 ===== |
| | |
| | * **[[Security:Authentication]] was written, and the //Assessed// row that gated it on an audit was replaced with the audit's result.** The gate was "dropped if the deployment-measurement slice is under ~20". It is **22** within the queued 46, and **45** once the candidate pool is widened to 184 — so the page was written, and the row now carries both numbers rather than the one it was queued with. |
| | * **The queued count was not wrong, it was one probe.** ''auth_tight'' still returns exactly 46 papers on the current run. What the row could not know is its precision: **24 of the 46 are not deployment measurements at all**, and three are homonyms — ''MFA'' expanded by its own paper as "Made for AdSense", ''OpenID'' matched inside "OpenIDEO", and an RPKI **relying party**, which is a route-origin validator. This is the ad-archives lesson (§3b) in a milder form: a candidate count justifies looking, and nothing more. |
| | * **The recall direction mattered more than the precision direction here.** Adding four probes and a full-text recall pass over every ''paper.cols.txt'' moved the population from the 22 the queued probe could see to 45 — and even the branded-mechanism slice the row described is 25, four of them invisible to the queued probe. The page's own subject — login-page discovery, login and password policies, session state — is invisible to a probe built from branded mechanism names, which is why the page was broadened from the row's scope. The weak step is recorded on the page's own log: **144 of the 155 papers over the full-text threshold were dismissed on their titles alone**. |
| | * **[[Security:Authentication]] became the seventh child of [[Security]]**, and the namespace page's "six children, all six written" sentence was updated in the same sitting. |
| | |
| | ===== 3d. Decisions taken on 2026-09-23 ===== |
| | |
| | * **[[Privacy:Advertising]] was written without ever being queued.** Karel chose it in the 2026-09-22 gap pass; the Queued table was empty and stays empty. An //Assessed// row records the result in the same shape as the others, because the gap pass's count (118) needed the same correction a queued count gets: **33.1%** precision and **59.2%** recall against the 71-paper population the page derived. Full audit on [[provenance:privacy:advertising]]. |
| | * **The recall miss has a shape worth remembering for the next gap pass.** A probe built from the //question// vocabulary ("ad targeting", "targeted ad", "RTB") misses the papers that answer it through a different observable — 22 of its 29 misses collect ad content, and the other seven include [[Design:Algorithm audits]]' two founding persona papers. A probe for the observable (the word "ads" itself, ≥ 40 times) found 64 of the 71. |
| | * **The //Sock-puppet and personalization audits// row was settled rather than left open.** It had been superseded since 2026-09-11 by [[Design:Algorithm audits]] and is now also covered, for ads, by [[Privacy:Advertising]] (22 of the 34 audits are in that page's population). The row's 20 was not re-derived. |
| | |
| | ===== 3e. Decisions taken on 2026-09-24 ===== |
| | |
| | * **[[Security:Online scams]] was written without ever being queued.** Karel chose it in the 2026-09-22 gap pass; the Queued table was empty and stays empty. An //Assessed// row records the result in the same shape as the other gap-pass pages, because the gap pass's reading of its own count did not survive the audit: the ''scams'' regex reproduces 43 papers, but only 7 of its 23 hits from 2024–2026 measure scam websites, and it misses 9 of the page's 27 papers. The page is the eighth child of [[:security]]. Full log: [[provenance:security:online_scams]]. |
| | |
| | ===== 3f. Decisions taken on 2026-09-25 ===== |
| | |
| | * **[[Security:Domain abuse]] was written without ever being queued.** Karel chose it in the 2026-09-22 gap pass; the Queued table was empty and stays empty. An //Assessed// row records the result in the same shape as the other gap-pass pages. The ''domain_abuse'' regex reproduces 85 papers and finds 38 of the page's 48; it misses the name-collision and delegation-hijack papers, and of its 27 hits from 2024–2026 only 11 are in the population. The page is the ninth child of [[:security]]. Full log: [[provenance:security:domain_abuse]]. |
| | * **One follow-up was filed rather than done:** a [[practices:ethics]] section on registering or claiming names to receive other people's traffic, which the new page flags as an open question. |
| | |
| | ===== 3g. Decisions taken on 2026-09-27 ===== |
| | |
| | * **[[Design:Platforms:Messaging channels]] was written without ever being queued.** Karel chose it in the 2026-09-22 gap pass; the Queued table was empty and stays empty. An //Assessed// row records the result in the same shape as the other gap-pass pages. The brief's three counts (40 / 41 / 13) reproduce exactly with case-sensitive name patterns; read against the page's 26-paper population, their union has 30.3% precision and 88.5% recall, and the WhatsApp count alone 9.8% precision. The page is the second child of [[Design:Platforms]], and [[:design]]'s table went from 18 to 19 rows. Full log: [[provenance:design:platforms:messaging_channels]]. |
| | * **Account enumeration was kept as a pointer section** on that page rather than proposed as a page of its own: four papers, one design (a census of accounts against the numbering plan), and [[Design:Platforms]] already carries its largest example. |
| | * **One follow-up was filed rather than done:** a [[practices:ethics]] section on entering and recording online groups whose members did not join a study. |
| | * **[[Design:Mobile and app measurement:Mini programs]] was written without ever being queued.** Karel chose it in the 2026-09-22 gap pass; the Queued table stays empty and an //Assessed// row records the result. The brief's count (37 papers, 21 from 2024–2026, ≥ 10 hits of ''%%/mini-?programs?|WeChat/%%'') reproduces exactly, case-insensitive. Read against the page's 16-paper population the rule has 40.5% precision and 93.8% recall; the ecosystem vocabulary alone reaches all 16. The page is the first child of [[Design:Mobile and app measurement]], which had no mention of mini-programs to replace; [[:design]]'s table went from 19 to 20 rows. Three mini-program papers from the seven venues are outside the extraction (one PDF never retrieved, two venue-years without OpenAlex abstracts) and are cited, not counted. Full log: [[provenance:design:mobile_and_app_measurement:mini_programs]]. |
| | |
| | ===== 3h. Decisions taken on 2026-10-08 ===== |
| | |
| | * **[[Security:Malicious web content]] was written without ever being queued.** Karel chose it in the 2026-09-22 gap pass; a research run on 2026-09-27 stopped before writing, and this sitting finished it. The Queued table was empty and stays empty; an //Assessed// row records the result in the same shape as the other gap-pass pages. The ''malicious_web_content'' regex reproduces 124 papers and finds 36 of the page's 39, but none of its 20 hits from 2024–2026 is in the population. The page is the tenth child of [[:security]]. Full log: [[provenance:security:malicious_web_content]]. |
| | * **The "nothing after 2023" finding was tested, not assumed:** wider full-text vocabularies over every 2024–2026 paper, a title-and-abstract probe over the whole venue index to find screening losses, and the outside-venue search. Three on-topic 2025 papers were screened out of the corpus (all defences); the page names them and does not count them. |
| | |
| | * **[[Privacy:PII leakage]] was written without ever being queued.** Karel chose it in the 2026-09-22 gap pass; the Queued table stays empty and an //Assessed// row records the result in the same shape as the other gap-pass pages. The brief's three counts (218 / 76 web / 96 from 2024–2026) re-run exactly from ''paper.norm.txt''; the page measures the same regex over ''paper.cols.txt'' (226 hits, 13 of 20 population papers). Full audit: [[provenance:privacy:pii_leakage]]. |
| | * **Decided as a new page, not a broadening of [[Privacy:Email tracking]]**, because 15 of the 20 population papers are not about an email address (search terms, wallet addresses, SSNs, sensitive visits, logged-in content, tag configurations). Both pages cite //Leaky Forms// and the contact-form study, for different purposes; each links the other. |
| |
| ===== 4. The red-link gate ===== | ===== 4. The red-link gate ===== |
| ===== 5. Limitations ===== | ===== 5. Limitations ===== |
| |
| The queued list is what the corpus supports **as of 2026-09-07** and it is not a schedule. Two rows are conditional in substance even though they read as commitments: ''statistics:annotation'' depends on the LLM-annotation slice surviving a hand audit of its 30 candidates, and [[Design:Connected TV]] will need its population derived from platform fields rather than the probe, because the probe's ''web''-platform column is exactly the wrong filter for it. The Assessed table's SSO/WebAuthn row is gated on an audit that has not been run: 46 candidates, unaudited, and the probe catches attacks on OAuth as well as deployment measurement. | The queued list is what the corpus supports **as of 2026-09-07** and it is not a schedule. Two rows are conditional in substance even though they read as commitments: ''statistics:annotation'' depended on the LLM-annotation slice surviving a hand audit of its 30 candidates — **it did: 21 annotate, 7 study LLMs, 2 use one as a generator, and the page's population turned out to be the 175-paper enum rather than the probe at all (written 2026-09-11)**, and [[Design:Connected TV]] will need its population derived from platform fields rather than the probe, because the probe's ''web''-platform column is exactly the wrong filter for it. The Assessed table's SSO/WebAuthn row **was** gated on an audit that had not been run. That audit ran on 2026-09-11 (§3c): the row now reports 21 of its own 46 as deployment measurements, and the page that came out of it derives a 45-paper population from a wider candidate pool. |
| |
| ~~DISCUSSION~~ | ~~DISCUSSION~~ |
| |
