| |
| privacy:policies [2026/09/10 17:43] – new page: measuring privacy policies, terms and store-declared labels; 102-paper POLICY enum population, tool lineage with repository dates, provenance mirror at provenance:privacy:policies (Authored by Claude) karel.kubicek.claude | privacy:policies [2026/09/10 18:06] (current) – generic review fixes: PrivaSeer 0-vs-12 (extraction fold is not a citation count), UNION share columns labelled, probe counts qualified, Discord denominator 15,525, tool table gains a full-text column, four papers get citekeys (Authored by Claude) karel.kubicek.claude |
|---|
| It is also a text-collection problem dressed up as a privacy problem, and almost everything that goes wrong goes wrong in the collection, not in the analysis. Four things break a web-trained intuition: | It is also a text-collection problem dressed up as a privacy problem, and almost everything that goes wrong goes wrong in the collection, not in the analysis. Four things break a web-trained intuition: |
| |
| - **The document has no fixed address.** There is no ''/.well-known/privacy-policy''. You find it by guessing at link text in a footer, and every study's denominator is really "the sites where my link heuristic worked". | - **The document has no fixed address.** There is no ''/.well-known/privacy-policy''. P3P once defined one (''/w3c/p3p.xml'') and P3P is dead — see [[#Which Methods Are Current]]. So you find the document by guessing at link text in a footer, and every study's denominator is really "the sites where my link heuristic worked". |
| - **"Has a privacy policy" is not a property of the web; it is a property of the ecosystem you sampled.** It runs from **94.2%** of websites with a data-collecting form {[cui2025_privacy]} to **4.35%** of Discord chatbots {[edu2022_exploring]}. Quoting a rate without naming the ecosystem says nothing. | - **"Has a privacy policy" is not a property of the web; it is a property of the ecosystem you sampled.** It runs from **94.2%** of websites with a data-collecting form {[cui2025_privacy]} to **4.35%** of Discord chatbots {[edu2022_exploring]}. Quoting a rate without naming the ecosystem says nothing. |
| - **The interesting claim is never "the policy says X".** It is "the policy says X and the traffic does Y". A full-text probe finds comparison-against-observed-behaviour language in **80 of the 123** candidate papers — an upper bound on a candidate set rather than a measured rate, but at that magnitude the shape is unmistakable: the comparison //is// the measurement, and the policy is only one of its two arms. | - **The interesting claim is never "the policy says X".** It is "the policy says X and the traffic does Y". A full-text probe finds comparison-against-observed-behaviour language in **80 of the 123** candidate papers — an upper bound on a candidate set rather than a measured rate, but at that magnitude the shape is unmistakable: the comparison //is// the measurement, and the policy is only one of its two arms. |
| - **Half of this literature is about Android apps, not websites.** Of the 123 corpus papers here, 43 measure only mobile, 39 only web and 17 both — and 24 measure neither, because a policy study can be about a VR store, an IoT companion app, a voice assistant or a chatbot ecosystem. The methods are genuinely different — an app policy is reached through a store listing, an app has a store-declared privacy label to contradict as well as a policy, and the tool lineage below was built for Android. | - **Half of this literature is about mobile apps, not websites.** Of the 123 corpus papers here, 43 measure only mobile, 39 only web and 17 both — and 24 measure neither, because a policy study can be about a VR store, an IoT companion app, a voice assistant or a chatbot ecosystem. The methods are genuinely different — an app policy is reached through a store listing, an app has a store-declared privacy label to contradict as well as a policy, the tool lineage below was built for Android, and the store-label half of the literature is mostly iOS. |
| |
| <WRAP important> | <WRAP important> |
| </WRAP> | </WRAP> |
| |
| Everything below about "the literature" is a claim about **seven venues** — CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026, 5,859 papers with extracted full text. Policy analysis also publishes heavily at ACL, EMNLP, CHI, SOUPS and in law reviews, none of which are in the corpus. Two of the field's foundation stones — the OPP-115 corpus (ACL 2016) and PrivaSeer (ACL 2021) — are outside it, and PrivaSeer is named by **zero** corpus papers. So read the counts as "how these seven security and privacy venues do policy analysis", not as the state of the field. See [[Literature:Corpus]]. | Everything below about "the literature" is a claim about **seven venues** — CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026, 5,859 papers with extracted full text. Policy analysis also publishes heavily at ACL, EMNLP, CHI, SOUPS and in law reviews, none of which are in the corpus. Two of the field's foundation stones — the OPP-115 corpus (ACL 2016) and PrivaSeer (ACL 2021) — are outside it: **no corpus paper's extraction records PrivaSeer as a tool or a data source**, though 12 corpus papers name it in their text. So read the counts as "how these seven security and privacy venues do policy analysis", not as the state of the field. See [[Literature:Corpus]]. |
| |
| ===== What to Read First ===== | ===== What to Read First ===== |
| - {[mhaidli2023researchers]} — **read this before you design anything.** Twenty-six interviews with researchers who have analysed policies. Its own discipline breakdown is 15 in computer science (10 of them in ML/NLP), and the rest spread over health, communication, policy, education, business, sociology and political science. Its finding is that //"no clear best practices or methodologies for privacy policy analysis have emerged so far"//, and it names the three places people get stuck: policy selection, policy retrieval, and content analysis. That is the shape of this page. | - {[mhaidli2023researchers]} — **read this before you design anything.** Twenty-six interviews with researchers who have analysed policies. Its own discipline breakdown is 15 in computer science (10 of them in ML/NLP), and the rest spread over health, communication, policy, education, business, sociology and political science. Its finding is that //"no clear best practices or methodologies for privacy policy analysis have emerged so far"//, and it names the three places people get stuck: policy selection, policy retrieval, and content analysis. That is the shape of this page. |
| - {[andow2020_actions]} — **the policy-versus-behaviour design, done properly.** PoliCheck introduced //entity-sensitive// consistency: it is not enough that the policy mentions "location", it must mention the //recipient// that received it. Re-running the analysis entity-insensitively falsely marked up to **37.1%** of inconsistent third-party flows as consistent, which is the single best argument for reading this paper before writing your own comparison. | - {[andow2020_actions]} — **the policy-versus-behaviour design, done properly.** PoliCheck introduced //entity-sensitive// consistency: it is not enough that the policy mentions "location", it must mention the //recipient// that received it. Re-running the analysis entity-insensitively falsely marked up to **37.1%** of inconsistent third-party flows as consistent, which is the single best argument for reading this paper before writing your own comparison. |
| - {[cui2023_poligraph]} — **the current tool, and the current critique.** PoliGraph builds a knowledge graph from the policy instead of classifying sentences, and it is what a 2026 study reaches for. Its own finding that **25.5%** (1,339/5,255) of policies define a term differently from the CCPA-based ontology is a warning about every ontology-matching pipeline including its own. | - {[cui2023_poligraph]} — **the current tool, and the current critique.** PoliGraph builds a knowledge graph from the policy instead of classifying sentences, and it is what a 2026 study reaches for: {[carrillo2026_traceability]} runs //PoliGraph-er// over GPT-ecosystem app policies and calls it "a state-of-the-art NLP tool". Its own finding that **25.5%** (1,339/5,255) of policies define a term differently from the CCPA-based ontology is a warning about every ontology-matching pipeline including its own. |
| - {[amos2021privacy]} — **the longitudinal corpus and how to build one.** 1,071,488 English policies from over 130,000 websites, reconstructed from the Wayback Machine. Read the curation section: it is the most careful published account of turning archived HTML into a policy dataset. | - {[amos2021privacy]} — **the longitudinal corpus and how to build one.** 1,071,488 English policies from over 130,000 websites, reconstructed from the Wayback Machine. Read the curation section: it is the most careful published account of turning archived HTML into a policy dataset. |
| - {[linden2020_privacy]} — **the natural experiment.** Matched pre/post-GDPR policy pairs, EU and Global. It shows what a before/after policy study looks like when you have controls, and it is honest that longer is not clearer. | - {[linden2020_privacy]} — **the natural experiment.** Matched pre/post-GDPR policy pairs, EU and Global. It shows what a before/after policy study looks like when you have controls, and it is honest that longer is not clearer. |
| * **The document is not in a language you handle.** **20.5% (9,523/46,472)** of the same set had no English version. A pipeline that silently drops non-English documents has silently changed its population; see [[Programming:Multilingual support]]. | * **The document is not in a language you handle.** **20.5% (9,523/46,472)** of the same set had no English version. A pipeline that silently drops non-English documents has silently changed its population; see [[Programming:Multilingual support]]. |
| |
| Only 7 of the 123 corpus papers say anything at all about dead or unresolvable policy links, and only 9 mention handling a policy served as a PDF.((Full-text probes over the 123 papers, narrow form; the wide forms are 64 and 22. A probe counts papers whose text contains the phrase, not papers that did the thing, so read these as upper bounds on a candidate set. Script and both forms on the provenance page.)) Report yours. | A narrow full-text probe for dead-or-unresolvable-link language matches **7** of the 123 candidate papers, and one for PDF handling matches **9**.((Both probes have a wide form too — 64 and 22 — and the wide form is a different, looser phrasing of the same question rather than a superset in every row. A probe counts papers whose text contains a phrase, not papers that did the thing, so neither number is a measurement of the field. The script and both forms are on the provenance page.)) Neither number is a measurement of how many papers //handled// these cases; the gap between 7 and 64 is exactly how much a probe's width decides its answer. Whatever the real rate is, report yours. |
| |
| ==== "How many have a policy" depends entirely on what you sampled ==== | ==== "How many have a policy" depends entirely on what you sampled ==== |
| | Alexa skills, later | 65,195 skills | 21,063 | {[yan2024_quality]} | | | Alexa skills, later | 65,195 skills | 21,063 | {[yan2024_quality]} | |
| | VR apps | 11,923 apps on 10 VR platforms | **29.5%** | {[zhan2024_vpvet]} | | | VR apps | 11,923 apps on 10 VR platforms | **29.5%** | {[zhan2024_vpvet]} | |
| | Discord chatbots requesting permissions | 15,528 chatbots | **4.35%** (676) — 14,852 had none | {[edu2022_exploring]} | | | Discord chatbots requesting permissions | **15,525** unique active chatbots, as the paper's own table states | **4.35%** (676) had a policy link. The paper separately reports 14,852 without one, which does not reconcile with its own total | {[edu2022_exploring]} | |
| | GPT Actions declaring a policy URL | Actions with a ''legal_info_url'' | 93.96% of those URLs were reachable | {[wu2025_depth]} | | | GPT Actions declaring a policy URL | Actions with a ''legal_info_url'' | 93.96% of those URLs were reachable | {[wu2025_depth]} | |
| |
| ==== Existing corpora, so you do not build another one ==== | ==== Existing corpora, so you do not build another one ==== |
| |
| Building a policy corpus is a month of work and three of the four corpora below are still downloadable. Check them before you crawl. | Building a policy corpus is a month of work and all four corpora below were still reachable on 2026-09-10. Check them before you crawl. |
| |
| ^ Corpus ^ What it is ^ Size ^ Where, and status as of 2026-09-10 ^ | ^ Corpus ^ What it is ^ Size ^ Where, and status as of 2026-09-10 ^ |
| | **OPP-115** | The reference annotated corpus. Each policy read and annotated by three law graduate students; the label set that Polisis, PrivBERT and most supervised work below is trained on | 115 website policies | ''usableprivacy.org/data'', live. CC BY-NC for research; a separate commercial licence exists via CMU Flintbox. **14 corpus papers use it** | | | **OPP-115** | The reference annotated corpus. Each policy read and annotated by three law graduate students; the label set that Polisis, PrivBERT and most supervised work below is trained on | 115 website policies | ''usableprivacy.org/data'', live. The site offers it for research //"in the spirit of"// a CC BY-NC licence; a separate commercial licence covers the **annotation files** via CMU Flintbox. **14 corpus papers' extractions record it; 30 name it in their text** | |
| | **APP-350** | The mobile-app counterpart, annotated with app privacy practices | 350 Android app policies | Same host, live, same //research// licence — but unlike OPP-115 it documents **no commercial-licence path**, only "contact the author". **1 corpus paper uses it** | | | **APP-350** | The mobile-app counterpart, annotated with app privacy practices | 350 Android app policies | Same host, live, same //research// wording — but unlike OPP-115 it documents **no commercial-licence path**, only "contact the author". **1 corpus paper's extraction records it; 9 name it in their text** | |
| | **PrivaSeer** | A crawled policy corpus and search engine, plus the ''PrivBERT'' language model pretrained on it | 3,967,487 policies | ''privaseer.ist.psu.edu'', live; corpus on PSU GitLab; model ''mukund/privbert'' on Hugging Face (CC BY-NC-SA). **Named by zero corpus papers**, though 3 use PrivBERT | | | **PrivaSeer** | A crawled policy corpus and search engine, plus the ''PrivBERT'' language model pretrained on it | 3,967,487 policies | ''privaseer.ist.psu.edu'', live; corpus on PSU GitLab; model ''mukund/privbert'' on Hugging Face, //"in the spirit of"// CC BY-NC-SA for research. **No corpus paper's extraction records it as a source**; 12 name it in their text and 3 use PrivBERT | |
| | **Princeton–Leuven longitudinal corpus** | Policies reconstructed from Wayback snapshots, 1997–2019, for before/after work | 1,071,488 English policies, >130,000 sites | Data: ''github.com/citp/privacy-policy-historical''; code: ''github.com/citp/PrivacyPoliciesOverTime''. Both alive, neither archived, but **last commits 2023-10-12 and 2022-06-06** and **no licence file on either** {[amos2021privacy]} | | | **Princeton–Leuven longitudinal corpus** | Policies reconstructed from Wayback snapshots, 1997–2019, for before/after work | 1,071,488 English policies, >130,000 sites | Data: ''github.com/citp/privacy-policy-historical''; code: ''github.com/citp/PrivacyPoliciesOverTime''. Both alive, neither archived, but **last commits 2023-10-12 and 2022-06-06** and **no licence file on either** {[amos2021privacy]} | |
| |
| |
| <WRAP important> | <WRAP important> |
| A fresh student searching for "privacy policy analysis tool" lands on **Polisis (2018)** because it has the most citations and a live demo site. It is eight years old, its label set is OPP-115's, and there is **no official code repository**. The third-party reproductions a name search returns are not maintained either — the two most-starred last moved on their default branches in **February and July 2020**, and the one whose GitHub activity date reads 2023 turns out to be a Dependabot security bump on a side branch. Do not start there. | A fresh student searching for "privacy policy analysis tool" lands on **Polisis (2018)** because it has the most citations and a live demo site. It is eight years old, its label set is OPP-115's, and there is **no official code repository**. The third-party reproductions a name search returns are not maintained either — the top two hits last moved on their default branches in **February and July 2020**, and the one whose GitHub activity date reads 2023 turns out to be a Dependabot security bump on a side branch. Do not start there. |
| </WRAP> | </WRAP> |
| |
| Counts below are papers in the whole 5,859-paper corpus whose extraction names the artefact anywhere (tool name, purpose, classifier resource, source list, or detection technique). Repository facts were fetched on **2026-09-10** from GitHub's commits API on each repository's own default branch — not from ''pushed_at'', which counts activity on any branch, and not from ''/releases/latest'' or ''/tags'', which are unreliable for repositories like these. The script and its unedited output are on the provenance page. | Two count columns, because they answer different questions and a fold over the extraction cannot answer the second. **Named in an extraction** is papers in the whole 5,859-paper corpus whose extraction record names the artefact somewhere (tool name, purpose, classifier resource, source list, or detection technique) — that is "somebody's methods section used it". **Named in full text** is a case-insensitive scan of all 5,869 readable ''paper.cols.txt'' files — that is "somebody cited it", which is nearly always the larger number. The MAPS full-text pattern is its title rather than its acronym, because ''MAPS'' also matches Google Maps and the Play Store's //Maps & Navigation// category; the override and its reason are printed by the probe script. Repository facts were fetched on **2026-09-10** from GitHub's commits API on each repository's own default branch — not from ''pushed_at'', which counts activity on any branch, and not from ''/releases/latest'' or ''/tags'', which are unreliable for repositories like these. The script and its unedited output are on the provenance page. |
| |
| ^ Artefact ^ First paper ^ Corpus papers ^ Used 2024+ ^ What it does, and whether to start there ^ | ^ Artefact ^ First paper ^ Named in an **extraction** ^ Named in **full text** ^ Used 2024+ ^ What it does, and whether to start there ^ |
| | **Privee** | USENIX 2014 | 1 | 0 | Regex rules plus naive Bayes. Historical interest only | | | **Privee** | USENIX 2014 | 1 | 25 | 0 | Regex rules plus naive Bayes. Historical interest only | |
| | **OPP-115** | ACL 2016 (outside corpus) | 14 | 6 | The label set, not a tool. Still the substrate under most supervised work | | | **OPP-115** | ACL 2016 (outside corpus) | 14 | 30 | 6 | The label set, not a tool. Still the substrate under most supervised work | |
| | **Polisis / PriBot** | USENIX 2018 {[harkous2018_polisis]} | 13 | 5 | Hierarchical CNNs over fastText embeddings, trained on 65 OPP-115 policies with 50 held out; average F1 **0.84**. Demo at ''pribot.org/polisis'' alive (it redirects HTTPS to plain HTTP), **no official repo** | | | **Polisis / PriBot** | USENIX 2018 {[harkous2018_polisis]} | 13 | 68 | 5 | Hierarchical CNNs over fastText embeddings, trained on 65 OPP-115 policies with 50 held out; average F1 **0.84**. Demo at ''pribot.org/polisis'' alive (it redirects HTTPS to plain HTTP), **no official repo** | |
| | **PolicyLint** | USENIX 2019 {[andow2019_policylint]} | 18 | 3 | Ontology-based extraction of collection/sharing statements from single sentences; finds //internal// contradictions. **14.2%** (1,618/11,430) of Play policies contradict themselves. Code at ''github.com/benandow/PrivacyPolicyAnalysis'' — **searching GitHub for "PolicyLint" or "PoliCheck" does not find it**; last commit 2022-10-05; ''LICENSE.txt'' is a three-clause BSD naming NC State, which GitHub's detector labels "Other" | | | **PolicyLint** | USENIX 2019 {[andow2019_policylint]} | 18 | 63 | 3 | Ontology-based extraction of collection/sharing statements from single sentences; finds //internal// contradictions. **14.2%** (1,618/11,430) of Play policies contradict themselves. Code at ''github.com/benandow/PrivacyPolicyAnalysis'' — **searching GitHub for "PolicyLint" or "PoliCheck" does not find it**; last commit 2022-10-05; ''LICENSE.txt'' is a three-clause BSD naming NC State, which GitHub's detector labels "Other" | |
| | **MAPS** | PETS 2019 {[zimmeck2019_maps]} | 2 | 1 | The million-app compliance pipeline. Its policy classifier is the thing people reuse | | | **MAPS** | PETS 2019 {[zimmeck2019_maps]} | 2 | 50 | 1 | The million-app compliance pipeline. Its policy classifier is the thing people reuse | |
| | **PoliCheck** | USENIX 2020 {[andow2020_actions]} | 9 | 4 | The **entity-sensitive** consistency check: matches an observed data flow against what the policy says about //that recipient//. Same repository as PolicyLint, and equally unfindable by name | | | **PoliCheck** | USENIX 2020 {[andow2020_actions]} | 9 | 63 | 4 | The **entity-sensitive** consistency check: matches an observed data flow against what the policy says about //that recipient//. Same repository as PolicyLint, and equally unfindable by name | |
| | **PurPliance** | 2021 | 5 | 0 | Adds //purpose// to the consistency check. ''github.com/ducalpha/PurPlianceOpenSource'', BSD-style (U. Michigan 2022); last code commit 2022-12-25, last commit of any kind 2024-03-11 — **the most recently touched repository in this table** | | | **PurPliance** | 2021 | 5 | 14 | 0 | Adds //purpose// to the consistency check. ''github.com/ducalpha/PurPlianceOpenSource'', BSD-style (U. Michigan 2022); last code commit 2022-12-25, last commit of any kind 2024-03-11 — **the most recently touched repository in this table** | |
| | **PrivBERT** | 2021 (PrivaSeer) | 3 | 3 | RoBERTa pretrained on ~1M policies. The drop-in sentence classifier | | | **PrivBERT** | 2021 (PrivaSeer) | 3 | 3 | 3 | RoBERTa pretrained on ~1M policies. The drop-in sentence classifier | |
| | **PoliGraph / poligraph-er** | USENIX 2023 {[cui2023_poligraph]} | 6 | 5 | Knowledge graph over the whole document instead of sentence classification; **70.6% recall at 96.9% precision** on collection tuples. MIT, last commit 2023-06-21, **no PyPI package** under either name; its README says a GPU is needed for hardware acceleration but that it runs without one, "significantly lower" performance | | | **PoliGraph / poligraph-er** | USENIX 2023 {[cui2023_poligraph]} | 6 | 14 | 5 | Knowledge graph over the whole document instead of sentence classification; **70.6% recall at 96.9% precision** on collection tuples. MIT, last commit 2023-06-21, **no PyPI package** under either name; its README says a GPU is needed for hardware acceleration but that it runs without one, "significantly lower" performance | |
| | **PolicyChecker** | CCS 2023 {[xiang2023_policychecker]} | 1 | 0 | GDPR //completeness// rather than consistency. ''github.com/AndyXiang945/PolicyChecker'', last commit 2023-11-20, **no licence file** | | | **PolicyChecker** | CCS 2023 {[xiang2023_policychecker]} | 1 | 9 | 0 | GDPR //completeness// rather than consistency. ''github.com/AndyXiang945/PolicyChecker'', last commit 2023-11-20, **no licence file** | |
| | **PolicyComp** | USENIX 2023 {[zhou2023_policycomp]} | 1 | 0 | Compares a policy against its //competitors'// policies to find overbroad collection | | | **PolicyComp** | USENIX 2023 {[zhou2023_policycomp]} | 1 | 10 | 0 | Compares a policy against its //competitors'// policies to find overbroad collection | |
| | **Lalaine** | USENIX 2023 {[xiao2023_lalaine]} | 2 | 1 | Apple privacy-label compliance specifically. ''github.com/xiaoyue10131748/Lalaine'', MIT, last commit 2023-09-25 | | | **Lalaine** | USENIX 2023 {[xiao2023_lalaine]} | 2 | 28 | 1 | Apple privacy-label compliance specifically. ''github.com/xiaoyue10131748/Lalaine'', MIT, last commit 2023-09-25 | |
| | **Calpric** | USENIX 2023 {[qiu2023_calpric]} | 1 | 0 | Crowdsourcing + active learning to get 16,856 labelled segments cheaply. ''github.com/dlgroupuoft/Calpric'', last commit 2023-06-21, **no licence file**, artifact "Available" only | | | **Calpric** | USENIX 2023 {[qiu2023_calpric]} | 1 | 3 | 0 | Crowdsourcing + active learning to get 16,856 labelled segments cheaply. ''github.com/dlgroupuoft/Calpric'', last commit 2023-06-21, **no licence file**, artifact "Available" only | |
| | **PrivaSeer** | ACL 2021 (outside corpus) | 0 | 0 | Corpus + search engine. Nobody in these seven venues cites it | | | **PrivaSeer** | ACL 2021 (outside corpus) | 0 | 12 | 0 | Corpus + search engine. Nobody's //methods section// records using it; twelve papers cite it | |
| |
| **What this table is really saying.** Not one repository in this table has a commit later than **March 2024**, and that one (PurPliance) is a README edit — its last code commit is December 2022. PoliGraph, the newest tool, has not moved since June 2023. Finding the code is its own problem: a GitHub name search for **PolicyLint** or **PoliCheck** returns only unrelated projects, because the repository is named after the paper series and not after either tool. Calpric and Lalaine are a milder version of the same: a plain web search on the tool name surfaces the paper and no repository, and you get to the code either through GitHub's own repository search or through the USENIX artifact-evaluation index, which links both directly.((Re-checked 2026-09-10: ''secartifacts.github.io/usenixsec2023/results'' (HTTP 200) links ''github.com/dlgroupuoft/Calpric'' and lists Lalaine; GitHub's repository search returns ''dlgroupuoft/Calpric'' and ''xiaoyue10131748/Lalaine'' as top name matches. PolicyChecker was named here in an earlier draft and should not have been — it is a CCS paper, is not on that index at all, and ''github.com/AndyXiang945/PolicyChecker'' is the first hit for its name. A general lesson about artifact discoverability, not a criticism of these authors.)) Budget for reading and porting code, not for running it. | **What this table is really saying.** Not one repository in this table has a commit later than **March 2024**, and that one (PurPliance) is a README edit — its last code commit is December 2022. PoliGraph, the newest tool, has not moved since June 2023. That is evidence of //unmaintained code//, which is not quite the same as an abandoned project: PoliGraph is still being cited into 2026, and none of these authors promised maintenance. Finding the code is its own problem: a GitHub name search for **PolicyLint** or **PoliCheck** returns only unrelated projects, because the repository is named after the paper series and not after either tool. Calpric and Lalaine are a milder version of the same: a plain web search on the tool name surfaces the paper and no repository, and you get to the code either through GitHub's own repository search or through the USENIX artifact-evaluation index, which links both directly.((Re-checked 2026-09-10: ''secartifacts.github.io/usenixsec2023/results'' (HTTP 200) links ''github.com/dlgroupuoft/Calpric'' and lists Lalaine; GitHub's repository search returns ''dlgroupuoft/Calpric'' and ''xiaoyue10131748/Lalaine'' as top name matches. PolicyChecker is //not// in this group: it is a CCS paper, is not on that index, and ''github.com/AndyXiang945/PolicyChecker'' is the first hit for its name. A general lesson about artifact discoverability, not a criticism of these authors. How this footnote reached its current form is on the provenance page.)) Budget for reading and porting code, not for running it. |
| |
| ===== Policy Versus Behaviour: the Actual Measurement ===== | ===== Policy Versus Behaviour: the Actual Measurement ===== |
| ==== Store-declared labels are a second, cheaper arm ==== | ==== Store-declared labels are a second, cheaper arm ==== |
| |
| Since 2020–2022 both mobile stores make the developer declare data practices in a structured form, which gives you a machine-readable disclosure to compare against **without** any NLP at all. Both are current and both are enforced: | Since 2020–2022 both mobile stores make the developer declare data practices in a structured form, which gives you a machine-readable disclosure to compare against **without** any NLP at all. Both requirements are current; neither store publishes figures on how often it enforces them, so treat "required" as a rule rather than as a fact about the population: |
| |
| * **Apple App Privacy labels**, mandatory for new and updated apps since December 2020. Since **1 May 2024** Apple additionally requires a **privacy manifest** (''PrivacyInfo.xcprivacy'') and a valid signature from listed third-party SDKs, declaring "required reasons" for certain APIs; Apple's own announcement says the requirement //"will expand to include the entire app binary"//.((Apple, [[https://developer.apple.com/news/?id=pvszzano|"Upcoming third-party SDK requirements"]], 26 April 2024. Fetched 2026-09-10; Apple's live //upcoming requirements// page carries no later deadline superseding it.)) | * **Apple App Privacy labels**, mandatory for new and updated apps since December 2020 — a date taken from contemporaneous reporting; Apple's current label documentation carries no date, so this one is not primary-sourced. Since **1 May 2024** Apple additionally requires a **privacy manifest** (''PrivacyInfo.xcprivacy'') and a valid signature from listed third-party SDKs, declaring "required reasons" for certain APIs; Apple's own announcement says the requirement //"will expand to include the entire app binary"//.((Apple, [[https://developer.apple.com/news/?id=pvszzano|"Upcoming third-party SDK requirements"]], 26 April 2024. Fetched 2026-09-10; Apple's live //upcoming requirements// page carries no later deadline superseding it.)) |
| * **Google Play Data safety**, mandatory since **20 July 2022**. Separately, Play's User Data policy requires a privacy policy from //every// app: //"Apps that do not access any personal and sensitive user data must still submit a privacy policy."//((Google Play, [[https://support.google.com/googleplay/android-developer/answer/10144311|User Data policy]]. Quote fetched 2026-09-10 and present verbatim. The **20 July 2022** date is //not// on Google's current page — it rests on contemporaneous reporting, which is why it is flagged on the provenance page as unverified against a primary source.)) | * **Google Play Data safety**, mandatory since **20 July 2022**. Separately, Play's User Data policy requires a privacy policy from //every// app: //"Apps that do not access any personal and sensitive user data must still submit a privacy policy."//((Google Play, [[https://support.google.com/googleplay/android-developer/answer/10144311|User Data policy]]. Quote fetched 2026-09-10 and present verbatim. The **20 July 2022** date is //not// on Google's current page — it rests on contemporaneous reporting, which is why it is flagged on the provenance page as unverified against a primary source.)) |
| |
| The label is not the policy and the two disagree: {[ali2024_honesty]} found 228,539 apps reporting linked-data practices in their policy but not their label. If your study covers mobile, collect both — comparing them is a measurement in itself, and it is where 2026 work is going: the PETS 2026 //Disclosure Divergence: Measuring Privacy Policy and Data Safety Misalignment at Scale// does exactly this, and the PETS 2026 study of why developers do not update Apple's privacy labels attacks the same gap from the developer side. | The label is not the policy and the two disagree: {[ali2024_honesty]} found 228,539 apps reporting linked-data practices in their policy but not their label. If your study covers mobile, collect both — comparing them is a measurement in itself, and it is where 2026 work is going: {[khatun2026_disclosure]} does exactly this at scale, and {[alsahdi2026_labels]} attacks the same gap from the developer side, by asking why they do not update the label. |
| |
| ===== What the Policy Text Itself Looks Like ===== | ===== What the Policy Text Itself Looks Like ===== |
| * **Semantic change is concentrated, not diffuse.** {[adhikari2023_evolution]} found the //User Choice/Control// category changed semantically in 26% of policies on average across two decades — most sections do not move. | * **Semantic change is concentrated, not diffuse.** {[adhikari2023_evolution]} found the //User Choice/Control// category changed semantically in 26% of policies on average across two decades — most sections do not move. |
| |
| Readability metrics are the one place this page will contradict a textbook: Flesch and Flesch–Kincaid are validated on prose for schoolchildren, not on legal documents with defined terms, and 17 of the 123 papers use one anyway. Report the raw length distribution alongside any readability score, because the score's absolute value means less than its trend. | Readability metrics are the one place this page will contradict a textbook: Flesch and Flesch–Kincaid are validated on prose for schoolchildren, not on legal documents with defined terms, and a narrow probe for readability language matches 17 of the 123 candidate papers (68 at the wide width). Report the raw length distribution alongside any readability score, because the score's absolute value means less than its trend. |
| |
| ===== Which Methods Are Current ===== | ===== Which Methods Are Current ===== |
| | **Hand-coding** by trained annotators | **Current, and it did not go away.** | Still the most common method in every era including 2025–2026 (45.8%). LLMs are being added beside it, mostly as the thing the humans validate | | | **Hand-coding** by trained annotators | **Current, and it did not go away.** | Still the most common method in every era including 2025–2026 (45.8%). LLMs are being added beside it, mostly as the thing the humans validate | |
| | **Supervised sentence classifiers** trained on OPP-115 (Polisis, PrivBERT, BERT variants) | **Declining, still the reproducible baseline.** | 42.9% (2014–2018) → 12.5% (2025–2026). Use one as the comparison arm even if your headline method is an LLM | | | **Supervised sentence classifiers** trained on OPP-115 (Polisis, PrivBERT, BERT variants) | **Declining, still the reproducible baseline.** | 42.9% (2014–2018) → 12.5% (2025–2026). Use one as the comparison arm even if your headline method is an LLM | |
| | **Ontology + dependency parsing** (PolicyLint, PoliCheck, PoliGraph) | **Current for consistency work, unmaintained as code.** | ''heuristic-rules'' peaked at 40% in 2022–2024, fell to 16.7%. PoliGraph is still cited into 2026 but has not been committed to since 2023-06-21 | | | **Ontology + dependency parsing** (PolicyLint, PoliCheck, PoliGraph) | **Current for consistency work, unmaintained as code.** | ''heuristic-rules'' was 42.9% in 2014–2018 and 40.0% in 2022–2024, and fell to 16.7% in 2025–2026. PoliGraph is still cited into 2026 but has not been committed to since 2023-06-21 | |
| | **Readability scores** as a headline result | **Historical.** | Established by 2021 and not a contribution on its own since. Report it, do not build on it | | | **Readability scores** as a headline result | **Historical.** | Established by 2021 and not a contribution on its own since. Report it, do not build on it | |
| | **Comparing policy against store-declared labels** | **New and growing.** | Apple labels mandatory since Dec 2020, Play Data safety since Jul 2022; the comparison papers start in 2023 and there are four in 2024–2026 | | | **Comparing policy against store-declared labels** | **New and growing.** | Apple labels required since Dec 2020, Play Data safety since Jul 2022; the comparison papers start in 2023 and the corpus's 2024–2026 slice holds several more. //How many// is a reading of titles, not a count a script produces, so no number is given here | |
| | **P3P** or any machine-readable policy //content// format | **Dead, and nothing replaced it.** | W3C retired P3P 1.1 on 30 August 2018 with an explicit "should not be referenced in this form or implemented as-is"; the P3P project home page's newest news item is dated **3 October 2007** and its own footer stamp reads 2018-02-02. GPC and IAB's GPP are consent //signals//, not policy content((Verified 2026-09-10 against the primary W3C pages with Playwright — ''w3.org'' answers ''curl'' with a Cloudflare 403 — see ''scripts/policies_w3c_p3p_check.mjs''. ADPC exists as a proposal and is not in production.)) | | | **P3P** or any machine-readable policy //content// format | **Dead, and nothing replaced it.** | W3C retired P3P 1.1 on 30 August 2018 with an explicit "should not be referenced in this form or implemented as-is"; the P3P project home page's newest news item is dated **3 October 2007** and its own footer stamp reads 2018-02-02. GPC and IAB's GPP are consent //signals//, not policy content((Verified 2026-09-10 against the primary W3C pages with Playwright — ''w3.org'' answers ''curl'' with a Cloudflare 403 — see ''scripts/policies_w3c_p3p_check.mjs''. ADPC exists as a proposal and is not in production.)) | |
| |
| |
| <WRAP tip> | <WRAP tip> |
| If you are choosing a method today: use an LLM for extraction, use a supervised OPP-115-trained classifier as your comparison arm, hand-label a validation sample yourself, and report all three. That is what the 2025–2026 papers that survived review actually did — {[xie2025_evaluating]} and {[chanenson2025_automating]} are the two to copy. | If you are choosing a method today: use an LLM for extraction, use a supervised OPP-115-trained classifier as your comparison arm, hand-label a validation sample yourself, and report all three. {[xie2025_evaluating]} is the closest model: an LLM pipeline with a Polisis-benchmark comparison arm and a hand-validated sample. {[chanenson2025_automating]} is worth reading beside it for the annotation design, but its comparison arm is a purpose-built RNN rather than an OPP-115-trained classifier. |
| </WRAP> | </WRAP> |
| |
| * **''UNION''** — **123** papers: ''POLICY'' plus a title-and-summary probe for policy/notice/terms/label vocabulary. A candidate set, used only for rankings and for "does this literature do X", never for a rate. | * **''UNION''** — **123** papers: ''POLICY'' plus a title-and-summary probe for policy/notice/terms/label vocabulary. A candidate set, used only for rankings and for "does this literature do X", never for a rate. |
| |
| The two overlap on 56 papers. **46 papers fire on the enum but not the title probe** — mostly Alexa-skill, IoT-companion-app and VR studies where the policy analysis is one component of a wider privacy audit, which is a good reminder that most policy measurement is not published under a policy-shaped title. | The two overlap on 56 papers. **46 papers fire on the enum but not the title probe.** Their platforms are ''other-online-service'' 21, ''mobile'' 20, ''web'' 18, ''iot'' 4, ''offline'' 1 (multi-valued) — voice-assistant skill audits, LLM app-store studies, IoT companion apps, children's-app compliance, browser extensions, and a porn-ecosystem study. In all of them the policy analysis is one component of a wider privacy audit, which is a good reminder that most policy measurement is not published under a policy-shaped title. All 46 are listed on the provenance page. |
| |
| ^ Year ^ corpus ^ ''POLICY'' ^ ''UNION'' ^ share of corpus ^ | ^ Year ^ corpus ^ ''POLICY'' ^ ''UNION'' ^ ''POLICY'' share of corpus ^ ''UNION'' share of corpus ^ |
| | 2014 | 166 | 1 | 1 | 0.6% | | | 2014 | 166 | 1 | 1 | 0.6% | 0.6% | |
| | 2016 | 182 | 1 | 2 | 1.1% | | | 2016 | 182 | 1 | 2 | 0.5% | 1.1% | |
| | 2017 | 231 | 3 | 3 | 1.3% | | | 2017 | 231 | 3 | 3 | 1.3% | 1.3% | |
| | 2018 | 254 | 2 | 2 | 0.8% | | | 2018 | 254 | 2 | 2 | 0.8% | 0.8% | |
| | 2019 | 402 | 6 | 6 | 1.5% | | | 2019 | 402 | 6 | 6 | 1.5% | 1.5% | |
| | 2020 | 404 | 8 | 9 | 2.2% | | | 2020 | 404 | 8 | 9 | 2.0% | 2.2% | |
| | 2021 | 379 | 7 | 9 | 2.4% | | | 2021 | 379 | 7 | 9 | 1.8% | 2.4% | |
| | 2022 | 546 | 17 | 19 | 3.5% | | | 2022 | 546 | 17 | 19 | 3.1% | 3.5% | |
| | 2023 | 719 | 14 | 16 | 2.2% | | | 2023 | 719 | 14 | 16 | 1.9% | 2.2% | |
| | 2024 | 690 | 19 | 24 | 3.5% | | | 2024 | 690 | 19 | 24 | 2.8% | 3.5% | |
| | 2025* | 770 | 16 | 20 | 2.6% | | | 2025* | 770 | 16 | 20 | 2.1% | 2.6% | |
| | 2026* | 415 | 8 | 12 | 2.9% | | | 2026* | 415 | 8 | 12 | 1.9% | 2.9% | |
| |
| ^ Venue ^ ''UNION'' ^ ''POLICY'' ^ venue's corpus slice ^ share of venue ^ | ^ Venue ^ ''UNION'' ^ ''POLICY'' ^ venue's corpus slice ^ ''POLICY'' share of venue ^ ''UNION'' share of venue ^ |
| | PETS | 54 | 43 | 510 | **10.6%** | | | PETS | 54 | 43 | 510 | **8.4%** | **10.6%** | |
| | USENIX Sec | 27 | 25 | 1,410 | 1.9% | | | USENIX Sec | 27 | 25 | 1,410 | 1.8% | 1.9% | |
| | CCS | 12 | 8 | 990 | 1.2% | | | CCS | 12 | 8 | 990 | 0.8% | 1.2% | |
| | NDSS | 9 | 8 | 701 | 1.3% | | | NDSS | 9 | 8 | 701 | 1.1% | 1.3% | |
| | TheWebConf | 9 | 7 | 843 | 1.1% | | | TheWebConf | 9 | 7 | 843 | 0.8% | 1.1% | |
| | IEEE S&P | 7 | 6 | 767 | 0.9% | | | IEEE S&P | 7 | 6 | 767 | 0.8% | 0.9% | |
| | IMC | 5 | 5 | 638 | 0.8% | | | IMC | 5 | 5 | 638 | 0.8% | 0.8% | |
| |
| **PETS is where this literature lives**, by a factor of five over the next venue. If you are writing a policy paper, that is your target and its reviewers are the ones whose expectations you should meet. | **PETS is where this literature lives** — 4.8× the next venue on the ''POLICY'' share, 5.5× on the ''UNION'' share. If you are writing a policy paper, that is your target and its reviewers are the ones whose expectations you should meet. |
| |
| ==== Where this literature differs from the corpus, tested ==== | ==== Where this literature differs from the corpus, tested ==== |
| |
| * **Every table names its own denominator.** 102 (''POLICY''), 123 (''UNION''), 4,439 (classified anything) and 5,859 (corpus) are four different populations and none of them is interchangeable with another. | * **Every table names its own denominator.** 102 (''POLICY''), 123 (''UNION''), 4,439 (classified anything) and 5,859 (corpus) are four different populations and none of them is interchangeable with another. |
| * **''POLICY'' is an enum and ''UNION'' is not.** The 102 needs no folding and is stable between extraction runs; the 123 depends on a regex I chose and would move if I chose differently. The only figures over the 123 anywhere on this page are the full-text probe table and the two places in the prose that quote its largest row (80 of 123); all three say in the same breath that they are an upper bound on a candidate set. Every //rate// is over 102, 4,439 or 5,859. | * **''POLICY'' is an enum and ''UNION'' is not.** The 102 needs no folding and is stable between extraction runs; the 123 depends on a regex I chose and would move if I chose differently. **Figures denominated on the 123 do appear**, and every one of them is either labelled ''UNION'' in its own column header or says in the same breath that it is a candidate set: the ''UNION'' share columns in the two tables above, the full-text probe table and the sentences that quote its rows, and the availability and open-question sentences that count probe hits. **No //rate// that the page asks you to believe about the field is over the 123**: the method table, the validation table and the significance table are all over the 102-paper enum. |
| * **Probe width decides the claim.** The title probe here is nine alternatives, widened from the four used in the gap analysis that proposed this page, because ''privacy label'', ''data safety'' and ''nutrition label'' are the store-declaration half of the literature and the narrow probe missed them. Both widths are on the provenance page. | * **Probe width decides the claim.** The title probe here is nine alternatives, widened from the four used in the gap analysis that proposed this page, because ''privacy label'', ''data safety'' and ''nutrition label'' are the store-declaration half of the literature and the narrow probe missed them. Both widths are on the provenance page. |
| * **Sentinels are never answers.** ''none-reported'' in the validation table is a real value reported as itself. | * **Sentinels are never answers.** ''none-reported'' in the validation table is a real value reported as itself. |
| * **Papers are counted, never tuples.** The 102 papers carry 179 privacy-policy classification tuples. | * **Papers are counted, never tuples.** The 102 papers carry 179 privacy-policy classification tuples. |
| * **Free-text names are folded** and the **complete unmapped residue** — 138 distinct source-list strings and 572 distinct tool names — is printed on the provenance page. The tool fold is deliberately narrow: it maps the policy-analysis lineage and the NLP substrate, and leaves general-purpose tools unmapped rather than guessing. | * **Free-text names are folded** and the **complete unmapped residue** — 138 distinct source-list strings and 572 distinct tool names — is printed on the provenance page. The tool fold is deliberately narrow: it maps the policy-analysis lineage and the NLP substrate, and leaves general-purpose tools unmapped rather than guessing. |
| * **Every measured figure above was checked against the paper's own text**, not against the extraction's evidence quote — 68 needles, 68 located, 0 missing. 67 were found in ''paper.cols.txt''; **one** ({[amos2021privacy]}'s //2009–2019// analysis window) is spliced by the de-columning and was located only in a ''pypdf'' rendering of the PDF. That is why the check runs four renderings rather than one: a ''.cols''-only check would have reported a false MISS. The full check and its unedited output are on the provenance page. | * **Every measured figure above was checked against the paper's own text**, not against the extraction's evidence quote — 70 needles, 70 located, 0 missing. 69 were found in ''paper.cols.txt''; **one** ({[amos2021privacy]}'s //2009–2019// analysis window) is spliced by the de-columning and was located only in a ''pypdf'' rendering of the PDF. That is why the check runs four renderings rather than one: a ''.cols''-only check would have reported a false MISS. The full check and its unedited output are on the provenance page. |
| * **2026 is provisional and 2025 is thin at the edges**; see the note under the method table. | * **2026 is provisional and 2025 is thin at the edges**; see the note under the method table. |
| * **Corpus-wide caveats**, including which venues are absent and the extraction's own error rates, are on [[Literature:Corpus]]. | * **Corpus-wide caveats**, including which venues are absent and the extraction's own error rates, are on [[Literature:Corpus]]. |
| A policy-measurement methods section that survives PETS review needs: | A policy-measurement methods section that survives PETS review needs: |
| |
| - **How you found the policy.** The seed phrases or link-text patterns, whether you followed redirects, how many hops, and whether you fell back to a site search. One paper in 123 publishes its seed list. Publish yours. | - **How you found the policy.** The seed phrases or link-text patterns, whether you followed redirects, how many hops, and whether you fell back to a site search. A probe for a published seed-phrase list matches **one** paper of the 123 — and there the probe is likely to be close to right, because a paper that publishes a seed list says so. Publish yours. |
| - **The retrieval funnel, numerically.** Sampled → link found → link resolved → document parsed → language accepted → length threshold passed → analysed. Every later percentage over the last number. {[pan2024_trap]} is the model here: 15.7% no link, 37.5% dead links, 20.5% non-English, 22.3% under threshold, all stated. | - **The retrieval funnel, numerically.** Sampled → link found → link resolved → document parsed → language accepted → length threshold passed → analysed. Every later percentage over the last number. {[pan2024_trap]} is the model here: 15.7% no link, 37.5% dead links, 20.5% non-English, 22.3% under threshold, all stated. |
| - **Which policy you analysed** when there is more than one. App policy or developer's generic corporate policy? Platform policy or vendor policy? Layered notice or the full document? Only 27 of 123 papers address this at all, and it silently changes results. | - **Which policy you analysed** when there is more than one. App policy or developer's generic corporate policy? Platform policy or vendor policy? Layered notice or the full document? A probe for "which policy applies" language matches 27 of the 123, so at most about a fifth address it; it silently changes results. |
| - **The date and the version.** A policy is a moving target. Give the retrieval date; if you compare across time, say how you matched versions and whether you used live crawls or archived snapshots ([[Design:Archives]], [[Design:Longitudinal]]). | - **The date and the version.** A policy is a moving target. Give the retrieval date; if you compare across time, say how you matched versions and whether you used live crawls or archived snapshots ([[Design:Archives]], [[Design:Longitudinal]]). |
| - **Language handling.** How you detected language, what you did with non-English documents, and whether you translated — machine translation changes what a sentence classifier sees ([[Programming:Multilingual support]]). | - **Language handling.** How you detected language, what you did with non-English documents, and whether you translated — machine translation changes what a sentence classifier sees ([[Programming:Multilingual support]]). |
| * **How good is LLM extraction, really?** Eleven of the 24 papers in 2025–2026 use one, and we found no head-to-head evaluation in these seven venues of an LLM against PoliGraph and against a PrivBERT baseline on the same documents with the same ground truth. Until someone runs that, the switch is a preference, not a finding. | * **How good is LLM extraction, really?** Eleven of the 24 papers in 2025–2026 use one, and we found no head-to-head evaluation in these seven venues of an LLM against PoliGraph and against a PrivBERT baseline on the same documents with the same ground truth. Until someone runs that, the switch is a preference, not a finding. |
| * **The retrieval step has never been measured as a step.** Every study reports the availability rate its own heuristic produced. Nobody has taken one site sample and compared five link-detection heuristics, so the field does not know how much of the 4.35%-to-94.2% spread in the availability table is ecosystem and how much is method. | * **The retrieval step has never been measured as a step.** Every study reports the availability rate its own heuristic produced. Nobody has taken one site sample and compared five link-detection heuristics, so the field does not know how much of the 4.35%-to-94.2% spread in the availability table is ecosystem and how much is method. |
| * **Non-English policy analysis is 30 papers and mostly bilingual.** {[hosseini2024_bilingual]} is English–German; the 2026 Swiss study adds a third. There is no multilingual policy corpus and no evaluation of whether OPP-115-derived label sets transfer to a language other than English at all. | * **Non-English policy analysis is thin, and mostly bilingual.** A probe for multi-language analysis matches 30 of the 123 candidate papers, but its printed list includes studies whose multilingual element is incidental (a ''robots.txt'' governance paper, a shopping-terms paper, a visual-notice user study), so the real number is smaller. {[hosseini2024_bilingual]} is English–German; {[nenadic2026_swiss]} adds a third language. There is no multilingual policy corpus, and we found no evaluation of whether OPP-115-derived label sets transfer to any language other than English. |
| * **Nobody has connected policy text to enforcement outcomes at scale.** 70 of the 123 papers assess a law, but we found none in these seven venues that tests whether a policy defect a tool flags predicts a regulator's decision. That is the missing external validity check for the whole compliance literature, and the material exists ([[Practices:Legal enforcement]]). | * **Nobody has connected policy text to enforcement outcomes at scale.** 64 of the 102 ''POLICY'' papers assess a law, but we found none in these seven venues that tests whether a policy defect a tool flags predicts a regulator's decision. That is the missing external validity check for the whole compliance literature, and the material exists ([[Practices:Legal enforcement]]). |
| * **Terms of service are almost unstudied here.** The probe finds one dedicated study in these seven venues, {[tsai2025_harmful]} on unfavourable financial terms in shopping sites. Terms are longer, more consequential and less machine-readable than privacy policies, and there is no OPP-115 for them. | * **Terms of service are almost unstudied here.** No script on this page probes for terms-of-service work specifically; reading the 123 candidate titles, the one study squarely on it is {[tsai2025_harmful]}, on unfavourable financial terms in shopping sites. That is a judgement from reading, not a count. Terms are longer, more consequential and less machine-readable than privacy policies, and there is no OPP-115 for them. |
| </WRAP> | </WRAP> |
| |
