This is an old revision of the document!
Table of Contents
Measuring Privacy Policies and Terms
A privacy policy is the only place where a service says, in its own words, what it does with your data. That makes it the natural control for every behavioural measurement on this wiki: you crawled the site and saw a tracker fire, and now you want to know whether the site said it would. Measuring policies is how you close that loop.
It is also a text-collection problem dressed up as a privacy problem, and almost everything that goes wrong goes wrong in the collection, not in the analysis. Four things break a web-trained intuition:
- The document has no fixed address. There is no
/.well-known/privacy-policy. You find it by guessing at link text in a footer, and every study's denominator is really “the sites where my link heuristic worked”. - “Has a privacy policy” is not a property of the web; it is a property of the ecosystem you sampled. It runs from 94.2% of websites with a data-collecting form [1Cui, Hao; Trimananda, Rahmadi; Markopoulou, Athina (2025): "Understanding Privacy Norms through Web Forms", in: Proceedings on Privacy Enhancing Technologies. (DOI)] to 4.35% of Discord chatbots [2Edu, Jide S.; Mulligan, Cliona; Pierazzi, Fabio; Polakis, Jason; Suarez-Tangil, Guillermo; Such, Jose M. (2022): "Exploring the security and privacy risks of chatbots in messaging services", in: Proceedings of the ACM Internet Measurement Conference. (DOI)]. Quoting a rate without naming the ecosystem says nothing.
- The interesting claim is never “the policy says X”. It is “the policy says X and the traffic does Y”. A full-text probe finds comparison-against-observed-behaviour language in 80 of the 123 candidate papers — an upper bound on a candidate set rather than a measured rate, but at that magnitude the shape is unmistakable: the comparison is the measurement, and the policy is only one of its two arms.
- Half of this literature is about Android apps, not websites. Of the 123 corpus papers here, 43 measure only mobile, 39 only web and 17 both — and 24 measure neither, because a policy study can be about a VR store, an IoT companion app, a voice assistant or a chatbot ecosystem. The methods are genuinely different — an app policy is reached through a store listing, an app has a store-declared privacy label to contradict as well as a policy, and the tool lineage below was built for Android.
Scope. This page is about policies, terms and store-declared privacy labels as text you collect and label. It is not about the banner: what a consent notice says and what happens when you click it is Consent. A cookie banner is a UI measurement; a privacy policy is a document-retrieval and NLP measurement, and the two literatures barely cite each other.
Everything below about “the literature” is a claim about seven venues — CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026, 5,859 papers with extracted full text. Policy analysis also publishes heavily at ACL, EMNLP, CHI, SOUPS and in law reviews, none of which are in the corpus. Two of the field's foundation stones — the OPP-115 corpus (ACL 2016) and PrivaSeer (ACL 2021) — are outside it, and PrivaSeer is named by zero corpus papers. So read the counts as “how these seven security and privacy venues do policy analysis”, not as the state of the field. See Corpus.
What to Read First
- [3Mhaidli, Abraham; Fidan, Selin; Doan, An; Herakovic, Gina; Srinath, Mukund; Matheson, Lee; Wilson, Shomir; Schaub, Florian (2023): "Researchers’ Experiences in Analyzing Privacy Policies: Challenges and Opportunities", Proceedings on Privacy Enhancing Technologies 2023:287-305. (DOI) (Link)] — read this before you design anything. Twenty-six interviews with researchers who have analysed policies. Its own discipline breakdown is 15 in computer science (10 of them in ML/NLP), and the rest spread over health, communication, policy, education, business, sociology and political science. Its finding is that “no clear best practices or methodologies for privacy policy analysis have emerged so far”, and it names the three places people get stuck: policy selection, policy retrieval, and content analysis. That is the shape of this page.
- [4Andow, Benjamin; Mahmud, Samin Yaseer; Whitaker, Justin; Enck, William; Reaves, Bradley; Singh, Kapil; Egelman, Serge (2020): "Actions Speak Louder than Words: Entity-Sensitive Privacy Policy and Data Flow Analysis with PoliCheck", in: Proceedings of the USENIX Security Symposium. (Link)] — the policy-versus-behaviour design, done properly. PoliCheck introduced entity-sensitive consistency: it is not enough that the policy mentions “location”, it must mention the recipient that received it. Re-running the analysis entity-insensitively falsely marked up to 37.1% of inconsistent third-party flows as consistent, which is the single best argument for reading this paper before writing your own comparison.
- [5Cui, Hao; Trimananda, Rahmadi; Markopoulou, Athina; Jordan, Scott (2023): "PoliGraph: Automated Privacy Policy Analysis using Knowledge Graphs", in: Proceedings of the USENIX Security Symposium. (Link)] — the current tool, and the current critique. PoliGraph builds a knowledge graph from the policy instead of classifying sentences, and it is what a 2026 study reaches for. Its own finding that 25.5% (1,339/5,255) of policies define a term differently from the CCPA-based ontology is a warning about every ontology-matching pipeline including its own.
- [6Amos, Ryan; Acar, Gunes; Lucherini, Eli; Kshirsagar, Mihir; Narayanan, Arvind; Mayer, Jonathan (2021): "Privacy Policies over Time: Curation and Analysis of a Million-Document Dataset", in: Proceedings of the Web Conference 2021, pp. 2165–2176. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] — the longitudinal corpus and how to build one. 1,071,488 English policies from over 130,000 websites, reconstructed from the Wayback Machine. Read the curation section: it is the most careful published account of turning archived HTML into a policy dataset.
- [7Linden, Thomas; Khandelwal, Rishabh; Harkous, Hamza; Fawaz, Kassem (2020): "The Privacy Policy Landscape After the GDPR", in: Proceedings on Privacy Enhancing Technologies. (DOI)] — the natural experiment. Matched pre/post-GDPR policy pairs, EU and Global. It shows what a before/after policy study looks like when you have controls, and it is honest that longer is not clearer.
Where the Policies Come From
You have to find the document first, and that step is the denominator
Nobody publishes a policy at a predictable URL. Every study locates it the same way: fetch a page, look at the anchors, and decide which link text means “privacy policy”. That heuristic is the study's real inclusion criterion, and it is usually described in one sentence.
What the criterion looks like when it is done carefully: [1Cui, Hao; Trimananda, Rahmadi; Markopoulou, Athina (2025): "Understanding Privacy Norms through Web Forms", in: Proceedings on Privacy Enhancing Technologies. (DOI)] matched link text and URLs against six seed phrases using embedding cosine similarity rather than exact strings, and reports policy links on 94.2% (9,559) of its 10,143 websites. What it looks like when it is not done carefully is a paper that says “we crawled the privacy policy” and reports no failure rate at all.
Three failure modes are worth budgeting for, because they are not symmetric — they all bias towards finding fewer, shorter, worse policies:
- The link exists and does not resolve. [8Pan, Shidong; Zhang, Dawen; Staples, Mark; Xing, Zhenchang; Chen, Jieshan; Xu, Xiwei; Hoang, Thong (2024): "Is It a Trap? A Large-scale Empirical Study And Comprehensive Assessment of Online Automated Privacy Policy Generators for Mobile Apps", in: Proceedings of the USENIX Security Symposium. (Link)] started from 99,194 usable Play apps and found 37.5% (37,150/99,194) of policy links led to an unavailable page. Note the denominator: the paper divides link failures by its app count, not by the apps that actually have a link (15.7% have none) — which is itself a lesson about reporting the funnel. If you count “has a link” as “has a policy” you will overstate availability by about a third.
- The document is not the policy you wanted. [8Pan, Shidong; Zhang, Dawen; Staples, Mark; Xing, Zhenchang; Chen, Jieshan; Xu, Xiwei; Hoang, Thong (2024): "Is It a Trap? A Large-scale Empirical Study And Comprehensive Assessment of Online Automated Privacy Policy Generators for Mobile Apps", in: Proceedings of the USENIX Security Symposium. (Link)] also found 22.3% (10,375/46,472) of the policies it did retrieve were under 2 KB or 200 words — placeholders, error pages, and cookie notices standing in for a policy.
- The document is not in a language you handle. 20.5% (9,523/46,472) of the same set had no English version. A pipeline that silently drops non-English documents has silently changed its population; see Multilingual support.
Only 7 of the 123 corpus papers say anything at all about dead or unresolvable policy links, and only 9 mention handling a policy served as a PDF.1) Report yours.
"How many have a policy" depends entirely on what you sampled
This is the table to look at before you quote an availability figure. Every row is a different ecosystem and a different denominator, and the spread is more than an order of magnitude.
| Ecosystem | Denominator, as the paper states it | Have a policy | Source |
|---|---|---|---|
| Websites in the EU | 6,357, the total of the paper's own availability table — the same paper also says 6,759 (its January domain lists) and 6,579 (its abstract) | 84.5% after 25 May 2018, up from 79.6% in January | [9Degeling, Martin; Utz, Christine; Lentzsch, Christopher; Hosseini, Henry; Schaub, Florian; Holz, Thorsten (2019): "We Value Your Privacy ... Now Take Some Cookies: Measuring the GDPR's Impact on Web Privacy", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] |
| Websites with a PI-collecting web form | 10,143 websites | 94.2% (9,559) | [1Cui, Hao; Trimananda, Rahmadi; Markopoulou, Athina (2025): "Understanding Privacy Norms through Web Forms", in: Proceedings on Privacy Enhancing Technologies. (DOI)] |
| Adult websites | 6,843 pornographic websites | 16% | [10Vallina, Pelayo; Feal, Álvaro; Gamba, Julien; Vallina-Rodriguez, Narseo; Anta, Antonio Fernández (2019): "Tales from the Porn: A Comprehensive Privacy Analysis of the Web Porn Ecosystem", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] |
| Android apps | 1,035,853 analysed apps (of 1,049,790 retrieved) | 50.5% had a Play Store policy link | [11Zimmeck, Sebastian; Story, Peter; Smullen, Daniel; Ravichander, Abhilasha; Wang, Ziqi; Reidenberg, Joel; Russell, N. Cameron; Sadeh, Norman (2019): "MAPS: Scaling Privacy Compliance Analysis to a Million Apps", Proceedings on Privacy Enhancing Technologies 2019(3). (DOI)] |
| Android apps (link resolves) | 99,194 usable apps (the paper's own denominator for this rate) | 37.5% (37,150) of policy links did not resolve | [8Pan, Shidong; Zhang, Dawen; Staples, Mark; Xing, Zhenchang; Chen, Jieshan; Xu, Xiwei; Hoang, Thong (2024): "Is It a Trap? A Large-scale Empirical Study And Comprehensive Assessment of Online Automated Privacy Policy Generators for Mobile Apps", in: Proceedings of the USENIX Security Symposium. (Link)] |
| Smart-home vendors | 596 vendors across 7 platforms | 48.99% had a device-applicable policy; 10.57% had none at all | [12Manandhar, Sunil; Kafle, Kaushal; Andow, Benjamin; Singh, Kapil; Nadkarni, Adwait (2022): "Smart Home Privacy Policies Demystified: A Study of Availability, Content, and Coverage", in: Proceedings of the USENIX Security Symposium. (Link)] |
| Alexa skills | 150,708 skills, 7 country stores | 24.2% (36,475) | [13Lentzsch, Christopher; Shah, Sheel Jayesh; Andow, Benjamin; Degeling, Martin; Das, Anupam; Enck, William (2021): "Hey Alexa, is this Skill Safe?: Taking a Closer Look at the Alexa Skill Ecosystem", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] |
| Alexa skills, later | 65,195 skills | 21,063 | [14Yan, Chuan; Xie, Fuman; Meng, Mark Huasong; Zhang, Yanjun; Bai, Guangdong (2024): "On the Quality of Privacy Policy Documents of Virtual Personal Assistant Applications", in: Proceedings on Privacy Enhancing Technologies. (DOI)] |
| VR apps | 11,923 apps on 10 VR platforms | 29.5% | [15Zhan, Yuxia; Meng, Yan; Zhou, Lu; Xiong, Yichang; Zhang, Xiaokuan; Ma, Lichuan; Chen, Guoxing; Pei, Qingqi; Zhu, Haojin (2024): "VPVet: Vetting Privacy Policies of Virtual Reality Apps", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)] |
| Discord chatbots requesting permissions | 15,528 chatbots | 4.35% (676) — 14,852 had none | [2Edu, Jide S.; Mulligan, Cliona; Pierazzi, Fabio; Polakis, Jason; Suarez-Tangil, Guillermo; Such, Jose M. (2022): "Exploring the security and privacy risks of chatbots in messaging services", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] |
| GPT Actions declaring a policy URL | Actions with a legal_info_url | 93.96% of those URLs were reachable | [16Wu, Yuhao; Jaff, Evin; Yang, Ke; Zhang, Ning; Iqbal, Umar (2025): "An In-Depth Investigation of Data Collection in LLM App Ecosystems", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] |
Two things follow. First, the web is the best-covered ecosystem in this table and every app-adjacent ecosystem is worse, which is the opposite of what a web-trained reader expects. Second, the last row is not comparable to the others: it is availability conditional on the developer having declared a URL, which is a different question. Watch for that conditioning; it is the most common way an availability figure gets inflated.
Existing corpora, so you do not build another one
Building a policy corpus is a month of work and three of the four corpora below are still downloadable. Check them before you crawl.
| Corpus | What it is | Size | Where, and status as of 2026-09-10 |
|---|---|---|---|
| OPP-115 | The reference annotated corpus. Each policy read and annotated by three law graduate students; the label set that Polisis, PrivBERT and most supervised work below is trained on | 115 website policies | usableprivacy.org/data, live. CC BY-NC for research; a separate commercial licence exists via CMU Flintbox. 14 corpus papers use it |
| APP-350 | The mobile-app counterpart, annotated with app privacy practices | 350 Android app policies | Same host, live, same research licence — but unlike OPP-115 it documents no commercial-licence path, only “contact the author”. 1 corpus paper uses it |
| PrivaSeer | A crawled policy corpus and search engine, plus the PrivBERT language model pretrained on it | 3,967,487 policies | privaseer.ist.psu.edu, live; corpus on PSU GitLab; model mukund/privbert on Hugging Face (CC BY-NC-SA). Named by zero corpus papers, though 3 use PrivBERT |
| Princeton–Leuven longitudinal corpus | Policies reconstructed from Wayback snapshots, 1997–2019, for before/after work | 1,071,488 English policies, >130,000 sites | Data: github.com/citp/privacy-policy-historical; code: github.com/citp/PrivacyPoliciesOverTime. Both alive, neither archived, but last commits 2023-10-12 and 2022-06-06 and no licence file on either [6Amos, Ryan; Acar, Gunes; Lucherini, Eli; Kshirsagar, Mihir; Narayanan, Arvind; Mayer, Jonathan (2021): "Privacy Policies over Time: Curation and Analysis of a Million-Document Dataset", in: Proceedings of the Web Conference 2021, pp. 2165–2176. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] |
The gap in that table is the point: there is no maintained, versioned, citable policy corpus that reaches 2026. Everything above either stops in 2019, or is a one-off crawl, or is 115 documents. See Open Questions.
The Tool Lineage, With Dates
A fresh student searching for “privacy policy analysis tool” lands on Polisis (2018) because it has the most citations and a live demo site. It is eight years old, its label set is OPP-115's, and there is no official code repository. The third-party reproductions a name search returns are not maintained either — the two most-starred last moved on their default branches in February and July 2020, and the one whose GitHub activity date reads 2023 turns out to be a Dependabot security bump on a side branch. Do not start there.
Counts below are papers in the whole 5,859-paper corpus whose extraction names the artefact anywhere (tool name, purpose, classifier resource, source list, or detection technique). Repository facts were fetched on 2026-09-10 from GitHub's commits API on each repository's own default branch — not from pushed_at, which counts activity on any branch, and not from /releases/latest or /tags, which are unreliable for repositories like these. The script and its unedited output are on the provenance page.
| Artefact | First paper | Corpus papers | Used 2024+ | What it does, and whether to start there |
|---|---|---|---|---|
| Privee | USENIX 2014 | 1 | 0 | Regex rules plus naive Bayes. Historical interest only |
| OPP-115 | ACL 2016 (outside corpus) | 14 | 6 | The label set, not a tool. Still the substrate under most supervised work |
| Polisis / PriBot | USENIX 2018 [17Harkous, Hamza; Fawaz, Kassem; Lebret, Rémi; Schaub, Florian; Shin, Kang G.; Aberer, Karl (2018): "Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learning", in: Proceedings of the USENIX Security Symposium. (Link)] | 13 | 5 | Hierarchical CNNs over fastText embeddings, trained on 65 OPP-115 policies with 50 held out; average F1 0.84. Demo at pribot.org/polisis alive (it redirects HTTPS to plain HTTP), no official repo |
| PolicyLint | USENIX 2019 [18Andow, Benjamin; Mahmud, Samin Yaseer; Wang, Wenyu; Whitaker, Justin; Enck, William; Reaves, Bradley; Singh, Kapil; Xie, Tao (2019): "PolicyLint: Investigating Internal Privacy Policy Contradictions on Google Play", in: Proceedings of the USENIX Security Symposium. (Link)] | 18 | 3 | Ontology-based extraction of collection/sharing statements from single sentences; finds internal contradictions. 14.2% (1,618/11,430) of Play policies contradict themselves. Code at github.com/benandow/PrivacyPolicyAnalysis — searching GitHub for “PolicyLint” or “PoliCheck” does not find it; last commit 2022-10-05; LICENSE.txt is a three-clause BSD naming NC State, which GitHub's detector labels “Other” |
| MAPS | PETS 2019 [11Zimmeck, Sebastian; Story, Peter; Smullen, Daniel; Ravichander, Abhilasha; Wang, Ziqi; Reidenberg, Joel; Russell, N. Cameron; Sadeh, Norman (2019): "MAPS: Scaling Privacy Compliance Analysis to a Million Apps", Proceedings on Privacy Enhancing Technologies 2019(3). (DOI)] | 2 | 1 | The million-app compliance pipeline. Its policy classifier is the thing people reuse |
| PoliCheck | USENIX 2020 [4Andow, Benjamin; Mahmud, Samin Yaseer; Whitaker, Justin; Enck, William; Reaves, Bradley; Singh, Kapil; Egelman, Serge (2020): "Actions Speak Louder than Words: Entity-Sensitive Privacy Policy and Data Flow Analysis with PoliCheck", in: Proceedings of the USENIX Security Symposium. (Link)] | 9 | 4 | The entity-sensitive consistency check: matches an observed data flow against what the policy says about that recipient. Same repository as PolicyLint, and equally unfindable by name |
| PurPliance | 2021 | 5 | 0 | Adds purpose to the consistency check. github.com/ducalpha/PurPlianceOpenSource, BSD-style (U. Michigan 2022); last code commit 2022-12-25, last commit of any kind 2024-03-11 — the most recently touched repository in this table |
| PrivBERT | 2021 (PrivaSeer) | 3 | 3 | RoBERTa pretrained on ~1M policies. The drop-in sentence classifier |
| PoliGraph / poligraph-er | USENIX 2023 [5Cui, Hao; Trimananda, Rahmadi; Markopoulou, Athina; Jordan, Scott (2023): "PoliGraph: Automated Privacy Policy Analysis using Knowledge Graphs", in: Proceedings of the USENIX Security Symposium. (Link)] | 6 | 5 | Knowledge graph over the whole document instead of sentence classification; 70.6% recall at 96.9% precision on collection tuples. MIT, last commit 2023-06-21, no PyPI package under either name; its README says a GPU is needed for hardware acceleration but that it runs without one, “significantly lower” performance |
| PolicyChecker | CCS 2023 [19Xiang, Anhao; Pei, Weiping; Yue, Chuan (2023): "PolicyChecker: Analyzing the GDPR Completeness of Mobile Apps' Privacy Policies", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)] | 1 | 0 | GDPR completeness rather than consistency. github.com/AndyXiang945/PolicyChecker, last commit 2023-11-20, no licence file |
| PolicyComp | USENIX 2023 [20Zhou, Lu; Wei, Chengyongxiao; Zhu, Tong; Chen, Guoxing; Zhang, Xiaokuan; Du, Suguo; Cao, Hui; Zhu, Haojin (2023): "POLICYCOMP: Counterpart Comparison of Privacy Policies Uncovers Overbroad Personal Data Collection Practices", in: Proceedings of the USENIX Security Symposium. (Link)] | 1 | 0 | Compares a policy against its competitors' policies to find overbroad collection |
| Lalaine | USENIX 2023 [21Xiao, Yue; Li, Zhengyi; Qin, Yue; Bai, Xiaolong; Guan, Jiale; Liao, Xiaojing; Xing, Luyi (2023): "Lalaine: Measuring and Characterizing Non-Compliance of Apple Privacy Labels", in: Proceedings of the USENIX Security Symposium. (Link)] | 2 | 1 | Apple privacy-label compliance specifically. github.com/xiaoyue10131748/Lalaine, MIT, last commit 2023-09-25 |
| Calpric | USENIX 2023 [22Qiu, Wenjun; Lie, David; Austin, Lisa (2023): "Calpric: Inclusive and Fine-grain Labeling of Privacy Policies with Crowdsourcing and Active Learning", in: Proceedings of the USENIX Security Symposium. (Link)] | 1 | 0 | Crowdsourcing + active learning to get 16,856 labelled segments cheaply. github.com/dlgroupuoft/Calpric, last commit 2023-06-21, no licence file, artifact “Available” only |
| PrivaSeer | ACL 2021 (outside corpus) | 0 | 0 | Corpus + search engine. Nobody in these seven venues cites it |
What this table is really saying. Not one repository in this table has a commit later than March 2024, and that one (PurPliance) is a README edit — its last code commit is December 2022. PoliGraph, the newest tool, has not moved since June 2023. Finding the code is its own problem: a GitHub name search for PolicyLint or PoliCheck returns only unrelated projects, because the repository is named after the paper series and not after either tool. Calpric and Lalaine are a milder version of the same: a plain web search on the tool name surfaces the paper and no repository, and you get to the code either through GitHub's own repository search or through the USENIX artifact-evaluation index, which links both directly.2) Budget for reading and porting code, not for running it.
Policy Versus Behaviour: the Actual Measurement
A full-text probe finds policy-versus-behaviour language in 80 of the 123 candidate papers — the largest of the thirteen probes below, and, like all of them, an upper bound on a candidate set rather than a rate over a population. Read it as: comparing a policy against something observed (traffic, static code, storage, or a store declaration) is what most of this literature is doing, and pure text studies are the minority.
The design has three parts and every one of them is a place to go wrong.
- The behavioural arm. Whatever you already measure — network flows (Traffic files), static taint (Mobile and app measurement), cookies (Cookies). Its false positives become “undisclosed collection” findings.
- The policy arm. Extract statements of the form (actor, action, data type, recipient, purpose). This is what PolicyLint → PoliCheck → PoliGraph progressively got better at.
- The matching rule. The hard part, and the part reviewers attack. “Location” in the policy and
GPSin the traffic are the same thing; “partners” anddoubleclick.netmight be. Every tool encodes an ontology to make that call, and the ontology is a research contribution in its own right, not an implementation detail.
| Study | Denominator, as stated | Finding |
|---|---|---|
| [23Zimmeck, Sebastian; Wang, Ziqi; Zou, Lieyong; Iyengar, Roger; Liu, Bin; Schaub, Florian; Wilson, Shomir; Sadeh, Norman; Bellovin, Steven M.; Reidenberg, Joel (2017): "Automated Analysis of Privacy Requirements for Mobile Apps", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] NDSS 2017 | 9,050 apps with policies | mean 1.83 potential inconsistencies per app |
| [24Libert, Timothy (2018): "An Automated Approach to Auditing Disclosure of Third-Party Data Collection in Website Privacy Policies", in: Proceedings of the ACM Web Conference. (DOI)] WWW 2018 | 1,807,491 third-party transmissions | only 14.80% were disclosed |
| [18Andow, Benjamin; Mahmud, Samin Yaseer; Wang, Wenyu; Whitaker, Justin; Enck, William; Reaves, Bradley; Singh, Kapil; Xie, Tao (2019): "PolicyLint: Investigating Internal Privacy Policy Contradictions on Google Play", in: Proceedings of the USENIX Security Symposium. (Link)] USENIX 2019 | 11,430 Play policies | 14.2% logically contradict themselves; 17.7% including narrowing definitions |
| [4Andow, Benjamin; Mahmud, Samin Yaseer; Whitaker, Justin; Enck, William; Reaves, Bradley; Singh, Kapil; Egelman, Serge (2020): "Actions Speak Louder than Words: Entity-Sensitive Privacy Policy and Data Flow Analysis with PoliCheck", in: Proceedings of the USENIX Security Symposium. (Link)] USENIX 2020 | 13,796 apps / 45,603 flows | 42.4% of apps had an omitted or incorrect disclosure; 31.1% of flows omitted; only 0.5% clearly disclosed |
| [25Trimananda, Rahmadi; Le, Hieu; Cui, Hao; Ho, Janice Tran; Shuba, Anastasia; Markopoulou, Athina (2022): "OVRseen: Auditing Network Traffic and Privacy Policies in Oculus VR", in: Proceedings of the USENIX Security Symposium. (Link)] USENIX 2022 | 1,135 flows, Oculus VR apps | 68% (776) inconsistent |
| [26Samarin, Nikita; Kothari, Shayna; Siyed, Zaina; Bjorkman, Oscar; Yuan, Reena; Wijesekera, Primal; Alomar, Noura; Fischer, Jordan; Hoofnagle, Chris; Egelman, Serge (2023): "Lessons in VCR Repair: Compliance of Android App Developers with the California Consumer Privacy Act (CCPA)", in: Proceedings on Privacy Enhancing Technologies. (DOI)] PETS 2023 | 69 apps with CCPA disclosures | 80% (55 apps) collected an identifier they did not disclose |
| [21Xiao, Yue; Li, Zhengyi; Qin, Yue; Bai, Xiaolong; Guan, Jiale; Liao, Xiaojing; Xing, Luyi (2023): "Lalaine: Measuring and Characterizing Non-Compliance of Apple Privacy Labels", in: Proceedings of the USENIX Security Symposium. (Link)] USENIX 2023 | 5,102 fully tested iOS apps | 3,423 had a non-compliant privacy label |
| [19Xiang, Anhao; Pei, Weiping; Yue, Chuan (2023): "PolicyChecker: Analyzing the GDPR Completeness of Mobile Apps' Privacy Policies", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)] CCS 2023 | 163,068 analysable policies | 99.3% incomplete under the GDPR; 98.1% violate a mandatory requirement |
| [27Ali, Mir Masood; Balash, David G.; Kodwani, Monica; Kanich, Chris; Aviv, Adam J. (2024): "Honesty is the Best Policy: On the Accuracy of Apple Privacy Labels Compared to Apps' Privacy Policies", in: Proceedings on Privacy Enhancing Technologies. (DOI)] PETS 2024 | iOS apps labelled “Data Not Collected” | 97% had policy text indicating they do collect data |
| [1Cui, Hao; Trimananda, Rahmadi; Markopoulou, Athina (2025): "Understanding Privacy Norms through Web Forms", in: Proceedings on Privacy Enhancing Technologies. (DOI)] PETS 2025 | sites collecting PI via web forms | policy disclosure and observed collection associate at phi < 0.20 for every data type |
Notice the ceiling effect. Once a compliance criterion is strict enough, essentially everything fails it — 99.3%, 98.1%, 97%. A finding of that shape is a statement about your criterion at least as much as about the population, and a reviewer will say so. The papers that survive that objection are the ones that report a graded result, validate the criterion against hand-labelled ground truth, and say what a passing policy looks like.
Store-declared labels are a second, cheaper arm
Since 2020–2022 both mobile stores make the developer declare data practices in a structured form, which gives you a machine-readable disclosure to compare against without any NLP at all. Both are current and both are enforced:
- Apple App Privacy labels, mandatory for new and updated apps since December 2020. Since 1 May 2024 Apple additionally requires a privacy manifest (
PrivacyInfo.xcprivacy) and a valid signature from listed third-party SDKs, declaring “required reasons” for certain APIs; Apple's own announcement says the requirement “will expand to include the entire app binary”.3) - Google Play Data safety, mandatory since 20 July 2022. Separately, Play's User Data policy requires a privacy policy from every app: “Apps that do not access any personal and sensitive user data must still submit a privacy policy.”4)
The label is not the policy and the two disagree: [27Ali, Mir Masood; Balash, David G.; Kodwani, Monica; Kanich, Chris; Aviv, Adam J. (2024): "Honesty is the Best Policy: On the Accuracy of Apple Privacy Labels Compared to Apps' Privacy Policies", in: Proceedings on Privacy Enhancing Technologies. (DOI)] found 228,539 apps reporting linked-data practices in their policy but not their label. If your study covers mobile, collect both — comparing them is a measurement in itself, and it is where 2026 work is going: the PETS 2026 Disclosure Divergence: Measuring Privacy Policy and Data Safety Misalignment at Scale does exactly this, and the PETS 2026 study of why developers do not update Apple's privacy labels attacks the same gap from the developer side.
What the Policy Text Itself Looks Like
The descriptive literature is smaller and largely settled. It is worth knowing so you do not re-measure it:
- Policies got longer after the GDPR, not clearer. [7Linden, Thomas; Khandelwal, Rishabh; Harkous, Hamza; Fawaz, Kassem (2020): "The Privacy Policy Landscape After the GDPR", in: Proceedings on Privacy Enhancing Technologies. (DOI)]: EU policies gained +35% words and +33% sentences, Global +25% and +22%, with no significant change in sentence structure.
- Length and reading grade rose steadily for two decades. [6Amos, Ryan; Acar, Gunes; Lucherini, Eli; Kshirsagar, Mihir; Narayanan, Arvind; Mayer, Jonathan (2021): "Privacy Policies over Time: Curation and Analysis of a Million-Document Dataset", in: Proceedings of the Web Conference 2021, pp. 2165–2176. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)]: median length 876 → 1,522 words between 2009 and 2019; median Flesch–Kincaid grade 11.9 → 13.2. [24Libert, Timothy (2018): "An Automated Approach to Auditing Disclosure of Third-Party Data Collection in Website Privacy Policies", in: Proceedings of the ACM Web Conference. (DOI)] put the cost at 84.7 minutes to read all first- and third-party policies applicable to an average site.
- Policies are heavily templated. 65% (306,404) of iOS apps' policies are template-like by sentence similarity [27Ali, Mir Masood; Balash, David G.; Kodwani, Monica; Kanich, Chris; Aviv, Adam J. (2024): "Honesty is the Best Policy: On the Accuracy of Apple Privacy Labels Compared to Apps' Privacy Policies", in: Proceedings on Privacy Enhancing Technologies. (DOI)]; 54.5% (1,919/3,521) of VR apps reuse a policy document outright [15Zhan, Yuxia; Meng, Yan; Zhou, Lu; Xiong, Yichang; Zhang, Xiaokuan; Ma, Lichuan; Chen, Guoxing; Pei, Qingqi; Zhu, Haojin (2024): "VPVet: Vetting Privacy Policies of Virtual Reality Apps", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)]. Deduplicate before you report a distribution, or you are measuring the template vendor.
- The policy under-describes the technology. [6Amos, Ryan; Acar, Gunes; Lucherini, Eli; Kshirsagar, Mihir; Narayanan, Arvind; Mayer, Jonathan (2021): "Privacy Policies over Time: Curation and Analysis of a Million-Document Dataset", in: Proceedings of the Web Conference 2021, pp. 2165–2176. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)]: 25.8% of 2019 policies mention web-beacon terms against 94.6% of top-10K sites observed using them.
- Semantic change is concentrated, not diffuse. [28Adhikari, Andrick; Das, Sanchari; Dewri, Rinku (2023): "Evolution of Composition, Readability, and Structure of Privacy Policies over Two Decades", in: Proceedings on Privacy Enhancing Technologies. (DOI)] found the User Choice/Control category changed semantically in 26% of policies on average across two decades — most sections do not move.
Readability metrics are the one place this page will contradict a textbook: Flesch and Flesch–Kincaid are validated on prose for schoolchildren, not on legal documents with defined terms, and 17 of the 123 papers use one anyway. Report the raw length distribution alongside any readability score, because the score's absolute value means less than its trend.
Which Methods Are Current
Read as of 2026-09. The population is the 102 corpus papers whose extraction records them classifying privacy-policy text — a structured enum, not a keyword match, so this table needs no folding. It is multi-valued: a paper using two methods appears in two rows.
classification.method | all 102 | 2014–2018 (n=7) | 2019–2021 (n=21) | 2022–2024 (n=50) | 2025–2026* (n=24) |
|---|---|---|---|---|---|
| manual-labelling | 43 | 57.1% | 42.9% | 38.0% | 45.8% |
| heuristic-rules | 33 | 42.9% | 28.6% | 40.0% | 16.7% |
| supervised-ml | 26 | 42.9% | 38.1% | 24.0% | 12.5% |
| third-party-service | 13 | 0% | 14.3% | 14.0% | 12.5% |
| llm | 12 | 0% | 0% | 2.0% | 45.8% |
| regex-or-signature | 9 | 14.3% | 9.5% | 6.0% | 12.5% |
other | 8 | 0% | 9.5% | 8.0% | 8.3% |
| static-analysis | 4 | 0% | 4.8% | 4.0% | 4.2% |
| unsupervised-ml | 3 | 0% | 0% | 6.0% | 0% |
| graph-analysis | 2 | 0% | 0% | 4.0% | 0% |
curated-database | 1 | 0% | 4.8% | 0% | 0% |
* 2025–2026 is provisional: CCS 2026 and IMC 2026 have not been held, and IEEE S&P 2026 and TheWebConf 2026 are under-selected by construction. The whole last column rests on 24 papers.
| Practice | Status | Evidence |
|---|---|---|
| LLM extraction and annotation of policy text | Current, and where this field went. | The lone 2022–2024 paper is [29Huang, Ziyuan; Tang, Jiaming; Karir, Manish; Liu, Mingyan; Sarabi, Armin (2024): "Analyzing Corporate Privacy Policies using AI Chatbots", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] (GPT-4, IMC 2024); 11 of 24 papers in 2025–2026 use one, 9 of them at PETS. [30Cory, Thomas; Rieder, Wolf; Krämer, Julia; Raschke, Philip; Herbke, Patrick; Küpper, Axel (2026): "Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models", Proceedings on Privacy Enhancing Technologies 2026(1):509-528. (DOI)] is the shape of the current work: word-level GDPR-transparency annotation by LLM, evaluated against hand annotation |
| Hand-coding by trained annotators | Current, and it did not go away. | Still the most common method in every era including 2025–2026 (45.8%). LLMs are being added beside it, mostly as the thing the humans validate |
| Supervised sentence classifiers trained on OPP-115 (Polisis, PrivBERT, BERT variants) | Declining, still the reproducible baseline. | 42.9% (2014–2018) → 12.5% (2025–2026). Use one as the comparison arm even if your headline method is an LLM |
| Ontology + dependency parsing (PolicyLint, PoliCheck, PoliGraph) | Current for consistency work, unmaintained as code. | heuristic-rules peaked at 40% in 2022–2024, fell to 16.7%. PoliGraph is still cited into 2026 but has not been committed to since 2023-06-21 |
| Readability scores as a headline result | Historical. | Established by 2021 and not a contribution on its own since. Report it, do not build on it |
| Comparing policy against store-declared labels | New and growing. | Apple labels mandatory since Dec 2020, Play Data safety since Jul 2022; the comparison papers start in 2023 and there are four in 2024–2026 |
| P3P or any machine-readable policy content format | Dead, and nothing replaced it. | W3C retired P3P 1.1 on 30 August 2018 with an explicit “should not be referenced in this form or implemented as-is”; the P3P project home page's newest news item is dated 3 October 2007 and its own footer stamp reads 2018-02-02. GPC and IAB's GPP are consent signals, not policy content5) |
The honest summary: the analysis modernised and the collection did not. LLMs went from nothing to nearly half the 2025–2026 papers in three years. Meanwhile nobody has fixed the retrieval step, no corpus reaches 2026, and every tool in the lineage is abandoned code.
If you are choosing a method today: use an LLM for extraction, use a supervised OPP-115-trained classifier as your comparison arm, hand-label a validation sample yourself, and report all three. That is what the 2025–2026 papers that survived review actually did — [31Xie, Qinge; Ramakrishnan, Karthik; Li, Frank (2025): "Evaluating Privacy Policies under Modern Privacy Laws At Scale: An LLM-Based Automated Approach", in: Proceedings of the USENIX Security Symposium, pp. 5797-5816. (Link)] and [32Chanenson, Jake; Pickering, Madison; Apthorpe, Noah (2025): "Automating Governing Knowledge Commons and Contextual Integrity (GKC-CI) Privacy Policy Annotations with Large Language Models", in: Proceedings on Privacy Enhancing Technologies. (DOI)] are the two to copy.
Use in Publications
Two populations, and they are not interchangeable:
POLICY— the 102 papers whose extraction records aclassificationtuple withtarget == “privacy-policy”. A structured enum: 2.3% of the 4,439 corpus papers that classified anything. This is the population for every method figure above.UNION— 123 papers:POLICYplus a title-and-summary probe for policy/notice/terms/label vocabulary. A candidate set, used only for rankings and for “does this literature do X”, never for a rate.
The two overlap on 56 papers. 46 papers fire on the enum but not the title probe — mostly Alexa-skill, IoT-companion-app and VR studies where the policy analysis is one component of a wider privacy audit, which is a good reminder that most policy measurement is not published under a policy-shaped title.
| Year | corpus | POLICY | UNION | share of corpus |
|---|---|---|---|---|
| 2014 | 166 | 1 | 1 | 0.6% |
| 2016 | 182 | 1 | 2 | 1.1% |
| 2017 | 231 | 3 | 3 | 1.3% |
| 2018 | 254 | 2 | 2 | 0.8% |
| 2019 | 402 | 6 | 6 | 1.5% |
| 2020 | 404 | 8 | 9 | 2.2% |
| 2021 | 379 | 7 | 9 | 2.4% |
| 2022 | 546 | 17 | 19 | 3.5% |
| 2023 | 719 | 14 | 16 | 2.2% |
| 2024 | 690 | 19 | 24 | 3.5% |
| 2025* | 770 | 16 | 20 | 2.6% |
| 2026* | 415 | 8 | 12 | 2.9% |
| Venue | UNION | POLICY | venue's corpus slice | share of venue |
|---|---|---|---|---|
| PETS | 54 | 43 | 510 | 10.6% |
| USENIX Sec | 27 | 25 | 1,410 | 1.9% |
| CCS | 12 | 8 | 990 | 1.2% |
| NDSS | 9 | 8 | 701 | 1.3% |
| TheWebConf | 9 | 7 | 843 | 1.1% |
| IEEE S&P | 7 | 6 | 767 | 0.9% |
| IMC | 5 | 5 | 638 | 0.8% |
PETS is where this literature lives, by a factor of five over the next venue. If you are writing a policy paper, that is your target and its reviewers are the ones whose expectations you should meet.
Where this literature differs from the corpus, tested
Four of these are real differences and one is not; all five are Fisher's exact, two-sided. The subgroup is POLICY (the 102-paper enum, not the 123-paper candidate set), restricted in each row to the papers for which the field exists at all. The comparison column is the rest of the corpus with those 102 papers removed — comparing a subgroup against a population that still contains it shrinks the difference.
| Property | This literature (POLICY) | The rest of the corpus | p |
|---|---|---|---|
| Uses the web archive as its temporal source | 11/101 (10.9%) | 57/5,241 (1.1%) | 3.99 × 10⁻⁸ |
| States an inter-annotator agreement metric | 38/101 (37.6%) | 474/3,217 (14.7%) | 2.65 × 10⁻⁸ |
| Assesses a law | 64/102 (62.7%) | 338/5,757 (5.9%) | 3.62 × 10⁻⁵⁰ |
| Releases artifacts publicly | 63/101 (62.4%) | 2,782/5,438 (51.2%) | 0.027 |
| Reports no classifier validation | 40/102 (39.2%) | 1,823/4,337 (42.0%) | 0.613 — no difference |
The first three are the character of the field: it is legal, it is longitudinal, and because it hand-codes documents it reports agreement two and a half times as often as the rest of the corpus does. The last row is the one to internalise: policy work is exactly as bad as everyone else at validating its classifiers. Two papers in five report no validation of the thing their whole result rests on. Do not read the strong agreement-reporting row as evidence the field is careful here; those are different classifiers.
What the field does, and does not, describe
Full-text probes over the 123 papers. Narrow form; each counts papers whose text contains the phrase, which is an upper bound on a candidate set, not a measurement.
| Does the paper's text mention… | papers | share of 123 |
|---|---|---|
| comparing the policy against observed behaviour | 80 | 65.0% |
| the policy changing / being versioned, or the web archive | 68 | 55.3% |
| duplicate or templated policies | 34 | 27.6% |
| analysing more than one language | 30 | 24.4% |
| which policy applies (app vs developer vs platform vs layered) | 27 | 22.0% |
| the language of the policies analysed | 27 | 22.0% |
| the sentence/segment segmentation step | 24 | 19.5% |
| measuring readability | 17 | 13.8% |
| stripping boilerplate from the policy page | 13 | 10.6% |
| finding the policy by link text or anchor keyword | 12 | 9.8% |
| a policy served as a PDF | 9 | 7.3% |
| dead or unresolvable policy links | 7 | 5.7% |
| naming its link-detection seed phrases | 1 | 0.8% |
The bottom five rows are the retrieval step, and they are the least-described part of a literature whose denominator is entirely determined by them.
Methodology and limitations of these figures
- Every table names its own denominator. 102 (
POLICY), 123 (UNION), 4,439 (classified anything) and 5,859 (corpus) are four different populations and none of them is interchangeable with another. POLICYis an enum andUNIONis not. The 102 needs no folding and is stable between extraction runs; the 123 depends on a regex I chose and would move if I chose differently. The only figures over the 123 anywhere on this page are the full-text probe table and the two places in the prose that quote its largest row (80 of 123); all three say in the same breath that they are an upper bound on a candidate set. Every rate is over 102, 4,439 or 5,859.- Probe width decides the claim. The title probe here is nine alternatives, widened from the four used in the gap analysis that proposed this page, because
privacy label,data safetyandnutrition labelare the store-declaration half of the literature and the narrow probe missed them. Both widths are on the provenance page. - Sentinels are never answers.
none-reportedin the validation table is a real value reported as itself. - Papers are counted, never tuples. The 102 papers carry 179 privacy-policy classification tuples.
- Free-text names are folded and the complete unmapped residue — 138 distinct source-list strings and 572 distinct tool names — is printed on the provenance page. The tool fold is deliberately narrow: it maps the policy-analysis lineage and the NLP substrate, and leaves general-purpose tools unmapped rather than guessing.
- Every measured figure above was checked against the paper's own text, not against the extraction's evidence quote — 68 needles, 68 located, 0 missing. 67 were found in
paper.cols.txt; one ([6Amos, Ryan; Acar, Gunes; Lucherini, Eli; Kshirsagar, Mihir; Narayanan, Arvind; Mayer, Jonathan (2021): "Privacy Policies over Time: Curation and Analysis of a Million-Document Dataset", in: Proceedings of the Web Conference 2021, pp. 2165–2176. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)]'s 2009–2019 analysis window) is spliced by the de-columning and was located only in apypdfrendering of the PDF. That is why the check runs four renderings rather than one: a.cols-only check would have reported a false MISS. The full check and its unedited output are on the provenance page. - 2026 is provisional and 2025 is thin at the edges; see the note under the method table.
- Corpus-wide caveats, including which venues are absent and the extraction's own error rates, are on Corpus.
- The full query log, both report scripts and their unedited output, the folds and their complete residue, the quote check, and the external sources accepted and rejected are at policies.
What to Report
A policy-measurement methods section that survives PETS review needs:
- How you found the policy. The seed phrases or link-text patterns, whether you followed redirects, how many hops, and whether you fell back to a site search. One paper in 123 publishes its seed list. Publish yours.
- The retrieval funnel, numerically. Sampled → link found → link resolved → document parsed → language accepted → length threshold passed → analysed. Every later percentage over the last number. [8Pan, Shidong; Zhang, Dawen; Staples, Mark; Xing, Zhenchang; Chen, Jieshan; Xu, Xiwei; Hoang, Thong (2024): "Is It a Trap? A Large-scale Empirical Study And Comprehensive Assessment of Online Automated Privacy Policy Generators for Mobile Apps", in: Proceedings of the USENIX Security Symposium. (Link)] is the model here: 15.7% no link, 37.5% dead links, 20.5% non-English, 22.3% under threshold, all stated.
- Which policy you analysed when there is more than one. App policy or developer's generic corporate policy? Platform policy or vendor policy? Layered notice or the full document? Only 27 of 123 papers address this at all, and it silently changes results.
- The date and the version. A policy is a moving target. Give the retrieval date; if you compare across time, say how you matched versions and whether you used live crawls or archived snapshots (Archives, Longitudinal).
- Language handling. How you detected language, what you did with non-English documents, and whether you translated — machine translation changes what a sentence classifier sees (Multilingual support).
- Deduplication. How you detected templated and reused policies, and whether your unit is the document or the service. 65% template rate means an undeduplicated distribution is mostly measuring template vendors.
- The extraction step in full. Segmentation unit (sentence, segment, paragraph), model and version — including the exact LLM snapshot and the prompt, published as an artefact (Artifacts) — and the ontology or label set.
- Validation of every classifier on your own data, with sample size and an agreement measure (Interrater agreement). Two papers in five do not, and that is the row a reviewer will find.
- For a consistency study: the matching rule, and a false-consistency check. [4Andow, Benjamin; Mahmud, Samin Yaseer; Whitaker, Justin; Enck, William; Reaves, Bradley; Singh, Kapil; Egelman, Serge (2020): "Actions Speak Louder than Words: Entity-Sensitive Privacy Policy and Data Flow Analysis with PoliCheck", in: Proceedings of the USENIX Security Symposium. (Link)] showed entity-insensitive matching falsely clears up to 37.1% of inconsistent flows. Report your rule and what happens under a weaker one.
- What a compliant policy would have looked like. If your criterion fails 99% of the population, show a passing example and a graded distribution, or the result reads as a criterion artefact.
Open Questions
- No maintained policy corpus reaches 2026. OPP-115 is 115 documents from 2016, APP-350 is 350, the Princeton–Leuven corpus stops in 2019 and has had no commit since 2022–2023. Every 2025–2026 paper crawls its own. A versioned, licensed, citable corpus with an annual refresh is the single highest-value piece of infrastructure this field is missing, and it is the policy analogue of what Tranco did for site rankings.
- Every tool in the lineage is abandoned code. PolicyLint/PoliCheck stopped in October 2022, PurPliance's last code commit is December 2022, PoliGraph stopped in June 2023, PolicyChecker in November 2023 — and two of them are not findable by searching GitHub for the tool's own name. Nobody has published a maintained, packaged, validated policy-extraction library; there is no PyPI package for any of them. A
pip installthat works would be cited by everything. - How good is LLM extraction, really? Eleven of the 24 papers in 2025–2026 use one, and we found no head-to-head evaluation in these seven venues of an LLM against PoliGraph and against a PrivBERT baseline on the same documents with the same ground truth. Until someone runs that, the switch is a preference, not a finding.
- The retrieval step has never been measured as a step. Every study reports the availability rate its own heuristic produced. Nobody has taken one site sample and compared five link-detection heuristics, so the field does not know how much of the 4.35%-to-94.2% spread in the availability table is ecosystem and how much is method.
- Non-English policy analysis is 30 papers and mostly bilingual. [33Hosseini, Henry; Utz, Christine; Degeling, Martin; Hupperich, Thomas (2024): "A Bilingual Longitudinal Analysis of Privacy Policies Measuring the Impacts of the GDPR and the CCPA/CPRA", in: Proceedings on Privacy Enhancing Technologies. (DOI)] is English–German; the 2026 Swiss study adds a third. There is no multilingual policy corpus and no evaluation of whether OPP-115-derived label sets transfer to a language other than English at all.
- Nobody has connected policy text to enforcement outcomes at scale. 70 of the 123 papers assess a law, but we found none in these seven venues that tests whether a policy defect a tool flags predicts a regulator's decision. That is the missing external validity check for the whole compliance literature, and the material exists (Legal enforcement).
- Terms of service are almost unstudied here. The probe finds one dedicated study in these seven venues, [34Tsai, Elisa; Mangaokar, Neal; Zheng, Boyuan; Zheng, Haizhong; Prakash, Atul (2025): "Harmful Terms and Where to Find Them: Measuring and Modeling Unfavorable Financial Terms and Conditions in Shopping Websites at Scale", in: Proceedings of the ACM Web Conference. (DOI)] on unfavourable financial terms in shopping sites. Terms are longer, more consequential and less machine-readable than privacy policies, and there is no OPP-115 for them.
Related Pages
- Consent — the banner and what it stores. Banner text is not policy text, and the two literatures are separate.
- Mobile and app measurement — half of this literature. Store listings, Data safety, privacy labels and the app-acquisition problem.
- Legal enforcement — what a regulator actually did about the violations these tools find.
- Website classification — the labelling protocol, and validating a classifier you did not train.
- Interrater agreement — hand-coding policy segments is the corpus's most agreement-reporting literature; this is how to do it.
- Archives and Longitudinal — recovering old policy versions from the Wayback Machine, which this field does about ten times as often as the rest of the corpus.
- Multilingual support — policies are not all in English, and your extractor probably assumes they are.
- Existing datasets — the general case of the corpora table above.
- Traffic files — the behavioural arm of a consistency study.
- Artifacts — publishing your policy set, your seed phrases and your prompts.
References
- [1]
- Cui, Hao; Trimananda, Rahmadi; Markopoulou, Athina (2025): "Understanding Privacy Norms through Web Forms", in: Proceedings on Privacy Enhancing Technologies. (DOI)
- [2]
- Edu, Jide S.; Mulligan, Cliona; Pierazzi, Fabio; Polakis, Jason; Suarez-Tangil, Guillermo; Such, Jose M. (2022): "Exploring the security and privacy risks of chatbots in messaging services", in: Proceedings of the ACM Internet Measurement Conference. (DOI)
- [3]
- Mhaidli, Abraham; Fidan, Selin; Doan, An; Herakovic, Gina; Srinath, Mukund; Matheson, Lee; Wilson, Shomir; Schaub, Florian (2023): "Researchers’ Experiences in Analyzing Privacy Policies: Challenges and Opportunities", Proceedings on Privacy Enhancing Technologies 2023:287-305. (DOI) (Link)
- [4]
- Andow, Benjamin; Mahmud, Samin Yaseer; Whitaker, Justin; Enck, William; Reaves, Bradley; Singh, Kapil; Egelman, Serge (2020): "Actions Speak Louder than Words: Entity-Sensitive Privacy Policy and Data Flow Analysis with PoliCheck", in: Proceedings of the USENIX Security Symposium. (Link)
- [5]
- Cui, Hao; Trimananda, Rahmadi; Markopoulou, Athina; Jordan, Scott (2023): "PoliGraph: Automated Privacy Policy Analysis using Knowledge Graphs", in: Proceedings of the USENIX Security Symposium. (Link)
- [6]
- Amos, Ryan; Acar, Gunes; Lucherini, Eli; Kshirsagar, Mihir; Narayanan, Arvind; Mayer, Jonathan (2021): "Privacy Policies over Time: Curation and Analysis of a Million-Document Dataset", in: Proceedings of the Web Conference 2021, pp. 2165–2176. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)
- [7]
- Linden, Thomas; Khandelwal, Rishabh; Harkous, Hamza; Fawaz, Kassem (2020): "The Privacy Policy Landscape After the GDPR", in: Proceedings on Privacy Enhancing Technologies. (DOI)
- [8]
- Pan, Shidong; Zhang, Dawen; Staples, Mark; Xing, Zhenchang; Chen, Jieshan; Xu, Xiwei; Hoang, Thong (2024): "Is It a Trap? A Large-scale Empirical Study And Comprehensive Assessment of Online Automated Privacy Policy Generators for Mobile Apps", in: Proceedings of the USENIX Security Symposium. (Link)
- [9]
- Degeling, Martin; Utz, Christine; Lentzsch, Christopher; Hosseini, Henry; Schaub, Florian; Holz, Thorsten (2019): "We Value Your Privacy ... Now Take Some Cookies: Measuring the GDPR's Impact on Web Privacy", in: Proceedings of the Network and Distributed System Security Symposium. (Link)
- [10]
- Vallina, Pelayo; Feal, Álvaro; Gamba, Julien; Vallina-Rodriguez, Narseo; Anta, Antonio Fernández (2019): "Tales from the Porn: A Comprehensive Privacy Analysis of the Web Porn Ecosystem", in: Proceedings of the ACM Internet Measurement Conference. (DOI)
- [11]
- Zimmeck, Sebastian; Story, Peter; Smullen, Daniel; Ravichander, Abhilasha; Wang, Ziqi; Reidenberg, Joel; Russell, N. Cameron; Sadeh, Norman (2019): "MAPS: Scaling Privacy Compliance Analysis to a Million Apps", Proceedings on Privacy Enhancing Technologies 2019(3). (DOI)
- [12]
- Manandhar, Sunil; Kafle, Kaushal; Andow, Benjamin; Singh, Kapil; Nadkarni, Adwait (2022): "Smart Home Privacy Policies Demystified: A Study of Availability, Content, and Coverage", in: Proceedings of the USENIX Security Symposium. (Link)
- [13]
- Lentzsch, Christopher; Shah, Sheel Jayesh; Andow, Benjamin; Degeling, Martin; Das, Anupam; Enck, William (2021): "Hey Alexa, is this Skill Safe?: Taking a Closer Look at the Alexa Skill Ecosystem", in: Proceedings of the Network and Distributed System Security Symposium. (Link)
- [14]
- Yan, Chuan; Xie, Fuman; Meng, Mark Huasong; Zhang, Yanjun; Bai, Guangdong (2024): "On the Quality of Privacy Policy Documents of Virtual Personal Assistant Applications", in: Proceedings on Privacy Enhancing Technologies. (DOI)
- [15]
- Zhan, Yuxia; Meng, Yan; Zhou, Lu; Xiong, Yichang; Zhang, Xiaokuan; Ma, Lichuan; Chen, Guoxing; Pei, Qingqi; Zhu, Haojin (2024): "VPVet: Vetting Privacy Policies of Virtual Reality Apps", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)
- [16]
- Wu, Yuhao; Jaff, Evin; Yang, Ke; Zhang, Ning; Iqbal, Umar (2025): "An In-Depth Investigation of Data Collection in LLM App Ecosystems", in: Proceedings of the ACM Internet Measurement Conference. (DOI)
- [17]
- Harkous, Hamza; Fawaz, Kassem; Lebret, Rémi; Schaub, Florian; Shin, Kang G.; Aberer, Karl (2018): "Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learning", in: Proceedings of the USENIX Security Symposium. (Link)
- [18]
- Andow, Benjamin; Mahmud, Samin Yaseer; Wang, Wenyu; Whitaker, Justin; Enck, William; Reaves, Bradley; Singh, Kapil; Xie, Tao (2019): "PolicyLint: Investigating Internal Privacy Policy Contradictions on Google Play", in: Proceedings of the USENIX Security Symposium. (Link)
- [19]
- Xiang, Anhao; Pei, Weiping; Yue, Chuan (2023): "PolicyChecker: Analyzing the GDPR Completeness of Mobile Apps' Privacy Policies", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)
- [20]
- Zhou, Lu; Wei, Chengyongxiao; Zhu, Tong; Chen, Guoxing; Zhang, Xiaokuan; Du, Suguo; Cao, Hui; Zhu, Haojin (2023): "POLICYCOMP: Counterpart Comparison of Privacy Policies Uncovers Overbroad Personal Data Collection Practices", in: Proceedings of the USENIX Security Symposium. (Link)
- [21]
- Xiao, Yue; Li, Zhengyi; Qin, Yue; Bai, Xiaolong; Guan, Jiale; Liao, Xiaojing; Xing, Luyi (2023): "Lalaine: Measuring and Characterizing Non-Compliance of Apple Privacy Labels", in: Proceedings of the USENIX Security Symposium. (Link)
- [22]
- Qiu, Wenjun; Lie, David; Austin, Lisa (2023): "Calpric: Inclusive and Fine-grain Labeling of Privacy Policies with Crowdsourcing and Active Learning", in: Proceedings of the USENIX Security Symposium. (Link)
- [23]
- Zimmeck, Sebastian; Wang, Ziqi; Zou, Lieyong; Iyengar, Roger; Liu, Bin; Schaub, Florian; Wilson, Shomir; Sadeh, Norman; Bellovin, Steven M.; Reidenberg, Joel (2017): "Automated Analysis of Privacy Requirements for Mobile Apps", in: Proceedings of the Network and Distributed System Security Symposium. (Link)
- [24]
- Libert, Timothy (2018): "An Automated Approach to Auditing Disclosure of Third-Party Data Collection in Website Privacy Policies", in: Proceedings of the ACM Web Conference. (DOI)
- [25]
- Trimananda, Rahmadi; Le, Hieu; Cui, Hao; Ho, Janice Tran; Shuba, Anastasia; Markopoulou, Athina (2022): "OVRseen: Auditing Network Traffic and Privacy Policies in Oculus VR", in: Proceedings of the USENIX Security Symposium. (Link)
- [26]
- Samarin, Nikita; Kothari, Shayna; Siyed, Zaina; Bjorkman, Oscar; Yuan, Reena; Wijesekera, Primal; Alomar, Noura; Fischer, Jordan; Hoofnagle, Chris; Egelman, Serge (2023): "Lessons in VCR Repair: Compliance of Android App Developers with the California Consumer Privacy Act (CCPA)", in: Proceedings on Privacy Enhancing Technologies. (DOI)
- [27]
- Ali, Mir Masood; Balash, David G.; Kodwani, Monica; Kanich, Chris; Aviv, Adam J. (2024): "Honesty is the Best Policy: On the Accuracy of Apple Privacy Labels Compared to Apps' Privacy Policies", in: Proceedings on Privacy Enhancing Technologies. (DOI)
- [28]
- Adhikari, Andrick; Das, Sanchari; Dewri, Rinku (2023): "Evolution of Composition, Readability, and Structure of Privacy Policies over Two Decades", in: Proceedings on Privacy Enhancing Technologies. (DOI)
- [29]
- Huang, Ziyuan; Tang, Jiaming; Karir, Manish; Liu, Mingyan; Sarabi, Armin (2024): "Analyzing Corporate Privacy Policies using AI Chatbots", in: Proceedings of the ACM Internet Measurement Conference. (DOI)
- [30]
- Cory, Thomas; Rieder, Wolf; Krämer, Julia; Raschke, Philip; Herbke, Patrick; Küpper, Axel (2026): "Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models", Proceedings on Privacy Enhancing Technologies 2026(1):509-528. (DOI)
- [31]
- Xie, Qinge; Ramakrishnan, Karthik; Li, Frank (2025): "Evaluating Privacy Policies under Modern Privacy Laws At Scale: An LLM-Based Automated Approach", in: Proceedings of the USENIX Security Symposium, pp. 5797-5816. (Link)
- [32]
- Chanenson, Jake; Pickering, Madison; Apthorpe, Noah (2025): "Automating Governing Knowledge Commons and Contextual Integrity (GKC-CI) Privacy Policy Annotations with Large Language Models", in: Proceedings on Privacy Enhancing Technologies. (DOI)
- [33]
- Hosseini, Henry; Utz, Christine; Degeling, Martin; Hupperich, Thomas (2024): "A Bilingual Longitudinal Analysis of Privacy Policies Measuring the Impacts of the GDPR and the CCPA/CPRA", in: Proceedings on Privacy Enhancing Technologies. (DOI)
- [34]
- Tsai, Elisa; Mangaokar, Neal; Zheng, Boyuan; Zheng, Haizhong; Prakash, Atul (2025): "Harmful Terms and Where to Find Them: Measuring and Modeling Unfavorable Financial Terms and Conditions in Shopping Websites at Scale", in: Proceedings of the ACM Web Conference. (DOI)
secartifacts.github.io/usenixsec2023/results (HTTP 200) links github.com/dlgroupuoft/Calpric and lists Lalaine; GitHub's repository search returns dlgroupuoft/Calpric and xiaoyue10131748/Lalaine as top name matches. PolicyChecker was named here in an earlier draft and should not have been — it is a CCS paper, is not on that index at all, and github.com/AndyXiang945/PolicyChecker is the first hit for its name. A general lesson about artifact discoverability, not a criticism of these authors.w3.org answers curl with a Cloudflare 403 — see scripts/policies_w3c_p3p_check.mjs. ADPC exists as a proposal and is not in production.