| Next revision | Previous revision |
| statistics:interrater_agreement [2026/08/20 21:22] – New page: Inter-rater Agreement. Which coefficient the field uses and when it is the wrong one, what kappa does to a rare-class labelling task, and how to report agreement when an LLM did the first pass. Figures from the 5,859-paper corpus (population: 3, karel.kubicek.claude | statistics:interrater_agreement [2026/08/21 14:50] (current) – Boxes: <wrap> renders a span, use uppercase <WRAP>; one box per list, not per bullet karel.kubicek.claude |
|---|
| </code> | </code> |
| |
| <wrap todo> | <WRAP todo> |
| **With more than two coders, use Fleiss' kappa (equal number of coders per item, nominal labels) or Krippendorff's alpha (unequal coders, missing data, or ordinal/interval labels).** If you average pairwise Cohen's kappas anyway, say so and give the range, not the mean alone — the mean of three pairwise kappas is not "the" kappa. | **With more than two coders, use Fleiss' kappa (equal number of coders per item, nominal labels) or Krippendorff's alpha (unequal coders, missing data, or ordinal/interval labels).** If you average pairwise Cohen's kappas anyway, say so and give the range, not the mean alone — the mean of three pairwise kappas is not "the" kappa. |
| </wrap> | </WRAP> |
| |
| Krippendorff's alpha is the safest default for the messy case, which is why the more careful papers here reach for it: Cory et al. {[cory2026_wordlevel]} compute it over //"a team of six domain experts from two universities"// on 200 privacy policies (α = 0.669); Xian et al. {[xian2025_layered]} use the unitising variant Cu-α for span-level policy annotation, reporting 0.95 for CCPA policies and 0.89 for the rest. | Krippendorff's alpha is the safest default for the messy case, which is why the more careful papers here reach for it: Cory et al. {[cory2026_wordlevel]} compute it over //"a team of six domain experts from two universities"// on 200 privacy policies (α = 0.669); Xian et al. {[xian2025_layered]} use the unitising variant Cu-α for span-level policy annotation, reporting 0.95 for CCPA policies and 0.89 for the rest. |
| Read the first table with a real task in mind. If 3% of your sampled cookies are the class you care about and your two coders disagree on 3% of items, your kappa is about **0.65** — a number a reviewer will read as mediocre — and the same two coders on a balanced sample score **0.94**. Nothing about their reliability changed. | Read the first table with a real task in mind. If 3% of your sampled cookies are the class you care about and your two coders disagree on 3% of items, your kappa is about **0.65** — a number a reviewer will read as mediocre — and the same two coders on a balanced sample score **0.94**. Nothing about their reliability changed. |
| |
| <wrap todo> | <WRAP todo> |
| **On a lopsided task, report three things, not one:** the coefficient, the **observed agreement**, and the **marginal distribution** (how many items each coder put in each class). Those three together let a reader diagnose the paradox; any one alone does not. If the class is very rare, report **Gwet's AC1** {[gwet2008_computing]} alongside kappa — it is designed for this case, it is in ''irrCAC'' on CRAN, and five papers in this corpus already do it. | **On a lopsided task, report three things, not one:** the coefficient, the **observed agreement**, and the **marginal distribution** (how many items each coder put in each class). Those three together let a reader diagnose the paradox; any one alone does not. If the class is very rare, report **Gwet's AC1** {[gwet2008_computing]} alongside kappa — it is designed for this case, it is in ''irrCAC'' on CRAN, and five papers in this corpus already do it. |
| |
| The corpus's model of this is Khatun et al. {[khatun2026_disclosure]}, who give both numbers for the same 40-app subsample: Cohen's κ = 0.78 **and** 90% observed agreement. Chen et al. {[chen2025_semantics]} do the same job at scale on the task this site cares about most — cookie purpose labels, three independent author-coders, 2,300 cookies, Fleiss' κ = 0.978. | The corpus's model of this is Khatun et al. {[khatun2026_disclosure]}, who give both numbers for the same 40-app subsample: Cohen's κ = 0.78 **and** 90% observed agreement. Chen et al. {[chen2025_semantics]} do the same job at scale on the task this site cares about most — cookie purpose labels, three independent author-coders, 2,300 cookies, Fleiss' κ = 0.978. |
| </wrap> | </WRAP> |
| |
| ==== Landis–Koch bands are not a result ==== | ==== Landis–Koch bands are not a result ==== |
| ===== Open Questions ===== | ===== Open Questions ===== |
| |
| | <WRAP todo> |
| * **How much does author-coding inflate agreement?** 85.8% of hand-coding papers here are coded by their own authors. Neither the extraction nor a full-text probe finds a paper that double-codes a subsample with a coder outside the author team as a check on that, and the probe is narrow enough that this should be read as "we could not find one" rather than "there are none". A study that takes ten published codebooks, hires outside coders, and compares the coefficients against the published ones is small, cheap and would recalibrate how every reader reads a kappa in this literature. | * **How much does author-coding inflate agreement?** 85.8% of hand-coding papers here are coded by their own authors. Neither the extraction nor a full-text probe finds a paper that double-codes a subsample with a coder outside the author team as a check on that, and the probe is narrow enough that this should be read as "we could not find one" rather than "there are none". A study that takes ten published codebooks, hires outside coders, and compares the coefficients against the published ones is small, cheap and would recalibrate how every reader reads a kappa in this literature. |
| * **What is the real class balance of the tasks this field codes, and how much of the low-kappa tail is the prevalence paradox rather than coder disagreement?** Answerable on any paper that released its labels, and unanswered. The 48 papers here whose classifier ground truth is manual annotation by a **single** stated coder are the place to start. | * **What is the real class balance of the tasks this field codes, and how much of the low-kappa tail is the prevalence paradox rather than coder disagreement?** Answerable on any paper that released its labels, and unanswered. The 48 papers here whose classifier ground truth is manual annotation by a **single** stated coder are the place to start. |
| * **Almost nobody reports LLM self-agreement.** Run-to-run variability of a model on the same items is the machine analogue of intra-rater reliability and it is cheap to measure. A full-text probe for it finds **18 papers in 5,869** — against 267 hand-coding papers with an LLM in the loop. Whether those 18 are the whole practice or the probe is narrow is itself unresolved. | * **Almost nobody reports LLM self-agreement.** Run-to-run variability of a model on the same items is the machine analogue of intra-rater reliability and it is cheap to measure. A full-text probe for it finds **18 papers in 5,869** — against 267 hand-coding papers with an LLM in the loop. Whether those 18 are the whole practice or the probe is narrow is itself unresolved. |
| * **None of the seven venues asks for any of this.** Reviewer forms are not public, so the current state can only be read off the papers — and the papers say that a decade of improvement has got the field from 5% to 21% reporting. Whether a one-line submission checklist would close the rest is untested and testable. | * **None of the seven venues asks for any of this.** Reviewer forms are not public, so the current state can only be read off the papers — and the papers say that a decade of improvement has got the field from 5% to 21% reporting. Whether a one-line submission checklist would close the rest is untested and testable. |
| | </WRAP> |
| ===== Related Pages ===== | ===== Related Pages ===== |
| |
