Cohen's Kappa: How to Calculate and Interpret It

Cohen's kappa (κ) measures agreement between two sets of categorical ratings after accounting for agreement expected from the raters' category-use patterns. It is most often used for inter-rater reliability, but it can also compare two classification systems or repeated categorical ratings of the same items.

The statistic matters because raw percentage agreement can look reassuring when both raters heavily favor the same category. Kappa asks a harder question: how much agreement remains after allowing for the agreement expected from the observed marginal distributions? That makes it useful for research, healthcare, annotation, education, and machine learning, but it also means a kappa value should never be interpreted without the underlying table and study context.

What Is Cohen's Kappa in Simple Terms?

Cohen's kappa is a chance-corrected agreement coefficient for two sets of categorical ratings. Jacob Cohen introduced the coefficient in 1960 for agreement on nominal scales. Instead of treating every matching label as equally informative, the method compares observed agreement with agreement expected under an independence model built from the two raters' marginal category proportions.

Imagine two doctors independently classifying the same patients as either depressed or not depressed. Percentage agreement tells you how often the labels match. Cohen's kappa goes further by considering how frequently each doctor uses each category. If both doctors almost always choose the same dominant category, some matching would be expected even without strong item-by-item discrimination.

This is why the phrase 'agreement by chance' needs care. The expected term is not a guess that both raters literally choose labels at random. It is a model-based expectation calculated from the observed category frequencies.

Why Cohen's Kappa Matters for Reliability

Inter-rater reliability concerns the consistency of judgments made about the same items. Kappa is useful when the outcome is categorical and subjective interpretation may affect classification, such as diagnostic categories, document coding, behavioral observations, image labels, or scoring decisions.

Reliability is not the same as validity. Two raters can agree perfectly and still both be wrong relative to a credible reference standard. A kappa of 1 therefore means perfect agreement between the two rating sets, not proof that the classifications are true or clinically accurate.

The same distinction matters in machine learning. Agreement between model predictions and reference labels can be informative, but it does not replace class-specific measures such as sensitivity, specificity, precision, recall, or other metrics tied to the actual decision problem.

When Should You Use Cohen's Kappa?

Standard Cohen's kappa is most natural when two rating sets classify the same independent items into the same mutually exclusive nominal categories. The two rating sets may come from two people, two systems, or the same rater at two different times when the question is test-retest or intra-rater agreement.

Typical examples include two clinicians classifying cases, two researchers coding interview excerpts, two annotators labeling images, or a classifier being compared with a reference label set. The key is that both rating sets evaluate the same units using compatible categories.

If the categories have a meaningful order, weighted kappa is often more informative. If there are more than two raters, a multi-rater method such as Fleiss' kappa may be considered, but it is not the only option. If the outcome is continuous, an intraclass correlation coefficient or another continuous-data agreement method is usually more appropriate than ordinary kappa.

Requirements to Check Before Calculating Kappa

The most important decision comes before the formula. Both rating sets should classify the same items, use compatible mutually exclusive categories, and produce one categorical outcome per item for the analysis being performed. The sampling units should also be sufficiently independent for the inferential method being used.

Do not choose kappa simply because it is familiar. First identify the measurement scale, the number of rating sets, whether category order matters, whether missing ratings exist, and whether the goal is agreement, accuracy against a reference standard, or another performance question.

A practical warning is that degenerate data can make kappa undefined. For example, if the label structure collapses so that the denominator in the kappa calculation becomes zero, statistical software may return an undefined value rather than a meaningful coefficient. Current scikit-learn documentation explicitly handles such cases as undefined metrics.

How Does Cohen's Kappa Work?

Cohen's kappa uses two core quantities. Observed agreement, p_o, is the proportion of items for which the two ratings match. Expected agreement, p_e, is calculated from the marginal proportions with which each rater uses each category.

The formula is κ = (p_o - p_e) / (1 - p_e). The numerator represents agreement beyond the expected amount, while the denominator scales that excess relative to the maximum additional agreement that would be possible above p_e.

If observed agreement is much greater than expected agreement, kappa is positive and can approach 1. If observed agreement equals expected agreement, kappa is 0. If observed agreement is below the expectation, kappa becomes negative. Although the theoretical bounds are -1 to 1, the values actually attainable in a particular table depend on its marginal distributions.

How to Calculate Cohen's Kappa Step by Step

Consider 50 patients independently classified by two doctors as either not depressed or depressed. The contingency table below contains every combination of the two ratings. The diagonal cells represent agreement; the off-diagonal cells represent disagreement.

Worked Example: From the Table to κ = 0.44

The doctors agree in 17 + 19 = 36 cases. With 50 patients, observed agreement is p_o = 36/50 = 0.72, or 72%.

Next calculate the marginal proportions. Rater 1 uses each category 25 out of 50 times, so the proportions are 0.50 and 0.50. Rater 2 classifies 23 of 50 patients as not depressed and 27 as depressed, giving marginal proportions of 0.46 and 0.54.

Expected agreement for not depressed is 0.50 × 0.46 = 0.23. Expected agreement for depressed is 0.50 × 0.54 = 0.27. Therefore p_e = 0.23 + 0.27 = 0.50.

Substituting the values gives κ = (0.72 - 0.50) / (1 - 0.50) = 0.44. The calculation is straightforward; the more difficult question is what 0.44 means for the actual study.

How Do You Interpret Cohen's Kappa Values?

A kappa of 1 means perfect item-by-item agreement. A value of 0 means the observed agreement equals the expected agreement used in the kappa model. A negative value means observed agreement falls below that expectation.

Landis and Koch's 1977 descriptive categories are still widely quoted, but they should be treated as a rough language convention rather than a universal scientific acceptance standard. Under that framework, κ = 0.44 would be described as moderate agreement.

Whether 0.44 is acceptable depends on the purpose of the ratings, consequences of disagreement, category prevalence, sample size, uncertainty, and professional standards in the field. A threshold that is tolerable for exploratory coding may be inadequate for a high-stakes clinical decision.

Standard, Weighted, and Multi-Rater Kappa

Unweighted Cohen's kappa treats every disagreement equally. That is appropriate for nominal categories such as positive versus negative or cat versus dog, where categories do not have an inherent ranking.

Weighted Cohen's kappa is designed for ordered categories. Cohen's 1968 weighted extension allows disagreements to receive different penalties, so a one-step disagreement on an ordinal scale can count as less severe than a disagreement across the full scale. Linear and quadratic weighting are common choices. Quadratic weights penalize distant disagreements more heavily relative to nearby ones, so the weighting rule should be chosen because it matches the meaning of the scale, not because it produces the largest number.

For more than two raters, Fleiss' kappa is a commonly used option, but it is not universally required. Other designs may call for alternatives such as Krippendorff's alpha, Conger-type coefficients, or Gwet-type agreement coefficients. Missing ratings, whether every rater evaluates every item, and the measurement scale all influence the best choice.

Cohen's Kappa vs Accuracy, Percentage Agreement, and ICC

Percentage agreement is the simplest measure: matched classifications divided by all classifications. Its strength is transparency. Its limitation is that it does not adjust for agreement expected from the marginal category distributions. Reporting raw agreement alongside kappa is often more informative than replacing one with the other.

Accuracy is usually defined relative to a reference label and asks how many predictions are correct. Cohen's kappa is symmetric between the two rating sets and summarizes chance-corrected agreement. In machine learning, this means kappa and accuracy may tell different stories, especially when label distributions are uneven.

Pearson correlation measures association, not agreement. Two measurement methods can correlate strongly while one systematically produces larger values than the other. For continuous measurements, an appropriate intraclass correlation coefficient is often a better reliability tool, although ICC selection itself depends on whether raters are fixed or random, whether absolute agreement or consistency is desired, and whether single or averaged measurements are evaluated.

The Main Limitation: High Agreement Can Still Produce Low Kappa

One of the most important limitations of Cohen's kappa is its sensitivity to prevalence. When one category is extremely common, expected agreement can become very high. The result can be a surprisingly modest kappa even when raw agreement looks excellent. Published methodological research describes this as a prevalence-related kappa paradox.

Consider an illustrative 100-item table in which the raters agree on 91 items. Raw agreement is 91%, but suppose the marginal category proportions produce expected agreement of 89.6%. Kappa then becomes (0.91 - 0.896) / (1 - 0.896) ≈ 0.135. The low value is not a calculation error: observed agreement only slightly exceeds what the marginal distributions already imply.

This is why the contingency table is essential. If a report gives only κ = 0.14, readers cannot see whether the problem is widespread disagreement, extreme prevalence, asymmetric category use, or some combination of these factors.

Marginal Imbalance and the Maximum Possible Kappa

Kappa is also affected when the two raters use categories at very different rates. If one rater assigns 70% of items to category A and the other assigns only 50%, they cannot agree on every item while keeping those marginal totals unchanged.

That means the maximum attainable kappa for fixed unequal marginals may be below 1. This is an advanced but useful diagnostic when a coefficient seems disappointingly low. The right response is not to manipulate the data to raise kappa; it is to examine whether category definitions, rater behavior, or the study design explain the marginal difference.

A low coefficient may therefore point to several different realities: genuinely ambiguous items, unclear coding rules, rater training problems, different decision thresholds, or highly uneven prevalence. Those causes require different remedies.

Cohen's Kappa With Imbalanced Data and Machine Learning

Kappa is sometimes recommended for imbalanced classification because it can reveal that high accuracy partly reflects a dominant class. That is useful, but the stronger claim that kappa simply 'solves class imbalance' is misleading. Kappa itself changes with prevalence and marginal distributions.

Do not automatically rebalance a representative test set just to obtain a higher kappa. If the real deployment population is 95% negative and 5% positive, that distribution may be part of the question the evaluation needs to answer. Artificially changing the test distribution changes the evaluation context and can change κ even if the model itself is unchanged.

For model evaluation, use kappa as one piece of evidence. A confusion matrix shows where errors occur, while sensitivity, specificity, precision, recall, calibration, or other task-specific measures can reveal whether the minority or high-cost class is being handled well. Current scikit-learn documentation provides cohen_kappa_score with unweighted, linear-weighted, and quadratic-weighted options for Python workflows.

How to Report Cohen's Kappa in Research

A defensible report should make the agreement pattern visible rather than presenting a single coefficient without context. At minimum, report the kappa estimate, the number of rated items, raw percentage agreement, category frequencies or the contingency table, and a clear description of the rating design.

When inference matters, report an appropriate confidence interval. The interval communicates uncertainty around the sample estimate and can change how confidently a study interprets the apparent strength of agreement. For formal work, use a validated statistical package or a variance method appropriate to the design rather than relying on a simplified standard-error shortcut.

A concise write-up for the worked example could read: 'The two raters agreed on 72% of classifications. Cohen's κ was 0.44, which falls in the moderate range under the Landis-Koch descriptive convention. The contingency table and marginal category frequencies were examined because kappa is sensitive to prevalence and category use.'

What to Do When Kappa Is Lower Than Expected

First, inspect the contingency table instead of recalculating the same coefficient. Determine which categories generate disagreement and whether one rater systematically uses a category more often than the other.

Next, inspect prevalence. If almost every item belongs to one category, high percentage agreement can be compatible with a modest kappa. Then review the coding definitions. Ambiguous categories, overlapping criteria, insufficient rater training, or different thresholds often create reliability problems that a statistical formula cannot fix.

Only after understanding the disagreement pattern should you consider changing the analysis method. Weighted kappa may fit ordered ratings; a multi-rater coefficient may fit a larger panel; an ICC may fit continuous measurements. Choosing a different statistic should follow the measurement design, not the desire for a larger reliability number.

Common Cohen's Kappa Mistakes to Avoid

The most consequential mistake is using the wrong statistic for the data. Standard kappa is not automatically appropriate for ordered ratings, many-rater designs, or continuous measurements.

Another mistake is treating a conventional interpretation label as a universal pass/fail rule. 'Substantial' is descriptive language, not proof that agreement is adequate for every clinical, educational, or research purpose.

Researchers should also avoid interpreting reliability as validity, comparing kappa values across very different prevalence settings without caution, reporting kappa without raw agreement, or concluding that a high overall kappa guarantees acceptable performance in every category.

FAQs

What is Cohen's kappa used for?

It is used to quantify agreement between two sets of categorical ratings while adjusting for agreement expected from their marginal category distributions. Common settings include research coding, clinical classifications, human annotation, and classification-system comparisons.

What is a good Cohen's kappa value?

There is no universal cutoff. Conventional labels such as slight, moderate, or substantial can describe magnitude, but acceptability depends on the application, uncertainty, prevalence, consequences of disagreement, and professional standards.

Is a kappa of 0.70 good?

A value of 0.70 falls in the 'substantial' range under the Landis-Koch convention. Whether it is sufficient still depends on the purpose of the ratings and the cost of disagreement.

Can Cohen's kappa be negative?

Yes. A negative value means observed agreement is lower than the agreement expected under the kappa model for those marginal distributions.

Why can kappa be low when agreement is high?

If one category dominates, expected agreement can also be very high. Kappa measures the improvement over that expectation, so a small gap between observed and expected agreement can produce a modest coefficient despite impressive raw agreement.

Is Cohen's kappa the same as accuracy?

No. Accuracy is generally reference-based correctness. Kappa is a symmetric chance-corrected agreement measure between two categorical rating sets.

Can Cohen's kappa be used for ordinal data?

Yes, but weighted kappa is usually more informative when category order and the distance between disagreements matter.

Can Cohen's kappa be used for more than two raters?

Standard Cohen's kappa is a two-rating-set coefficient. Multi-rater studies typically use a method designed for multiple raters, such as Fleiss' kappa or another coefficient matched to the design.

Can kappa be undefined?

Yes. Degenerate label configurations can make the denominator of the calculation zero or otherwise leave the statistic undefined. Software may return NaN or another explicitly undefined result in those cases.

Does Cohen's kappa measure reliability or validity?

It measures agreement or reliability between rating sets. Strong agreement does not prove that the ratings are correct against an external reference standard.

Conclusion: Use Cohen's Kappa as Evidence, Not a Verdict

Cohen's kappa is valuable because it asks a better question than raw percentage agreement: how much agreement exists beyond the amount expected from the raters' observed category distributions? That makes it useful for two sets of categorical ratings in research, healthcare, annotation, education, and machine learning.

The most important practical insight is that κ is not self-interpreting. Examine the contingency table, observed agreement, marginal category frequencies, prevalence, confidence interval, and the real consequences of disagreement. Use weighted kappa when ordered disagreements matter, a suitable multi-rater method when more than two raters are involved, and an appropriate continuous-data agreement method such as an ICC when the outcome is quantitative.

Before calculating Cohen's kappa, identify the data type, number of rating sets, category structure, and purpose of the analysis. After calculating it, diagnose the disagreement pattern rather than treating a cutoff table as the final decision.

References and Authoritative Sources

The article uses foundational papers, peer-reviewed methodological literature, and current official software documentation for claims that benefit from verification.

1. Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 37-46.

2. Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213-220.

3. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174.

4. Zec, S., et al. (2017). High Agreement and High Prevalence: The Paradox of Cohen's Kappa. The Open Nursing Journal, 11, 211-218.

5. McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia Medica, 22(3), 276-282.

6. Sim, J., & Wright, C. C. (2005). The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Physical Therapy, 85(3), 257-268.

7. Koo, T. K., & Li, M. Y. (2016). A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine, 15(2), 155-163.

8. scikit-learn developers. cohen_kappa_score documentation (current stable documentation checked August 2026).