Cohen's kappa

Cohen's kappa coefficient is a statistical measure of inter-annotator agreement. It is generally thought to be a more robust judge than simple percent agreement calculation.

The equation for kappa is:

<math>\kappa = \frac{\Pr(a) – \Pr(e)}{1 – \Pr(e)}, \!<math>

where <math>\Pr(a)<math> is the relative observed annotator agreement, and <math>\Pr(e)<math> is the probability that agreement is due to chance.

The seminal paper introducing kappa as a novel technique was published by Jacob Cohen in the journal Educational and Psychological Measurement in 1960.

References

• Jacob Cohen, A coefficient of agreement for nominal scales, Educational and Psychological Measurement 20: 37–46, 1960.