statsmodels.stats.inter_rater.krippendorff_alpha#

statsmodels.stats.inter_rater.krippendorff_alpha(data, level='nominal')[source]#

Compute Krippendorff’s alpha reliability coefficient.

Agreement among any number of raters, for nominal, ordinal, interval or ratio data, with any number of missing ratings.

Added in version 0.15.1.

Parameters:
dataarray_like

Numeric ratings in a 2-D array with subjects in rows and raters in columns, the layout used by statsmodels.stats.inter_rater.aggregate_raters. Missing ratings are coded as nan. Nominal categories can be any numeric codes. See examples.

level{“nominal”, “ordinal”, “interval”, “ratio”}, optional

Level of measurement of the ratings. It defines the squared difference \(\delta^2_{ck}\) between two ratings \(c\) and \(k\). The default is “nominal”.

  • “nominal”: 0 if \(c = k\) and 1 otherwise.

  • “ordinal”: \((\sum_{g=c}^{k} n_g - (n_c + n_k) / 2)^2\), which depends on how often the ratings between \(c\) and \(k\) are used, not on their numeric distance.

  • “interval”: \((c - k)^2\).

  • “ratio”: \(((c - k) / (c + k))^2\). The ratings must be nonnegative.

Returns:
float

Krippendorff’s alpha. It is 1 for perfect agreement and 0 for agreement at the level of chance. It is nan if no subject has two or more ratings, or if all those ratings are equal, because alpha is 0 / 0 then.

See also

fleiss_kappa

Fleiss’ kappa for nominal ratings without missing values.

cohens_kappa

Cohen’s kappa for two raters.

Notes

Only subjects with at least two ratings are used. Following Krippendorff (2011), the ratings within each subject \(u\) with \(m_u\) ratings are tabulated in the coincidence matrix \(o_{ck}\): each of the \(m_u (m_u - 1)\) ordered pairs of ratings adds \(1 / (m_u - 1)\) to the cell of its two values. With the margins \(n_c = \sum_k o_{ck}\) and \(n = \sum_c n_c\),

\[\alpha = 1 - (n - 1) \frac{\sum_c \sum_k o_{ck} \delta^2_{ck}} {\sum_c \sum_k n_c n_k \delta^2_{ck}}\]

For nominal data without missing ratings, alpha is a function of Fleiss’ kappa, \(\alpha = 1 - (1 - \kappa) (n - 1) / n\) (see fleiss_kappa). For two raters, Fleiss’ kappa is Scott’s pi, so nominal alpha is 1 - (1 - pi) (n - 1) / n, which is Scott’s pi with a correction for the finite number n of ratings.

If all ratings of the subjects with at least two ratings have the same value, the agreement is perfect but alpha is 0 / 0 and nan is returned, for example for [[1, 1], [1, 1]], while [[1, 1], [2, 2]] gives 1. R’s irr::kripp.alpha returns 1 in that case.

The ratings are held in a matrix of subjects by distinct values and in matrices of distinct values by distinct values. Memory and time therefore grow with the number of subjects times the number of distinct ratings plus the square of the number of distinct ratings. This is small for ratings on a scale with a few levels but large for continuous ratings.

References

Krippendorff, Klaus. 2011. “Computing Krippendorff’s Alpha-Reliability.” Annenberg School for Communication, University of Pennsylvania. https://repository.upenn.edu/asc_papers/43

Hayes, Andrew F., and Klaus Krippendorff. 2007. “Answering the Call for a Standard Reliability Measure for Coding Data.” Communication Methods and Measures 1 (1): 77-89. https://doi.org/10.1080/19312450709336664

Examples

The example of Krippendorff (2011): 12 subjects rated by 4 raters, with 7 missing ratings.

>>> import numpy as np
>>> from statsmodels.stats.inter_rater import krippendorff_alpha
>>> nan = np.nan
>>> data = [[1, 1, nan, 1], [2, 2, 3, 2], [3, 3, 3, 3], [3, 3, 3, 3],
...         [2, 2, 2, 2], [1, 2, 3, 4], [4, 4, 4, 4], [1, 1, 2, 1],
...         [2, 2, 2, 2], [nan, 5, 5, 5], [nan, nan, 1, 1],
...         [nan, 3, nan, nan]]
>>> round(krippendorff_alpha(data), 3)
0.743
>>> round(krippendorff_alpha(data, level="interval"), 3)
0.849

Subjects are in rows. Data with raters in rows, as used by R, has to be transposed. Here 6 subjects are rated by 3 raters. The transposed matrix is accepted as 3 subjects with 6 ratings each and gives a different alpha without an error.

>>> ratings = np.array([[1, 1, 2], [2, 2, 2], [3, 3, 1],
...                     [1, 2, 1], [3, 3, 3], [2, 2, 3]])
>>> round(krippendorff_alpha(ratings), 3)
0.364
>>> round(krippendorff_alpha(ratings.T), 3)
-0.112