statsmodels.stats.inter_rater.krippendorff_alpha#
- statsmodels.stats.inter_rater.krippendorff_alpha(data, level='nominal')[source]#
Compute Krippendorff’s alpha reliability coefficient.
Agreement among any number of raters, for nominal, ordinal, interval or ratio data, with any number of missing ratings.
Added in version 0.15.1.
- Parameters:
- dataarray_like
Numeric ratings in a 2-D array with subjects in rows and raters in columns, the layout used by
statsmodels.stats.inter_rater.aggregate_raters. Missing ratings are coded asnan. Nominal categories can be any numeric codes. See examples.- level{“nominal”, “ordinal”, “interval”, “ratio”},
optional Level of measurement of the ratings. It defines the squared difference \(\delta^2_{ck}\) between two ratings \(c\) and \(k\). The default is “nominal”.
“nominal”: 0 if \(c = k\) and 1 otherwise.
“ordinal”: \((\sum_{g=c}^{k} n_g - (n_c + n_k) / 2)^2\), which depends on how often the ratings between \(c\) and \(k\) are used, not on their numeric distance.
“interval”: \((c - k)^2\).
“ratio”: \(((c - k) / (c + k))^2\). The ratings must be nonnegative.
- Returns:
floatKrippendorff’s alpha. It is 1 for perfect agreement and 0 for agreement at the level of chance. It is
nanif no subject has two or more ratings, or if all those ratings are equal, because alpha is 0 / 0 then.
See also
fleiss_kappaFleiss’ kappa for nominal ratings without missing values.
cohens_kappaCohen’s kappa for two raters.
Notes
Only subjects with at least two ratings are used. Following Krippendorff (2011), the ratings within each subject \(u\) with \(m_u\) ratings are tabulated in the coincidence matrix \(o_{ck}\): each of the \(m_u (m_u - 1)\) ordered pairs of ratings adds \(1 / (m_u - 1)\) to the cell of its two values. With the margins \(n_c = \sum_k o_{ck}\) and \(n = \sum_c n_c\),
\[\alpha = 1 - (n - 1) \frac{\sum_c \sum_k o_{ck} \delta^2_{ck}} {\sum_c \sum_k n_c n_k \delta^2_{ck}}\]For nominal data without missing ratings, alpha is a function of Fleiss’ kappa, \(\alpha = 1 - (1 - \kappa) (n - 1) / n\) (see
fleiss_kappa). For two raters, Fleiss’ kappa is Scott’s pi, so nominal alpha is 1 - (1 - pi) (n - 1) / n, which is Scott’s pi with a correction for the finite number n of ratings.If all ratings of the subjects with at least two ratings have the same value, the agreement is perfect but alpha is 0 / 0 and
nanis returned, for example for[[1, 1], [1, 1]], while[[1, 1], [2, 2]]gives 1. R’sirr::kripp.alphareturns 1 in that case.The ratings are held in a matrix of subjects by distinct values and in matrices of distinct values by distinct values. Memory and time therefore grow with the number of subjects times the number of distinct ratings plus the square of the number of distinct ratings. This is small for ratings on a scale with a few levels but large for continuous ratings.
References
Krippendorff, Klaus. 2011. “Computing Krippendorff’s Alpha-Reliability.” Annenberg School for Communication, University of Pennsylvania. https://repository.upenn.edu/asc_papers/43
Hayes, Andrew F., and Klaus Krippendorff. 2007. “Answering the Call for a Standard Reliability Measure for Coding Data.” Communication Methods and Measures 1 (1): 77-89. https://doi.org/10.1080/19312450709336664
Examples
The example of Krippendorff (2011): 12 subjects rated by 4 raters, with 7 missing ratings.
>>> import numpy as np >>> from statsmodels.stats.inter_rater import krippendorff_alpha >>> nan = np.nan >>> data = [[1, 1, nan, 1], [2, 2, 3, 2], [3, 3, 3, 3], [3, 3, 3, 3], ... [2, 2, 2, 2], [1, 2, 3, 4], [4, 4, 4, 4], [1, 1, 2, 1], ... [2, 2, 2, 2], [nan, 5, 5, 5], [nan, nan, 1, 1], ... [nan, 3, nan, nan]] >>> round(krippendorff_alpha(data), 3) 0.743 >>> round(krippendorff_alpha(data, level="interval"), 3) 0.849
Subjects are in rows. Data with raters in rows, as used by R, has to be transposed. Here 6 subjects are rated by 3 raters. The transposed matrix is accepted as 3 subjects with 6 ratings each and gives a different alpha without an error.
>>> ratings = np.array([[1, 1, 2], [2, 2, 2], [3, 3, 1], ... [1, 2, 1], [3, 3, 3], [2, 2, 3]]) >>> round(krippendorff_alpha(ratings), 3) 0.364 >>> round(krippendorff_alpha(ratings.T), 3) -0.112