Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Compute Fleiss' kappa using numpy.
def fleiss_kappa(M):
"""
See `Fleiss' Kappa <https://en.wikipedia.org/wiki/Fleiss%27_kappa>`_.
:param M: a matrix of shape (:attr:`N`, :attr:`k`) where `N` is the number of subjects and `k` is the number of categories into which assignments are made. `M[i, j]` represent the number of raters who assigned the `i`th subject to the `j`th category.
:type M: numpy matrix
"""
N, k = M.shape # N is # of items, k is # of categories
n_annotators = float(np.sum(M[0, :])) # # of annotators
p = np.sum(M, axis=0) / (N * n_annotators)
P = (np.sum(M * M, axis=1) - n_annotators) / (n_annotators * (n_annotators - 1))
Pbar = np.sum(P) / N
PbarE = np.sum(p * p)
kappa = (Pbar - PbarE) / (1 - PbarE)
return kappa
#end def
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.