Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
Compute Fleiss' kappa using numpy.
def fleiss_kappa(M):
See `Fleiss' Kappa <>`_.
:param M: a matrix of shape (:attr:`N`, :attr:`k`) where `N` is the number of subjects and `k` is the number of categories into which assignments are made. `M[i, j]` represent the number of raters who assigned the `i`th subject to the `j`th category.
:type M: numpy matrix
N, k = M.shape # N is # of items, k is # of categories
n_annotators = float(np.sum(M[0, :])) # # of annotators
p = np.sum(M, axis=0) / (N * n_annotators)
P = (np.sum(M * M, axis=1) - n_annotators) / (n_annotators * (n_annotators - 1))
Pbar = np.sum(P) / N
PbarE = np.sum(p * p)
kappa = (Pbar - PbarE) / (1 - PbarE)
return kappa
#end def

This comment has been minimized.

Copy link

alexwal commented Jan 13, 2020

Is it correct that fleiss_kappa(np.array([[1, 10000000]])) should return -1.0047520386044162e-07? I expected this to have a pretty high agreement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.