Skip to content

Instantly share code, notes, and snippets.

@skylander86
Created April 7, 2016 02:08
  • Star 14 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save skylander86/65c442356377367e27e79ef1fed4adee to your computer and use it in GitHub Desktop.
Compute Fleiss' kappa using numpy.
def fleiss_kappa(M):
"""
See `Fleiss' Kappa <https://en.wikipedia.org/wiki/Fleiss%27_kappa>`_.
:param M: a matrix of shape (:attr:`N`, :attr:`k`) where `N` is the number of subjects and `k` is the number of categories into which assignments are made. `M[i, j]` represent the number of raters who assigned the `i`th subject to the `j`th category.
:type M: numpy matrix
"""
N, k = M.shape # N is # of items, k is # of categories
n_annotators = float(np.sum(M[0, :])) # # of annotators
p = np.sum(M, axis=0) / (N * n_annotators)
P = (np.sum(M * M, axis=1) - n_annotators) / (n_annotators * (n_annotators - 1))
Pbar = np.sum(P) / N
PbarE = np.sum(p * p)
kappa = (Pbar - PbarE) / (1 - PbarE)
return kappa
#end def
@alexwal
Copy link

alexwal commented Jan 13, 2020

Is it correct that fleiss_kappa(np.array([[1, 10000000]])) should return -1.0047520386044162e-07? I expected this to have a pretty high agreement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment