Instantly share code, notes, and snippets.

# even4void/reliability.R

Created October 17, 2010 19:36
Some illustrations of reliability analysis.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
 # Some illustrations of reliability analysis. # Time-stamp: <2010-10-17 21:35:51 chl> x <- c(12,8,22,10,10,6,8,4,14,6,2,22,12,7,24,14,8,4,5,6,14,5,5,16) GHQ <- data.frame(subject=gl(12,1,24), time=gl(2,12), score=x) # ---------------------------------------------------- # Computing reliability (Rx) by hand from ANOVA tables # ---------------------------------------------------- # one-way ANOVA (assuming no time trends) summary(aov(score ~ subject, data=GHQ)) # Rx is (73.758-3.667)/(73.758+3.667) ## 0.905 # two-way ANOVA summary(aov(score ~ subject + time, data=GHQ)) # Assuming raters are sampled from a larger pool of raters, Rx is 12*(73.758-3.939)/(12*73.758+2*0.667+10*3.939) ## 0.905 # If a single rater is to be used for all subjects, Rx becomes (73.758-3.939)/(73.758+3.939) ## 0.899 # ------------------------------------ # Computing ICC with the `psy` package # ------------------------------------ library(psy) with(GHQ, icc(cbind(score[time=="1"],score[time=="2"]))) # 'consistency' is the same as Rx computed from two-way ANVOA ## 0.899 # 'agreement' is the same as Rx computed from one-way ANOVA ## 0.905 # ---------------------- # Computing ICC from LMM # ---------------------- library(lme4) # a random-intercept model (equivalent to the one-way ANOVA) summary(lmer(score ~ 1 + (1|subject), data=GHQ)) # ICC is computed as 5.9199^2/(5.9199^2+1.9149^2) ## 0.905 # a mixed-effects model (equivalent to the two-way ANOVA) summary(lmer(score ~ time + (1|subject), data=GHQ)) # ICC is computed as 5.9084^2/(5.9084^2+1.9848^2) ## 0.899 detach(package:lme4) # the same models as above with the `nlme` package library(nlme) intervals(lme(score ~ 1, random= ~1|subject, data=GHQ)) # ICC is computed as 5.919918^2/(5.919918^2+1.914855^2) ## 0.905 intervals(lme(score ~ time, random= ~1|subject, data=GHQ)) # ICC is computed as 5.908387^2/(5.908387^2+1.984792^2) ## 0.899

### even4void commented Oct 17, 2010

This R script illustrates how to compute reliability from test-retest study using ANOVA or mixed-effects models. Data consist in GHQ scores for 12 clinical psychology students, taken at two occasions three-days apart (Dunn, 1992).

Dunn, G (1992). Design and analysis of reliability studies. Statistical Methods in Medical Research, 1, 123-157.