Meta-analysis of Cohen’s kappa

Abstract

Cohen’s κ is the most important and most widely accepted measure of inter-rater reliability when the outcome of interest is measured on a nominal scale. The estimates of Cohen’s κ usually vary from one study to another due to differences in study settings, test properties, rater characteristics and subject characteristics. This study proposes a formal statistical framework for meta-analysis of Cohen’s κ to describe the typical inter-rater reliability estimate across multiple studies, to quantify between-study variation and to evaluate the contribution of moderators to heterogeneity. To demonstrate the application of the proposed statistical framework, a meta-analysis of Cohen’s κ is conducted for pressure ulcer classification systems. Implications and directions for future research are discussed.

Appendices

Appendix A

An example illustrating how to calculate Cohen’s κ, its variance and confidence intervals using real data from Nixon et al. (2005)

	Ward nurse
	No pressure ulcer	Pressure ulcer	Total
Clinical research nurse
No pressure ulcer	(a) 2175	(b) 35	(f 1) 2210
Pressure ulcer	(c) 42	(d) 144	(f 2) 186
Total	(g 1) 2217	(g 2) 179	(n) 2396

$$ p_{0} = \frac{a + d}{n} = \frac{2175 + 144}{2396} \approx 0.97 $$

$$ p_{c} = \frac{{\frac{{f_{1} \times g_{1} }}{n} + \frac{{f_{2} \times g_{2} }}{n}}}{n} = \frac{{\frac{2210 \times 2217}{2396} + \frac{186 \times 179}{2396}}}{2396} \approx 0.86 $$

$$ \hat{\kappa } = \frac{{p_{0} - p_{c} }}{{1 - p_{c} }} = \frac{0.97 - 0.86}{1 - 0.86} \approx 0.79 $$

$$ Var(\hat{\kappa }) = \frac{1}{{(1 - p_{c} )^{2} }}\frac{{p_{0} (1 - p_{0} )}}{n} = \frac{1}{{(1 - 0.86)^{2} }}\frac{0.97(1 - 0.97)}{2396} \approx 6.20 \times 10^{ - 4} $$

$\hat{\kappa } \pm 1.96 \times \sqrt {\text{var} (\kappa )} \approx 0.79 \pm 1.96 \times 0.02$. Therefore, 95% CI for Cohen’s κ is [0.75, 0.83].

Note that, the calculation of Var($\hat{\kappa }$) depends on p _c which is usually not reported in research articles. When p ₀ and $\hat{\kappa }$ are reported, p _c can be derived by $p_{c} = \frac{{p_{0} - \hat{\kappa }}}{{1 - \hat{\kappa }}}$.

Appendix B

R codes for all analyses conducted in the meta-analysis of Cohen’s κ for PU classification systems

#Install the package metafor

library(metafor)

#Derive vi, the variance for each kappa estimate

#p0i and ki are the percentage agreement and Cohen’s kappa estimate for study i

# pci is the chance agreement derived from p0i and ki (as shown in Appendix A)

pci=(p0i-ki)/(1-ki)

vi=p0i*(1-p0i)/((1-pci)^2*ni)

#Fit the random-effects model using rma() and get confidence intervals for parameter estimates

res<-rma(ki,vi,data=PUdata)

confint(res)

#Get the forest plot

forest(res,at=c(-0.2,0,0.2,0.4,0.6,0.8,1,1.2), slab = paste(PUdata$Study))

op<-par(cex=1,font=2)

text(-1.25,17, “Study”,pos=4)

text(1.5,17, “Cohen’s Kappa [95% CI]”,pos=4)

#Get the funnel plot to check publication bias

funnel(res, main = “Random-Effects Model”)

#Sort the data matrix by the moderator variable rater for the mixed-effects model

data<-PUdata[order(PUdata$rater),]

#Fit the mixed-effects model using rma() and get confidence intervals for parameter estimates

mix<-rma(ki,vi,mods=~factor(rater),data=data)

confint(mix)

#Get the forest plot for mixed-effect model and add group estimates to the bottom of the plot

forest(data$ki,data$vi, at=c(-0.2,0,0.2,0.4,0.6,0.8,1,1.2),ylim=c(-3,18),slab = paste(data$Study))

preds <- predict(mix, newmods = c(0, 1))

op<-par(cex=1.1,font=1)

addpoly(preds$pred, sei = preds$se, mlab = c(“Raters without training”, “Raters with training”))

abline(h=0)

abline(h=10.5)

text(1.2,0.4,”Raters with training”)

text(1.2,11,”Raters without training”)

op<-par(cex=1.1,font=2)

text(-1.25,17, “Study”,pos=4)

text(1.5,17, “Cohen’s Kappa [95% CI]”,pos=4)

# Use trim and fill method to adjust for publication bias under the random-effects model

re<-rma(ki,vi,data=PUdata,method=“REML”)

rtf <- trimfill(re)

# Use trim and fill method to adjust for publication bias under the fixed-effects model

fe<-rma(ki,vi,data=PUdata,method=“FE”)

ftf <- trimfill(fe)

#Get the funnel plot with augmented data

funnel(ftf)

abline(v=1)