Abstract
Intraclass correlation coefficients (ICCs) are frequently employed in health science research, often to assess intrarater and interrater reliability. In many cases, insufficient details are provided about these ICCs and there seem to be misunderstandings about their selection and how they should be interpreted. This paper is intended primarily to provide a clear, accessible description of ICCs, including how they should be selected, interpreted, and reported. Emphasis is given to areas where researchers seem to encounter the greatest conceptual difficulties and to exhibit the greatest misconceptions. Two extended examples are used to support the points being made. Major additional aims of this paper are to raise the awareness of authors, reviewers, and editors concerning the importance of using appropriate ICCs, and to encourage them to ensure that complete and accurate information about ICCs is reported in journal articles. Failure to do so perpetuates a risk that incorrect decisions might be made about matters that are of crucial importance for people’s health.
Similar content being viewed by others
References
Brozek, J., Alexander, H.: Components of variance and the consistency of repeated measurements. Res. Q. 18, 152–166 (1947)
Carrasco, J.L., Jover, L.: Estimating the generalized concordance correlation coefficient through variance components. Biometrics 59, 849e58 (2003)
Fleiss, J.: The Design and Analysis of Clinical Experiments. Wiley, New York (1986)
Franco, A., Malhotra, N., Simonovits, G.: Publication bias in the social sciences: unlocking the file drawer. Science 345, 1502–1505 (2014). doi:10.1126/science.1255484
Hallgren, K.A.: Computing inter-rater reliability for observational data: an overview and tutorial. Tutor. Quant. Methods Psychol. 8(1), 23–34 (2012)
Hayen, A., Dennis, R., Finch, C.: Determining the intra- and inter-observer reliability of screening tools used in sports injury research. J. Sci. Med. Sport 10, 201–210 (2007). doi:10.1016/j.jsams.2006.09.002
Kottner, J., Audigé, L., Brorson, S., Donner, A., Gajewski, B.J., Hróbjartsson, A., Roberts, C., Shoukri, M., Streiner, D.L.: Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. J. Clin. Epidemiol. 64, 96–106 (2011). doi:10.1016/j.jclinepi.2010.03.002
Krebs, D.E.: Declare your ICC type. Phys. Ther. 66, 1431 (1986)
Landers, R.N.: Computing intraclass correlations (ICC) as estimates of interrater reliability in SPSS. The Winnower 2, e143518.81744 (2015). doi:10.15200/winn.143518.81744
Lee, K.M., Lee, J., Chung, C.Y., Ahn, S., Sung, K.H., Kim, T.W., Lee, H.J., Park, M.S.: Pitfalls and important issues in testing reliability using intraclass correlation coefficients in orthopaedic research. Clin. Orthop. Surg. 4, 149–155 (2012). doi:10.4055/cios.2012.4.2.149
McAra, S.J.: Glygeryl trinitrate and toe-brachial indexes in pedal ischaemia. Unpublished Ph.D. thesis. Charles Sturt University, Albury, Australia (2015). http://primo.unilinc.edu.au/CSU:CSU_ALL:dtl_csu80101
McGraw, K.O., Wong, S.P.: Forming inferences about some intraclass correlation coefficients. Psychol. Methods 1, 30–46 (1996)
Müller, R., Büttner, P.: A critical discussion of intraclass correlation coefficients. Stat. Med. 13, 2465–2476 (1994)
Nunnally, J.C., Bernstein, I.H.: Psychometric Theory, 3rd edn. McGraw Hill, New York (1994)
Portney, L.G., Watkins, M.P.: Foundations of clinical research: applications to practice, 3rd edn. Pearson Education, Upper Saddle River (2009)
Rankin, G., Stokes, M.: Reliability of assessment tools in rehabilitation: an illustration of appropriate statistical analyses. Clin. Rehabil. 12, 187–199 (1998)
Romanos, M.T., Raspovic, A., Perrin, B.M.: The reliability of toe systolic pressure and the toe brachial index in patients with diabetes. J. Foot Ankle Res. 3, 31 (2010). doi:10.1186/1757-1146-3-31
Raykov, T., Dimitrov, D.M., von Eye, A., Marcoulides, G.A.: Interrater agreement evaluation: a latent variable modeling approach. Educ. Psychol. Meas. 73, 512–531 (2013). doi:10.1177/0013164412449016
Scanlon, C., Park, K., Mapletoft, D., Begg, L., Burns, J.: Interrater and intrarater reliability of photoplethysmography for measuring toe blood pressure and toe-brachial index in people with diabetes mellitus. J. Foot Ankle Res. 5, 13 (2012). doi:10.1186/1757-1146-5-13
Shrout, P.E., Fleiss, J.L.: Intraclass correlations: uses in assessing rater reliability. Psychol. Bull. 86, 420–428 (1979)
Sonter, J., Chuter, V., Casey, S.: Intratester and intertester reliability of toe pressure measurements in people with and without diabetes performed by podiatric physicians. J. Am. Podiatr. Med. Assoc. 105, 201–208 (2015a). doi:10.7547/0003-0538-105.3.201
Sonter, J., Sadler, S., Chuter, V.: Inter-rater reliability of automated devices for measurement of toe systolic blood pressure and the toe brachial index. Blood Press. Monit. 20, 47–51 (2015b). doi:10.1097/MBP.0000000000000083
Tehan, P.E., Bray, A., Chuter, V.H.: Non-invasive vascular assessment in the foot with diabetes: sensitivity and specificity of the ankle brachial index, toe brachial index and continuous wave Doppler for detecting peripheral arterial disease. J. Diabetes Complicat. 30, 155–160 (2015). doi:10.1016/j.jdiacomp.2015.07.019
Wrobel, J.S., Armstrong, D.G.: Reliability and validity of current physical examination techniques of the foot and ankle. J. Am. Podiatr. Med. Assoc. 98, 197–206 (2008)
Acknowledgments
I thank Michela Betta, Sylvia McAra, Melissa Nott, Rod Pope, Therese Schmid, and two anonymous reviewers for valuable comments and feedback concerning earlier drafts of this paper. Associate Professor Pope also engaged in thought-provoking interchanges, drew my attention to literature about ICCs that I had not been aware of, and shared computer output that provided confirming insights. Dr McAra granted permission to use data that had been acquired as part of her doctoral research.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
None.
Human and animal rights
This paper does not contain any studies with human participants performed by the author.
Rights and permissions
About this article
Cite this article
Trevethan, R. Intraclass correlation coefficients: clearing the air, extending some cautions, and making some requests. Health Serv Outcomes Res Method 17, 127–143 (2017). https://doi.org/10.1007/s10742-016-0156-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10742-016-0156-6