Abstract
Reliability of well-known instruments was examined in 202 people with severe mental illness participating in a multisite vocational study. We examined interrater reliability of the Positive and Negative Syndrome Scale (PANSS) and the internal consistency and test-retest reliability of the PANSS, the Rosenberg Self-Esteem Scale, the Medical Outcomes Study Short Form-36 (SF-36), and the Quality of Life Interview. Most scales had good levels of reliability, with intraclass correlation coefficients (ICCs) and coefficient alphas above .70. However, the SF-36 scales were generally less stable over time, particularly Social Functioning (ICC = .55). Test-retest reliability was lower among less educated respondents and among ethnic minorities. We recommend close monitoring of psychometric issues in future multisite studies.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
REFERENCES
Alsawalmeh, Y. M., & Feldt, L. S. (1992). Test of the hypothesis that the intraclass reliability coefficient is the same for two measurement procedures. Applied Psychological Measurement, 16, 195–205.
Alsawalmeh, Y. M., & Feldt, L. S. (1994).Testing the equality of two related intraclass reliability coefficients. Applied Psychological Measurement, 18, 183–190.
Arns, P. G., & Linney, J. A. (1993). Work, self, and life satisfaction for persons with severe and persistent mental disorders. Psychosocial Rehabilitation Journal, 17, 63–79.
Bartko, J. J., & Carpenter, W. T. (1976). On the methods and theory of reliability. Journal of Nervous and Mental Disease, 163, 307–317.
Bond, G. R., Drake, R. E., Mueser, K. T., & Becker, D. R. (1997). An update on supported employment for people with severe mental illness. Psychiatric Services, 48, 335–346.
Bond, G. R., Resnick, S. R., Drake, R. E., Xie, H., McHugo, G. J., & Bebout, R. R. (2001). Does competitive employment improve nonvocational outcomes for people with severe mental illness? Journal of Consulting and Clinical Psychology, 69.
Burke, J. D., Burke, K. C., Baker, J. H., and Hillis, A. (1995). Testretest reliability in psychiatric patients of the SF-36 health survey. International Journal of Methods in Psychiatric Research, 5, 189–194.
Charter, R. A., & Feldt, L. S. (1996). Testing the equality of two alpha coefficients. Perceptual and Motor Skills, 82, 763–768.
De Vellis, R. F. (1991). Applied social research methods series: Vol. 26. Scale development: Theory and applications. Newbury Park, CA: Sage.
Drake, R. E., McHugo, G. J., & Biesanz, J. C. (1995). The Testretest reliability of standardized instruments among homeless persons with substance use disorders. Journal of Studies on Alcohol, 56, 161–167.
Feldt, L. S. (1969). A test of the hypothesis that Cronbach's alpha or Kuder-Richardson coefficient twenty is the same for two tests. Psychometrika, 34, 363–373.
Ferring, D., & Filipp, S. (1996). Measurement of self-esteem: Findings on reliability, validity, and stability of the Rosenberg Scale. Diagnostica, 42, 284–292.
Guenzel, P. J., Berckmans, T. R., & Cannell, C. F. (1983). General Interviewing Techniques: A Self-Instructional Workbook for Telephone and Personal Interviewer Training. Ann Arbor, MI: Survey Research Center of the Institute for Social Research, The University of Michigan.
Hays, R.D., Sherbourne, C.D., & Mazel, R. M. (1993). TheRAND 36-item health survey 1.0. Health Economics, 2, 217–227.
Kay, S. R., Fiszbein, A., & Opler, L. A. (1987). The Positive and Negative Syndrome Scale (PANSS) for schizophrenia. Schizophrenia Bulletin, 13, 261–276.
Kay, S. R., Opler, L. A., & Lindenmayer, J. P. (1989). The positive and negative syndrome scale (PANSS): Rationale and standardisation. British Journal of Psychiatry, 155(Suppl. 7), 59–65.
Kelly, J. R., & McGrath, J. E. (1988). Applied social research methods series: Vol. 13. On time and method. Newbury Park, CA: Sage.
Lehman, A. F. (1988).Aquality of life interview for the chronically mentally ill. Evaluation and Program Planning, 11, 51–62.
Lehman, A. F. (1996). Quality of life interview. In L. I. Sederer & B. Dickey (Eds.), Outcomes assessment in clinical practice (pp. 117–119). Baltimore, MD: Williams & Wilkins.
Leidy, N. K., Palmer, C., Murray, M., Robb, J., & Revicki, D. A. (1998). Health-related quality of life assessment in euthymic and depressed patients with bipolar disorder: Psychometric performance of four self-report measures. Journal of Affective Disorders, 48, 207–214.
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46.
McHorney, C. A., Ware, J. E., Lu, R., & Sherbourne, C. D. (1994). The MOS 36-item short-form health survey (SF-36): III. Tests of data quality, scaling assumptions, and reliability across diverse patients groups. Medical Care, 32, 40–66.
Nunnally, J.C. (1978). Psychometric theory (2nd ed.). NY: McGraw-Hill. Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Erlbaum.
Pollner, M. (1998). The effects of interviewer gender in mental health interviews. The Journal of Nervous and Mental Disease, 186, 369–373.
Rosenberg, M. (1965). The measurement of self-esteem. In Society and the adolescent self-image (pp. 16–36). Princeton, NJ: Princeton University Press.
Russo, J., Roy-Byrne, P., Reeder, D., Alexander, M., Dwyer-O'Connor, E., Dagadakis, C., Ries, R., & Patrick, D. (1997). Longitudinal assessment of quality of life in acute psychiatric inpatients: Reliability and validity. The Journal of Nervous and Mental Disease, 185, 166–175.
Russo, J., Trujillo, C. A., Wingerson, D., Decker, K., Ries, R., Wetzler, H., & Roy-Byrne, P. (1998). The MOS 36-item short form health survey: Reliability, validity, and preliminary findings in schizophrenic outpatients. Medical Care, 36, 752–756.
Salyers, M. P., Bosworth, H. B., Swanson, J.W., Lamb-Pagone, J., & Osher, F.C. (2000). Reliability and validity of the SF-12 Health Survey among people with severe mental illness. Medical Care, 38, 1141–1150.
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.
Torrey, W. C., Mueser, K. T., McHugo, G. J., & Drake, R. E. (2000). Self-esteem as an outcome measure in vocational rehabilitation studies of adults with severe mental illness. Psychiatric Services, 51, 229–233.
Tracy, K., Adler, L. A., Rotrosen, J., Edson, R., Lavori, P., & the Veterans Affairs Cooperative Study #394 Study Group. (1997). Interrater reliability issues in multicenter trials, Part I: Theoretical concepts and operational procedures used in department of veterans affairs cooperative study #394. Psychopharmacology Bulletin, 33, 53–57.
VanDongen, C. J. (1996). Quality of life and self-esteem in working and nonworking persons with mental illness. Community Mental Health Journal, 32, 535–548.
Ware, J. E., Kosinski, M., Bayliss, M. S., McHorney, C. A., Rogers, W. H., & Raczek, A. (1995). Comparison of methods for the scoring and statistical analysis of SF-36 health profile and summary measures: Summary of results from the Medical Outcomes Study. Medical Care, 33, AS264–279.
Ware, J. E., & Sherbourne, C. D. (1992). The MOS 36-item short health survey (SF-36): I. Conceptual framework and item selection. Medical Care, 30, 473–481.
Wood, P. A., Hurlburt, M. S., Hough, R. L., & Hofstetter, C. R. (1997). Health status and functioning among the homeless mentally ill: An assessment of the Medical Outcomes Study SF-36 scales. Evaluation and Program Planning, 20, 151–161.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Salyers, M.P., McHugo, G.J., Cook, J.A. et al. Reliability of Instruments in a Cooperative, Multisite Study: Employment Intervention Demonstration Program. Ment Health Serv Res 3, 129–139 (2001). https://doi.org/10.1023/A:1011519514465
Issue Date:
DOI: https://doi.org/10.1023/A:1011519514465