Skip to main content

Advertisement

Log in

Middle and High School Student Perspectives on Digitally-Delivered Mental Health Assessments and Measurement Feedback Systems

  • Original Article
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

Implementation of measurement-based care (MBC) by child-serving community mental health providers, particularly school-based providers, is low. To inform user-centered design of measurement feedback systems (MFSs) and MBC implementation more broadly, semi-structured interviews were conducted with 61 middle and high school students. Interviews explored student preferences for and perceived helpfulness of different assessment methods and use of MFS in counseling. Results indicate that student preference for digitally-administered assessment is equivocal, with preferences being influenced by student perceptions of the ease of use, impersonalization, and confidentiality. Students with exposure to the MFS found it helpful when used by their provider to share assessment feedback.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Alpert, J. S. (2016). The electronic medical record in 2016: Advantages and disadvantages. Digital Medicine,2, 48–51. https://doi.org/10.4103/2226-8561.189504.

    Article  Google Scholar 

  • American Psychological Association. (2008). Task Force on Evidence-Based Practice with Children and Adolescents.

  • Bergman, H., Kornor, H., Nikolakopoulou, A., Hanssen-Bauer, K., Soares-Weiser, K., Tollefesen, T. K., & Bjorndal, A. (2018). Client feedback in psychological therapy for children and adolescents with mental health problems. Cochrane Database of Systematic Reviews 2018, Issue 8. Art. No.: CD011729. https://doi.org/10.1002/14651858.CD011729.pub2.

  • Bickman, L. (2008). A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child and Adolescent Psychiatry,47(10), 1114. https://doi.org/10.1097/CHI.0b013e3181825af8.

    Article  PubMed  Google Scholar 

  • Bickman, L., Douglas, S., & R.,…, Hoagwood, K., (2016). Implementing a measurement feedback system: A tale of two sites. Administration and Policy in Mental Health,43, 410–425. https://doi.org/10.1007/s10488-015-0647-8.

    Article  Google Scholar 

  • Bickman, L., Kelley, S. D., Breda, C., de Andrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomized trial. Psychiatric Services,62(12), 1423–1429. https://doi.org/10.1176/appi.ps.002052011.

    Article  PubMed  Google Scholar 

  • Boo, J., & Vispoel, W. (2012). Computer versus paper-and-pencil assessment of educational development: A comparison of psychometric features and examinee preferences. Psychological Reports,111(2), 443–460. https://doi.org/10.2466/10.03.PRO.111.5.443-460.

    Article  PubMed  Google Scholar 

  • Bowling, A. (2005). Mode of questionnaire administration can have serious effects on data quality. Journal of Public Health,27(3), 281–291. https://doi.org/10.1093/pubmed/fdi031.

    Article  PubMed  Google Scholar 

  • Chorpita, B. F., & Weisz, J. R. (2009). Modular approach to therapy for children with anxiety, depression, trauma, or conduct problems (MATCH-ADTC). Satellite Beach, FL: PracticeWise.

    Google Scholar 

  • Connors, E. H., Arora, P., Curtis, L. N., & Stephan, S. H. (2015). Evidence-based assessment in school mental health. Cognitive and Behavioral Practice,22(1), 60–73. https://doi.org/10.1016/j.cbpra.2014.03.008.

    Article  Google Scholar 

  • Courage, C., & Baxter, K. (2005). Understanding your users: A practical guide to user requirements methods, tools, and techniques. San Francisco: Morgan Kaufmann Publishers.

    Google Scholar 

  • Duong, M. T., Lyon, A. R., Ludwig, K., Wasse, J. K., & McCauley, E. (2016). Student perceptions of the acceptability and utility of standardized and idiographic assessment in school mental health. International Journal of Mental Health Promotion,18(1), 19–63. https://doi.org/10.1080/14623730.2015.1079429.

    Article  PubMed  Google Scholar 

  • Dyrborg, J., Larsen, F. W., Nielson, S., Byman, J., Nielson, B. B., & Gautre-Delay, F. (2000). The Children’s Global Assessment Scale (CGAS) and Global Assessment of Psychosocial Disability (GAPD) in clinical practice: Substance and reliability as judged by intraclass correlations. European Child and Adolescent Psychiatry,9(3), 195–201. https://doi.org/10.1007/s007870070043.

    Article  CAS  PubMed  Google Scholar 

  • Eisen, S. V., Dickey, B., & Sederer, L. I. (2000). A self-report symptom and problem rating scale to increase inpatients’ involvement in treatment. Psychiatric Services,51(3), 349–353. https://doi.org/10.1176/appi.ps.51.3.349.

    Article  CAS  PubMed  Google Scholar 

  • Fleming, M. (1997). Evaluating adolescent responses to a computer health information system. Journal of Adolescent Health,20, 166. https://doi.org/10.1019/S1054-139X(97)87625-2.

    Article  Google Scholar 

  • Fouladi, R. T., McCarthy, C. J., & Moller, N. (2002). Paper-and-pencil or online? Evaluating mode effects on measures of emotional functioning and attachment. Assessment,9, 204–215. https://doi.org/10.1177/10791102009002011.

    Article  PubMed  Google Scholar 

  • Gleacher, A. A., Olin, S. S., …, Hoagwood, K. (2016). Implementing a measurement feedback system in community mental health clinics: A case study of multilevel barriers and facilitators. Administration and Policy in Mental Health,43, 426–440. https://doi.org/10.1007/s10488-015-0642-0.

    Article  Google Scholar 

  • Gwaltney, C. J., Shields, A. L., & Shiffman, S. (2008). Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: A meta-analytic review. Value in Health,11(2), 322–333. https://doi.org/10.1111/j.1524-4733.2007.00231.x.

    Article  PubMed  Google Scholar 

  • Haynes, S. N., Mumma, G. H., & Pinson, C. (2009). Idiographic assessment: Conceptual and psychometric foundations of individualized behavioral assessment. Clinical Psychology Review,29, 179–191. https://doi.org/10.1016/j.cpr.2008.12.003.

    Article  PubMed  Google Scholar 

  • Hsieh, H.-F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research,15(9), 1277–1288. https://doi.org/10.1177/1049732305276687.

    Article  PubMed  Google Scholar 

  • Jensen-Doss, A., Becker, E. M., Smith, A. M., Lyon, A. R., Lewis, C. C., Stanick, C. F., et al. (2018). Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Administration and Policy in Mental Health and Mental Health Services Research,45, 48–61. https://doi.org/10.1007/s10488-016-0763-0.

    Article  PubMed  Google Scholar 

  • Jenson-Doss, A., & Hawley, K. M. (2011). Understanding clinicians’ diagnostic practices: Attitudes toward the utility of diagnosis and standardized diagnostic tools. Administration and Policy in Mental Health,38(6), 476–485. https://doi.org/10.1007/s10488-011-0334-3.

    Article  Google Scholar 

  • Jensen-Doss, A., & Weisz, J. (2008). Diagnostic agreement predicts treatment process and outcomes in youth mental health clinics. Journal of Consulting and Clinical Psychology,76(5), 711–722. https://doi.org/10.1037/0022-006X.76.5.711.

    Article  PubMed  Google Scholar 

  • Kazak, A. E., Hoagwood, K., Weisz, J. R., Hood, K., Kratochwill, T. R., Vargas, L. A., et al. (2010). A meta-systems approach to evidence-based practice for children and adolescents. American Psychologist,65(2), 85–97. https://doi.org/10.1037/a0017784.

    Article  PubMed  Google Scholar 

  • Kendrick, T., El-Gohary, M., Stuart, B., Gilbody, S., Churchill, R., Aiken, L.,…, Moore, M. (2016). Routine use of patient reported outcome measures (PROMs) for improving treatment of common mental health disorders in adults. Cochrane Database of Systematic Reviews, Issue 7, Art. No.: CD011119. https://doi.org/10.1002/14651858.CD011119.pub2.

  • Kokkonen, E. W. J., Davis, S. A., Lin, H.-C., Dabade, T. S., Feldeman, S. R., & Fleischer, A. B. (2013). Use of electronic medical records differs by specialty and office settings. Journal of American Medical Informatics Association,20, 33–38. https://doi.org/10.1136/amiajnl-2012-001609.

    Article  Google Scholar 

  • Lambert, M. J., Whipple, J. L., Vermeersch, D. A., Smart, D. W., Hawkins, E. J., Nielsen, S. L., et al. (2002). Enhancing psychotherapy outcomes via providing feedback on client progress: A replication. Clinical Psychology and Psychotherapy,9(2), 91–103. https://doi.org/10.1002/cpp.324.

    Article  Google Scholar 

  • Lavin, P., Barry, L., & Williams, S. (2017). Measurement-based care in behavioral health [power point slides]. https://www.jointcommission.org/assets/1/6/bhc_Joint_Commission_measures_webinar_041117.pdf.

  • Lewis, C. C., Boyd, M., Puspitasari, A., Navarro, E., Howard, J., Kassab, H., …, Kroenke, K. (2019). Implementing measurement-based care in behavioral health: A review. JAMA Psychiatry, 76(3), 324–335. https://doi-org.flagship.luc.edu/10.1001/jamapsychiatry.2018.3329.

    Article  Google Scholar 

  • Lindhiem, O., Bennett, C. B., Orimoto, T. E., & Kolko, D. J. (2016). A meta-analysis of personalized treatment goals in psychotherapy: A preliminary report and call for more studies. Clinical Psychology: Science and Practice,23(2), 165–176.

    Google Scholar 

  • Lyon, A. R., Dorsey, S., Pullmann, M., Silbaugh-Cowdin, J., & Berliner, L. (2015). Clinician use of standardized assessments following a common elements psychotherapy training and consultation program. Administration and Policy in Mental Health and Mental Health Services Research,42(1), 47–60.

    Article  Google Scholar 

  • Lyon, A. R., Knaster Wasse, J., Ludwig, K., Zachry, M., Bruns, E. J., Unutzer, J., et al. (2016a). The Contextualized Technology Adaptation Process (CTAP): Optimizing health information technology to improve mental health systems. Administration and Policy in Mental Health and Mental Health Services Research,43(3), 394–409. https://doi.org/10.1007/s10488-015-0637-x.

    Article  PubMed  Google Scholar 

  • Lyon, A. R., & Lewis, C. C. (2016). Designing health information technologies for uptake: Development and implementation of measurement feedback systems in mental health service delivery. Administration and Policy in Mental Health,43, 344–349. https://doi.org/10.1007/s10488-015-0704-3.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lyon, A. R., Lewis, C. C., Boyd, M. R., Hendrix, E., & Liu, F. (2016b). Capabilities and characteristics of digital measurement feedback systems: Results from a comprehensive review. Administration and Policy in Mental Health and Mental Health Services Research,43, 441–466. https://doi.org/10.1007/s10488-016-0719-4.

    Article  PubMed  Google Scholar 

  • Lyon, A. R., Lewis, C. C., Melvin, A., Boyd, M., Nicodimos, S., Liu, F. F., et al. (2016c). Health Information Technologies-Academic and Commercial Evaluation (HIT-ACE) methodology: Description and application to clinical feedback systems. Implementation Science,11(1), 1–18. https://doi.org/10.1186/s13012-016-0495-2.

    Article  Google Scholar 

  • Lyon, A. R., Ludwig, K., Knaster Wasse, J., Bergstrom, A., Hendrix, E., & McCauley, E. (2016d). Determinants and functions of standardized assessment use among school mental health clinicians: A mixed methods evaluation. Administration and Policy in Mental Health and Mental Health Services Research,43, 122–134. https://doi.org/10.1007/s10488-015-0626-0.

    Article  PubMed  Google Scholar 

  • Lyon, A. R., Pullmann, M. D., Whitaker, K., Ludwig, K., Knaster Wasse, J., & McCauley, E. (2019). A digital feedback system to support implementation of measurement-based care by school-based mental health clinicians. Journal of Clinical Child and Adolescent Psychology,48, 168–179. https://doi.org/10.1080/15374416.2017.1280808.

    Article  Google Scholar 

  • Mangunkusomo, R. T., Duisterhout, J. S., De Graaff, N., Maarsingh, E. J., de Koning, H. J., & Raat, H. (2006). Internet versus paper mode of health and health behavior questionnaires in elementary schools: Asthma and fruit as examples. Journal of School Health,76, 80–86.

    Article  Google Scholar 

  • Mangunkusomo, R. T., Moorman, P. W., Berg-de Ruiter, A. E., & Van Der Lei, J. (2005). Internet-administered adolescent questionnaires compared with a paper version in a randomized study. Journal of Adolescent Health,36(1), 71–76. https://doi.org/10.1016/j.jadohealth.2004.02.020.

    Article  Google Scholar 

  • Menachemi, N., & Collum, T. H. (2011). Benefits and drawbacks of electronic record systems. Risk Management and Healthcare Policy,4, 47–55. https://doi.org/10.2147/RMHP.S12985.

    Article  PubMed  PubMed Central  Google Scholar 

  • Merikangas, K. R., He, J., Burstein, M., Swendsen, J., Avenevoli, S., Case, B., …, Olfson, M. (2011). Service utilization for lifetime mental disorders in U.S. adolescents: Results of the national comorbidity survey-adolescent supplement (NCS-A). Journal of the American Academy of Child and Adolescent Psychiatry, 50(1), 32–45. https://doi.org/10.1016/j.jaac.2010.10.006.

    Article  Google Scholar 

  • Muhr, T. (2004). User’s manual for ALTAS.ti 5.0. Berlin: ATLAS.ti. Scientific Software Development GmbH.

  • Nadeem, E., Cappella, E., Holland, S., Coccaro, C., & Crisonino, G. (2016). Development and piloting of a classroom-focused measurement feedback system. Administration and Policy in Mental Health,43, 379–393.

    Article  Google Scholar 

  • Norman, D. A., & Draper, S. W. (Eds.). (1986). User-centered system design: New perspectives on human–computer interaction. Hillsdale, NJ: Lawrence Earlbaum Associates.

    Google Scholar 

  • Palinkas, L. A., Aarons, G. A., Horwitz, S., Chamberlain, P., Hurlburt, M., & Landsverk, J. (2011). Mixed methods designs in implementation research. Administration and Policy in Mental Health,38, 44–53. https://doi.org/10.1007/s10488-010-0314-z.

    Article  PubMed  Google Scholar 

  • Poston, J. M., & Hanson, W. E. (2010). Meta-analysis of psychological assessment as a therapeutic intervention. Psychological Assessment,22, 203–212. https://doi.org/10.1037/a0018679.

    Article  PubMed  Google Scholar 

  • Powell, B. J., Beidas, R. S., Lewis, C. C., Aarons, G. A., McMillen, J. C., Proctor, E. K., et al. (2017). Methods to improve the selection and tailoring of implementation strategies. The Journal of Behavioral Health Services and Research,44(2), 177–194.

    Article  Google Scholar 

  • Rakovshik, S. G., & McManus, F. (2010). Establishing evidence-based training in cognitive behavioral therapy: A review of current empirical findings and theoretical guidance. Clinical Psychology Review,30(5), 496–516. https://doi.org/10.1016/j.cpr.2010.03.004.

    Article  PubMed  Google Scholar 

  • Scott, K., & Lewis, C. C. (2015). Using measurement-based care to enhance any treatment. Cognitive and Behavioral Practice,22, 49–59. https://doi.org/10.1016/j.cbpra.2014.01.010.

    Article  PubMed  PubMed Central  Google Scholar 

  • Seidman, E., Chorpita, B. F., Reay, W. E., Stelk, W., Garland, A. F., Jutash, K., …, Ringeisen, H. (2010). A framework for measurement feedback to improve decision-making in mental health. Administration Policy in Mental Health,37, 128–131. https://doi.org/10.1007/s10488-009-0260-9.

    Article  Google Scholar 

  • Shaffer, D., Gould, M. S., Brasic, J., Ambrosini, P., Fisher, P., Bird, H., et al. (1983). A children’s global assessment scale (CGAS). Archives of General Psychiatry,40(1), 1223–1231. https://doi.org/10.1001/archpsyc.1983.017901000074010.

    Article  Google Scholar 

  • Tharinger, D. J., Finn, S. E., Gentry, L., Hamilton, A., Fowler, J., Matson, M., …, Walkowiak, J. (2009). Therapeutic assessment with children: A pilot study of treatment acceptability and outcome. Journal of Personality Assessment,91, 238–244. https://doi.org/10.1080/00223890902794275.

    Article  Google Scholar 

  • Unützer, J., Chan, Y. F., Hafer, E., Knaster, J., Shields, A., Powers, D., et al. (2012). Quality improvement with pay-for performance incentives in integrated behavioral health care. American Journal of Public Health,102(6), e41–e45.

    Article  Google Scholar 

  • Unützer, J., Katon, W., Callahan, C. M., Williams, J. W., Jr., Hunkeler, E., Harpole, L., et al. (2002). Collaborative care management of late-life depression in the primary care setting: A randomized controlled trial. JAMA,288(22), 2836–2845.

    Article  Google Scholar 

  • Vallejo, M. A., Mananes, G., Comeche, M. I., & Diaz, M. I. (2008). Comparison between administration via internet and paper-and-pencil administration of two clinical instruments: SCL-90-R and GHQ-28. Journal of Behavioral Therapy and Experimental Psychiatry,39(3), 201–208. https://doi.org/10.1016/j.jbtep.2007.04.001.

    Article  Google Scholar 

  • Van de Looij-Jansen, P. M., & Wilde, E. J. D. (2008). Comparison of web-based versus paper-and-pencil self-administered questionnaire: Effects on health indicators in Dutch adolescents. Health Services Research,43(5), 1708–1721. https://doi.org/10.1111/j.1475-6773.2008.00860.

    Article  PubMed Central  Google Scholar 

  • Vispoel, W. P., Boo, J., & Bieiler, T. (2001). Computerized and paper-and-pencil versions of the Rosenberg Self-Esteem Scale: A comparison of psychometric features and respondent preferences. Educational and Psychological Measurement,61(3), 461–474. https://doi.org/10.1177/00131640121971329.

    Article  Google Scholar 

  • Williams, J. W., Katon, W., Lin, E. H., Nöel, P. H., Worchel, J., Cornell, J., et al. (2004). The effectiveness of depression care management on diabetes-related outcomes in older patients. Annals of Internal Medicine,140(12), 1015–1024.

    Article  Google Scholar 

Download references

Acknowledgements

This publication was made possible in part by funding from Grant K08MH095939, awarded to the last author (Aaron R. Lyon) from the National Institute of Mental Health (NIMH). The authors would also like to thank the participating students, Seattle Children’s Hospital, and Public Health of Seattle and King County for their support of this project. This paper was not previously presented at a meeting.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ashley M. Mayworm.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the Ethical Standards of the Institutional Review Board of the University of Washington (43941 UW IRB) and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in this study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mayworm, A.M., Kelly, B.M., Duong, M.T. et al. Middle and High School Student Perspectives on Digitally-Delivered Mental Health Assessments and Measurement Feedback Systems. Adm Policy Ment Health 47, 531–544 (2020). https://doi.org/10.1007/s10488-020-01010-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-020-01010-9

Keywords

Navigation