Skip to main content
Mijn bibliotheek
Shop
Nieuws Boeken Tijdschriften E-learning Web-tv Video Audio
Tips voor Mijn BSL
UITGEBREID ZOEKEN
Inloggen
Top
Quality of Life Research
Gepubliceerd in: Quality of Life Research 3/2012

01-04-2012

The inclusion of ‘then-test’ questions in post-test questionnaires alters post-test responses: a randomized study of bias in health program evaluation

Auteurs: Sandra Nolte, Gerald R. Elsworth, Andrew J. Sinclair, Richard H. Osborne

Gepubliceerd in: Quality of Life Research | Uitgave 3/2012

Log in om toegang te krijgen
share
DELEN

Deel dit onderdeel of sectie (kopieer de link)

  • Optie A:
    Klik op de rechtermuisknop op de link en selecteer de optie “linkadres kopiëren”
  • Optie B:
    Deel de link per e-mail
    Link delen
publish
Publiceer nu

Abstract

Objectives

Program evaluations are frequently based on ‘then-test’ data, i.e., pre-test collected in retrospect. While the application of the then-test has practical advantages, little is known about the validity of then-test data. Because of the collection of then-test in close proximity to post-test questions, this study was aimed at exploring whether the presence of then-test questions in post-test questionnaires influenced subjects’ responses to post-test.

Patients and methods

To test the influence of then-test questions, we designed a randomized three-group study in the context of chronic disease self-management programs. Interventions had comparable goals and philosophies, and all 949 study participants filled out identical Health Education Impact Questionnaires (heiQ) at pre-test. At post-test, participants were then randomized to one of the following three groups: Group A responded to post-test questions only (n = 331); Group B filled out transition questions in addition to post-test (n = 304); and Group C filled out then-test questions in addition to post-test (n = 314).

Results

Significant post-test differences were found in six of eight heiQ scales, with respondents who filled out then-test questions reporting significantly higher post-test scores than respondents of the other groups.

Conclusions

This study provides evidence that the inclusion of then-test questions alters post-test responses, suggesting that change scores based on then-test data be interpreted with care.
vorige artikel Neuro-QOL: quality of life item banks for adults with neurological disorders: item development and calibrations based upon clinical and general population testing
volgende artikel Correlates of quality of life in China rural–urban female migrate workers
Literatuur
1.
Golembiewsky, R., Billingsley, K., & Yeager, S. (1976). Measuring change and persistence in human affairs: Types of change generated by OD designs. Journal of Applied Behavioral Science, 12, 133–157.CrossRef
2.
Howard, G. S., & Dailey, P. R. (1979). Response-shift bias: A source of contamination of self-report measures. Journal of Applied Psychology, 64(2), 144–150.CrossRef
3.
Sprangers, M. A. G., & Schwartz, C. E. (1999). Integrating response shift into health-related quality of life research: A theoretical model. Social Science and Medicine, 48(11), 1507–1515.PubMedCrossRef
4.
Schwartz, C. E., & Sprangers, M. A. G. (1999). Methodological approaches for assessing response shift in longitudinal health-related quality-of-life research. Social Science and Medicine, 48(11), 1531–1548.PubMedCrossRef
5.
Cronbach, L. J., & Furby, L. (1970). How should we measure “change”—or should we? Psychological Bulletin, 74(1), 68–80.CrossRef
6.
Lorig, K. R., Sobel, D. S., Stewart, A. L., Brown, B. W., Bandura, A., Ritter, P., et al. (1999). Evidence suggesting that a chronic disease self-management program can improve health status while reducing hospitalization. Medical Care, 37(1), 5–14.PubMedCrossRef
7.
Elzen, H., Slaets, J., Snijders, T., & Steverink, N. (2007). Evaluation of the chronic disease self-management program (CDSMP) among chronically ill older people in the Netherlands. Social Science and Medicine, 64, 1832–1841.PubMedCrossRef
8.
Lorig, K. R. (2003). Editoria—Self-management education: more than a nice extra. Medical Care, 41(6), 699–701.PubMed
9.
Lorig, K. R., & Holman, H. R. (1989). Long-term outcomes of an arthritis self-management study: Effects of reinforcement efforts. Social Science and Medicine, 29(2), 221–224.PubMedCrossRef
10.
Gibbons, F. X. (1999). Social comparison as a mediator of response shift. Social Science and Medicine, 48(11), 1517–1530.PubMedCrossRef
11.
Osborne, R. H., Hawkins, M., & Sprangers, M. A. G. (2006). Change of perspective: A measurable and desired outcome of chronic disease self-management intervention programs that violates the premise of preintervention/postintervention assessment. Arthritis Care and Research, 55(3), 458–465.PubMedCrossRef
12.
Sprangers, M. A. G. (1989). Response-shift bias in program evaluation. Impact Assessment Bulletin, 7, 153–166.
13.
Schwartz, C., & Sprangers, M. (2010). Guidelines for improving the stringency of response shift research using the thentest. Quality of Life Research, 19, 455–464.PubMedCrossRef
14.
Byrne, B. M., Shavelson, R. J., & Muthén, B. (1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105, 456–466.CrossRef
15.
McGaw, B., & Jöreskog, K. G. (1971). Factorial invariance of ability measures in groups differing in intelligence and socioeconomic status. British Journal of Mathematical, 24, 154–168.
16.
Nolte, S., Elsworth, G. R., Sinclair, A. J., & Osborne, R. H. (2009). Tests of measurement invariance failed to support the application of the “then-test”. Journal of Clinical Epidemiology, 62(11), 1173–1180.PubMedCrossRef
17.
Howard, G. S., Schmeck, R. R., & Bray, J. H. (1979). Internal invalidity in studies employing self-report instruments: a suggested remedy. Journal of Educational Measurement, 16(2), 129–135.CrossRef
18.
Lorig, K., & Fries, J. (2000). The arthritis helpbook (5th ed.). Cambridge, MA: Perseus Books.
19.
Lorig, K. R., González, V. M., & Laurent, D. D. (1999). The chronic disease self-management program: Leaders manual. Palo Alto: Stanford University.
20.
Osteoporosis Victoria. (2001). The osteoporosis prevention and self management course—Leaders manual. Elsternwick: Osteoporosis Victoria, A division of Arthritis Victoria.
21.
Nolte, S., Elsworth, G. R., Sinclair, A. J., & Osborne, R. H. (2007). The extent and breadth of benefits from participating in chronic disease self-management courses: A national patient-reported outcomes survey. Patient Education and Counseling, 65(3), 351–360.PubMedCrossRef
22.
Osborne, R. H., Elsworth, G. R., & Whitfield, K. (2007). The health education impact questionnaire (heiQ): An outcomes and evaluation measure for patient education and self-management interventions for people with chronic conditions. Patient Education and Counseling, 66, 192–201.PubMedCrossRef
23.
Nolte, S. (2008). Approaches to the measurement of outcomes of chronic disease self-management interventions using a self-report inventory. Melbourne: RMIT University.
24.
Brown, M. B., & Forsythe, A. B. (1974). Robust tests for the equality of variances. Journal of the American Statistical Association, 69(346), 364–367.CrossRef
25.
Rubin, D. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701.CrossRef
26.
27.
Krosnick, J. A., & Alwin, D. F. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51, 201–219.CrossRef
28.
Lam, T. C. M., & Bengo, P. (2003). A comparison of three retrospective self-reporting methods of measuring change in instructional practice. American Journal of Evaluation, 24(1), 65–80.
29.
Schwarz, N., & Strack, F. (1985). Cognitive and affective processes in judgements of subjective well-being: a preliminary model. In H. Brandstatter & E. Kirchler (Eds.), Economic psychology (pp. 439–447). Linz, Austria: R. Tauner.
30.
Crowne, D. P., & Marlowe, D. (1964). The approval motive: Studies in evaluative dependence. New York, London, Sydney: Wiley.
31.
Howard, G. S., Ralph, K. M., Gulanick, N. A., Maxwell, S. E., Nance, S. W., & Gerber, S. K. (1979). Internal invalidity in pretest-posttest self-report evaluations and a re-evaluation of retrospective pretests. Applied Psychological Measurement, 3, 1–23.CrossRef
32.
Hill, L. G., & Betz, D. L. (2005). Revisiting the retrospective pretest. American Journal of Evaluation, 26(4), 501–517.CrossRef
33.
Sprangers, M. A. G., & Hoogstraten, J. (1988). On delay and reassessment of retrospective preratings. Journal of Experimental Education, 56, 148–153.
34.
Norman, G. (2003). Hi! How are you? Response shift, implicit theories and differing epistemologies. Quality of Life Research, 12, 239–249.PubMedCrossRef
35.
Ross, M. (1989). Relation of implicit theories to the construction of personal histories. Psychological Review, 96, 341–357.CrossRef
36.
Nimon, K., Zigarmi, D., & Allen, J. (2011). Measures of program effectiveness based on retrospective pretest data: are all created equal? American Journal of Evaluation, 32(8), 8–28.CrossRef
37.
Terborg, J., & Davis, G. (1982). Evaluation of a new method of assessing change to planned job redesign as applied to Hackman and Oldham’s job characteristic model. Organizational Behavior and Human Performance, 29, 112–128.CrossRef
38.
Sprangers, M., & Hoogstraten, J. (1989). Pretesting effects in retrospective pretest-posttest designs. Journal of Applied Psychology, 74, 265–272.CrossRef
39.
Stone, G. R., & Packer, T. L. (2010). Evaluation of a rural chronic disease self-management program. Rural and Remote Health, 10(1), 1203.PubMed
Metagegevens
Titel
The inclusion of ‘then-test’ questions in post-test questionnaires alters post-test responses: a randomized study of bias in health program evaluation
Auteurs
Sandra Nolte
Gerald R. Elsworth
Andrew J. Sinclair
Richard H. Osborne
Publicatiedatum
01-04-2012
Uitgeverij
Springer Netherlands
Gepubliceerd in
Quality of Life Research / Uitgave 3/2012
Print ISSN: 0962-9343
Elektronisch ISSN: 1573-2649
DOI
https://doi.org/10.1007/s11136-011-9952-1

Andere artikelen Uitgave 3/2012

Quality of Life Research 3/2012 Naar de uitgave

OriginalPaper

A mixed methods approach to adapting health-related quality of life measures for use in routine oncology clinical practice

OriginalPaper

Measuring outcomes in Parkinson’s disease: a multi-perspective concept mapping study

OriginalPaper

Correlates of quality of life in China rural–urban female migrate workers

OriginalPaper

Palliative care outcomes in surgical oncology patients with advanced malignancies: a mixed methods approach

OriginalPaper

Neuro-QOL: quality of life item banks for adults with neurological disorders: item development and calibrations based upon clinical and general population testing

OriginalPaper

Factors associated with quality of life in older adults in the United States