Skip to main content
Log in

Using Propensity Scores to Construct Comparable Control Groups for Disease Management Program Evaluation

  • Original Research Article
  • Published:
Disease Management & Health Outcomes

Abstract

Introduction

The ability of observational studies to draw conclusions on causal relationships between covariates and outcomes can be improved by incorporating randomly matched controls using the propensity scoring method. This procedure controls for pre-program differences between the enrolled and non-enrolled groups by reducing each participant’s set of covariates into a single score, which makes it feasible to match on what are essentially multiple variables simultaneously. This paper introduces this concept using the first year results of a congestive heart failure (CHF) disease management (DM) program as an example.

Methods

This study employed a case-control pre-post study design with controls randomly matched to patients based on the propensity score. There were 94 patients with CHF enrolled in a DM program for at least 1 year (cases), who were matched to 94 patients with CHF drawn from a health plan’s CHF population (controls). Independent variables that estimated the propensity score were pre-program: hospital admissions, emergency department (ED) visits, total costs, and risk level. Baseline (1 year prior to program commencement) and 1-year outcome variables were compared for the two groups.

Results

The results indicated that, at post-program, program participants had significantly lower hospitalization rates (p = 0.005), ED visit rates (p = 0.048), and total costs (p = 0.003) than their matched controls drawn from the CHF population.

Conclusions

Because of its simplicity and utility, propensity scoring should be considered as an alternative procedure for use with current non-experimental designs in evaluating DM program effectiveness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Table I
Fig. 1
Fig. 2
Fig. 3
Table II
Fig. 4
Table III

Similar content being viewed by others

References

  1. Kleijen J, Gotzche P, Kunz RA, et al. So what’s so special about randomization? In: Maynard A, Chalmers I, editors. Non-random reflections on health services research. London: BMJ Publishing, 1997: 93–106

    Google Scholar 

  2. D’Arcy Hart P. Early controlled clinical trials. BMJ 1996; 312(2): 769–82

    Google Scholar 

  3. Linden A, Adams J, Roberts N. An assessment of the total population approach for evaluating disease management program effectiveness. Dis Manag 2003; 6(2): 93–102

    Article  PubMed  Google Scholar 

  4. Linden A, Adams J, Roberts N. Evaluating disease management program effectiveness: an introduction to time series analysis. Dis Manag 2003; 6(4): 243–55

    Article  PubMed  Google Scholar 

  5. Linden A, Adams J, Roberts N. Evaluating disease management program effectiveness: an introduction to survival analysis. Dis Manag. 2004; 7(3): 180–190

    Article  PubMed  Google Scholar 

  6. Linden A, Adams J, Roberts N. Evaluation methods in disease management: determining program effectiveness. Position Paper for the Disease Management Association of America (DMAA). 2003 Oct

  7. Campbell DT, Stanley JC. Experimental and quasi-experimental designs for research. Chicago (IL): Rand McNally, 1966

    Google Scholar 

  8. Cook TD, Campbell DT. Quasi-experimentation: design and analysis issues for field settings. Chicago (IL): Rand McNally College Publishing Company, 1979

    Google Scholar 

  9. Shadish SR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Boston (MA): Houghton Mifflin, 2002

    Google Scholar 

  10. American Healthways and the John Hopkins Consensus Conference. Consensus report: standard outcome metrics and evaluation methodology for disease management programs. Dis Manag 2003; 6(3): 121–38

    Article  Google Scholar 

  11. Dehejia RH, Wahba S. Propensity score-matching methods for non-experimental causal studies. Rev Econ and Stats 2002; 84: 151–61

    Article  Google Scholar 

  12. Rosenbaum P, Rubin D. The central role of the propensity score in observational studies for causal effects. Biometrika 1983; 70: 41–55

    Article  Google Scholar 

  13. Rosenbaum P, Rubin D. Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. Am Stat 1985; 39: 33–8

    Google Scholar 

  14. Rubin D. Estimating causal effects of treatments in randomized and non-randomized studies. J Educ Psychol 1974; 66: 688–701

    Article  Google Scholar 

  15. Rubin D. Assignment to treatment group on the basis of a covariate. J Educ Stats 1977; 2: 1–26

    Article  Google Scholar 

  16. Cox DR. The analysis of binary data. London: Methuen, 1970

    Google Scholar 

  17. Cox DR. The analysis of multivariate binary data. Appl Stat 1972; 21: 113–20

    Article  Google Scholar 

  18. Dehejia RH, Wahba S. Causal effects in nonexperimental studies: reevaluating the evaluation of training studies. J Am Stat Assoc 1999; 94: 1053–62

    Article  Google Scholar 

  19. Heckman J, Ichimura J, Todd P. Matching as an econometric evaluation estimator: evidence from evaluating a job training programme. Rev Econ Stud 1997; 64: 605–54

    Article  Google Scholar 

  20. Rubin DB, Thomas N. Matching using estimated propensity scores: relating theory to practice. Biometrics 1996; 52: 249–64

    Article  PubMed  CAS  Google Scholar 

  21. Rubin DB. Estimating causal effect from large data sets using propensity scores. Ann Intern Med 1997; 127: 757–63

    PubMed  CAS  Google Scholar 

  22. Canner P. How much data should be collected in clinical trials? Stat Med 1984; 3: 423–32

    Article  PubMed  CAS  Google Scholar 

  23. Canner P. Covariate adjustment of treatment effects in clinical trials. Control Clin Trials 1991; 12: 359–66

    Article  PubMed  CAS  Google Scholar 

  24. Drake C. Effects of misspecification of the propensity score on estimators of treatment effects. Biometrics 1993; 49: 1231–6

    Article  Google Scholar 

  25. Cochran WG. The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics 1968; 24: 205–13

    Article  Google Scholar 

  26. Rosenbaum PR, Rubin DB. Reducing bias in observational studies using subclassification on the propensity score. J Am Stat Assoc 1984; 79: 516–24

    Article  Google Scholar 

  27. Aday L, Andersen RM. Equity in access to medical care: realized and potential. Med Care 1981; 19(12 Suppl.): 4–27

    Article  CAS  Google Scholar 

  28. Andersen RM. Behavioral model of families: use of health services. Research Series No. 25. Chicago (IL): Center for Health Administration Studies, University of Chicago, 1968

    Google Scholar 

  29. Linden A, Adams J, Roberts N. Strengthening the case for disease management effectiveness: unhiding the hidden bias. Am J Eval. In press

  30. Rosenbaum P. Sensitivity analysis for certain permutation tests in matched observational studies. Biometrika 1987; 74: 13–26

    Article  Google Scholar 

  31. Fisher ES, Barton JA, Malenka DJ, et al. Overcoming potential pitfalls in the Medicare data for epidemiologic research. Am J Public Health 1990; 80: 533–46

    Article  Google Scholar 

  32. Romano PS, Mark DH. Bias in the coding of hospital discharge data and its implication for quality assessment. Med Care 1994; 32: 81–90

    Article  PubMed  CAS  Google Scholar 

  33. Roos LL, Sharp SM, Cohen MM. Comparing clinical information with claims data: some similarities and differences. J Clin Epidemiol 1991; 44: 881–8

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgments

No sources of funding were used to assist in the preparation of this study. The authors have no conflicts of interest that are directly relevant to the content of this study.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ariel Linden.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Linden, A., Adams, J.L. & Roberts, N. Using Propensity Scores to Construct Comparable Control Groups for Disease Management Program Evaluation. Dis-Manage-Health-Outcomes 13, 107–115 (2005). https://doi.org/10.2165/00115677-200513020-00004

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.2165/00115677-200513020-00004

Keywords

Navigation