Skip to main content
SpringerLink
Log in

Clinical Significance

  • Chapter
  • First Online:
Behavioral Clinical Trials for Chronic Diseases

Abstract

This chapter first considers the strengths and limitations of statistical inference. Its limitations are often overlooked relative to its convenience and immense popularity as a decision-making metric. This background provides the basis for the argument that statistical significance is not the only, or the best, judge for making the case that a behavioral treatment can move a chronic disease endpoint. More persuasive is a demonstration of clinical significance where a behavioral treatment is strong enough to neutralize the risk incurred by a behavioral risk factor. The second part of the chapter is directed toward how to identify clinically significant treatment targets, incorporate clinical significance into the analysis of trial results, and integrate clinical significance into the calculation of sample size. The contribution of any single behavioral trial is assessed by the strength of the scientific conclusions drawn from it. This includes a clinically significant benefit of treatment, quantified certainty that this benefit is greater than a relevant comparator, biologic plausibility, and consistency with results from past studies. The conclusion that a behavioral treatment improves health can only be answered progressively, through replication.

“To say that a great deal of mischief has been associated with the test of significance is hardly original. It is what ‘everybody knows.’. . . . To say it ‘out loud’ is to assume the role of the child who pointed out that the emperor was really outfitted in his underwear.”

Bakan 1966 [1]

“If this comment was hardly original in 1966, it can hardly be original now. Yet this naked emperor has been shamelessly running around for a long time.”

Cohen, 1994 [2]

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 119.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bakan D (1966) The test of significance in psychological research. Psych Bulletin 66:423–437

    Article  Google Scholar 

  2. Cohen J (1994) The earth is round (p < .05). Am Psych 49:997–1003

    Article  Google Scholar 

  3. Hume D (1739-1740) A treatise of human nature, V1, Norton DF, Norton MJ (eds) (2000), Oxford University Press, Oxford

    Google Scholar 

  4. Popper KR (1959) The logic of scientific discovery. Basic Books, New York

    Google Scholar 

  5. Fisher RA (1935) The design of experiments. Oliver and Boyd, Edinburgh

    Google Scholar 

  6. Goodman SN (1999) Toward evidence-based medical statistics. 1: The P-value fallacy. Ann Intern Med 130:995–1004

    Article  Google Scholar 

  7. Fisher RA (1973) Statistical methods and scientific inference, 3rd edn. Macmillan, New York

    Google Scholar 

  8. Bayes T (1763) Essay towards solving a problem in the doctrine of chances. Phil Trans Roy Soc 53:370–418. (Reprinted: Biometrika 1958:45:293–315)

    Article  Google Scholar 

  9. Fisher RA (1956) Statistical methods and scientific inference, 1st edn. Oliver & Boyd, Edinburg

    Google Scholar 

  10. Neyman J, Pearson E (1933) On the problem of the most efficient tests of statistical hypotheses. Phil Trans Roy Soc 231:289–337

    Google Scholar 

  11. Tyler RW (1931) What is statistical significance? Ed Res Bull 10:115–118

    Google Scholar 

  12. Wasserstein RL, Schirm AL, Lazar NA (2019) Moving to a world beyond “p < 0.05”. Am Stat 73:1–19

    Article  Google Scholar 

  13. Amrhein V, Greenland S, McShane B (2019) Retire statistical significance. Nature 567:305–307

    Article  Google Scholar 

  14. Skeelam J (1969) Models, inference, and strategy. Biometrics 25:457–475

    Article  Google Scholar 

  15. Armstrong JS (2007) Significance tests harm progress in forecasting. Intl J Forecasting 23:321–327

    Article  Google Scholar 

  16. Ioannidis JPA (2005) Why most published research findings are false. PLoS Med 2:e124. https://doi.org/10.1371/journal.pmed.0020124

    Article  PubMed  PubMed Central  Google Scholar 

  17. Bacchetti P (2010) Current sample size conventions: flaws, harms, and alternatives. BMC Med 8:17. https://doi.org/10.1186/1741-7015-8-17

    Article  PubMed  PubMed Central  Google Scholar 

  18. Kraemer HC, Morgan GA, Leech NL, Gliner JA, Vaske JJ, Harmon RJ (2003) Measures of clinical significance. J Am Acad Child Adolesc Psychiat 42:1524–1529

    Article  Google Scholar 

  19. Silva-Aycaguer LC, Suarez-Gil P, Fernandez-Somoano A (2010) The null hypothesis significance test in health sciences research (1995–2006): statistical analysis and interpretation. BMC Med Res Meth 10:44. https://doi.org/10.1186/1471-2288-10-44

    Article  Google Scholar 

  20. Jacobson NS (1991) Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consul Clin Psych 59:12–19

    Article  Google Scholar 

  21. Lecoutre B, Lecoutre MP, Poiteveneau J (2001) Uses, abuses, and misuses of significance tests in the scientific community: Won’t the Bayesian choice be unavoidable? Intl Stat Rev 69:399–417

    Article  Google Scholar 

  22. Rutledge T, Loh C (2004) Effect sizes and statistical testing in the determination of clinical significance in behavioral medicine research. Ann Behav Med 27:138–145

    Article  Google Scholar 

  23. Hubbard R, Ryan PA (2000) The historical growth of statistical significance testing in psychology — and its future prospects. Educ Psych Meas 60:661–681

    Google Scholar 

  24. Schmidt FL, Hunter JE (1997) Eight common but false objections to the discontinuation of significance testing in analysis of research data. In: Harlow LL, Mulnik SA, Steiger JH (eds) What if there were no significance tests? Erlbaum, London, pp 35–60

    Google Scholar 

  25. Gigerenzer G (2004) Mindless statistics. J Socio-Econ 33:587–606

    Article  Google Scholar 

  26. Gigerenzer G, Swijtink Z, Porter T, Daston L, Beatty J, Kruger L (1989) The empire of chance. How probability changed science and everyday life. Cambridge University Press, Cambridge

    Google Scholar 

  27. Efron B, Hastie T (2016) Computer age statistical inference. Algorithms, evidence, and data science. Cambridge University Press, New York

    Google Scholar 

  28. Goodman SN (1999) Toward evidence-based medical statistics. 2: The Bayes factor. Ann Intern Med 130:1005–1013

    Article  Google Scholar 

  29. Efron B (August 2018) Personal communication

    Google Scholar 

  30. Czajkowski SM, Powell LH, Adler N, Naar-King S, Reynolds KD, Hunter CM, Laraia B, Olster DH, Perna FM, Peterson JC, Epel E, Charlson M (2015) From ideas to efficacy: the ORBIT model for developing behavioral treatments for chronic diseases. Health Psychol 34:971–982

    Article  Google Scholar 

  31. 2018 Physical Activity Guidelines Advisory Committee (2018) 2018 Physical Activity Guidelines Advisory Committee Scientific Report. US Department of Health and Human Services, Washington DC

    Google Scholar 

  32. McGlothlin AE, Lewis RJ (2014) Minimal clinically important difference defining what really matters to patients. JAMA 312:1342–1343

    Article  Google Scholar 

  33. Nezu AM, Nezu CM (2008) Evidence-based outcome research: a practical guide to conducting randomized controlled trials for psychosocial interventions. Oxford University Press, New York

    Google Scholar 

  34. Lambert MJ, Hansen NB, Bauer S (2008) Assessing the clinical significance of outcome results. In: Nezu AM, Nezu CM (eds) Evidence-based outcome research: a practical guide to conducting randomized controlled trials for psychosocial interventions. Oxford University Press, New York, pp 359–378

    Google Scholar 

  35. Gitlin LN, Czaja SJ (2016) Behavioral intervention research. Designing, evaluating, and implementing. Springer, New York

    Google Scholar 

  36. Calvert M, Kyte D, Mercieca-Bebber R, Slade A, Chan AW, King MT, SPIRIT-PRO Group (2018) Guidelines for inclusion of patient-reported outcomes in clinical trial protocols: the SPIRIT-PRO Extension. JAMA 319:483–494

    Article  Google Scholar 

  37. De Geest S, Sabate E (2003) Adherence to long-term therapies: evidence for action. Europ J Cardiovasc Nursing 2:323. https://doi.org/10.1016/S1474-5151(03)00091-4

  38. Colman E (2012) Food and Drug Administration’s obesity drug guidance document. A short history. Circulation 125:2156–2164

    Google Scholar 

  39. Kraemer HC, Mintz J, Tinklenberg J, Yesavage JA (2006) Caution regarding the use of pilot studies to guide power calculations for study proposals. Arch Gen Psychiat 63:484–489

    Article  Google Scholar 

  40. Bacchetti P, McCulloch CE, Segal MR (2008) Simple, defensible sample sizes based on cost efficiency. Biometrics 64:577–594

    Article  Google Scholar 

  41. Willan AR (2008) Optimal sample size determinations from an industry perspective based on the expected value of information. Clin Trials 5:587–594

    Article  Google Scholar 

  42. Everson SA, Goldberg DE, Kaplan GA, Julkunen J, Salonen JT (1998) Anger expression and incident hypertension. Psychosom Med 60:730–735

    Article  Google Scholar 

  43. Everson SA, Lynch JW, Kaplan GA, Lakka TA, Sivenius J, Salonen JT (2001) Stress-induced blood pressure reactivity and incident stroke in middle-aged men. Stroke 32:1263–1270

    Article  Google Scholar 

  44. Mosnaim G, Li H, Martin M, Richardson D, Belice PJ, Avery E, Silberstein A, Leigh J, Kenyon R, Jones S, Bender B, Powell LH (2015) A tailored mobile health intervention to improve adherence and asthma control in minority adolescents. J Allergy Clin Immunol Pract 3:288–290

    Google Scholar 

  45. Mosnaim G, Li H, Martin M, Richardson D, Belice PJ, Avery E, Ryan N, Bender B, Powell LH (2013) The impact of peer support and mp3 messaging on adherence to inhaled corticosteroids in minority adolescents with asthma: a randomized, controlled trial. J Allergy Clin Immunol Pract 1:485–493

    Article  Google Scholar 

  46. Berkman LF, Blumenthal J, Burg M, Carney RM, Catellier D, Cowan MJ, Czajkowski SM, DeBusk R, Hosking J, Jaffe A, Kaufmann PG, Mitchell P, Norman J, Powell LH, Raczynski JM, Schneiderman N, Enhancing Recovery in Coronary Heart Disease Patients Investigators (ENRICHD) (2003) Effects of treating depression and low perceived social support on clinical events after myocardial infarction: the Enhancing Recovery in Coronary Heart Disease Patients (ENRICHD) Randomized Trial. JAMA 289:3106–3116

    Google Scholar 

  47. Frasure-Smith N, Lesperance F (2003) Depression – A cardiac risk factor in search of a treatment. JAMA 289:3171–3173

    Google Scholar 

  48. Howard BV, Van Horn L, Hsia J, Manson JE, Stefanick ML, Wassertheil-Smoller S, Kuller LH, LaCroix AZ, Langer RD, Lasser NL, Lewis CE, Limacher MC, Margolis KL, Mysiw WJ, Ockene JK, Parker LM, Perri MG, Phillips L, Prentice RL, Robbins J, Rossouw JE, Sarto GE, Schatz IJ, Snetselaar LG, Stevens VJ, Tinker LF, Trevisan M, Vitolins MZ, Anderson GL, Assaf AR, Bassford T, Beresford SA, Black HR, Brunner RL, Brzyski RG, Caan B, Chlebowski RT, Gass M, Granek I, Greenland P, Hays J, Heber D, Heiss G, Hendrix SL, Hubbell FA, Johnson KC, Kotchen JM (2006) Low-fat dietary pattern and risk of cardiovascular disease: the Women’s Health Initiative Randomized Controlled Dietary Modification Trial. JAMA 295:655–666

    Article  Google Scholar 

  49. Knowler WC, Barrett-Connor E, Fowler SE, Hamman RF, Lachin JM, Walker EA, Nathan DM, Diabetes Prevention Program Research Group (2002) Reduction in the incidence of type 2 diabetes with lifestyle intervention or metformin. New Engl J Med 346:393–403

    Google Scholar 

  50. Kosaka K, Noda M, Kuzuya T (2005) Prevention of type 2 diabetes by lifestyle intervention: a Japanese trial in IGT males. Diab Res Clin Pract 67:152–162

    Google Scholar 

  51. Saito T, Watanabe M, Nishida J, Izumi T, Omura M, Takagi T, Fukunaga R, Bandai Y, Tajima N, Nakamura Y, Ito M, Zensharen Study for Prevention of Lifestyle Diseases Group (2011) Lifestyle modification and prevention of type 2 diabetes in overweight Japanese with impaired fasting glucose levels. Arch Intern Med 171:1352–1360

    Article  Google Scholar 

  52. Sakane N, Sato J, Tsushita K, Tsujii S, Kotani K, Tsuzaki K, Tominaga M, Kawazu S, Sato Y, Usui T, Kamae I, Yoshida T, Kiyohara Y, Sato S, Kuzuya H, Japan Diabetes Prevention Program (JDPP) Research Group (2011) Prevention of type 2 diabetes in a primary healthcare setting: three-year results of lifestyle intervention in Japanese subjects with impaired glucose tolerance. BMC Public Health 11:40. https://doi.org/10.1186/1471-2458-11-40

    Article  PubMed  PubMed Central  Google Scholar 

  53. Lindahl B, Nilsson TK, Borch-Johnsen K, Oder ME, Soderberg S, Widman L, Johnson O, Hallmans G, Jansson J (2009) A randomized lifestyle intervention with 5-year follow-up in subjects with impaired glucose tolerance: pronounced short-term impact but long-term adherence problems. Scand J Pub Health 37:434–442

    Article  Google Scholar 

  54. Pan XR, Li GW, Hu YH, Wang JX, Yang WY, An ZX, Hu ZX, Lin J, Xiao JZ, Cao HB, Liu PA, Jiang XG, Jiang YY, Wang JP, Zheng H, Zhang H, Bennett PH, Howard BV (1997) Effects of diet and exercise in preventing NIDDM in people with impaired glucose tolerance. Diabetes Care 20:537–544

    Article  Google Scholar 

  55. Penn L, White M, Lindstrom J, den Boer AT, Blaak E, Eriksson JG, Feskens E, Ilanne-Parikka P, Keinanen-Kiukaanniemi SM, Walker M, Mathers JC, Uusitupa M, Tuomilehto J (2013) Importance of weight loss maintenance and risk prediction in the prevention of type 2 diabetes: analysis of European Diabetes Prevention Study RCT. PLoS ONE 8:e57143. https://doi.org/10.1371/journal.pone.0057143

  56. Ramachandran A, Snehalatha C, Ram J, Selvam S, Simon M, Nanditha A, Shetty AS, Godsland IF, Chaturvedi N, Majeed A, Oliver N, Toumazou C, Alberti KG, Johnston DG (2013) Effectiveness of mobile phone messaging in prevention of type 2 diabetes by lifestyle modification in men in India: a prospective, parallel-group, randomized controlled trial. Lancet 1:191–198

    Google Scholar 

  57. Tuomilehto J, Lindström J, Eriksson JG, Valle TT, Hämäläinen H, Ilanne-Parikka P, Keinänen-Kiukaanniemi S, Laakso M, Louheranta A, Rastas M, Salminen V, Uusitupa M, Finnish Diabetes Prevention Study Group (2001) Prevention of type 2 diabetes mellitus by changes in lifestyle among subjects with impaired glucose tolerance. New Engl J Med 344:1343–1350

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Preventive Medicine, Rush University Medical Center, Chicago, IL, USA

    Lynda H. Powell

  2. College of Nursing, Villanova University, Villanova, PA, USA

    Peter G. Kaufmann

  3. Department of Psychiatry, Washington University in St. Louis, St. Louis, MO, USA

    Kenneth E. Freedland

Authors
  1. Lynda H. Powell
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter G. Kaufmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kenneth E. Freedland
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Powell, L.H., Kaufmann, P.G., Freedland, K.E. (2021). Clinical Significance. In: Behavioral Clinical Trials for Chronic Diseases. Springer, Cham. https://doi.org/10.1007/978-3-030-39330-4_5

Download citation

Publish with us

Policies and ethics

Search

Navigation