Choose (and use) your tools wisely: “Validated” measures and advanced analyses can provide invalid evidence for/against a theory

Excerpt

Doyle and Mullan’s letter (2016) provides an opportunity to clarify some important issues in methods and theory not explicit in Leventhal, Phillips, and Burns (2016). Doyle and Mullan (2016) claim that (1) Leventhal et al. (2016) were biased in their overview of the Commonsense Model (CSM) by not citing the meta-analyses showing weak relationships between certain measures of CSM constructs and measures of treatment adherence (thereby harming the science by not including null results of the meta-analyses); and (2) the lack of meta-analytic support for the CSM means that the theory should be abandoned. Both claims are false for two main, related reasons: first, the meta-analyses were not relevant to the purpose of Leventhal et al. (2016) and so were not left out due to bias; second, and more importantly, even if the purpose of Leventhal et al. (2016) had been to provide a systematic review of research evidence for the CSM, the meta-analyses cited by Doyle and Mullan do not provide adequate or even valid evidence against the CSM. We will provide three specific arguments in support of these issues after re-stating the goals of the article targeted for criticism. …