The concepts of mediation and incremental validity are interrelated but this is often overlooked in the literature. This paper introduces a thematic issue of the Child and Youth Care Forum with papers that focus on mediation, incremental validity, and novel intervention development in youth anxiety and related problems. While the individual papers each make an important contribution to the respective topics, taken together, the papers highlight a too often underappreciated connection across studies which are on similar topics (e.g., anxiety as in this issue) but with apparently disparate methods or goals (e.g., in this issue two papers on theory testing, one on the development of a measure, one on the specificity of associations, and one on the treatment of co-occurring anxiety and headache). However, recognition of similarities between the concept of mediation and the concept of incremental validity can help in clarifying the implications of findings obtained from such research, clarify the nature of processes associated with youth problems such as anxiety, foster an understanding of the co-occurrence of problems, and possibly even aid in the theoretical development of novel therapeutic strategies.

The Oxford English Dictionary defines the word mediate as a verb meaning to “intervene between people in a dispute in order to bring about an agreement or reconciliation” or to “bring about.” While the latter definition is more central, both connotations are implied by the concept of mediation in psychological research. The concept of mediation has produced a vast literature in the fields researching children and youth but typically involves the desire by researchers to identify the process or processes responsible for an existing or previously established relationship (See e.g., Holmbeck 2002), or similarly to identify a theoretically important intermediary process or processes. Variables that serve as the putative mechanism or theoretical process linking independent and dependent variables are considered mediators. Mediators are also typically understood as distinct from moderators. While moderation is not the focus of this paper or the papers in this issue, distinguishing mediators from moderators is conceptually useful, as the terms are sometimes confused. Variables that influence the nature (e.g., direction or strength) of a relationship between independent and outcome/dependent variable or that help specify the conditions or context where a relationship exists between and an independent variable and an outcome/dependent variable are considered moderators (see Baron and Kenny 1986; Holmbeck 2002).

Based on a seminal paper by Baron and Kenny (1986) research on mediation often tests mediation through the causal steps approach (see also MacKinnon et al. 2002). This approach proceeds through four steps. The first step is testing whether an independent variable is associated with or predicts the outcome/dependent variable. The statistical significance of this first link is not as important in the related concept of “indirect models” also termed “indirect path models” in these models a mediator is simply an intermediary variable and not necessarily the reason for or an explanation for an existing link between independent and dependent variable (an indirect causal model may not posit a direct link between the independent variable and the outcome). The second condition (in Baron and Kenny 1986) is that the independent variable is associated with or otherwise predicts the putative mediator variable. The third condition is that the mediator variable also predicts the outcome/dependent variable. Condition four is tested by determining if an independent variable is no longer predictive (full mediation) or substantially less predictive (partial mediation) of the outcome/dependent variable when controlling for the mediator variable (again see MacKinnon et al. 2002; Preacher and Hayes 2008; Sobel 1982 for related methods/discussion). When this set of findings occurs it is thought that the mediator is responsible for the association. Panel “a” of Fig. 1 shows a common representation of mediation.

Fig. 1
figure 1

Conceptual similarity between mediation and incremental validity

Full size image

In causal theories the mediator is thought to be the reason for the association or an intermediary link or process in a causal chain. Longitudinal designs allow for temporal conclusions about mediators as intermediary links but only experimental designs allow for strong causal conclusions about the role of a mediator. In non-experimental cross-sectional studies, the main conclusion than can be drawn is that the posited mediator is accounting for (functionally all) the variance in the association between an independent and dependent variable in full mediation or that the mediator and independent variable are each showing incremental prediction with the mediator carrying a substantial (e.g., statistically significant) portion of the variance in partial mediation. For example, Affrunti and Ginsburg (2012, this issue) found that interpretation biases partially mediated the relationship between parental over-control and child anxiety (correlation dropped but was still significant) and completely mediated the relationship between parental anxiety and child anxiety (correlation dropped and was no longer significant). Similarly, Ng et al. (2012, this issue) reported that facets of resilience (positive thinking, help seeking) accounted for a significant portion of the association between coping and anxiety. Both studies employed non-experimental cross sectional designs, as such, the set of associations revealed supplies initial evidence for the theoretical propositions detailed in the reports, but does not allow directional or causal conclusions. However, the findings of Affrunti and Ginsburg (2012, this issue) and Ng et al. (2012, this issue) encourage additional longitudinal and experimental studies of the putative mediators.

Affrunti and Ginsburg (2012, this issue) and Ng et al. (2012, this issue) also highlight advances in contemporary approaches to mediation (i.e., applying formal statistical tests to compare the size of direct and indirect effects). Reliance solely on the causal steps approach has been criticized on multiple grounds. One serious limitation of the causal steps approach is that it does not include a direct statistical test of the indirect or mediated effect. That is, contemporary approaches test whether the direct paths are significant and whether the direct effect (e.g., the regression coefficient when only the independent and dependent variables are in the model) of the independent variable on the dependent variable is smaller than its total effect (e.g., the regression coefficient when the mediator is also included in the model, see Hayes 2009 for additional detail and discussion).

The concept of incremental validity was introduced by Sechrest (1963). In general, validity refers to the extent to which a concept, conclusion, measurement, measurement instrument or other assessment is well-founded and corresponds accurately to the construct in question (see Cronbach and Meehl 1955 for a classic discussion). Incremental validity is thought to be shown when a measure, assessment, or assessment instrument provides information over-and-above what is already obtained by other means (e.g., an existing measure of the same or a similar construct). Studies that examine incremental validity typically test whether a statistically significant amount of unique variance is accounted for when controlling for other theoretical predictors of some outcome (though other considerations apply, see Stickle and Weems 2006). For a measure to have “validity” the concept of “incremental validity” implies not just demonstration of correlation/association with other measures of the construct but also additive prediction beyond what other constructs and measures provide. Panel “b” of Fig. 1 illustrates incremental validity. Incremental validity in this regard really addresses the potential utility or usefulness of a measure. Thomas et al. (2012, this issue), is a prime example of this issue in measurement development. Specifically, in demonstrating the utility of assessing safety behaviors among adolescents, Thomas et al. not only tested if the measure, the Subtle Avoidance Frequency Examination (SAFE), correlated with anxiety but also tested if it was uniquely associated with anxiety. For example, the paper reports that SAFE scores predicted if the child was clinic referred for social anxiety (versus non-referred comparison youth) beyond established measures of depression and attention problems. Gardner and Epkins (2012, this issue) examine a related question of the uniqueness of links between two constructs and the measures used to assess them. Specifically, they report that the association between girls’ rumination and anxiety sensitivity remains even after controlling for a number of possible links between the two constructs (i.e., the girls’ depressive and anxiety symptoms, their perceptions of maternal depression and rejecting parenting, and maternal report of rejecting parenting, depression, and rumination).

In testing incremental validity, researchers test whether there is unique variance in an outcome that can be attributed the variable in question. To place this in terms of mediation the researchers are hoping that an existing measure or measures are not mediating or better accounting for an association between the new measure and an outcome. This is often accomplished by conducting the same type of analysis as condition four noted above in mediation which is determining if an independent variable (the measure being examined for incremental validity) is no longer predictive/associated (as in full mediation) of the outcome/dependent variable when controlling for some other variable (e.g., an established measure). The conceptual similarity between mediation and incremental validity is highlighted in Fig. 1. It is worth noting that the two models are mathematically equivalent. That is, the correlation between two measures in the incremental validity model is statistically equivalent to the correlation between the IV and mediator in the mediation model. The typical hope (of the developer of a new measure) is that an established measure does not explain the association between the new measure and an outcome. In incremental validity, longitudinal and experimental designs foster the strength of conclusions about validity of a measure, but research investigating incremental validity is typically less concerned with testing a causal theory than research on mediation. In other words, individuals testing incremental validity are concerned with how much additional variance the measure predicts, and typically less about explaining the reason for an association.

Recognition of the similarities between questions related to mediation and questions regarding incremental validity has a number of potentially important implications. The first implication is that researchers interested in incremental validity might draw profitably from causal theories in developing tests of the incremental validity of measures. Specifically, if a new measure or assessment predicts additional variance in an outcome beyond an existing assessment or measure theorized or otherwise identified to be a mediator of the outcome’s link to an existing correlate, then the new measure is supplying not just additional variance but is also adding to the theoretical understanding of the outcome variable (for empirical examples in anxiety research see Lau et al. 2011; Watts and Weems 2006; Weems et al. 2002). In this way, tests of the incremental validity of measures or assessments are potentially just as important to theory development as are research studies that explicitly focus on identifying variables termed “mediators.”

Future research on incremental validity might also draw from the statistical methodology used in contemporary approaches to mediation. For example, the Sobel test (1982) provides a statistical test of the mediated (indirect) effect. If a new measure’s association with an outcome does not show incremental validity, it may be that the new measure has a significant indirect effect on the outcome. The new measure may therefore still be very useful if it helps identify important intermediary processes. Methodologists and statisticians have noted that there are inherent limitations to the Sobel test (e.g., it assumes that the indirect path has normally distributed errors and this assumption is rarely met see Hayes 2009; Preacher and Hayes 2008). One recommendation for the future then is bootstrapping (Hayes 2009). This approach not only includes a formal test of the indirect path, it tests the statistical significance of the indirect effect using confidence intervals and also corrects for potential bias in the error distribution (see Hayes 2009; Preacher and Hayes 2008). Bootstrapped approaches are now incorporated into common statistical software such as SPSS, SAS, and Mplus, and can be applied to cross-sectional and to longitudinal data.

The second implication of the similarity between mediation and incremental validity is the other side of the same coin. Specifically, that researchers testing mediators might also profit by couching their findings not just in terms of consistency with theoretical predictions but also in terms of the incremental validity (or lack thereof) of the measures used as independent variables and mediators. To be clear, partial mediation implies that both the mediator and the independent variable are uniquely associated with the outcome (i.e., both are showing incremental validity). In non-experimental and non-longitudinal research this is particularly important because cause and effect and temporal ordering cannot be known regardless of the order of the analysis or theory. Knowing that both the mediator and independent variable predict unique variance suggests that the assessment of both mediator and independent variable may aid in prediction and thereby aid in understanding the nature of a particular client or set of clients’ problems. For example, in Affrunti and Ginsburg (2012, this issue) both parental over-control and interpretation bias were associated with child anxiety. Both appear to show incremental validity over each other. Full mediation implies that the independent variable is not adding to the prediction identification of the outcome/dependent variable. In such a case, client time and other resources might profitably be saved by focusing on the more useful assessment. Again drawing from Affrunti and Ginsburg (2012, this issue) parental anxiety level was no longer associated with child anxiety when controlling for interpretation bias. To place this in the incremental validity wording, the parents’ State-Trait Anxiety Inventory scores (the measure of parent anxiety) did not show incremental validity beyond child interpretation bias.

Intersecting research on mediation and incremental validity also has the potential to foster practice by improving assessment and theoretical understanding related to intervention development. This is because the identification of useful (incrementally valid) assessment tools as well as the identification of theoretical intermediary processes and reasons for associations are potentially important to developing intervention techniques that address the process or processes underlying apparently multiple problems. When measures are identified that provide information above and beyond other measures (that provide similar information), but in less time or for less cost, children and families in need of help benefit. This is nicely illustrated in the paper by Drake and Ginsburg (2012, this issue). The authors developed cognitive and behavioral intervention techniques that focus on the cognitive and behavioral processes that appear to underlie both pediatric headaches and anxiety disorders (i.e., the possible reasons for their co-occurrence/correlation). This included cognitive restructuring to reduce maladaptive cognitions such as interpretation biases, somatic focused relaxation training, and behavioral strategies where the child is helped to engage in fun activities despite pain and to face anxiety provoking situations. While the case study methodology employed does not allow for causal conclusions, once again the study encourages a more labor intensive experimental test (e.g., using randomized designs or other treatment mediation designs see Eichas et al. 2010 for an example). Such randomized designs can be used to test if changes in interpretation biases, for example, are evident in the intervention and if these changes in biases mediate any reduction seen in both anxiety and headache and thereby help explain the association between pediatric anxiety and headache. If mediation is found, the measure used to assess “interpretation biases” in such a test would have considerable evidence to back its utility and usefulness (incremental validity) in practice.

In summary, the papers in this special issue of the Child and Youth Care Forum help to highlight the intersection of mediation, incremental validity, and intervention development in youth anxiety. Recognition of similarities between the concept of mediation and the concept of incremental validity can help in clarifying the implications of findings obtained from research on the assessment of and processes associated with problematic anxiety and aid in the theoretical development of novel therapeutic strategies.