Improving the psychometric properties of dot-probe attention measures using response-based computation

https://doi.org/10.1016/j.jbtep.2018.01.009Get rights and content

Highlights

  • Response-based measures demonstrated strong internal consistency.

  • Standard measures demonstrated no evidence of internal consistency.

  • Response-based measures demonstrated improved stability across time.

  • Standard measures demonstrated no evidence of stability across time.

  • State anxiety was uniquely associated with the ratio of response-based orientation.

Abstract

Background and objectives

Abnormal threat-related attention in anxiety disorders is most commonly assessed and modified using the dot-probe paradigm; however, poor psychometric properties of reaction-time measures may contribute to inconsistencies across studies. Typically, standard attention measures are derived using average reaction-times obtained in experimentally-defined conditions. However, current approaches based on experimentally-defined conditions are limited. In this study, the psychometric properties of a novel response-based computation approach to analyze dot-probe data are compared to standard measures of attention.

Methods

148 adults (19.19 ± 1.42 years, 84 women) completed a standardized dot-probe task including threatening and neutral faces. We generated both standard and response-based measures of attention bias, attentional orientation, and attentional disengagement. We compared overall internal consistency, number of trials necessary to reach internal consistency, test-retest reliability (n = 72), and criterion validity obtained using each approach.

Results

Compared to standard attention measures, response-based measures demonstrated uniformly high levels of internal consistency with relatively few trials and varying improvements in test-retest reliability. Additionally, response-based measures demonstrated specific evidence of anxiety-related associations above and beyond both standard attention measures and other confounds.

Limitations

Future studies are necessary to validate this approach in clinical samples.

Conclusions

Response-based attention measures demonstrate superior psychometric properties compared to standard attention measures, which may improve the detection of anxiety-related associations and treatment-related changes in clinical samples.

Introduction

Maladaptive attention patterns in response to threatening information are proposed to play a central role in anxiety disorders (Bar-Haim et al., 2007, Cisler and Koster, 2010). Most commonly, anxiety-related differences in threat-related attention are assessed using the dot-probe paradigm (Fig. 1). Poor psychometric properties of standard dot-probe attention measures may yield inconsistent findings (Cristea et al., 2015, Schmukle, 2005). In standard computation approaches, threat-related attention bias is measured by comparing the average reaction times observed between experimentally-defined conditions. To capture variability across responses, new computation approaches have calculated attention bias using a nearest neighbor approach that dynamically compares reaction times between consecutive trials (Zvielli et al., 2014a, Zvielli et al., 2014b). However, this nearest neighbor approach reduces the number of trials available for psychometric analysis and limits decomposition of attention bias into orientation and disengagement of attention (Koster et al., 2004, Zvielli et al., 2014a). In this study, we propose a novel response-based method that captures intra-individual variability of threat-related attention while maximizing the data used for computation. To validate this method, we assess and compare the psychometric properties (i.e., reliability, stability, and criterion validity) of standard computation and response-based computation measures.

Developing psychometrically improved computations of attention measures offers important clinical implications. Although promising, examination of anxiety-related perturbations in attention and the clinical utility of manipulating these perturbations using Attention Bias Modification (ABM) have produced mixed findings across studies (Cristea et al., 2015, Heeren et al., 2015, Van Bockstaele et al., 2014). Inconsistent findings may be attributable to the poor psychometric properties of attention measures computed within the dot-probe paradigm. Numerous studies have demonstrated that dot-probe attention measures demonstrate both poor internal consistency and test-retest reliability (Schmukle, 2005), which may attenuate both anxiety-related associations and detection of treatment-related changes, respectively.

Standard computation methods characterize attention patterns between experimental conditions based on the average reaction times, which may fail to capture intra-individual variability in threat-related attention. In a standard computation approach, reaction times are categorized, aggregated, and compared according to experimental conditions (e.g., RTIncongruentMean – RTCongruentMean). This reaction time difference characterizes the direction in which attention is generally biased. Using standard attention bias calculations, for example, an individual may demonstrate an overall bias towards threat (i.e., RTIncongruentMean > RTCongruentMean), bias away from threat (i.e., RTIncongruentMean < RTCongruentMean), or no bias in either direction (i.e., RTIncongruentMean = RTCongruentMean). Other calculations use neutral trials as a reference to separately examine orientation and disengagement of threat-related attention (Cisler & Koster, 2010). For example, researchers often utilize neutral baseline trials to decompose attention bias into orientation (RTNeutralMean – RTCongruentMean) and disengagement (RTIncongruentMean – RTNeutralMean) of attention. However, these standard approaches may fail to capture intra-individual variability of threat-related attention and contribute to the poor psychometric properties of attention measures (Cortina, 1993, Loevinger, 1954, Zvielli et al., 2014b, Zvielli et al., 2014a).

Rather than consistently demonstrating bias towards threat or away from threat, responses vary widely within each experimental condition; therefore, dynamic computation methods aim to capture temporal dynamics of threat-related attention (Iacoviello et al., 2014, Zvielli et al., 2014a). Dynamic methods utilize a nearest-neighbor approach which compares consecutive responses across trials (e.g., RTIncongruentTrial1 – RTCongruentTrial1). For example, an individual may demonstrate attention towards threat (RTIncongruentTrial1 > RTCongruentTrial1 = +50 ms) on one trial pair and attention away from threat (RTIncongruentTrial2 - RTCongruentTrial2 = −50 ms) on the next trial pair. Thus, two separate attention measures are generated according to the type of response (i.e., bias towards or bias away) demonstrated on each consecutive trial pair (e.g., Zvielli et al., 2014a, Zvielli et al., 2014b).

Although dynamic computation methods characterize intra-individual variability across trials, such methods are limited. For example, overall number of consecutive congruent and incongruent trials available for analysis is reduced because of the randomized sequence of trials commonly employed in dot-probe paradigms. Moreover, inclusion of neutral baseline trials exacerbates trial loss associated with a nearest-neighbor approach in two ways. First, the addition of neutral trials further decreases the number of truly consecutive congruent and incongruent trials, thereby minimizing the number of analyzable trials. Second, sub-dividing the number of neutral reference trials typically prevents decomposition of attention bias into separate indices of orientation and disengagement of attention, which may play distinct roles in anxiety (Cisler and Koster, 2010, Evans et al., 2016). Finally, simulation studies suggest that existing trial-level measures do not capture patterns of threat-related attention, but instead capture intra-individual variability in reaction time more generally (Kruijt, Field, & Fox, 2016). In light of these issues, alternative computation methods are necessary to overcome limitations of existing approaches.

To address these issues, we developed a novel response-based computation that minimizes data loss and allows separation of orientation and disengagement of attention. Rather than referencing against the next consecutive trial, response-based computation separately compares individual trial reaction times to a mean reference reaction time. To obtain a distribution of vigilance and avoidance responses for each participant, RT from individual congruent trials are referenced against the mean RT of incongruent trials (i.e., RTIncongruentMean – RTCongruent [Trial1 … Trial2 … Trialn]; see Fig. 2).

To illustrate this approach, two participants may each demonstrate an average reaction time of 500 ms for incongruent trials and congruent trials. In a standard computation approach, both individuals would exhibit an average lack of attention bias (i.e., 500 ms–500 ms = 0 ms bias) that characterizes non-anxious individuals. Despite demonstrating identical mean reaction times in experimentally-defined conditions, however, each participant may demonstrate a markedly distinct distribution of threat-related attention. For example, one individual may demonstrate a lack of attention bias consistently across trials (e.g., 500 ms – RTCongruent [500 ms, 500 ms, Trialn]), whereas the other individual may demonstrate the occurrence of both vigilance and avoidance on separate trials (e.g., 500 ms – RTCongruent [400 ms, 600 ms, Trialn]; see Fig. 3A). Using this approach, the first individual exhibits a distribution of non-biased responses (0 ms), whereas the other individual exhibits a distribution of vigilance (+100 ms) and avoidance (−100 ms) responses. After categorizing individual responses in this fashion, difference scores can subsequently be averaged within response-based conditions to create separate measures of both vigilance and avoidance severity (see Fig. 3B). Moreover, neutral trials can be included as a reference condition to separately examine attentional orientation (RTNeutralMean – RTCongruent [Trial1 … Trial2 … Trialn]) and attentional disengagement (RTIncongruent[Trial1 … Trial2 … Trialn] – RTNeutralMean). By including every trial with this response-based approach, inter-individual differences in threat-related attention can be accurately captured without the data loss that accompanies nearest-neighbor computation approaches.

However, isolated measures of vigilance and avoidance may fail to capture the interplay between these threat-related attention mechanisms and are purportedly confounded by variability in reaction time (Kruijt et al., 2016). To address both issues, response-based measures can be utilized to characterize individual differences in the relative magnitude of vigilance and avoidance (|Vigilance|:|Avoidance|). For example, one individual may demonstrate vigilance responses (100 ms) that are double in absolute magnitude relative to avoidance responses (50 ms), which would produce a 2:1 Vigilance:Avoidance ratio. In contrast, a second individual may demonstrate vigilance responses (75 ms) that are triple in absolute magnitude of avoidance responses (25 ms), which would produce a 3:1 Vigilance:Avoidance ratio (see Fig. 3B). Beyond characterizing the interplay between threat-related attention mechanisms, the direct contrast removes common variance associated with reaction-time variability. As a result, such ratio measures may improve detection of anxiety-related associations compared to standard computation measures while also controlling for factors that confound dynamic computation measures.

To validate our response-based computation approach, we first compared the internal consistency of attention measures computed using standard and response-based approaches. Additionally, we compared the number of trials required to demonstrate internal consistency benchmarks using each computation method. Given that detection of treatment-related changes requires stability of a measurement across time, we also compared the test-retest reliability of attention measures. Based on preliminary psychometric findings using dynamic approaches (Zvielli et al., 2014b), we hypothesized that response-based attention measures would demonstrate superior internal consistency and test-retest reliability as well as stronger anxiety-related associations that are independent of global reaction time variability.

Section snippets

Participants

Dot-probe data were analyzed from 148 undergraduate students (19.22 ± 1.42 years old, 84 women) enrolled in an introductory psychology course at the University of Miami. Participants were compensated with research credit. The present study combined two larger attention samples in their entirety. Participants were either unselected (Sample 1: n = 89, 19.39 ± 1.45 years old, 49 women) or had high trait levels of social anxiety (Sample 2: n = 59, 18.91 ± 1.33 years old, 38 women). In Sample 2,

Standard attention measures

When computed using a standard approach, all attention measures demonstrated unacceptable levels of internal consistency (See Fig. 3). Specifically, attention bias demonstrated unacceptable levels of internal consistency across estimates of both split-half reliability [r(146) = 0.51, p < 0.001] and Cronbach's alpha [max: a = 0.25 at 16 trials]. Similarly, both the orientation [r(146) = 0.39, p = 0.002; max: a = 0.41 at 12 trials] and disengagement components [r(146) = 0.40, p < 0.001; max: a

Discussion

Our novel response-based computation approach enabled a systematic comparison of the clinical utility of standard and response-based computation approaches across measures of attention bias, orientation and disengagement of attention. Compared to standard computation approaches, response-based attention measures demonstrated markedly improved psychometric properties. Standard computational measures invariably demonstrated unacceptable internal consistency, no test-retest reliability, and no

Conflicts of interest

Dr. Britton received support from National Institute of Mental Health (R00 MH091183) during the conduct of the study.

Acknowledgements

We would like to thank Ilana Seager, Katherine Walukevich, Brittany Tripp, Lee Kissel, Marigloria Maldonado-Puebla, Steve Gomez, and Juliana Berhane for their help with data collection. Additionally, we would like to thank Dr. Stewart Shankman and Emily Meissel for their helpful comments.

References (35)

  • Y. Bar-Haim et al.

    Threat-related attentional bias in anxious and nonanxious individuals: A meta-analytic study

    Psychological Bulletin

    (2007)
  • M.J. Briggs-Gowan et al.

    Attention bias and anxiety in young children exposed to family violence

    Journal of Child Psychology and Psychiatry

    (2015)
  • J.C. Britton et al.

    Response to learned Threat: An fMRI study in adolescent and adult anxiety

    American Journal of Psychiatry

    (2013)
  • D. Broadbent et al.

    Anxiety and attentional Bias: State and trait

    Cognition & Emotion

    (1988)
  • J.M. Cortina

    What is coefficient Alpha? An examination of theory and applications

    Journal of Applied Psychology

    (1993)
  • I.A. Cristea et al.

    Efficacy of cognitive bias modification interventions in anxiety and depression: meta-analysis

    British Journal of Psychiatry

    (2015)
  • D. Foti et al.

    Differentiating neural responses to emotional pictures: Evidence from temporal-spatial PCA

    Psychophysiology

    (2009)
  • Cited by (24)

    • Ethnic differences in behavioral and physiological indicators of sensitivity to threat

      2022, Journal of Anxiety Disorders
      Citation Excerpt :

      Response-based measures of AB are able to assess this kind of intra-individual variability as they compare the individual’s response on each trial to the individuals’ own average responding. Interestingly, response-based measures that incorporate intra-individual variability in AB to threat have exhibited better reliability and validity compared to traditional measures (Evans and Britton, 2018; Meissel et al., 2021), are associated with separate neural circuitry (Evans et al., 2020) and may, therefore, be more sensitive to detecting between group differences. Indeed, in the present paper, H/L individuals exhibited both slower orientation to threat (which may be indicative of a tendency to direct attention away from threat in an attempt to avoid the negative affect elicited by threat) and slow disengagement from threat (which may reflect more elaborative processing of threats that stem from poor regulatory attention; Cisler & Koster, 2010; Evans et al., 2020).

    • Influence of acute stress on attentional bias toward threat: How a previous trauma exposure disrupts threat apprehension

      2021, International Journal of Psychophysiology
      Citation Excerpt :

      In order to assess attentional biases to threat, the dot-probe task has been widely used (Bardeen and Orcutt, 2011; Bullock and Bonanno, 2013; Chan et al., 2013). In this task, the computation of reaction times provides indices of orientation of attention, disengagement of attention and avoidance (Evans and Britton, 2018). To complement these RT analyses, event related potentials (ERPs) provide a high temporal resolution measure of attentional processes during EEG recording as well as a representation of processing efficiency (the mobilization of additional resources needed to accurately perform the task).

    • Neural mechanisms underlying heterogeneous expression of threat-related attention in social anxiety

      2020, Behaviour Research and Therapy
      Citation Excerpt :

      For example, an anxious individual may orient attention toward threat on some trials, but orient attention away from threat on other trials. The conjoint expression of both orientation patterns would suggest a more complex pattern of threat-related attention mechanisms in social anxiety (Evans & Britton, 2018). To systematically investigate the expression of threat-related attention and its neural mechanisms in social anxiety, the present study used both standard and novel dot-probe computation approaches in conjunction with fMRI methodology.

    View all citing articles on Scopus
    View full text