Is a Positive Developmental-Behavioral Screening Score Sufficient to Justify Referral? A Review of Evidence and Theory

doi:10.1016/j.acap.2017.01.016

Academic Pediatrics

Volume 17, Issue 5, July 2017, Pages 464-470

https://doi.org/10.1016/j.acap.2017.01.016 Get rights and content

Abstract

In their recommendations on screening for autism and developmental disabilities, the American Academy of Pediatrics recommends referral subsequent to a positive screening result. In this article, we argue that positive screening results are not always sufficient to justify a referral. We show that although positive predictive values are often low, they actually overstate the probability of having a disorder for many children who screen positive. Moreover, recommended screening thresholds are seldom set to ensure that the benefits of referral will equal or exceed the costs and risk of harm, which is a necessary condition for an optimal threshold in decision analysis. Drawing on recent recommendations for the Institute of Medicine/National Academy of Medicine, we discuss the implications of this argument for pediatric policy, education, and practice. In particular, we recommend that screening policies be revised to ensure that the costs and benefits of actions recommended in the event of a positive screen are appropriate to the screening threshold. We recommend greater focus on clinical decision-making in the education of physicians, including shared decision-making with patients and their families. Finally, we recommend broadening the scope of screening research to encompass not only the accuracy of specific screening instruments, but also their ability to improve decision-making in the context of systems of care.

Section snippets

Overstated Values

Group-level statistics like PPV overstate the value of recommended screening thresholds. Screening instruments are often evaluated according to the proportion of children with disabilities who score positive (ie, sensitivity), and the proportion of children without disabilities who score negative (ie, specificity). Many guidelines—including the AAP's—recommend minimum values of 70% for sensitivity as well as specificity.1, 6 Statistics like sensitivity and specificity are extremely useful for

Unaccounted Costs

Recommended screening thresholds often fail to account for the expected costs and benefits of available referral options. How high must threshold probability be to justify a referral? Decision analysis offers a clear response: A child should be referred if the probability of disability is high enough that the benefits of referral are likely to outweigh costs and potential for harm.¹⁰ Because of uncertainty about the precise extent and severity of a child's symptoms, it is often unclear whether

Screening Policy

We argue that policy statements should not recommend referral for all children with positive screening results unless there is evidence that the benefits are likely to outweigh costs and harms for children who score at recommended screening thresholds. Because of the absence of such evidence for developmental-behavioral screening, current recommendations should be significantly revised to offer more specific guidance regarding expectations for individual practitioners. Referrals for all

Education

With increased emphasis on clinical judgement, improvements in education are also warranted. To improve the diagnostic process, the IOM/NAM specifically recommends enhancing professional education and training in “clinical judgement” and “appropriate use of diagnostic tests and the application of these results on subsequent decision-making.”¹¹^pp364 If screening is not sufficient to drive referral decisions, then enhanced training in clinical reasoning might be needed. A large literature in

Research

Most research on screening focuses on the standardization, construct validity, and diagnostic accuracy of screening instruments. In contrast, the IOM/NAM describes diagnosis as a longitudinal, iterative process that often involves multiple team members and close engagement with families. This perspective suggests that research on screening should not be confined to the accuracy of screening instruments when used alone, but should also address their ability to improve clinical decision-making in

Acknowledgments

Financial disclosure: Research support was provided by NICHD R01 HD072778. The opinions expressed in this article are the author's own and do not reflect the view of the National Institutes of Health, the Department of Health and Human Services, or the United States government.

References (47)

K.E. Zuckerman et al.
Incomplete specialty referral among children in community health centers
J Pediatr
(2011)
M. Stuart et al.
Caregiver burden after receiving a diagnosis of an autism spectrum disorder
Res Autism Spectrum Disord
(2009)
E.D. Crawford et al.
An approach using PSA levels of 1.5 ng/mL as the cutoff for prostate cancer screening in primary care
Urology
(2016)
D.M. Lloyd-Jones et al.
Framingham risk score and prediction of lifetime risk for coronary heart disease
Am J Cardol
(2004)
Identifying infants and young children with developmental disorders in the medical home: an algorithm for developmental surveillance and screening
Pediatrics
(2006)
Centers for Disease Control and Prevention. Developmental Monitoring and Screening for Health Professionals. Available...
R.C. Sheldrick et al.
Identification of developmental-behavioral problems in primary care: a systematic review
Pediatrics
(2011)
L.S. Wissow et al.
Universal mental health screening in pediatric primary care: a systematic review
J Am Acad Child Adolesc Psychiatry
(2013)
R.C. Sheldrick et al.
A system dynamics model of clinical decision thresholds for the detection of developmental-behavioral disorders
Implement Sci
(2016)
F. Glascoe
Screening for developmental and behavioral problems
Ment Retard Dev Disabil Res Rev
(2005)

R.C. Sheldrick et al.

Thresholds and accuracy in screening tools for early detection of psychopathology

J Child Psychol Psychiatry

(2015)

J.V. Lavigne et al.

Screening for mental health problems: addressing the base rate fallacy for a sustainable screening program in integrated primary care

J Pediatr Psychol

(2016)

P.E. Meehl et al.

Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores

Psychol Bull

(1955)

S.G. Pauker et al.

The threshold approach to clinical decision making

N Engl J Med

(1980)

Improving Diagnosis in Health Care

(2015)

R.J. Baron

What's keeping us so busy in primary care? A snapshot from one practice

N Engl J Med

(2010)

P.F. Belamarich et al.

Drowning in a sea of advice: pediatricians and American Academy of Pediatrics policy statements

Pediatrics

(2006)

K.E. Zuckerman et al.

Barriers to specialty care and specialty referral completion in the community health center setting

J Pediatr

(2013)

C.B. Forrest et al.

Specialty referral completion among primary care patients: results from the ASPN Referral Study

Ann Fam Med

(2007)

M.L. McPheeters et al.

Screening for Autism Spectrum Disorder in Young Children: A Systematic Evidence Review for the U.S. Preventive Services Task Force. Evidence Synthesis No. 129. AHRQ Publication No. 13-05185-EF-1

(2016)

J.P. Guevara et al.

Effectiveness of developmental screening in an urban setting

Pediatrics

(2013)

A.G. Jutel et al.

Putting a Name to It: Diagnosis in Contemporary Society

(2014)

B. Djulbegovic et al.

When is it rational to order a diagnostic test, or prescribe treatment: the threshold model as an explanation of practice variation

Eur J Clin Invest

(2015)

Cited by (38)

Assessing the Validity of the Baby Pediatric Symptom Checklist Using a Nationally Representative Household Survey
2023, Academic Pediatrics
The Baby Pediatric Symptom Checklist (BPSC) is a screening tool developed for detecting behavioral or emotional concerns among parents of children younger than 18 months. Nationally representative survey data have not yet been used to assess the validity of the BPSC, nor to evaluate its appropriateness for use among children between 18 and 23 months old. The current study assesses the validity of the BPSC using data from the National Health Interview Survey (NHIS).
Data from the 2019 NHIS were used to evaluate the 12-item BPSC screening tool among a nationally representative sample of children 2 to 23 months. Confirmatory factor analysis (CFA) and differential item functioning (DIF) were used to assess construct and predictive validity and test how response items differed by selected sociodemographics. Quantile regression was used to calculate 50th, 70th, and 90th percentiles for age-based normative curves of the previously established domains of irritability, inflexibility, and difficulty with routines.
A 3-factor CFA produced comparable results to the original study. Tests of DIF did not reveal any significant effects for the child's sex, race and Hispanic origin, household urbanization level, number of children in family, or respondent type (mother, father, other). In addition, DIF was not found between children aged 2 to 17 months and 18 to 23 months. Age-based normative data were calculated for each subscale.
The use of the BPSC in a nationally representative survey produced findings comparable to those of the original-validation study. The NHIS can be used to track BPSC scores over time at the population-level.
Screening Accuracy of the Parent-Report Preschool Strengths and Difficulties Questionnaire in Primary Care
2023, Academic Pediatrics
To evaluate the screening test accuracy and reliability of the parent-report preschool Strengths and Difficulties Questionnaire (P-SDQ) in primary care settings.
Children 24 to 48 months were recruited at scheduled primary care visits in Toronto, Canada. Parents completed the P-SDQ at baseline, 2, and 12 weeks. At 12 weeks, parents were invited to a semistructured diagnostic phone interview, the Preschool Age Psychiatric Assessment (PAPA). Criterion validity between baseline P-SDQ scores (Total Difficulties Score [TDS], internalizing and externalizing subscale) and Diagnostic and Statistical Manual, 5th edition diagnoses on PAPA was evaluated using area under the curve (AUC) and calculating screening test properties (sensitivity and specificity). Test-retest reliability at baseline and 2 weeks was evaluated using intraclass correlation coefficient.
A total of 183 children were enrolled, mean age 39.3 (SD 7.4) months, 46.4% male, 120 (66%) completed P-SDQ at 2 weeks, 107 (58%) completed PAPA at 12 weeks. Of those with a PAPA, 26 (24%) had any psychiatric diagnosis, 17 (16%) had internalizing disorders and 4 (4%) had externalizing disorders. TDS identified any diagnosis with AUC = 0.67 (95% confidence intervals (CI): 0.55, 0.79); internalizing subscale identified internalizing disorders with AUC = 0.61 (95% CI: 0.47, 0.74); externalizing subscale identified externalizing disorders with AUC = 0.77 (95% CI: 0.60, 0.94). Sensitivity and specificity, and test-retest reliability were satisfactory for TDS and externalizing subscale, and less satisfactory for the internalizing subscale.
The externalizing subscale has sufficient accuracy and reliability to identify children aged 2 to 4 years at risk for attention deficit/hyperactivity disorder and disruptive behavior disorders in primary care.
Math Matters: Why Positive Screening Rates Cannot Substitute for Prevalence
2022, Academic Pediatrics
Systematic screening of autism spectrum disorders in French Maternal-Child Health Services in pre-school children: Which care pathway? What feasibility?
2022, Neuropsychiatrie de l'Enfance et de l'Adolescence
L’objectif de cette étude était de décrire le parcours de soins des enfants de 18 à 30 mois à risque élevé de troubles du spectre de l’autisme (TSA) suivis en protection maternelle et infantile (PMI) dans les Bouches du-Rhône. L’objectif secondaire était d’évaluer la faisabilité du dépistage systématique des TSA en consultation de PMI.
Les enfants inclus dans l’étude descriptive du parcours de soins avaient 18 à 30 mois et un risque élevé de TSA au M-CHAT-R. Les données de la consultation de dépistage, 6, 12 et 24 mois après leur consultation ont été recueillies rétrospectivement. Pour l’étude qualitative transversale, des entretiens ont été menés auprès de médecins de PMI.
Parmi les 2458 enfants dépistés entre septembre 2015 et décembre 2018, 76 enfants (3,1 %) avaient un risque élevé de TSA (M-CHAT-R ≥ 8). Ils ont bénéficié rapidement de mesures de socialisation, pour atteindre une importante intégration en école ordinaire. Le délai médian de consultation d’un médecin spécialiste à partir du dépistage était de 81 jours [40 ; 145]. Les principaux freins au dépistage systématique des TSA exprimés par les médecins de PMI étaient : le temps nécessaire au dépistage, la barrière de la langue, les difficultés de compréhension, l’offre de diagnostic et de soins limitée. Des leviers spécifiques au contexte de consultation de PMI ont été rapportés.
De nombreuses actions peuvent être entreprises entre les âges de deux et trois ans pour ces enfants dépistés à risque élevé de TSA en PMI.
The objective of this study was to describe the care pathway, socialization, and the modalities of scholarization of children aged 18 to 30 months at high risk of Autism Spectrum Disorders (ASD) followed in Maternal-Child Health Services (PMI) in the Bouches du-Rhône area in France. The secondary objective was to evaluate the feasibility of systematic screening for ASDs in PMI consultations.
A mixed observational (quantitative-qualitative) study was carried out in the Bouches-du-Rhône department in France. The children included in the care pathway descriptive study were 18 to 30 months of age and at high risk for ASD at the M-CHAT-R (M-CHAT-R ≥ 8). Data from the screening visit, and 6, 12 and 24 months thereafter were collected retrospectively. For the cross-sectional qualitative study, interviews were conducted with PMI medical practitioners.
From September 2015 to December 2018, of the 2458 children screened, 76 children (3.1%) were at high risk for ASD (M-CHAT-R ≥ 8) and the socialization of these children was initially poor since it was done only within the family (69% at the time of screening). These children quickly benefited from socialization measures, achieving a significant integration into regular school. Indeed, the schooling rate was very high at 24 months after the screening since 89% of the children were enrolled in school, most in regular schools, with human support. The care provided by a speech therapist, a psychomotrician or a psychologist increased over time. However, the speech therapist's follow-up was a little later than that of the psychomotrician. The median time to see a child psychiatrist or neuropaediatrician or pediatrician from screening was 81 days [40; 145]. Nine interviews were conducted with the referring PMI medical practitioners in the Bouches-du-Rhône department. The main barriers to routine screening for ASDs expressed by PMI medical practitioners were: the time required for screening, language and comprehension barriers, and limited diagnostic and care options. Levers specific to the PMI consultation setting were reported.
This study provides a glimpse of certain characteristics of the care, socialization and schooling pathway of these very young children, which is little documented to date in France. In view of the elements resulting from the interviews, as well as the large number of children having been screened over three years (n = 2458), the feasibility of systematic screening for ASD at 18–30 months in the context of consultations with PMIs in the department seems to be established. Although specialized care management is confronted with structures, and the time frames described and studied were long, many actions can be undertaken between the ages of two and three years for these children.
Combining Two Developmental Screening Tests to Improve Predictive Accuracy
2020, Academic Pediatrics
Citation Excerpt :
Further improvement in predictive value was obtained when the referral threshold was increased while keeping the nonreferral threshold stable and defining an in-between group with LR's too weak to qualify for referral and too strong to dismiss it. Variants of this approach have been championed by Sheldrick et al8,9,19 and others20 as resembling what happens in clinical practice. Clinicians often feel very confident about referring some children and not referring others but seek additional information to resolve uncertainty in making a decision for individuals in the in-between group.
Single developmental screening tests are insufficient for supporting clinical decisions, showing less than 50% of those referred are actually delayed. Combining two tests can address this problem. We used likelihood ratios (LR) to evaluate the predictive value (PV) of combining the Ages and Stages Questionnaires (ASQ) and Parents Evaluation of Developmental Status (PEDS) screening tests.
We used data from Limbos and Joyce that included information on the ASQ, PEDS, and an age-appropriate diagnostic assessment of delay on 328 children (ages 12–60 months) drawn from primary care. LRs were calculated for each of the available ASQ/PEDS combinations then rank ordered. PV was examined using different LRs to define referral and nonreferral thresholds.
Setting the referral threshold at LR 2.0 for combined scores decreased the number of false positives from 66 to 28, increased false negatives from 6 to 9 with positive predictive value (PPV) improving significantly from 30% to 47% (P < .05). Sensitivity dropped from 82% to 74% and specificity improved from 72% to 90%. When the LR referral threshold increased, PV improved but an inconclusive group emerged that required more information for decision-making.
Combining the ASQ and PEDS significantly improved predictive accuracy of a referral in a single threshold analysis. Predictive value improved further as requirements for referral increased and an inconclusive group needing further study was recognized. Systematic research is needed to identify what readily available information will assist decision making in this group.
Psychometric Properties of Two Developmental Screening Instruments for Hispanic Children in the Philadelphia Region
2019, Academic Pediatrics
Citation Excerpt :
The generally poor positive predictive values for both instruments imply that a large number of false positives will be associated with both tests, and the generally good negative predictive values suggest that both instruments functioned quite well in ruling out the presence of any developmental delays. Positive predictive value is a common characteristic of screening tools, especially for lower incidence concerns.32,33 These findings support that screening should be a part of a broader process of developmental surveillance which includes consideration of parental concerns, medical history, and the family and neighborhood environment.
To assess the validity of Spanish versions of the Survey of Well-being of Young Children (SWYC) Milestones and the Ages & Stages Questionnaire, Third Edition (ASQ-3), and to document the rates of developmental delays in an urban cohort of children with Hispanic parents.
Spanish-speaking families with a child 9 to 60 months of age (N = 991) were initially screened using Spanish translations of the SWYC Milestones and the ASQ-3. A stratified random sample of 494 of these children subsequently received standardized clinical assessment to confirm the presence of developmental delays. Reverse weighting corrected for the selection bias inherent in the stratification scheme.
Fifty-five percent of toddlers (9 to 41 months of age) and 34.8% of preschoolers (42 to 60 months of age) scored in the moderately to severely delayed range, most frequently in language. Sensitivity and specificity for toddlers with severe delays associated with the SWYC were 0.69 and 0.64, respectively, and 0.55 and 0.75 for the ASQ-3. Sensitivity and specificity for preschoolers with severe delays associated with the SWYC were 0.87 and 0.58, respectively, and 0.71 and 0.86 for the ASQ-3.
Although psychometric properties of the Spanish translated versions are not as strong as the English versions, the findings suggest that both the SWYC Milestones and ASQ-3 represent promising tools for identifying Hispanic children with developmental delays. The rate of delays were consistent with other studies showing a high percentage of Hispanic children with developmental delays, most frequently in language skills.

View all citing articles on Scopus

: The authors have no conflicts of interest to disclose.

View full text

Narrative ReviewIs a Positive Developmental-Behavioral Screening Score Sufficient to Justify Referral? A Review of Evidence and Theory

Abstract

Section snippets

Overstated Values

Unaccounted Costs

Screening Policy

Education

Research

Acknowledgments

J Pediatr

Res Autism Spectrum Disord

Urology

Am J Cardol

Identifying infants and young children with developmental disorders in the medical home: an algorithm for developmental surveillance and screening

Pediatrics

Identification of developmental-behavioral problems in primary care: a systematic review

Pediatrics

Universal mental health screening in pediatric primary care: a systematic review

J Am Acad Child Adolesc Psychiatry

A system dynamics model of clinical decision thresholds for the detection of developmental-behavioral disorders

Implement Sci

Screening for developmental and behavioral problems

Ment Retard Dev Disabil Res Rev

Thresholds and accuracy in screening tools for early detection of psychopathology

J Child Psychol Psychiatry

Screening for mental health problems: addressing the base rate fallacy for a sustainable screening program in integrated primary care

J Pediatr Psychol

Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores

Psychol Bull

The threshold approach to clinical decision making

N Engl J Med

Improving Diagnosis in Health Care

What's keeping us so busy in primary care? A snapshot from one practice

N Engl J Med

Drowning in a sea of advice: pediatricians and American Academy of Pediatrics policy statements

Pediatrics

Barriers to specialty care and specialty referral completion in the community health center setting

J Pediatr

Specialty referral completion among primary care patients: results from the ASPN Referral Study

Ann Fam Med

Screening for Autism Spectrum Disorder in Young Children: A Systematic Evidence Review for the U.S. Preventive Services Task Force. Evidence Synthesis No. 129. AHRQ Publication No. 13-05185-EF-1

Effectiveness of developmental screening in an urban setting

Pediatrics

Putting a Name to It: Diagnosis in Contemporary Society

When is it rational to order a diagnostic test, or prescribe treatment: the threshold model as an explanation of practice variation

Eur J Clin Invest

Narrative Review
Is a Positive Developmental-Behavioral Screening Score Sufficient to Justify Referral? A Review of Evidence and Theory