Figures
Abstract
Although there is evidence of a distinct profile of executive dysfunction in Williams syndrome (WS), a rare genetically based neurodevelopmental disorder, the utility of informant reports of everyday executive function (EF) impairments and their relation to intelligence is not yet clear. Here we aimed to evaluate the functional impact of executive dysfunction in adults with WS and to establish the validity of child and adult versions of the most commonly used rating scale for EF assessment, the Behaviour Rating Inventory of Executive Function (BRIEF). We were also interested in whether distinct components of everyday EF relate to intelligence in WS. Parent report child (BRIEF-C) and adult (BRIEF-A) ratings were collected on 20 adults with WS (aged 18.5 to 53 years), with a mean IQ of 60.95 (SD = 17.67). Neuropsychological measures of EF included: The Shape School Test (Espy, 2007); select subdomains of EF from the Woodcock-Johnson III Tests of Cognitive Abilities, Australian Adaptation (WJ III COG); and select subdomains from the Vineland Adaptive Behaviour Scales, Second Edition—Parent Survey (Vineland-II). Results showed that the BRIEF-A, but not the BRIEF-C, was the most highly correlated with neuropsychological measures of EF, suggesting that it was a valid measure of the profile of EF impairments in adults with WS. The profile of everyday EF dysfunction revealed relative impairments in monitoring, working memory, planning and organisation in WS. In addition, both neuropsychological and rating scale measures showed an association between the shifting component of EF and intelligence. These findings indicate that the BRIEF-A is a valid measure of the multidimensional nature of real-world impairments in EF, and highlight its utility as a less labor intensive and low-cost screening tool for measuring specific EF impairments that could become the focus of targeted intervention in adults with WS.
Citation: Hocking DR, Reeve J, Porter MA (2015) Characterising the Profile of Everyday Executive Functioning and Relation to IQ in Adults with Williams Syndrome: Is the BRIEF Adult Version a Valid Rating Scale? PLoS ONE 10(9): e0137628. https://doi.org/10.1371/journal.pone.0137628
Editor: Bert De Smedt, University of Leuven, BELGIUM
Received: November 20, 2014; Accepted: August 19, 2015; Published: September 10, 2015
Copyright: © 2015 Hocking et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: Apex Foundation for Research Into Intellectual Disability awarded to DH and MP, (http://www.afrid.com.au). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
It is often the case that clinicians or researchers need to utilise rating scales and questionnaires as a practical way of quantifying cognitive or psychological functioning in both typical and atypical populations. Although traditional neuropsychological measures arguably provide objective and standardised cognitive assessments, the use of self-report or parent-report questionnaires can provide a more multifaceted measure of everyday behaviours in real life contexts [1]. However, in atypically developing populations with intellectual disability (ID), the extent to which reliability and validity is affected based on the developmental appropriateness of a particular rating scale, and its application to a specific neurodevelopmental disorder, is an important yet neglected issue in developmental neuropsychology.
One domain of cognitive functioning that currently has no universally accepted method of assessment in neurodevelopmental disorders is executive function (EF) skills. EF is an umbrella term for a set of higher-order cognitive processes that organise and regulate goal-directed behaviour in novel and challenging situations [2–4]. EF includes complex cognitive functions such as: cognitive flexibility; task initiation and response inhibition; planning and organisation; working memory and regulation of emotion and behaviour [5–7]. Even within each aspect of EF, such as working memory, there are often presumed to be dissociable components. For example, one influential theory of working memory is Baddeley’s model [8,9], which identifies two domain-specific temporary storage systems for verbal (“phonological loop”) and visuospatial (“visuospatial sketchpad”) information, and a limited-capacity central executive involved in the control and regulation of cognitive processes. It is well established that the central executive is further fractionated into three separable but not completely independent component processes, namely: the ability to shift between mental sets or tasks (‘shifting’); selectively attending to stimuli and inhibiting prepotent responses (‘inhibition’); and updating and monitoring in working memory (‘updating’) [10]. There has been considerable controversy as to whether these components of EF can be viewed as separable or rather as a unitary construct in early development [11–13], and this controversy equally applies to neurodevelopmental disorders associated with ID.
Willner et al. [14] employed child versions of performance-based measures of EF in adults with mild to moderate IDs, and found three factors (i.e. inhibition, shifting and updating working memory) that resembled the tripartite model of EF proposed in typical development [10]. Thus, there appears to be some utility in adopting this multi-dimensional model of EF in neurodevelopmental disorders where it is possible that some components of EF may be more impaired than others relative to overall mental age (MA) level. This has important implications for targeted interventions in neurodevelopmental disorders such as autism where select deficits in behavioural flexibility or shifting have been observed [15–17]. For example, preliminary studies in autism provide support for improvements in core behaviour regulation problems (i.e. ability to follow rules, make transitions and be flexible) when behavioural flexibility is directly targeted in contextually-based EF interventions [18].
Williams syndrome (WS) is a rare (1 in 7,500) neurodevelopmental disorder resulting from a microdeletion of approximately 25–28 genes on the long arm of chromosome 7q11.23 [19,20]. The syndrome is characterised by distinctive facial morphology, cardiovascular problems, mild to moderate intellectual disability, and gregarious and empathic social personalities. Individuals with WS have a high frequency of attention-deficit/hyperactivity disorder (ADHD) and/or Generalised anxiety disorder as co-morbid diagnoses, with a higher than expected prevalence compared to the general population in both instances [21,22].
The disorder has attracted a great deal of interest from developmental neuropsychologists due to a unique profile of strengths and weaknesses in the neuropsychological and EF profile. These include a relative proficiency in some aspects of language and short-term verbal memory, alongside weaknesses in non-verbal domains associated with visuospatial and visuomotor abilities [23–25]. Recent studies have generally been consistent with a fractionation of EFs that are more or less compromised in individuals with WS [26–29]. There are an increasing number of studies that suggest poor performance in WS relative to both MA matched typically developing (TD) individuals and task norms in: response inhibition [30–32]; dual-task coordination [33,34]; shifting, working memory, planning and organisation [27]. These deficits across a number of EF domains appear to be more specific to the visuospatial modality in WS when compared to the verbal domain [27,31]. With respect to working memory (WM), previous studies have indicated that individuals with WS have relatively weaker visuospatial WM compared to MA, whereas their performance on verbal WM tasks is comparable to MA matched TD children [35]. This has been suggested as a syndrome-specific profile in a recent study by Carney et al. [36], who examined the effect of task modality on executive function in WS and Down syndrome (DS) when compared to TD children. Although the typical profile of weaker visuospatial WM was indicated in the WS group, there were no developmental delays in verbal short-term memory between individuals with WS and TD children matched for MA or chronological age. These findings suggest that EF is not uniformly impaired in individuals with WS.
In addition to limitations of face-to-face performance measures of EF in WS and other neurodevelopmental disorders, there is considerable controversy regarding the precise relationships between IQ and performance-based EF skills. Friedman et al. [37] examined performance on three types of EF (inhibiting, working memory and shifting) in a large sample of typically developing adolescents aged 16 to 18 years, and found working memory to be highly correlated with IQ measures, but not shifting or inhibitory control. However, in children with specific learning and intellectual disabilities, evidence is consistent with the notion that working memory is also associated with specific learning disabilities after taking into account variations in IQ [38]. When comparing children with mild learning disabilities to a group with more severe intellectual disability (normal vs low IQ), Maehler and Schuchardt [38] revealed no significant differences in working memory between the two groups of children, despite their varying levels of intelligence. To further add to these conflicting findings, Osório et al. [3] recently showed that working memory (but not shifting or inhibition) was most strongly associated with IQ in both individuals with WS and in typically developing (TD) controls, albeit that the strength of the association was considerably higher in the WS group. Notably, the association among the three EFs was substantially greater in individuals with WS than in the TD group, suggestive of a global compromise of EFs in WS [3]. However, the relationships between the everyday profile of EF skills and intelligence are, as yet, unclear in WS, and are therefore worthy of further investigation.
In addition to the theoretical debate regarding the multi-dimensional nature of EF in neurodevelopmental disorders, there are several inherent limitations of performance-based measures of EF in neurodevelopmental disorders. One particular issue is that many performance-based tests of EF rely heavily on lower level abilities that are typically impaired in individuals with compromised intellectual functions, such as processing speed and fine motor control. Another important limitation is that relating to the structured and interactive nature of performance-based tests of EF that may limit their ecological validity in terms of measuring the control of everyday real-world behaviour. That is, standardised one-on-one testing environments of performance-based tests of EF provide structure and cues, and monitoring that may reduce cognitive demands associated with critical EF processes that otherwise would be impaired in less structured settings [1]. Thus, performance-based measures may not be representative of more complex everyday EF in the real-world environment. These limitations can be overcome with the development of rating scale approaches that measure EF demands in the everyday environment (e.g. the BADS-C, [39]), and parent-report behaviour rating scales such as the Behaviour Rating Inventory of Executive Function for children (BRIEF-C, [40]), which can complement face-to-face performance-based measures. Importantly, a related issue is whether rating scales developed for children can be considered developmentally appropriate when employing them for adults with ID, especially when overall MA level would be more commensurate with the use of a child version of a rating scale. This is a particularly relevant issue for studies using rating scales in WS that typically include both children and adults across a wide age range [27,41], and where child versions of rating scales are often utilised, even for adults with WS.
The Behavior Rating Inventory of Executive Function—Adult version (BRIEF-A) was originally developed and adapted for use with typically developing adults, but has since been utilised for a range of acquired and neurodevelopmental disorders including: autism [42–44][; attention deficit hyperactivity disorder (ADHD; Rotenberg-Shpigelman et al., 2008), and individuals with Traumatic Brain Injury (TBI, [45]). The adult version of the BRIEF assesses functional problems related to everyday executive functioning and is separated into two indices (Metacognition and Behavioural Regulation), and nine scale scores. The Metacognitive Index (MI) consists of five scales: task monitor; organisation of materials; plan/organise; working memory and behavioural initiation. The Behavioural Regulation Index (BRI) contains four scales: emotional control; shift; inhibit and self-monitor. Although the psychometric properties of the child BRIEF have been well documented [46,47], there is very little information on the validity of the BRIEF-A version in adult samples with neurodevelopmental disorders associated with ID. The existing studies that have examined EF profiles using the BRIEF in young children with ID are very limited. Lee et al. [48] examined EF profiles with the preschool version of the BRIEF in young children with Down syndrome (DS) and found elevations on the global EF composite, as well as significant deficits in working memory and planning beyond those expected for overall developmental level (see also [49]). Given that very little is known about how EF skills may change in adulthood in people with WS, it is crucial that further studies examine the functional impact of EF deficits in adulthood using validated measures of everyday EFs that are developmentally appropriate for the specific age group under investigation.
Current study
The primary aim of this study was to examine the functional impact of executive dysfunction in an adult sample of WS individuals to establish the most valid version of the BRIEF. In order to achieve this aim, we explored the following research questions. The first question was whether parent ratings differed depending on whether they used the child or adult version of the BRIEF. This was examined through the utilisation of the BRIEF-C and the BRIEF-A questionnaires in adults with WS. The second question related to whether the BRIEF-C or the BRIEF-A was most related to performance-based measures of EF. Finally, once the more valid version of the BRIEF had been established, the third research question related to the extent to which the BRIEF questionnaire captured the multi-dimensional nature of EF, and the possible interrelationships between EF components and IQ in adults with WS.
Based on the lack of previous research in this area, no specific hypotheses were made with respect to the first two research questions. However, with regard to the third research question, and consistent with previous findings [3,26,27], it was hypothesised that adults with WS would display elevated levels of executive dysfunction in inhibition, planning, and working memory (H1). In relation to the extent to which BRIEF ratings are related to IQ, Osório and colleagues’ [3] study reported both working memory and inhibiting to be associated with IQ, but working memory was found to explain a greater proportion of variance in IQ in the WS group. Therefore, it was hypothesised that while both working memory and inhibition would be significantly correlated with IQ, it was predicted that there would be a stronger association between the EF component of working memory and IQ in adults with WS (H2).
Method
Participants
Twenty adults with a diagnosis of WS (11 males and 9 females) and their parents were recruited through the Williams Syndrome Family Support Group (Victoria) and the Williams Syndrome Association Australia. All participants with WS had their diagnosis confirmed with the positive fluorescent in situ hybridisation (FISH) test and displayed the typical ~1.6 Mb heterozygous microdeletion at 7q11.23 [50]. Participants were screened for a history of psychological, developmental or neurological impairments unrelated to the syndrome, such as ADHD, epilepsy or traumatic brain injury assessed by an experienced clinical neuropsychologist (last author). No participant was excluded from the study based on these criteria, as no potential participant displayed the above. The participants’ chronological age ranged from 18.5 to 53 years (M = 27.74, SD = 8.53). The mental age and IQ for each participant was assessed using the Woodcock-Johnson III Tests of Cognitive Abilities, Australian Adaptation (WJ III COG; [51]). Four participants were unable to complete neuropsychological testing at the time of their participation in the current study due to illness or misadventure. An IQ test was conducted (the Woodcock-Johnson Tests of Cognitive Abilities–Revised, Australian Adaptation or WJ-R COG) approximately three to five years prior with each of these individuals as part of another study. Mental age for these four participants ranged from 5.75 to 8.08 years (M = 6.61, SD = 1.06) and overall IQ ranged from 50 to 64 (M = 53, SD = 7.72). These four participants were, therefore, representative of the larger study cohort. Mental age ranged from 5 to 10.83 years (M = 7.15, SD = 1.83). Participants performed in the mild to moderate range of intellectual disability, on average, with IQs ranging from 25 (severe) to 86 (low average) (M = 60.84, SD = 18.15). The mean and range of IQs in the current sample was highly consistent with that reported for intelligence in WS (Mervis et al., 2000), and with previous studies of cognitive heterogeneity in WS (Martens et al., 2008; Porter & Coltheart, 2005). Due to low reading ability and IQ, all participants with WS provided verbal consent after a brief description of the study, and parents provided signed “surrogate” consent according the declaration of Helinski. Ethics approval for the consent procedure and this study was gained from the Macquarie University Human Research Ethics Committee (reference number: 5200900071).
Materials
The Child Behaviour Rating Inventory of Executive Functioning (Parent Form)—BRIEF-C.
The BRIEF-C is designed to assess the executive function behaviours of children aged from 5 to 18 years in a home and school environment [40]. The questionnaire consists of 86 standardised items that measure different aspects of executive function. Seventy-two of those items fall within eight clinical scales (Inhibit, Shift, Emotional Control, Initiate, Working Memory, Plan/Organise, Organisation of Materials, and Monitor). These scales combine to form two indexes (Behavior Regulation Index [BRI] and the Metacognition Index [MI]) and one composite summary score (Global Executive Composite [GEC]). The structure of the BRIEF-C rating scale is outlined in S1 Table. For each clinical scale and index, a T score can be derived with higher scores indicating greater degrees of executive dysfunction, with scores at or above 65 suggesting clinical significance [40]. Psychometric properties show high internal consistency with Cronbach alpha coefficients ranging from .80 to .98 and good test-retest reliability ranging from .76 to .85 [40].
Behaviour Rating Inventory of Executive Function—Adult Version (Parent Report Form)—BRIEF-A.
The BRIEF-A is designed to assess the everyday executive functioning behaviours of adults aged from 18 to 90 years [52]. The questionnaire consists of 75 standardised items, of which items measure different aspects of executive function within nine clinical scales (Inhibit, Shift, Emotional Control, Self-Monitor, Initiate, Working Memory, Plan/Organise, Organisation of Materials, and Task Monitor). The nine clinical scales combine to form two indices (BRI and the MI) and one composite summary score (GEC). The structure of the BRIEF-A rating scale is outlined in S2 Table. For each scale, T scores can be derived with higher scores reflecting greater degree of executive dysfunction and levels of impairment, with scores at or above 65 suggestive of clinical significance. Internal consistency is high with alpha coefficients ranging from .80 to .98, and high test-retest reliability ranging from .91 to .94 [52].
The Shape School test.
Inhibitory control was assessed using the Shape School test [53]. Four conditions (control, inhibit, switch, and both) were presented in the same fixed order from easiest to most difficult conditions as per recommendations [53]. The participant is shown rows of coloured squares and circles, which are depicted as pupils in “The Shape School”. The first condition required each participant to name the colour of each pupil (Control Condition). The second condition required participants to inhibit salient yet irrelevant information and only name the colours of pupils with a happy facial expression while ignoring the sad pupils (Inhibit Condition). In the third condition, some pupils are drawn with hats and the participants are required to switch between naming the hatted pupil’s colour and the hatless pupil’s shape (Switch Condition). The final condition consists of both the happy/sad faced pupils and the hat/hatless pupils (Both Condition). The participant is required to name the shape of the happy pupil with the hat, and name the colour of the happy pupil without a hat while refraining from naming the sad pupil (hatted or hatless). Responses and the time taken to complete each of the conditions were recorded. An efficiency score was calculated from the naming speed and accuracy scores as per recommendations [53].
The Vineland Adaptive Behaviour Scales, Second Edition (Parent Survey)—Vineland-II.
The Vineland-II is a standardised semi-structured parent interview that measures adaptive functioning [54]. The Vineland-II consists of 11 subdomains grouped into four domain composites (Communication, Daily Living Skills, Socialisation, and Motor Skills), with the domain composites used to derive the adaptive behaviour composite. In addition, the Vineland-II provides a Maladaptive Behaviour Index (comprising three subscales; Internalising, Externalising, and Other) that measures undesirable behaviours that may inhibit the development of an individual’s adaptive functioning. Raw scores were converted into standard scores (with population M = 100, SD = 15) based on chronological age. Lower scores reflect greater maladaptive behaviour with scores 2 standard deviations below the normative mean (score of 69) indicating low level of adaptive functioning. The Vineland-II has excellent levels of reliability and internal consistency [54].
Woodcock-Johnson III Tests of Cognitive Abilities, Australian Adaptation—WJ III COG.
The WJ III COG is a standardised measure of intelligence designed for individuals between the ages of 2 and 90 years. The WJ III COG is based on the Cattell-Horn-Carroll (CHC) theory proposing two types of intelligence—fluid (Gƒ) and crystallised (Gc) intelligence, or innate or learned intelligence [55]. The WJ III COG can be explored at three levels (known as Stratum I, II, and III) [55]. Stratum I consists of 20 individual tests, each measuring many of the specific abilities of Gf and Gc intelligence. Further, the battery provides a measurement of seven Stratum II (CHC) factors that are derived from two qualitatively different Stratum I abilities. In addition to Stratum II, several clinically useful clusters may also be obtained by combining several of the Stratum I abilities. Finally, the WJ III COG yields an overall (Stratum III) General Intellectual Ability (GIA) or single g factor, which is similar to the Full Scale IQ (FSIQ) of the Wechsler Scale of Intelligence. In accordance with the WJ III COG manual [55], executive functioning is measured through the Broad Attention, Working Memory, and Executive Process Clinical Clusters.
The raw scores on the WJ III can be converted into standard scores (with population M = 100, SD = 15) based on the age of the participant, with lower scores reflecting greater cognitive difficulties (a score of 69 or below is classified as “very low”) (Mather & Woodcock, 2001). WJ III COG is psychometrically sound with reliability coefficients above .80. Median reliabilities for each cluster are typically at .90 or higher (McGrew & Woodcock, 2001). The GIA measures also display high reliability, with the GIA-Ext ranging from .98 to .99 [55].
Data analyses
To compare ratings on BRIEF-C and BRIEF-A clinical scales, indices and GEC scores, pairwise comparisons and cross-tabulations were employed. In addition, Cohen’s kappa coefficient [56] was reported to provide a measure of agreement, corrected for chance agreement, between the scores on the BRIEF-C and BRIEF-A (0 = poor agreement, .20 –slight agreement, 0.4 = fair agreement, 0.6 = moderate agreement, 0.8 substantial agreement, 1.0 = almost perfect agreement). T scores were utilised in the analyses of the BRIEF-C and BRIEF-A measures, and were calculated based on chronological age. Item analysis was then conducted to compare each item pair on the BRIEF-C and BRIEF-A rating scales in order to investigate how similar each item pair was rated. Identical items were chosen for analysis as these were expected to be highly correlated. Identical items were also investigated using pairwise comparisons and Cohen’s kappa coefficient. Overall, the BRIEF-C Inhibit (D(20) = .20, p = .033) and Shift (D(20) = .23, p = .009) clinical scales and all identical items violated the assumption of normality, and hence nonparametric tests were employed for these scales.
Due to the small sample size, and in order to minimise the likelihood of a Type-II error the p value was set to 0.05 for all analyses (see [57]). For any coefficients at or near the .05 level of significance, moderate to large effect sizes would rule out the likelihood of a Type-I error [57], and, as such, effect sizes were reported for all analyses (0.2 = small effect size, 0.5 = medium effect size, 0.8 = large effect size) [58]. To examine the profile of EF deficits on the BRIEF, repeated measures analyses of variance (ANOVA) was carried out to compare the performance of the adults with WS on each of the BRIEF-A clinical scales, indices and GEC scores. Pearson correlation coefficients were used to investigate whether BRIEF-A ratings were associated with IQ.
Results
Descriptive analysis
Table 1 displays descriptive statistics for the clinical scales, indices and GEC scores of the BRIEF-C and BRIEF-A T scores. The BRIEF-C revealed four clinical scales to be in the clinically significant range (≥65), namely Initiate, Working Memory, Plan/Organise, and Monitor, as well as the MI and GEC indexes. The BRI and its corresponding scales were all found to be within normal limits. The BRIEF-A revealed five clinical scales to be clinically significant, namely Shift, Initiate, Working Memory, Plan/Organise, and Task Monitor, as well as the MI and GEC indexes.
Item analyses and reliability for BRIEF-C and BRIEF-A
To examine the differences between each of the clinical scales, indices and GEC scores across the BRIEF-C and BRIEF-A, paired samples t-tests were performed. Overall, parents rated their son or daughter with WS as significantly higher on the BRIEF-C Monitor scale than on the BRIEF-A Self-Monitor scale (t(19) = 4.61, p < .001, r = .73). Since the BRIEF-C Inhibit and Shift scales were not normally distributed, the Wilcoxon signed-rank test was performed, but no other significant differences emerged (see Table 1).
The ratings of the identically worded items were analysed in order to determine whether the scores of the BRIEF-C and BRIEF-A questionnaires are interpreted by parents in the same way given the context of the adult and child questionnaire and the difference in the surrounding items (Table 2). Paired samples t-tests revealed no significant differences in parent ratings on the nine identical item pairs. Cohen κ statistics for interrater agreement for parent ratings between the BRIEF-C and BRIEF-A identical item pairs can be seen in Table 2. The strength of agreement ranges from poor to moderate, indicating that parents showed different ratings across the identical items on the BRIEF-C and BRIEF-A. The lowest Cohen κ statistic (indicating a poor strength of agreement) can be seen on the item “Has good ideas but cannot get them on paper”. The highest Cohen κ statistic is seen for the item “Is impulsive”, indicating a moderate agreement between the identical pair.
Validity of BRIEF-C and BRIEF-A rating scales in measuring EF in WS
Relationship between the Shape School, WJ III COG, Vineland-II, and BRIEF-C.
No significant relationships were identified between the performance-based measures and the BRIEF-C clinical scales, indices, and GEC scores (p > .05) (see S3, S4, S5 Tables).
Correlational analyses for Shape School and BRIEF-A.
Correlations were examined between the Shape School and the BRIEF-A using Pearson’s correlation coefficients. There was a significant negative correlation between the Shape School Switch condition and the BRIEF-A Shift clinical scale (r = -.47, p = .042) (Table 3). In addition, the Shape School Both condition was significantly negatively related to Shift (r = -.60, p = .006), Working Memory (r = -.51, p = .026), Plan/Organise (r = -.64, p = .003), MI (r = -.59, p = .009), and the GEC (r = -.49, p = .034). There were no other significant relationships between the Shape School conditions and BRIEF-A scales and indices (p > .05).
Correlational analyses for WJ III COG and BRIEF-A.
Correlations were explored between the WJ III COG measures of EF and the BRIEF-A using Pearson’s correlation coefficients (Table 4). The WJ III COG Broad Attention clinical cluster was significantly and negatively correlated with the BRIEF-A clinical scales Shift (r = -.64, p = .007), Working Memory (r = -.52, p = .038), and Plan/Organise (r = -.55, p = .026). The Executive Processes clinical cluster was significantly and negatively related with BRIEF-A Shift (r = -.71, p = .003), Working Memory (r = -.54, p = .039), and Plan/Organise (r = -.64, p = .010) scales. There were no other significant relationships between the WJ III COG tests of EF and the BRIEF-A clinical scales and indices (p > .05).
Correlational analyses for Vineland-II and BRIEF-A.
Correlations between the Vineland-II adaptive behaviour domains and the BRIEF-A revealed a significant positive correlation between Vineland-II Externalising Behaviours and BRIEF-A Emotional Control (r = .54, p = .039) (Table 5).
Behavioural profile of everyday executive functioning in WS
To examine BRIEF-A score profiles, repeated-measures ANOVA was conducted to ascertain differences between the clinical scales, indices and GEC scores. Mauchly’s test indicated that the assumption of circularity was not satisfied (χ2(65) = 180.24, p < .001), and therefore degrees of freedom were corrected using the Huynh-Feldt estimate of sphericity (ε = .64). There were significant differences between the BRIEF-A scores (F(7.08, 134.55) = 13.05, p = < .001, partial η2 = .41), with ratings significantly highest for Task-Monitor, Working Memory, Plan/Organise, MI, and GEC. Moreover, 70% of the current sample scored in the clinically significant range (T ≥ 65) on Task-Monitor, 65% on Working Memory, 60% on Shift, 70% on Plan/Organise, 65% on MI, and 55% on GEC; compared to 35% on Inhibit, 60% on Shift, 30% on Emotional Control, 35% on Self Monitor, 55% on Initiate and 25% on the Behavioural Regulation Index. See S6 Table for behavioural profile on the Shape School, WJ III COG and Vineland-II.
Relation between IQ and the BRIEF-A
The level of associations between BRIEF-A and IQ were explored using Pearson correlation coefficients. The BRIEF-A Shift clinical scale was significantly and negatively correlated with IQ (r = -.52, p = .021), indicating that lower IQ score is related to poorer EF abilities in the component of shifting (Table 6). No other significant relationships were identified.
Discussion
The primary goal of this study was to examine the functional impact of everyday EF impairments in adults with WS using an in-depth analysis of both neuropsychological measures and parent reported EF rating scales: the BRIEF. We compared the validity of parent reports on the BRIEF adult (BRIEF-A) and child (BRIEF-C) rating scales with respect to their relation to neuropsychological measures of EF, and the behavioural profile of everyday EF in adults with WS. This study also explored interrelationships between distinct components of everyday EF and intelligence. The current study is the first to expand on previous performance-based neuropsychological studies in WS by establishing the validity of adult parent reports of everyday EF in adults with WS [3,26,27].
The current findings indicated that BRIEF-A parent ratings were a more valid measure of EF impairments than the BRIEF-C in adults with WS. This was most apparent when examining identical item level ratings between the BRIEF-C and BRIEF-A, with evidence of poor to moderate interrater agreement between the two BRIEF rating scales. There are several possible explanations for the discrepancies in ratings across the BRIEF-C and BRIEF-A in adults with WS. Firstly, it is possible that the context of the surrounding items may influence parent reported ratings for each item on the child and adult BRIEF scales. Secondly, the discrepancies in parent ratings between the BRIEF-C and BRIEF-A might reflect differences in rating scale instructions: that is, the BRIEF-C directs respondents to consider problem behaviours over ‘the past six months’, while the BRIEF-A requires consideration of problem behaviours only ‘over the past month’. Although it is unlikely that EF abilities would change dramatically over time in adults with WS, it remains feasible that the elevated parent reported BRIEF-C scores could be a reflection of the parents’ inclusion of more problem behaviours over a longer period of time, with greater opportunity for poor behaviour exemplars to occur over a six month time period versus a one month time period. Finally, it is conceivable that expectations vary according to the developmental level (child or adult version) in which EF performance is compared by parents relative to typically developing peers. However, this seems unlikely to account for these discrepancies as the parent ratings indicated a greater degree of EF impairments on the BRIEF-C than the BRIEF-A in adults with WS.
The current findings revealed strong associations between individually administered neuropsychological measures of EF and the BRIEF-A scales and, therefore, provide support for the validity of the adult BRIEF in capturing EF impairments reflected in everyday functioning in adults with WS. These findings are important in view of the lack of correlations between performance measures and rating scales on the BRIEF-C in previous studies, leading to debate as to whether rating scales of EF are actually measuring the same underlying construct [59]. However, the current study findings have revealed distinct associations between the putative EF skills measured by individual BRIEF-A clinical scales and performance measures that tap specific components of EF. Performance on the Shape School switch condition (a measure of cognitive flexibility) was negatively associated with ratings on the BRIEF-A Shift, but not Inhibit scale, while the WJ III COG measure of Executive Processes showed negative associations with several BRIEF-A scales including shifting, working memory and planning/organisation. The Shape School is a measure of two main components of EF commonly included in theoretical accounts of executive control [60,61]: the ability to inhibit a prepotent response (response inhibition), and ability to flexibly adapt to changes in the environment (cognitive flexibility). Diamond [11] has argued that the type of mixed switching is perhaps the most difficult because the neural systems operate in a global, diffuse way, making it easier to shift global behaviours than repeatedly invoke different response sets across trials. We conclude that parent reports of everyday shifting as measured by the BRIEF-A appear to tap into these higher demands on cognitive flexibility in a structured context in adults with WS.
Consistent with previous studies from individually administered neuropsychological measures [2,26,27], we found adults with WS exhibited scores in the clinically significant range on several of BRIEF-A subscales. Contrary to previous reports using neuropsychological assessments of EF in WS [3,30,32,35,36], we did not find greater parent reported ratings on the BRIEF-A Inhibit Scale which measures inhibitory control and impulsivity. The discrepancy between inhibitory control deficits between parent report and performance-based neuropsychological findings could reflect the previous samples that included a wide age range including children with WS (e.g. [26,32]). Additionally, we found no specific deficits on the Shape School Inhibit condition in adults with WS, which contrasts with a previous report of poorer response inhibition relative to developmental and IQ level on the same measure in individuals aged 5 to 46 years with WS [32]. Another possible explanation for these discrepancies may relate to inhibitory control impairments known to be associated with attention deficit hyperactivity disorder (ADHD), which is the most common comorbid diagnosis (up to 67.4%) in children with WS [21,22]. While previous studies have typically included children without assessment for ADHD characteristics, in the current study we screened adults with WS and none had an ADHD diagnosis. Thus, the greater difficulties in inhibitory control in previous studies in children with WS [26,32] may be secondary to ADHD characteristics. This interpretation is consistent with a previous study that found a general preservation of inhibition irrespective of the modality in WS individuals without a diagnosis of ADHD [35], and with previous evidence that impulsivity and ADHD characteristics are more common in children with WS [62].
The results of the current study did not support our hypothesis that working memory would be most strongly associated with intelligence in adults with WS. We expected that previous evidence of a link between working memory and IQ in both WS and typically developing children [3,37] would translate to a greater degree of association between BRIEF-A ratings of working memory and intelligence in adults with WS. Rather, our results indicated that in adults with WS lower IQ score was associated with greater EF deficits only on the BRIEF-A Shift clinical scale. The discrepancy between performance measures of working memory and parent reports of EF and their interrelationships with IQ might reflect a lack of overlap between standardised measures and behavioural ratings of EF within the everyday setting. However, this is unlikely as our results indicate that the Switch condition of the Shape School, a performance-based measure of cognitive flexibility, also showed the strongest association with IQ. Additionally, there is evidence that performance on the Dimensional Change Card Sort or Wisconsin Card Sorting Test, measures of shifting ability, are strongly correlated with IQ in young typically developing children and adolescents [63,64]. Alternatively, there is also debate as to whether aspects of fluid and crystallised intelligence as measured on psychometric intelligence tests accurately reflect underlying constructs of EF [63]. Given the previously reported cognitive variability in WS [50,65,66], more research is needed with larger samples to explore different subgroups to further clarify associations between shifting, working memory, and IQ in adults with WS.
The current study findings have significant clinical implications for a model of EF assessment that situates the adult with a neurodevelopmental disorder within the context of their unique environment. To date, the validity of child and adult rating scales of EF have not been compared for the assessment of EF in adults when mental age and IQ level would be commensurate with the use of a child version. These data confirm that the adult version of the BRIEF is the most valid clinical and research tool for assessment of EF in adults with WS, but assumptions about the validity of rating scales for use in adults should not be premised on findings in typically developing children. Another important clinical implication is the appropriateness of individual items on rating scales of EF for adults with an ID. More research is needed to develop rating scales that capture the profile of EF impairments that are specific to intellectual level in adults, in the same way the Developmental Behaviour Checklist is specific to behavioural and emotional disturbance in children with ID [67].
There are several limitations of the current study that should be acknowledged. Firstly, the small sample size of adults with WS is a limitation in capturing variability in ratings of EF in the everyday setting, as well as the lack of correction for multiple correlations. Equally as limiting as a strict correction for multiple comparisons however, is the risk of Type II errors which are arguably more important to avoid in research into neurodevelopmental disorders [57]. It should also be acknowledged that data reduction techniques such as latent class cluster analysis would have been useful in identifying subtypes of related cases when examining the EF profile in WS. Although the small sample size precluded the application of these data reduction techniques, an analysis of the effect sizes revealed moderate to large effects, suggesting that a larger sample could have provided a better picture of inter-correlations in the EF profile in WS. Future studies should attempt to overcome the issue of small sample size, for example by using international, multi-site studies to explore the profile of everyday EF in adults with WS. Secondly, the BRIEF rating scales may be biased as they relied solely on parent report of everyday EF. Future studies are needed to examine rating scales from multiple informants to validate these findings across different settings. Furthermore, the potential for parent report bias should be acknowledged, as some parents may view the behaviour of their son or daughter in an unreasonably negative (or positive) way. Finally, since previous studies have not considered the impact of comorbid ADHD on EF impairments in individuals with WS [27], it would be beneficial to compare EF profiles of adults with WS with and without a comorbid diagnosis of ADHD, and compare EF profiles in WS with other neurodevelopmental disorders with distinct EF profiles (e.g. Down syndrome, fragile X syndrome, autism).
In summary, the current study is the first to examine the validity of child and adult rating scale measures of EF difficulties in everyday settings in adults with WS. Our results confirm that the BRIEF-A is more sensitive than the BRIEF-C in capturing distinct components of EF within this population. The pattern of associations between performance measures and rating scales are suggestive of a fractionised profile of relative strengths and weaknesses across individual subcomponents of EF in adults with WS. Contrary to previous studies from performance-based assessments [3], the current findings show that shifting was the component of EF most closely associated with IQ in adults with WS. These findings have significant clinical implications for the appropriateness of child measures of EF for use in adults with ID, and for intervention approaches that focus on relative strengths in EF to improve areas of specific weakness in WS.
Supporting Information
S1 Table. The BRIEF-C Rating Scale Structure.
https://doi.org/10.1371/journal.pone.0137628.s001
(DOCX)
S2 Table. The BRIEF-A Rating Scale Structure.
https://doi.org/10.1371/journal.pone.0137628.s002
(DOCX)
S3 Table. Correlations Between the Shape School Test and the BRIEF-C (T Scores).
https://doi.org/10.1371/journal.pone.0137628.s003
(DOCX)
S4 Table. Correlations Between the WJ III COG and the BRIEF-C (T Scores).
https://doi.org/10.1371/journal.pone.0137628.s004
(DOCX)
S5 Table. Correlations Between the Vineland-II and the BRIEF-C (T Scores).
https://doi.org/10.1371/journal.pone.0137628.s005
(DOCX)
S6 Table. Behavioural profile for Adult WS on the Shape School, WJ III COG and Vineland-II.
https://doi.org/10.1371/journal.pone.0137628.s006
(DOCX)
Acknowledgments
We would like to thank participants and their families for their ongoing support with our research. During the preparation of this paper Melanie Porter was funded by a research grant from the Williams Syndrome Australia Limited, and Darren Hocking and Melanie Porter were funded by a research grant from the APEX Foundation.
Author Contributions
Conceived and designed the experiments: MAP DRH. Performed the experiments: JR. Analyzed the data: JR. Wrote the paper: DRH JR MAP.
References
- 1. Isquith PK, Roth RM, Gioia G. Contribution of rating scales to the assessment of executive functions. Appl Neuropsychol Child. 2013;2(2):125–32. pmid:23442015
- 2. Anderson V.Assessing executive functions in children: Biological, psychological and developmental considerations. Pediatr Rehabil. 2001;4(3):119–36. pmid:11831564
- 3. Osório A, Cruz R, Sampaio A, Garayzabal E, Martinez-Regueiro R, Goncalves OF, et al. How executive functions are related to intelligence in Williams syndrome. Res Dev Disabil. 2012;33(4):1169–75. pmid:22502842
- 4. Waid-Ebbs JK, Wen P, Heaton SC, Donovan NJ, Velozo C. The item level psychometrics of the behavior rating inventory of executive function-adult version (BRIEF-A) in a TBI sample. Brain Inj. 2012;26(13–14):1646–57. pmid:22876936
- 5. Gioia GA, Isquith P, Guy S, Kenworthy L. Behavior Rating Inventory of Executive Function. Child Neuropsychol. 2000;6(3):235–8 pmid:11419452
- 6. Grattan L, Eslinger P. (1989). High cognition and social behavior. Changes in cognitive flexibility and empathy after cerebral lesions. Neuropsychology. 1989; 3:175–185.
- 7. Willner P, Bailey R, Parry R, Dymond S. Evaluation of executive functioning in people with intellectual disabilities. J Intellect Disabil Res. 2010;54(4):366–79. pmid:20433573
- 8.
Baddeley AD. Working memory, Thought, and Action. New York: Oxford University Press; 2007.
- 9.
Baddeley AD, Hitch G. Working memory. In: Bower GH, editor. The psychology of learning and motivation: Advances in research and theory New York: Academic Press; 1974. pp. 47–89.
- 10. Miyake A, Friedman NP, Emerson MJ, Witzki AH, Howerter A. The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cogn Psychol. 2000;41(1):49–100. pmid:10945922
- 11. Diamond A. All or none hypothesis: a global-default mode that characterizes the brain and mind. Dev Psychol. 2009;45(1):130–8. pmid:19209996
- 12. Diamond A. Close interrelation of motor development and cognitive development and of the cerebellum and prefrontal cortex. Child Dev. 2000;71(1):44–56. pmid:10836557
- 13. Duncan J, Seitz RJ, Kolodny J, Bor D, Herzog H, Ahmed A. et al. A neural basis for general intelligence. Am J Ophthalmol. 2000;130(5):687. pmid:11078860
- 14. Willner P, Bailey R, Parry R, Dymond S. Evaluation of executive functioning in people with intellectual disabilities. J Intellect Disabil Res. 2010 Apr;54(4):366–79. pmid:20433573
- 15. Russell J. Jarrold C. Memory for actions in children with autism: self versus other. Cogn Neuropsychiatry. 1999;4(4):303–31. pmid:16571510
- 16. Lawson RA, Papadakis AA, Higginson CI, Barnett JE, Wills MC, Strang JF, et al. Everyday executive function impairments predict comorbid psychopathology in autism spectrum and attention deficit hyperactivity disorders. Neuropsychology. 2014;29(3):445–53. pmid:25313979
- 17. Ozonoff S, Jensen J. Specific executive function profiles in three neurodevelopmental disorders. J Autism Dev Disord. 1999;29:171–177. pmid:10382139
- 18. Kenworthy L, Anthony LG, Naiman DQ, Cannon L, Wills MC, Luong-Tran C, et al. Randomized controlled effectiveness trial of executive function intervention for children on the autism spectrum. J Child Psychol Psychiatry. 2014; 55(4):374–83. pmid:24256459
- 19. Donnai D, Karmiloff-Smith A. Williams syndrome: from genotype through to the cognitive phenotype. Am J Med Genet. 2000;97(2):164–71. pmid:11180224
- 20. Stromme P, Bjornstad PG, Ramsted K. Prevalence estimation of Williams syndrome. J Child Neurol. 2002;17(4):269–71. pmid:12088082
- 21. Leyfer OT, Woodruff-Borden J, Klein-Tasman BP, Fricke JS, Mervis, CB. Prevalence of psychiatric disorders in 4 to 16 year olds with Williams syndrome. Am J Med Genet B Neuropsychiatr Genet. 2006;141B(6):615–22. pmid:16823805
- 22. Rhodes SM, Riby DM, Matthews K, Coghill DR. Attention-deficit/hyperactivity disorder and Williams syndrome: shared behavioral and neuropsychological profiles. J Clin Exp Neuropsychol. 2011;33(1):147–56. pmid:20700845
- 23. Bellugi U. Korenberg JR, Klima ES (2001). Willams syndrome: An exploration of neurocognitive and genetic features. Clin Neurosci Res. 2001;1:217–229.
- 24. Hocking DR, Bradshaw JL, Rinehart NJ. Fronto-parietal and cerebellar contributions to motor dysfunction in Williams syndrome: a review and future directions. Neurosci Biobehav Rev. 2008;32(3):497–507. pmid:18006060
- 25. Mervis CB, Robinson BF, Bertrand J, Morris CA, Klein-Tasman BP. Armstrong SC. The Williams syndrome cognitive profile. Brain Cogn. 2000;44(3):604–28. pmid:11104544
- 26. Menghini D, Addona F, Costanzo F, Vicari S. Executive functions in individuals with Williams syndrome. J Intellect Disabil Res. 2010;54(5):418–32. pmid:20537048
- 27. Rhodes SM, Riby DM, Park J, Fraser E, Campbell LE. Executive neuropsychological functioning in individuals with Williams syndrome. Neuropsychologia. 2010;48(5):1216–26. pmid:20026085
- 28. Rhodes SM, Riby DM, Fraser E. Campbell LE. The extent of working memory deficits associated with Williams syndrome: Exploration of verbal and spatial domains and executively controlled processes. Brain Cogn. 2011 Nov;77(2):208–14. pmid:21889249
- 29. Vicari S, Bellucci S, Carlesimo GA. Evidence from two genetic syndromes for the independence of spatial and visual working memory. Dev Med Child Neurol. 2006;48(2):126–31. pmid:16417668
- 30. Atkinson J, Braddick O, Anker S, Curran W, Andrew R, Wattam,-Bell J, et al. (2003). Neurological models of visuospatial cognition in children with Williams syndrome: Measures of dorsal-stream and frontal function. Dev Neuropsychol. 2003;23(1–2):139–72. pmid:12730023
- 31. Greer J, Riby DM, Hamilton C, Riby LM. Attentional lapse and inhibition control in adults with Williams Syndrome. Res Dev Disabil. 2013;34(11):4170–7. pmid:24076981
- 32. Porter MA, Coltheart M, Langdon R. The neuropsychological basis of hypersociability in Williams and Downs syndrome. Neuropsychologia. 2007;45(12):2839–49. pmid:17597166
- 33. Hocking DR, Thomas D, Menant JC, Porter MA, Smith S, Lord SR, et al. The interplay between executive control and motor functioning in Williams syndrome. Dev Sci. 2013;16(3):428–42. pmid:23587040
- 34. Hocking DR, Menant JC, Kirk HE, Lord S, Porter MA. Gait profiles as indicators of domain-specific impairments in executive control across neurodevelopmental disorders. Res Dev Disabil. 2014;35(1):203–14. pmid:24176260
- 35. Costanzo F, Varuzza C, Menghini D, Addona F, Gianesini T, Vicari S. Executive functions in intellectual disabilities: A comparison between Williams syndrome and Down syndrome. Res Dev Disabil. 2013;34(5):1770–80. pmid:23501586
- 36. Carney DPJ, Brown JH, Henry LA. Executive function in Williams and Down syndrome. Res Dev Disabil. 2013;34(1):46–55. pmid:22940158
- 37. Friedman NP, Miyake A, Corley RP, Young SE, DeFries JD, Hewitt JK. Not all executive functions are related to intelligence. Psychol Sci. 2006;17(2):172–9. pmid:16466426
- 38. Maehler C, Schuchardt K. Working memory functioning in children with learning disabilities: Does intelligence make a difference? J Intellect Disabil Res. 2009;53(1):3–10. pmid:19093981
- 39.
Emslie H, Wilson FC, Burden V, Nimmo-Smith I, Wilson BA. Behavioral Assessment of the Dysexecutive Syndrome for Children (BADS-C). Edmunds: Thames Valley Test Company, Bury St; 2003.
- 40.
Gioia GA, Isquith P, Guy S, Kenworthy L. Behavior Rating Inventory of Executive Function. Odessa, FL: Psychological Assessment Resources, Inc; 2000.
- 41. Mervis CB, John AE. Cognitive and behavioral characteristics of children with Williams syndrome: implications for intervention approaches. Am J Med Genet C Semin Med Genet. 2010;154C(2):229–48. pmid:20425784
- 42. Gioia GA, Isquith PK, Kenworthy L, Barton RM. Profiles of everyday executive function in acquired and developmental disorders. Child Neuropsychol. 2002;8(2):121–37. pmid:12638065
- 43. Kenworthy LE, Black DO, Wallace GL, Ahluvalia T, Wagner AE, Sirian LM. Disorganization: the forgotten executive dysfunction in high-functioning autism (HFA) spectrum disorders. Dev Neuropsychol. 2005;28(3):809–27. pmid:16266250
- 44. Rosenthal M, Wallace GL, Lawson R, Wills MC, Dixon E, Yerys BE, Kenworthy L. Impairments in real-world executive function increase from childhood to adolescence in autism spectrum disorders. Neuropsychology. 2013;27(1):13–8. pmid:23356593
- 45. Waid-Ebbs JK, Wen PS, Heaton SC, Donovan NJ, Velozo C. The item level psychometrics of the behaviour rating inventory of executive function-adult (BRIEF-A) in a TBI sample. Brain Inj. 2012;26(13–14):1646–57. pmid:22876936
- 46. Blijd-Hoogewys EM, Bezemer ML, van Geert PL. Executive functioning in children with ASD: an analysis of the BRIEF. J Autism Dev Disord. 2014;44(12):3089–100. pmid:24996868
- 47. Gioia GA, Isquith PK. Ecological assessment of executive function in traumatic brain injury. Dev Neuropsychol. 2004;25(1–2):135–58. pmid:14984332
- 48. Lee NR, Fidler DJ, Blakeley-Smith A, Daunhauer L, Robinson C, Hepburn SL. Caregiver report of executive functioning in a population-based sample of young children with Down syndrome. Am J Intellect Dev Disabil. 2011;116(4):290–304. pmid:21740257
- 49. Daunhauer LA, Fidler DJ, Hahn L, Will E, Lee NR, Hepburn S. Profiles of everyday executive functioning in young children with down syndrome. Am J Intellect Dev Disabil. 2014;119(4):303–18. pmid:25007296
- 50. Porter MA, Dobson-Stone C, Kwok JB, Schofield PR, Beckett W, Tassabehji M. A role for transcription factor GTF2IRD2 in executive function in Williams-Beuren syndrome. PLoS One. 2012;7(10):e47457 pmid:23118870
- 51.
Woodcock RW, McGrew KS, Mather N. Woodcock-Johnson III Tests of Cognitive Abilities. Itasca, IL: Riverside Publishing; 2001.
- 52.
Roth RM, Isquith PK, Gioia GA. Behaviour Rating Inventory of Executive Function-Adult Version. Lutz, FL: Psychological Assessment Resources, Inc; 2005.
- 53. Espy KA, Bull R, Martin J, Stroup W. Measuring the development of executive control with the shape school. Psychol Assess. 2006;18(4):373–81. pmid:17154758
- 54.
Sparrow SS, Cicchetti DV, Balla DA. Vineland Adaptive Behavior Scales, Second Edition: Survey Forms Manual. Circle Pine, MN: AGS Publishing; 2005.
- 55.
Mather N, Woodcock RW. Examiners Manual. Woodcock-Johnson III Tests of Cognitive Abilities. Itasca, IL: Riverside Publishing; 2001.
- 56. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20:37–46.
- 57. Rothman KJ. No adjustments are needed for multiple comparisons. Epidemiology. 1990;1(1):43–46. pmid:2081237
- 58.
Cohen J. Statistical power analysis for the behavioral sciences (2nd ed.). New Jersey: Lawrence Erlbaum; 1988.
- 59. McAuley T, Chen S, Goos L, Schachar R, Crosbie J. Is the behavior rating inventory of executive function more strongly associated with measures of impairment or executive function? J Int Neuropsychol Soc. 2010;16(3):495–505. pmid:20188014
- 60. Bari A., & Robbins T.W. (2013). Inhibition and impulsivity: Behavioral and neural basis of response control. Prog Neurobiol. 2013;108:44–79. pmid:23856628
- 61. Friedman N.P., & Miyake A. (2004). The relations among inhibition and interference control functions: a latent-variable analysis. J Exp Psychol Gen. 2004 Mar;133(1):101–35. pmid:14979754
- 62. Porter MA, Dodd H, Cairns D. Psychopathological and behavior impairments in Williams-Beuren syndrome: the influence of gender, chronological age, and cognition. Child Neuropsychol. 2009;15(4):359–74. pmid:19125360
- 63. Ardila A, Galeano LM, Rosselli M. Toward a model of neuropsychological activity. Neuropsychol Rev. 1998;8(4):171–90. pmid:9951709
- 64. Hongwanishkul D, Happaney KR, Lee WS, Zelazo PD. Assessment of hot and cool executive function in young children: age-related changes and individual differences. Dev Neuropsychol. 2005;28(2):617–44. pmid:16144430
- 65. Martens MA, Wilson SJ, Reutens DC. Research review: Williams syndrome: A critical review of the cognitive, behavioral, and neuroanatomical phenotype. J Child Psychol Psychiatry. 2008;49(6):576–608. pmid:18489677
- 66. Porter MA, Coltheart M. Cognitive heterogeneity in Williams syndrome. Dev Neuropsychol. 2005;27(2):275–306. pmid:15753050
- 67. Einfeld SL, Tonge BJ. The Developmental Behaviour Checklist: The development and validation of an instrument to assess behavioural and emotional disturbance in children and adolescents with mental retardation. J Autism Dev Disord. 1995 Apr;25(2):81–104. pmid:7559289