Introduction

An internationally agreed upon and validated patient reported outcome measure (PROM) in which the outcome of genetic counselling is evaluated by counselees is needed to measure the outcomes of genetic counselling and compare results between countries [1]. Such a PROM should be applicable to individuals receiving counselling for a range of genetic conditions and useful for assessment at multiple clinical time points, e.g. before and after genetic counselling.

In a systematic review, Payne et al. (2008) explored existing measures that evaluate the outcome of clinical genetics services and the key domains these measures capture [2]. They identified 37 non-genetics-specific and 30 genetics-specific outcome measures, but did not find any outcome measure that encompassed all the potential benefits patients may experience when using clinical genetics services. Payne et al. did identify a variety of different domains that did apply, including anxiety and depression, coping, decision-making, distress, family environment, health status, knowledge, mood, perception of risk, perceived personal control, psychological impact, quality of life, satisfaction and expectations, self-esteem, spiritual well-being and worry. One construct that covers many of these domains is “empowerment”, which has been defined as a combination of cognitive, decisional, and behavioural control; emotional regulation; and hope [3]. Empowerment has been identified as a key patient outcome goal of genetic counselling and may therefore be a useful overarching construct that represents many PROMs in clinical genetics services [4].

To measure empowerment, McAllister et al. operationalized the Genetic Counselling Outcome Scale (GCOS) [3]. The GCOS is a 24-item PROM developed from an earlier 84-item version based on qualitative data, items from the Perceived Personal Control questionnaire, and the emotional representations subscale of the revised illness perceptions questionnaire adapted for use in genetic conditions [3]. The GCOS has a seven-factor structure that exists under a single higher order construct “empowerment” [3]. Validation of the GCOS showed good internal consistency, test-retest reliability, responsiveness and some evidence of construct validity [3]. However, to date, the psychometric properties of the GCOS have only been investigated in one study, which had some limitations. In particular, the exploratory factor analysis was performed on the 84-item version rather than the final 24-item GCOS, the results were not confirmed in another sample, and the sample consisted of support-group members instead of counselees [3]. Nor was the responsiveness of the GCOS compared to that of other PROMs. Thus there is a need for a validation study based on counselees using the 24-item version of the questionnaire to further validate the GCOS for use in clinical practice, research, quality improvement cycles and policy.

The aim of this study is therefore to evaluate the psychometric properties of the GCOS and to contribute in the international validation process of the GCOS. To do this, we first examined the construct validity of the GCOS using exploratory factor analysis (EFA), then confirmed the factor structure of our EFA with a confirmatory factory analysis (CFA). We then evaluated the reliability of the GCOS in terms of test-retest reliability and internal consistency. To explore the GCOS’s convergent validity, we explored the association between the GCOS and other PROMs. Based on the findings of McAllister et al. [3], we expected the GCOS to be positively correlated with PROMs that measure perceived personal control, mastery and positive affect, and to be negatively correlated with PROMs that measure negative affect and anxiety. Finally, to measure the responsiveness of the GCOS, we measured the change in scores of the GCOS and other PROMs before and after the first genetic counselling session.

Materials and methods

Participants

Participants were included from the Department of Genetics of the University Medical Center Groningen (UMCG) and the University Medical Center Utrecht (UMCU). Participants were referred for genetic counselling by their general practitioner, medical specialist or midwife. All counselees were eligible for inclusion if they spoke sufficient Dutch to complete the questionnaires. If children were referred (<16 years of age), their parents were considered as the counselees, and one of the parents was asked to complete the questionnaires from his or her perspective. Referrals included affected counselees, unaffected counselees and parents of referred children. Genetic counselling was provided by four genetic counselling teams specialized in different type of diseases: neuro-genetics (most referrals consisted of movement disorders, neurodegenerative diseases and muscular diseases), onco-genetics (most referrals consisted of breast cancer, ovarian cancer and colorectal cancer) and cardio-genetics (most referrals consisted of cardiomyopathies and cardiac arrhythmias). Referrals to the fourth team, which specializes in other diseases, included intellectual disabilities, congenital syndromes, prenatal pathology and hereditary diseases not covered by the other teams. Ethical approval for this study was granted by the Medical Ethical Review Committee of the University Medical Center Groningen (M13.139274).

Study design and procedure

This research has a pre-post observational study design. Participants were included from September 2014 until February 2016. All counselees received a starting package sent to their home address that included an information letter about the study, an invitation letter for a first consultation at the hospital, an informed consent form, and the first questionnaire (T0). Participants who gave informed consent received a second questionnaire (T1) in the week following their first consultation. During this first consultation the (diagnostic) questions were investigated, expectations of the counselees were discussed and the procedure was clarified. Information from the counselee was collected, to get to know more about the (possible) genetic disease involved (in the counselee, the family or the child). If necessary, a physical examination was performed, photographs were taken and/or blood samples were collected. Sometimes counselees needed additional examinations, like X-rays or a referral to another medical specialist. In most cases two counselling sessions were sufficient to answer the question(s). Sometimes the last session was performed by telephone or web-consultation and in some cases no second session was needed. All counselees received a concluding letter afterwards. A third questionnaire (T2) was sent a week after T1. T2 was only sent to the first half of the participants because it was only used to measure the test-retest reliability, and we stopped offering T2 once our sample size had sufficient power. The results of genetic counselling were often communicated during a second consultation, which took place after T2. The time between invitation letter and intake was around two weeks. The time between intake and result disclosure was a few weeks for carrier testing and around three to six months for index patients in whom genetic testing was performed.

Translation process of the GCOS

The Dutch version of the GCOS is a translation of the 24-item UK version of the GCOS [3]. The UK version was translated from English to Dutch by two different Dutch first-language translators with an expertise in clinical genetics (IMvL) and health psychology (AVR). Any discrepancies between the two translations were resolved by discussion. The resulting Dutch version of the GCOS-24 was back-translated into English by an English first-language translator naive to the outcome measurement. Adaptations were made if necessary. The original English and Dutch versions of the GCOS-24 are shown in Supplementary information 1 and 2, respectively.

Measurement instruments

To evaluate the psychometric properties, we compared the GCOS with other PROMs that assess various aspects of empowerment. All these measurement instruments are described more extensively below.

Empowerment was measured with the Dutch translation of the GCOS [3]. This instrument consists of 24 items on a 7-point scale, where items 4, 5, 10, 11, 12, 13, 17, 18, 21 and 22 need to be reversed to calculate a total score. The total score (empowerment) is calculated by adding up all the items. It ranges from 24 to 168, with a higher score indicating more empowerment.

Perceived personal control was measured with the validated Dutch version of the Perceived Personal Control questionnaire (PPC) [5]. The PPC consists of nine items on a 3-point scale, and a total score is calculated by summing the item scores and dividing by the total number of items (range 0–2). Higher PPC scores indicate more control. The internal consistency for our study sample was Cronbach’s α 0.81–0.82 for T0 and T1.

Anxiety was measured with the short form of the Spielberger State-Trait Anxiety Inventory (STAI) [6]. The STAI consists of six items on a 4-point scale, and the total score ranges from 6 to 24, with higher scores indicating more symptoms of anxiety. The internal consistency was Cronbach’s α 0.87 for both T0 and T1.

Positive and negative affect was measured with the Positive And Negative Affect Schedule (PANAS) [7]. The PANAS consists of two subscales (positive and negative affect), with each subscale consisting of 10 items on a 5-point scale (range 10–50), and a higher score indicates more positive or negative affect. The internal consistency was Cronbach’s α 0.88–0.90 for T0 and T1 for the positive affect subscale and Cronbach’s α 0.90–0.91 for the negative affect subscale.

Mastery was measured with the Pearlin Mastery Scale [8]. Mastery refers to the degree to which people perceive they can control factors that influence their life situation and has been found to be important for quality of life and well-being. The Pearlin Mastery scale consists of seven items on a 5-point scale, and the total score ranges from 7 to 35, with higher scores indicating a higher feeling of self-mastery. The internal consistency was Cronbach’s α 0.79–0.80 for T0 and T1.

Statistical analysis

Respondents with missing values on the GCOS at T0 were removed from the analyses. To check for sample selection bias, we compared the socio-demographic and clinical variables of included versus excluded participants using a Chi-square test for categorical variables and a T-test for continuous variables. An EFA was conducted on the outcomes of the GCOS of the T0 Utrecht subset with a factor analysis based upon maximum likelihood and oblique rotation (as factors are expected to correlate with each other). The number of relevant factors was determined by scree plot, parallel analysis, and Velicer’s MAP test [9]. Based on the EFA, it was possible to distinguish the subscales of the GCOS. To check the reproducibility of the factor structure we found in the T0 Utrecht subset, we performed an additional factor analysis on the outcomes of the GCOS of the T0 Groningen subset and the entire T1 dataset.

We then carried out a confirmatory factor analysis (CFA) using the maximum likelihood method in AMOS 23 [10]. Confirmation of the factor structure of our EFA was evaluated by goodness-of-fit indices using the chi-squared to degrees of freedom ratio (χ2/df), the comparative fit index (CFI), the Tucker–Lewis index (TLI) and the root mean square error of approximation (RMSEA). χ2/df values close to or <2 indicate good fit of the model and values <5 an acceptable fit [11]. CFI and TLI values of ≥ 0.90 and ≥ 0.95 indicate acceptable and good model fit, respectively [11]. RMSEA values ≤ 0.06 indicate a good model fit and values in the range of 0.06–0.08 indicate an acceptable model fit [11]. If a given model had a non-acceptable fit, we made modifications to the model to reach an acceptable fit on all (or most) goodness-of-fit indices. These modifications were based on the largest modification indices, which show how much the model fit improves by including additional paths between errors or residuals in the model.

We next examined inter-correlations between the GCOS subscales and their relation with the main scale at T0. Correlation coefficients below 0.3 are considered small, those between 0.3 and 0.5 moderate, and those above 0.5 strong [12]. Internal consistency was measured with Cronbach’s α. Test-retest reliability was based on the Intraclass correlation coefficient (ICC) statistics (two-way random with absolute agreement) between T1 and T2. An accepted guideline for a good Cronbach’s α is that it should be between 0.70 and 0.90 [13]. For ICC, a value ≥ 0.70 is considered good [13]. Convergent validity was measured with Pearson correlation coefficients between the outcomes of the GCOS and other PROMs on T0. Correlation coefficients below 0.3 are considered small, those between 0.3 and 0.5 moderate, and those above 0.5 strong [12]. Responsiveness was measured by determining the mean change in the GCOS and other PROMs between T0 and T1 using a paired samples t-test. Both statistical significance (p < 0.05) and effect sizes (Cohen’s d) were used as indicators of change. An effect size of 0.2 is considered small, 0.5 medium and 0.8 large [13]. Statistical analyses were carried out using IBM SPSS Statistics 23.

Results

Sample

Of the 5300 eligible counselees, 2502 completed T0 (response rate: 47%). Of these 2502 participants, 2194 completed all items of the GCOS on T0 (88%) and were included in the analyses. Table 1 shows the characteristics of the participants for the total sample as well as stratified by reason for referral. T1 was completed by 1740 participants (79%). T2 was completed by 1021 participants, the number required to measure the test-retest reliability of the GCOS. Included participants were then compared with the total sample (including counselees with missing GCOS values on T0) to check if sample corrections were necessary. There were no significant differences between these groups on all socio-demographical and clinical variables.

Table 1 Characteristics of participants at T0 for the total sample and stratified by reason for referral
Full size table

Exploratory factor analysis

EFA was performed on the 24 items of the GCOS. Based on different factor extraction procedures (MAP test, parallel analysis, scree plot), and after comparing different factor structures, a six-factor structure was considered the best choice for the GCOS. In all the factor structures encountered, items 13, 15, 22 and 24 were removed because they had low factor loadings (<0.3) and low inter-item correlations (<0.3) with the other items. Although items 6 and 7 were initially extracted as a seventh factor, we ultimately removed these items because of very low internal consistency (Cronbach’s α = 0.50) and because they produced negative variances in the confirmatory factor analysis.

Table 2 shows the factor loadings of the remaining 18 items on the Utrecht subset at T0. Eigenvalues of the unrotated factors, with corresponding percentages of explained variance, are shown at the bottom of the table. The total amount of variance explained by the six-factor solution is 64%. Based on the EFA, six subscales are suggested: uncertainty about heredity, hope, negative emotions, knowledge about the condition, knowledge about genetic services and uncertainty about the treatment. A main scale, empowerment, can be calculated as a sum of all the items. Figure 1 illustrates the relation between the main scale, the subscales, and the items of the GCOS-18.

Table 2 Item-factor loadings of the GCOS-18 (T0 Utrecht)
Full size table
Fig. 1
figure 1

Main scale, subscales and items of the GCOS-18

Full size image

To check the reproducibility of our six-factor structure derived from the Utrecht T0 subset, we performed an additional factor analysis on the Groningen T0 subset. The Groningen T0 subset showed the same six-factor structure as the Utrecht T0 subset. We then performed an additional factor analysis on the entire T1 dataset and found the same six-factor structure. Only one item (item 1) belonged to a different subscale in the T1 data (moving from knowledge about genetic services to knowledge about the condition). The item-factor loading of these two additional datasets are presented in Supplementary Information 3–4.

Inter-correlations between the subscales and main scale

Table 3 displays the inter-correlations between the six subscales of the GCOS-18 and their correlations with the main scale (empowerment) at T0. The main scale correlates moderately to strongly with all the subscales. In general, the subscales of the GCOS-18 correlate much stronger with the main scale than with each other, suggesting that the subscales represent different facets of empowerment. Most correlations between the subscales are weak, with four exceptions. The subscale hope correlates moderately with negative emotions and knowledge about the condition. Knowledge about the condition correlates moderately with knowledge about genetic services. Negative emotions correlates moderately with uncertainty about the treatment. This means that these subscales share some overlap with each other.

Table 3 Inter-correlations of the subscales and main scale of the GCOS-18 at T0
Full size table

Confirmatory factor analysis

The goodness-of-fit indices showed mixed results for the six-factor solution: χ2/df = 11.07 (>5 indicating a non-acceptable fit), CFI = 0.86 (<0.90 indicating a non-acceptable fit), TLI = 0.84 (<0.90 indicating a non-acceptable fit) and RMSEA = 0.07 (<0.08 indicating an acceptable fit). After two modifications, the resulting model could be confirmed to have sufficient fit by most indices: χ2/df = 7.47 (> 5 still indicating a non-acceptable fit), CFI = 0.91 (≥ 0.90 indicating an acceptable fit), TLI = 0.90 (≥ 0.90 indicating an acceptable fit) and RMSEA = 0.05 (≤ 0.06 indicating a good fit). Figure 2 shows the final model after two modifications with two additional paths between residuals 2/3 and 2/5. Although the model reached a goodness-of-fit on all indices after seven modifications, this model seems less useful because such a large number of modifications makes it less reproducible for other datasets. Therefore the model with only two modifications was ultimately selected in the CFA. Supplementary Information 5 shows the improvement of the goodness-of-fit values after each modification.

Fig. 2
figure 2

Six-factor model of the GCOS-18 confirmed sufficiently by most fit indices using confirmatory factor analysis

Full size image

Reliability

The internal consistency of the GCOS-18 was satisfactory for the main scale empowerment (Cronbach’s α = 0.77) and the subscales hope (Cronbach’s α = 0.74) and knowledge about the condition (Cronbach’s α = 0.75). It was modest for the subscales uncertainty about heredity (Cronbach’s α = 0.67), knowledge about genetic services (Cronbach’s α = 0.66), and negative emotions (Cronbach’s α = 0.66). It was low for the subscale uncertainty about the treatment (Cronbach’s α = 0.59), indicating that this subscale should be interpreted with caution.

Comparing the data from the 923 participants completing the GCOS at T1 and T2, the test-retest reliability was excellent for the main scale empowerment (ICC = 0.92); good for the subscales hope (ICC = 0.89), negative emotions (ICC = 0.89), knowledge about the condition (ICC = 0.82) and uncertainty about heredity (ICC = 0.80); and satisfactory for uncertainty about the treatment (ICC = 0.79) and knowledge about genetic services (ICC = 0.75). These results suggest that the scores on the GCOS main scale and subscales remain sufficiently stable within counselees when measured twice within a short time-period.

Convergent validity

Table 4 shows the correlations between the scores of the GCOS-18 and other PROMs. Convergent validity was confirmed for the main scale of the GCOS-18. As we hypothesized, the GCOS-18 main scale showed moderate correlations with perceived personal control (PPC), positive emotions (PANAS), and mastery (Pearlin Mastery Scale). As expected, the main scale of the GCOS-18 had a moderate correlation with negative emotions (PANAS) and a strong negative correlation with state anxiety (STAI). Next, we studied the relation between the subscales of the GCOS-18 and other PROMs. Some subscales had many moderate correlations with other PROMs, e.g. hope, negative emotions and uncertainty about the treatment. Other subscales had low correlations with other PROMs, e.g. uncertainty about heredity and knowledge about genetic services. These results show that all the PROMs are related to the main scale of the GCOS, while the subscales of the GCOS correlate differently with the other PROMs, again suggesting that these subscales represent different facets of empowerment.

Table 4 Convergent validity and responsiveness of the GCOS-18
Full size table

Responsiveness

Table 4 also shows participants’ scores on the GCOS-18 and other PROMs before (T0) and after (T1) their first genetic counselling session. There was a significant increase in the main scale of the GCOS (empowerment) after the first genetic counselling session. The effect size of 0.30 was small but comparable to that of the other PROMs. There was a significant increase in perceived personal control and mastery after the first genetic counselling session, accompanied by significant decreases in state anxiety and negative emotions. There was also a significant decrease in positive emotions, which was unexpected. The effect sizes of all these changes were small. Regarding the GCOS subscales, there was a significant improvement of all the subscales after the first genetic counselling session, with the subscale hope improving least. The effect sizes of all these changes were, however, also small. These results show that the responsiveness of the GCOS-18 is comparable to that of other PROMs, indicating that the GCOS-18 is capable of measuring the influence of genetic counselling on counselees.

Discussion

In this study we examined the psychometric properties of the GCOS for a Dutch study sample to further validate it for future use in research and clinical practice. The main scale of the GCOS, which is indicative of empowerment, proved to be valid and reliable, and the test’s responsiveness was comparable to that of other validated PROMs used in the evaluation of genetic counselling. One important difference compared to the original GCOS is the reduced number of items: 18 instead of 24. Although a seven-factor structure as found by McAllister et al. [3] could be replicated, we propose a six-factor structure for the GCOS as the seventh factor had very low internal consistency and could not be reproduced in a confirmatory factor analysis. The responsiveness of the GCOS-24 from McAllister et al. [3] had a medium-large effect size (d = 0.70). The effect size of the GCOS-18 in our study was small (d = 0.30). As the other PROMs in our sample had small effect sizes as well, it seems that the differences between these effect sizes could be explained mostly by differences in the study sample. Together, our results suggest that this Dutch version of the GCOS is a suitable questionnaire to measure patient outcomes in genetic counselling with empowerment as an outcome measure.

In addition to the main empowerment scale, we could distinguish six subscales: uncertainty about heredity, hope, negative emotions, knowledge about the condition, knowledge about genetic services and uncertainty about the treatment. Although the definitions of the subscales were based on the item composition of the six factors extracted from the exploratory factor analysis, it seems possible to retrieve the theoretical constructs of empowerment in the subscale items. Empowerment has been defined as a combination of cognitive, decisional and behavioural control, emotional regulation and hope [3]. Cognitive, decisional and behavioural control could be regarded as components of perceived personal control, and perceived personal control was most related to the knowledge subscales of the GCOS. It might be expected that emotional regulation would be most strongly associated with the negative emotions and uncertainty subscales. Hope is represented by the hope subscale.

Interestingly, the subscales of the GCOS showed moderate to strong correlations with the main scale (empowerment), but only small correlations with each other for the most part. Furthermore, the subscales of the GCOS showed different patterns of correlations with other validated PROMs. These findings suggest that the subscales represent different facets of empowerment. Thus it seems useful to measure these facets of empowerment as well, because these subscales provide specific and unique information about empowerment that would be missed if only considering the main scale. These subscales could help to further tailor subsequent genetic counselling.

Recently, Costal Tirado et al. [14] also mentioned different subscales of the GCOS. These subscales were based on the original version of the GCOS. Three subscales were similar (hope, negative emotions, knowledge about genetic services) but sometimes named differently (hope, emotional regulation, referral clarity). Other subscales consisted of items that belong to several subscales in this study (support, family impact, powerlessness). Also a subscale was mentioned that was similar (adaptation), but excluded in our study, because of insufficient psychometric properties. Other validation studies are needed to understand which subscales could be considered as most suitable for the GCOS. It would be interesting to know if in other countries where the psychometric properties of the GCOS-24 are studied, the same subscales can be distinguished as in the Dutch version of the GCOS. In genetic counselling two aspects are important: provision of relevant information and psychological support. It might be expected that psychological support could lead to a reduction in the uncertainty and negative emotions subscales of the GCOS, especially for counselees who experience strong psychological distress. In turn, the information provision task of a genetic counsellor could be expected to improve the knowledge subscales of the GCOS, especially for counselees who initially know less about genetics. Outcomes of the subscales allow us to look at the influence of genetic counselling on empowerment in a more detailed way.

Strengths of our study include the very large, diverse, and representative Dutch study sample, who were being counselled for a broad range of genetic conditions (heart diseases, neurological disorders, cancer, congenital syndromes, intellectual disability and prenatal pathology), and the different types of counselees (e.g. affected/unaffected) coming from different areas of the country. Furthermore, we used a range of statistical approaches to examine the psychometric properties of this questionnaire. The six-factor structure of the GCOS, for example, was independently demonstrated in two different datasets (Groningen and Utrecht) and in two different measurements (T0 and T1), and replicated in a confirmatory factor analysis, which provides evidence for the reproducibility of our findings.

Our study also had some limitations. To reach a solid factor structure, six items were removed, making the questionnaire less comparable to the psychometric properties of the original questionnaire. Although convergent validity has been measured with a sufficient and diverse number of PROMs, there were no questionnaires included to examine divergent validity (i.e. questionnaires that were expected not to be related to the GCOS). Of the 2502 participants, 308 had to be excluded because they did not complete all the items on the GCOS. A high rate of unanswered items has also been mentioned in other studies using the GCOS [14, 15]. One explanation for this could be the fact that counselees are asked to choose a middle response (neither agree or disagree) if no answer is applicable, an instruction that seems to be missed by many counselees.

Future research could clarify the responsiveness on empowerment after the second genetic counselling session, which is when the genetic risk assessment and its implications are often discussed. The change in all outcome measures for the total group was small after the first consultation, which is explainable by the fact that this first session is usually only exploratory and includes counselees who do not require further genetic testing. Another topic for future research is how the influence of genetic counselling is experienced by different subgroups. This could provide more information about which counselees experience genetic counselling as (most) helpful and which subgroups profit less, as well as what could be improved in the counselling process for these subgroups. A future goal is to use the GCOS on an individual level instead of group level. When reference groups are developed, it becomes possible to better understand and compare GCOS results, including those derived at an individual level.