Background

Generalisability (external validity) is the extent to which the results of a study can be applied to other populations. The many threats to the external validity of a study's results include choice of sampling frame, representativeness of the initial sample, and attrition. These issues were discussed in a previous paper [1], and reporting methods were proposed that would enable the reader to assess - at least qualitatively - the generalisability of results from a cohort or longitudinal study. These reporting methods have since been taken further with the publication of the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) initiative [2].

A common method of assessing generalisability is to compare demographics, health characteristics, and health service variables between a study sample and population of interest at baseline. Over time, this comparison should be repeated to see if biases are changing. This process relies on data from people who enrol and remain in the cohort, enrol and later drop out (using data before withdrawal), were invited but never participated, and those in the population of interest (i.e., all people who might have been selected for inclusion in the study). Repeated assessments of generalisability are particularly appropriate in longitudinal cohort studies where drop out is potentially not a random phenomenon (e.g., related to characteristics of people who cease to participate).

Among cohorts of older people, drop out is frequently due to death. This potential source of bias is different from biases that lead to other types of attrition among participants who are still alive.

Relative survival is the ratio of survival that is observed in the study sample in comparison to that of the population from which it was drawn [3]. This method, which was originally developed to measure survival of cancer patients, has not been used previously to assess bias due to deaths in cohort studies. The main purpose of this paper is to explain and illustrate relative survival as a tool for assessing generalisability of results from a cohort of older people among whom death is a potential threat to generalisability.

We illustrate the method using data from women born in 1921-26 who first participated in the Australian Longitudinal Study on Women's Health (ALSWH) in 1996. We also consider possible reasons for differences in relative survival using data from the ALSWH, data collected from the reference population in the national five yearly census, and periodic national health surveys.

Materials and methods

Participants

The ALSWH is a longitudinal study of factors affecting the health and well-being of three national cohorts of women who were born in 1973-78, 1946-51, and 1921-26. The women were selected randomly from the national Medicare health insurance database (which includes all citizens and permanent residents of Australia), with intentional over-sampling of women living in rural and remote areas. In 1996, more than 40 000 women responded to the initial survey; they were reasonably representative of the general population of Australian women in each age group, although compared with data from the 1996 Australian Census there was over-representation of women who were born in Australia, employed and had a university education [4]. More details about the study can be found at http://www.alswh.org.au. Ethical clearance for the study was obtained from the Universities of Newcastle and Queensland.

This paper focuses on the 12 432 women in the 1921-26 cohort who participated in the baseline survey in 1996. Although these women had a nominal age range of 70 to 74 years when the sample was selected, 5% of women were aged 75 years. Due to the small number of participants from the Northern Territory this jurisdiction is not included in the State/Territory comparisons.

Mortality data

Using personal identifying information provided by the participants, vital status was ascertained by probabilistic linkage to the National Death Index (NDI) for all participants from baseline (1996) to 31 October 2008 [5]. The expected mortality of the study population was ascertained using annual life tables produced by the Australian Bureau of Statistics (ABS) for each State and Territory of Australia [6].

National Health Survey

The Australian National Health Survey (NHS) is conducted periodically by trained interviewers from the ABS. In addition to demographic information, the survey provides detailed information about the health status of Australians; their use of health services, facilities, and medications; and health-related aspects of their lifestyle. It consists of a representative sample of residents of private and non-private dwellings in all States and Territories, but excludes special dwellings such as hospitals, institutions, and nursing homes.

The 1995 NHS was conducted during the 12-month period of January 1995 to January 1996. It comprised about 23 800 households, representing approximately 57 600 persons [7]. A total of 894 women aged 70 to 74 years (i.e., women born in 1921-25) participated in the 1995 NHS. Using unit record data supplied by the ABS, these women were compared against the ALSWH cohort participants at the first survey for selected characteristics [8].

Relative survival analysis

Relative survival - the ratio of survival observed in the study sample to the survival to that it should have experienced - can be calculated based on the life table of the population from which it was drawn [3]. In this instance, the study sample was those ALSWH participants born in 1921-26 and the reference population was all Australian women of the same age and State or Territory of residence. It was assumed that the expected mortality experienced by the study sample during a particular period would be the same as mortality in the general population of the same sex, age, and State or Territory of residence from which they are drawn. The Ederer II method was used to calculate interval-specific relative survival [9].

Firstly the population (L) at the start of each interval in each birth cohort and the number of deaths (D) and those lost to follow-up (W) during the interval were determined. From this information, both the population at risk (L') and interval-specific survival (P) were estimated by assuming withdrawals and deaths were evenly distributed over the interval via the formulae:

L ' = L − W / 2  and P = 1 − D / L ' .

The cumulative survival (CP) for a particular interval (i) was then obtained by the cumulative product of the interval-specific survival terms where the initial cumulative survival (CP(0)) was equal to one, and:

CP ( i + 1 ) = CP ( i ) ⋅ P ( i ) .

The expected interval-specific and expected cumulative survival (P* and CP*) were calculated similarly, using expected deaths (D*) obtained from appropriate life tables. Finally the interval-specific and cumulative relative survival ratios (R and CR) were calculated as the ratios of the observed and expected interval-specific and cumulative survivals:

R = P / P *  and CR = CP / CP *

The effect of oversampling women in rural and remote areas in the ALSWH was accounted for via the use of sampling weights, wherein individual weighted deaths both observed and expected were summed to derive the survival estimates.

Separate analyses were carried out for each State and Territory of residence, each category of the Accessibility/Remoteness Index of Australia (ARIA) classification [10], and initial age in years. ARIA categorises areas as 'highly accessible', 'accessible', 'moderately accessible', 'remote' and 'very remote' based on the road distance from the closest service centre. Relative survival was calculated using SAS macros created by Paul Dickman [11].

Comparison to NHS

Selected demographic, health behaviour, and health status characteristics of the ALSWH sample were compared to those of participants of the 1995 NHS of the same age in order to explore possible reasons for observed differences in survival. These comparisons were presented as percentages and analysed using the χ2 statistic.

Effects of factors associated with mortality

A proportional hazards model was used in order to assess the effects of initial differences in potential factors associated with mortality between the study sample and the population [12]. The model was of the form:

λ i ( a , s , z i ( a ) ) = λ * i ( a , s ) exp [ β ' z i ( a ) ]

where λi(a, s, zi(a)) is the death intensity at age a and State/Territory (s) for the ith individual with covariates zi(a) and λ*i(a, s) represent the population mortality at age a for an individual of the same sex and State/Territory as the ith individual in the study who is born in the same year as i. The State and Territory specific life tables [6] were used to obtain values for λ*i(a, s).

A multiplicative model was used because in the more widely used additive model it is assumed that, at all times and for all values of covariates, the mortality in the study sample is always either higher or lower than that of the general population. This assumption was not justifiable in this context. The effect of oversampling in rural areas was accounted for by including place of residence as it was defined in the original sample (urban, rural and remote) as a covariate in the model. Factors considered in this analysis were based on the results of a previous study of the survival of this cohort [13]. These factors included were: age, marital status, country of birth, State or Territory of residence, Accessibility/Remoteness Index (ARIA), education, smoking status, physical activity, body mass index and self rated health. The proportional hazards model was conducted using the SAS PHREG procedure; a hazard ratio of less than one indicates better relative survival. All analyses were performed using SAS version 9.1 [14].

Results

Relative survival

There were 3661 deaths (29.4%) amongst the 12 432 women born in 1921-26 who participated in the baseline survey of the ALSWH. Over the 12-year period of 1996 to 2008, the ALSWH sample had a relative cumulative survival 9.5% (95% confidence interval, 8.3% - 10.7%) greater than their peers in the general Australian population matched for age and State or Territory of residence (Table 1). Interval-specific relative survival remained relatively constant over the whole period, varying between 0.3% and 1.2%.

Table 1 Life Table Estimates of Survival of the Australian Longitudinal Study on Women's Health 1921-26 Cohort Relative to Women in the Australian Population Born in the Same Period and Resident in the Same State or Territory.
Full size table

ALSWH participants had significantly better survival than the general population in all jurisdictions, with their relative cumulative survival advantage ranging from 6% in South Australia to 23% in the Australian Capital Territory (Table 2). Their relative cumulative survival was consistently higher than for the general population across all ARIA groups, although in the remote/very remote areas the difference was not statistically significant (Table 2). The relative survival advantage of ALSWH participants increased with initial age: among those assessed in 1996, a 6% advantage among women aged 70 approached 22% among women aged 75 (Table 2).

Table 2 Life Table Estimates of Relative Cumulative Survival Over the Period 1996-2008 Among Participants in the Australian Longitudinal Study on Women's Health 1921-26 Cohort by State or Territory of Residence, Accessibility/Remoteness Index (ARIA), and Age at Baseline (1996).
Full size table

Comparison of ALSWH cohort to NHS sample at baseline

The comparison of the ALSWH 1921-26 cohort at baseline (1996) with the 1995 NHS for selected socio-demographic characteristics and health related behaviours is shown in Table 3. The ALSWH cohort was less likely than their NHS counterparts to be widowed, more likely to be married, and more likely to have a tertiary education. There were more women born in 'Other English speaking background (ESB)'countries in the ALSWH cohort and less born in Europe. Among health related behaviours, ALSWH participants were less likely to be current smokers, report fair or poor self-rated health, and report that their health limited their ability to exercise (walk 100 metres).

Table 3 Comparison of Selected Characteristics Between the Australian Longitudinal Study on Women's Health 1921-26 Cohort and the 1995 National Health Survey.
Full size table

Effects of factors on mortality (relative survival)

The results of the multiplicative relative survival model are shown in Table 4. Relative survival was significantly associated with initial age, country of birth, State or Territory of residence, marital status, body mass index (BMI), smoking status, physical activity, and self rated health. For example, age at baseline was positively associated with better relative survival, implying that - even after adjusting for major risk factors - older members of the cohort were healthier than their younger counterparts in the general population.

Table 4 Multivariate Analysis of Relative Survival of the Australian Longitudinal Study on Women's Health 1921-26 Cohort, 1996-2008.
Full size table

Discussion

In the 12 years (1996-2008) under consideration, the ALSWH 1921-26 cohort had significantly better survival than the general population. This better relative survival was consistent across all jurisdictions for the duration of the study. There was also no indication that the survival of the sample converged to that of the general population over time. This result was unexpected because most sample groups tend to become more like the general population over time (or, even if they differ at baseline, the effect of that initial difference becomes less important). Other longitudinal studies have found the mortality of sampled respondents and non-respondents converging over time [15, 16].

That the ALSWH 1921-26 cohort had significantly better survival was expected, as participants were self selected from an initial sample of randomly selected Australian women. From an initial sample of 39 000 women selected from the Medicare database, 12 432 responded [17]. It would be expected that older women - in particular those with better health ('the healthy volunteer') or who were more interested in their health - would be more likely to participate in such a survey. Similar effects have been observed in other longitudinal studies [15, 16, 18].

Consistent with previous analysis comparing the ALSWH cohort to census data [19], systematic differences were observed between the ALSWH 1921-26 cohort and the participants of the 1995 National Health Survey. A previously published study of survival among the 1921-26 cohort showed that self rated health was a strong predictor of long term survival [13]. Other variables found to be associated with survival in this current study (e.g., marital status, country of birth, smoking, physical activity and BMI) were also associated in the previous study. This study found significant differences between the ALSWH cohort and the NHS participants with respect to several variables, namely: self rated health, smoking status, marital status, and country of birth. Given the magnitude and direction of these differences, these factors could explain a major portion of the observed survival advantage in the ALSWH cohort. For example, if the distribution of self rated health observed in the NHS was applied to the ALSWH sample then the cumulative mortality of the ALSWH cohort as a whole would be increased by 10%, thereby reducing the survival advantage observed in the ALSWH cohort by about a half.

Limitations

An alternative possible reason for the observed difference in survival between the ASLWH sample and NHS participants could be incomplete ascertainment of deaths in the study population. Death information was obtained from both linkage to the NDI as well as notification by family or carers of participants. A previous paper examining linkage of the study population to the NDI in 1998 showed that such a linkage identified 95% of the deaths [5], however this has not been reassessed since that time and it is possible that this capture method has worsened over the last decade. Other studies of the accuracy of the NDI found the false negative rate ranging from 3% to 11% (compared to 5% in the ALSWH study) [20–22]. If the false negative rate is as high as 10%, this could account for about half the difference observed in this study. However, such a high false negative rate is considered unlikely because NDI linkage is supplemented by other information, particularly data that is obtained at the time of the triennial ALSWH surveys. Still, the systematic difference in survival observed over the study period suggests some under ascertainment of deaths may have occurred.

Another limitation to the use of relative survival as a tool for assessing generalisability is the categorisation of available life tables. For Australia, life tables are available by age, sex and State/Territory of residence only; it would be useful if they were available for other factors such as smoking status. Indeed, if other population data on survival were available with stratification by other variables (e.g. from other cohort studies), then the relative survival approach with weighting by strata would be feasible.

Conclusion

This study has shown that relative survival can be a useful and relatively easily obtained measure of generalisability (external validity) in a longitudinal study of the health of a population-based sample, particularly when participants are of advanced years. The advantage of this method is that, through a single measure, it can indicate the degree to which a study sample corresponds to the general population with respect to health status. Along with the other comparisons of the study population to census and survey data, this measure provides information relevant to the reporting requirements of the STROBE statement [2]. In the case of the ALSWH cohort studied here, it seems likely that most of the difference in their survival over that of the general population is attributable to the better health of the sample at baseline.

Implications

It is essential that any future analysis of this cohort considers the results of this investigation, but if and how any adjustments to design must occur will depend on the objectives of future work. If the analysis involves examining the associations of various factors with some outcome, then it may be sufficient to control for the factors that were found to be associated with improved survival. On the other hand, if population estimates are required then it would be necessary to employ some type of weighting scheme involving these factors, as well as the weights that account for the deliberate oversampling in rural areas. For example, if one was estimating the population prevalence of diabetes in older women then it may be necessary to use weights for area of residence and perhaps other factors such as self rated health.