How to do (or not to do) … Designing a discrete choice experiment for application in a low-income country

Abstract

Introduction

A discrete choice experiment (DCE) is a quantitative technique for eliciting individual preferences. It allows researchers to uncover how individuals value selected attributes of a programme, product or service by asking them to state their choice over different hypothetical alternatives. DCEs have been applied to a range of health policy, planning and resource allocation decisions in high-income settings. These include the elicitation of views on diagnosis, treatment and care (Coast et al.2006; King et al.2007; Lancsar et al.2007; Kjaer and Gyrd-Hansen 2008), access to services (Gerard and Lattimer 2005; Gerard et al.2006; Longo et al.2006) and the employment preferences of health personnel (Scott 2001; Ubach et al.2003; Wordsworth et al.2004).

Although DCEs have been frequently applied in health economic research in high-income countries, there are comparatively few examples of DCEs being used elsewhere. Searches of the published literature identified only four papers that had employed a DCE to consider a health issue in a low- and middle-income country (LMICs) (Hanson et al.2005; Baltussen et al.2006; Christofides et al.2006; Baltussen et al.2007) and a further three papers were identified by undertaking selective searches of the internet (Chomitz et al. 1997; Penn-Kekana et al.2005; Baltussen and Niessen 2006). Moreover, we are aware of several DCEs currently underway in LMICs. These include: an examination of public preferences for maternity services in Ghana (Ternent et al.2006) and an estimation of women's demand for barrier methods for HIV prevention in South Africa (Terris-Prestholt et al.2008a, b). In addition, there are ongoing DCEs to elicit the employment preferences of health workers, including of doctors and nurses in Ethiopia (Hanson and Jack 2007), clinical officers in Tanzania (O Maestad and J Riise Kolstad, personal communication) and several cadres in Malawi, South Africa and Tanzania (C Normand, personal communication). In LMICs, DCEs have been more frequently used outside of the health sector, with academic articles reporting preferences in the agriculture, water, transport and tourism sectors (Baidu-Forson et al.1997; Tiwari and Kawakami 2001; Tano et al.2003; Hope and Garrod 2004). Internet searches also returned several discussion papers, reports and other sources of grey literature that had applied the technique in these settings (Carlsson et al.2005; Porras and Hope 2005). Nonetheless, the use of DCEs in LMICs remains relatively recent.

This paper aims to provide an introduction to the DCE methodology for those working on health issues in low-income countries. We describe the technique, outline the stages involved in its application and emphasize the design considerations pertinent to conducting a DCE in a low-income setting. In discussing the design process, we draw on our experiences of conducting DCEs to elicit patient preferences of hospital services in Zambia (Hanson et al.2005) and employment preferences of health workers in Malawi (Mangham and Hanson, 2008).

What is a Discrete Choice Experiment?

Techniques for eliciting preferences have primarily emerged from a desire to understand consumer demand for goods and services where it was not possible to use revealed preference data on the actual choices made by individuals. This would arise for products and services not traded on a market, such as for a new product under development and not yet commercially available. Similarly, where there is no variation in the products available (or services provided), it is not possible to isolate the contribution of each product attribute to the overall utility derived from the product.

DCEs require respondents to state their choice over sets of hypothetical alternatives. Each alternative is described by several characteristics, known as attributes, and responses are used to infer the value placed on each attribute. In comparison to other stated preference techniques that require the individual to rank or rate alternatives, a DCE presents a reasonably straightforward task and one which more closely resembles a real-world decision.

DCEs are used to determine the significance of the attributes that describe the good or service and the extent to which individuals are willing to trade one attribute for another (Drummond et al.2005). Information on the relative importance of the selected attributes can be useful for those involved in policy decisions and setting resource allocation priorities, and may be designed with that in mind (Baltussen and Niessen 2006; Baltussen et al.2007). For example, a DCE on the employment preferences of registered nurses in Malawi was designed in light of the Malawi government's programme to recruit and retain health personnel, which included salary increases to improve motivation and discourage attrition (Palmer 2006). The results of the DCE showed that remuneration had a significant impact on how nurses viewed their employment, and that they were willing to trade-off pay increases to obtain improvements in non-monetary benefits or working conditions (Mangham and Hanson, 2008).

The method can be used to estimate the marginal valuations of attributes or the willingness to pay (WTP) for a unit change in each attribute estimated (Drummond et al.2005). In comparison with standard techniques, which estimate WTP for the good or service as a whole, this more detailed information on WTP by attribute may be useful, though some evidence suggests that the levels of the cost attribute can affect the estimates (Radcliffe 2000; Drummond et al.2005). DCE results can also be analysed by subgroup and it is possible to consider the extent to which individuals’ characteristics impact on the marginal valuations. For instance, a DCE applied to elicit the patient preferences for hospital quality in Zambia estimated WTP for the different attributes for the whole sample and by socio-economic group. They found the highest willingness to pay was for a thorough, rather than superficial, examination and that the WTP for this attribute in the lowest socio-economic group was only about half that in the highest group.

How to design a DCE

Discrete choice analysis involves the construction of an experimental design to study the effects of the attribute levels on the stated preference (or dependent variable). The attributes of an experimental design are variables that have two or more fixed levels.

There are several stages to the design of a DCE, which we outline below. In our discussion the design considerations pertinent to conducting a DCE on a health-related issue in a low-income setting are highlighted. We also use examples from our own experience of undertaking DCEs in sub-Saharan Africa, to elicit patient preferences on the quality of hospital services in Zambia (Hanson et al.2005), and the employment preferences of registered nurses in Malawi (Mangham and Hanson, 2008).

Establishing attributes

The first stage of a DCE involves identifying the attributes relevant to the stated research question and then assigning levels for each of these attributes (Ryan 2001; Hensher et al.2005). Since these attributes and attribute levels describe the hypothetical scenarios under consideration in the DCE, this is a critical aspect of the design. The underlying validity of the study depends, therefore, on the researcher's ability to correctly specify the relevant attributes. Despite the importance of this stage in the design, there is often sparse explanation in the DCE literature of how attributes and levels are established (Coast and Horrocks 2007).

Selecting and defining the attributes requires a good understanding of the target population's perspective and experience (Hall et al.2004; Coast and Horrocks 2007). Researchers working in a different cultural or language setting may find obtaining the necessary depth of understanding a challenge and may want to involve local experts. Policy concerns may also shape the choice of attributes, and it is often advisable to engage local institutions and policy-makers during this preparatory stage (Baltussen and Niessen 2006). For example, in the Malawi study on nurses’ employment preferences, the decision to include an attribute on the provision of government housing was influenced by the Ministry of Health's interest in how the availability and quality of government housing affects the retention of health personnel.

Published and grey literature, such as policy documents and government reports, are a useful starting point for identifying attributes. For example, the literature on the global shortage of health workers (Buchan and Sochalski 2004; Joint Learning Initiative 2004; World Health Organization 2006) provided a background to the Malawi study, while Malawi government policy and programme documents provided detail on the country-specific situation (Buchan and Sochalski 2004; Joint Learning Initiative 2004; World Health Organization 2006; Coast and Horrocks 2007).

It is often necessary, however, to augment secondary sources with primary data to ensure that the DCE is tailored to the study setting. Primary qualitative data were essential for selecting and defining the attributes that registered nurses considered important. Semi-structured interviews were conducted with public sector nurses to obtain information on their current working conditions, preferences, reasons for the shortage of nurses and priorities for reform. The sample included views of nurses based in urban and in rural settings in three geographically distinct districts, working in primary, secondary and tertiary health facilities, and with differing degrees of seniority and experience. Additional interviews were also held with senior officials in the Ministry of Health to obtain their perceptions, policy concerns and remuneration data. In Zambia, qualitative data were collected using focus group discussions on what dimensions of the quality of hospital services were important to individuals when choosing a hospital. In total, 16 separate discussions were held with men and women from two districts, and from different socio-demographic groups: university students, market sellers, residents from high and low density areas (Hanson et al.2005).

In both settings, the discussions were facilitated by a sociologist from a local research institute and were tape recorded, transcribed, translated and supplemented by detailed written notes. Local researchers brought a valuable perspective, knowledge and experience to the design of the DCE. They led the collection of qualitative data, which not only encouraged respondents to be more open, but also allowed them to express themselves in their local languages.

In both studies content analysis was used to obtain possible attributes for the DCE. This involved reading all transcripts and written notes to identify major themes, followed by a more detailed review, during which additional themes and sub-themes emerged. The transcripts were coded manually and relevant extracts recorded in a summary table. Software packages can also assist the coding and retrieval of data, and are particularly useful where the number of interviews is large or for more complex qualitative analysis (Lewando-Hundt et al.1997).

There are no design restrictions on the number of attributes that could be included in a DCE, though in practice most DCEs have contained fewer than 10 to ensure that respondents are able to consider all attributes listed when making their choice (DeShazo and Fermo 2002). The greater the number of attributes, the greater the cognitive difficulty of completing a DCE. With too many attributes, the participants may be encouraged to apply a simple decision rule in which they base their response on a single or subset of attributes. In establishing attributes it is also important to avoid conceptual overlap between two or more of the attributes, known as inter-attribute correlation, since it would prevent the accurate estimation of the main effect of a single attribute on the dependent variable.

In both the Malawi and Zambia studies, six attributes were established that were important to the respondents and were policy relevant. For example, the attributes relating to the quality of hospital services identified in the Zambia study were: (1) the likelihood that the child will receive all the drugs s/he needs at the hospital, (2) the likelihood that the hospital staff will examine the child properly, (3) staff attitudes, (4) cleanliness of the wards and toilets, and (5) the waiting time between arrival at the outpatients department and admission to the ward. In addition a sixth cost attribute was included that covered the costs of the child's examination and treatment. Other issues frequently mentioned by discussants were not included in the DCE because of inter-attribute correlation with the established attributes. They were: the availability of diagnostic services, quality of nursing care, staff dedication and availability of staff (Hanson et al.2005).

In specifying attributes, care should be taken to ensure that definitions are appropriate for the setting and are not ambiguous. For example, the Malawian nurses frequently used the term ‘upgrading’ when referring to obtaining additional professional qualifications. These long-term educational opportunities are distinct from in-service training courses, and in defining the attribute on access to training it was therefore important to use the ‘upgrading’ expression to avoid attribute ambiguity.

Assigning attribute levels

Once the attributes are established, attribute levels need to be assigned. Typically the levels chosen should reflect the range of situations that respondents might expect to experience, and again qualitative data are valuable. Ensuring the levels are realistic and meaningful will increase the precision of parameter estimates (Hall et al.2004).

In the Malawi study, qualitative data were used to determine base levels that reflected the prevailing working conditions for the public sector registered nurses. Additional levels were then established that represented a reasonable improvement from the base. For ease of cognition we sought to establish no more than three levels for each attribute, initially opting for two levels and then adding a third where there was no single base level. For example, variation in the actual provision of government housing meant both ‘no government housing’ and ‘basic government housing’ were possible base levels. Three levels were also used for the pay attribute as we were interested in the preferences over the magnitude of the pay increase. A net monthly pay of K30,000 Malawi kwacha (approximately US$240) was an average prevailing salary for public sector registered nurses, and two higher levels were included, K40,000 ($320) and K50,000 ($400). These higher pay levels were similar to those in the private sector and in line with what the interviewed nurses had indicated they would consider a reasonable salary.

Designing the choice sets

The next stage in the design of a DCE is to generate the hypothetical alternatives and to combine them to create choice sets. A full factorial design can be generated which consists of all possible combinations of the levels of the attributes, and permits estimation of main effects and interactions. A main effect refers to the direct independent effect on the choice variable of the difference in attribute levels (e.g. difference in price). An interaction effect is the effect on the choice variable obtained by varying two or more attribute levels together (e.g. difference in price combined with difference in colour).

In most practical situations it is considered too cost-prohibitive and tedious to have subjects rate all possible combinations in a full factorial design (Kuhfeld 2005). A design with five attributes, each with three levels would, for example, generate 243 possible alternatives (3⁵). Thus, fractional factorial designs are often used to consider a selection of possible alternatives.

In selecting a fractional factorial design, researchers should seek to obtain a design that is both orthogonal and balanced (Huber and Zwerina 1996; Kuhfeld 2005). In orthogonal fractional factorial designs, the parameter estimates in the linear model are uncorrelated, which means that the attributes of the design are statistically independent of each other (Hensher et al.2005; Kuhfeld 2005). A balanced design has each attribute level occurring equally often, and this minimizes the variance in the parameter estimates (Kuhfeld 2005). Fractional factorial designs that are both orthogonal and balanced are known as orthogonal arrays and these can be obtained from design websites such as http://www.research.att.com/~njas/oadir (Burgess and Street 2005). However, orthogonal arrays only exist for certain combinations of attributes and attribute levels (Kuhfeld 2005). For other combinations there will be a trade-off between the degrees of orthogonality and balance. Researchers should select the most efficient design, using a measure known as D-efficiency (Carlsson and Martinsson 2003; Burgess and Street 2005; Kuhfeld 2005; Street et al.2005).

A third property that characterizes efficient choice designs is minimal overlap (Huber and Zwerina 1996; Maddala et al.2003). Each attribute level is only meaningful in comparison to others within the choice set, or in other words no information is obtained on an attribute's value when its levels are the same across all alternatives within a choice set. Researchers should, therefore, seek to minimize the probability that an attribute level repeats itself in each choice set. Finally, Huber and Zwerina (1996) have argued for the importance of utility balance, which refers to balancing the utilities of the alternatives offered in the choice set, though in practice the lack of prior information on the utility of attributes limits the applicability of this criterion (Huber and Zwerina 1996).

The recommended approach to constructing an experimental design continues to evolve (DeShazo and Fermo 2002; Street et al.2005; Adamowicz 2006; Louviere 2006) and ensuring that the generated DCE meets best practice can be a challenge. Consequently, we encourage researchers planning to undertake a DCE to review the latest publications, within and beyond the discipline of health economics. We are also aware that access to the latest information may be an obstacle for those undertaking research in LMICs.

Generating and pre-testing the questionnaire

The created choice sets form the basis for the DCE questionnaire. The number of choice sets presented to each individual will depend on the size of the fractional factorial design and the strategy employed in designing the choice sets (Street et al.2005). Typically DCEs ask respondents to consider up to 18 choice sets, with 18 representing a practical limit of how many comparisons can be completed before boredom sets in (Hanson et al.2005; Christofides et al.2006). This boredom threshold level is likely to vary, and will depend on the number of choice sets, their complexity and the characteristics of the target population.

In both the Zambia and the Malawi studies we applied a pairwise design, such that respondents were asked to consider a choice set with two alternatives and state their preference for each pair. For illustration, an example of a choice set from the Malawi study is shown in Figure 1. A dichotomous choice has frequently been applied in health services research and the statistical information obtained represents the demand conditional on accepting one of the two scenarios.

Figure 1

Open in new tabDownload slide

Example of a choice set from the study of employment preferences of Malawian registered nurses

In the Malawi study we also asked a second question that introduced a ‘choose neither’ option. This allowed respondents to reject both alternatives and provided data to estimate unconditional demand. In many situations the inclusion of a non-choice option will more closely resemble a real world context, since individuals are not required to make a choice, though these non-choice responses do not provide any information on how individuals trade-off the attribute levels of the available alternatives. In our design, the motivation behind asking both which job they considered superior and which one they would choose was to ascertain the extent to which the personal circumstances of respondents were an explicit constraint in their decision-making. For example, we hypothesized that married women would be more constrained in making their choice over place of work.

The internal consistency of responses can be considered by including one or two choice pairs in which one option is superior to the other on all characteristics. In the Zambia study this assumed that people prefer a lower cost, more thorough examination, a shorter waiting time, and better drug availability, cleanliness and staff attitudes. Individuals that fail to choose the superior hospital job may have misunderstood the task or were unable to provide consistent answers because of problems of communication or translation. While it is useful to know the extent to which individuals respond rationally, Lancsar and Louviere (2006) advise against excluding apparently ‘irrational’ choices from the analysis of DCEs as that may cause statistical bias and/or affect statistical efficiency.

The questionnaire should be clearly presented and contain a standard introduction to the DCE with choice set examples. To minimize any bias caused by the order in which the choice sets occur or the attributes are described, it is good practice to produce several versions of the questionnaire in which choice sets and attributes are presented in different orders (Kjaer et al.2006). Pictures, diagrams and symbols may aid comprehension, and are particularly relevant for conducting a DCE in low-income countries where literacy cannot be assumed. Similarly, in some settings the questionnaire will need to be translated into one or more local languages. For example, picture boards and verbal descriptions were used when eliciting preferences on hospital quality in Zambia and interviews were administered in both Bemba and English (Hanson et al.2005). Finally, it is usual to collect data on socio-economic indicators to allow analysis of the impact of individual characteristics on the choices made.

In preparing the questionnaire it is also important to consider how the DCE will be administered. It is possible for the questionnaire to be self-administered, undertaken in examination conditions (Chomitz et al. 1997) or, as we opted, to have trained fieldworkers administer the questionnaire to respondents individually. In many low-income countries, postal or online surveys will not be feasible because of the infrastructure, and given the lower levels of literacy, there are also likely to be practical advantages to having fieldworkers explain what is asked of consenting respondents and work through examples.

Piloting the questionnaire is a key stage in most survey designs and is particularly relevant when working across cultures and several languages. Moreover, pre-testing provides an opportunity to review several elements of the design process, including the selection and definition of attributes and their levels (Hall et al.2004). This is important given the extensive role of the researcher in the design of the DCE. For example, pre-testing in the Malawi study identified the conceptual overlap between the type of hospital and the typical workload, which led to the rewording of the levels relating to place of work (Mangham and Hanson, 2008). Minor modifications were also made to the definition of the attribute levels for typical workload. Other aspects of the design which should be reviewed during pre-testing include the respondent's understanding of the task, their ease of comprehension and whether the number of choice sets can be managed by the target population (Hall et al.2004).

Analysing of DCE data

Once the choice sets and questionnaire design are finalized, the DCE questionnaire can be administered to collect data for subsequent analysis. Our discussion on data analysis is restricted to a few key aspects, since the focus of this paper is on how to design a DCE. Furthermore, the same methods for analysing DCE data apply irrespective of the study context and are well documented elsewhere (Louviere et al.2000; Hensher et al.2005).

Moreover, as respondents are asked to consider multiple choice pairs, it cannot be assumed that the error terms are independent and panel data estimation techniques are required. The estimated parameters represent the marginal utility associated with a change in the attribute level in moving from one alternative to the other.

Discussion

In developing countries the application of DCEs to consider questions of health policy and planning is relatively recent, but appears to be of growing interest (Chomitz et al. 1997; Hanson et al.2005; Penn-Kekana et al.2005; Baltussen et al.2006; Christofides et al.2006; Ternent et al.2006; Mangham and Hanson 2008). Moreover, Baltussen and Niessen (2006) argue that a multi-criteria approach to health priority setting is essential and DCEs, as a technique for undertaking multi-attribute analysis, should be used more routinely to guide resource allocation decisions.

The stages involved in the design of a DCE are the same regardless of study setting: establishing attributes and attribute levels, designing choice sets, and generating and pre-testing the DCE questionnaire. There are, however, some particular challenges in conducting a DCE in a developing country context, and in describing each element of the design we have sought to highlight the additional design considerations. These challenges may relate to working in different cultural or language settings, or surveying populations that have a lower level of literacy or are less accustomed to market research techniques. Many of these challenges will be common across a range of research methodologies and will be familiar to those experienced in undertaking research in LMICs. Nevertheless, we believe that there is merit in considering aspects of the DCE design that may require more attention when the method is to be applied in a low-income setting.

For example, the use of primary data when establishing attributes and attribute levels may be more critical in a low-income setting. The validity of the research findings depends on the analyst's ability to correctly specify the relevant programme, product or service attributes and levels, and this requires a detailed understanding of the target populations’ experience and point of view (Hall et al.2004). Access to relevant information on the policy context or specific health programmes can be a challenge, particularly in developing countries, and the perspectives and concerns of individuals are often poorly articulated. Thus, although secondary literature can be used to identify an initial set of attributes, in low-income settings additional primary research is almost always necessary to ensure that the final set of attributes is appropriate and valid.

Similarly, pre-testing the questionnaire is likely to be even more important in a context where there are cultural and language differences between the researchers and study participants, or where the population surveyed has a lower level of literacy. Pre-testing provides an opportunity to ensure that the information is presented in a comprehensible way, and that the choices are realistic and meaningful. It also provides an opportunity to observe how cognitively demanding the questionnaire is for respondents to complete. Researchers may, for example, need to reduce the number of choice sets reviewed, adjust the number of attributes in each alternative or include pictorial information and verbal descriptions. Such amendments should help to ensure that responses are a better reflection of individual preferences and improve the precision in the parameter estimates.

Given some of these additional challenges of designing a DCE for a low-income country, it is reasonable to ask whether a research methodology that was originally developed to understand demand for consumer products in high-income countries readily translates to questions of public policy and health resource allocation in low-income countries. Our experience, and that from the literature, suggests that participants are able to state their preferences over health service provision and areas for policy reform (Chomitz et al. 1997; Hanson et al.2005; Penn-Kekana et al.2005; Baltussen et al.2006; Christofides et al.2006; Ternent et al.2006; Mangham and Hanson, 2008). The results also suggest that the preferences are reasoned and deliberate. For instance, the internal consistency of responses was high in both the Zambia and Malawi studies, from which we inferred that the vast majority of individuals were making rational choices. Similarly, the theoretical validity of the valuations, which is assessed by determining whether the estimated coefficients are of the anticipated sign, found that the results were consistent with prior expectations.

DCEs that elicit preferences on the provision of health services or strategies for policy reform should be useful for health policy-makers and planners involved in identifying priorities for resource allocation. Although the task of choosing between alternative scenarios is reasonably straightforward to comprehend, some elements of the design are complex. It will be important, therefore, that researchers are able to communicate the research findings and policy implications in a way that can be easily understood.

DCEs, and other stated preference techniques, also have the advantage that they are a means of obtaining preferences on situations that are not traded explicitly in markets, as is often the case with health care, or have public good characteristics, such as a vaccination programme (Pokhrel 2006). Similarly, their ability to be used in situations that are purely hypothetical means that it is possible to elicit preferences over potential policy reform or health system changes prior to their implementation. Moreover, as the design of DCEs is controlled and the attributes are varied systematically, it is straightforward to identify the effect of an attribute on the choice variable (Baltussen and Niessen 2006; Pokhrel 2006).

As the benefits of the DCEs become more widely understood we expect the technique to be increasingly applied to health policy and planning questions in developing countries. We hope this paper provides a useful introduction for those wanting to gain a better understanding of the methodology and the process of designing a DCE.

References