Using Concept Mapping to design an indicator framework for addiction treatment centres

Abstract

Objective. The objective of this study is to determine an indicator framework for addiction treatment centres based on the demands of stakeholders and in alignment with the European Foundation for Quality Management (EFQM) Excellence Model.

Setting. The setting is the Jellinek Centre based in Amsterdam, the Netherlands, which serves as a prototype for an addiction treatment centre.

Method. Concept mapping was used in the construction of the indicator framework. During the 1-day workshop, 16 stakeholders generated, prioritized and sorted 73 items concerning quality and performance. Multidimensional scaling and cluster analysis was applied in constructing a framework consisting of two dimensions and eight clusters.

Results. The horizontal axis of the indicator framework is named ‘Organization’ and has two poles, namely, ‘Processes’ and ‘Results’. The vertical axis is named ‘ Task’ and the poles are named ‘Efficient treatment’ and ‘Prevention programs’. The eight clusters in the two-dimensional framework are arranged in the following, prioritized sequence: ‘Efficient treatment network’, ‘Effective service’, ‘ Target group’, ‘Quality of life’, ‘Efficient service’, ‘Knowledge transfer’, ‘Reducing addiction related problems’, and ‘Prevention programs’. The most important items in the framework are: ‘patients are satisfied with their treatment’, ‘early interventions’, and ‘efficient treatment chain’.

Conclusion. The indicator framework aligns with three clusters of the results criteria of the EFQM Excellence Model. It is based on the stakeholders’ perspectives and is believed to be specific for addiction treatment centres. The study demonstrates that concept mapping is a suitable strategy for generating indicator frameworks.

Today health care organizations have to operate in a constantly changing environment. In the past, government regulations determined the policy, the infrastructure, and the activities of the services, but in recent years the influence of regulatory bodies has been reduced. Financing departments implement cost control programs, patients become critical customers, and the public wants to see results [1]. In order to cope with the demand, health care organizations have to find a way to clarify and specify those demands in order to meet the requirements. In many cases health care organizations use the European Foundation for Quality Management (EFQM) Excellence Model [2] as an aid.

However, working with the EFQM Excellence Model means that the demands of the different stakeholders have to be clarified and transformed into indicators or measures. Moreover, the EFQM Excellence Model does not specify the results that have to be achieved in order to survive and thrive as a health care organization. Given the absence of health care as an area of results in the EFQM Excellence Model, this study aims to develop, by way of concept mapping, an indicator framework for addiction treatment centres that can be aligned to the EFQM Excellence Model.

The EFQM excellence model

The roots of the EFQM Excellence Model lie firmly within Total Quality Management [3], and are related to the American Malcolm Baldridge National Quality Award [4]. During the last few years the EFQM Excellence Model has been widely used within health care, which may be due to its simple structure and the similarities with the Donabedian [5] model of structure, process, and outcome. Since 1992 there have been three revisions of the EFQM Excellence Model that have progressively shifted the emphasis from the classical view of quality management, to the view of excellence and performance management. Details are documented in the EFQM manual [6] and in the EFQM handbook for hospitals [7].

The Jellinek Centre

One health care organization that has used the EFQM Excellence Model extensively is the Jellinek Centre [8]. The Jellinek Centre is a treatment centre responsible for providing a wide range of services for alcohol-dependent, drug-dependent, and addicted clients [9] in the region of Amsterdam in the Netherlands. The organization runs 25 in-patient and outpatient treatment services. The Jellinek Centre treats about 5000 clients annually. With a staff of 500 the Jellinek Centre has the reputation of being a leading edge organization in relation to quality, innovation, and the management of addiction within the Netherlands.

Research problem

In general, addiction treatment centres often have difficulties in defining indicators [10], meaning that determining the progress towards excellence can be problematic, particularly when abstinence from the addictive substance is no longer the overall treatment goal. Moreover, addiction centres operate within a complex setting of stakeholders who often have conflicting demands. Therefore the Jellinek Centre wanted to find a solution to monitoring the progress towards excellence by way of constructing an indicator framework that contained quality, performance, and result indicators based on the specific demands of the stakeholders. To do this the following research questions were formulated:

• What are the demands of the different stakeholders at the Jellinek Centre?

• How can the demands be logically grouped in order to construct a framework?

• What are the specific quality and performance indicators?

• In which way can the framework be aligned to the EFQM Excellence Model?

Method

To answer the research question the authors identified an empirical approach in which the various demands of the stakeholders could be explored and integrated into one framework. Reviewing several methods [11–13], the authors believed that concept mapping [14] would best meet the requirements because of the following advantages: it uses an empirical method to clarify ambiguous questions and problems; it combines opinion polls and statistical procedures; it can be carried out within a small group and over a 1-day workshop.

Concept mapping has been used in the fields of health, social, and management science [15–17] in order to clarify questions, proposals, and concepts. The method is developed by Trochim and consists of a combination of qualitative and quantitative procedures. The knowledge of the participants is collected and processed using multidimensional scaling and cluster analysis to create a conceptual framework that provides clarity for ambiguous, multidimensional, and controversial concepts. The technique is a consensus procedure based on the input of statements generated by participants who represent different perspectives. Concept mapping is carried out in six defined steps. The statistical procedures are standardized, contributing to high internal validity [18].

1. Step: Preparation

Based on the formulated research problem, the authors identified 16 stakeholders at the Jellinek, eight internal and eight external. The most important selection criterion for the stakeholders was diversity. Through a diverse group of participants at the brainstorming session, a wide variety of statements were generated covering the total conceptual domain. The external stakeholders in the case of the Jellinek Centre included the clients who come for help and have expectations concerning the service and treatment they will receive. The patient council at the Jellinek represented the patients. Funding representatives were included in order to express the distinct requirements concerning results and performance. Additionally, representatives who would voice the needs and expectations of regulatory bodies at central government and local authorities were selected. The internal stakeholders included nurses, social workers, psychiatrists, managers, and administrative staff, who all had their own perspectives of the Jellinek. All 16 stakeholders were invited for a 1-day workshop, which commenced with the introduction of the method.

2. Step: Generation of statements

To stimulate the creativity of the participants during the 1 hour brainstorming session the following questions were tabled: What are in your opinion the critical success criteria for the Jellinek? What must the Jellinek do in order to survive? How can the Jellinek improve its position in a constantly changing environment?

These questions enabled the 16 stakeholders to generate 73 statements, which were fed directly into a personal computer. The composition of the heterogeneous group triggered some exceptional and unconventional items (Table 1).

3. Step: Structuring statements

The seventy-three statements were printed onto cards and rated by each of the participants who had to follow two instructions. Firstly, the items had to be sorted into homogeneous groups. The instruction was to sort the cards into piles, ‘in a way that makes sense to you’. Secondly, a priority score between one (low priority) and five (high priority) had to be given for each statement. This rating process took approximately 2 hours, after which the ratings were fed into the personal computer for statistical analysis.

4. Step: Representation of statements

The statistical analysis was carried out using the software package triadne [19], a program developed to combine multidimensional scaling and cluster analysis. The ratings given by each participant were handled equally. The data from the sorted items were transferred into an association matrix, which represented the probabilities of paired statements. The priority rating data were then transformed into a priority matrix with five priority categories. The association matrix was used to calculate the point and the cluster maps. The priority matrix delivered the ranking of the items and the clusters.

Based on the association matrix, a non-metric multidimensional scaling analysis was carried out [20]. The values of the 73 statements were plotted onto a two-dimensional point map (Figure 1). After the first two factors (19.43, 15.17, 8.28), the values of the five-dimensional solution showed a sharp decrease, which legitimized the use of the two-dimensional solution. The coordinates of the items on the point map served as input for the hierarchical cluster analysis. The result of the analysis was a cluster map that contained grouped items, with the position of each cluster on two dimensions (Figure 2).

5. Step: Interpretation of the maps

During this stage of the workshop, the stakeholders discussed, named, and interpreted the clusters (Table 2). All participants received a copy of the cluster map showing details of the items of each cluster, and the group finally agreed on eight cluster names.

6. Step: Utilization of maps

A small expert group then used the findings of this exercise to make a report, specifying the instruments for the clusters, and aligning the concept map with the EFQM Excellence Model.

Results

The statements

The first result of the concept mapping procedure was the list of statements and their priorities generated during the brainstorming session (Table 1). The three statements with the highest priorities were: ‘patients are satisfied with their treatment (29)’, ‘early interventions to prevent addiction problems (24)’, and ‘efficient treatment link to other services outside the Jellinek Centre (52)’. On a five-point scale, the three items have a priority higher than four, and a standard deviation of less than one.

The point map

Based on the sorting procedure, the point map (see Figure 1) is the first level of analysis and shows the items distributed in a circle, with no items at the centre of the two dimensions. There are more items located at the eastern and western poles of the map, than at the northern and southern poles.

By using the wording agreed by the participants, the authors can state the following:

At the northern pole of the map, statements can be found that relate to treatment efficiency such as ‘low dropout rate and short treatment (statement number 5)’. Items that relate to the results and effectiveness of treatment, such as ‘physical improvement of patients (71), improving patient’s quality of life (31), patients are satisfied with their treatment (29)’ are all located at the eastern pole. Prevention topics and community programs are located at the southern pole, as demonstrated by the statements, ‘openly discuss substance abuse (10)’, preventing addicted persons becoming isolated (8)’, ‘avoiding addict ghetto-forming in Amsterdam (6)’. Items relating to the efficiency of the service network (i.e. beyond the treatment of the Jellinek Centre) are located at the western pole, and include: ‘efficient treatment chain with other services (52)’, ‘insight into the treatment results for each target group (44)’.

The following headings were assigned to the two axes and the four poles:

Vertical axis—Task:

Northern pole—Efficient treatment; southern pole—Prevention programs

Horizontal axis—Organization:

Eastern pole—Results; western pole—Processes

Cluster map

The cluster map constitutes the second level of representation, the thickness of the line surrounding the cluster (Figure 2) indicates the priority given to that cluster. There are five clusters identified that have a priority higher than three. Located on the upper left quadrant of the map, are the clusters concerning service and general treatment process, having the highest priority followed by the client-oriented clusters in the upper right quadrant.

The sequence of the clusters shown in Table 2 is based on the prioritization ratings of the stakeholders. The priority of the cluster can range between one and five.

The five most important clusters are found to be ‘Efficient treatment network’, ‘Effective service’, ‘ Target group’, ‘Quality of life’, and ‘Efficient service’. Those clusters were all located in the northern part of the map. All the stakeholders gave low priority to the clusters concerning the community as a whole. ‘Knowledge transfer’, ‘Reducing addiction related problems’, and ‘Prevention programs’ are seen as less important for addiction centres like the Jellinek. Relevant instruments for indicators for each cluster are provided in Table 2.

Discussion

There are several ways to utilize the findings of the concept mapping strategy. The dimensions identified in the point map, i.e. ‘organization’ (horizontal) with the poles ‘processes’ and ‘results’ and ‘task’ (vertical), with the poles ‘efficient treatment’ and ‘prevention programs’, possibly being used for a fundamental discussion about the outcome of an addiction treatment. The dimensions and poles reflect the demands of the stakeholders, meaning that the Jellinek Centre should have transparent processes and explicit results. Before the concept mapping exercise, the authors expected the internal and external stakeholders to have different views on the priorities for the Jellinek Centre. However, this expectation is not confirmed by the findings. Both the internal and the external stakeholders wanted to see a well-organized, transparent, result- and patient-oriented treatment centre. No stakeholder-specific clusters were found.

When comparing the findings of the point map with the EFQM Excellence Model, the authors found some interesting parallels. The participants of the workshop were not familiar with the EFQM Excellence Model, and most of them had no business or quality measurement background. In particular, the two horizontal pole processes and results resemble the two dimensions of the EFQM Excellence Model enablers and results.

In addition to the general results of the point map, eight clusters were identified that are specific to the addiction treatment centre, but not identical to the four results criteria of the EFQM Excellence Model. However, the authors do believe that they relate to it, and can be utilized to fill in the results criteria with the indicators that are relevant for addiction treatment services. For instance, the clusters ‘Efficient treatment network’, ‘Efficient service’, and ‘Effective service’, form a group concerning efficiency and effectiveness, which cover the content of criterion ‘Key Performance Results’ of the EFQM Excellence Model. The EFQM criterion ‘People Result’ is not represented in the cluster map. Whilst the eight clusters gave a specific indicator framework that could be used to fulfil the results criteria of the EFQM Excellence Model minus the people results, the enabler criteria ‘Leadership’, ‘Policy and Strategy’, ‘People’, and ‘Partnerships and Resources’ were not represented in the concept map.

The emphasis on ‘Efficient treatment network’ as the most important cluster is interesting as it reflects the complexity of the problems faced by patients, i.e. physical, psychological, social, legal, and financial, which should be covered by the specialized treatment facilities. Furthermore, the stakeholders expressed the need for a well-integrated chain of services that operate efficiently and effectively. Indicators with the highest priority for this chain include patient flow between services, and patient profiles which can be based on the Addiction Severity Index [21], or information from administrative databases [22]. The efficiency of the chain depends upon the effectiveness of each single service, which is deemed to be the second most important cluster. Items for the second cluster included dropout rates and time-in-program, both of which could be directly transferred to performance indicators. The clusters that were given third and fourth priority concerned the clients and not the services. Performance indicators for ‘Target group’ could be specified by clinical interviews based on the Addiction Severity Index and Symptom Checklists [23]. The high priority of the cluster ‘Quality of life’ demonstrates that a broad spectrum of measures is needed concerning clinical improvements and patient satisfaction.

Finally there is the question of validity and the generalization of the findings in the study. In this article it is shown that the procedure and the statistical analysis of the concept mapping strategy are systematic, structured, and replicable, which supports the internal validity of the indicator framework. The external validity is much more difficult to estimate. Even so, the first findings of the concept maps of other health care organizations show similar frameworks for indicators [24,25]. Questions such as: ‘Are the items generated during the brainstorming sessions representative of all the indicators relevant for an addiction treatment centre?’ and, ‘Are the participants representative of all the stakeholders?’ have to be explored further. The discussion of these questions exceeds the remit of this article, although further insight into them can be gleaned from some of the first publications [26,27] on those methodological topics.

Conclusion

Concept mapping achieved by means of a 1-day workshop was used to create a framework of indicators for the Jellinek Centre. Based upon 73 statements generated in a brainstorming session that involved 16 internal and external stakeholders, a two-dimensional framework and eight clusters was designed. The framework was considered to be particularly useful to the Jellinek Centre, incorporating three of the four results criteria of the EFQM Excellence Model. Whilst the external validity of concept mapping has to be confirmed by further studies, the authors believe that the internal validity is high and that the framework is meaningful.

Therefore the authors conclude that whilst the findings cannot be used to generalize about other addiction treatment services directly, they could prove useful as a starting point for other addiction treatment centres wishing to develop an indicator framework relative to the EFQM Excellence Model. With this in mind, the authors would encourage other centres to apply this concept of mapping strategy to help them clarify the demands of their stakeholders.

Acknowledgements

M. Donker and P. ten Cate.

References

Author notes

¹Amsterdam Institute for Addiction Research, Amsterdam, The Netherlands, ²University of Amsterdam, Amsterdam, The Netherlands, ³Free University of Amsterdam, Amsterdam, The Netherlands