The use of concept mapping in measurement development and evaluation: Application and future directions
Introduction
Concept mapping is framed as an inclusive, participatory, collaborative, and inductive social science research process (Kane & Trochim, 2009). The methodology’s flexibility is recognized as a strength, and the number of topics for which the method could be applied seems virtually limitless. It enables both detailed idea generation by stakeholders and higher-level conceptual representation. Although sophisticated multivariate analyses are employed, the results are visual and intuitive, thereby enhancing interpretation and use (Kane and Trochim, 2007, Trochim, 1989). Over its 25 year history, concept mapping has been used in an array of fields to develop theory, plan for programs and social interventions, evaluate social programs, and develop measures and scales (Kane & Trochim, 2009).
The foundation for the use of concept mapping in measurement was outlined in the early development and articulation of the method. Drawing from Campbell (1966); Campbell (1986) ideas about the natural coherence between observable patterns in both theory and reality, Trochim (1985) framed social research as a pattern matching exercise that involves correspondence between conceptual and operational domains. This provided the philosophical and epistemological basis for concept mapping as a technique for explicating a conceptual domain. Later writings concentrated on the methodological tenets of concept mapping as an integrated, mixed-methods approach that enabled groups to conceptualize an issue of relevance (Trochim and Linton, 1986, Trochim, 1989). The publication of a special issue in Evaluation and Program Planning in 1989 introduced concept mapping to the broader community of evaluators and researchers, and offered a practical and useful conceptualization tool for managing diverse perspectives and distributed group knowledge. Two early studies highlighted the application of concept mapping to measurement development (e.g., scales, measures, questionnaires). Galvin (1989) used the method to organize a stakeholder-produced conceptual framework from which an evaluation questionnaire was directly constructed. In generating additional content of relevance, vander Waal, Casparie, and Lako, 1996 used concept mapping to purposefully include representatives of the intended targets of the measure, and emphasized their contributions to the clarity and validity of the instrument. Although these studies lacked a complete description of the practical, step-wise application of concept mapping in the context of traditional scale development and psychometric testing procedures, they suggested the flexibility of the method to support and enhance measurement quality.
Since 2000, a body of measurement development research within the social, behavioral, and health sciences that includes concept mapping as a primary technique has emerged. Several aspects of the method have likely contributed to its presence in the literature. The generation of a large set of ideas, structuring of ideas based on judgments made about their interrelationships, graphical representations of scaled similarities among theoretical ideas, and identification of clustered sets of like items are some practical features that align concept mapping with general approaches for measurement development (Kane & Trochim, 2009). In their contemporary 8-stage mixed-methods framework for instrument development, Velozo et al. (2012) recommended concept mapping as a structured qualitative method for conceptualizing the construct(s) to be measured and developing representative items. They further emphasized the method's value in synthesizing literature findings, operationally defining constructs, and generating hypotheses about the scope and content of the scale. Previous research has demonstrated concept mapping to be a valid and reliable conceptualization approach in general (Rosas & Kane, 2012). However, little guidance or understanding is available on how concept mapping can and should be integrated in the measurement development process. Furthermore, despite the purported epistemological, methodological, and ontological value of concept mapping, little has been done to critically review how researchers have approached the application of the method in applied measurement development research.
To that end, we systematically reviewed the literature on concept mapping to identify where and how the method was applied in the context of measurement development and evaluation. In this review we examine the practice of integrating the concept mapping methodology into processes for establishing new measurement tools in accordance with generally accepted development and testing procedures based on established psychometric principles. From this examination we assess the current practice of using concept mapping in applied measurement development research, noting the strengths, limitations, and future directions for the field.
To begin, it is useful to broadly outline the measurement development and psychometric evaluation process within social, behavior, and health sciences research. This multi-step process generally involves the (a) articulation of construct(s) of interest and their context, (b) specification of the response format and selection of the initial items, (c) collection of data from a set of target respondents, and (d) examination of the psychometric properties and determination of quality (DeVellis, 2011, Furr, 2011, Simms, 2008).
Formal development activities are conducted to protect against two types of error: measuring less than the proposed construct (i.e., construct underrepresentation) and measuring more than the proposed construct (i.e., construct irrelevant invariance). Rigor in the process of conceptualization and definition is required to avoid the first type of error. Establishing content validity – the minimum psychometric requirement for measurement adequacy – relies on sufficiently capturing the specific domain of interest, while simultaneously containing no extraneous content (Netemeyer et al., 2003; Schriesheim, Powers, Scandura, Gardiner, & Lankau, 1993). Rigor in psychometric analysis is required to avoid the second type of error. Reliability and validity are fundamental facets of psychometric quality and researchers strive to provide evidence regarding the nature and strength of these characteristics (Furr, 2011). Psychometric quality is further demonstrated in the assessment of the instrument's performance in the sample being studied through results that truly reflect the hypothetical construct(s) it purports to measure (DeVellis, 2011). It is within this ongoing, iterative process, that use of concept mapping as a core method in measurement development and evaluation is reviewed.
Section snippets
Review sample selection
Our review began with a literature search to identify a sample of published studies where concept mapping was employed as a principal method in the measurement development process. Due to the range of fields where concept mapping has been used, we determined a broad search was warranted using several highly-cited publications as sources. We identified three seminal publications, Trochim and Linton (1986), Trochim (1989), and Kane and Trochim (2007) as the most frequently referenced source
Results
This review focused on the 23 measurement development studies listed in Table 1. Two subsets are included in this final sample. In one set (n = 15), concept mapping, scale and measure construction, and psychometric testing and evaluation processes were reported within a single manuscript. In the other set (n = 8), the aforementioned processes were published separately. In these paired publications, the first described the use of concept mapping to generate conceptual structure of the content for a
Discussion
According to McGrath (2005), the conceptual complexity of constructs and their associated measures have played an important role in the failure to develop more accurate measurement systems. The studies included in this review integrated concept mapping in a variety of ways to establish a clear conceptual measurement framework. The new instruments produced in these studies demonstrated adequate to strong psychometric properties through the use of multiple analytical methods following
References1 (81)
- et al.
Validation of a screener and opioid assessment measure for patients with chronic pain
Pain
(2004) - et al.
Development and validation of the current opioid misuse measure
Pain
(2007) - et al.
Improving treatment with methotrexate in rheumatoid arthritis: Development of a multimedia patient education program and the MiRAK, a new instrument to evaluate methotrexate-related knowledge
Seminars in Arthritis and Rheumatism
(2014) - et al.
The patient’s perspective of the quality of breast cancer care: The development of an instrument to measure quality of care through focus groups and concept mapping with breast cancer patients
European Journal of Cancer
(2007) Concept mapping for planning and evaluation of a big brother/big sister program
Evaluation and Program Planning
(1989)A review of scale development practices in the study of organizations
Journal of Management
(1995)- et al.
Creating a quality of life assessment measure for residents in long term care
Journal of the American Medical Directors Association
(2012) - et al.
The Health Literacy Management Scale (HeLMS): A measure of an individual's capacity to seek, understand and use health information within the healthcare setting
Patient Education and Counseling
(2013) - et al.
The Health Education Impact Questionnaire (heiQ): An outcomes and evaluation measure for patient education and self-management interventions for people with chronic conditions
Patient Education and Counseling
(2007) - et al.
Development and validation of the influenza intensity and impact questionnaire (FluiiQ™)
Value in Health
(2011)
The use of concept mapping for scale development and validation in evaluation
Evaluation and Program Planning
Quality and rigor of the concept mapping methodology: a pooled study analysis
Evaluation and Program Planning
Improving construct measurement in management research: Comments and a quantitative approach for assessing the theoretical content adequacy of paper-and-pencil survey-type instruments
Journal of Management
Integration of primary health care services: Perceptions of Australian general practitioners, non- general practitioner health service providers and consumers at the general practice–primary care interface
Evaluation and Program Planning
Conceptualization for evaluation and planning
Evaluation and Program Planning
An introduction to concept mapping for planning and evaluation
Evaluation and Program Planning
Improving measurement methods in rehabilitation: Core concepts and recommendations for scale development
Archives of Physical Medicine and Rehabilitation
eTailQ: Dimensionalizing, measuring and predicting etail quality
Journal of Retailing
Quality of health care in inflammatory bowel disease: Development of a reliable questionnaire (QUOTE-IBD) and first results
The American Journal of Gastroenterology
Quality of care: A comparison of preferences between medical specialists and patients with chronic diseases
Social Science & Medicine
The recovery promotion fidelity scale: Assessing the organizational promotion of recovery
Community Mental Health Journal
Construction of a GP integration model
Social Science & Medicine
Defining community readiness for the implementation of a system of care
Administration and Policy in Mental Health and Mental Health Services Research
Conflict in small groups: The meaning and consequences of process conflict
Small Group Research
Psychometric properties of the consumer quality index to assess shelter and community care services
The European Journal of Public Health
A validity-driven approach to the understanding of the personal and societal burden of low back pain: Development of a conceptual and measurement model
Arthritis Research &Therapy
Pattern matching as an essential in distal knowing
Relabeling internal and external validity for applied social scientists
New Directions for Program Evaluation
The psychosocial references of older adults: A pilot examination of content and structure
The Gerontologist
Is Rasch model analysis applicable in small sample size pilot studies for assessing item characteristics? An example using PROMIS pain behavior item bank data
Quality of Life Research
Constructing validity: Basic issues in scale development
Psychological Assessment
The Oregon mental health referral checklists: Concept mapping the mental health needs of youth in the juvenile justice system
Brief Treatment and Crisis Intervention
Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis
Practical Assessment, Research & Evaluation
Self-report measure of financial exploitation of older adults
The Gerontologist
Self-report measure of psychological abuse of older adults
The Gerontologist
Conceptual model and map of psychological abuse of older adults
Journal of Elder Abuse & Neglect
Conceptual model and map of financial exploitation of older adults
Journal of Elder Abuse and Neglect
Construct validity in psychological tests
Psychological Bulletin
Multidimensional scaling
Cited by (51)
Exploring actors' perceptions of the precision agriculture innovation system – A Group Concept Mapping approach in Germany and Switzerland
2023, Technological Forecasting and Social ChangeA conceptual model for mental health and performance of North American athletes: A mixed methods study
2022, Psychology of Sport and ExerciseCitation Excerpt :The conceptual map in this survey can serve as the foundation for these initiatives, as it addresses social, psychological, and physiological factors that contribute to optimal mental health and performance for athletes. Given its rigorous methodology, group concept mapping is a core technique used in measurement development and assessment (Rosas & Ridings, 2017). The items in this conceptual model could thus be used to create a psychometrically sound assessment for athlete wellness and performance.
Conceptualization and Inventory of the Sexual and Psychological Burden of Women With Pelvic Floor Complaints; A Mixed-Method Study
2022, Sexual MedicineCitation Excerpt :Typically, GCM consists of a number of successive steps, namely idea generation such as brainstorming, the synthesis of ideas, the sorting and rating of statements, multivariate analyses, and the interpretation of the output.43,46 As a post hoc analysis of the emergent conceptual model, a selection process was undertaken to identify statements that can be used in future research for inventory development.42,44 By following the procedure for item selection in measure development outlined by Nunnally,47 a core set of statements from the conceptual model were selected.
A valid and reliable scale for measuring middle managers’ trustworthiness in continuous improvement
2021, International Journal of Production EconomicsCitation Excerpt :In our case, following previous recommendations, the techniques of concept mapping (Rosas, 2017) and expert panel (Landeta, 2006) were used. The use of concept mapping allows a particular concept to be made more explicit and operational based on the personal knowledge of a group of participants (Kane and Trochim, 2007; Rosas, 2017; Trochim, 1989). Following the five main steps developed by Trochim (1989), we developed two different concept mappings: one with six operators and another with ten middle managers.
“Sensing” the destination: Development of the destination sensescape index
2021, Tourism Management
- 1
Astricks denotes studies included in the review.