Quality of cost evaluations of physician continuous professional development: Systematic review of reporting and methods

We worked with an experienced reference librarian to create a search strategy (see e‑Box in Electronic Supplementary Material [ESM]) that included terms related to the population (e.g., “Physicians”), intervention (e.g., “Education, Continuing”), and outcomes (e.g., “Economics, Medical,” “Costs and Cost Analysis”). We searched MEDLINE, Embase, PsycInfo, and the Cochrane Database, from each database’s inception through April 23, 2020. We identified three additional studies from included article reference lists.

We included all original comparative studies that reported the cost of physician CPD, without restrictions (such as language or date). We defined physician CPD as

Activities intended to promote or measure the clinical knowledge/skills of physicians in independent medical practice through a) courses or assessments delivered in any modality or venue, whether or not [continuing medical education] credit is awarded, or b) self-directed learning or self-assessment activities for which credit is awarded [19].

As comparative studies, we included studies with two or more groups, single-group studies with pre- and post-intervention assessment, cross-sectional studies that simultaneously evaluated training cost and training consequences (outcomes), and economic modeling studies based on a specific, real (not hypothetical) training intervention.

Each article was screened for inclusion independently by two reviewers (the authors or coinvestigators CRS, SM, and BBT recognized in the Acknowledgements), who first reviewed the title and abstract and then reviewed the full text if needed (inter-reviewer reliability, kappa = 0.73). Reviewers resolved all disagreements by consensus.

Two reviewers (DAC, JF) independently reviewed each included study to extract information on reporting and methodological quality.

We coded reporting quality using the CHEERS statement [10], iteratively discussing and operationalizing each reporting element (see Tab. S1 in ESM for operational considerations). We extracted additional detailed information about reporting of abstracts [14] and research questions [11, 23].

We appraised methodological quality in three ways. First, we appraised methodological quality generally using the MERSQI [18], which comprises domains of study design, sampling, outcomes (type, objectivity, and validity), and statistical analyses. Building on operational definitions used in a previous systematic review [15], we revised the coding criteria to accommodate specific features of cost evaluations, as outlined in Tab. 1. Second, we appraised cost-specific methods by coding key methodological features of cost evaluations using the “reference case” domains defined in our scoping review [19], namely: perspective, comparison, description of intervention(s), non-economic measures of effectiveness, training costs, valuation of monetary amounts, analysis, sensitivity analyses, and discounting (see details in Results). Third, we rendered an overall subjective rating of methodological quality using a framework developed by Clune for a review of cost evaluations of education [24]. Clune classified the study methods as supporting results that are “Plausible” (strong quantitation and pricing, effectiveness outcomes, and comparison), “Substantial” (substantial data but serious flaws), “Minimal” (minimal data; list costs, allege affordability, claim feasibility), or “Rhetoric” (claims of cost-effectiveness without data).

As has been done previously [23], we calculated a completeness of reporting index for each report section (title/abstract, introduction, methods, results, and discussion) reflecting the unweighted average percentage of elements reported. We summed the section-specific reporting indices to yield an overall reporting index (all sections, maximum 500) and a main text reporting index (all except title/abstract, maximum 400).

We calculated inter-rater agreement on quality codes using kappa. We used the Kruskall-Wallis or Wilcoxon rank sum test to explore differences in reporting quality and methodological quality over time (dividing articles into four groups of roughly equal size according to publication year), and for structured vs unstructured abstracts. We used Spearman’s rho to explore correlations among the various reporting quality and methodological quality scores. We used SAS 9.4 (SAS Institute, Cary, NC) for all analyses. Statistical significance was defined by a two-sided alpha of 0.05.

As per inclusion criteria, all studies involved practicing physicians. Specialties included family, internal, or general medicine (49 studies), surgery (6 studies), and pediatrics (3 studies). Twenty-one studies additionally involved other health professionals, including nurses (10 studies), nurse practitioners/physician assistants (8 studies), and postgraduate physician trainees (5 studies). CPD topics included general medicine (35 studies), drug prescribing (11 studies), procedural skills (7 studies), and pediatrics (3 studies).

The frequency of reporting of each CHEERS element is shown in Fig. 1. Inter-rater reliability was kappa ≥0.67 (“substantial” or “near-perfect”) for all these items. The reporting index (percentage of CHEERS elements reported in a given article section, maximum 100) ranged from 43 to 66 per section, namely: Title/Abstract mean (SD) 43 (20), Introduction 56 (34), Methods 66 (19), Results 61 (17), and Discussion 66 (30). The overall reporting index (sum of the five section-specific reporting indices, maximum 500) was 292 (83), ranging from 94 to 432 per study.

Only 28 studies (45%) mentioned economics or cost (or a similar term) in the title, and 21 (34%) mentioned the need to study the costs or economics of education in the Introduction. Forty-eight studies (77%) reported a specific statement of study intent (research question, purpose, goal, aim, or hypothesis). Among these, we further coded for four key elements of the statement of study intent, as defined in a previous review [11]: the population (reported in 33 [69% of 48]), the primary intervention (40 [83%]), the comparison or anticipated association (13 [27%]), and the outcome (44 [92%]). Thirty-eight (79%) reported a cost-related outcome in the statement of study intent, and 20 (42%) mentioned an effectiveness outcome.

The number of study participants was reported inconsistently: 42 studies (68%) reported the number of physicians trained and 23 (37%) reported the number of patients (i.e., in outcomes analyses); either physicians or patients were numbered in 48 (77%). The “intervention” training activity was well-described in 40 studies (65%). Among 36 studies with a separate comparison arm, the comparison activity was well-described in 26 (72%); this included 15 of 22 studies (68%) with no-intervention comparisons (i.e., explicit mention that the comparison group received no training) and 14 of 17 studies (82%) with an active training comparison. Given the critical role of context in cost evaluations, we coded both the training setting (physical location of training, experience/qualifications of CPD provider, or relationship between provider and physicians; reported in 42 [68%]) and the larger clinical context (patient demographics, or institution type, size, or infrastructure; reported in 38 [61%]).

The selection, quantitation, and pricing of resources (collectively, the valuation process) constitute the core of an economic analysis, yet these methods were reported infrequently. Although 40 (65%) listed the resources counted in valuation, only 10 (16%) reported the method for selecting these resources for inclusion. Likewise, methods of resource quantitation were reported infrequently: 10 studies (16%) reported both quantities and data sources, another 8 (13%) reported quantities without specifying the source, and 44 (71%) reported no quantities (i.e., reported the cost alone). The source of price information was mentioned in 26 (42%). Fifty-seven studies (92%) reported effectiveness (non-cost) outcomes; of these, 28 (49%) reported in detail the methods used to measure such outcomes.

As required for inclusion in this review, all studies reported the cost of at least one training intervention. However, only 13 (21%) detailed the quantities and prices of resources used. The CHEERS statement recommends reporting both the total costs for each intervention, and the incremental cost between interventions or relative to baseline (the cost difference or ratio). Fifty-five studies (89%) reported the total cost of at least one intervention, and 16 of 17 studies with an active comparison intervention (94%) reported the comparison’s total cost. Nineteen studies (31%) reported the incremental cost.

Although 44 studies (71%) reported at least a partial list of study limitations, only 30 (48%) discussed the generalizability of findings (i.e., applicability to other contexts). Forty-three (69%) listed the funding source, and 24 (39%) mentioned potential conflicts of interest.

The CHEERS guidelines recommend “reporting a structured summary [abstract] of objectives, perspective, setting, methods (including study design and inputs), results (including base-case and uncertainty analyses), and conclusions” [10]. We coded each abstract for these elements, and a description of participants and the primary and comparison training interventions (Fig. 2; Tab. S1 in ESM). Six studies (10%) had no abstract. Among the remaining 56, on average 5.8 (2.1) of 12 elements (median 6, range 2–10) were reported. Only 7 of 56 (13%) used a highly structured abstract (>5 headings), and none of these used headings specific to economic studies. Most (32 [57%]) used a four- or five-heading variation of purpose-methods-results-conclusions, and 17 (30%) were unstructured. The cost “inputs” (specific resources included in cost analyses, reported in 4 abstracts [7%]), economic perspective (7 [13%]), and sensitivity analyses (10 [18%], including 6 for training costs and 4 for clinical cost outcomes) were reported least often. Structured abstracts contained more elements than unstructured abstracts (mean [SD] 6.6 [1.7] vs 3.8 [1.7], p < 0.001), and highly structured abstracts contained more elements than less or unstructured abstracts (7.7 [2.1] vs 5.5 [2.0], p = 0.02).

We appraised general methodological quality with the MERSQI, using operational definitions summarized in Tab. 1. Scores ranged from 6.0 to 16.0, with mean (SD) 11.2 (2.4), median 11.0 (IQR, 9.5, 13.0). The domain with lowest scores was validation of the evaluation instrument: 50 studies (81%) reported zero evidence sources, and only 1 study (2%) reported all 3 of the sources appraised in the MERSQI (content, internal structure, and relations with other variables). Appropriate data analysis (operationalized as quantitation of resources) was also infrequent (25 [40%]).

We appraised cost-specific methodological quality according to the following “reference case” domains [19]:

Finally, we rendered a subjective judgment of overall methodological quality (the strength of each study’s findings). We rated 6 studies (10%) as having “Plausible” results, 15 (24%) as “Substantial,” and 41 (66%) as “Minimal.” No studies were coded as “Rhetoric,” since such studies did not meet inclusion criteria. Scoring these as Plausible = 3, Substantial = 2, Minimal = 1, the mean (SD) was 1.4 (0.7).

We divided studies by publication date into four groups of similar size. Reporting quality improved over time (see Tab. S3 in ESM), with the overall reporting index rising from mean (SD) 241 (105) in 16 studies before 1999 to 321 (52) in 17 studies after 2010; however, this trend did not reach statistical significance (p = 0.08). Methodological quality also varied over time, with total MERSQI scores rising from 9.8 (2.7) to 11.9 (1.9) and subjective methodological quality rising from 1.3 (0.6) to 1.7 (0.8); however, these changes likewise did not reach statistical significance (p ≥ 0.15). We also report differences in the frequency of reporting conflicts of interest (0/16 vs 12/17, p < 0.001), quantitation of resources (3/16 vs 4/17, p = 0.70), pricing of resources (4/16 vs 11/17, p = 0.09), and sensitivity analyses (1/16 vs 4/17, p = 0.61) over the same time periods (see Tab. S3 in ESM for details, including the interim chronological subgroups).

We found moderate to large, positive, statistically significant correlations between reporting and methodological quality, namely: overall reporting and MERSQI total score, rho = 0.62; overall reporting and subjective methodological quality, rho = 0.58; and MERSQI score and subjective quality, rho = 0.50 (all p < 0.001). We found a slightly lower correlation between abstract quality (number of items reported) and main text reporting (rho = 0.45, p < 0.001). Studies with structured abstracts had higher main text reporting (271 [47], maximum 400) than those with unstructured abstracts (229 [87]), p = 0.04.

Most of the outcomes in this study required reviewer judgments of quality; to facilitate rigor and minimize subjectivity, we defined and applied specific operational criteria (Tab. S1 in ESM; Tab. 1). The CHEERS statement provides suggestions for additional details that we did not code (such as analytic methods, probability distributions, and base rates) because we viewed them as subordinate. Some included studies had cost as the primary focus, while for others costs were a secondary outcome; for the latter studies, reporting might have been appraised more favorably using reporting guidelines for other research designs (e.g., for randomized [9] and non-randomized [25] trials). The correlation between reporting and methodological quality likely reflects, in part, the fact that methods can only be judged from what is reported. Strengths of our study include the systematic search, duplicate review, use of a widely accepted reporting standard, and methodological appraisal using both cost-specific and general criteria.

Two previous systematic reviews appraised cost evaluations in HPE, one using the BMJ Guidelines [15], the other focusing on cost ingredients [6], and both applying the MERSQI. Our review, which employed the more comprehensive CHEERS to appraise reporting and two additional approaches to appraise methodological quality, confirms those findings and identifies new areas of strength and deficiency. These findings also provide quantitative data elaborating the qualitative themes identified in our previous scoping review [19]. Our findings of significant deficiencies in reporting and methods parallel those from previous reviews of non-cost research in HPE [11, 14, 23, 26, 27]. Finally, looking outside HPE, in 2002 Clune rated the methodological quality of 56 cost evaluations as rhetoric (23 [41%]), and the remaining 33 as minimal (16 [48% of 33]), substantial (13 [39%]), and plausible (4 [12%]) [24]. Our findings roughly parallel his.

Our findings have implications for research in CPD and for HPE research broadly. First, these data highlight specific gaps in reporting of CPD cost evaluations that warrant particular attention, such as valuation (resource selection, quantitation, and pricing), number trained, setting, description of the intervention and comparison, and discussions of generalizability (application in other training and clinical contexts). Unfortunately, poor reporting is ubiquitous, persistent, and refractory to remediation [28‐32]. Indeed, our data show that reporting quality has improved only slightly since publication of the BMJ Guidelines [16] in 1996 and CHEERS [10] in 2013 (Tab. S3 in ESM). Clearly, the solution is not simple, and merely publishing reporting guidelines is insufficient. We call upon authors, editors, reviewers, and faculty development educators to collaborate to improve reporting quality.

Our findings also highlight specific deficiencies in titles and abstracts, and complement studies showing shortcomings of abstracts in other fields [14, 23, 33‐35]. These deficiencies are not trivial: the abstract is critical in attracting consumers to read the full text, and essential to appropriate indexing for and searching of literature databases [36]; and titles are the “shortest possible abstract” [37]. Moreover, we found that structured abstracts are associated with more complete reporting. More informative and more structured abstracts have been requested for over 30 years [12, 13]. Tips for writing effective titles and abstracts are available [38, 39].

Our findings identify potential improvements in methodological quality. We emphasize the critical importance of resource quantitation, which is typically considered more important than actual costs [40‐42], since prices vary widely by locale and over time, and quantities can be re-priced as needed using locally accurate values. Selection of relevant resources is also essential yet rarely reported and, we suspect, often done with little deliberate attention. Other potential methodological improvements include choosing an active comparison intervention, articulating the economic perspective, calculating combined expressions of cost and impact, formally analyzing incremental costs and cost-effectiveness, and performing sensitivity analyses.

Researchers and peer reviewers will benefit from our operational clarifications of MERSQI criteria for application to cost evaluations (Tab. 1) and our operational considerations for coding reporting quality with CHEERS (Tab. S1 in ESM), as they plan, report, and appraise such studies.

Finally, we note the overall paucity of cost-related studies. During the 63 years encompassed in this review, over 3520 original research studies were indexed under the MeSH term “Education, Medical, Continuing.” The 62 studies of CPD costs that we identified comprise only 1.8% of these (a proportion comparable to that found in previous reviews [6, 15]). We suggest that more studies that robustly measure and report the costs of CPD (and HPE generally) will contribute to informed decisions, efficient resource allocation, and ultimately a better-trained workforce.