Do workshops in evidence-based practice equip participants to identify and answer questions requiring consideration of clinical research? A diagnostic skill assessment

Abstract

Evidence-based practice (EBP) requires practitioners to identify and formulate questions in response to patient encounters, and to seek, select, and appraise applicable clinical research. A standardized workshop format serves as the model for training of medical educators in these skills. We developed an evaluation exercise to assess the ability to identify and solve a problem requiring the use of targeted skills and administered it to 47 North American junior faculty and residents in various specialties at the close of two short workshops in EBP. Prior to the workshop, subjects reported prior training in EBP and completed a previously validated knowledge test. Our post-workshop exercise differed from the baseline measures and required participants to spontaneously identify a suitable question in response to a simulated clinical encounter, followed by a description of a stepwise approach to answering it. They then responded to successively more explicitly prompted queries relevant to their question. We analyzed responses to identify areas of skill deficiency and potential reasons for these deficiencies. Twelve respondents (26%) initially failed to identify a suitable question in response to the clinical scenario. Ability to choose a suitable question correlated with the ability to connect an original question to an appropriate study design. Prior EBP training correlated with the pretest score but not with performance on our exercise. Overall performance correlated with ability to correctly classify their questions as pertaining to therapy, diagnosis, prognosis, or harm. We conclude that faculty and residents completing standard workshops in EBP may still lack the ability to initiate and investigate original clinical inquiries using EBP skills.

Appendices

Appendix A Example scenario and target questions used in the exercise

Scenario A

It is a very busy shift in the emergency department of your hospital. You are attending to several patients with potentially significant illness. At 4 pm, a 53-year-old male is brought in by ambulance with a chief complaint of chest pain. He has been given sublingual nitroglycerin by paramedics together with O2 and they report that he experienced relief of pain. In the ED his vital signs are BP 150/95 P 100 R 18 T 98.9 O2 sat 97% on room air.

The patient tells you that the pain began about 6 h ago after arising from bed and persisted until the ambulance arrived. He has had respiratory discomfort associated with the pain but is unclear about whether the pain is actually worse during inspiration. He denies other associated symptoms such as nausea, vomiting or diaphoresis. The pain was relatively mild when he first noticed it and he dismissed it as unimportant. He describes the pain as located over the left precordium and extending to the shoulder but not radiating to arms, back, neck or jaw.

The patient has a history of hypertension for which he takes a diuretic and has recently been told that his cholesterol is ‘a bit high.’ He takes medicine sporadically for allergic rhinitis and had an appendectomy some years ago. He denies previous episodes of chest pain. He admits to social drinking and occasional use of recreational drugs. He tells you that his father died of a ‘heart attack’ at age 61 and his mother of breast cancer. There is no other remarkable family or personal history. The patient appears stable and not in distress. His physical exam is unremarkable. EKG shows a NSR at rate of 98 and no acute ST changes.

As you finish your evaluation, the nurse reports to you that the bedside troponin I is negative, and asks if you would like to give something to the patient.

Appendix B Rubric used to score responses to the evaluation exercise

Point Allocation Part 1 (Open ended)	Points
1. Picks foreground question	1
Picks background question	0
2. Completes PICO appropriately for index question	2
Identifies components of PICO, or equivalent	1
No recognition of PICO or equivalent	0
3. Identifies appropriate foreground question type (therapy, diagnosis etc)	2
Identifies inappropriate foreground question type (therapy, diagnosis etc)	1
No recognition of question type	0
4. Identifies or describes appropriate study design	2
Identifies or describes inappropriate study design or hierarchy	1
No recognition of study design	0
5. Identifies appropriate filtered resource	2
Identifies appropriate non-filtered resource or inappropriate filtered	1
No recognition of choice of resource	0
6. Defines search strategy	1
No recognition of search strategy	0
7. Identifies appropriate specific appraisal criteria	2
Mentions critical appraisal of studies selected	1
No recognition of critical appraisal of studies	0
8. Mentions applicability assessment	1
No recognition of applicability assessment	0
9. Correctly Identifies original question as foreground	1
Identifies as background and selects a true foreground question	1
Identifies as background but poses another background query	0
Incorrectly identifies original question as foreground	0