Using bookmarking methods with orthopedic clinicians and patients with fractures produces score interpretation labels for patient-reported outcome measures

We constructed vignettes based on three PROMIS item banks [13]. Item banks were previously calibrated using an item response theory model, which allows for any combination of items to produce a T-score on the same metric. Briefly, to create vignettes of hypothetical patients, we calculated the most likely response to every item in each item bank for all possible T-scores. For example, a person with very poor physical function (e.g., T = 30) would most likely respond “unable to do” to an item about one’s ability to climb up five steps whereas a high functioning person (e.g., T = 60) would most likely respond “without any difficulty.” We then identified the lowest and highest T-scores that had variability in the most probable response options across items. This range varied by item bank. Vignettes were constructed at 5-point intervals within that range. For Upper Extremity Function, the lowest score selected for a vignette was 17.5. Seven additional vignettes (22.5, 27.5, 32.5, 37.5, 42.5, and 47.5) were then created by selecting items with their most likely response at a given T-score. For Physical Function, nine vignettes were created for T-scores between 17.5 and 57.5. For Pain Interference, seven vignettes were constructed for T-scores between 47.5 and 77.5. Each vignette included six items with only one common item across vignettes for a given domain. Items were selected to maximize variation in response options within a vignette and represent the range of content within an item bank. Upper Extremity Function and Pain Interference vignettes were created de novo and included a total of 33 and 32 items, respectively, across all vignettes. Physical Function vignettes were modified from vignettes used in a previous study [14]. First, to minimize conceptual overlap between the vignettes for Physical Function and Upper Extremity Function, items in the Physical Function vignettes that focused on upper extremity activities were removed and replaced with non-upper extremity items. Second, we added items reflecting better physical function that had been added to the PROMIS Physical Function item bank [23] after the original bookmarking study. A total of 44 items were used. Each vignette’s hypothetical patient was named with a common surname based on the 2010 US Census (see Fig. 1). We pilot tested the vignettes for all three domains to assess how well they reflected distinct levels of function/pain by asking blinded team members and colleagues to order vignettes from least to most severe.

A study coordinator screened sequential patients receiving orthopedic care at an academic medical center in the Northeastern US for eligibility. Eligibility criteria included the following: fracture of the hip, proximal humerus, distal radius, tibial shaft, ankle, or wrist confirmed radiographically or by an attending physician in the past year, age over 18 years old, proficient with oral and written English, ability to send and receive email, and ability to join a focus group from a laptop or desktop computer using a videoconference platform. In order to include diverse perspectives, each group consisted of at least one male and one female, at least 1 person with an injury 1–3 months ago, and at least 2 Latinx, Black, Indigenous, or person of color participants. The study coordinator then approached eligible patients, confirmed eligibility, and provided additional information about the study. Enrolled participants received study materials (e.g., paper copies of vignettes and bookmarks) and were scheduled for a video call in advance of the focus group. During the video call, the study coordinator provided instruction, if needed, on how to use the platform, tested their connectivity, answered questions about the study, and collected sociodemographic information.

Of the 118 patients screened, 86 met clinical eligibility criteria, and 58 were approached. Reasons for not approaching potential participants included the patient missing their appointment (5), the coordinator missing the patient due to schedule conflicts (10), and having already met targets for patients with similar sociodemographic and clinical characteristics (13). Eighteen patients (31%) agreed to participate. Of these, 11 attended one of two focus groups. Participants were compensated for their time.

Clinician members of the study team emailed study information and an invitation to participate to institutional listservs comprised of clinicians serving patients with fractures (e.g., departments of physical therapy, occupational therapy, orthopedics, physical medicine, and rehabilitation; orthopedic advanced practice providers). Additional invitations were distributed to a regional orthopedic society and clinicians in their personal networks. Targeted participants encompassed both the West Coast and Northeast regions of the United States. The study coordinator contacted interested clinicians to complete an eligibility screen and collect sociodemographic information. Similar to the patient group, to include diverse perspectives, each group included a minimum of at least one surgeon and one non-surgeon, and at least one clinician from each region. Enrolled clinician participants were provided study materials in-person or via mail.

Forty-two clinicians expressed interest and were contacted to determine the eligibility. Seven did not complete the eligibility screening. Nine did not meet the eligibility criteria, including treating patients with fractures for more than 1 year post-training, ability to participate in a videoconference from a laptop or desktop computer, and availability at the time of the focus group. Ten responded to the request to complete the eligibility screening after enrollment targets were met. The 16 remaining clinicians were approached and all agreed to participate. All attended one of two focus groups. Clinician participants were compensated for their time and received continuing medical education credit.

All focus groups were three hours long and began with reviewing guidelines for using the videoconference platform including using cameras so all participants were visible. After an explanation of the aims and methods, participants practiced the methods in a warm-up exercise. Next, they were asked to define the severity labels “within normal limits,” “mild,” “moderate,” and “severe.” They were then instructed to complete a seven to ten item short form from the target item bank on paper. These were not collected but used to make participants more familiar with the types of items used to measure a domain and think more deeply about the concept being assessed. Using a paper document that displayed each vignette in order from least severe to most severe, participants placed bookmarks separating vignettes that reflected hypothetical patients who were “within normal limits” or who had “mild,” “moderate,” or “severe” dysfunction or interference. Participants worked independently and reported their results via REDCap, a web-based data collection platform, through a personalized URL sent via the videoconference platform’s messaging tool. Participants then engaged in a moderated discussion sharing how they described each vignette and how they arrived at their conclusion. The moderator displayed the vignettes in the videoconference platform and moved bookmarks in real time to reflect where participants suggested they should be placed. The moderator used a semi-structured guide with specific prompts (e.g., “How did you decide on the location of the bookmark?”, “Knowing that we want to reach an agreement, is anyone persuaded to move their bookmark?”). The discussion continued until there was consensus on bookmark placement. This study was approved by the Institutional Review Board at the lead study site.