One report, multiple aims: orthopedic surgeons vary how they use patient-reported outcomes with patients

The ASK study is a pragmatic, cluster randomized controlled trial (ClinicalTrials.gov identifier: NCT03102580; 2018–2023) evaluating decision quality using a shared decision report (see Supplemental Material) designed through iterative consultation and engagement with clinicians and patients [7]. The ASK study used group level randomization at the site (office) level. A total of 36 surgeons at 10 different orthopedic physician groups (or practices) participated across 14 different locations including included medical school-based practices (8), community-based arthroplasty specialty practices (2), and community-based general orthopedic practices (4).

The intervention arm receives a 3-page report including individual level PROM and risk factor data collected from patients prior to the clinic visit. The report uses hip and knee specific PROMs (HOOS12, KOOS12) [8, 9] as well as measures of global physical and emotional health (SF-36 Physical Component Summary (PCS), Mental Component Summary (MCS)) [10]. It also includes anticipated 12-month pain and function outcomes based upon a validated predictive analytic algorithm using norms from the national Function and Outcomes Research for Comparative Effectiveness in Total Joint Replacement (FORCE-TJR) registry [7]. Specifically, our team generated six prediction models for primary knee and hip patients respectively as well as prediction models for three outcomes at 12 months: KOOS or HOOS pain, KOOS or HOOS function/ADL; and SF-36 PCS. The candidate predictors included demographic factors (age, sex, race, ethnicity), medical comorbidities (BMI, modified Charlson Comorbidity Index), musculoskeletal comorbidities (moderate or severe pain in low back (Oswestry) or non-operative hips and knees), and baseline status for the relevant outcome (pain, function/ADL, or PCS). These candidate predictors were identified based on orthopedic and OA literature and prior models [11‐13].

The final page includes information about non-operative treatment options drawn from the American Association of Orthopedic Surgeons guidelines [14, 15]. The usual care arm receives a brief report of PROM and risk factor data without the estimated pain and function outcomes and non-operative treatment information.

Eligibility criteria included surgeons participating in the ASK study for at least 6 months and actively enrolling patients into the study. All received a 3-page printed report with descriptive and predictive PROM information for patients considering total knee or hip arthroplasty. Of the participating 24 surgeons, 16 met criteria. All were invited to participate in a 30-min interview via video call. Interviews were recorded with verbal consent from all participants.

We created a semi-structured interview guide that addressed each section of the report. Open-ended questions explored how the surgeon used report content, features that were helpful, confusing, or could be improved, and how report use fit into the surgeon’s workflow. At the conclusion of the interview, participants were asked about number of years in clinical practice, experience with PROs during residency training, and if they collected PROMs for other patient populations or procedures outside of the ASK study. Two investigators (DCL, NER) used the interview guide with one surgeon participant and then revised the guide to improve clarity. The investigators then independently interviewed the remaining 8 participants via video call. Interviews were recorded, transcribed, and de-identified.

We used a conventional content analysis to analyze data [16]. Both interviewers read all transcripts and independently created initial codes. Together they discussed and reconciled codes to produce a final codebook and definitions. Each independently coded all transcripts, then together reviewed coded transcripts to reconcile discrepancies. Next, both interviewers reviewed all excerpts assigned a given code to confirm or revise assignment as well as categorize concepts. For two concepts, sub-themes were identified. Again, each interviewer assigned sub-themes independently, then reconciled discrepancies together. To increase accuracy of data interpretation, we presented preliminary results to ASK surgeons and staff via webinar. Their feedback was then used to refine data interpretation. We used Dedoose v9.0.18 (Los Angeles, CA) for data management.

All surgeons shared positive feedback about using the report. They described it as providing an easy, quick synopsis of helpful information similar to a case presentation. Surgeons liked the quantification of pain and physical function (e.g., “Giving some objective numbers to the patient [sic] symptoms, I think, is very, very important.” I). They reported patients liked the report and felt it increased patients’ knowledge of and confidence in the treatment plan. Surgeons noted that the report may be particularly helpful for older adults, patients with higher health literacy or numeracy, patients who are overly optimistic about their treatment outcomes, and patients that are seeking information or highly engaged in their care. Surgeons indicated the report may be less helpful for patients who are skeptical, don’t use the word “pain” to describe their symptoms, have lower health literacy or numeracy, have high self-efficacy (as no additional support or information is needed), and when a patient has already made a treatment decision (e.g., “And some patients are like, well, I already knew I wanted surgery anyways, so this is not changing my mind or making an difference in terms of my decision or outcomes.” D).

Surgeons described using the report to achieve seven different goals (Table 3). All surgeons used different components of the report to educate patients. Education included giving patients context for how their pain and function compared with other patients (e.g., “…using the pain numbers and the function numbers to explain where they are in their disease process in relation to other patients with the same disease.” I). Surgeons also used the report’s information about surgical risk factors to highlight their impact on predicted pain, function, and general physical health outcomes (e.g., “I really go over what their predicted score changes will be and show them what comorbidities are affecting them that we can’t change. So, like low back pain, or whatever.” A).

Education and expectation setting accounted for 60% of all communication goals excerpted, with the remaining excerpts describing five other goals. All surgeons used the report for at least one of these additional goals, but no two surgeons had an identical set of communication goals. Goals were not correlated with whether or not the report was used prior to the visit. Five surgeons (56%) used the report to support shared decision-making (e.g., “They don’t want to undergo surgery. And I go, ‘I totally recognize that. Let’s see if there’s any other modalities that we can do.’” D). Four surgeons (44%) used the report as an additional support for the treatment plan, both for surgery and for non-surgical interventions (e.g., “I’ll kind of use this [report] as one more piece of information, to kind of support the decision that now is not the time [for surgery], or they’re not a good candidate for a joint replacement.” C). Four surgeons used the report as a source of information (e.g., “I start talking to the patient based off what I’m seeing [in the report].” G). One surgeon noted that sometimes the report identifies pain and functional problems on the contralateral side that the patient didn’t mention during their discussion. Three surgeons described utilizing the report’s indication of the presence of risk factors for surgery (e.g., body mass index [BMI], smoking, mental health) to coach patients for behavior change (e.g., “If these are the factors that we can modify, then we need to modify it to get a better outcome.” H). They also noted that patients may be more forthright about these factors in a patient-completed assessment than in conversation with the surgeon (e.g., “[patients] may not say that they are smokers, or they have emotional health issues, or narcotics use, so sometimes as they keep answering questions, it comes out.” H). Surprisingly, although the report captures and displays a patient’s pain and function prior to surgery, three surgeons referenced the report following TJR to re-frame progress after surgery. Specifically, they highlight how today’s pain and function compares to the pre-surgical report noting the timeline for improvement is 12 months (e.g., “You were at a 19, and this is where your pain is now. You’re now at a 97. That shows an incredible improvement.” I).

Surgeons identified components of the report that were helpful for specific communication aims (e.g., graphs of hip and knee pain used to educate patients; predicted outcomes post-surgery used for expectation setting). Each component of the “Factors Affecting Your Arthritis Joint and Joint Replacement Outcomes” (i.e., BMI, emotional health, narcotics use, low back pain, medical comorbidity index, age) was identified by at least one surgeon as useful. Surgeons also identified aspects of the report that they didn’t use, which was most commonly information on non-surgical treatment options.

The report configuration was identified as helpful. This included having information that compared a patient to other patients. They also liked being able to compare hip pain to knee pain and the left side to the right side rather than having the report only include the joint discussed at that visit. Second, they found the use of color to convey score meaning (i.e., green, yellow, red) helpful, particularly when displayed in distinct tiers versus color gradations (e.g., greenish-yellow). Surgeons reported that patients benefited from having information displayed graphically and that both patients and surgeons liked having a physical, printed report (e.g., “I have patients who love this. They love the feedback of it because it gives them one, something tangible that leaves the office, which I think is very useful sometimes.” D).

Surgeons identified 8 challenges in using the report in practice. PROM score interpretation was the most frequently identified challenge in terms of number of surgeons (n = 8) and number of comments (n = 22). Surgeons described difficulty interpreting “overall physical function” and “general physical health” scores (e.g., “I don’t know exactly what that [PCS score] means for some patients. Because they’ll have – you know, you’re coming in for knee pain, and the knee pain is not too bad but the overall physical function is really bad. Or the opposite. And then, what exactly does that mean?” F). Similarly, some noted difficulty interpreting a “joint function” score that was a summary of joints, not specific to the joint being discussed (e.g., “But a lot of patients have complaints that are very specific to one joint, so I think that the function and physical health is not quite as concrete that the patient can understand.” A). Another noted challenge included different scales (e.g., range 0–100 for pain with 100 as goal (KOOS12/HOOS12); and 20–80 with 50 as norm for overall physical function (SF-36)) and lack of clarity on whether scores represented percentiles (e.g., “What does that 19 mean? You know, does that mean the 19^th percentile out of 100 people similar to it? Is it just an asterisk score which is 19?” F). Although all graphs had indications of the direction of scores, it was not always intuitive (e.g., “It looks like pain is increasing, and I have to say, pain is improving…But normally, when you think of your pain getting better, on a 10-point scale, the number goes down.” D). Surgeons reported that patients asked about the specific meaning of PROM items included in the report (e.g., “Sometimes patients talk about walking on a flat surface. They talk about, ‘Well, walking how far?…ten feet or…five miles?’” I).

A second commonly noted challenge related to workflow in clinic. The parent study’s staff were responsible for ensuring patient assessments were completed in advance of seeing the surgeon. Two sites provided an in-office option for patients to complete the assessment if not completed prior to arrival at the surgeon office. However, without this pre-visit study support, as one surgeon said, “We need an army of people of volunteers to get this information from patients.” (H). One surgeon described limitations in not having longitudinal reports (e.g., “The next visit when they come back, I don’t have [the report] with me and we finished talking about it, so there’s no way for me to kind of check back on it and see if they’ve addressed these things or not.” H). When asked to reflect on expanding use of PROM assessment in practice, surgeons raised concerns about PROM assessment on all patients slowing down clinic workflow, stressing the need for patients to complete PROMs before coming into the clinic. However, this is difficult, as some patients decline (e.g., “There are some patients who refuse to fill out the tablet if they’re there, or some who never filled it out at home…Some people say no to that.” D). They also pointed out the burden of accessing a report for all patients (e.g., “It’s going to be a little complicated to try and keep pulling this form out for every patient because on an average, if I see maybe somewhere like 40 to maybe 50 on a bad day, it would be hard to pull this up.” H).

Four surgeons expressed concern about the burden on patients to complete PROMs. This included concerns for multiple requests for different goals (e.g., “They’re normally bombarded by a lot of things, they’re bombarded by PROMs, they’re bombarded by questionnaires and things like that before even coming into the office when I see them.” D) and having those requests be unclear (e.g., “I still had, over the past couple of weeks, two or three patients coming and asking me that, ‘Did you want me to participate in this?’” H).

Another challenge was limitations for tailoring report content to a particular patient. One surgeon commented that he may weigh the use of narcotics differently for a patient with sickle cell disease or HIV, but the lack of that specific comorbidity information in the report limited the ability to fully tailor the report to each patient. Additionally, when the report was generated for a patient whose presenting issue was not related to OA, predicting outcomes post-surgery is no longer useful or relevant (e.g., “I think the problem is that the data is designed for people with hip arthritis. When you capture people that don’t have hip arthritis, the data doesn’t work correctly.” C).

Other challenges included times when a patient was seen for one joint (i.e., left hip), but the contralateral side (i.e., right hip) indicated worse pain leading to concern about underlying reason for lack of concordance in report information and patient report. In addition, the lack of standardization in what PROMs are used in clinical care was a noted challenge for efficiency in data capture and ability for scalability. Surgeons expressed concerns about the expected increased mandates for capturing PROMs for billing requirements, and one described a divide within the field about whether or not PROMs are sufficiently useful to warrant implementation costs (e.g., “There are still leaders in the field that are arguing that it’s a complete waste of time.” I).

Surgeons identified modifications to improve interpretability, such as modifying figure labels, increasing visibility of specific information, use of consistent PROM scales (e.g., all 0–100), and supporting printing in black/white. Other recommendations focused on adding additional medical information such as x-ray grade, diagnosis, and hemoglobin A1c. An interest existed for displaying the timeline for improvement (e.g., adding expected outcomes at 6 months) and creating longitudinal reports showing patient PROM scores over time. Finally, there were suggestions to expand information on non-surgical treatment options with predictive analytics indicating expected outcomes and duration of treatment impact.

Most surgeons (78%) had used PROMs in clinic prior to the study. However, when asked about training needs, surgeons suggested additional training beyond the study-provided standard, one-time training on the value of engaging patients in treatment decisions, as well as the content and use of the information in the report. Surgeons suggested training on how the report can help them, how to use it with a patient, and specific score interpretation guidance. They suggested training should be brief (e.g., 10–15 min) and utilize a case-based approach (e.g., trainer demonstrating report use with a hypothetical patient).