LLM Diagnostic Accuracy, State of ASC Labor Costs

Erik Sunset: [00:00:00] Hey everybody. Thanks for listening. Thanks for watching from wherever you are. I’m Erik sunset, your host to the DocBuddy journal. This is episode 48. It’s being recorded on Thursday, January 11th. And whether this is your first time seeing the show, or you’re a regular listener, be sure that you’re subscribed on apple podcasts, Spotify, or YouTube ad, to be sure that you get the newest episodes of the doc Bundy journal as they are released. This is our first show being recorded in 2024.

Yep. We’ll pull back the curtain on our production schedule for you there. Last week’s rate episode with Lisa blue was recorded just before going into that week of Christmas. And we published it at the beginning of the month to be sure that we were continuing the feed of the doc, but a journal for ya. We’re also going to be recording another episode with a guest later this week, which I think you’ll find very interesting.

We’ve actually got nearly a handful of [00:01:00] guests will be reporting episodes with. Next week to get your really diversified look and view into health. It from those who are experts on the front lines. But for this week, it’s just me. I’m all you got this time. Hopefully it didn’t scare you off. If it did though be sure to take a look at our backlog of episodes, which you can get to at docbuddy.com. Slash blog. And if there is anything else we can do for you in terms of your physician workflow solutions, being able to reduce the amount of burden that your providers undergo to generate documentation, whether they’re at the clinic, at the surgery center or at any place of care in between. You can also find out more about that on doc, buddy.com. So let’s get into it. It’s been sort of a quiet month, at least from us here.

And the doc put a journal talking about. AI and talking about large language models or LLMs in healthcare, the most [00:02:00] famous LLM of course, being chat GPT. But our friends at JAMA J. J a M a released a new reports. And it was titled diagnostic accuracy of enlarge language model in pediatric case studies. And depending on which side of the line that you fall on, this is either going to be surprising results or not surprising results at all. So we’ll start with that.

Heading from the reports. We’ll cite directly from it and provide some commentary as well. Hopefully I’ll be able to make that clear for you. The chat bot had a diagnostic error rates of 83%. So that’s 83 out of a hundred. In this case among the incorrect diagnoses. 72 out of a hundred were incorrect and 11% were clinically related, but too broad to be considered a correct diagnosis. Most of the incorrect diagnoses generated by the chat bot belong to the same organ system as the correct diagnosis.

So this was 47 out of the [00:03:00] 83 that it got wrong. And for instance, psoriasis and seborrheic dermatitis. Close, but not the same. But there were not specific enough to be considered. Correct. In the incidence of hypoparathyroidism and hungry bone syndrome. Additionally 36% of the final case report diagnoses were included in the LLM generated differential list.

So despite the high error rate of the chat bot physicians. Can and should continue to investigate the applications of Pell labs to medicine. LLMs and chatbots have potential as an administrative tool for physicians demonstrating proficiency in writing research articles and generating patient instructions. And as we all know, Certain electronic health records like epic have collaborated with Microsoft’s open AI to incorporate GPT four with the goal of automating components of clinical documentation from within their EHR. [00:04:00] However, this is on the flip side, coming from the authors of the study.

However, the underwhelming diagnostic performance of chatbots observed in this particular study underscores the invaluable role that clinical experience holds. The chat bot evaluated in this study. Unlike physicians. I was not able to identify some relationships such as that between autism and vitamin deficiencies. And I’m going to paraphrase a little bit here to improve the generative AI chatbots, diagnostic accuracy, more selective training is likely required. Yeah. You think because Chad GBT is generalized LLM, it’s trained across all different media and content found across the web.

It’s not being trained on medical textbooks on medical encyclopedias, something though, like Google’s med pal. To have specifically been trained on medical data. And you’d expect it to perform better. Then Chad GPT in this particular study with [00:05:00] an incorrect. Or better put an error rate for diagnostics at 83 out of a hundred. So stated the other way.

It only got 17, right out of a hundred. That’s an F F minus minus, minus, minus, minus. So not even close. Is that really a surprise to those that are paying attention? No, because Chad GBT is a generalized word predictor, and I made a little tongue in cheek here. If you haven’t listened to prior episodes of the DocBuddy journal, or you’re not. Up to speed on how these large language models work. You initiate a prompt, you ask for something. Or like in this case to diagnose an issue based on a set of observations, And then the machine uses a word prediction. Model word prediction algorithm to give you what looks like a complete thought, but it’s doing just that.

It’s a next word in a sentence predictor at its core. And it, you know what that sounds derogatory. But that’s the reality of how these softwares are [00:06:00] built and how they work. It’s giving you the most likely next word in the sentence. A lot of times it works out really well. Just obviously not in this case. We’re chatting GPT.

Isn’t trained on pediatric diagnoses and pediatric problems while on us. To get more than 17 out of a hundred. Great. So to put a bow on this topic before we move on to the next one. The states, the obvious for me, a lot of times, Chad GPT and LLMs are a solution in search of a problem. There’s no replacement for providers, medical decision-making and experienced at least at this juncture.

And if we ever get to a juncture where. A software can do a good job providing with diagnose diagnostic accuracy. Excuse me. I’d still want a provider in place to be sure that my inputs were correct. From which we’re deriving that particular output, like a diagnosis.

So a big thank you to JAMA that is sort of an [00:07:00] ungated article on their site. We’ll be sure to provide a link to it in the show notes so that you’re able to take a look at it for yourself and see what you think. Our second topic of the day comes from our very good friends at Becker’s. We love the work they do over there. And this particular article the state of labor costs, three stats to know from Becker’s asc.com. The number one stat.

If you work at an ASC or associated with an ASC or you own an ASC, you’re going to know medical groups are spending more per physician in 20, 23 than previous years. A recent report by the American medical group association found that the medium expense per physician in 2023 increased from $905,000 and change, we’re going to round it down, men. Five grand. I do over a million dollars at 1,000,030 6,000. And this is despite a slight uptake in revenue compared to the previous year. And the median loss faced by medical groups per physician was just shy of a [00:08:00] quarter million dollars. The AMDA attributed the increase to issues with ongoing labor shortages. Hold that thought we’re going to jump right into that. But one quick stat first compared to pre pandemic levels, median, total revenue per position increased 9.1%.

While median total expense per physician increased over 26% over the same period.

An ongoing labor shortage. Yeah, that’d be putting it very, very mildly. Not only are we looking at a labor shortage for those in nonclinical roles in healthcare facilities. From what we hear and from what we’re seeing firsthand, it is very difficult to staff, to adequate levels from everything from front desk, surgical techs. Physicians and mid-levels included here as well. What, what we’re really talking about with this ongoing labor shortage is the slow motion cataclysm of just simply not having enough physicians in this country. We’ve covered it [00:09:00] before. But in the next five years, we’re going to be down. A significant amount of physicians that we. Expect to have per capita.

And this issue is. It’s twofold. Number one, physicians are retiring early. They’re feeling the pressure of their career, the pressure of the bad health. It that they’re forced to use. They’re just simply leaving the profession and none of the front end of our physician supply chain. We hardly have anybody applying and going through medical school to want to become a physician. You have big issues here? And, you know, when you look at some of these numbers, That the spend per physician is going up or meeting an expense per physician is going up. You’re going to see that number just as A matter of the statistics continue to go up as more and more physicians leave the workforce and they’re not replaced with those graduating from medical school. Number two of three stats from Becker’s ASC. The cost of practicing medicine is [00:10:00] rising inflation specifically in healthcare continues to grow. The Medicare economic index, which major is medical practice cost inflation increased 4.6% in 2023. The highest in the last 23 years. Despite this CMS has finalized fee schedule reduced overall physician pay by one and a quarter points and updated the conversion factor to $32 and 74 cents. The 3.4% decrease over the last year. What else can you say here?

We’re, we’re getting squeezed. Our wallets are getting pinched in at every turn at every corner. But some of these stats are interesting to keep an eye on the cost of practicing medicine is rising. Medical practice costs, inflation increased almost five points in 2023.

Not great. Something’s gotta be done. And then finally, the third point from Becker’s ASC and this particular article, some ASTs have had to spend one [00:11:00] quarter or more of their net operating revenue on employees to stay ahead of shortages. The average percentage of operating revenue ASC spent on paying employees was 21.3%.

According to the report from BMG health. However, the highest vendors paid 29% of their earnings to compensate staff.

No man, really those facts and figures kind of tell you everything that you need to know. It is really tough to hire good help. I mean, it was tough to find good help before. COVID obviously it’s always tough to find good help. But now it’s really tough to find it. To find even. S like serviceable help and then to go a step further as even hang on to it. The average percentage of operator of new was 21.3.

And then at the top end, almost a third. Of your operating revenue being spent on staffing.

Food for thought there, like I said, we’re looking forward to bringing you a great slate of guests through this week and the next, and then into the future as [00:12:00] well. We also look forward to bringing you the highest quality. Findings and insights for around the health. It arena, particularly around physician workflow, particularly around burnout.

So on behalf of the entire doc, buddy team, we want to thank you for listening. Be sure that you’re subscribed on apple podcast, Spotify and YouTube. And. If you’re interested in taking a look at any of doc equity solutions, like op notes, like enterprise, like on-call like on-call calendar, like doc, buddy. Voice. Even. Drop a sign over a dock, buddy.com until next time.

I’m your host, Erik sunset. We’ll talk to you again soon.