Error by AI scribe during medical appointment leaves patient devastated
doctor forced to apologise after ai makes 'scary' error about illegal drugs
artificial intelligence is listening to many people's medical appointments, but it is not always hearing things correctly.
Show me one single AI user that checks the output of their AI, and I will show you my unicorn
Not gunna get me with the ole “wanna see a unicorn” trick twice.
I just asked SlopGPT a question and checked the output. Can I see your unicorn?
(That was a lie. I just really want to see a unicorn.)
en.wiktionary.org/wiki/toupee_fallacy
Thank you.
As far as I’m concerned, using AI of any sort inside of a doctor’s office is a grave violation of HIPAA
Let me tell you about how medical transcriptionists used to do it: Sister in law was one, she didn’t seem to care about HIPAA too much, she and all her colleagues did lose their jobs about a 15-20 years ago when AI transcription software took over their jobs, but before that she was frequently given jobs because her predecessors made serious mistakes in their transcriptions.
If she didn’t care about HIPAA, I’m glad she lost the job lol.
If only we could fire AI too.
She’d tell stories (without the names)- many of her colleagues would get deep into the personal details…
I don’t think that’s a hipaa violation, but I’m no expert.
EMTs, etc, do that too. You’d go crazy if you couldn’t.
I work in the digital handling of PHI (HIPAA regulated) it’s pretty insane what is considered HIPAA protected in that world.
In practice, yeah, real people can’t just be all "oh, that’s a national secret, I can’t tell you anything about my work or the weird worms people had in weirder places…
Ai transcription is everywhere. You cant get away from it
Yeah? Watch me.
Watch me and millions like me push back everywhere and slowly change the world.
TL;DR: AI transcript said I’d been a smoker since the age of 11, never smoked a day in my life.
My doctor asked me if she could use AI transcription software because she had hand surgery and couldn’t type. I said yes and reviewed the statements and while it was fairly accurate, it did have one egregious error on it.
When asked if I had ever smoked, in person, I replied with a simple “no,” and did not proceed to have any follow up conversation until her next question. The AI transcript for mysterious reasons decided my answer was “yes, since the age of 11.” 💀💀💀
Had to message them to fix it and while they did fix it asap, my first instinct wasn’t to check the transcript and I only did so out of curiosity. I can’t imagine what other people’s experiences may be.
Article:
Just say no.
We use a software to help us take notes is that okay? FUCK NO.
The patient was devastated? You should have seen how many times the AI apologized.
I’ve used AI for medical transcription before and it is absolutely incredible. It’s not just a word for word text, but a detailed clinical breakdown of everything discussed.
All you have to do is simply read the damn thing thoroughly and correct whatever minor errors pop up.
Takes 95% less time.
I’m unsure if there’s an ethical issue with putting patients medical records in the data machine.
There exists online and offline ML.
Transcription is not a task that needs a full blown LLM, summarization probably does, and if being done offline or on a server with trusted verifiable lack of logging should be fine. That does not mean send it to ChatGPT API.
I would hope they store the raw conversation file encrypted with patient records so they can retranscribe later as tools improve.
It needs to be a system and a contractual framework that complies with the medical privacy regulations of the place where it’s being used (in the US, HIPAA). I have some doubts about whether specific companies or people will actually comply, or the enforcement of those laws by certain governmental authorities, but I don’t think it’s conceptually impossible to have this kind of tech be compliant with those types of regulations.
Medical AI has guardrails out the wazoo. By the time it’s made available to a physician, it’s probably been reviewed by a council, vetted by IT, and negotiated heavily by Legal. If the data leaves the premises, it’ll be thoroughly protected and isolated.
That said, fuck AI. I’d give up medical transcription to get the environment, economy, and sanity back. Musk’s data center is running dozens of natural gas turbines in a low-income, primarily Black neighborhood 11 miles down the road from St. Jude Children’s Research Hospital. It’s positively ghoulish.
Horseshit. Medical AI is using Oracle services. Larry mother fucking Elison is storing and processing this data.
The medical industry seems so ignorant about this. They think the argument is just about AI. It’s also about the fact that I don’t trust the people you’re partnered with.
I get so pissed when the AI consent is baked into other paperwork with no opt out. I will leave.
Edit: www.oracle.com/customers/beacon-health/
Edit: they might not all be, but I’m not authorizing your arbitrary “AI use” clause. You might as well be, now or at some point down the road.
One Rich Asshole Called Larry Elison
The agreement is only as good as the hospital’s vetting, so I may be a little optimistic after seeing a very thorough and protracted process on my side. If you work for a healthcare system that’s okay with Beacon and Oracle, then it may be time to evaluate whether they’re a good healthcare system to work for.
Clue: Medical transcription has been done by “AI” for the past 20 years. Source: sister in law was a medical transcriptionist making $80K per year, until Dragon Naturally Speaking and friends decimated her job market.
All you have to do is fire the human who is trained and rarely makes a mistake and replace them with a paid machine that almost always makes a mistake and double check everything yourself (or hire another human to double check it) because someone’s life literally depends on the entirety of the text being absolutely correct.
AI does work at a level that makes unpaid interns look like career professionals. All for the low cost of completely fucking the environment.
Glad you like it.
Exactly. It is a tool for professionals to use to speed up existing processes. It is not a replacement for people, regardless of what the marketers tell you so you buy their product. I don’t understand how there aren’t thousands of lawsuits against these AI companies for their blatantly false advertising.
AI keeps proving this day by day when people try to make it a replacement and have it blow up in their faces every day.
SEC is neutered, what makes you think anyone in charge of advertising rules hasn’t been replaced by a 17 year old called Goebbels Gobbler or something with a DOGE laptop.
This article is about an incident in Australia. The SEC has no bearing on what Australia can do.
The AI bubble is an American phenomenon and their SEC isn’t slowing down the insanity. Their lack of a functional SEC is everyones problem.
Hell, I just found out a few months ago my father’s office (he’s a lawyer) still did transcriptions of court proceedings manually! Full blown lawyers wasting expensive man hours doing that. As soon as I found out I got some random whisper cpp frontend and slapped it on a random computer
As long as they still have someone verifying and correcting mistakes, that’s a decent solution. It doesn’t need to be complicated. Whisper is by no means perfect, but from my experience it handles 90% fine, only having to listen and correct 10% is a huge improvement over transcribing everything manually.
The issues with so much of the current AI use is implementations that completely remove trained human oversight from the equation because the AI companies position it as a complete replacement for employees.
I’m curious how this is supposed to work? How do you only listen to 10%? How are you supposed to know which 10% to listen to? Wouldn’t you still need to listen to it all to find that 10%?
Yes, but listening to it all once and correcting the mistakes as you hear/see them would be faster than transcribing the entire thing, then proof reading.
I hope his clients are aware
Don’t forget where you are. They kill people around here for that kind of talk.
I read “medieval” appointment and was thoroughly confused, and intrigued.
leeches, blood letting, letting miasma out, holes in the skull, the usual.
[deleted]
Sounds like an AI hallucination
Wow. They are streamlining US “healthcare” by putting the AI claim-denier 5000 INSIDE the exam rooms.
Please just bring back traditional voice to text, with its dumb but decipherable errors instead of these plausible left field LLM “hallucinations”.
I’d rather the computer write that I’m coming in for a “score throat and fever” than the sort of random ass shit LLMs can throw in there.
I’m looking at her and telling you she’s done shrooms in her college days.
Luckily, I know something that would probably help her get over the trauma…
All these articles, nobody posting error budgets or Delta’s with real doctors in this shit.
But we all know doctors are perfect with perfect notes every time and every nurse who gives you a dose 🤷♂️
God forbid that doctor take 2 minutes to confirm accuracy.
Ignoring that this means that one in every ten patients is getting inaccurate notes.
This will literally lead to people dying.
How accurate are the doctors? How accurate are they both in tandem? Are they higher than 90%?
Dual systems working on a decision could be significantly safer.
You can hold a doctor accountable.
Try to hold a machine accountable and you’re met with 🤷🏼.
👌👍🤡
Very much so.
It pleases me to see nerds screech the second anyone challenges their fan fiction going on in their head and the very obvious racism, inconsistent expectations based on class and race, and prejudices they display.
And if I can do it by bullying with the same punching down and judgement based on how they look or interact I’m even more than excited.
This is a stupid “science” and I’m happy I made someone feel bad for latching on and their shitty opinions on how effective they “think” (or don’t) an LLM they haven’t used, trained by doctors, might be.
Ok, so be the change you want to see and you post the data.
[deleted]
Blocked and reported.
Ohhhh no, me calling out your ignorance and heading to a power mod cause it hurts your internal fan fic 🤣
Classic Reddit.
I mean, they should, but they won’t. I’ll take human typos over LLM “hallucinations” most days.
👌👍
But Lemmy hates AI and loves to huff its own farts
It’s a fair point. Unpopular in this community, but fair. Of course, nuance is rarely rewarded in online discussion
“If you’re not with us, you’re against us. Huh. If you’re not an apple, you’re a banana.”
-Terry Pratchett, Thud
I like to eat apples and bananas.
I like to iat ipples and banini’s
Fediverse has turned into the truth social of disenfranchised progressives. Don’t get me wrong I agree with many themes in this place, but the tribalism and unwillingness to refrain from immediately labeling a topic or person a total bad actor is pretty high. Everyone is a mustache twirling billionaire, redneck who hates trans and are Nazis, or a progressive hero.
You have a single example of it going wrong.
Meanwhile, outside of fantasy land, mistake rates are going down as doctors use AI because it’s more accurate than humans. Things are better because technology is better.
I can tell you didn’t read the article if you think it only discusses a single incident. Odd how you can so enthusiastically claim that it’s just overblown nonsense without even reading it. Sounds exactly like the doctor here making claims about something they even didn’t read or verify.
An AI booster is too lazy or stupid to read and understand something they have very strong opinions about? Im shocked.
You’re blindly following propaganda and pretending you’re on the right side of history