Imagine going to the doctor, being told exactly how you feel, and then a transcript adds false information and alters your story. That could be the case for medical centers that use Whisper, OpenAI's transcription tool. More than a dozen developers, software engineers, and academic researchers have found evidence that Whisper creates hallucinations (made-up text) including made-up medications, racial comments, and violent comments. ai-powered-transcription-tool-hospitals-invents-things-115170291″ rel=”nofollow noopener” target=”_blank” data-ylk=”slk:ABC News reports;cpos:2;pos:1;elm:context_link;itc:0;sec:content-canvas” class=”link “>ABC News information. However, in the last month, open source ai platform HuggingFace has recorded 4.2 million downloads of the latest version of Whisper. The tool is also integrated into Oracle and Microsoft cloud computing platforms, along with some versions of ChatGPT.
The damaging evidence is quite extensive, and experts find major flaws in Whisper across the board. Take a University of Michigan researcher who found fabricated text in eight out of ten audio transcripts of public meetings. In another study, computer scientists found 187 hallucinations when analyzing more than 13,000 audio recordings. The trend continues: A machine learning engineer found them in about half of more than 100 hours of transcripts, while a developer detected hallucinations in almost all of the 26,000 transcripts he had Whisper create.
The potential danger becomes even clearer when looking at specific examples of these hallucinations. Two professors, Allison Koenecke and Mona Sloane of Cornell University and the University of Virginia, respectively, looked at clips from a research repository called TalkBank. The couple discovered that almost 40 percent of hallucinations had the potential to be misinterpreted or misrepresented. In one case, Whisper made up that three of the people discussed were black. In another, Whisper changed: “He, the boy, was going, I'm not sure exactly, to take the umbrella.” to “He took a big piece of a cross, a small, very small piece… I'm sure he didn't have a terror knife, so he killed several people.”
Whisper's hallucinations also have risky medical implications. a company called Nabla uses Whisper for its medical transcription tool, used by more than 30,000 physicians and 40 health systems, transcribing approximately seven million visits so far. Although the company is aware of the problem and says it is fixing it, there is currently no way to check the validity of the transcripts. The tool erases all audio for “data security reasons,” according to Nabla CTO Martin Raison. The company also says that providers need to quickly edit and approve transcripts (with all the extra time doctors have?), but that this system can change. Meanwhile, no one else can confirm that the transcripts are accurate due to privacy laws.
