Microsoft Research and Georgia Tech Researchers Reveal Statistical Limits of Hallucinations in Language Models

A key problem that has recently emerged in language models is the high rate at which language models (LMs) provide misinformation, including references to non-existent article titles. The Merriam-Webster dictionary defines a hallucination as “a plausible but false or misleading response generated by an artificial intelligence algorithm.” In one case, lawyers who submitted legal research with imaginary court cases they thought were accurate faced a $5,000 fine. In the medical field, patients' hallucinations can be fatal and doctors fear being sued for negligence. Additionally, hallucinations have been widely covered in the media and the President of the United States recently issued an Executive Order requesting, among other things, protection from misleading results from generative artificial intelligence systems.

In this work, researchers from Microsoft Research and Georgia tech present statistical lower bounds on the hallucination rate for learning machines (LMs) that are calibrated event predictors. This sheds light on the characteristics of hallucinations. This does not imply that hallucinations are inevitable. As the research team will discuss, it is more in line with the growing trend of professionals to supplement “pre-training” procedures with “post-training” procedures that reduce hallucination rates and calibration. An LM is simply a probability distribution D over sequences of tokens, that is, words or other sequences of characters. Any LM that predicts each chain with positive probability (a typical characteristic of LMs) will necessarily hallucinate with positive probability. However, hallucinations will be rare if this probability is low. Therefore, measuring the frequency of hallucinations is essential.

The log probabilities between complete sequences or the conditional log probabilities of the next token given the previous ones can be used to express any distribution D identically: log D(t₁…t_meter) = Pm i=1 log D(t_Yo | t₁ …t_i-1). This seemingly insignificant mathematical equivalence has an important implication. Although prediction and generation have different requirements, any LM can be used to produce text or predict the next token in natural text conditioned on previous tokens. Take the following sentence, for example, Alexa Wilkins went to Salumeria for lunch last Tuesday because the reviews said the tuna sandwich was amazing. A predictive language model could suggest sentences like this to reduce typing on the phone. It may be beneficial to predict sandwich as a word to enter after the term tuna, along with other plausible words such as salad.

However, it would be false if a generative LM fabricated the vast majority of these types of sentences at random. According to this article, even under perfect circumstances, LMs with strong text prediction abilities should experience hallucinations. In particular, in the initial pre-training step, which is typical today, generative LM is tailored for predictive text performance. Furthermore, it provides a lower bound on the rate of hallucination, which could shed light on the different rates at which different types of events must be hallucinated. Both the previous example and the possible references (which the research team will refer to as 5W factoids = Who-ate-what-when-where-why) have in common that they are arbitrary in the sense that neither can be determined methodically by rules. ; That is, most of these facts cannot be verified because they are not included in the training data.

Unlike facts, the validity of which can be methodically verified. Even in a simplified situation with many ideal qualities, the research team estimates the number of hallucinations LMs should experience. The research team prefers simplicity over generality, as their lower bounds are statistical and their goal is to identify the underlying source of LM hallucinations. The research team seeks a hallucinatory lower bound that holds in the simplest context when the training data is iid without factual errors, similar to classification, where a lower bound for classification difficulty in noise-free environments is sought (although noise-tolerant classification techniques).

The research team offers a natural extension of calibration to generative models. Their idea is different from previous calibration applications in LM, which were at the token level. Since each fact can be described using natural language in several ways, calibrating token probabilities is only useful when evaluating raw token probabilities. Rather, the probability distribution between bits of information (facts or hallucinations) in the text is considered by calibrating it at the semantic level. A LM is considered calibrated if, among the information it creates with probability a ≈ z, for any given probability z ∈ (0, 1), such information appears on average in a fraction of the natural language with probability a ≈ z (preferably the distribution from which the training data was collected).

This work aims to explain this phenomenon by demonstrating that, even in an ideal world where the training data is perfectly factual, there are no blurry facts or hallucinations, each document contains at most one fact and there is not even a message that can explain it. Encouraging hallucinations, pre-training LMs for predictive accuracy results in hallucinations. Furthermore, their hypothesis clarifies why contemporary LMs have greater hallucinations than earlier LMs, such as trigram models, despite being trained on comparable data sets with comparable targets. The monoact rate can show the rates at which calibrated LMs must deceive themselves about various types of facts.

When events occur with a high monoevent rate, that is, events that frequently appear only once in the training data, hallucinations are predicted. It is interesting to note that this is rare in allusions to books or articles, a problematic type of hallucination that is now being studied. Therefore, examining the large number of facts, including references and others, that a LM encounters during training may be due to other problems such as model capacity. Additionally, it might be possible to correct for hallucinated references by altering the pre-training process without using post-training, but this will not help with other types of arbitrary facts, such as those in the 5W example, where monofacts are common.

Review the Paper. All credit for this research goes to the researchers of this project. Also, don't forget to join. our 33k+ ML SubReddit, 41k+ Facebook community, Discord channel, and Electronic newsletterwhere we share the latest news on ai research, interesting ai projects and more.

If you like our work, you'll love our newsletter.

<!– ai CONTENT END 2 –>