There is much research into the causes of generative model hallucinations. But in fact, generative ANNs based on statistical learning have fundamental limitations that lead to hallucinations. The first limitation is related to the phenomenon of the entropy gap (the probabilistic gap between accuracy and creativity), which is based on Shannon's information entropy. Low entropy means that the distribution is concentrated on frequent patterns, which gives high accuracy but little novelty. High entropy expands the inference space, generating creative, but often false or hallucinatory answers. In addition, any statistical model, seeking to minimize the average prediction error, inevitably runs into a limit set by the entropy of the real language or actual data. The second limitation is related to Gödel's incompleteness theorem, which states that any sufficiently powerful formal system capable of expressing the arithmetic of natural numbers (and therefore more complex systems, including AI, working with formalized knowledge) cannot be both complete and consistent. And although generative models do not operate with proofs in the sense of first-order logic, as soon as we try to implement a built-in validation of its own conclusions into the system, especially concerning statements about the world or other knowledge, the question arises: can the system verify the truth of its own statements using only its own internal apparatus? And Gödel's theorem says no, it cannot. This means that the model will either leave the statement unverified, or it will be necessary to go beyond the model to verify it (human supervision, external knowledge bases, evidence systems, etc.). Thus: the entropy gap prevents novelty and reliability from being combined, and the incompleteness of formal systems does not allow a reliable mechanism for complete self-verification to be built into any generative system. That is, even the most theoretically advanced language model cannot guarantee that it can validate any of its outputs within its own rules, probabilities, and learning space. But what if we somehow create some kind of multi-agent symbiosis to generate verified data? We will again get a closed system and the only way out of this circle is self-learning. What do you think about this? More http://dx.doi.org/10.13140/RG.2.2.31172.74885
Dear Yurii Parzhyn ,
AI hallucinations are incorrect or misleading results that AI models generate. These errors can be caused by a variety of factors, including insufficient training data, incorrect assumptions made by the model, or biases in the data used to train the model. AI hallucinations can be a problem for AI systems that are used to make important decisions, such as medical diagnoses or financial trading.
https://cloud.google.com/discover/what-are-ai-hallucinations
Regards,
Shafagat
По вашей тематике может возможно ограничить неблагоприятные эффекты, исследовав функцию негэнтропии, как показателя полезности и надёжности для результата. Также предлагается ограничить повышение энтропии подходящими критериями, информацией по Шеннону (т.е. для снятия неопределённости при получении результата). Моё предложение основано на том, что я у себя рассматриваю вопросы энтропии-информации-негэнтропии в контексте информации об уровне безопасности в технической системе и тут, по моему, большую роль играет не энтропия, а негэнтропия и информация.
Indeed, there are many reasons why generative models hallucinate. However, I am talking about fundamental limitations that cannot be addressed within existing learning paradigms.
Well, future is unpredictable. There are many advancements in the fields of AI till now which pave a path to have AI-models free from hallucinations. However, there may be various reasons like inadequate training of model which cause inappropriate results and hallucinations. So, it still can take time to free AI from any errors or hallucinations.
Hallucinations seem to come from fundamental limits in how these models work, not just data issues. The balance between accuracy and creativity creates a natural tension, and models can't fully verify their own outputs without external help. Even with advanced systems, some form of grounding or oversight might always be necessary
Hello Yurii Parzhyn
You’ve raised significant points about the limitations of generative models, particularly the entropy gap and Gödel’s incompleteness theorem, both of which are foundational challenges.
Entropy Gap and Trade-off between Accuracy and Creativity
The entropy gap is indeed a key factor. Generative models, when tuned for accuracy, concentrate on frequent patterns, leading to high precision but low novelty. On the flip side, boosting entropy to generate creative responses results in a wider inference space, producing more diverse but often inaccurate (hallucinatory) outputs. This trade-off is inherent in statistical learning models, where balancing novelty and accuracy remains a major challenge.
Gödel’s Incompleteness Theorem and Self-Validation
Gödel’s incompleteness theorem highlights a critical issue: any sufficiently powerful system cannot be both complete and consistent. While generative models don’t operate like formal proof systems, when we attempt internal self-validation, we face the problem that these models cannot verify the truth of their own statements using only internal mechanisms. They depend on external validation (e.g., human oversight or external knowledge bases) to ensure factual accuracy.
Multi-Agent Systems and Self-Learning
The idea of multi-agent symbiosis could help cross-check generated outputs, but it risks creating a closed system with the same validation issue. Self-learning, in theory, could improve validation, but it still requires external sources to ensure truth, as the model cannot independently verify its outputs without a broader knowledge framework.
Conclusion
The limitations you outlined—entropy-based trade-offs and Gödel’s incompleteness—are fundamental. Generative models can't self-validate without external inputs, and novelty comes at the cost of reliability. The future lies in combining AI with external verification and human oversight, or perhaps evolving meta-learning techniques to address these gaps.
Dear Anil Shukla,
I can't help but agree with you (or ChatGTP :)). Indeed, self-learning based solely on statistical methods is unlikely to lead to a significant breakthrough. Yann LeCun also constantly talks about the problems of statistical learning. He also says that learning based on considering the internal energy of the system can be a solution. The well-known Hopfield neural network model also points to the prospects of this direction, although it also has many limitations. You can read more about this issue in the study https://doi.org/10.48550/arXiv.2503.21794
Can researchers stop AI making up citations?
OpenAI’s GPT-5 hallucinates less than previous models do, but cutting hallucination completely might prove impossible...
"Eliminating hallucinations entirely is likely to prove impossible, says Mushtaq Bilal, a researcher at Silvi, a Copenhagen-based firm that makes an AI app to aid the creation of systematic reviews in science. “I think if it was possible, AI labs would have done it already.”..."
https://www.nature.com/articles/d41586-025-02853-8
- Badges
- Science topic