AI glossary · Basics

Hallucination

Also called: AI hallucination · Made-up AI answer

What is a hallucination?

Anthropic, the company behind Claude, describes hallucinations as text that is factually wrong or does not match the given context. Even top models produce them.

A wrong answer sounds as confident as a right one, with no hesitation and no warning. Typical examples: invented figures, legal sections, sources, product names or opening hours.

How does it work?

A language model learned in training to predict the next word. It writes what is likely to fit; it has no reference book. If information is missing, it writes on anyway: plausible, but wrong.

It cannot be prevented completely, only contained:

  • Allow “I don't know”: Anthropic recommends explicitly letting the AI admit uncertainty.
  • Quotes first: with long documents, have it pull out exact quotes, then build on them.
  • Add your own knowledge: with RAG, the system finds matching passages in your documents and passes them on.
  • Ask for sources: every claim with a reference. Anything unreferenced counts as unchecked.
  • A person checks: someone reads whatever goes out (human in the loop).

Why does it matter for your business?

Passing AI answers to customers unchecked puts the risk on you. A wrong delivery date in an email or an invented legal section in a quote is enough for trouble.

A website chatbot can end up in court. On 12 May 2026 (case 4 UKl 3/25), the Higher Regional Court of Hamm in Germany ruled on a chatbot that named invented medical specialist titles, and held the operator responsible for its statements. More in the German guide on chatbot liability (in German).

How does OsAI use it?

OsAI builds bots and workflows that answer from stored knowledge, for example from project files via RAG. Customer emails only go out once a person has approved them. If a detail is missing, the AI should say so rather than guess.

Spotting hallucinations and checking results is part of the AI training under Art. 4 of the EU AI Act: half a day for up to 10 people, €890.

Frequently asked questions

Why does ChatGPT make up sources?

Because a language model predicts text, and to the model a citation is a text pattern like any other. Without web search or stored documents, it does not check whether a source exists. Open every link yourself first.

Can hallucinations be prevented completely?

No, only made less frequent. What helps: your own knowledge via RAG, permission to say “I don't know”, and a person checking everything that goes to customers.

Does RAG help against hallucinations?

Yes, it makes them rarer, but not impossible. The AI receives matching passages from your documents and can cite them. According to Anthropic, how well this works depends on the quality of that knowledge base: outdated documents produce outdated answers.

Sources

Last updated: