New research examines how “hallucinations”: confident but incorrect or unsupported answers, can be identified in AI tools used for legal question answering. The study presents a new method (“LEGALscore”) that uses signals from within the AI system itself to flag when an answer is likely to be fabricated or not properly grounded in legal source material
The researchers tested the approach on a dataset of real-world contract questions derived from the Contract Understanding Atticus Dataset (CUAD), and across several widely used open-source AI models, including LLaMA, Qwen, and Mistral. They find that combining multiple internal indicators improves the ability to distinguish reliable legal answers from “hallucinated” ones, compared with relying on one method.
Because the method does not depend on searching external databases, it can be applied quickly and integrated into interactive legal tools. The authors caution, however, that some errors such as plausible but incorrect figures or subtle misinterpretations remain difficult to detect automatically, reinforcing the need for human review and professional judgement when AI is used in legal work.
For legal regulators, this research provides empirical evidence and a technical benchmark for evaluating and standardising the reliability and risk controls of AI systems used in legal services, informing future regulatory guidance, certification and compliance requirements.