Scale and Singapore Launch Benchmark for AI Legal Reasoning

Scale AI and Singapore's Infocomm Media Development Authority have introduced SG-LegalClarify, a benchmark testing how well AI models reason about Singapore law and recognize when they need more information. The benchmark evaluates 19 models across 1,860 legal scenarios in 10 areas of Singaporean law.
- SG-LegalClarify tests 19 AI models across 1,860 Singapore legal scenarios.
- The benchmark covers 10 legal domains and includes incomplete-information cases.
- With web search, the highest-performing model satisfied 77.8% of mandatory criteria.
- Models reached the correct conclusion in 55.6% of trials after asking all relevant questions.
- Scale plans to release a public subset for research and independent analysis.
Testing legal reasoning beyond simple answers
SG-LegalClarify was developed as the first research initiative under Scale's collaboration with IMDA. It focuses on a problem that standard legal evaluations may miss: real users often provide incomplete information, so an AI assistant must identify relevant facts and ask questions before giving an answer.
The benchmark includes 1,100 original questions written by qualified legal professionals with expertise in Singapore law. The questions cover areas including criminal law, employment, property, family law and judicial processes. Of the questions, 760 deliberately leave out information needed to determine the correct answer. Each of those is paired with a version containing the missing facts, producing 1,860 evaluation scenarios.
Models can ask a simulated user for missing details. The benchmark then compares their performance when facts must be gathered with their performance when the same information is supplied from the start.
Web search improved results, but gaps remained
A three-model AI judge panel, selected through comparison with human expert assessments, evaluates responses against 12,546 expert-written grading criteria. The criteria measure issue identification, rule recall, application of the law to specific facts and the final conclusion.
According to Scale, without web search the highest-performing model satisfied 69.8% of mandatory grading criteria, while the average across all 19 models was 53.8%. With web search, the average rose to 63.0% and the highest-performing model reached 77.8%.
Rule recall was a particular weakness. Even with web access, models satisfied only 56.5% of applicable rule-recall criteria. Scale said models also failed to ask clarifying questions in approximately 39% of scenarios with missing facts. They asked unnecessary questions in approximately 28% of scenarios where enough information was already available, and asked about an average of 39% of the relevant facts identified by legal experts.
Asking questions did not ensure a correct result. In web search-enabled trials involving missing facts, models that asked about every relevant missing fact reached the correct legal conclusion in 55.6% of cases. They reached an incorrect conclusion in 32.1% and provided no conclusion in 12.3%.
Why local AI evaluation matters
Scale said Singapore provides a distinct test because its legal system draws on English common law while incorporating Singapore-specific statutes and principles. A model familiar with general common-law concepts may still produce an incorrect answer if it misses where Singapore law differs.
The company said SG-LegalClarify shows why governments need ways to independently assess AI systems against their own laws, institutions and standards. The benchmark is the first result of the collaboration announced in July 2026, which also includes work being explored in agentic AI, model evaluation methods and AI safety.
Scale plans to release a public subset of the benchmark for further research and independent analysis. Although SG-LegalClarify is focused on Singapore law, Scale said its approach could offer a model for other countries evaluating AI systems against national requirements.
Scale AI: Scale and IMDA Launch First AI Benchmark Under MOU


