Artificial intelligence news.
Straight from the original announcements.

Scale and Singapore Launch Benchmark for AI Legal Reasoning

November 12, 2026By the AI News Bites Editorial TeamSource: Scale AI
Scale and Singapore Launch Benchmark for AI Legal Reasoning
Image: Scale AI

Scale AI and Singapore's Infocomm Media Development Authority have introduced SG-LegalClarify, a benchmark testing how well AI models reason about Singapore law and recognize when they need more information. The benchmark evaluates 19 models across 1,860 legal scenarios in 10 areas of Singaporean law.

Key points
  • SG-LegalClarify tests 19 AI models across 1,860 Singapore legal scenarios.
  • The benchmark covers 10 legal domains and includes incomplete-information cases.
  • With web search, the highest-performing model satisfied 77.8% of mandatory criteria.
  • Models reached the correct conclusion in 55.6% of trials after asking all relevant questions.
  • Scale plans to release a public subset for research and independent analysis.

Testing legal reasoning beyond simple answers

SG-LegalClarify was developed as the first research initiative under Scale's collaboration with IMDA. It focuses on a problem that standard legal evaluations may miss: real users often provide incomplete information, so an AI assistant must identify relevant facts and ask questions before giving an answer.

The benchmark includes 1,100 original questions written by qualified legal professionals with expertise in Singapore law. The questions cover areas including criminal law, employment, property, family law and judicial processes. Of the questions, 760 deliberately leave out information needed to determine the correct answer. Each of those is paired with a version containing the missing facts, producing 1,860 evaluation scenarios.

Models can ask a simulated user for missing details. The benchmark then compares their performance when facts must be gathered with their performance when the same information is supplied from the start.

Web search improved results, but gaps remained

A three-model AI judge panel, selected through comparison with human expert assessments, evaluates responses against 12,546 expert-written grading criteria. The criteria measure issue identification, rule recall, application of the law to specific facts and the final conclusion.

According to Scale, without web search the highest-performing model satisfied 69.8% of mandatory grading criteria, while the average across all 19 models was 53.8%. With web search, the average rose to 63.0% and the highest-performing model reached 77.8%.

Rule recall was a particular weakness. Even with web access, models satisfied only 56.5% of applicable rule-recall criteria. Scale said models also failed to ask clarifying questions in approximately 39% of scenarios with missing facts. They asked unnecessary questions in approximately 28% of scenarios where enough information was already available, and asked about an average of 39% of the relevant facts identified by legal experts.

Asking questions did not ensure a correct result. In web search-enabled trials involving missing facts, models that asked about every relevant missing fact reached the correct legal conclusion in 55.6% of cases. They reached an incorrect conclusion in 32.1% and provided no conclusion in 12.3%.

Why local AI evaluation matters

Scale said Singapore provides a distinct test because its legal system draws on English common law while incorporating Singapore-specific statutes and principles. A model familiar with general common-law concepts may still produce an incorrect answer if it misses where Singapore law differs.

The company said SG-LegalClarify shows why governments need ways to independently assess AI systems against their own laws, institutions and standards. The benchmark is the first result of the collaboration announced in July 2026, which also includes work being explored in agentic AI, model evaluation methods and AI safety.

Scale plans to release a public subset of the benchmark for further research and independent analysis. Although SG-LegalClarify is focused on Singapore law, Scale said its approach could offer a model for other countries evaluating AI systems against national requirements.

Scale AIInfocomm Media Development AuthoritySG-LegalClarifybenchmarksresearch findingsgovernment and AI

Related news

Study finds $360 billion AI gains gap among Commonwealth nations
ResearchNovember 5, 2026, 07:00

Study finds $360 billion AI gains gap among Commonwealth nations

A new study from the Ayre Think Tank calculates a $360 billion difference in potential economic gains from AI between the Commonwealth countries best prepared for the…

Scale launches free 12-hour AI course and microcredential for K-12 teachers
BusinessOctober 30, 2026

Scale launches free 12-hour AI course and microcredential for K-12 teachers

Scale has launched Fundamentals of AI for Educators, a free 12-hour, self-paced course and microcredential that helps K-12 teachers use AI effectively and ethically in…

Scale AI's DualChem benchmark tests how LLMs handle dual-use chemistry questions
Safety and ethicsOctober 28, 2026

Scale AI's DualChem benchmark tests how LLMs handle dual-use chemistry questions

Scale AI has introduced DualChem, a benchmark that measures whether large language models help with chemistry questions that could serve either legitimate research or…