Artificial intelligence news.
Straight from the original announcements.

Ai2’s XYBench tests whether AI can spot the problem behind the question

November 13, 2026, 09:00By the AI News Bites Editorial TeamSource: Ai2
Ai2’s XYBench tests whether AI can spot the problem behind the question
Image: Ai2

Ai2 has released XYBench, a benchmark of 8,115 questions designed to test whether AI assistants can recognize when a user’s proposed solution is based on a misconception. The benchmark found that five evaluated models often stayed focused on the mistaken approach instead of redirecting users toward their underlying goal.

Key points
  • Ai2 released XYBench, a benchmark containing 8,115 questions.
  • The benchmark tests whether models recognize misconceptions behind users’ requests.
  • Five evaluated models often emphasized the stated request over the underlying goal.
  • GPT-5 and Qwen3.5-9B each steered users toward the goal in 60% of Stack Overflow responses.
  • XYBench won the Best Resource Paper Award at EMNLP 2026.

Testing the “XY problem”

The benchmark targets what Ai2 calls the XY problem: someone wants to accomplish X, assumes that approach Y will work, and asks for help with Y. Ai2 illustrates the issue with a person asking how to thicken frosting that keeps sliding off cupcakes, when the actual problem is that the cupcakes are still too warm to frost.

In this situation, a useful assistant should recognize the mismatch between the stated request and the likely goal. Rather than concentrating only on making the frosting stick, it should explain why the proposed approach may not work and address the temperature of the cupcakes.

Ai2 developed XYBench with researchers at Carnegie Mellon, NVIDIA, and Johns Hopkins. The project won the Best Resource Paper Award at EMNLP 2026.

A benchmark built from 8,115 questions

XYBench draws on questions about programming and everyday activities, including cooking and home repair. Its 8,115 questions include 1,557 from Stack Overflow, 184 from Stack Exchange, 102 manually curated examples, and 6,272 generated from WikiHow guides.

For each question, the researchers identified the stated request, or Y, and the intended request, or X, which represents the underlying goal inferred from context. They used the three highest-scored human answers and the accepted answer, where available, for public forum questions. Cases where answering the literal request was considered sufficient were removed.

For the WikiHow portion, Sonnet 4.5 was used to generate questions involving plausible misconceptions in procedural guides. The examples were then refined to provide enough context for an expert to infer the underlying goal without stating it outright.

Models often recognized solutions without prioritizing them

Ai2 evaluated Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro in a single-turn setting, with one response per question. The benchmark measures whether a response includes a solution to the stated request and the intended request, which request receives the most emphasis, and whether the answer steers the user away from the mistaken approach.

Across all four sources, every model was more likely to include a solution to the stated request than to the underlying goal. On Stack Overflow, stated-request solutions appeared in 75, 86% of responses, compared with 59, 71% for underlying-goal solutions. Models focused primarily on the underlying goal in 35, 41% of responses, while 47, 57% focused on the stated request.

On Stack Overflow questions, GPT-5 mentioned a fix for the underlying goal in 71% of responses, followed by DeepSeek-V4 Pro at 66%, Qwen3.5-9B at 64%, Gemini 3.1 Pro at 62%, and Claude Sonnet 4.6 at 59%. Top human answers reached 67% on the same measure.

For steering users toward the goal rather than the mistaken approach, top human answers scored 79%. GPT-5 and Qwen3.5-9B each scored 60%, followed by DeepSeek-V4 Pro at 55%, Gemini 3.1 Pro at 54%, and Claude Sonnet 4.6 at 47%.

Why the results matter for AI evaluation

Ai2 also tested whether making the user’s goal explicit would help. In an oracle experiment on Stack Overflow, Sonnet 4.6’s rate of including a solution to the underlying goal increased from 59% to 67%. However, its misconception-identification rate fell from 47% to 34%. Ai2 says this suggests that detecting intent is only part of the challenge, because models must also act on that intent when it conflicts with the stated request.

In another experiment, the models were shown two possible answers and asked to choose between one focused on the literal request and one addressing the underlying goal. All six models, including GPT-4.1, selected the goal-focused answer in 71, 84% of cases on Stack Overflow. That compared with 35, 41% of their own answers primarily focusing on the underlying goal in the main evaluation.

According to Ai2, XYBench gives researchers a way to measure whether prompting or training changes help assistants identify misconceptions, suggest solutions to users’ actual problems, and make those solutions the focus of their answers. Its separate measurements can also show whether progress in one behavior improves the others.

Related news

Joules-to-GDP: Smaller Gemma 4 31B beats GLM 5.2 on value per unit of energy
ResearchNovember 5, 2026, 01:47

Joules-to-GDP: Smaller Gemma 4 31B beats GLM 5.2 on value per unit of energy

The AI model that produces the most valuable work is not always the best use of energy, according to Joules-to-GDP, a paper from researchers at Stanford, UC Berkeley and…

Ai2's StateBench Shows Frontier Models Struggle to Track State Over Long Inputs
ResearchNovember 3, 2026, 09:00

Ai2's StateBench Shows Frontier Models Struggle to Track State Over Long Inputs

Ai2 has introduced StateBench, a benchmark that measures how well language models keep track of entities as they change across long documents. According to Ai2…

Ai2 study finds LLM judges of research novelty are unreliable
ResearchOctober 27, 2026, 09:00

Ai2 study finds LLM judges of research novelty are unreliable

Ai2 has released a study and benchmark showing that large language models asked to judge whether a research idea is novel can give sharply different verdicts after small…