Artificial intelligence news.
Straight from the original announcements.

GPT-6 Astra led coding agents, but finished only 26.4% of tasks

November 10, 2026, 17:08By the AI News Bites Editorial TeamSource: Qodo
GPT-6 Astra led coding agents, but finished only 26.4% of tasks
Image: Qodo

OpenAI’s GPT-6 Astra passed an average of 69.9% of hidden acceptance criteria in a benchmark of coding agents, yet fully finished only 26.4% of tasks. The Co-Design Bench study suggests that agent-generated code can appear complete in a pull request while still missing requirements from the original ticket.

Key points
  • GPT-6 Astra passed 69.9% of hidden criteria on average.
  • It fully finished 26.4% of the 273 benchmark tasks.
  • Opus 5 fully finished 18.3% of tasks.
  • The benchmark used 2,331 hidden acceptance criteria.
  • Qodo says reviews should compare pull requests with their linked tickets.

A strong pass rate hid incomplete tasks

Co-Design Bench was published on 26 Oct 2026 by researchers at Alibaba and five universities. It tested ten coding agents on 273 tasks drawn from 37,463 deduplicated queries across 26,314 real user coding sessions.

The tasks asked agents to revise working web applications, sometimes using an attached image, audio clip, video or document. The researchers created 2,331 hidden, expert-authored acceptance criteria. Agents received the user request and a runnable application, but not the criteria.

After each agent completed its work, judges built and ran the application in a browser and assessed every criterion as Pass, Partial or Fail. A task counted as fully finished only when every criterion passed. GPT-6 Astra ranked first, passing 69.9% of criteria per task on average and completing 26.4% of tasks in full. Opus 5 passed 66.1% of criteria and fully finished 18.3% of tasks, while Qwen3.8-Max recorded 65.3% and 16.5%, respectively.

The benchmark focused on difficult, realistic requests

The researchers selected requests using difficulty, verifiability and past failure signals, including reformulations and rollbacks. The benchmark therefore was designed to test demanding work rather than routine coding prompts. Its tasks covered web applications, not backend services, and each was a single fixed request without follow-up questions.

GPT-6 Astra fully finished 41.0% of 100 multimodal editing tasks and 17.9% of 173 repository editing tasks. Across all ten models, repository edits involving three or four change types passed 59.1% of criteria, compared with 65.4% for edits involving one or two types. Of the 173 repository tasks, 166 required more than one change type.

The results also depended on the software harness used to run the agents. Every model was tested in the Claude Code harness, while changing harnesses moved Qwen3.8-27B’s repository-edit score from 56.0% to 74.9%. The researchers used an LLM judge, a separate LLM verifier that challenged positive verdicts, and two experts who checked 200 outputs.

Why reviewing only the diff can miss the problem

Qodo’s analysis is that an agent can complete most of a task while leaving an important requirement untouched. A change that meets nine of ten criteria may look complete in a pull request because the missing tenth requirement produces no incorrect line for a reviewer to inspect. Qodo gives the example of a discount that never reaches the order summary: the necessary code is absent, rather than visibly broken.

The Co-Design Bench paper describes this pattern by saying that “Substantial coverage of individual requirements can coexist with limited complete-task success.” According to Qodo, the benchmark measures agents rather than code review, so the conclusion about review is Qodo’s inference.

Qodo says it reviews pull requests against their linked tickets from systems including Jira, Linear, Azure DevOps, GitHub Issues, GitLab Issues and Monday.com. Its stated approach is to compare the code with the ticket’s intent, acceptance criteria and subtasks, then report missing or incomplete requirements as findings in the pull request. That approach depends on having written requirements: a vague ticket gives reviewers little to verify.

The benchmark’s authors also identify limits, including the difficult task selection and the possibility of judgment error. They write that retained evidence “cannot eliminate judgment error.” An earlier test reported a different result: the paper says Opus 4.8 reached 99.4% feature completion on WebCompass-edit when it could run and fix its own code, while also noting that the two benchmarks used different task pools and grading.

Related news

JetBrains asks 12 AI models for 1,200 programming hot takes
ResearchNovember 15, 2026, 18:02

JetBrains asks 12 AI models for 1,200 programming hot takes

Twelve AI models overwhelmingly attacked Python, Rust and TypeScript in a JetBrains experiment that collected 1,200 opinions about programming languages. The models also…

Products and tools
Products and toolsNovember 13, 2026, 09:00

Shield hands legacy IT portal tasks to OpenAI-powered agents

Shield says its Forge platform can now hand supported browser tasks to AI agents powered by OpenAI’s GPT-6 Astra, allowing IT providers to automate work in legacy…

For developers
For developersNovember 2, 2026

Qodo's Codex plugin brings code review context into coding sessions

Qodo, a company that builds AI tools for code review and software quality, has built a plugin in the Codex marketplace that feeds organizational knowledge and…