Artificial intelligence news.
Straight from the original announcements.

Snorkel AI boosts Qwen3.8-27B on difficult professional tasks

November 19, 2026, 15:32By the AI News Bites Editorial TeamSource: Snorkel AI
Snorkel AI boosts Qwen3.8-27B on difficult professional tasks
Image: Snorkel AI

Snorkel AI says it raised Qwen3.8-27B's pass@1 score on OpenAI's GDPval benchmark from 31.9% to 40.7% by combining expert-built training data with a modified reinforcement learning method. The work targets AI agents that produce professional deliverables, rather than simply answer questions.

Key points
  • Qwen3.8-27B reached 40.7% pass@1 on OpenAI's GDPval gold set.
  • The base model's pass@1 score was 31.9%.
  • Binary-gated GRPO used about 102K training rollouts.
  • Expert-comparison performance rose from 36.4% to 44.0%.
  • Budget-exhaustion rates fell from 17.2% to 7.0%.

Training agents to complete professional work

Snorkel AI trained Qwen3.8-27B on 1,105 professional tasks from its GDPval+ series, using about 102K training rollouts. On OpenAI's GDPval gold set, the model reached 40.7% pass@1, compared with 31.9% for the base model.

GDPval evaluates tasks drawn from experienced professionals across nine sectors of the US economy. Each task ends with a deliverable, such as a financial model, regulatory filing, or slide deck created from source documents. Tasks include a human-produced reference deliverable and a detailed rubric describing what a complete result must contain.

Snorkel AI says a single attempt averaged about 50,000 response tokens and could involve numerous tool calls. For evaluation, an attempt passes when it satisfies at least 90% of the rubric items. The company uses that threshold because LLM judges can produce inconsistent verdicts when grading individual items.

Why standard GRPO often stalls

The training method, called binary-gated GRPO, is designed for tasks where successful attempts are uncommon. Standard GRPO samples eight rollouts for each task and compares each result with the group average. When every rollout fails, all receive a zero return, leaving the group with no learning signal.

In Snorkel AI's runs, 79% of generated groups contained no passing rollout. The company says the discarded groups were not uniformly poor: in the median discarded group, the best rollout reached a training score of 78%, and 66% of discarded groups contained a rollout scoring at least 70%.

Binary-gated GRPO keeps the normal pass-or-fail signal whenever a group contains at least one pass. If every attempt fails, it uses a fallback training score that reflects how much of the rubric the deliverable satisfies and whether its key results and conclusions agree with the expert reference. Scores at or below 40% receive no fallback credit, while higher scores are scaled toward the 90% pass threshold.

With this approach, 95% of generated groups carried a training signal, compared with 21% under GRPO. Snorkel AI says standard GRPO, trained on the same data and with a similar number of rollouts, stayed close to the base model.

The gains extended beyond the main score

In LLM-judged head-to-head comparisons, the trained model's deliverables were rated above those of human experts in 44.0% of attempts, up from 36.4% for the base model. Counting ties, the trained model matched or beat the expert in 47.6% of attempts.

Snorkel AI reports that binary-gated GRPO was ahead of the base model in all nine GDPval sectors on rubric pass and in eight of nine sectors on expert comparison. The improvement appeared at both rubric thresholds and in expert comparison for pass@1 and pass@8, where a task is counted as solved if any of eight attempts succeeds.

The model also ran out of response budget less often. That share fell from 17.2% to 7.0%, while it stayed above 11% in every evaluation under standard GRPO. With a 64K-token cap, about half the full budget, the trained model already matched the base model's full-budget pass rate.

For users and developers building agents for professional workflows, the results suggest that training methods can make partial failures more useful when successful completions are rare. Snorkel AI's findings are limited to the reported Qwen3.8-27B experiments and GDPval evaluations, but they point to a practical focus for agent development: teaching systems to finish complex, rubric-based work while preserving the distinction between a completed deliverable and partial progress.

Related news

Paper2Agent turns research papers into interactive AI agents
ResearchNovember 19, 2026, 16:23

Paper2Agent turns research papers into interactive AI agents

Paper2Agent converts a research paper and its codebase into an interactive AI agent that can run the paper’s methods. The framework packages each agent as an MCP server…

Snorkel AI says gated training lifts Qwen3.8-27B on professional tasks
ResearchNovember 19, 2026, 15:32

Snorkel AI says gated training lifts Qwen3.8-27B on professional tasks

Snorkel AI says it raised Qwen3.8-27B's pass@1 performance on OpenAI's GDPval gold set from 31.9% to 40.7% by combining expert-built training data with binary-gated…

Fable 5.2 lifts coding benchmark score to 71% as terminal failures fade
ResearchNovember 12, 2026, 18:16

Fable 5.2 lifts coding benchmark score to 71% as terminal failures fade

Fable 5.2 scored 71.4% on single attempt coding tasks in Snorkel AI’s comparison with Fable 5.1, up from 53.1% for its predecessor. The analysis found that terminal and…