Snorkel AI boosts Qwen3.8-27B on difficult professional tasks

Snorkel AI says it raised Qwen3.8-27B's pass@1 score on OpenAI's GDPval benchmark from 31.9% to 40.7% by combining expert-built training data with a modified reinforcement learning method. The work targets AI agents that produce professional deliverables, rather than simply answer questions.
- Qwen3.8-27B reached 40.7% pass@1 on OpenAI's GDPval gold set.
- The base model's pass@1 score was 31.9%.
- Binary-gated GRPO used about 102K training rollouts.
- Expert-comparison performance rose from 36.4% to 44.0%.
- Budget-exhaustion rates fell from 17.2% to 7.0%.
Training agents to complete professional work
Snorkel AI trained Qwen3.8-27B on 1,105 professional tasks from its GDPval+ series, using about 102K training rollouts. On OpenAI's GDPval gold set, the model reached 40.7% pass@1, compared with 31.9% for the base model.
GDPval evaluates tasks drawn from experienced professionals across nine sectors of the US economy. Each task ends with a deliverable, such as a financial model, regulatory filing, or slide deck created from source documents. Tasks include a human-produced reference deliverable and a detailed rubric describing what a complete result must contain.
Snorkel AI says a single attempt averaged about 50,000 response tokens and could involve numerous tool calls. For evaluation, an attempt passes when it satisfies at least 90% of the rubric items. The company uses that threshold because LLM judges can produce inconsistent verdicts when grading individual items.
Why standard GRPO often stalls
The training method, called binary-gated GRPO, is designed for tasks where successful attempts are uncommon. Standard GRPO samples eight rollouts for each task and compares each result with the group average. When every rollout fails, all receive a zero return, leaving the group with no learning signal.
In Snorkel AI's runs, 79% of generated groups contained no passing rollout. The company says the discarded groups were not uniformly poor: in the median discarded group, the best rollout reached a training score of 78%, and 66% of discarded groups contained a rollout scoring at least 70%.
Binary-gated GRPO keeps the normal pass-or-fail signal whenever a group contains at least one pass. If every attempt fails, it uses a fallback training score that reflects how much of the rubric the deliverable satisfies and whether its key results and conclusions agree with the expert reference. Scores at or below 40% receive no fallback credit, while higher scores are scaled toward the 90% pass threshold.
With this approach, 95% of generated groups carried a training signal, compared with 21% under GRPO. Snorkel AI says standard GRPO, trained on the same data and with a similar number of rollouts, stayed close to the base model.
The gains extended beyond the main score
In LLM-judged head-to-head comparisons, the trained model's deliverables were rated above those of human experts in 44.0% of attempts, up from 36.4% for the base model. Counting ties, the trained model matched or beat the expert in 47.6% of attempts.
Snorkel AI reports that binary-gated GRPO was ahead of the base model in all nine GDPval sectors on rubric pass and in eight of nine sectors on expert comparison. The improvement appeared at both rubric thresholds and in expert comparison for pass@1 and pass@8, where a task is counted as solved if any of eight attempts succeeds.
The model also ran out of response budget less often. That share fell from 17.2% to 7.0%, while it stayed above 11% in every evaluation under standard GRPO. With a 64K-token cap, about half the full budget, the trained model already matched the base model's full-budget pass rate.
For users and developers building agents for professional workflows, the results suggest that training methods can make partial failures more useful when successful completions are rare. Snorkel AI's findings are limited to the reported Qwen3.8-27B experiments and GDPval evaluations, but they point to a practical focus for agent development: teaching systems to finish complex, rubric-based work while preserving the distinction between a completed deliverable and partial progress.
Snorkel AI: Training agents for professional work when success is rare


