Fable 5.2 lifts coding benchmark score to 71% as terminal failures fade

Fable 5.2 scored 71.4% on single attempt coding tasks in Snorkel AI’s comparison with Fable 5.1, up from 53.1% for its predecessor. The analysis found that terminal and output failures nearly disappeared, leaving planning and reasoning as the main sources of failure.
- Fable 5.2 scored 71.4% pass@1 on 26 matched coding tasks.
- Fable 5.1 scored 53.1% pass@1 on the same matched set.
- Fable 5.2 passed every trial on 15 of 26 tasks, compared with 6 for Fable 5.1.
- Fable 5.2 recorded 77 successful recoveries from 90 recovery episodes.
- Its remaining failures centered on strategy, faulty inference, and task understanding.
Fable 5.2 leads on the matched task set
Snorkel AI evaluated Fable 5.1 and Fable 5.2 on an expert-created set of 30 frontier coding tasks. Each model received five attempts per task, producing 150 trials per model. For the side-by-side comparison, Snorkel AI included tasks where both models had at least three clean rollouts. That requirement was met by 26 tasks.
The matched comparison covered 106 trials from Fable 5.1 and 121 from Fable 5.2, with trials lost to environment or endpoint errors excluded from the reported results. Fable 5.2 achieved a 71.4% pass@1 score, compared with 53.1% for Fable 5.1, a gain of 18.3 points.
Fable 5.2 also led when each model had three attempts per task. Its pass@3 score was 80.4%, versus 68.8% for Fable 5.1. The trial-level pass rate increased from 54.7%, or 58 of 106 trials, to 70.2%, or 85 of 121 trials.
At the task level, Fable 5.2 solved at least one attempt on 21 of the 26 tasks, while Fable 5.1 did so on 18. Both models solved 16 tasks at least once, Fable 5.2 alone solved another 5, Fable 5.1 alone solved 2, and 3 remained unsolved by both.
Reliability improved as mechanical failures receded
The benchmark report identifies reliability as one of the clearest differences between the models. Fable 5.1 passed every trial on only 6 of the 26 matched tasks. Fable 5.2 passed every trial on 15 tasks.
According to Snorkel AI, the earlier model often lost attempts to fragile large-file writes, malformed outputs, failed tool calls, or other terminal failures, even when the work was close to correct. In the Fable 5.2 run, there were no premature terminations, no format failures, and one wrong-arguments trace across the analyzed root-caused trials.
Fable 5.2 also showed stronger recovery behavior. The review recorded 90 recovery episodes, with 77 successful recoveries, or 86%. The median recovery took one step, and 62 of the 77 successful recoveries were resolved within two steps. A failed command was usually corrected on the next step, rather than triggering a run-ending cascade.
Verification was the model’s most frequent and highest-rated skill, with 302 recorded instances and 71% rated as strong. The report says Fable 5.2 checked its work without being prompted, including by building adversarial test fixtures, counting expected results before comparing them with its output, and rerunning completed steps to confirm identical results.
Remaining failures point to strategy and reasoning
The improvement did not remove every weakness. In failing Fable 5.2 trials, 96% of 295 skill ratings were adequate or strong, and only 10 were rated weak. Snorkel AI says this indicates that the model often executed a flawed plan competently rather than failing during execution.
The failure profile shifted from mechanical problems to planning and reasoning. Wrong strategy accounted for 35% of Fable 5.2’s root-caused trials, faulty inference for 28%, and misunderstanding the task for 23%. By comparison, Fable 5.1’s most frequent causes included premature termination at 42%, wrong output format at 25%, and wrong tool arguments at 26%.
Two tasks that Fable 5.1 passed reliably failed on every Fable 5.2 attempt, while a third fell from five of five successful attempts to two of five. In one case, the model repeatedly made the same incorrect inference about expected behavior and then verified its work against that interpretation. In another, it misunderstood the task from the beginning.
What the results mean for coding users
For users, the comparison suggests that Fable 5.2 is less likely to lose otherwise completed coding work to terminal interaction or delivery problems on this task set. Its stronger consistency also means fewer repeated runs may be needed when the model has already found a workable solution, according to the benchmark analysis.
The remaining limitations are harder to address with execution safeguards. When a model cleanly follows the wrong strategy or verifies against a stable misreading of the task, additional sampling may not correct the problem. Snorkel AI says the mechanical failures receded in this evaluation, while reasoning failures remained, and that whether this pattern holds across future generations remains untested.
Snorkel AI: Fable 5.2 vs Fable 5.1: Coding Benchmark Results


