Cline-Bench ranks AI agents on 47 real-world coding tasks

Cline has introduced Cline-Bench, a benchmark designed to measure how reliably AI agents complete real user requests in existing software projects. Its initial evaluation covers 47 coding tasks, with each model run three times per task.
- Cline-Bench contains 47 agentic coding tasks.
- Each model was run three times per task.
- Gemini 4 Argon had the highest success rate at 51.1%.
- The benchmark reports pass@1, pass@3 and pass^3.
- Cline plans to add multi-turn and non-coding tasks.
A benchmark built around real software projects
Each Cline-Bench task includes a user request, a runnable starting repository, and tests covering both the requested changes and relevant existing behavior. Cline says it invested substantial engineering and review effort in constructing, repairing and validating these task packages.
The current benchmark focuses on single-turn coding tasks. Cline plans to extend it to multi-turn interactions, where users provide feedback or additional instructions, as well as to non-coding use cases.
To measure both capability and repeatability, Cline runs each model three times on every task. The benchmark reports pass@1, the mean success rate across trials, alongside pass@3, the share of tasks with at least one successful trial, and pass^3, the share of tasks solved successfully in all three trials.
Gemini 4 Argon leads the initial results
Gemini 4 Argon recorded the highest success rate, at 51.1%. Opus 5.5 followed at 48.9%, tied with Gemini 4 Carbon, while Sonnet 5.5 reached 48.2%.
GPT 6 Astra and DeepSeek V4.1 Flash 1017 each posted 44.7%, followed by GPT 6.1 Sol at 44.0% and Kimi K3.1 Pro at 43.3%. GLM 5.3 reached 34.0%, Mistral Large 4 reached 29.1%, and Nemotron 3.5 Super VL recorded 9.2%.
Cline describes the results as preliminary. The reported 95% task-level bootstrap confidence intervals are conditional on the selected task set, and the company says they do not establish that small ranking differences reflect differences in model capability.
Testing consistency, safeguards and efficiency
Cline says the benchmark is designed to preserve the original request, its constraints and the context available to the agent. It also reviews tests and graders against required behavior, repairs unsupported assumptions and scoring defects, and validates changes with correct and deliberately incorrect implementations.
Solutions are evaluated in a separate clean environment, protected tests are restored, and missing or skipped required tests count as failures. Cline also tests the grader with deliberately tampered solutions, measures intended to prevent agents from receiving credit by weakening tests, skipping checks or fabricating passing output.
The evaluation used Cline CLI 3.0.74 and a four-hour solver limit. Runtime and interaction data include mean and median solver time and assistant turns, excluding environment setup and grading. Gemini 4 Argon had a mean solver time of 27.8 minutes and a mean of 80.5 assistant turns, while GPT 6 Astra averaged 14.2 minutes and 59.9 assistant turns.
For users and developers, Cline-Bench offers a view of agent performance that goes beyond a single successful attempt. Its repeated trials, repository-based tasks and grader safeguards are intended to show how often agents complete required changes and how consistently they do so. The results remain limited to the selected coding tasks, with broader evaluations left for future extensions.


