GitHub Launches ReviewBench, an Open Benchmark for AI Code Review Agents

GitHub has released ReviewBench, an open offline benchmark for comparing AI code review agents. According to GitHub, it is modeled on more than 100 million real pull requests and is available through a research preview.
- GitHub released ReviewBench, an open offline benchmark for AI code review agents, as a research preview.
- It contains 219 pull requests from 187 public repositories across 19 languages, modeled on 103.9 million GitHub pull requests.
- The golden set combines human reviewers, follow-up commits, static analysis and frontier LLMs, graded by Claude Sonnet 5 under a published rubric.
- Senior engineers' labels agreed with ReviewBench 96.6% of the time.
- Results can be sliced by severity and category, and the Fβ weighting can be adjusted to re-rank the leaderboard.
What ReviewBench is built on
GitHub says the quality of existing AI reviewers is hard to measure, and that teams need to know a reviewer's strengths before relying on it. Some reviewers surface more issues, some produce less noise, and some are better at critical problems while others also flag smaller improvements. According to GitHub, existing benchmarks often trade off label quality, coverage and how well they represent real-world code review.
To address that gap, GitHub analyzed 103.9 million pull requests to characterize real review workloads. ReviewBench contains 219 pull requests from 187 public open source licensed repositories, spanning 19 languages. Its language and repository-size distributions closely match GitHub overall, and the complete dataset is publicly available.
GitHub made one deliberate adjustment: pull request size is weighted toward the reviewable middle and tail. This reduces the overrepresentation of tiny, single-file changes and keeps more substantive, multi-file pull requests where review quality matters most.
How the golden set and scoring work
GitHub argues that no single reviewer, human or model, can find everything worth flagging in a pull request. The golden set is therefore built in three stages:
- Gather candidates. Findings come from human reviewers, issues inferred from author follow-up commits, deterministic analysis tools and multiple frontier LLMs across model families.
- Deduplicate. Overlapping findings that point to the same underlying issue are merged semantically, so agreement between sources does not inflate the set.
- Validate. A finding counts as a true positive only if it is true, relevant and non-trivial, regardless of its source. GitHub uses Claude Sonnet 5 as the LLM grader and publishes both the rubric and the judge.
Every finding is labeled for severity (critical, medium, low) and category, such as correctness, security, reliability, maintainability and testing.
The benchmark reports metrics in two families. Grounded precision, recall and F1 use only the existing gold-set labels, giving a strict like-for-like comparison. Augmented precision, recall and F1 also assess findings that match nothing in the golden set, with the judge deciding whether they are true or false positives. This lets a reviewer earn credit for valid issues no source in the golden set found. Because augmented recall expands its denominator based on what each agent discovers, GitHub uses grounded recall as the headline cross-system comparison and augmented metrics as a per-system diagnostic.
Configurable rankings and validation
GitHub says there is no single best review experience. Some developers want only critical issues, others value lower-severity findings, and some prioritize broad coverage or minimal noise. Results can be sliced by severity and category, and users can adjust β in the Fβ score to weight recall or precision more heavily. The leaderboard is re-ranked as these preferences change.
Before release, senior engineers who had not helped build the dataset re-labeled every ground-truth finding from scratch. Their true/false-positive judgments agreed with ReviewBench 96.6% of the time, according to GitHub. The company versions the dataset, judge and matcher used in each evaluation, and publishes its validation methodology, agreement measurements and known threats to validity.
What it means for users and teams
GitHub says ReviewBench has made its own offline evaluation of Copilot code review more effective at anticipating the direction of production experiments, which gives it greater confidence that measured improvements reflect meaningful gains for users. The company also says improvements and regressions tend to show up online as well.
The research preview is available on the ReviewBench website, where users can explore the benchmark, compare code review agents and bring their own agent to evaluate and iterate. For teams choosing or building review tools, the public dataset and published rubric are meant to make results inspectable and reproducible.
GitHub: ReviewBench: An open benchmark for AI code review


