OpenRouter outlines a practical method for benchmarking LLMs

OpenRouter has published a guide to benchmarking large language models with application-specific tasks, code-based grading and repeatable runs. Its example study found GPT-6.1 Sol passed 180/180 trials, while DeepSeek recorded 157/180 passes at the lowest response cost.
- OpenRouter recommends application-specific tasks with script-checkable answers.
- Graders should test correct, incorrect and malformed model responses.
- GPT-6.1 Sol passed 180/180 trials on OpenRouter’s 60 synthetic tasks.
- DeepSeek had the lowest response cost and passed 157/180 trials.
- The public ranking showed weak agreement with the study, with a 0.109 Spearman correlation.
Build tests around the application
OpenRouter recommends choosing models using evidence from the inputs and output formats used by an application, rather than relying only on public rankings. Each benchmark task should define an input, an expected answer and an output contract that a script can check.
The guide describes tasks covering invoice arithmetic, dates and schedules, email extraction, Python functions, output constraints and refund decisions. For the study described by OpenRouter, the team wrote 40 baseline tasks and 20 hard tasks before the final run. The examples include requirements such as returning a number with exactly two decimal places, producing a JSON object with specified keys and values, or returning only APPROVE or DENY.
OpenRouter also advises computing references independently and keeping prompt-tuning cases separate from final evaluation. In its refund examples, the benchmark tests policy boundaries, including exactly $200 and 30 days, as well as values just above each threshold.
Code-based grading needs its own checks
The guide recommends exact matching for fixed outputs and schema or value checks for JSON. Graders should normalize only permitted differences. Accepting prose around a number, OpenRouter says, can conceal a failure when the application requires only a number.
Graders should be tested against known-correct, wrong and malformed responses. OpenRouter reports that a development checker initially rejected valid Python because it omitted ord and type. The team corrected the issue before the final study and specified CPython 3.12.10/WASI with resource limits. The full-study runner isolates generated code and stops when sandbox setup is missing, while the teaching example executes no generated code.
These safeguards matter because a faulty checker can turn infrastructure or grading errors into apparent model failures. The guide says its offline fixtures cover boundary decisions, invalid formats and truncated answers without calling a model.
The example supports controlled comparisons
OpenRouter provides a companion ZIP with a standard-library teaching runner, a JSON example and comparison tools. Users are told to compare the archive’s SHA-256 checksum with a38ee17a2eb7090f9f4849a605cc8214691c889ca0ae40c121259603592178e8 before extracting it. The example requires Python 3.12 or later and its offline checks passed on Windows x64 and Linux. macOS remains untested, while the bundled full-study sandbox and native Ori setup support Windows x64 only.
The three-task example checks an invoice total, UTC conversion and a refund decision. Its default check performs zero inference calls. Live inference requires an explicit live option and an allowance, helping keep validation separate from spending.
In the benchmark cited by OpenRouter, GPT-6.1 Sol passed 180/180 trials across three repeats on 60 synthetic tasks. DeepSeek had the lowest response cost and passed 157/180 trials. Agreement with the public ranking was weak, with a Spearman correlation of 0.109. For users, the message is to compare identical tasks using graded accuracy, billed cost and request time, while checking run counts, missing answers and verified bills before trusting a score.
OpenRouter: How to Benchmark LLMs


