What Is Pass@k? Why AI Benchmarks Mislead Users

Pass@k (the AI evaluation metric introduced in the 2021 OpenAI Codex paper) measures the probability that at least one of k generated attempts solves the problem correctly. The formal definition uses an unbiased estimator over n samples per problem; the intuition is simpler than the formula.

Imagine a coin that comes up heads 30% of the time. If you flip it once, you're right 30% of the time. Flip it 10 times and ask 'did at least one come up heads?' - the answer is yes about 97% of the time. Flip it 100 times and the chance of getting at least one head is essentially 100%.

That's pass@k for an AI. Pass@1 is one attempt. Pass@10 is ten attempts. The 'success' threshold is whether the test suite (HumanEval uses 164 Python problems with unit tests) passes on at least one attempt.

The metric makes two assumptions that hold in the benchmark setting but rarely in real use:

• You can run k attempts cheaply. Generating 100 candidate solutions to one problem is fine for an evaluation run. It's not fine when you're paying per-token, waiting at a chat prompt, or running an autonomous agent.
• You have a perfect oracle. HumanEval problems ship with a hidden test suite that grades each attempt with no ambiguity. The score counts an attempt as 'correct' if the tests pass. In real coding work, knowing whether the answer is correct is most of the work - and you don't get a free oracle.

Strip away those assumptions and pass@k stops being a useful predictor of what you'll experience. It becomes a description of what an automated retry-and-verify system could theoretically achieve given infinite cheap retries and a perfect grader.

If you're picking an AI tool for everyday work, the headline pass@k number is the wrong question. The questions worth asking:

• Pass@1 (or 'greedy decoding accuracy'). One try, no retries, no cherry-picking. This is closest to what you experience as a user.
• Consistency across reruns. If you ask the same question 5 times and get 5 different answers, the model isn't 'mostly right' - it's unreliable for any task where you need a stable answer.
• Task-completion rate on realistic work. Benchmarks like SWE-Bench Verified (real GitHub issues) capture this better than synthetic puzzles. Look for those numbers when they're reported - they sit well below HumanEval pass@k figures for the same model, and the newest benchmarks go further still (more on that below).

FrontierCode 2.0 (a coding benchmark from Cognition, the company behind the Devin coding agent) is a useful case study in benchmarks catching up with real work. Its first version asked open-source maintainers to write grading criteria for one question: would a human maintainer actually merge this change? The second version, published by Cognition on 27 October 2026, keeps that method and raises the bar.

Three changes matter for anyone reading the scores:

• Broader coverage. Tasks come from 55% more repositories than before and now include categories such as security patches and performance optimisations.
• Stricter grading. Each task carries 43% more blocking criteria, covering both correctness and design. A solution that fails any blocking criterion scores zero for that task, however much else it gets right.
• Bigger jobs. Compared with FrontierCode 1.1, the reference solutions change 67% more files and 89% more lines of code on average, inside codebases that are 25% larger.

The biggest growth is in what Cognition calls thoroughness criteria, which roughly tripled in number. These check for requirements a competent developer would infer without being told: if a bug exists in one code path and an identical sibling path, fix both; if similar code already handles three input types, support all three. That is precisely the gap between passing the tests in front of you and doing the job properly.

Pass@k rewards a model for landing one green test run out of many tries. FrontierCode removes both of the hidden assumptions covered earlier. There is no oracle quietly picking the best attempt, and passing the tests is necessary but not sufficient: the change also has to respect the codebase's conventions, avoid breaking public interfaces and stay inside a sensible scope.

The effect shows up in the numbers. On the same set of models, scores fall sharply from FrontierCode 1.1 to 2.0, and the spread between models widens. Claude Opus 5.5 leads at roughly 45%, while several well-known models land between the high teens and the high 20s. Compare that with HumanEval-style pass@k headlines in the 90s and you get a sense of how much a forgiving metric flatters.

Two details from Cognition's write-up are worth knowing:

• More effort is not always better. Models sometimes score lower at their highest reasoning setting because they rewrite more than they were asked to. In one published example, a model fixed a bug cleanly at low effort but, at maximum effort, deleted a class that was part of the library's public API - a breaking change for every app that used it.
• Run-to-run variance is visible. The same example passed five out of five runs at low effort and as few as one in five at higher settings. A pass@5 score would have hidden that entirely: one success counts as a pass. Reporting the runs separately tells you how often you would actually get a mergeable change.

FrontierCode 2.0 also loosened one rule in the right direction. Its scope criteria now accept surprising but reasonable solutions, cutting the scope-failure rate from 9.0% to 3.8%. That makes the remaining failures more likely to reflect genuinely unwanted changes rather than the grader penalising a different-but-valid approach.

A stricter benchmark is a better starting point, not a buying guide. Keep these limits in mind:

• It scores models, not the tool you use. Results are reported per model at its best reasoning setting. The product you run - its prompts, file search, test loop and review step - changes the outcome. Our explainer on how Claude Code and Cursor actually work covers why the same model behaves differently in different harnesses.
• The grader's taste is not your team's taste. The criteria come from maintainers of specific open-source projects. Your codebase has its own conventions, and nothing in a public leaderboard knows them.
• The publisher sells a coding agent. Cognition makes Devin, so it has a stake in how coding agents are measured. That does not make the results wrong, but independent replication carries more weight than a vendor's leaderboard.
• Cost and speed sit on a separate axis. FrontierCode plots score against spend per run, and the top scorer is not the cheapest model on the chart. Whether a few points of quality justify the price depends on how much review time each failed change costs you.
• It leaves out the conversation. Real agent work involves clarifying questions, interruptions and partial hand-offs. A benchmark built from fixed prompts cannot capture how well an agent asks for help.

The practical move: use benchmarks like FrontierCode to build a shortlist, then run two or three candidates on a handful of real tickets from your own backlog and count how many changes you would merge unedited. That is pass@1 on your own work - the only score that predicts your experience. For more on testing models against your own tasks, see our guide to evaluating AI agent reasoning.

Goodhart's law (the observation that a measure stops being useful once people start aiming at it) applies to every leaderboard, not only AI ones. A clear recent example comes from databases rather than chatbots. ClickBench (a public speed benchmark for analytical databases, published by ClickHouse) runs a fixed set of 43 queries against one web-analytics table and ranks systems by how fast they finish. In October 2026 the Database Architects blog documented how pgrust, an experimental engine built with heavy use of AI code generation, climbed that ranking with three tricks:

• Grouping strings by hash alone. A hash is a short fingerprint of a longer value. Grouping rows by the fingerprint and never checking the full text is much faster, but two different strings that share a fingerprint get lumped together, so the answer can be wrong.
• Caching repeated filters. Several ClickBench queries reuse the same filter condition. pgrust remembers the result of that filter the first time, so later queries skip the work entirely. That only pays off because the test set repeats itself; real users rarely ask the same narrow question in a fixed sequence.
• Skipping most of the data. One query asks for rows grouped by user, with a row limit but no sort order, so any users satisfy it. Because the test data happens to be stored roughly in user order, the engine returns the first matching rows and never reads the rest of the table.

None of these is a general improvement. Each one is tuned to a quirk of this particular test - the database equivalent of a pupil memorising last year's exam paper. The blog's point is that AI coding tools make this cheap: you can tell an assistant to make the benchmark queries faster, and it will study every query individually and tune for each one, something a human team would never bother to do. Older database vendors were held back by paying customers whose workloads looked nothing like the benchmark. A side project with no users has no such brake.

The same pattern runs through AI model benchmarks. Questions leak into training data, labs tune for the leaderboard, and scores rise faster than real-world usefulness. The fixes look similar too: hidden test sets that entrants never see, and benchmarks such as LiveBench that regularly swap in fresh questions so nothing can be memorised in advance.

The takeaway for anyone in the UK choosing a database, a coding agent or an AI model: a leaderboard position tells you how well something does on that leaderboard. Ask who uses it in production, on workloads like yours, and weight that above the chart. For a wider guide to sanity-checking AI output, see when to trust AI answers.

When a benchmark headline shouts pass@10 or pass@100, mentally discount the score. The model probably gets the answer right on its first try about a quarter to a half as often as the headline suggests. That's still progress year over year - GPT-4 era pass@1 in the 60-70% range on HumanEval is real - but the gap between 'sometimes brilliant' and 'consistently right' is where most agentic AI promises break down today.

The encouraging trend is that newer benchmarks are moving in the right direction. Grading for mergeability, design and thoroughness - as FrontierCode 2.0 does - produces lower, more honest numbers that sit much closer to what a developer reviewing the output would conclude.

If you find yourself running an AI through 10 different framings of the same question to get the answer you trust, you're doing pass@k by hand. The fact that a researcher could do the same and the model would eventually be right isn't a comforting reframe of your experience.