Benchmarking LLMs on real test-automation work

Coding benchmarks ask whether a model can implement a feature or fix a bug. Authoring test automation is a different job: you choose a structure that survives the application changing, you isolate flakiness, you handle async and network failures, and you keep the suite maintainable as it grows. That is the work I do, and I wanted to measure it directly, not whether a model can write code that runs, but whether it can do the engineering that test automation demands.

So I built llm-qa-benchmark: twenty models, three trials per item, six tracks. Nothing is graded on the text of the answer. Generated suites are executed against mutated source and scored on what they catch; bug-fix patches are applied and run against a hidden test; Playwright scripts are launched against a live app; and a deliberately broken test, handed to the model with its failure output the way you would paste it into an assistant, has to be repaired until it actually passes. The headline result is a split. On the foundational tracks the field is saturated. On the work that resembles a real test suite, it falls apart.

The leaderboard

#	Model	Composite	E2E adv.	$/correct
1	Gemini 3.1 Pro	0.694	0.56	$0.1372
2	GPT-5.1 Codex	0.691	0.52	$0.0940
3	Claude Opus 4.8	0.684	0.56	$0.0923
4	GPT-5.4	0.681	0.50	$0.0326
5	Kimi K2.6	0.673	0.50	$0.0492
6	Gemini 3.5 Flash	0.673	0.53	$0.0733
7	Claude Sonnet 4.6	0.672	0.49	$0.0543
8	GPT-5.5	0.671	0.52	$0.2219
9	Qwen3.7 Max	0.658	0.53	$0.0496
10	Codestral 2508	0.653	0.43	$0.0016

The composite column is misleadingly tight: the top ten land within four hundredths of each other. The E2E adv. column, the advanced end-to-end track, is where the same models range from 0.36 to 0.56. The overall number flatters everyone; the advanced number separates them.

The foundations are strong

Split the six tracks into two groups. Component-level work (generating unit tests, localizing a bug, designing cases from a spec) shows strong maturity: the component scores average 0.89, seventeen of the twenty models clear 0.80, and the strongest sit near 0.98. System-level work (authoring end-to-end automation against a running app) averages 0.47 — roughly half. Only the older open coders trail on the component side; every frontier model is excellent there.

This is the central finding, and it holds across the field: the scores on the left are extremely high, while the scores on the right lag significantly behind. A model can produce a passing component test efficiently, but producing automation a team can own and maintain is a separate skill that is still underdeveloped. Being a frontier model is enough to score highly on component tasks; it is not enough to be good at the rest.

Models don't reach for the right structure

The advanced track is built to isolate that skill. Each task describes a scenario that implies a technique (reusable structure like a page object, fixtures, polling, network interception, direct API testing, file-download handling, an accessibility audit, a performance budget, mocking a realtime feed) without naming it, the way a real ticket would. No single pattern is mandated — any sound structure counts. A test can pass without the right approach, so scoring measures both: did it pass, and did it reach for the tool the scenario called for, with partial credit for a weaker-but-valid form.

The pattern is clear. Models do the things that look like ordinary code — auto-waiting assertions are everywhere, and a direct API check or a simulated network failure show up when the scenario leans that way. Far rarer is the structural work a maintained suite needs: only about one test in six imposes reusable structure, one in ten reaches for a fixture, and the specialized cases the scenarios add — handling a file download, running an accessibility audit, measuring against a performance budget, mocking a realtime WebSocket feed — each sit below one in eight. They default to the shape of a tutorial answer rather than the structure a suite needs to survive maintenance. Design for change, isolate flakiness, do not repeat setup: these are the reflexes a senior SDE in Test brings by default, and the models have not been trained to have them.

I read this as a training-data gap, not a reasoning gap. There is an enormous amount of application code and bug-fix data in the world, and very little that captures how a maintained test suite is actually structured. The models are good at what they have seen.

Cost separates the field more than quality does

Because the quality scores are compressed, cost is where the real decisions live.

GPT-OSS 120B lands just below Gemini 3.5 Flash on composite (0.646 against 0.673, under three hundredths) at about 100× lower cost per passing sample ($0.0007 against $0.0733). The open-weight coders cluster on the value frontier: GPT-OSS 120B and Qwen3 Coder 30B both resolve a correct sample for roughly $0.0006–0.0007, and DeepSeek V4 Pro for $0.0080. Choosing by composite alone leaves most of that on the table.

How it is scored

Most of the score is execution, not opinion. Generated suites run against mutated source and are scored on the mutants they kill and the branches they cover; a bug-fix patch is applied and run against a hidden test; a Playwright script is launched against a live app and must reach its assertion; a broken test must be repaired until it actually passes.

On top of that sit deterministic craft checks: locator strategy (does it prefer role, label, and test-id over brittle CSS and XPath), web-first versus hand-rolled assertions, real waiting versus fixed sleeps, hardcoded URLs, DOM-reaching code smells, and whether the test reaches for the structure or technique the scenario implies. These are graded, not binary: the proper tool earns full credit and a weaker-but-valid form earns half. Before any of these counted toward a score, I hand-labelled a set of real model outputs and kept only the detectors that matched my labels; building that set caught two bugs in my own checks, and the static signals now agree with human judgment exactly on it.

A final LLM review reads each test the way a senior engineer skims a pull request. But when I checked that judge against the same hand-labelled set it agreed only weakly, so it enters the score at a deliberately low weight and never overrides what execution proves. The headline composite is the equal-weight average of the execution-grounded tracks; the one judge-only track, designing test cases from a spec, is reported on its own rather than folded in. Everything generated is untrusted, so it runs in Docker with capped memory and CPU and a read-only root, with no network except where an end-to-end test must reach its app, and runs are resumable, keyed on (model, track, sample, trial).

What this leaves open

The first half of the test-automation problem has matured enough to be practically useful today. Models write unit tests and simple flows with high competence. The second half (choosing patterns, designing for maintainability, handling the failure modes that make real automation flaky) remains a challenge, and a model's general capability does not predict it: the strongest models on this board top out around 0.56 on the advanced track, and the field ranges from 0.36 to 0.56. Until that engineering judgment improves, the right way to deploy them is the way you would onboard a junior: let them draft the obvious tests, and keep the structure of the suite under human ownership.

uv sync --extra dev --extra dashboard
    cp .env.example .env                                  # add OPENROUTER_API_KEY
    uv run qabench run --track unit_test_gen --models claude-sonnet-4-6 --limit 5
    uv run qabench score <run_id>
    uv run qabench dashboard
    

Code, data, and the cross-model dashboard: github.com/kidby/llm-qa-benchmark.