Reliability is not capability

The false picture here is the demo. A system does something impressive once, and the observer concludes it can do that thing. But the number that governs delegation is not what a system can do once; it is what it does nine times in ten. A coworker who produces brilliant work half the time and confident failures the other half is not half as useful as a reliable one. It is a liability, because you must now inspect everything, and inspection at that depth costs nearly what the work cost.

METR runs a measurement programme built on about 170 tasks, mostly software and research work, that asks a precise question: how long a task, measured in human labour time, can a system complete at a given success rate. As of early 2026, tasks a frontier system completes at 50 per cent success ran about 12 hours of human labour, with a 95 per cent confidence interval of roughly 5 to 61 hours. At 80 per cent success, the figure is about 1.5 hours, with an interval from 50 minutes to 2 hours 40 (METR, 2026). Read those two numbers again. The 80 per cent horizon is roughly one eighth of the 50 per cent horizon. The same system, the same week, and demanding four-in-five reliability instead of a coin flip divides the deliverable task size by eight. The long-run trend is a doubling roughly every 6 to 7 months, so both horizons move, but the ratio between them is the operational fact. Two caveats the measurement states itself: visual computer-use tasks score 40 to 100 times lower, and the programme does not establish autonomous working duration or transfer outside software-like work.

The second study answers a different question: when a system passes the stated check, is the work actually acceptable? METR had 296 AI-generated pull requests, plus 47 human ones as controls, reviewed blind by 4 active open-source maintainers who did not know which was which. About half of the test-passing AI submissions would not have been merged, a gap of roughly 24 percentage points below the benchmark pass rate. The rejections were for core functionality failures, breaking other functionality, and quality (METR, 2026). The submissions had passed their tests. The tests were not the standard. The maintainers were.

What you do differently, in two rules. First, quote your delegation appetite at the 80 per cent number, never the 50. When you size what a coworker can take, the honest figure is the smaller one, currently around an hour and a half of equivalent human work in software-like domains, less in domains without a measurement. Second, define acceptance as "a named person would accept this", not "the check passed". For every delegated task from Level 1 onward, you will name the person, usually yourself, sometimes a client, whose acceptance is the standard, and review against that person's judgement rather than against whatever automated check exists.

The horizon numbers arrive with caveats written by the measurers themselves, and the caveats change how you may use the numbers. The error bars are roughly a factor of two in each direction, so a 1.5 hour figure honestly means somewhere between 45 minutes and 3 hours. The measure is serial human labour replaced at a stated success rate, not how long a system can run unattended: a 12 hour horizon does not mean 12 hours of autonomous operation, it means the system succeeds half the time on tasks that would take a person 12 hours of connected work. The task suite is mostly software-shaped, and visual computer-use tasks score 40 to 100 times lower, so a horizon quoted for your domain, if your domain involves screens, forms and portals, may be two orders of magnitude smaller than the headline. And the long-run doubling of roughly every 6 to 7 months means any printed figure, including the ones in this lesson, is stale within a year of printing (METR, 2026). What you do differently: never repeat a horizon figure without its date and its domain attached, and treat every capability figure you meet as a floor with a date, not a fact.

The blinded merge study carries its own limits, stated by its authors. The agents were given no iteration loop: they submitted once, with no chance to respond to review, which is not how a supervised coworker works and plausibly understates what supervised delegation achieves. The reviewers worked without continuous integration, so some mechanical checks a real project would run were absent. And the study covered about a quarter of the candidate repositories, so the 24 point gap is a measurement of a sample, not a law. None of this rescues the headline conclusion, that test-passing work is frequently unacceptable work, but it does mean the exact gap in your own delegations could sit above or below it.

The discipline that follows is small and permanent. First, when the model behind a seat changes, re-test your own delegations rather than reading the release notes, because the horizon data describes a population of tasks and your seat carries a sample of one. Second, keep your own acceptance records: for each delegated task, what was submitted, what you accepted, what you corrected, what you rejected. Public figures decay on a 6 to 7 month clock. Your acceptance rate on your own work, measured against your own standard, is the only capability figure that is never stale, never off-domain, and never someone else's task suite. The register you begin in Level 1 is where those records live, and this lesson is the reason it earns its keep.