Generating a prototype that runs used to be the expensive part of starting. It is now close to free, and that lowered what a prototype is worth. A demo proves the idea can exist. It says nothing about what happens when the idea meets real traffic, a partner API that fails at night, or an assessor asking who approved a change.
That gap is where most AI-built software now stalls. The first version arrives in days. The version a business can run takes as long as it always did, and teams that expected the first speed to continue are surprised by the second.
What the tools are good at
None of that is an argument against the tools. They are the largest gain in delivery throughput most of us will see in a career, and the gain is concrete in places that are easy to name.
- Breadth: an unfamiliar repository can be read in minutes instead of a fortnight
- Volume: the migration nobody wanted to do by hand now gets done
- Conformance: a rule can be applied to each file in a codebase with the same care on the last one as on the first, which is more than a person manages after lunch
- Tests: a suite that would have been skipped for lack of time gets written alongside the code
That last one matters most for this article. Agents write tests readily, quickly and in volume. A team can go from no coverage to a large green suite in a week. The question is what that suite is checking.
What a suite checks
A generated test asserts what somebody expected. A suite passes when the system still matches yesterday's understanding of it and fails when it drifts. That is worth a great deal, and it is also the limit: a suite can only fail against something someone already predicted. It holds no opinion about the prediction.
A green suite says the system still does what somebody once expected. It cannot say whether the expectation was right.
When the tests and the code come from the same session, the limit gets sharper. The session that misread the requirement writes code that follows the misreading and tests that confirm it. Both are consistent with each other and both are wrong. The suite is green, coverage is high, and nothing in it will ever fail for the reason that matters.
A worked example
Take a refund screen. The ticket says a customer can request a refund within the return window. An agent builds the screen, the endpoint and a full suite of tests in an afternoon. The tests check the window boundary, the currency rounding, the error messages and the audit log entry. All of them pass.
A person from the support team then uses the screen. They notice it asks for the order number, and customers calling about refunds rarely have it. They have the email address and roughly when they bought. Technically the screen is correct. In practice nobody will get through it on the first try, and the support queue will absorb the difference.
No assertion in the suite could have caught that, because until somebody used the screen, nobody knew there was anything to assert. The fix is small: search by email and date range. The test that defends it takes minutes to write. The expectation behind it took a person using the product.
Why a person still runs the product
That is why a person still runs the product. It is not a slower substitute for the suite, and it should not be a ceremony in the last week before release. A human pass is the step in the process that produces new expectations rather than checking old ones. Somebody uses the thing and says it is technically correct and nobody will use it. Somebody notices the screen does exactly what the ticket described and the ticket was wrong.
So the order matters, and it runs the opposite way round from how it is usually done. Judgement first, automation second. A person exercises the product and forms an expectation. Whatever in that expectation is worth keeping gets locked into a test, and from then on the machine defends it on each commit, at no cost, without getting bored. Automation encodes judgement. A team that automates first ends up with a fast, green, thorough suite guarding assumptions nobody examined.
What the test lead does now
This changes the test lead's job more than it removes it. Writing regression tests by hand is mostly gone. Three things replace it:
- Deciding what counts as finished, in a form the agent can check before it claims a unit is done
- Reading generated suites for the patterns that make them hollow: mocks around the code under test, assertions that restate the implementation, coverage that counts the easy paths twice
- Running the product as a user would, early and often, and turning what they find into new expectations
The second one is where most of the value hides. A suite that mocks its own subject is measuring the mock. It passes regardless of what the code does, and it looks identical to a useful suite in a coverage report. Finding that takes someone who reads the tests with suspicion.
Where a person belongs in the loop
The same reasoning settles where a person belongs in an agent's loop. Reading each line does not scale, and no team does it honestly past the first fortnight. A person belongs wherever a decision is being taken that the machine has no standing to take.
- What the work is for, and what is deliberately outside it
- The design the change has to hold, and the behaviour that must not move
- How the work is cut up so that nothing falls between the pieces
- Whether the code and its tests are finished in the sense a person would defend
Those are the 4 sign-offs on our projects, each with a name against it, and the first one is yours.4sign-offs, and the first one is yours
None of this is a position we reasoned our way into from first principles. It is what the work kept teaching, usually at our own expense. Review runs in a session that did not write the code, because a session that wrote it will defend it. Checks run inside the loop rather than in a document, because a rule enforced only by good intentions decays within a quarter. Acceptance criteria exist in a checkable form before anything is built, because each suite is downstream of somebody's judgement and inherits whatever was wrong with it.
The speed is real and we are not selling a brake. What makes the speed usable is knowing which decisions are not the machine's to take, and putting a name against each one before the work starts.