TL;DR
Software does the same thing twice: same input, same result, every time. A coding agent does not: reliability no longer accumulates. An agent that got a task right nine times starts the tenth from zero, and the run that goes wrong gives no warning. Experience narrows the spread of results: well-written instructions and skills land where they were meant to far more often. But the spread never closes. What catches the rest has to be the deterministic part of your system, not a person paying attention.
Run the same code twice and the result is identical. Run the same agent twice and it is not. People vary too. With people, there is an assumption: the more often someone does the same thing, the more reliably they do it. You check a colleague's first delivery hard. Once they have done the task right many times, you rely on it. An agent that has done the same task right many times starts from zero the next time. With a colleague, the learning lives in the person. With an agent, it lives in the instructions you wrote, and nowhere else. So the check you used to run on the first few has to run every time.
The same prompt returns a different answer each run, by construction.
#Prose hides this: two phrasings of one correct answer read as the same answer. A picture cannot. The same property sits behind the that runs as intended for weeks and then once does not.
Ask an for the same user interface twice and you need no test to see it: two layouts, two sets of choices, both defensible, neither the one you saw before. A language model composes its answer by choosing, token after token, among continuations that are all plausible, and nothing in that process obliges it to choose the same way twice. is a property of coding agents.
What made a run diverge never reaches the output you see.
#Nothing in the run that goes wrong looks different while it happens. It comes back with the same confidence and shape as the runs that were right. What made it diverge never reaches the output: what was already in the session, which instructions loaded, what the project looked like at that moment.
Experience shows where a run can go wrong, never which run will.
#An experienced person can point at the places in a task an agent could get wrong, and still cannot tell you which run will be the one that does.
Someone with enough experience of agents knows which instructions tend to go wrong and when the context is too full for the agent to work reliably, and learns to write a skill as reliably as it can be written.
Over a long stretch of , that is what changed: a sharper sense of where a run can go wrong and what to say to keep it from going there.
What experience cannot remove, a deterministic check has to catch.
#An automated, deterministic check that returns pass or fail without interpretation does the same thing every time, precisely the property the agent lacks. Nobody has to watch the agent work. What gets judged is the output. Such checks limit the effects of non-deterministic behaviour, and putting them in place is your job.
Keep exploring