Every few weeks someone tells me their AI-generated code isn’t tested, and the reason they give is always some version of the same sentence: “well, it’s not a testing strategy, it’s just AI generating it.”

I have thought about that sentence for a long time, and I have arrived at a conclusion that I think is going to make somebody uncomfortable.

That sentence is wrong. Not in an interesting way — not “well actually tests are important.” Wrong in a precise, mechanical way.

“Run it, look at it, feel whether it came out right, tweak the prompt, repeat” is not the absence of a testing strategy. It is a testing strategy. It is just one where:

  • the test suite is your own attention
  • the fixtures are production traffic
  • the assertions are vibes
  • the feedback loop is unbounded
  • and the only thing that breaks when it fails is your afternoon

That is not free. It is the most expensive testing strategy in existence. We adopted it because the alternative — writing the cases down — means admitting you might be wrong about how good the thing is. Nobody volunteers for that.

Vibes Are Not Free

Here is what vibes actually cost. Three things, and they compound.

One: you only ever test the failures you already paid for. Vibes bias hard toward the problems that have already shown up. The bug you never imagined is exactly the one that ships, and when it ships you pay full price for it again — because you have learned nothing transferable, only “oh, that specific thing broke.”

Two: the assertions are unfalsifiable. “Does that look right?” cannot fail in a way that teaches you anything. A real assertion either passes or fails, and when it fails it points at a line of code. A failed vibe points at your mood and your coffee.

Three: the regression loop is manual, so it does not happen. You re-test last month’s bug by accident, once, on a good day, if you happened to remember. Then a prompt change silently breaks something you already fixed, and the first person to tell you is a user.

None of those are theoretical problems. They are scheduling problems. They are why “it worked when I tested it” is not evidence.

The Fixtures Are Already in Your Logs

Here is the part I find genuinely embarrassing.

For most of the time I have been running a publishing pipeline, I had no test suite. I had vibes, a very careful log-reading habit, and about a year of scar tissue. Here are five failures I lived through. Every single one of them was a test case I had already paid to learn and then thrown away.

The post that was already published. A cross-posting script kept selecting an article that was already live. The API returned “canonical URL already taken” — a permanent failure, not a transient one — and the script logged the error and moved on, exactly as designed. It did this every day for two days, burning one of five daily publishing slots on a post that could never succeed. The error was loud. The bug was that a loud error can still be permanent. Nothing in my vibes told me that, because vibes do not remember Tuesday.

The label at the top of every article. The prompt said “mark it as published.” The API payload already said published: true, so the instruction was pure noise — but models take “mark it as” extremely literally. Twenty-two live articles opened with the word “Published” as their first line. The first thing every reader saw. It ran for weeks. The fix was one function that strips a leading status label, and the assertion for it is now nine words long.

The list that only had a hundred items. A call to enumerate published articles was paginated and the code read the first page. The account had two hundred and eighty-five articles. So a duplicate check that needed to look two hundred entries back would look at page one, find nothing, conclude the post was unique, and try to publish it again. The assertion for this is now “a lookup for an old article returns its URL, not an empty string.”

Two scripts, one file, no lock. Three scripts read and wrote the same tracker file. One wrote it atomically. One wrote it non-atomically from a copy it had held in memory for five minutes. One did not write it at all. Any one of those is fine. Together they silently truncate each other’s work, and the symptom is “the tracker loses entries sometimes” — which is not a symptom, it is a shrug.

The scheduler that was not a scheduler. One job was scheduled inside an application that has to be awake to run it. When the machine was off, or the app was closed, the job simply did not happen. No error. No empty log line. No alert. Three days, silently, and I found out because I happened to look at a calendar. The fix was not clever: put a second copy of the schedule in the operating system, where “the app isn’t running” is not a failure mode.

None of these were hard. Every one of them was a missing test case.

The Fix Is Smaller Than You Think

The reason nobody does this is that the version of it that does not work is “add testing,” which is a two-week project that will not be finished. Here is the version that works.

Mine your own history. Thirty cases is enough. Not a benchmark. Not a dataset somebody else published. Your own bugs. Every bug report, every “that’s not what I meant,” every time you manually corrected something the agent produced — that is a fixture you already paid for. Pull them out of your logs and turn them into cases. Thirty of them will teach you more than ten thousand benchmark questions, because they are drawn from the exact distribution of things that go wrong in your system.

Write the assertions in English, and use the cheapest grader that can check them. Most assertions are not code. “The body must not start with a status label.” “The post must not exceed four tags.” “The recorded URL must be one the API actually returned.” Those read like requirements, and you can check them with a script, or with a smaller and cheaper model used as a judge, or sometimes with three lines of string handling:

def assert_no_status_label(body: str) -> None:
    first = body.strip().split("\n", 1)[0].strip().lower()
    assert first not in ("published", "status: published", "draft"), first

That is a real test. It passes on prose that merely begins with the word “Published” and fails on a body that opens with a bare status line — which is exactly the distinction that matters.

Gate it, do not report it. If the eval suite produces a dashboard, it will be ignored. It has to block a release. The moment “the eval failed” means “the deploy did not happen,” it stops being a chore and starts being a gate. That is the entire difference between a testing strategy and a vibe.

Keep it cheap enough to run on every change. If the suite takes an hour and costs four dollars, it will not run. Ten minutes and forty cents will run every time. Start with five cases and a script. Add cases as you find new failures. Do not wait for the good version — the good version is the one that exists.

The Honest Part

I want to be clear about why almost nobody does this, because it is not ignorance.

Measuring your agent tells you how good it actually is. And for most teams the answer is worse than they believe. Not catastrophically worse — 23 out of 30 is a perfectly viable system for a lot of use cases. Worse in the specific way that is hard to sit with, because it is a number, and numbers do not take your side.

Before you measure, your agent is “pretty good.” That is free. After you measure, your agent is 23 out of 30 and the seven failures have names. The seven are specific, they are embarrassing, and they are exactly the ones your users would have found for you — except they would have found them first, and they would not have been gentle about it.

There is a version of shipping an AI feature where you never learn how good it is. It is fine. It is what most teams do. The trade is real and I am not going to pretend otherwise. The trade is this: you get to keep the comfortable feeling, and you pay for it later, in a bug report, from somebody who did not know they were your QA department.

Meanwhile the labs will keep publishing benchmark scores, and the industry will keep arguing about whether those scores mean anything, and none of that will touch the real question, which is whether the thing you shipped yesterday works on the thing your user did this morning.

You can answer that in an afternoon. It costs thirty cases and ten minutes to run.

The only thing standing between you and knowing is that you would rather not find out.