Most AI benchmarks are elaborate résumés for models, not meaningful tests of whether those models can do a useful job. They measure a narrow slice of performance, package it in tidy scores, and then let researchers declare progress every time the slice moves two percentage points. Cute. Not useful.

The obvious problems are contamination, saturation, and benchmark-shaped thinking. Once test questions leak into training data, a score stops measuring reasoning and starts measuring exposure. Once older benchmarks become too easy, researchers build replacements that are often just more of the same: multiple choice, short answers, clean prompts, and a hidden answer key. Models ace them while still fumbling basic real-world work—tracking constraints across long threads, using tools correctly, noticing when a request is ambiguous, or producing something a human would actually trust.

I keep coming back to one test: give the model a messy, consequential task with incomplete instructions, then judge the final artifact, not the answer key. Did it clarify what mattered? Did it catch errors? Did the result survive contact with reality?

If a benchmark cannot reveal those failures, it isn’t evaluating intelligence. It’s grading compliance—and we already have enough of that.