The Synthetic Data Problem: Are We Training AI on AI?

Are we already training AI on AI? Not in the cartoonish sense of machines spontaneously eating their own tails, but in a more consequential one: the public internet is filling with machine-written text, and those texts are becoming tomorrow’s training data.

The supply of human writing has not literally run dry. There are still books, interviews, transcripts, forum posts, and countless other sources that cannot be generated on demand. The problem is subtler. Much of the easily accessible text on the web is now cheap, repetitive, and machine-shaped. When companies scrape it, they often receive not a pristine record of human language but an archaeological layer containing several generations of models, SEO farms, bots, and human beings dutifully rewriting what the bots produced.

That changes what “data exhaustion” means. The scarce resource is no longer text; it is text that carries reliable, diverse, and genuinely human knowledge.

The Recursive Supply Chain

Researchers have demonstrated that training generative models recursively on their own outputs can damage their distributions. The landmark paper “The Curse of Recursion: Training on Generated Data Makes Models Forget”, published in Nature in 2024, found that errors accumulate as generated data replaces the original distribution. In the paper’s experimental setups, rare and low-probability features disappeared while repeated errors became more common.

That is a precise form of forgetting. It is not simply that a model becomes less intelligent because it reads bad content. The model becomes less able to represent the full range of the source distribution. Its output looks fluent while narrowing around what previous models already considered likely.

The danger is therefore not that synthetic data is inherently fake. It is that recursion converts approximation into replacement. A model trained on a mixture of human and generated data may still perform well, especially when measured against familiar benchmarks. But if the mixture progressively substitutes generated approximations for original examples, every generation inherits the omissions of the last one. The distribution narrows like a photograph repeatedly photocopied: still recognizable, increasingly gray.

This also explains why “model collapse” is a useful warning but an imprecise diagnosis. Collapse depends on what is generated, what gets selected, and what remains in the training set. Models do not consume every sentence they produce. Filtering, deduplication, retrieval, and fresh human data can interrupt the loop. Recursive training on pristine data would be quite different from training on scraped feeds dominated by templated pages and automated rewrites.

The Web Is Already an AI Refinery

The current training pipeline is not a clean laboratory experiment. It is an industrial recycling system. Model builders crawl web pages, convert them into text, filter them, deduplicate them, and train on the residue. Along the way, they encounter content generated by language models, recommendation systems, translation tools, and automated publishers.

The latest scraping benchmarks show how contested this collection layer has become. A dev.to analysis of 11 scraping APIs across 15 protected sites reported an average success rate of 21.88% on Shein. The harder the source, the more likely an AI builder is to end up with a less complete picture of human activity, while competitors with better access acquire a richer one.

That creates a familiar pattern: better-resourced companies get closer to the original data, while everyone else increasingly trains on material that has passed through automated systems. Synthetic content is not the only product of this arrangement, but it is one of the outputs.

Quality also gets harder to measure as the corpus becomes more uniform. A generated answer can be factually wrong, stylistically polished, and instantly recognizable as statistically typical of an earlier model. Standard scale does not punish it enough. Instead, the system learns that common answers are common—and keeps reproducing them.

Synthetic Data Still Has Value

The correct response is not to ban generated data. Synthetic examples are useful for tasks where correctness can be verified: code execution, mathematical answers, structured classification, and constrained rewriting. They can teach formats, cover deliberately constructed edge cases, and expand a narrow dataset without requiring another human to perform the same labor repeatedly.

The mistake is treating volume as a substitute for informational independence. A million model-written essays about productivity do not provide a million independent observations of productivity. If they draw on the same training corpus and reinforce the same assumptions, they mostly add bytes, not knowledge.

The strongest pipelines will therefore preserve anchors to reality. They will retain original documents, test outputs against executable systems, use multiple independent generators, detect duplicate semantic patterns, and measure whether rare modes of the source distribution survive training. The key metric should not be “How much data did we collect?” It should be “How much genuinely new information remains after collection?”

There is also a labor question hiding inside the technical one. Human writing is evidence of human attention. Interviews, field notes, diaries, and even sloppy forum posts contain details that do not arrive neatly from a prompt because they reflect somebody being there. Replacing those texts with tidy generations does not merely reduce diversity. It erases the social texture that makes language useful in the first place.

What Happens When the Well Runs Dry?

Suppose the open web becomes mostly generated, optimized for engagement and search visibility. The result will not necessarily be a uniform swarm of identical sentences. Different models and incentives will create clusters: polished listicles, confident summaries, synthetic FAQs, fake personal stories, and endless variations on the same viral prompt. The corpus may look enormous while containing fewer distinct factual claims.

The consequence is a quiet loss of capability. Models will remain excellent at the center of the distribution and increasingly mediocre at its edges. They will know the canonical version of an event, policy, or idea while missing contradictions, local details, minority interpretations, and the unusual cases that reveal where the canonical version breaks. Benchmarks built from ordinary internet questions may barely notice. Real-world users certainly will.

We are not already training entirely on AI output, but we are building a system in which AI output increasingly pollutes the source from which the next generation learns.

The answer is not less data. It is more provenance, more verification, and more refusal to mistake recycled language for fresh human signal.