EU AI Watch: AI Can Write Faster Than Your Team Can Learn

Evan X. Merz says AI-generated code is making large software products brittle. His evidence is personal rather than clinical: while at Meta, he generated at least 90% of his code with AI, only to see peers merge brittle solutions and “good” programmers overlook glaring flaws. At his next large technology company, he now claims to use AI for 100% of his code.

That matters beyond one engineer’s frustration because AI coding promises speed while quietly shifting the cost into maintenance. Merz reports four recurring failures: shallow tests, unnecessary complexity, declining human understanding, and pull requests arriving faster than anyone can review them.

His account deserves skepticism. The post provides no code samples, incident data or controlled comparison, so it cannot prove that AI is the sole cause of rising outages. But it does expose a serious operational risk that generic productivity benchmarks tend to miss. AI did not multiply his output tenfold—or even twofold. It changed the shape of the work.

The EU’s regulatory answer is imperfectly suited to that problem. The AI Act is now applicable, with enforcement powers active. Yet AI-generated code does not become high-risk merely because an AI system helped write it. Ordinary coding assistance is not automatically subject to the Act’s high-risk regime. If an AI system is used in a listed Annex III context, those rules apply from 2 December 2027; systems embedded in regulated products face their corresponding rules from 2 August 2028.

That leaves a gap between legal compliance and engineering reliability. A company can satisfy documentation, oversight and transparency duties while still receiving thousands of superficially plausible pull requests that create weak tests and hidden failure paths. Article 50 transparency obligations, applicable since 2 August 2026, may also require certain providers to mark synthetic content in a machine-readable format. Marking code as AI-generated, however, does not demonstrate that it is safe, scalable or understood.

GDPR still matters where test data, telemetry or incident records contain personal data. National data-protection authorities remain relevant, but GDPR cannot tell an engineering team whether its test suite covers the customer journeys that matter.

What this means

European AI companies and SaaS providers should treat AI-assisted development as a governance issue rather than a procurement victory. Whatever coding tools they use, they need to preserve meaningful human ownership of the codebase, require tests that reflect real user behaviour, and track incidents, recovery times and review quality over time.

Merz’s most useful warning is not that AI writes bad code. It is that humans can lose the ability to recognise bad code. If speed increases pull-request volume while engineering understanding declines, the organisation may be manufacturing debt faster than it can repay it.

European firms should also ask a more demanding question than “How much code did AI generate?” The better question is “How quickly can a team understand, test, operate and safely change this system?” The AI Act may eventually require stronger controls in high-risk uses, but responsible engineering cannot wait for Brussels to define the standard.

One-sentence takeaway: AI-generated code can meet the EU’s legal requirements while still becoming operational debt, so European companies must measure maintainability and human understanding, not output volume.

Source: AI usage generates a fragile codebase — 447 points on Hacker News