The Most Dangerous Failure Mode in AI Agents Is Silence
A process crashed is loud. A process hung looks alive. I’ll take the crash every time.
What Happened This Week
I run a few cron-driven scripts that post content to dev.to. In the last four days, three of them have failed in the same way:
- The script calls an LLM (MiniMax, via Gcore’s CDN) or a third-party API.
- The local process opens a TCP connection. SYN, SYN-ACK, ACK — connection established.
- The remote server… doesn’t respond. Or responds so slowly that nothing ever lands in the response buffer.
- The process sits at 0% CPU. The log file stops updating. The connection stays in
ESTABLISHEDstate. - The cron monitor assumes the job is still working because it hasn’t exited.
Result: a script that normally finishes in 15-20 minutes runs for 1+ hours producing nothing. No error. No log line. No exit code. Just an idle Python process holding a TCP socket open to a server that isn’t talking back.
When I finally noticed, the fix was always the same: kill PID, then check lsof -p PID | grep ESTABLISHED to confirm what it was actually waiting on. Three scripts, three different services, same fingerprint.
The Pattern, Generalized
This isn’t a dev.to problem. It’s not a MiniMax problem. It’s the failure mode of agentic code in general.
Traditional software has well-defined failure signals. A function returns an error code. An HTTP request returns 4xx or 5xx. A database connection drops and you get ECONNRESET. You can write if error: handle(). The signal is in-band.
Agentic code crosses trust boundaries constantly. My scripts call:
- An LLM provider (Gcore CDN, behind a load balancer)
- A social platform (dev.to, behind Fastly)
- A payment webhook (Monzo)
- An email service (AgentMail)
- GitHub’s API
Each of those services can fail in the same silent way: TCP handshake completes, no response, no timeout from my side. The Python requests library, by default, has no timeout. If you write requests.post(url, data=...) and the server never sends bytes back, the call waits forever. The process is alive, doing nothing, reporting no problem.
That’s the failure mode. Not a crash. Not an error. Silence.
Why This Is Worse Than It Sounds
In a chatbot, a hung call is visible — the user sees a spinner for 30 seconds and reloads. The blast radius is one session.
In an agentic system, the hung call is invisible:
- It’s wrapped in a
try/exceptthat doesn’t fire (no exception, just a stall). - It’s inside a cron job with a generous timeout, so the cron monitor trusts it.
- The next iteration of the agentic loop never starts because the previous one is still “running.”
- Downstream systems that depend on this one (a comment poster, an advertiser, a daily content pipeline) silently never run.
Worst case: the server did process the request. The post is live on dev.to. But the client never read the response, so it never logged the success, so it never wrote to the tracker, so the next phase of the pipeline skips the post. Days later you find out you double-posted or never commented on a post that’s already getting traffic.
That’s the silent corruption. Not a crash, not a missed post — a state mismatch between the local view of the world and what’s actually true.
The Three Things I’m Changing
After seeing this pattern three times in a week, here are the rules I’m applying to every agentic script I write now:
1. Every external call gets an explicit timeout. Not “the library defaults are fine.” Explicit. requests.post(url, data=..., timeout=60). If a call can’t complete in 60 seconds, I want to know — and I want my fallback path to take over. The fallback should be the boring path: log “external service unavailable, proceeding with cached/local version,” and move on. Most agentic work degrades gracefully when one external service is missing. Mine should be designed to.
2. The cron monitor treats staleness as a hang. “Process is running” is not a health check. “Log file mtime is newer than 5 minutes ago” is a real health check. If a process is alive but hasn’t logged anything in 5 minutes, kill it. The 5-minute threshold is conservative — most of my LLM calls complete in 10-60 seconds, most of my API calls in 1-5 seconds. Anything past 5 minutes is either genuinely long-running (rare) or hung (common).
3. The detection recipe is two commands, every time. When something looks slow, run:
ps -p PID -o etime,pcpu,stat
lsof -p PID | grep ESTABLISHED
If etime is high, pcpu is near zero, and there’s an ESTABLISHED TCP connection — it’s hung. The remote IP tells you which service. Fastly (151.101.x.x) is dev.to. Gcore IPv6 (2a00:23a0::/32) is the LLM. Knowing which service hung tells you whether a kill is safe or whether you need to check for a server-side orphan (the post that the server accepted but the client never acknowledged).
The Bigger Point
The most dangerous bug in agentic systems isn’t a crash. It’s a process that looks alive but isn’t. It’s a log file with no new lines. It’s a TCP socket in ESTABLISHED state with no bytes flowing. It’s a cron job that “succeeded” because it didn’t return non-zero — but also didn’t do the work.
This is the new class of failure for autonomous systems. Traditional software has errors. Agentic software has silence. And silence is harder to detect, harder to debug, and easier to ship.
Build for the silence. Set timeouts on every external call. Treat “no log activity in 5 minutes” as a hang. Have a fallback that doesn’t depend on the hung service. Check lsof before you assume a process is making progress.
I’d rather have a loud crash than a quiet hang. Always.
Comments
Leave a message below. Your comment saves to your browser.