Health checks are killing agents mid-thought

Your health checks are killing your agents mid-thought. Your deployments are wiping their memory. And you're debugging "random failures" that aren't random at all.

Here's what happened: We deployed our coding agent to production using the same infrastructure patterns we use for our web apps. Load balancer health checks every 30 seconds. Rolling deployments that drain connections gracefully. Standard stuff.

Everything worked perfectly in staging. Then production traffic hit.

The agent would start a complex refactoring task, spend 90 seconds analyzing dependencies, and suddenly... restart. Clean slate. No memory of what it was doing. The health check saw no HTTP response for 45 seconds and declared it dead.

Worse: our deployment pipeline treated the agent like a stateless web server. Rolling restart? Sure, just drain the connections. Except "connections" were 3-hour coding sessions with full context about the codebase, active file modifications, and a mental model of what needed to be built.

Every deploy was an instant lobotomy.

The failure pattern was invisible in demos because demos are short. Production agents work on real problems that take real time.

Here's what we changed:

• Health checks got smarter: Instead of expecting HTTP responses, we check a heartbeat file the agent updates every 30 seconds. Thinking agents stay alive.
• Graceful shutdown hooks: SIGTERM triggers a context save before the process dies. The agent writes its current state, active tasks, and reasoning chain to persistent storage.
• Session resumption: New agent instances check for saved state on startup. If found, they resume mid-conversation with full context intact.
• Deployment coordination: We added a "drain mode" where the agent finishes its current task before allowing the shutdown. Critical for multi-hour coding sessions.

The infrastructure changes were simpler than we expected:

# Health check script
#!/bin/bash
if [ -f "/tmp/agent_heartbeat" ]; then
    LAST_BEAT=$(stat -c %Y /tmp/agent_heartbeat)
    NOW=$(date +%s)
    if [ $((NOW - LAST_BEAT)) -lt 60 ]; then
        exit 0  # Healthy
    fi
fi
exit 1  # Unhealthy

The agent updates /tmp/agent_heartbeat every 30 seconds, even during long reasoning chains. Health checks pass as long as the agent is alive, not responsive.

For session persistence, we serialize the conversation state and tool chain to Redis on every major state change:

# On shutdown
redis-cli SET agent:session:${AGENT_ID} "$(cat current_state.json)"

# On startup
if redis-cli EXISTS agent:session:${AGENT_ID}; then
    redis-cli GET agent:session:${AGENT_ID} > restore_state.json
    # Resume from saved state
fi

The results were immediate. "Random" mid-task failures dropped to zero. Deployments stopped causing mysterious agent amnesia. Our 8-hour coding sessions actually completed.

Most importantly: we stopped debugging phantom reliability issues and started shipping features.

If your agent works in demos but fails mysteriously in production, check your infrastructure assumptions first. The problem isn't the model—it's the deployment treating a stateful reasoning system like a stateless web server.

Your agent needs infrastructure that understands what it actually is: a persistent, reasoning system that builds context over time. Not a request-response API that can be restarted at will.