Our agent debugged phantom failures for 47 minutes because it trusted terminal history

Last week our engineering agent tried to "fix" a deployment that was already running perfectly. It spent 47 minutes debugging phantom errors, restarting healthy services, and eventually broke our staging environment trying to solve problems that didn't exist.

The issue? It was reading old terminal output as current system state.

Most agents treat terminal history like a real-time dashboard. They see an error from three commands ago and assume it's still happening. They read "connection refused" from a failed service start and miss the "service started successfully" message two lines down.

Here's the verification pattern that stopped our agent from chasing ghosts:

1. Separate diagnostic commands from action commands

Before doing anything, your agent should verify current state with dedicated status checks:

// Bad: assumes the error is current
if terminal_output.contains("connection refused"):
    restart_service()

// Good: verifies current state first
status = check_service_status()
if status != "running":
    restart_service()

2. Use exit codes, ignore error text

Error messages lie. Exit codes don't. Train your agent to check return values, not parse error descriptions:

// Check if service is actually running
result = subprocess.run(["systemctl", "is-active", "myservice"], 
                       capture_output=True)
if result.returncode == 0:
    # Service is running, don't "fix" it
    return "Service already healthy"

3. Build a current-state cache

Your agent should maintain a simple status cache that gets refreshed with each verification command:

system_state = {
    "api_server": "running",
    "database": "running", 
    "queue_worker": "stopped",
    "last_check": "2024-01-15 14:30:22"
}

4. Timestamp everything

Add timestamps to your agent's decision log so it can distinguish between old problems and current ones:

[14:25] ERROR: Connection refused on port 8080
[14:26] INFO: Restarting service...
[14:27] INFO: Service started successfully
[14:30] STATUS CHECK: Service running, port 8080 responding

The key insight: verification before action. Your agent should never fix something without first confirming it's actually broken.

Pro tip: Create a "system health" command that your agent runs before any deployment or infrastructure changes. Make it return structured JSON with current service states, not just "everything looks good."

Since implementing this pattern, our agent hasn't chased a single phantom error. It still finds and fixes real problems, but it stopped creating new ones by "solving" issues that only existed in terminal history.

The verification pattern works because it treats your infrastructure like a database — you check current state before making changes, not assume state from old logs.