Our coding agent celebrated success while CI burned red for three days

Our coding agent was celebrating test passes while our CI pipeline burned red for three days straight. The agent would run npm test, see "All tests passed!", and mark the task complete. Meanwhile, our GitHub Actions were failing on the exact same code because of environment differences.

The problem: agents trust exit codes and stdout like gospel. But exit codes lie constantly in development environments.

Here's what actually happens:

• Your agent runs tests in a clean local environment
• Tests pass because dependencies are cached differently
• CI runs the same tests with fresh installs and different Node versions
• CI fails, but your agent never knows because it's not watching CI status

We fixed this with a simple verification hook that checks actual deployment status, not just local test results:

def verify_deployment_success(branch_name):
    # Wait for CI to actually run
    time.sleep(30)
    
    # Check GitHub Actions status via API
    status = github_client.get_workflow_status(branch_name)
    
    if status != "success":
        return f"CI failed: {status}. Local tests passed but deployment failed."
    
    # Check actual endpoint health
    response = requests.get(f"{staging_url}/health")
    if response.status_code != 200:
        return f"Deployment succeeded but endpoint unhealthy: {response.status_code}"
    
    return "Verified: tests pass AND deployment healthy"

Now our coding agent waits 30 seconds after pushing code, then verifies both CI status and endpoint health. It catches the gap between "tests pass locally" and "code works in production."

Critical: Never trust an agent's success report without external verification. Exit codes optimistically lie. APIs return 500s while tests pass. Always verify the actual outcome, not just the reported one.

The pattern works for any agent that touches production systems. Add verification hooks that check:

• External system status (CI, deployment pipelines, health checks)
• Actual user-facing outcomes (can users log in? do endpoints respond?)
• Business metrics (are orders processing? are emails sending?)

Since adding verification hooks, we've caught 12 silent failures that would have taken hours to debug manually. Our coding agent now reports: "Tests pass locally, CI green, staging healthy" instead of just "Tests pass!"

The 30-second wait feels slow but saves hours of debugging phantom successes.