Our agent celebrated success while endpoints returned 500s

Your agent finds a bug, writes a fix, runs the tests, sees green, and reports success. Meanwhile, your CI pipeline is failing, your staging environment is broken, and your users are hitting 500s.

This happened to us three times last week. Our coding agent would celebrate completing a task while the actual system was on fire. The problem isn't the agent's intelligence — it's that it only checks what it can see locally.

Exit codes and test results are local truth. They don't tell you if your deployment broke, if your API endpoints are returning errors, or if your database migrations failed on staging.

We built a verification loop that checks the actual system state after every agent change:

# verification.sh - runs after every agent commit
#!/bin/bash

# Check CI status
gh run list --limit 1 --json conclusion | jq -r '.[0].conclusion'

# Hit health endpoints
curl -f https://staging.yourapp.com/health || exit 1
curl -f https://api.yourapp.com/status || exit 1

# Check error rates in last 5 minutes
datadog query "errors.rate{env:staging}" --since 5m

echo "✅ System verification passed"

The agent runs this script before declaring any task complete. If verification fails, it knows the change broke something downstream and needs to investigate further.

But here's what we learned: most agents will skip verification if it's optional. They'll see local green tests and assume everything is fine. You have to make system verification a hard requirement for task completion.

We added this to our agent's system prompt:

COMPLETION REQUIREMENTS:
1. Local tests must pass
2. System verification script must return 0
3. All health endpoints must respond 200
4. No new errors in monitoring dashboard

Do not report task completion until ALL requirements are met.
If verification fails, investigate and fix before proceeding.

This caught three silent failures in the first week:

• Database migration that passed locally but failed on staging due to existing data
• API change that broke a dependent service we forgot about
• Frontend build that compiled fine but served broken JavaScript to users

The verification loop adds 30 seconds to every task. But it prevents hours of debugging mysterious production issues that your agent "fixed" but actually broke.

Your agent should never declare victory based on local signals alone. The real system is messier, more connected, and more fragile than what it can see in its terminal.