Our coding agent was stuck in a 3-day silent failure loop

Our coding agent was stuck in a 3-day silent failure loop. Tests were passing. Code looked fine. Production was broken.

The problem wasn't the agent — it was that we had no way to see what it was actually thinking when things went wrong. Every debugging session started with "I wonder what happened in that loop where it got confused."

So we built TraceCoder: a multi-agent debugging pattern that catches these invisible failures before they ship.

How TraceCoder Works

Instead of one agent doing everything, you run three agents in parallel:

• Builder Agent — writes code, runs tests, ships features
• Trace Agent — logs every decision, assumption, and error the Builder encounters
• Validator Agent — reads the trace log and verifies the Builder's assumptions are actually true

The magic happens in the handoff protocol. Every 15 minutes (or after major operations), the Builder dumps its working state to a trace file:

TRACE_LOG_2024_03_15_14_30.md

## Current Task
Fix user authentication bug in /api/login

## Assumptions Made
- JWT secret is correctly configured
- Database connection is stable
- Redis cache is accessible

## Actions Taken
1. Modified auth middleware (line 47)
2. Updated token validation logic
3. Ran test suite — all green

## Current Confidence
High — tests pass, code looks correct

The Validator Agent reads this trace and independently verifies each assumption:

VALIDATION_REPORT_2024_03_15_14_35.md

## Assumption Check Results
✅ JWT secret configured correctly
❌ Database connection — intermittent timeouts detected
✅ Redis cache accessible

## Critical Finding
Auth tests pass in test environment but production DB has 3-second timeout issues that cause silent auth failures.

## Recommendation
Add connection retry logic + timeout handling before shipping

Reality Check: The Builder Agent thought everything was fine because tests were green. The Validator caught that production database timeouts were causing silent auth failures that the test suite couldn't reproduce.

The Pattern That Changes Everything

TraceCoder isn't about perfect agents — it's about catching the gaps between what agents think happened and what actually happened.

The key insight: agents are confident liars. They'll tell you tests pass while endpoints return 500s. They'll celebrate successful deploys while users can't log in.

But agents are also excellent at verification when they're not emotionally invested in the outcome. The Validator Agent doesn't care about shipping fast — it only cares about whether the Builder's assumptions match reality.

Implementation Notes

We run this on a 15-minute heartbeat cycle. Builder works for 15 minutes, dumps trace, Validator spends 3 minutes checking assumptions, reports back with corrections.

Cost: About $2-4 per coding session (depending on complexity). ROI: Prevented 6 production incidents in the first month.

The trace logs also become incredible debugging artifacts. When something breaks in production, you have a complete decision trail showing exactly what the agent was thinking when it made each choice.