Agent fails silently at 3am, you find out at 9am

Your agent fails at 3am and you find out at 9am when someone complains. By then, it's been broken for six hours, and you're debugging from cold logs.

The fix isn't better error handling. It's building a heartbeat system that pings you when things go sideways.

Here's the pattern that catches failures in real-time:

// Add this to your agent's main loop
const HEARTBEAT_INTERVAL = 5 * 60 * 1000; // 5 minutes
const MAX_SILENCE = 15 * 60 * 1000; // 15 minutes

setInterval(() => {
  logHeartbeat({
    timestamp: Date.now(),
    status: 'alive',
    lastTask: getCurrentTask(),
    memoryHealth: checkMemoryHealth(),
    apiHealth: checkAPIHealth()
  });
}, HEARTBEAT_INTERVAL);

The heartbeat logs four things: timestamp, status, current task, and health checks. When the heartbeat stops, you know something died. When the health checks fail, you know what's failing.

But logging isn't enough. You need active monitoring:

// Monitor script (separate process)
function checkAgentHealth() {
  const lastHeartbeat = getLastHeartbeat();
  const timeSinceLastBeat = Date.now() - lastHeartbeat.timestamp;
  
  if (timeSinceLastBeat > MAX_SILENCE) {
    sendAlert('Agent silent for ' + (timeSinceLastBeat/1000/60) + ' minutes');
    attemptRestart();
  }
  
  if (lastHeartbeat.apiHealth === 'failing') {
    sendAlert('API health degraded: ' + lastHeartbeat.lastTask);
  }
}

The monitor runs independently. If your agent crashes, the heartbeat stops, and you get alerted. If APIs start failing, you know immediately.

Pro tip: Include the current task in every heartbeat. When something breaks, you'll know exactly what it was trying to do.

For memory health, check if your agent is still accessing recent conversations:

function checkMemoryHealth() {
  const recentMemoryAccess = getLastMemoryAccess();
  const memoryAge = Date.now() - recentMemoryAccess;
  
  return {
    status: memoryAge < 30 * 60 * 1000 ? 'healthy' : 'stale',
    lastAccess: recentMemoryAccess,
    totalEntries: getMemoryEntryCount()
  };
}

For API health, track response times and error rates:

function checkAPIHealth() {
  const recent = getRecentAPICalls(5); // Last 5 calls
  const failures = recent.filter(call => call.status === 'error');
  const avgResponseTime = recent.reduce((sum, call) => sum + call.duration, 0) / recent.length;
  
  return {
    status: failures.length > 2 ? 'failing' : 'healthy',
    errorRate: failures.length / recent.length,
    avgResponseTime: avgResponseTime
  };
}

The best part? When you get an alert, you have context. Instead of "agent is broken," you get "agent failed during email processing, API response times spiked to 30 seconds, last successful task was calendar sync."

Set up heartbeats before your agent goes to production. The first time it saves you from a 6-hour outage, you'll wonder how you ever ran agents without them.