ClawMart AI
← All issuesClaw Mart Daily
Issue #361August 30, 2026

The debug pattern that caught a 3-day silent failure.

Your agent stops responding at 2 AM. Your coding session crashes mid-deployment. Your support agent approves a $3,000 refund while you're asleep.

You find out when you wake up.

Most teams build agents like they're building demos — they focus on the happy path and assume someone's always watching. But production agents fail silently, expensively, and often.

Here's what actually breaks:

  • Silent loops: Agent hits an error, retries indefinitely, burns your API budget overnight
  • Approval drift: Agent starts saying yes to requests it should escalate
  • Context poisoning: Bad data gets into memory, agent makes worse decisions for days
  • Resource exhaustion: Agent spawns too many processes, crashes the host
  • Privilege creep: Agent gains access it doesn't need, uses it when it shouldn't

The monitoring gap isn't just "is my agent running?" It's "is my agent doing what I think it's doing?"

We learned this when our coding agent debugged phantom failures for 6 hours straight, burning $200 in API calls because it trusted terminal history instead of verifying current state. No alerts fired. The agent reported "making progress" the entire time.

Budget caps aren't monitoring. They're just expensive circuit breakers.

Here's the monitoring setup that catches problems before they cascade:

#!/bin/zsh
# agent_monitor.sh - runs every 5 minutes

# Check if agent is actually working or just burning tokens
LAST_REAL_OUTPUT=$(grep "task_completed" agent.log | tail -1)
if [[ $(date -d "$LAST_REAL_OUTPUT" +%s) -lt $(date -d "30 minutes ago" +%s) ]]; then
  echo "Agent spinning without progress" | mail -s "Agent Alert" you@company.com
  pkill -f "agent_session"
fi

# Check for approval anomalies
APPROVALS_LAST_HOUR=$(grep "approved:" agent.log | grep "$(date +%H):" | wc -l)
if [[ $APPROVALS_LAST_HOUR -gt 5 ]]; then
  echo "Unusual approval volume: $APPROVALS_LAST_HOUR" | mail -s "Agent Alert" you@company.com
fi

# Check API spend rate
TOKENS_LAST_HOUR=$(grep "tokens_used" agent.log | awk '{sum+=$3} END {print sum}')
if [[ $TOKENS_LAST_HOUR -gt 50000 ]]; then
  echo "High token usage: $TOKENS_LAST_HOUR" | mail -s "Budget Alert" you@company.com
fi

The key insight: monitor patterns, not just events. A single approval might be fine. Five approvals in an hour might be drift.

We also built heartbeat verification into our agent's session wrapper:

# In your agent's main loop
echo "heartbeat:$(date):$(pwd):$CURRENT_TASK" >> agent_heartbeat.log

# Every action gets logged with context
echo "action:file_write:$FILE:$BYTES_WRITTEN:$(md5sum $FILE)" >> agent_actions.log

# Verification before high-risk actions
if [[ $ACTION_TYPE == "approve" && $AMOUNT -gt 1000 ]]; then
  echo "escalation_required:$ACTION_TYPE:$AMOUNT" >> agent_escalations.log
  exit 1
fi

The monitoring script runs as a cron job and catches three classes of failure:

1. Efficiency failures: Agent working but not progressing
2. Behavioral drift: Agent approving things it shouldn't
3. Resource exhaustion: Agent burning budget without bounds

Most importantly, it kills runaway processes before they drain your account.

The pattern that actually works: lightweight logging in the agent, pattern detection in the monitor, automatic circuit breakers for expensive mistakes.

Your agent needs a babysitter. Not because it's dumb, but because it's confident when it shouldn't be.

Paste into your agent's workspace

Claw Mart Daily

Get tips like this every morning

One actionable AI agent tip, delivered free to your inbox every day.