8-hour agent shifts are distributed systems, not long conversations

We started running 8-hour autonomous coding sessions last month. The first week was brutal — agents would work for 6 hours, hit one API timeout, and lose everything. Success rates dropped from 88% in our 30-minute demos to 44% in production shifts.

The problem wasn't the models. It was treating agents like single processes instead of distributed systems.

Your agent isn't a script. It's a fleet.

Every long-running agent spawns unpredictable subprocesses: tool calls that retry, memory writes that buffer, API clients that maintain connections. When one subprocess crashes, it shouldn't take down the entire shift.

Here's the isolation pattern that fixed our success rates:

#!/bin/bash
# Agent launcher with process isolation
AGENT_ID="agent_$(date +%s)"
AGENT_HOME="/tmp/agents/$AGENT_ID"

# Isolated workspace
mkdir -p $AGENT_HOME/{workspace,logs,checkpoints}
cd $AGENT_HOME

# Process limits
ulimit -n 256  # file descriptors
ulimit -u 32   # processes
ulimit -v $((2*1024*1024))  # 2GB virtual memory

# Launch with supervision
while true; do
  python agent.py --checkpoint-dir=checkpoints 2>&1 | tee logs/session.log
  if [ $? -eq 0 ]; then break; fi
  echo "Agent crashed, restarting from checkpoint..."
  sleep 5
done

But isolation is just the start. You need checkpoint discipline:

• State snapshots every 15 minutes — not just conversation history, but file states, environment variables, and active tool connections
• Atomic checkpoints — write to temp files, then rename. Never corrupt a recovery point.
• Recovery verification — when restarting, verify the checkpoint actually represents a valid state before continuing

The third piece is per-agent cost tracking. Long-running agents generate variable inference chains that can spiral into budget disasters:

# Cost governor in your agent loop
class CostGovernor:
    def __init__(self, budget_per_hour=10.00):
        self.budget = budget_per_hour
        self.spent = 0
        self.start_time = time.time()
    
    def check_spend(self, estimated_cost):
        elapsed_hours = (time.time() - self.start_time) / 3600
        budget_used = self.spent / (self.budget * elapsed_hours)
        
        if budget_used > 0.9:  # 90% budget used
            return "HALT"  # Stop execution
        elif budget_used > 0.7:  # 70% budget used
            return "CHEAP"  # Switch to cheaper model
        return "CONTINUE"

We run 7 concurrent agents on a single 14GB machine now. Each agent gets its own process tree, cost budget, and recovery protocol. When one agent hits a memory leak or infinite retry loop, the others keep working.

Key insight: Your 8-hour agent shift is actually 480 separate 1-minute distributed transactions. Design for partial failures, not perfect execution.

The difference is dramatic. Our 8-hour success rates went from 44% to 91%. More importantly, partial failures now preserve 6 hours of work instead of losing everything.

Long-running agents aren't just bigger prompts. They're infrastructure.