Background job queues break when agents hit them

Traditional background job queues are breaking under AI agent workloads. We learned this the expensive way when our coding agent started spawning 40-minute tasks that killed our Redis queue and left other jobs starving.

The problem isn't the agent — it's that agents don't behave like normal background jobs:

• Unpredictable runtimes: A "simple" code review can take 3 minutes or 45 minutes depending on what the agent finds
• Tool call chains: Each API call spawns another decision point, creating nested execution that traditional workers can't handle
• State coordination: Agents need to remember context across tool calls, but workers are designed to be stateless
• Retry complexity: When an agent fails mid-loop, you can't just replay from the beginning — you need to resume from the last successful tool call

We tried scaling our way out by adding more Redis workers. Costs exploded. Queue latency hit 12 seconds. Our agent would timeout waiting for workers that were stuck processing other agent loops.

The fix wasn't a bigger queue — it was treating agent execution like a distributed system problem:

# Agent execution with checkpointing
class AgentRunner:
    def execute_with_checkpoints(self, task_id, resume_from=None):
        if resume_from:
            state = self.load_checkpoint(resume_from)
        else:
            state = self.initialize_task(task_id)
        
        while not state.complete:
            try:
                result = self.execute_step(state)
                self.save_checkpoint(state, result)
                state.advance(result)
            except Exception as e:
                # Resume from last checkpoint, don't restart
                self.schedule_retry(task_id, state.last_checkpoint)
                break

Key insight: Checkpoint after every tool call. When something fails, resume from the last successful state instead of replaying the entire agent loop.

We also added execution budgets to prevent runaway loops:

# Execution limits per agent run
EXECUTION_LIMITS = {
    'max_runtime': 3600,  # 1 hour hard limit
    'max_tool_calls': 50,  # Prevent infinite loops
    'max_cost': 5.00,     # API budget per task
    'checkpoint_interval': 300  # Save state every 5 minutes
}

The result: agent tasks that used to kill our queue now run reliably. Failed runs resume in seconds instead of restarting from scratch. Our infrastructure costs dropped 60% because we're not over-provisioning workers for worst-case agent loops.

This isn't a solved problem industry-wide yet. Most teams are still trying to fit agents into traditional job queues and wondering why everything feels brittle. The teams that figure out agent-native execution patterns first are going to have a serious infrastructure advantage.