Event-driven agents eat themselves alive in production
Our coding agent was spinning in circles for hours, and the logs looked perfect. API calls succeeding, tasks completing, no errors anywhere. But our Slack was getting flooded with duplicate PR notifications, and our AWS bill was climbing fast.
The problem? A feedback loop we never saw coming.
Here's what happened: Our agent listens for GitHub webhook events → processes the PR → updates the status → GitHub sends a webhook about the status change → agent processes that as a new event → updates status again → infinite loop.
The insidious part is that everything works. The agent responds correctly to each event. GitHub returns 200s. The status updates are valid. But the system is eating itself.
Event-driven agents create invisible feedback loops that standard monitoring doesn't catch. Your logs show success while your budget burns.
We've seen this pattern emerge in multiple places:
- Slack agents that react to their own messages
- File watchers that trigger on their own output files
- Multi-agent teams where Agent A's output triggers Agent B, whose output triggers Agent A
- Calendar agents that reschedule meetings, triggering calendar events, triggering more rescheduling
The fix isn't rate limiting or retries — it's causal awareness.
Here's the pattern that stopped our loops:
// Every action gets a causal ID
const actionId = `${agentId}-${timestamp}-${taskHash}`;
// Tag all outputs with the causal chain
const payload = {
...data,
causedBy: actionId,
causalChain: [...previousChain, actionId]
};
// Before processing any event, check causality
if (event.causalChain?.includes(myAgentId)) {
console.log('Loop detected, skipping');
return;
}We also added execution budgets:
// Kill any workflow that exceeds reasonable bounds
const budget = {
maxActions: 50,
maxDuration: 300000, // 5 minutes
maxCost: 5.00 // dollars
};And deduplication at the event level:
// Hash the meaningful parts of each event
const eventSignature = hash({
type: event.type,
resourceId: event.resource.id,
timestamp: Math.floor(event.timestamp / 60000) // 1-min buckets
});
if (recentEvents.has(eventSignature)) {
return; // Skip duplicate
}The most important safeguard is self-effect detection. Before your agent acts on any event, it should ask: "Did I cause this?"
If you're running event-driven agents, audit your loops now. Check your recent API usage for suspicious patterns — repeated calls to the same endpoints, identical payloads, or usage spikes that correlate with agent deployments.
The agents that survive production are the ones that know when to ignore themselves.