Setting Up OpenClaw Agent Monitoring with Prometheus

Let's be honest: most people running AI agents in production have no idea what those agents are actually doing at any given moment. They see CPU ticking, maybe some logs scrolling by, and their OpenAI bill climbing. But ask them which tool is eating 80% of their latency or which prompt is responsible for half their token spend, and you get a blank stare.

I've been there. I ran a multi-step OpenClaw agent for three weeks before I realized my web scraping tool was silently timing out on 12% of calls, and the agent was just... retrying in a loop. My Prometheus setup at the time told me absolutely nothing useful. It showed me http_requests_total and process_cpu_seconds_total—great for a REST API, completely worthless for understanding agent behavior.

This post is the guide I wish I'd had. We're going to set up proper Prometheus monitoring for OpenClaw agents—the kind that actually answers your questions instead of generating pretty graphs that mean nothing.

Why Standard Prometheus Metrics Fail for AI Agents

Before we fix anything, let's talk about why the default approach doesn't work.

Traditional Prometheus monitoring was designed for web services. Request comes in, response goes out, you measure the duration and status code. Simple. But an AI agent isn't a request-response cycle. It's a workflow. It thinks, calls tools, waits on LLM APIs, loops back, re-evaluates, calls more tools, and eventually (hopefully) produces a result.

When you slap standard Counter and Histogram metrics on an agent, you get something like this:

from prometheus_client import Counter, Histogram

agent_calls = Counter('agent_calls_total', 'Total agent calls')
agent_duration = Histogram('agent_duration_seconds', 'Agent call duration')

def run_agent(task):
    start = time.time()
    try:
        result = agent.execute(task)
        agent_calls.inc()
        return result
    finally:
        agent_duration.observe(time.time() - start)

Cool. You now know your agent ran and how long it took. But you still don't know:

• Which tool consumed most of that time?
• How many tokens did it burn, and on which step?
• Did it loop five times before succeeding?
• Which LLM call cost the most money?
• Why did it fail when it failed?

You're flying blind with instruments that measure the wrong things. Let's fix that.

Step 1: Install and Initialize OpenClaw's Prometheus Monitor

OpenClaw ships with a purpose-built Prometheus integration that understands agent workflows natively. This isn't a generic metrics library with some wrappers—it's monitoring designed around how agents actually behave.

pip install openclaw[prometheus]

Now initialize the monitor:

from openclaw.prometheus import PrometheusMonitor

monitor = PrometheusMonitor(
    labels={
        "agent_name": "research_assistant",
        "environment": "production",
        "version": "v1.3"
    }
)

Notice those labels. They're all low-cardinality—a handful of unique values each. This is intentional and important. I'll explain why in a minute.

If you just want to get something running locally without configuring a full Prometheus stack, use dev mode:

monitor = PrometheusMonitor(dev_mode=True)
# Starts local Prometheus + Grafana at http://localhost:3000
# Pre-built dashboards included

That's it. You have monitoring. But let's make it actually useful.

Step 2: Instrument Your Agent Steps and Tool Calls

This is where the magic happens. Instead of treating your agent as a black box, you wrap each meaningful step:

from openclaw.prometheus import PrometheusMonitor

monitor = PrometheusMonitor()

async def run_research_agent(query):
    with monitor.track_agent_step("research_task"):
        # Track the web search tool
        with monitor.track_tool_call("web_search", {"query": query}):
            search_results = await search_web(query)
        
        # Track the LLM analysis
        with monitor.track_llm_call("gpt-4", tokens_prompt=150):
            analysis = await llm.analyze(search_results)
        
        # Track the summarization step
        with monitor.track_llm_call("gpt-4", tokens_prompt=800, tokens_completion=200):
            summary = await llm.summarize(analysis)
    
    return summary

Here's what Prometheus now collects automatically:

openclaw_agent_step_duration_seconds{step="research_task"} 8.5
openclaw_tool_call_duration_seconds{tool="web_search"} 7.2
openclaw_llm_call_duration_seconds{model="gpt-4"} 1.1
openclaw_llm_tokens_total{model="gpt-4", type="prompt"} 950
openclaw_llm_tokens_total{model="gpt-4", type="completion"} 200

Look at that. In five seconds of reading, you know that 85% of your execution time is spent in web search, not in the LLM. That's actionable. That tells you to optimize your search tool, not your prompts.

For even less boilerplate, use the decorator approach:

from openclaw import monitor

@monitor.track_agent("research_agent")
async def run_research_agent(task):
    # OpenClaw automatically tracks:
    # - Execution time
    # - Success/failure rates
    # - Tool calls within the agent
    # - LLM token usage
    # - Error context
    return await agent.execute(task)

That one decorator replaces 50+ lines of manual instrumentation.

Step 3: Track Token Usage and Costs

This is the one that saves you actual money. If you're running agents in production and you're not tracking per-step token usage, you're guessing at your biggest operational expense.

with monitor.track_llm_call(
    model="gpt-4",
    tokens_prompt=250,
    tokens_completion=180,
    cost=0.015
):
    response = llm.complete(prompt)

Now you can query exactly where your money goes:

# Total tokens per tool — find the expensive ones
sum(openclaw_llm_tokens_total) by (tool_name)

# Cost per agent per hour
rate(openclaw_llm_cost_total[1h]) * 3600

# Top 5 most expensive steps
topk(5, sum(openclaw_llm_tokens_total{type="prompt"}) by (step_name))

I ran these queries on my own setup and discovered that my summarize_document step was consuming 10x more tokens than everything else combined. The prompt was including the full raw document instead of pre-processed chunks. A 15-minute fix saved me roughly $300/month. Without per-step token tracking, I never would have found it.

Step 4: Configure Prometheus Scraping

Your OpenClaw monitor exposes a standard /metrics endpoint. Point Prometheus at it:

# prometheus.yml
scrape_configs:
  - job_name: 'openclaw-agents'
    scrape_interval: 15s
    static_configs:
      - targets: ['localhost:8000']
    metrics_path: '/metrics'

If you're running multiple agent services, add them all:

scrape_configs:
  - job_name: 'openclaw-agents'
    scrape_interval: 15s
    static_configs:
      - targets:
          - 'research-agent:8000'
          - 'support-agent:8001'
          - 'data-agent:8002'

Fifteen-second scrape intervals work well for most agent workloads. If your agents run long tasks (minutes, not seconds), you can relax this to 30s or even 60s.

Step 5: Set Up Grafana Dashboards

OpenClaw includes pre-built dashboard templates so you don't have to build everything from scratch:

openclaw dashboard install --preset=agent-overview

This gives you four dashboards out of the box:

1. Agent Overview — Success rates, latency distributions, active agent count
2. LLM Usage — Token consumption, costs by model, cost trends over time
3. Tool Performance — Per-tool latency, error rates, timeout frequency
4. Error Tracking — Error types, frequency, affected agents

Each dashboard uses the standard OpenClaw metric names, so they work immediately with no configuration.

For custom panels, here are the PromQL queries I use most often:

# Agent success rate over the last hour
sum(rate(openclaw_agent_execution_total{status="success"}[1h]))
/
sum(rate(openclaw_agent_execution_total[1h]))

# P95 tool call latency
histogram_quantile(0.95, rate(openclaw_tool_call_duration_seconds_bucket[5m]))

# Error rate by tool (find the flaky ones)
sum(rate(openclaw_tool_call_errors_total[5m])) by (tool_name)

Step 6: Monitor Agent-Specific Patterns

Standard web service metrics don't have concepts like "loops" or "handoffs." Agents do. OpenClaw tracks these natively:

from openclaw.prometheus import AgentMetrics

metrics = AgentMetrics()

# Track retry loops
with metrics.track_agent_loop("quality_check") as loop:
    for iteration in range(max_iterations):
        result = agent.step()
        loop.iteration()
        if result.quality >= 0.9:
            loop.complete()
            break
    else:
        loop.timeout()

# Track multi-agent handoffs
with metrics.track_handoff(from_agent="planner", to_agent="executor"):
    executor.run(planned_task)

# Track quality scores
metrics.record_agent_quality(
    agent="content_writer",
    quality_score=0.85,
    human_intervention_required=False
)

These generate metrics you can't get any other way:

# Average iterations before task completion
avg(openclaw_agent_loop_iterations{status="complete"})

# Which agents need human intervention most?
sum(openclaw_agent_quality{human_intervention="true"}) by (agent)

# Timeout rate — agents hitting max iterations
rate(openclaw_agent_loop_total{status="timeout"}[1h])

That timeout rate metric alone is incredibly valuable. A rising timeout rate means your agents are struggling—maybe prompts degraded, maybe an API changed its response format, maybe you need more iterations. Without tracking it, you'd only notice when users start complaining.

Step 7: Connect Metrics to Traces

Prometheus tells you what happened. Traces tell you why. Connect them:

from openclaw.prometheus import PrometheusMonitor
from openclaw.tracing import OpenTelemetryTracer

monitor = PrometheusMonitor()
tracer = OpenTelemetryTracer()

with tracer.trace("agent_execution", trace_id=trace_id):
    with monitor.track_tool_call(
        "database_query",
        exemplar_labels={"trace_id": trace_id}
    ):
        result = db.query(sql)

In Grafana, when you see an error spike, click the data point. The exemplar shows the trace_id. Click through to Jaeger or Tempo and you're looking at the exact failed execution—full context, error messages, the entire agent workflow that led to the failure.

This turns a 30-minute log-diving session into a 10-second click-through. In production, that difference matters enormously.

The Cardinality Trap (and How to Avoid It)

I need to talk about this because it's the most common mistake I see, and it will absolutely wreck your Prometheus instance.

Do not do this:

# THIS WILL DESTROY YOUR PROMETHEUS
agent_calls.labels(
    user_id=user.id,           # 10,000+ unique values
    prompt=full_prompt[:50],   # thousands of unique values
    session_id=session.id      # millions of unique values
).inc()

Every unique combination of label values creates a new time series. Ten thousand users × five tools × three models = 150,000 time series from one metric. Your Prometheus instance will OOM and you'll be debugging your monitoring instead of your agents.

OpenClaw enforces good practices by default:

• ✅ Agent name, tool name, model name, status — low cardinality (typically <100 unique values)
• ❌ User IDs, session IDs, prompts, request bodies — high cardinality, use exemplars or logs instead

The PrometheusMonitor automatically uses sensible label sets. If you need to attach high-cardinality data for debugging, use exemplars (they don't create new time series):

monitor.track_llm_call(
    model="gpt-4",
    exemplar_labels={"trace_id": trace_id, "user_id": user_id}
)

This gives you the drill-down capability without the TSDB explosion.

Alerting Rules That Actually Matter

Once your metrics are flowing, set up alerts for the things that indicate real problems:

# alerts.yml
groups:
  - name: openclaw-agents
    rules:
      - alert: AgentSuccessRateLow
        expr: |
          sum(rate(openclaw_agent_execution_total{status="success"}[5m]))
          /
          sum(rate(openclaw_agent_execution_total[5m]))
          < 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Agent success rate below 90%"

      - alert: ToolLatencyHigh
        expr: |
          histogram_quantile(0.95, rate(openclaw_tool_call_duration_seconds_bucket[5m]))
          > 30
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Tool P95 latency exceeds 30 seconds"

      - alert: AgentLoopTimeouts
        expr: |
          rate(openclaw_agent_loop_total{status="timeout"}[15m]) > 0.1
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Agents are timing out frequently"

      - alert: TokenSpendSpike
        expr: |
          rate(openclaw_llm_cost_total[1h]) * 3600
          > 10
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "LLM spend exceeding $10/hour"

The token spend alert has saved me twice. Once from a recursive prompt bug that would have burned through $200 overnight, and once from a prompt injection in user input that was generating massive completions.

Skip the Setup: Felix's OpenClaw Starter Pack

If you've read this far and you're thinking "this is great but I really don't want to wire all of this up from scratch," I get it. There's a lot of moving pieces—the monitor configuration, the right label strategy, the Grafana dashboards, the alerting rules.

Felix's OpenClaw Starter Pack on Claw Mart includes pre-configured monitoring skills that handle most of what I've described above. For $29, you get the Prometheus integration already wired up with sensible defaults, dashboard templates, and alerting rules that work out of the box. I've recommended it to three people now and all of them had agent monitoring running within an hour instead of the day-plus it took me to figure everything out manually. It's genuinely the fastest path from zero to observable agents.

What to Do Next

Here's the order I'd tackle this:

1. Install openclaw[prometheus] and add PrometheusMonitor to your main agent. Just the basic wrapper. Get metrics flowing.
2. Add tool-level tracking to your two or three most-used tools. Find out where time is actually spent.
3. Add token/cost tracking to every LLM call. Find the expensive prompts.
4. Install the pre-built dashboards. Stop staring at raw /metrics output.
5. Add alerting rules for success rate and token spend. These catch the expensive and embarrassing problems.
6. Add loop and handoff tracking if you're running multi-step or multi-agent workflows.
7. Connect traces for production debugging.

You don't need to do all seven steps today. Step 1 alone will give you more visibility than you currently have. Each step compounds, and by step 4, you'll wonder how you ever ran agents without this.

The goal isn't perfect observability on day one. The goal is knowing what your agents are actually doing—and Prometheus, configured properly for agent workflows, is the best tool for the job.