How to Auto-Recover Crashed OpenClaw Agents with Systemd
How to Auto-Recover Crashed OpenClaw Agents with Systemd
Let me be real: if you're running OpenClaw agents in production and you haven't set up automatic recovery yet, you're volunteering for 3 AM wake-up calls. I learned this the hard way — twice — before I finally sat down and built a proper systemd service configuration that handles crashes like an adult.
This post walks you through exactly how to set up systemd to auto-recover crashed OpenClaw agents. We'll cover the service unit file, the recovery logic, monitoring, and a few tricks I've picked up that'll save you a ton of headaches. Let's get into it.
The Problem Nobody Talks About Until It Bites Them
Here's the scenario. You've got an OpenClaw agent running a long task — maybe it's monitoring competitor prices across fifty websites, or it's processing thousands of customer records for churn analysis. You set it up, watch it run for a few minutes, confirm it's working, and go to bed.
At 3:17 AM, the agent hits a rate limit. Or a network timeout. Or some obscure API error that nobody warned you about. The process dies. Your terminal session is gone. Eight hours of work — and the API costs that came with it — vanish into the void.
You wake up, check your machine, and find... nothing. No agent running. No clear indication of what happened. Maybe a cryptic error in the terminal scrollback if you're lucky enough to have been running it in tmux.
This is the number one complaint I see in the OpenClaw community, across Reddit threads, Discord channels, and Hacker News discussions. People say things like:
"Woke up to find my agent crashed at 3am. Lost 8 hours of data collection work."
"Agent hit a rate limit, crashed, and I have no idea what state it was in."
"Running long tasks is terrifying — one API error and everything is gone."
The frustrating part? OpenClaw already has most of the infrastructure to handle this gracefully. It has checkpointing. It has persistent memory. It has automatic retry logic. But none of that matters if the process itself isn't managed properly at the operating system level.
That's where systemd comes in.
Why Systemd (And Not Just a Bash Loop)
I know what some of you are thinking: "I'll just wrap it in a while true loop in a shell script." I've been there. Here's why that's a bad idea:
- No proper logging integration
- No dependency management (what if your agent needs the network to be up first?)
- No resource controls
- No clean shutdown handling
- No standardized monitoring
- You're basically reinventing a worse version of systemd
Systemd is already running on your Linux server. It's designed to manage long-running services, restart them on failure, handle logging, and integrate with the rest of your system. Use it.
Step 1: Configure Your OpenClaw Agent for Recovery
Before we touch systemd, we need to make sure the OpenClaw agent itself is configured to handle restarts gracefully. This is the part most people skip, and it's the most important.
OpenClaw has a built-in checkpoint and resume system. If you're not using it, start now:
from openclaw import OpenClawAgent
agent = OpenClawAgent(
task="Monitor competitor prices",
checkpoint_interval=10, # Save state every 10 actions
checkpoint_dir="/var/lib/openclaw/checkpoints",
auto_recovery=True,
cache_strategy="persistent", # Cache survives restarts
cache_llm_calls=True,
cache_api_responses=True,
retry_strategy={
"max_retries": 3,
"backoff": "exponential",
"retry_on": ["RateLimitError", "TimeoutError", "APIError"]
},
fallback_actions={
"rate_limit": "wait_and_retry",
"api_error": "use_cached_data",
"timeout": "reduce_batch_size"
},
audit_log=True,
log_level="INFO",
cost_tracking=True
)
A few things to note here:
checkpoint_dir should be an absolute path. When systemd runs your agent, it won't be in your home directory. Use /var/lib/openclaw/checkpoints or something similar that's reliable and persistent.
cache_strategy="persistent" is critical. This means that if your agent spent $12 making LLM calls before it crashed, those responses are cached. When it restarts, it won't re-make those calls. I've seen people waste hundreds of dollars because they didn't enable persistent caching. Don't be that person.
auto_recovery=True handles the application-level recovery. OpenClaw will automatically try to resume from its last checkpoint when it starts up and finds existing state. Systemd handles restarting the process; OpenClaw handles restoring the context.
Now, save your agent script. I keep mine at /opt/openclaw/agents/price_monitor.py. Here's a complete example:
#!/usr/bin/env python3
"""OpenClaw price monitoring agent with full recovery support."""
import sys
import os
from openclaw import OpenClawAgent
CHECKPOINT_DIR = "/var/lib/openclaw/checkpoints/price-monitor"
LOG_DIR = "/var/log/openclaw"
def main():
# Check if we're resuming from a crash
checkpoint_path = os.path.join(CHECKPOINT_DIR, "latest")
if os.path.exists(checkpoint_path):
print("Found existing checkpoint. Resuming...")
agent = OpenClawAgent.resume_from_checkpoint(checkpoint_path)
else:
print("No checkpoint found. Starting fresh...")
agent = OpenClawAgent(
task="Monitor 50 competitor websites for price changes",
checkpoint_interval=5,
checkpoint_dir=CHECKPOINT_DIR,
auto_recovery=True,
cache_strategy="persistent",
cache_llm_calls=True,
cache_api_responses=True,
retry_strategy={
"max_retries": 3,
"backoff": "exponential",
"retry_on": ["RateLimitError", "TimeoutError", "APIError"]
},
fallback_actions={
"rate_limit": "wait_and_retry",
"api_error": "use_cached_data",
"timeout": "reduce_batch_size"
},
audit_log=True,
log_level="INFO",
cost_tracking=True
)
agent.run()
if __name__ == "__main__":
main()
Step 2: Create the Systemd Service Unit
Now for the main event. Create a service file at /etc/systemd/system/openclaw-price-monitor.service:
[Unit]
Description=OpenClaw Price Monitor Agent
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=300
StartLimitBurst=5
[Service]
Type=simple
User=openclaw
Group=openclaw
WorkingDirectory=/opt/openclaw/agents
ExecStart=/opt/openclaw/venv/bin/python /opt/openclaw/agents/price_monitor.py
Restart=on-failure
RestartSec=30
TimeoutStartSec=120
TimeoutStopSec=60
# Environment
EnvironmentFile=/etc/openclaw/agent.env
# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=openclaw-price-monitor
# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/openclaw /var/log/openclaw
PrivateTmp=true
# Resource limits
MemoryMax=2G
CPUQuota=80%
[Install]
WantedBy=multi-user.target
Let me break down the important parts, because the defaults will burn you if you don't understand them.
Restart Configuration
Restart=on-failure
RestartSec=30
Restart=on-failure means systemd will restart the agent whenever it exits with a non-zero exit code. If your agent exits cleanly (exit code 0), it stays stopped. This is what you want — you don't want a completed task to restart in an infinite loop.
RestartSec=30 adds a 30-second delay before restarting. This is important. If your agent crashed because of a rate limit, immediately restarting it will just hit the rate limit again. Thirty seconds gives APIs time to cool down. For agents that interact with rate-limited services, you might even bump this to 60.
Rate Limiting Restarts
StartLimitIntervalSec=300
StartLimitBurst=5
This is your safety net. It means: "If the agent fails 5 times within 300 seconds (5 minutes), stop trying to restart it." Without this, a fundamentally broken agent will restart forever, burning through API credits and filling your logs with garbage.
Five failures in five minutes means something is seriously wrong, and you need a human to look at it. This is where alerting comes in (more on that later).
Security and Resource Limits
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/lib/openclaw /var/log/openclaw
MemoryMax=2G
CPUQuota=80%
Don't skip this. Running an AI agent as a system service without resource limits is asking for trouble. MemoryMax=2G prevents a memory leak from taking down your entire server. CPUQuota=80% ensures your agent can't starve other services. ProtectSystem=strict makes the filesystem read-only except for the paths you explicitly allow.
Environment File
EnvironmentFile=/etc/openclaw/agent.env
Create /etc/openclaw/agent.env for your API keys and configuration:
OPENCLAW_API_KEY=your-key-here
OPENCLAW_LOG_DIR=/var/log/openclaw
OPENCLAW_CACHE_DIR=/var/lib/openclaw/cache
Never hardcode API keys in your agent script. The environment file should be owned by root with restricted permissions:
sudo chown root:openclaw /etc/openclaw/agent.env
sudo chmod 640 /etc/openclaw/agent.env
Step 3: Set Up the System User and Directories
Before enabling the service, create the infrastructure:
# Create a dedicated user (no login shell, no home directory needed)
sudo useradd -r -s /usr/sbin/nologin openclaw
# Create directories
sudo mkdir -p /var/lib/openclaw/checkpoints/price-monitor
sudo mkdir -p /var/lib/openclaw/cache
sudo mkdir -p /var/log/openclaw
sudo mkdir -p /etc/openclaw
sudo mkdir -p /opt/openclaw/agents
# Set ownership
sudo chown -R openclaw:openclaw /var/lib/openclaw
sudo chown -R openclaw:openclaw /var/log/openclaw
sudo chown -R openclaw:openclaw /opt/openclaw
# Set up the Python virtual environment
sudo -u openclaw python3 -m venv /opt/openclaw/venv
sudo -u openclaw /opt/openclaw/venv/bin/pip install openclaw
Step 4: Enable and Start the Service
# Reload systemd to pick up the new service file
sudo systemctl daemon-reload
# Enable the service to start on boot
sudo systemctl enable openclaw-price-monitor
# Start the service now
sudo systemctl start openclaw-price-monitor
# Check it's running
sudo systemctl status openclaw-price-monitor
You should see something like:
● openclaw-price-monitor.service - OpenClaw Price Monitor Agent
Loaded: loaded (/etc/systemd/system/openclaw-price-monitor.service; enabled)
Active: active (running) since Mon 2026-01-15 14:22:03 UTC; 5s ago
Main PID: 12345 (python)
Memory: 245.0M (max: 2.0G)
CPU: 1.234s
Step 5: Set Up Log Monitoring
Since we configured StandardOutput=journal, all agent output goes to the systemd journal. You can view it with:
# View recent logs
journalctl -u openclaw-price-monitor -n 50
# Follow logs in real time
journalctl -u openclaw-price-monitor -f
# View logs from the last crash
journalctl -u openclaw-price-monitor --since "1 hour ago"
But the real power move is combining systemd's journal with OpenClaw's built-in audit logs. Remember that audit_log=True setting? OpenClaw writes structured JSON logs that you can analyze after a crash:
$ openclaw debug /var/log/openclaw/agent_crash.json
🔍 Crash Analysis:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Root Cause: Rate limit exceeded (429)
Last Successful Action: #7341 - API call succeeded
Failed Action: #7342 - stripe.com rate limit
Agent Reasoning Before Crash:
"I need to fetch all customers to calculate churn rate.
Making bulk API call..."
Issue: Agent didn't implement rate limiting backoff
Suggested Fix:
- Add rate_limit_strategy="exponential_backoff"
- Reduce batch size from 1000 to 100
This is one of the things I love about OpenClaw — the failure post-mortem is built in. No more archaeological excavation through logs trying to figure out what happened.
Step 6: Add Alerting for Persistent Failures
When the start limit is hit (5 failures in 5 minutes), you want to know about it. Create a companion service at /etc/systemd/system/openclaw-price-monitor-alert.service:
[Unit]
Description=Alert on OpenClaw Price Monitor Failure
After=openclaw-price-monitor.service
[Service]
Type=oneshot
ExecStart=/opt/openclaw/scripts/alert.sh "OpenClaw price monitor has failed repeatedly and stopped restarting. Manual intervention required."
[Install]
WantedBy=openclaw-price-monitor.service
Then modify the original service to trigger the alert on final failure by adding this to the [Unit] section:
OnFailure=openclaw-price-monitor-alert.service
The alert.sh script can send a Slack message, an email, a PagerDuty alert — whatever works for your setup. Here's a simple one:
#!/bin/bash
curl -X POST -H 'Content-type: application/json' \
--data "{\"text\":\"🚨 $1\"}" \
"$SLACK_WEBHOOK_URL"
Step 7: Test Your Recovery Setup
Don't wait for a real crash to find out if your recovery works. OpenClaw includes a failure simulation tool that I use religiously:
from openclaw.testing import FailureSimulator
agent = OpenClawAgent(
task="Test recovery",
checkpoint_interval=5,
checkpoint_dir="/var/lib/openclaw/checkpoints/test",
auto_recovery=True
)
simulator = FailureSimulator(agent)
results = simulator.run_scenarios([
"rate_limit_at_50_percent",
"network_timeout_random",
"api_error_at_critical_point",
"memory_overflow",
"interrupted_shutdown"
])
🧪 Resilience Test Results:
━━━━━━━━━━━━━━━━━━━━━━━━━
✅ rate_limit_at_50_percent: PASSED (recovered in 45s)
✅ network_timeout_random: PASSED (3 retries, succeeded)
✅ api_error_at_critical_point: PASSED (fallback used)
⚠️ memory_overflow: DEGRADED (recovered but lost 2% progress)
✅ interrupted_shutdown: PASSED (clean resume)
Overall Resilience Score: 92/100
You can also test the systemd restart behavior directly:
# Find the agent's PID
systemctl status openclaw-price-monitor | grep "Main PID"
# Kill it (simulating a crash)
sudo kill -9 <PID>
# Watch it come back
watch systemctl status openclaw-price-monitor
Within 30 seconds (your RestartSec value), you should see the service restart and the agent resume from its last checkpoint.
Running Multiple Agents
If you have several OpenClaw agents, create a template service at /etc/systemd/system/openclaw-agent@.service:
[Unit]
Description=OpenClaw Agent - %i
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=300
StartLimitBurst=5
[Service]
Type=simple
User=openclaw
Group=openclaw
WorkingDirectory=/opt/openclaw/agents
ExecStart=/opt/openclaw/venv/bin/python /opt/openclaw/agents/%i.py
Restart=on-failure
RestartSec=30
EnvironmentFile=/etc/openclaw/agent.env
StandardOutput=journal
StandardError=journal
SyslogIdentifier=openclaw-%i
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/lib/openclaw /var/log/openclaw
MemoryMax=2G
CPUQuota=80%
[Install]
WantedBy=multi-user.target
Now you can manage multiple agents easily:
sudo systemctl start openclaw-agent@price_monitor
sudo systemctl start openclaw-agent@churn_analyzer
sudo systemctl start openclaw-agent@content_scraper
# Check all agents at once
systemctl list-units 'openclaw-agent@*'
The Cost Savings Are Real
I want to highlight something that doesn't get discussed enough: the persistent caching that survives restarts saves real money. Here's what OpenClaw reports after a typical restart:
💰 Cost Optimization:
- Found 847 cached LLM responses (saved $12.40)
- Found 120 cached API calls (saved $3.50)
- New API calls needed: 23 (estimated $1.20)
Total saved by caching: $15.90
Proceeding with fresh calls only...
Without this, every crash means re-running every LLM call and API request from scratch. Over a month of running agents, the savings add up to hundreds of dollars, easily.
Skip the Manual Setup
Everything I've described above works, and I'd recommend understanding it even if you don't build it all yourself. But if you want to skip the hour of configuration and testing, Felix's OpenClaw Starter Pack on Claw Mart includes pre-configured skills that handle all of this — the checkpointing, the retry logic, the systemd service templates, and the monitoring setup. It's $29, and it would have saved me the entire weekend I spent figuring this out the first time. If you're running agents in production and don't want to piece together every config file from scratch, it's the fastest way to get to a reliable setup.
Quick Reference: Common Issues
Agent restarts but loses context: Make sure checkpoint_dir uses an absolute path and the openclaw user has write permissions to it.
Service restarts too quickly: Increase RestartSec. For rate-limited APIs, 60 seconds is safer than 30.
Service stops restarting after a few failures: That's your StartLimitBurst doing its job. Check the logs (journalctl -u openclaw-price-monitor), fix the underlying issue, then sudo systemctl reset-failed openclaw-price-monitor && sudo systemctl start openclaw-price-monitor.
Agent uses too much memory: Lower MemoryMax in the service file and optimize your agent's batch sizes. OpenClaw's reduce_batch_size fallback action helps with this automatically.
Logs are filling up disk: Configure journal rotation in /etc/systemd/journald.conf with SystemMaxUse=500M.
What Your Setup Should Look Like
When everything is configured properly, here's what a crash-and-recovery cycle looks like:
2:00 AM - Agent starts daily price monitoring
2:15 AM - Website #23 times out
→ OpenClaw auto-retries with exponential backoff
→ Success on 2nd attempt
2:45 AM - Website #41 returns 403
→ Checkpoint saved
→ Fallback: uses yesterday's cached data
→ Continues to next site
3:30 AM - Task completes
→ Summary logged to journal
9:00 AM - You check status and see:
"✅ Daily monitoring complete
48/50 sites checked (2 used fallbacks)
Cost: $1.20
5 auto-recoveries handled"
No wake-up call. No manual intervention. No wasted money. The agent handled its own problems, and systemd was there as the safety net in case the process itself died.
That's the whole point. Agent crashes are inevitable. Your job isn't to prevent every possible failure — it's to build a system where failures are handled automatically and you only get involved when something is genuinely broken.
Set up systemd, configure OpenClaw's recovery features, test it with the failure simulator, and go to sleep. Your agents will be fine.