---
created: 2026-03-12
tags: [sop, fleet-ops, recovery]
related: ["[[02-Fleet/Fleet Map]]", "[[04-Operations/Fleet Doctor]]", "[[04-Operations/SOPs/Model Change SOP]]"]
---

# SOP: Agent Recovery

When an agent is down or unresponsive, follow these steps in order. **Do not skip to restart** — diagnose first.

## Step 1: Check Service Status

```bash
systemctl status clawdbot-{agent}
```

If `inactive` or `failed` → go to Step 4 (restart).
If `active` but unresponsive → continue to Step 2.

## Step 2: Check Config

```bash
# Validate JSON
python3 -c "import json; json.load(open('/root/.openclaw-{agent}/clawdbot.json'))"

# For Rivet (main gateway):
python3 -c "import json; json.load(open('/root/.openclaw/clawdbot.json'))"
```

If invalid JSON → fix the config, then restart.

## Step 3: Check for Session Overrides

```bash
# Look for stale model overrides that bypass config:
node -e "const s=JSON.parse(require('fs').readFileSync('/root/.openclaw-{agent}/agents/main/sessions/sessions.json'));for(const[k,v] of Object.entries(s)){if(v.model)console.log(k,v.model)}"
```

If session overrides exist → delete `model` and `authProfileOverride` keys from the session entries, then restart.

## Step 4: Restart

```bash
systemctl restart clawdbot-{agent}
# Wait 8 seconds
sleep 8
# Verify it's running and using correct model:
journalctl -u clawdbot-{agent} --since "10 sec ago" | grep -E "model|started|error"
```

## Step 5: Verify

```bash
# Check port is responding (agent-specific port from Fleet Map):
curl -s http://localhost:{port}/health || echo "Port not responding"
```

## Step 6: If Still Down — Switch Model

If the agent is failing due to API issues (rate limits, auth errors):

1. Check current model and provider in logs
2. Switch to fallback model — see [[04-Operations/SOPs/Model Change SOP]]
3. Ensure fallback uses a **different provider** than primary
4. Update [[02-Fleet/Fleet Map]] and agent profile

## Step 7: Escalate

If none of the above works:
- Write to `rivet/CC-VPS-INBOX.md` (if you're CC VPS)
- Alert via Telegram (if you're Rivet)
- Check [[04-Operations/Fleet Doctor]] logs: `/home/ccuser/rateright-growth/rivet/memory/fleet-doctor.log`

---

## Agent Ports (Quick Reference)

| Agent | Port | Service |
|---|---|---|
| [[02-Fleet/Rivet]] | 18789 | clawdbot-gateway |
| [[02-Fleet/Builder]] | 18790 | clawdbot-builder |
| [[02-Fleet/Susan]] | 18792 | clawdbot-susan |
| [[02-Fleet/Harper]] | 18796 | clawdbot-harper |
| [[02-Fleet/Sentinel]] | 18800 | clawdbot-sentinel |
| [[02-Fleet/Radar]] | 18804 | clawdbot-radar |
| [[02-Fleet/Herald]] | 18808 | clawdbot-herald |
| [[02-Fleet/Cog]] | 18812 | clawdbot-cog |

---

## Key Lesson

> **Never blind-restart.** Diagnose first — blind restarts corrupted Telegram offsets on Feb 18 and caused a worse outage than the original problem. See [[04-Operations/Fleet Doctor]] for the diagnostic philosophy.