---
created: 2026-03-12
tags: [sop, incident, operations]
related: ["[[04-Operations/Fleet Doctor]]", "[[02-Fleet/Sentinel]]", "[[04-Operations/Incidents/2026-03-12 OpenRouter Rate Limit]]"]
---

# SOP: Incident Response

## 1. Detect

Incidents are detected via:
- [[04-Operations/Fleet Doctor]] (automated, every 5 min)
- [[02-Fleet/Sentinel]] security monitoring
- Telegram alerts from [[02-Fleet/Rivet]]
- Manual observation (Michael, CC VPS, CC Local)

---

## 2. Assess Severity

| Level | Definition | Response Time |
|---|---|---|
| **P0** | Fleet down, revenue-blocking, data loss | Immediate |
| **P1** | Single agent down, degraded service | Within 1 hour |
| **P2** | Non-critical, workaround exists | Within 24 hours |

---

## 3. Fix

### Agent Down
→ See [[04-Operations/SOPs/Agent Recovery SOP]]

### API Down (Growth Engine)
1. Check Railway dashboard / logs
2. Check `curl https://rateright-growth-production.up.railway.app/health/ping`
3. If deploy failed → check git log, revert if needed
4. If Railway issue → wait or redeploy

### App Down (rateright.com.au)
1. SSH to VPS: `ssh root@134.199.153.159`
2. Check service: `systemctl status rateright-app`
3. Check nginx: `systemctl status nginx`
4. Check logs: `journalctl -u rateright-app -f`
5. Restart if needed: `systemctl restart rateright-app`

### Full Fleet Outage
1. Check shared provider (was it OpenRouter? → [[04-Operations/Incidents/2026-03-12 OpenRouter Rate Limit]])
2. Switch affected agents to fallback models
3. Clear session overrides in `sessions.json`
4. Restart agents one at a time: `systemctl restart clawdbot-{agent}`

---

## 4. Postmortem

After resolution, create an incident report in `04-Operations/Incidents/`:
- Use [[Templates/Incident Report]] template
- Include: Summary, Root Cause, Fix Applied, Lessons, Time to Resolve
- Link from relevant agent profiles and [[07-Command-Centre/Decision Log]]

---

## Key Lesson

> **Diagnose first, fix specifically, verify, escalate if failed.**
> Blind restarts corrupted Telegram offsets and CAUSED a fleet outage on Feb 18.
> — [[04-Operations/Fleet Doctor]]