# Incident: Gateway Crash Loop
**Date:** 2026-02-06
**Duration:** Minutes (quick fix via CC Local)
**Severity:** Critical — Rivet offline briefly
**Resolved by:** Michael + CC Local (Claude Opus 4.6)

## Root Causes (3 stacked failures)
1. **gateway.mode deleted** from clawdbot.json during Tailscale/model migration edits
2. **Telegram config wiped** — plugin + channel + bot token all gone
3. **Auth store mismatch** — DeepSeek/Moonshot keys in models.json but not auth-profiles.json

## Secondary Issues
- Tailscale exit node routed all VPS traffic through Michael's phone (broke SSH)
- Invalid `channels.whatsapp.enabled` key caused config validation failure
- Systemd restart race condition (port 18789 held by orphaned child)

## What We've Done Since
- [x] Health watchdog deployed (15-min checks, alerts on critical)
- [x] Config backup script (snapshots all 3 config files: main, auth, models)
- [x] Config integrity check in watchdog (gateway.mode, Telegram, auth profiles)
- [x] Initial known-good snapshot taken post-recovery
- [x] Brave Search API key renewed
- [x] Notion API key fixed and centralized in secrets.json

## What's Still Missing
- [ ] Pre-restart config validation (validate BEFORE applying, not after crash)
- [ ] Automatic rollback on crash loop detection (if restart count > 5 in 10 min, restore last backup)
- [ ] Config change audit log (who changed what, when)
- [ ] Runbook: "Gateway won't start" troubleshooting steps

## Lessons
1. Never edit multiple config files in one session without snapshots between changes
2. Config files that must agree (clawdbot.json ↔ auth-profiles.json ↔ models.json) are a design risk
3. The crash loop counter reaching 2,222 means there was NO circuit breaker for hours
4. Always use `gateway config.patch` — never raw file edits