# Incident Report — Clawdbot Gateway Down
**Date:** 2026-02-06
**Duration:** ~12 hours (05:42 AEDT crash loop detected → 21:30 AEDT fully resolved)
**Severity:** Critical — Rivet completely offline, no Telegram or WhatsApp
**Resolved by:** CC Local (Claude Opus 4.6)

---

## Summary

Clawdbot gateway entered a crash loop (restart counter reached 2,222) after configuration changes were made during a Tailscale exit node install and model migration to Kimi K2. Multiple config keys were corrupted or removed, preventing the gateway from starting. All issues have been identified and resolved.

---

## Root Causes

Three separate config issues stacked on top of each other:

### 1. Missing `gateway.mode` (Critical — blocked startup)
The `gateway.mode=local` setting was deleted from `clawdbot.json` during config edits. Without this, the gateway refuses to start with:
```
Gateway start blocked: set gateway.mode=local (current: unset)
```
**Fix:** Added `"gateway": {"mode": "local"}` back to config.

### 2. Telegram channel removed from config (Critical — no comms)
The Telegram plugin and channel configuration were dropped from both `plugins.entries` and `channels` sections. The bot token file (`/root/.clawdbot/telegram/bot_token`) also contained an expired/revoked token.
**Fix:** Re-added Telegram to plugins and channels config, obtained new bot token (`8294933660:...`) from BotFather, configured via `clawdbot channels add`.

### 3. Model provider API keys not in auth store (Critical — no model access)
DeepSeek and Moonshot API keys existed in `models.json` (provider config) but were not registered in `auth-profiles.json` (auth store). The gateway only checks the auth store, so all non-Anthropic models failed with "No API key found."

Additionally, `anthropic:default` (API key) was disabled due to billing, and `anthropic:claude-cli` (OAuth) had accumulated cooldowns from the crash loop.

**Fix:** Registered `moonshot:default` and `deepseek:default` API key profiles in `auth-profiles.json`. Cleared all cooldowns and disabled flags from `usageStats`.

---

## Secondary Issues Encountered

### Tailscale Exit Node Broke SSH
After Rivet enabled Tailscale exit node routing, ALL VPS traffic routed through Michael's phone, making the VPS unreachable via SSH from external IPs. Rivet self-diagnosed and disabled the exit node, restoring SSH access.

### Invalid WhatsApp Config Key (Final crash)
Rivet attempted to update WhatsApp channel config and added `"enabled": true` to `channels.whatsapp`, which is not a valid key in that section (WhatsApp enablement is handled via `plugins.entries`). This caused a config validation failure on the next restart.
**Fix:** Removed `channels.whatsapp.enabled` from config.

### Systemd Restart Race Condition
The gateway forks a child process (`clawdbot-gateway`) that holds port 18789. When the parent exits, systemd restarts and the new instance finds the port occupied. This is a design quirk, not a bug — using `clawdbot gateway stop` (which cleanly shuts down the child) before `clawdbot gateway start` avoids it.

---

## Changes Made

| File | Change |
|------|--------|
| `/root/.clawdbot/clawdbot.json` | Added `gateway.mode=local`, added Telegram channel/plugin config, removed invalid `channels.whatsapp.enabled` |
| `/root/.clawdbot/agents/main/agent/auth-profiles.json` | Added `moonshot:default` and `deepseek:default` API key profiles, cleared cooldowns |
| `/root/.clawdbot/telegram/bot_token` | Updated with new bot token |

---

## Current State (Post-Fix)

| Component | Status |
|-----------|--------|
| Gateway | Running, stable |
| Telegram | Running, polling, sending/receiving |
| WhatsApp | Running, connected |
| SSH | Accessible on 0.0.0.0:22 |
| Primary model | `anthropic/claude-opus-4-6` (upgraded by Rivet) |
| Fallbacks | `claude-opus-4-5` → `kimi-k2-0905-preview` |
| Sub-agents | Kimi K2 (default) |
| Heartbeat | DeepSeek (30m interval) |
| `/restart` command | Enabled |
| Tailscale | Installed, exit node disabled |

---

## Lessons Learned

1. **Always validate config after manual edits.** Run `clawdbot doctor` before restarting to catch invalid keys.
2. **Model provider keys must be in BOTH `models.json` AND `auth-profiles.json`.** The gateway reads provider config from one and auth from the other.
3. **Use `clawdbot gateway stop/start` not `systemctl restart`.** The CLI handles child process cleanup; systemd doesn't.
4. **Tailscale exit node routes ALL traffic** — including inbound SSH. Use it only for specific routes, not as a blanket exit node on a server.
5. **Keep `/restart` enabled** so Rivet can self-recover from Telegram without needing SSH access.

---

*Report generated by CC Local (Claude Opus 4.6) — 2026-02-06*