# Fleet Outage Lessons Learned — 2026-02-18
*Written by CC Local after full fleet recovery. For ALL agents.*

> **UPDATE (Mar 2026):** The OpenClaw mapOptionsForApi bug was fixed in v2026.3.2. Non-Anthropic providers (MiniMax, DeepSeek, Moonshot/Kimi, OpenRouter) now work correctly. The fleet has migrated to multi-provider (zero Anthropic dependency). The session archive lesson and Telegram offset lesson remain fully valid.

## The Incident
5 of 8 agents stopped responding to Telegram DMs. Messages were consumed by polling but silently dropped — no errors, no replies, no pairing codes. Only Rivet, Builder, and (intermittently) Radar worked.

## Root Causes (3 layered problems)

### 1. Broken Non-Anthropic Model Fallbacks (CRITICAL)
**Problem:** OpenClaw has a known bug — DeepSeek, MiniMax, and Kimi providers all crash with `Error: Unhandled API in mapOptionsForApi: undefined`. When the primary model (Opus) hit any issue, agents fell back to these broken providers and silently crashed mid-message processing.

**Affected configs:**
- Harper: MiniMax M2.5 as FIRST fallback (before Sonnet!)
- Builder: Kimi K2 fallback
- Herald: Kimi K2.5 fallback
- Rivet: DeepSeek fallback
- Susan/Radar: Sessions stuck on `authProfileOverride: "deepseek:default"`

**Fix:** Removed ALL non-Anthropic fallbacks from all 8 agents. Every agent now runs:
```
primary: anthropic/claude-opus-4-6
fallbacks: [anthropic/claude-sonnet-4-20250514]
```

**Rule: NEVER add non-Anthropic models as fallbacks until the OpenClaw mapOptionsForApi bug is fixed. Anthropic-only fallback chains.**

### 2. Stale Telegram Update Offsets (THE SILENT KILLER)
**Problem:** Each agent stores a `lastUpdateId` in `/root/.clawdbot-{agent}/telegram/update-offset-default.json`. The stall-detector crons (running every 2-5 min) were force-restarting agents mid-polling, corrupting these offsets. Result: the gateway's `shouldSkipUpdate()` function silently skipped ALL new messages because their update IDs were less than or equal to the stored offset.

**Evidence:** Harper, Sentinel, and Radar all had the EXACT same offset (370526109) despite having different bot tokens — clear cross-contamination from the restart scripts.

**Fix:** Reset offset files to `{"version":1}` (no lastUpdateId) and restart. Gateway picks up fresh from Telegram's current state.

**Key file:** `/root/.clawdbot-{agent}/telegram/update-offset-default.json`

**Rule: If an agent polls Telegram but silently drops messages, reset the update offset FIRST. It is the most common silent failure.**

### 3. Destructive Cron Jobs (ROOT CAUSE of problem 2)
**Problem:** Four cron jobs were running as root, all restarting agents aggressively:
- `stall-detector.js --auto-wake` (every 5 min)
- `crash-loop-guard.sh` (every 2 min)
- `buddy-check.js --auto-wake` (every 30 min)
- `health-check.sh` (every 5 min)

These caused: restart storms, mid-poll interruptions, offset corruption, message loss during restart windows, and port conflicts from orphaned processes.

**Fix:** All 4 crons DISABLED in root crontab.

**Rule: NEVER auto-restart agents on a timer. If monitoring is needed, alert but do not act. Let a human or coordinator decide.**

## Secondary Issues Found

### Session Auth Overrides
Susan and Radar had `authProfileOverride: "deepseek:default"` stuck in their session state, forcing every message through the broken DeepSeek provider even though the config said Anthropic.

**Fix:** Clear `authProfileOverride`, `authProfileOverrideSource`, and `authProfileOverrideCompactionCount` from `sessions.json`.

**Rule: After changing model configs, check session state for stale auth overrides.**

### Missing sessions.json
Harper's `sessions.json` was archived (moved to .bak) during debugging without creating a replacement. Gateway silently failed to create new sessions.

**Rule: sessions.json must always exist. Even an empty `{}` works.**

## Recovery Playbook (When Agent Stops Responding)

```
Step 1: Check basics
  systemctl status clawdbot-{agent}
  ss -tlnp | grep {port}

Step 2: Check for broken model fallbacks
  python3 -c "import json; c=json.load(open('/root/.clawdbot-{agent}/clawdbot.json')); print(c['agents']['defaults']['model'])"
  -> Must be Anthropic-only

Step 3: Reset Telegram offset (most common fix)
  systemctl stop clawdbot-{agent}
  echo '{"version":1}' > /root/.clawdbot-{agent}/telegram/update-offset-default.json
  systemctl start clawdbot-{agent}

Step 4: Check session state
  cat /root/.clawdbot-{agent}/agents/main/sessions/sessions.json | python3 -m json.tool | grep -i "authProfile"
  -> Clear any non-Anthropic overrides

Step 5: Verify sessions.json exists
  ls -la /root/.clawdbot-{agent}/agents/main/sessions/sessions.json
  -> If missing: echo '{}' > sessions.json

Step 6: Nuclear option — fresh start
  systemctl stop clawdbot-{agent}
  Reset offset (step 3)
  Archive session: mv sessions.json sessions.json.bak.$(date +%s)
  echo '{}' > sessions.json
  systemctl start clawdbot-{agent}
```

## Current Fleet Status (Post-Recovery)

| Agent | Port | Status | Model | Service |
|-------|------|--------|-------|---------|
| Rivet | 18789 | UP | opus-4-6, sonnet fallback | clawdbot-gateway |
| Builder | 18790 | UP | opus-4-6, sonnet fallback | clawdbot-builder |
| Susan | 18792 | UP | opus-4-6, sonnet fallback | clawdbot-susan |
| Harper | 18796 | UP | opus-4-6, sonnet fallback | clawdbot-harper |
| Sentinel | 18800 | UP | opus-4-6, sonnet fallback | clawdbot-sentinel |
| Radar | 18804 | UP | opus-4-6, sonnet fallback | clawdbot-radar |
| Herald | 18808 | UP | opus-4-6, sonnet fallback | clawdbot-herald |
| Cog | 18812 | UP | opus-4-6, sonnet fallback | clawdbot-cog |

## Disabled Crons (DO NOT RE-ENABLE without review)
```
# DISABLED 2026-02-18: */5 * * * * stall-detector.js --auto-wake
# DISABLED 2026-02-18: */2 * * * * crash-loop-guard.sh
# DISABLED 2026-02-18: */30 * * * * buddy-check.js --auto-wake
# DISABLED 2026-02-18: */5 * * * * health-check.sh
```

## Key Takeaways
1. **Silent failures are the worst failures** — No errors in logs, messages just vanish
2. **Telegram offset is the number 1 thing to reset** when DMs stop working
3. **Auto-restart scripts cause more damage than downtime** — Alert, don't act
4. **Anthropic-only fallback chains** until OpenClaw fixes non-Anthropic providers
5. **Session state can override config** — Always check both