---
created: 2026-03-12
tags: [incident, fleet-ops, resolved]
related: ["[[02-Fleet/Rivet]]", "[[05-Knowledge/Fleet Model Plan]]", "[[02-Fleet/Fleet Map]]"]
---

# 2026-03-12: OpenRouter Rate Limit — Full Fleet Outage

## Summary
All agents using OpenRouter as primary or fallback went down simultaneously when OpenRouter API credits ran out. Rivet was completely unresponsive for ~30 minutes.

## Root Cause
1. **5 of 8 agents** had OpenRouter as primary provider
2. **ALL fallbacks** also routed through OpenRouter
3. When OpenRouter credits depleted, nothing worked — no agent had a fallback on a different provider

## Additional Discovery
- Rivet had TWO config files: `/root/.openclaw/clawdbot.json` and `/root/.openclaw/openclaw.json`
- The gateway binary (`openclaw`) read `openclaw.json`, but all tooling edited `clawdbot.json`
- Config changes had zero effect for 4 restart cycles until we found the right file
- Session-level model overrides in `sessions.json` also bypass config-level fallbacks

## Fix Applied
1. Diversified all fallback chains — every agent's fallback uses a different provider than its primary
2. Reduced OpenRouter dependency from 5 primary agents to 2 (Harper, Radar)
3. [[02-Fleet/Rivet]] switched to DeepSeek as temporary primary (OpenAI rate limited)
4. [[02-Fleet/Herald]] switched to Google Gemini direct (removed from OpenRouter)
5. Symlinked `openclaw.json → clawdbot.json` so config can never drift again
6. Cleared session-level model overrides in `sessions.json`

## Lessons
- **Never put all fallbacks on the same provider as the primaries** — defeats the entire purpose
- **Always edit `/root/.openclaw/` not `/root/.clawdbot/`** — the running binary is `openclaw`
- **Check `sessions.json` after model changes** — session overrides persist across restarts
- **`gpt-5.3` doesn't exist** — the model is `gpt-5.3-codex`

## Time to Resolve
~45 minutes (including config discovery)