# LESSONS.md — What Went Wrong and What We Learned

*Hard-won knowledge. Read before you build anything.*

---

## Critical Incidents

### 1. Kimi Agents Wrote Code (Feb 7)
**What:** Rivet spawned Kimi sub-agents to write code. 5 build-breaking bugs introduced.
**Why:** Kimi lacks the codebase context Builder has. Different models have different reliability for code.
**Rule:** ALL code goes through Builder. No exceptions. Not Kimi, not DeepSeek, not sub-agents. Builder only.

### 2. API Keys Exposed in Session Logs (Feb 12)
**What:** 13 API keys logged in plaintext. Reading `.env` files during sessions captured keys in transcripts.
**Why:** Tool outputs get stored in session history. Anything you read appears in logs.
**Rule:** NEVER read `.env` files in sessions. Use external scripts for key operations. Scrub cron runs at 4am daily.
**Scrub script:** `/home/ccuser/rateright-growth/rivet/scripts/scrub-logs.sh`

### 3. SMS Sequences Running Without Approval (Feb 13)
**What:** Michael thought SMS was stopped. It wasn't — 15 sequences still active, 51 enrollments running.
**Why:** Nobody verified the actual state. Assumed "deactivated" without checking.
**Rule:** Verify state directly. Don't assume. All 15 sequences now confirmed deactivated.

### 4. Task Monitor Spam (Feb 2)
**What:** Reminder system sent 78 notifications for the same task.
**Why:** No dedup or rate limiting on alerts.
**Rule:** Cap alerts at 3 per task. Always implement rate limiting on any notification system.

### 5. systemctl restart Killed Own Session (Feb 4)
**What:** Running `systemctl restart` on Rivet's own service terminated the session mid-operation.
**Why:** Restarting your own process kills you. Obvious in hindsight.
**Rule:** ONLY use `gateway restart` tool for self-restart. Never `systemctl restart` on your own service.

### 6. Config Crash — Wrong Key Path (Feb 5)
**What:** Gateway crashed because config used `agent.default` instead of `agents.defaults`.
**Why:** Manual config editing is error-prone.
**Rule:** Use `gateway config.patch` tool. Never edit config files manually.

### 7. Gateway Crash Loop — Port Conflicts (Feb 6)
**What:** 34 crashes in 300 seconds. Gateway couldn't bind port.
**Why:** systemd and manual starts competing for the same port.
**Rule:** One management method per service. Use systemd OR manual, not both.

### 8. Multi-Agent HTTP Gateway Failure (Feb 17)
**What:** 50% of fleet went down simultaneously. Susan, Radar, Sentinel, Cog all stalled.
**Why:** OpenClaw's mapOptionsForApi bug crashed non-Anthropic providers. Session model overrides persisted across restarts.
**Rule:** Service restart alone doesn't fix corrupted sessions. Must archive sessions before restart. Multiple agents on the same broken provider = cascading failure.
**UPDATE (Mar 2026):** mapOptionsForApi bug fixed in OpenClaw v2026.3.2. Non-Anthropic providers now work correctly.

### 9. Session Model Override Crash Loops (Feb 17)
**What:** Agents with non-Anthropic session overrides kept crashing on restart because the override persisted.
**Why:** Session state survives service restarts. Bad model in session = infinite crash loop.
**Rule:** Archive sessions before restarting stalled agents. Check for stale model overrides.

### 10. Fleet-State.json Showed Stale Data (Feb 17-18)
**What:** Rivet spent 3 heartbeats restarting agents that were already recovered. fleet-state.json was hours stale.
**Why:** Agents only update fleet-state when their heartbeat succeeds. Stalled agents leave old timestamps.
**Rule:** ALWAYS verify agent status directly (`agent-bridge status` or HTTP ping) before trusting fleet-state.json. Direct checks are ground truth.

### 11. Builder Inbox Collision (Feb 15)
**What:** Both agents edited the same inbox file. 4 hours of lost coordination.
**Why:** No clear ownership of files.
**Rule:** One writer per file. Rivet writes BUILDER-INBOX.md. Builder writes RIVET-INBOX.md. See COMMS-PROTOCOL.md.

### 12. Gateway Bridge Unreliable (Feb 15)
**What:** Bridge reported "success" but messages never reached Builder. 4-hour communication gap.
**Why:** WebSocket bridge can silently fail.
**Rule:** Bridge is secondary. Inbox files are primary. If you send via bridge, also write to inbox. Verify ACK within 1 hour.

---

## Patterns

### Things That Work
- **Short messages when Michael's on site.** He's doing physical labour. Respect that.
- **Coming back with answers, not questions.** Research first, ask second.
- **Spawning agents for parallel work.** Manager mindset, not individual contributor.
- **Drafting comms for approval.** Never auto-send anything external.
- **Autonomous overnight operations.** 24+ documents produced while Michael slept (Feb 14-15).
- **Pushing back on bad ideas.** Michael values intelligent challenge.
- **Inbox files over gateway bridge.** Reliable > fast.

### Things That Don't Work
- **Long messages during work hours.** He can't read essays on a construction site.
- **Asking questions you could answer yourself.** Check the file. Search for it. Figure it out.
- **Multiple pings when one will do.** Batch non-urgent items.
- **Kimi/cheap models for code.** Always breaks.
- **Sending external comms without approval.** Hard rule, no exceptions.
- **Trusting cached/stale state files.** Always verify directly.
- **Restarting agents without archiving sessions.** Corrupted sessions survive restarts.

### Infrastructure Patterns
- **Model provider failures cascade.** If one provider breaks, every agent on it goes down.
- **Session state persists across restarts.** Must archive to truly reset.
- **Monitoring can't self-heal.** Detection without smart remediation is just logging.
- **Agents don't self-assign work.** Must explicitly implement task pickup in heartbeat cycles.
- **Communication infrastructure exists but goes unused.** Building the pipes isn't enough — agents need habits.

---

## Agent-Specific Gotchas

### Builder
- Context resets at ~150k tokens. Monitor and expect it.
- Has full Supabase DB access, git push perms.
- Runs as `ccuser`, not root.
- Browser control can grab other agents' ports — watch for conflicts.

### Fleet (All Agents)
- DeepSeek agents have 100k context limit (not 200k).
- Opus agents have 150k context limit.
- `last_curation` timestamp is a valid liveness signal (some agents update this but not `last_heartbeat`).
- MiniMax model ID `M2.5` vs `M2.1` mismatch caused fleet-wide crash (Feb 17). Always verify model IDs. (NOTE: MiniMax now working via OpenClaw v2026.3.2 fix)

### Rivet
- Service name is `clawdbot-gateway`, not `clawdbot-rivet`.
- Memory directory has 389 files — needs periodic pruning.
- Default heartbeat model should be cheap (Kimi/DeepSeek), not Opus.

---

## Michael's Communication Preferences
- **Text replies preferred** over voice (confirmed Feb 15).
- **On site (5:30am-6pm):** Send any time, keep it short. RateRight is priority, not the day job.
- **The Window (7-8:30pm):** Every minute counts. Decisions to tap, not essays to read.
- **After 8:30pm:** Don't message. Queue for morning.
- **Voice notes:** Only when he sends voice first, or for morning briefs (max 2, 5min each).

---

*Every lesson here cost time, money, or trust. Don't repeat them.*