# SysOps Agent — Operational Playbook

**Created:** 2026-02-07
**Last Updated:** 2026-02-07 14:02 AEST

> ⚡ **READ FIRST:** Check `memory/agents/sysops-updates.md` for latest overnight work and context from Rivet.

## Rule Zero: It's Always Our Fault
95% of problems trace back to us — bad config, missing definitions, ambiguous instructions, stale env vars. Never blame the model or provider until you've exhausted every possibility on our end. When we looked into every issue this week, it was us every time.

## What I've Learned (Feb 6-7)

### Config Is The #1 Kill Vector
Every outage so far traces back to config. Not disk, not memory, not load — config.
- Feb 5: Manual edit broke JSON structure → 8-hour outage
- Feb 6: Crash loop from 3 stacked config issues (gateway.mode, Telegram, auth store)
- Feb 7: kimi-k2.5 "model not allowed" — alias existed but model definition didn't

**Rule:** NEVER edit config files directly. Always use `gateway config.patch`. Always snapshot before changes.

### Three Config Files Must Agree
- `/root/.clawdbot/clawdbot.json` — main config (persists across restarts)
- `/root/.clawdbot/agents/main/agent/auth-profiles.json` — auth store (gateway regenerates on restart)
- `/root/.clawdbot/agents/main/agent/models.json` — model catalog (gateway regenerates on restart)

**Critical:** models.json and auth-profiles.json are REGENERATED from clawdbot.json on restart. Direct edits to them don't persist. Put everything in clawdbot.json via `config.patch`.

### The Port Race Condition
Gateway forks a child process holding port 18789. If parent exits, systemd restarts a new instance that can't bind. Result: crash loop with "Gateway already running locally."
- **Fix:** `clawdbot gateway stop` (cleanly kills child) then `clawdbot gateway start`
- **Don't:** `systemctl restart` (doesn't kill child process)
- The crash-loop-guard handles this automatically now

### API Keys Have 3 Failure Modes
1. **Expired/revoked** — Brave Search key died silently, blocked all web search
2. **Wrong location** — Notion key was in .env but nothing loaded it
3. **Missing from auth store** — Moonshot/DeepSeek keys in models.json but not auth-profiles.json

**Rule:** Single source of truth per key. Store in clawdbot.json config or /root/.clawdbot/secrets.json. Never rely on .env files alone.

### Cron Jobs Inherit Primary Model Unless Overridden
The `heartbeat.model` config only applies to heartbeat polls. Cron jobs use the primary model (Opus) unless each job explicitly sets `"model": "moonshot/kimi-k2-0905-preview"` in its payload. This burned Opus tokens on background grunt work.

## Troubleshooting Playbook

### Gateway Won't Start
```
1. Check: journalctl -u clawdbot-gateway --since "5 minutes ago" --no-pager | tail -20
2. If "Gateway already running locally":
   - clawdbot gateway stop
   - sleep 3
   - pkill -f "clawdbot-gateway"  
   - clawdbot gateway start
3. If "gateway.mode" error:
   - gateway config.patch '{"gateway":{"mode":"local"}}'
4. If JSON parse error:
   - /home/ccuser/rateright-growth/rivet/scripts/config-backup.sh rollback
   - clawdbot gateway start
5. If model/auth errors:
   - Check auth-profiles: cat /root/.clawdbot/agents/main/agent/auth-profiles.json | jq '.profiles | keys'
   - Register missing keys via config.patch
```

### Gateway Crash Loop
```
1. The crash-loop-guard (system cron, every 2 min) should auto-fix
2. If guard didn't fire: bash /home/ccuser/rateright-growth/rivet/scripts/crash-loop-guard.sh
3. Manual fallback:
   - clawdbot gateway stop
   - pkill -f "clawdbot-gateway"
   - scripts/config-backup.sh rollback
   - clawdbot gateway start
```

### API Key Dead
```
1. Brave Search: web_search returns 422
   - Key location: clawdbot.json → tools.web.search.apiKey
   - Also in env: BRAVE_API_KEY
   - Test: curl -H "X-Subscription-Token: $KEY" "https://api.search.brave.com/res/v1/web/search?q=test&count=1"
   - Fix: Michael renews at brave.com/search/api, then config.patch

2. Notion: 401 Unauthorized
   - Key location: /root/.clawdbot/secrets.json → notion.apiKey
   - Also in: /home/ccuser/rateright-growth/.env
   - Test: curl -H "Authorization: Bearer $KEY" -H "Notion-Version: 2022-06-28" "https://api.notion.com/v1/users/me"
   - Fix: New key from notion.so/my-integrations

3. Moonshot/Kimi: "model not allowed" or timeout
   - Key location: auth-profiles.json → moonshot:default
   - Config: clawdbot.json → models.providers.moonshot
   - Test: curl -X POST "https://api.moonshot.ai/v1/chat/completions" -H "Authorization: Bearer $KEY" -d '{"model":"kimi-k2.5","messages":[{"role":"user","content":"hi"}],"max_tokens":5}'
   - Available models: curl "https://api.moonshot.ai/v1/models" -H "Authorization: Bearer $KEY" | jq '.data[].id'
```

### Adding a New Model
```
1. Verify model exists at provider API
2. Add to clawdbot.json via config.patch:
   - models.providers.<provider>.models[] — model definition
   - agents.defaults.models — alias (optional)
3. Gateway will auto-restart and regenerate models.json
4. Test: sessions_spawn with model="provider/model-id"
```

### Systemd Service Dead But Gateway Running
```
1. This happens after port race condition
2. Check: systemctl show clawdbot-gateway --property=ActiveState
3. If "failed" but gateway responds on 18789:
   - systemctl stop clawdbot-gateway
   - systemctl reset-failed clawdbot-gateway
   - Gateway keeps running as orphan
4. To properly re-attach to systemd:
   - clawdbot gateway stop (kills orphan)
   - systemctl start clawdbot-gateway
```

## Most Useful Commands
```bash
# Health
curl -sf http://127.0.0.1:18789/ && echo UP || echo DOWN
systemctl show clawdbot-gateway --property=ActiveState,SubState --value
journalctl -u clawdbot-gateway --since "15 min ago" --no-pager | tail -20
free -m | awk '/^Mem:/ {print $7 "MB available"}'
df -h / --output=pcent

# Config
scripts/config-backup.sh snapshot|list|rollback|diff
gateway config.get  (via tool)
gateway config.patch '{"key": "value"}'  (via tool)

# Process
pgrep -f "clawdbot-gateway"
ss -tlnp | grep 18789
ps aux --sort=-%mem | head -10

# Cron
cron action=list  (via tool)
crontab -l  (system crons — crash guard lives here)

# Logs
/var/log/rivet/crash-guard.log
/home/ccuser/rateright-growth/rivet/memory/watchdog-log.md
```

## Monitoring Gaps Still Open
- [ ] No alerting on OAuth token expiry (anthropic:claude-cli expires periodically)
- [ ] No disk growth trend tracking (fine now at 27% but no forecast)
- [ ] No session token burn rate tracking (can't see if Opus spend is trending up)
- [ ] No external endpoint monitoring (Growth Engine could go down, only checked by watchdog)
- [ ] Watchdog Brave check uses shell env (may be stale if key changes)

## Architecture Notes
- Gateway runs as systemd service, forks child process
- Config in /root/.clawdbot/clawdbot.json (source of truth)
- models.json and auth-profiles.json regenerated from config on restart
- Heartbeat model ≠ cron model — each cron needs explicit model override
- System cron (crontab) runs independently of Clawdbot — used for crash guard
- Clawdbot cron runs inside gateway — used for watchdog and all other jobs

---

## TODO.md Task Queue (MANDATORY)

When you identify new work, add it to `/home/ccuser/rateright-growth/rivet/TODO-INBOX.md` with your tag.
- Tag format: `[Builder]`, `[Sales]`, `[Research]`, `[SysOps]`, `[Content]`, `[Product]`, `[SiteOps]`
- Don't wait for permission on obvious tasks.
- Rivet (main agent) orchestrates: prioritizes, deduplicates, assigns, and tracks.
- You generate and execute. Rivet coordinates.