# Infrastructure Lessons Learned — Agent Recovery
*Written 2026-02-16 by Builder after Rivet + Susan outage*

## The Incident
Rivet and Susan both went offline. Root cause: Susan's service was misconfigured and kept stealing Rivet's port, killing Rivet's process in the process.

## Key Issues & Fixes

### 1. Port Conflicts Between Agents
**Problem:** All three agents (Rivet, Builder, Susan) run as root on the same VPS. Clawdbot defaults to port 18789. Without explicit port config, agents fight over the same port.

**Fix:** Each agent MUST have `gateway.port` set in its config:
- Rivet: 18789 (default, `/root/.clawdbot/clawdbot.json`)
- Builder: 18790 (`/root/.clawdbot-builder/clawdbot.json`)
- Susan: 18792 (`/root/.clawdbot-susan/clawdbot.json`)

**Rule:** Never rely on `--port` CLI flag or `CLAWDBOT_PORT` env var alone. The config file `gateway.port` key is what clawdbot actually reads.

### 2. The `--profile` Flag is Essential
**Problem:** Susan's wrapper ran `clawdbot gateway run` without `--profile susan`. Without this, clawdbot loads the DEFAULT config (`/root/.clawdbot/clawdbot.json` = Rivet's config), ignoring Susan's config entirely.

**Fix:** Every agent wrapper MUST use `--profile <name>`:
```bash
exec /usr/bin/clawdbot --profile susan gateway run --port 18792 --bind 127.0.0.1
exec /usr/bin/clawdbot --profile builder gateway run --port 18790 --bind 127.0.0.1
```

The `--profile <name>` flag tells clawdbot to use `/root/.clawdbot-<name>/clawdbot.json`.

### 3. Shared Lock Files Cause Cross-Agent Blocking
**Problem:** All root-user gateways share lock files in `/tmp/clawdbot-0/`. When Susan tried to start, it saw Builder's lock and refused — even though they're on different ports.

**Fix:** The `--profile` flag creates separate lock namespaces per config hash. Stale locks from crashed agents must be cleaned: `rm -f /tmp/clawdbot-0/gateway.*.lock`

**Monitoring tip:** If an agent won't start and the port is free, check for stale locks first.

### 4. Wrapper Scripts Kill Other Agents
**Problem:** Susan's wrapper script had an "orphan cleanup" that checks its port, finds nothing, but then the gateway startup finds Rivet on 18789 (because it's reading Rivet's config). Susan's wrapper then kills whatever it finds on the target port.

**Fix:** Remove aggressive orphan-killing from wrappers. Use `--force` flag if needed, but better to have correct config so port conflicts don't happen.

### 5. Config Validation Matters
**Problem:** Susan's config had invalid values (`tts.auto: "disabled"` should be `"off"`, `topicFilter` is not a valid key, `gateway.http.port` is not a valid key). Clawdbot refused to start with invalid config.

**Fix:** After editing any config, validate with: `clawdbot --profile <name> doctor --fix`

Valid `tts.auto` values: `"off"`, `"always"`, `"inbound"`, `"tagged"`

## Architecture: Current Agent Setup

| Agent | Service | Profile | Port | Config | Workspace |
|-------|---------|---------|------|--------|-----------|
| Rivet | clawdbot-gateway | (default) | 18789 | /root/.clawdbot/clawdbot.json | /home/ccuser/rateright-growth/rivet |
| Builder | clawdbot-builder | builder | 18790 | /root/.clawdbot-builder/clawdbot.json | /home/ccuser/the-50-dollar-app |
| Susan | clawdbot-susan | susan | 18792 | /root/.clawdbot-susan/clawdbot.json | /home/ccuser/susan |

## Recovery Checklist (For Infra Bot)

When an agent is down:
1. `systemctl status clawdbot-<service>.service` — check why it died
2. `ss -tlnp | grep <port>` — is the port held by something else?
3. `ls /tmp/clawdbot-0/gateway.*.lock` — stale locks?
4. If stale lock: `rm -f /tmp/clawdbot-0/gateway.<hash>.lock`
5. If port stolen: identify the thief process, check its `--profile` flag
6. `systemctl restart clawdbot-<service>.service`
7. Verify: `journalctl -u clawdbot-<service>.service --since "30 seconds ago" | grep "listening on ws"`

## Startup Order
Start agents in this order to avoid port conflicts:
1. Rivet (18789) — first, as others may depend on it
2. Builder (18790)
3. Susan (18792)

## Key Commands
```bash
# Check all agents
for svc in clawdbot-gateway clawdbot-builder clawdbot-susan; do echo "$svc: $(systemctl is-active $svc)"; done

# Check ports
ss -tlnp | grep -E "1879[0-9]"

# Clear stale locks
rm -f /tmp/clawdbot-0/gateway.*.lock

# Restart an agent
systemctl restart clawdbot-<service>.service

# Check logs
journalctl -u clawdbot-<service>.service --since "5 minutes ago" --no-pager

# Validate config
clawdbot --profile <name> doctor --fix
```