# OpsMan Recovery Plan — 2026-03-04

Owner: Builder  
Sponsor: Michael (Rocky)  
Objective: Restore OpsMan to reliable day-to-day operation, remove known blockers, and harden to product-grade reliability.

## Scope (full system)
1. Runtime/services (systemd, ports, process health)
2. Model/provider/auth chain
3. Session + memory + context lifecycle
4. Filesystem/config hygiene
5. Comms channels (Telegram/WhatsApp/voice)
6. Inter-agent comms (agent-bridge, inbox, queue)
7. Fleet coordination + heartbeats
8. Ops control center/dashboard + observability
9. Security hardening baseline
10. Runbooks, SLOs, and ongoing maintenance

---

## Current Findings Snapshot

### Working now
- All 8 agent gateways are running under systemd and listening on expected loopback ports.
- Telegram channel is online across fleet.
- Rivet now runs MiniMax M2.5 with valid MiniMax auth profile.
- Shared JSONL inbox system works for cross-agent messages.

### Not working / degraded
- `scripts/agent-bridge.js wake` path was broken (404: missing `cron` tool).
- Some session histories are bloated/stale and can trigger quality drops or context overflow.
- Config drift risk exists across multiple config files/state dirs.
- Security posture warnings still present (permissions/token hygiene).
- Fleet has stale unread backlog and high alert/decision noise.

---

## Layered Work Plan

## Phase 1 — Critical restore (P0) [IN PROGRESS]

### 1.1 Fix inter-agent wake path
- [x] Replace unsupported `cron wake` in `scripts/agent-bridge.js` with inbox-based wake (`inbox.js send`).
- [x] Validate wake to rivet + builder succeeds.
- [ ] Validate wake success for all agents.

### 1.2 Rivet model/fallback/auth stabilization
- [x] Set Rivet primary model to `minimax/MiniMax-M2.5`.
- [x] Ensure MiniMax auth profile exists in Rivet auth store.
- [x] Remove invalid fallback references causing 400 invalid model id errors.
- [ ] Live message sanity check with Michael on Rivet (3 consecutive normal replies).

### 1.3 Stop active failure loops
- [ ] Identify top 3 agents with repeated failover/context errors.
- [ ] Reset/rotate affected sessions safely.
- [ ] Confirm no repeated provider failover errors in journald for 30+ min.

Exit criteria P0:
- Wake path works for all agents.
- Rivet responds normally (no invalid model errors).
- No repeated critical provider failures in logs.

---

## Phase 2 — Reliability hardening (P1)

### 2.1 Config consistency checks
- [ ] Build `scripts/opsman-config-audit.js` to verify per agent:
  - service exists/running
  - port listening
  - primary/fallback model IDs valid for configured provider
  - auth profile exists for primary provider
  - workspace path correctness
- [ ] Add pass/fail summary output (machine-readable JSON + human table).

### 2.2 Session hygiene controls
- [ ] Define session thresholds (max age, max message count, max token depth).
- [ ] Add safe archival helper for stale/bloated sessions.
- [ ] Add recommended model upgrades for high-context agents.

### 2.3 Heartbeat and queue integrity
- [ ] Ensure every agent updates status.json each heartbeat.
- [ ] Add stale heartbeat detector + auto-escalation to Cog/Sentinel.
- [ ] Clear unread backlog via ack policy and escalation routing.

Exit criteria P1:
- Config audit script passes across all 8 agents.
- No unresolved context-overflow loops.
- Heartbeats consistently fresh.

---

## Phase 3 — Control Center (P2)

### 3.1 Build Ops Control Dashboard (single pane)
- [ ] Fleet status matrix (service, port, model, auth, heartbeat age)
- [ ] Comms matrix (telegram/whatsapp/voice status)
- [ ] Inter-agent comms test panel (wake/ping/send checks)
- [ ] Session health panel (active sessions, token pressure, compactions)
- [ ] Incident timeline panel (last 24h errors)

### 3.2 Add operational test suite
- [ ] `npm run opsman:test` with:
  - bridge wake all agents
  - status all agents
  - model/auth lint
  - queue/inbox sanity checks

Exit criteria P2:
- Dashboard shows accurate live state.
- One-command ops test runs green.

---

## Phase 4 — Product-grade ops (P3)

### 4.1 SLOs and alerting
- [ ] Define SLOs:
  - wake success >= 99.5%
  - message delivery success >= 99.9%
  - heartbeat freshness < 10 min
  - MTTR stuck-agent < 5 min
- [ ] Build SLO monitors and alert routing.

### 4.2 Runbooks and drills
- [ ] Create runbooks for:
  - invalid model/fallback
  - auth expiry/failover
  - context overflow
  - bridge outage
  - channel outage
- [ ] Run monthly chaos drills.

Exit criteria P3:
- Measurable reliability with on-call runbooks.
- Demonstrably product-ready OpsMan operations.

---

## Risks
- Config changes across multiple agent state dirs can drift without automation.
- Model provider auth changes may fail silently until runtime.
- Session reset actions can interrupt active workflows if not coordinated.

## Mitigations
- Use config audit scripts + backups before edits.
- Enforce model/auth lint before service restart.
- Use append-only incident logs and rollback checkpoints.

---

## Change Log (today)
- Fixed `agent-bridge.js` wake path to use inbox-based wake, replacing unsupported cron invocation.
- Stabilized Rivet model/auth path around MiniMax M2.5.
- Restarted Rivet gateway after config fixes.

---

## Definition of Done (10/10 target)
- Fleet comms robust (no wake/path failures).
- Model/auth/session failures auto-detected before user impact.
- Dashboard gives real-time truth of system health.
- Runbooks + SLOs make outages fast to detect and fast to recover.
- OpsMan reliable enough to be sold as core product infrastructure.