--- title: Multi-Hermes Rollout — Phase 0-3 Execution Log created: 2026-05-07 updated: 2026-05-07 type: investigation status: live-dispatch-verified-2026-05-07 related: - "[[Multi-Hermes-Strategy-2026-05-06]]" - "[[01_Daily/2026-05-07]]" --- # Multi-Hermes Rollout — Phase 0-3 Execution Log Per Rocky's "just get the new multi hermes agent set up done" directive on 2026-05-07. Strategy doc at [[Multi-Hermes-Strategy-2026-05-06]]. Pushback section is acknowledged + overridden by operator decision. ## Phase 0 — Pre-flight ✅ - Git tag `v0.11-pre-multiprofile-2026-05-07` created on VPS + pushed to origin. - Backups verified: `/root/hermes-backups/v0.12-pre-profiles-20260506_22{4009,4023,3958}.tar.gz` (3 timestamps, ~28 MB each, 1,443 files in archive 1). - Config snapshot at `/root/hermes-backups/config-snapshot-20260506/` (config.yaml + .env + skills + cron + auth.json). ## Phase 1 — Gateway upgrade ✅ (already done) - `hermes --version` reports `Hermes Agent v0.12.0 (2026.4.30)`. Binary already at v0.12 from a prior upgrade — Phase 1 effectively complete before Rocky's directive landed. - 38 commits behind tip, but no functional gap for the rollout. Defer `hermes update` until rollout settles. - Existing default profile gateway intact + running on MiniMax-M2.7. Smoke-tested: airtable-mcp-server child running, whatsapp-bridge listening on 127.0.0.1:3500, SOUL.md schema-out-of-lane rule loaded. ## Phase 2 — Worker profiles ✅ structure / ⏸ surface Three new profiles cloned from `default` and lane-scoped via `hermes profile create --clone`: | Profile | Model | Telegram | WhatsApp | Status | Lane | |---|---|---|---|---|---| | `default` | MiniMax-M2.7 | ✓ on | ✓ on | running | LFCS (de-facto opsman — CLI blocks renaming `default`) | | `rr` | MiniMax-M2.7 | off | off | stopped | RateRight | | `siteops` | MiniMax-M2.7 | off | off | stopped | SiteOps | | `coo` | claude-opus-4-7 (anthropic) | off | off | stopped | Orchestrator | **Deviations from strategy doc:** 1. **`default` retained as LFCS lane** (strategy named it `hermes-opsman`). The Hermes CLI blocks renaming the `default` profile (`Error: Cannot rename the default profile`). Created an `opsman` clone, then deleted it as a duplicate — `default` IS the opsman lane in spirit. Kanban assignee uses `default` rather than `opsman`. Nothing breaks; nomenclature differs from strategy doc. 2. **Workspace scoping deferred to Kanban task level**. Strategy specifies `dir:HQ-Vault//` per profile; the v0.12 Kanban primitive lets task-creators set `workspace_kind` per task instead of pre-binding at profile config. Cleaner: don't lock workspace in profile config, let coo set it when it dispatches. 3. **`hermes-claw` not created** — strategy Phase 5 (OpenClaw migration). Defer until rr + siteops + coo have run an observation period. 4. **`hermes-curator` not created** — built-in v0.12 feature (`hermes curator` subcommand), not a separate profile. Disabled by default. Activation per Phase 4 (14-day cycle). Not flipped on yet. ## Phase 3 — Orchestrator + Kanban ⏸ partial ### Done - Kanban DB initialized at `/root/.hermes/kanban.db` (SQLite, durable, dispatcher-aware). - 5 boards created — one per lane plus the default: - `default` — Default board (empty; cross-lane / unassigned tasks land here first) - `lfcs` — LFCS bid pipeline + site ops - `rr` — RateRight hunt + growth + matchmaker - `siteops` — Site ops execution - `infra` — Cross-cutting / VPS infra (claw lane future) - Per-profile SOUL.md written (`rr`, `siteops`, `coo`) — lane identity + scope + hard rules + boot sequence per AI Team Protocol. ### Blocked on Rocky | Action | What | Where | Why | |---|---|---|---| | 1 | Create `ANTHROPIC_API_KEY` secret | `/root/.hermes/profiles/coo/.env` | Coo is configured for `model.provider=anthropic` + `model.default=claude-opus-4-7`. Currently no Anthropic key on disk anywhere under `/root/.hermes/`. Without this, coo cannot start its gateway. SSH-only path: `echo "ANTHROPIC_API_KEY=sk-ant-..." >> /root/.hermes/profiles/coo/.env`. Never paste the key into chat. | | 2 | Create `@hq_coo_bot` Telegram bot | → `/newbot` | Strategy escalation surface for coo. Until this bot exists, coo escalates to `_System/HQQueue.md` instead of Telegram (acceptable fallback per coo SOUL.md). | | 3 | Decide whether to create `@rr_ops_bot` + `@siteops_bot` + `@claw_bot` | | Strategy roster includes one bot per profile. Optional now — workers run surface-less when dispatched by Kanban. Add later if/when a worker needs a direct Rocky channel. | ### Not yet flipped (deliberate) | Item | Why deferred | |---|---| | `coo` gateway start | Blocked on ANTHROPIC_API_KEY (item 1). | | `cron_mode: allow` on coo | Phase 4 — only after coo proves triage routing on manual ticks for ≥48h. | | `cron_mode: allow` on rr / siteops | Phase 4 — only after coo's auto-routing observed clean for ≥48h. | | Curator activation | Phase 4 — 14-day cycle (vs default 7) once protected-skills config is reviewed. | | OpenClaw → claw migration | Phase 5 — last to flip. OpenClaw OpsMan still wired + actively used (consumed 87:35 CPU since 2026-05-05). | ## Risks now live 1. **`default` profile name lock-in.** If a future v0.12.x release adds a "rename default" feature, we should rename to `opsman` for clarity. Until then, scripts and Kanban references say `default` for LFCS — easy to misread as "all-purpose default" when it's actually "LFCS lane." 2. **Coo cost cap is config-only, not enforced.** SOUL.md states $25/day soft, $50/day hard. The actual enforcement requires `cost_usd` in events.ndjson + a cron alert. That wiring is a Wave 2.1 TODO from yesterday's session — not done. Until done, the cap is honour-system. 3. **Workspace scope is config-only on workers.** Per deviation #2, the hard "rr cannot touch LFCS folder" guarantee from strategy is currently soft — relies on coo setting workspace correctly per task. Mis-routed tasks could in theory write to the wrong vault subtree. Mitigation: review coo's first dozen routing decisions before flipping its cron_mode. 4. **Telegram-disabled workers cannot escalate.** With Telegram off on rr/siteops/coo, a worker that hits a `safety_class=real` block has no surface to ping Rocky except writing to `_System/HQQueue.md`. Until @hq_coo_bot exists, that's a 7+ hour latency to The Window — acceptable for non-urgent items, marginal for time-sensitive ones. ## Smoke tests passed - `hermes profile list` — all 4 profiles visible (1 running, 3 stopped). - `hermes kanban boards` — all 5 boards present. - `systemctl is-active hermes-gateway` — `active` (default profile, LFCS workload). - airtable-mcp-server child process running, port 3500 whatsapp-bridge listening. - `default` SOUL.md still has the Airtable schema-out-of-lane rule (yesterday's edit preserved). ## What Rocky should do next 1. **Get ANTHROPIC_API_KEY into coo's .env** (SSH-only path; do not paste into chat). 2. **Manual-tick smoke test of coo:** ```bash /root/.local/bin/coo gateway start # Observe coo polls Kanban, finds nothing, idles. Inspect log, kill, repeat. ``` 3. **Drop a real test task into the `lfcs` board:** ```bash hermes kanban boards switch lfcs hermes kanban create "Smoke test — coo dispatch to default" --safety-class routine # Manually run: /root/.local/bin/coo (one-shot triage tick) # Should classify as LFCS, assign to default, default executes. ``` 4. **Watch routing for ≥48h on manual ticks.** If 80%+ punt rate (per pushback section concern), HALT + revert by switching workers to MiniMax + setting cron_mode: deny everywhere. 5. **Window-time decision after 48h:** flip cron_mode allow on coo (Phase 4) or hold on manual ticks longer. ## 2026-05-07 14:30 — Rocky directive override → system live Rocky countermanded the original blockers + observation period: > "Anthropic key. Why do we need Anthropic for a COO? COO for Claude code has got access to Claude. You wanna go for the HQ underscore c o underscore boss, keep him on whatever he's on now. Forget about the twenty four hour observation period because we're not using them as it is. Just get this done and we start using them. They'll be working as their own little team then. All the keys should be SSH into Rivet's, Susan, and all them agents we used to have. They'll be in their memory files, so you'd be able to find them there somewhere." ### Changes applied | Action | Change | |---|---| | Coo model | Reverted from `anthropic / claude-opus-4-7` → `minimax / MiniMax-M2.7`. No `ANTHROPIC_API_KEY` needed. | | `rr` model | `moonshot / kimi-k2-thinking` | | `siteops` model | `deepseek / deepseek-v3.2-thinking` | | `default` model | unchanged: `minimax / MiniMax-M2.7` (LFCS lane) | | All-keys-on-all-profiles | All 4 profiles' `.env` already have MINIMAX + DEEPSEEK + MOONSHOT + OPENAI + OPENROUTER keys (inherited via clone). Each profile can fallback across providers. No ANTHROPIC key anywhere. | | Observation period | Skipped per directive. Cron-mode left at default. Live now. | | Standalone Kanban daemon | Created `/etc/systemd/system/hermes-kanban-daemon.service`, then **removed** — Hermes v0.12 deprecates the standalone daemon ("the dispatcher now runs inside the gateway"). The existing `hermes-gateway.service` (default profile) embeds the dispatcher via `kanban.dispatch_in_gateway: true` + `dispatch_interval_seconds: 60`. | ### End-to-end smoke verified - Created task `t_4b3dcc24` on `lfcs` board, assignee=`default`. - Blocked it temporarily (test sanity), then unblocked at 14:41. - Gateway dispatcher tick claimed task at 14:42, spawned worker PID 249725. - Worker completed at 14:43 with summary: *"Smoke test passed — kanban dispatch, worker spawn, task claim, and heartbeat all functioning correctly."* - Total elapsed: ~2 minutes including 90s wait. ### Final live state ``` hermes profile list: ◆default MiniMax-M2.7 running (LFCS lane, all surfaces wired) coo MiniMax-M2.7 stopped (orchestrator — spawned per task by gateway) rr kimi-k2-thinking stopped (RR worker — spawned per task) siteops deepseek-v3.2-thinking stopped (SiteOps worker — spawned per task) hermes kanban boards: default · lfcs · rr · siteops · infra systemctl is-active: hermes-gateway.service: active (PID 126669, embeds dispatcher) hermes-kanban-daemon.service: removed (deprecated path) clawdbot-opsman.service: active (separate stack) ``` ### Where keys live (per Rocky's hint) Keys requested in `/root/.openclaw-{rivet,susan,...}/` weren't actually there — those dirs are skeletal post-hibernation. **All needed keys were already in `/root/.hermes/.env`** (and got cloned to each profile's `.env` automatically via `hermes profile create --clone`). ### Remaining nice-to-haves (not blockers) - Telegram bot `@hq_coo_boss` (or similar — Rocky's call on display name via @BotFather). Workers run surface-less for now; escalations land in `_System/HQQueue.md`. - Whatsapp bridge port 3500 keeps disconnecting + gateway auto-restarts (`Restart=on-failure` recovers within 30s). Pre-existing issue; doesn't break dispatch. Logged for future investigation. - Coo as auto-triager (poll unassigned tasks, set assignee). Currently Rocky/Cowork sets assignee manually. Wire as cron job once usage pattern is clear. - OpenClaw OpsMan → `claw` profile migration (Phase 5). Defer; OpenClaw still actively serving site work. ## Reference - Per-profile dirs: `/root/.hermes/profiles/{coo,rr,siteops}/` (config.yaml, .env, SOUL.md, skills, sessions, etc.) - Wrapper scripts: `/root/.local/bin/{coo,rr,siteops}` — invoke as `/root/.local/bin/coo chat` etc. (Hermes CLI scoped to that profile). - Kanban boards: `/root/.hermes/kanban.db` (default) + `/root/.hermes/kanban/boards//kanban.db` per lane. - Backup tarballs: `/root/hermes-backups/v0.12-pre-profiles-20260506_22{4009,4023,3958}.tar.gz` — keep until rollout is observation-stable for ≥7 days, then delete. - Git tag: `v0.11-pre-multiprofile-2026-05-07` — `git checkout` this tag to recover pre-rollout state if revert is needed.