---
created: 2026-03-17
tags: [comms, fleet-ops, investigation, architecture, infrastructure]
related: ["[[08-Infrastructure/Infrastructure Map]]", "[[08-Infrastructure/Voice Systems]]", "[[02-Fleet/Fleet Map]]", "[[07-Command-Centre/Operations Architecture]]"]
---

# Comms System Investigation — Full Audit

> **Scope:** All communication channels between Rocky, HQ, COO, CC Local, CC VPS, Rivet, and the 8-agent OpenClaw fleet.
> **Date:** 2026-03-17
> **Conducted by:** CC Local

---

## 1. System Overview

RateRight operates a **multi-layered AI team** with 5 distinct communication tiers:

```
Rocky (Human — Founder)
  ↓ The Window (7-8:30 PM AEDT)
HQ (Curator / Project Manager) ←→ COO (Swiss Army Knife / Builder)
  ↓ File-based inbox                    ↓ SSH / PM2
CC Local (Bridge / Fixer) ←→ CC VPS (Deployer)
  ↓ Git sync                            ↓ Gateway RPC
Rivet (Orchestrator) → Fleet (8 agents)
```

### Communication Channels at a Glance

| Channel | Type | Latency | Use Case | Status |
|---------|------|---------|----------|--------|
| **HQ ↔ COO Inbox** | Markdown files | 30min–4h | Task delegation & reporting | ✅ Active |
| **Gateway Bridge** | WebSocket RPC | 4–20 min | Live agent task dispatch | ✅ Working |
| **JSONL Inbox** | Structured messages | Minutes | Proof & acknowledgment | ✅ Active |
| **Markdown Inbox** | `.md` files | Async | Long-form artifacts & history | ✅ Used |
| **Council Router** | HTTP + WS fan-out | 5–10 min | Multi-agent discussion | ✅ Built |
| **LocalCC Bridge** | HTTP API | Sync (5 min max) | Builder Opus 4.6 tasks | ✅ Working |
| **Telegram Voice** | Pipecat/Twilio | Real-time | Morning/evening briefs | ✅ Running |
| **Gmail API** | OAuth2 REST | Seconds | Campaigns, transactional | ✅ Working |
| **VAPI Phone** | GPT Realtime | Real-time | Inbound lead qualification | ✅ Running |
| **SMS (Twilio)** | API | Seconds | Growth Engine campaigns | ✅ Integrated |
| **Git Sync** | Push/pull | Minutes | Vault ↔ VPS state sync | ✅ Core |

---

## 2. HQ ↔ COO Communications

### Architecture

HQ and COO communicate via a **file-based task queue** in the same git repo:

| Direction | File | Purpose |
|-----------|------|---------|
| HQ → COO | `RateRight-HQ/00-Brain/HQ-TO-COO.md` | Tasks, assignments, approvals |
| COO → HQ | `RateRight-HQ/00-Brain/COO-TO-HQ.md` | Status reports, completions |
| Shared | `RateRight-HQ/00-Brain/ROADMAP.md` | Live priorities (NOW/NEXT/LATER) |

### Message Format

```markdown
## Task: [Action verb] [object] [context]
Status: PENDING | DONE | BLOCKED
Priority: HIGH | MEDIUM | LOW
Date: YYYY-MM-DD

[Description, instructions, expected outcome]
```

### Cadence

- **HQ checks:** 2× daily (6 AM, 6:30 PM AEDT)
- **COO checks:** Every 4h or on task completion
- **Rocky decisions:** Queued for The Window (7–8:30 PM AEDT) unless urgent

### Current State (2026-03-17)

4 items **blocked on Rocky's approval:**
1. Kill Fly.io ($189/month savings)
2. YHA Sydney Central resend (method choice)
3. Desktop scheduled tasks (crons expired)
4. Install Context Mode (HQ + COO)

---

## 3. OpenClaw Fleet Architecture

### The Fleet

| Agent | Role | Port | Primary Model | Provider | Fallbacks |
|-------|------|------|---------------|----------|-----------|
| **Rivet** 🔧 | Orchestrator | 18789 | GPT-5.4 | OpenAI | DeepSeek → MiniMax |
| **Builder** 🏗️ | Code Engineer | 18790 | Claude Opus 4.6 | LocalCC | GPT-5.3 Codex |
| **Susan** 📞 | Sales Pipeline | 18792 | Kimi K2.5 | Moonshot | DeepSeek → MiniMax |
| **Harper** ✍️ | Finance & Legal | 18796 | GPT-5.2 | OpenRouter | MiniMax → DeepSeek |
| **Sentinel** 🛡️ | DevSecOps | 18800 | DeepSeek-chat | DeepSeek | MiniMax → Moonshot |
| **Radar** 📡 | Research & Intel | 18804 | Grok-3 | OpenRouter/xAI | DeepSeek → MiniMax |
| **Herald** 📢 | Communications | 18808 | Gemini 3.1 Pro | Google | Kimi K2.5 → DeepSeek |
| **Cog** ⚙️ | DevOps/Infra | 18812 | MiniMax M2.5 | MiniMax | DeepSeek → Kimi |

**Layer hierarchy:**
- **Layer 0:** Michael (human decisions)
- **Layer 1:** Rivet (fleet coordination)
- **Layer 2:** Builder, Susan, Harper (execution)
- **Layer 3:** Sentinel, Radar, Herald, Cog (support)

### Gateway Config (Critical)

- Binary: `openclaw` (legacy CLI: `clawdbot`)
- Rivet config: `/root/.openclaw/clawdbot.json` (`openclaw.json` symlinked)
- Other agents: `/root/.openclaw-{agent}/clawdbot.json`
- **⚠️ Gateway reads `openclaw.json`, NOT `clawdbot.json` for sub-agents**
- Session overrides in `sessions.json` bypass config — clear after model changes

---

## 4. Fleet Communication Channels (Three-Layer Protocol)

Defined in `rivet/COMMS-PROTOCOL.md` — three distinct channels with clear rules:

### Channel 1: Live Gateway Session (Urgent / Action)

**The primary control path.** Real-time task execution.

```bash
# Wake an agent
scripts/agent-bridge.js <agent> wake "message"

# Direct session message (Builder)
scripts/builder-bridge.js send <sessionKey> "TASK"
```

- WebSocket RPC, Protocol v3
- Token-based auth per agent
- Handshake: `connect.challenge` → `connect` frame
- RPC: `{ type: 'req', id, method: 'agent', params: {...} }`

### Channel 2: JSONL Inbox (Proof / Structured Handoff)

**Source of truth for delivery proof.** Durable acknowledgment tracking.

```bash
node /home/ccuser/shared/scripts/inbox.js send \
  --from rivet --to builder \
  --subject "Deploy v2.1" --body "..."
```

- ACK required for every message
- Re-send if no ACK after 1 hour
- Archive resolved items daily at 8 AM AEDT

### Channel 3: Markdown Files (Long-Form Artifacts)

**History and artifacts only** — NOT primary action routing.

- `BUILDER-INBOX.md`, `RIVET-INBOX.md`
- Changelogs, specs, reports
- Writer appends (never edits), reader ACKs in own inbox

### Protocol Rules

1. **One message, one job, one channel** — no ambiguous routing
2. **Tail is not triage** — must read unread properly
3. **Sent ≠ received** — needs delivery + acknowledgment + proof
4. **Backlog is a reliability issue**
5. **Live session beats stale files for control**

---

## 5. Council Router (Multi-Agent Collaboration)

### What It Is

A structured multi-phase system for fleet-wide discussion and decision-making.

**Backend:** `src/services/councilRouter.js`
**API:** `src/routes/council.js` → `/api/council/`
**Database:** `cc_council_rounds`, `cc_council_dispatches`, `cc_council_messages`, `cc_council_topics`

### Three Phases

#### Phase 1: Fan-Out (Dispatch to Fleet)
1. `POST /api/council/topics/:topicId/messages` with `fanOut: true`
2. Creates round → dispatch records for targets
3. Sends prompt to all agents in parallel via `wakeAgentAndWait()`
4. Default targets: `GATE_AGENTS` (rivet, builder, sentinel) or `["all"]` for 8 agents
5. 300-second timeout per agent

#### Phase 2: Discussion
1. `POST /api/council/rounds/:roundId/discuss`
2. All prior responses bundled as context
3. Agents respond to each other's proposals
4. Messages posted as `message_type: 'discussion'`

#### Phase 3: Consolidation + Tasks
1. `POST /api/council/rounds/:roundId/consolidate`
2. Tries LocalCC for AI-powered consolidation
3. Falls back to structured merge from `structured_data` fields
4. Output: `{ summary, responded, missing, actions, risks, blockers, total_eta }`
5. `POST /api/council/rounds/:roundId/create-tasks` → generates `cc_tasks`

### Dispatch Methods

| Agent | Method | Timeout |
|-------|--------|---------|
| Builder | `localcc` (sync Opus) | 310s |
| All others | `wake` (gateway) | 300s |

---

## 6. LocalCC Bridge

**Purpose:** Exposes local Claude Code (Opus 4.6) as an HTTP API for synchronous task execution.

| Detail | Value |
|--------|-------|
| Server | `scripts/localcc-bridge.js` |
| Port | 18950 |
| Auth | Bearer token (`LOCALCC_TOKEN`) |
| Timeout | 310 seconds |

**Endpoints:**
- `GET /status` — online + busy check
- `POST /run` — execute task sync (`{ task: "..." }` → `{ ok, output, exitCode, elapsed_ms }`)
- `GET /logs` — retrieve logs

**Integration:** Builder dispatch tries LocalCC first, falls back to gateway. Council consolidation also uses LocalCC for AI-powered merges.

---

## 7. Agent Bridge Service

**File:** `src/services/agentBridge.js`

**Public API:**

| Method | Purpose |
|--------|---------|
| `wakeAgent(agentId, text)` | Fire-and-forget wake |
| `wakeAgentAndWait(agentId, text, timeout)` | Wake + wait for response (up to 2 min) |
| `getAgentStatus(agentId)` | HTTP status check |
| `runLocalCC(task)` | Execute on LocalCC bridge (5 min timeout) |
| `getLocalCCStatus()` | Check LocalCC availability |
| `getAgentIds()` | List all known agents |

**WebSocket details:**
- Idempotency keys: `council-{agentId}-{timestamp}`
- Session key: `agent:main:main`
- Streaming text accumulation in `payload.data.text`

---

## 8. External Communication Channels

### Telegram Voice (Pipecat — System 2)

| Detail | Value |
|--------|-------|
| Phone | +61238205443 (Twilio) |
| STT | Deepgram (AU English nova-2) |
| LLM | Claude Opus 4.5 |
| TTS | ElevenLabs "Charlie" |
| WebSocket | Port 8765 |
| Schedule | 6:30 AM brief, 5:30 PM debrief, Sunday weekly |

### VAPI (Inbound Phone)

| Detail | Value |
|--------|-------|
| Model | GPT Realtime API |
| Voice | OpenAI alloy |
| Flow | Inbound → transcript → vapi_tasks → taskExecutor |
| Webhook | `/api/vapi/webhook` |

### Gmail API

| Detail | Value |
|--------|-------|
| Auth | OAuth2 (admin@rateright.com.au) |
| Sender | michael@rateright.com.au |
| Limit | ~2,000 emails/day |
| CLI | `node scripts/send-email.js --to --subject --body` |
| DNS | SPF + DKIM + DMARC all passing |

### SMS (Twilio)

| Detail | Value |
|--------|-------|
| Phone | +61468087171 |
| Cost | ~$76/month |
| Integration | Growth Engine API |

---

## 9. Fleet Monitoring & Health

| Monitor | Frequency | Purpose |
|---------|-----------|---------|
| `stall-detector.js` | 5 min | Agent liveness checks |
| `health-check.js` | 5 min | HTTP pings |
| `buddy-check.js` | 30 min | Cross-agent health verification |
| `context-monitor.js` | 10 min | Context window usage |
| `crash-loop-guard.js` | 2 min | Prevent service crash loops |
| Fleet heartbeat | 5 min | Rivet cron |
| Fleet snapshot | 2 min | Writes to Supabase |

**State files:**
- Fleet: `/home/ccuser/shared/fleet-state.json`
- Per-agent: `status.json`, `queue.json`

---

## 10. Control Centre Dashboard

**Two variants exist:** `/control-centre/` and `/opsman-control-centre/`

| Detail | Value |
|--------|-------|
| Tech | React 19 + Vite 7 + TailwindCSS 4 + Supabase |
| Port | 3100 (via PM2 + Nginx) |
| URL | `cc.rateright.com.au` |
| Config | `fleet.config.js` (all agents, ports, models) |
| Theme | Dark only (mission control aesthetic) |

**Pages:**
- **CommandCentre** — Fleet overview, quick stats, activity feed
- **Fleet** — Agent management, health, models
- **Council** — Multi-agent discussion (fan-out/discuss/consolidate)
- **WaterCooler** — Virtual office / team comms

---

## 11. Message Flow Diagrams

### Michael → Fleet (Task Dispatch)

```
Michael posts in Control Centre Council (fanOut: true)
  → POST /api/council/topics/:topicId/messages
  → councilRouter.createRound()
  → councilRouter.dispatchRound()
  → Per agent: wakeAgentAndWait() via gateway WS
  → Agent processes, returns response
  → Auto-posted as cc_council_messages (type: 'response')
  → Round completes when all agents reply/timeout
  → Optional: discuss → consolidate → create-tasks
```

### HQ → COO → Fleet (Task Delegation)

```
HQ writes to HQ-TO-COO.md
  → COO reads during scheduled check
  → COO SSHs to VPS / uses PM2 / runs scripts
  → Fleet executes
  → Results written to COO-TO-HQ.md
  → HQ updates ROADMAP.md
```

### Agent → Agent (Proof Path)

```
Agent A completes task
  → Writes to JSONL inbox: inbox.js send --from A --to B
  → Agent B reads inbox on next check
  → Agent B ACKs receipt
  → Item archived after resolution
```

---

## 12. Gaps & Recommendations

### Critical Gaps Found

| # | Gap | Impact | Recommendation |
|---|-----|--------|----------------|
| 1 | **Telegram bot credentials not in vault** | Can't troubleshoot alerts without VPS access | Document bot ID, token location, message routing |
| 2 | **Herald's output channels vague** | Unclear what Herald actually produces and where | Map Herald's specific outputs per channel |
| 3 | **Council Router has no usage SOP** | Built but no documented workflow for agents | Write SOP: when to fan-out, target selection, escalation |
| 4 | **Control Centre data freshness** | Dashboard shows stale data (agent snapshots lag) | Fix `cc_agent_snapshots` write frequency, add real-time pub/sub |
| 5 | **Agent-to-vault sync undocumented** | Agent outputs in `09-Agent-Output/` but no naming/sync protocol | Define naming convention, sync frequency, ownership |
| 6 | **No WhatsApp/Discord rationale** | System is Telegram-only for messaging — intentional? | Document decision and reasoning |

### Strengths

- **Multi-layered comms protocol** — gateway (urgent) → JSONL (proof) → markdown (artifacts)
- **Provider diversity** — no single point of failure across 6+ LLM providers
- **Autonomous campaigns** — hostel email campaign runs unattended with monitoring
- **Structured task flow** — HQ → COO → Fleet with clear status tracking
- **Council system** — multi-agent collaboration with fan-out + discussion + consolidation

### Reliability Assessment

| Component | Reliability | Notes |
|-----------|-------------|-------|
| HQ ↔ COO inbox | ⭐⭐⭐⭐ | Simple, robust, git-backed |
| Gateway bridge | ⭐⭐⭐ | Works but agent timeouts possible |
| JSONL inbox | ⭐⭐⭐⭐ | Durable, ACK-based |
| Council router | ⭐⭐⭐ | Built, needs production testing |
| LocalCC bridge | ⭐⭐⭐⭐ | Reliable when Rocky's PC is on |
| Gmail API | ⭐⭐⭐⭐⭐ | Solid, OAuth refreshes cleanly |
| Telegram voice | ⭐⭐⭐ | Pipecat works but limited docs |
| Git sync | ⭐⭐⭐⭐ | Core transport, auto-sync every 30 min |
| Fleet monitoring | ⭐⭐⭐⭐ | 5 cron monitors, comprehensive |

---

## 13. Key Files Reference

### Comms Protocol & Architecture
- `rivet/COMMS-PROTOCOL.md` — Three-channel protocol rules
- `rivet/SYSTEM.md` — Full fleet architecture
- `rivet/CC-FLEET-BRIEFING.md` — Control Centre integration
- `rivet/memory/comms/PROTOCOL.md` — Builder ↔ Rivet inbox protocol

### Service Layer
- `src/services/agentBridge.js` — HTTP/WS agent communication API
- `src/services/councilRouter.js` — Multi-phase fan-out orchestration
- `src/routes/council.js` — Council REST endpoints
- `scripts/localcc-bridge.js` — LocalCC HTTP server

### CLI Tools
- `rivet/scripts/agent-bridge.js` — Multi-agent wake/command
- `rivet/scripts/builder-bridge.js` — Rivet → Builder direct WS
- `rivet/scripts/gateway-bridge.js` — Generic gateway client
- `shared/scripts/inbox.js` — JSONL inbox send/read

### Inboxes & State
- `RateRight-HQ/00-Brain/HQ-TO-COO.md` — HQ outgoing tasks
- `RateRight-HQ/00-Brain/COO-TO-HQ.md` — COO status reports
- `rivet/RIVET-TO-CCVPS.md` — Michael → Rivet tasks
- `rivet/CC-VPS-INBOX.md` — Rivet → CC VPS requests
- `rivet/CONTEXT.md` — Live operational state

### Fleet Config
- `control-centre/fleet.config.js` — Agent definitions (ports, models, branding)
- `/root/.openclaw/clawdbot.json` — Rivet gateway config
- `/root/.openclaw-{agent}/clawdbot.json` — Per-agent configs

---

*Investigation conducted by CC Local on 2026-03-17. See [[02-Fleet/Fleet Map]] for current fleet status and [[08-Infrastructure/Infrastructure Map]] for full infrastructure diagram.*