# Fleet Remediation Plan — 2026-03-09

## Objective
Restore trustworthy multi-agent operation by fixing the health model, fleet-state generation, CLI stability, and live queue discipline.

## What was verified live
- All core agent services are running under systemd: builder, susan, harper, sentinel, radar, herald, cog.
- All local ports respond on 18789/18790/18792/18796/18800/18804/18808/18812.
- Builder status is fresh at `/home/ccuser/the-50-dollar-app/status.json`.
- Susan, Radar, Sentinel, Cog have fresh/fresher status updates.
- Harper and Herald remain stale at `status.json` level despite services being up/restarted.
- `fleet-cli.js status` crashes on `fleet.tasks.in_progress` when `fleet.tasks` is missing.
- `fleet-state.json` is not a trustworthy source of truth; it can be stale/incomplete and disagree with status files.
- Agent queues are stale and inconsistent across the fleet.

## Root causes
1. No single source of truth for fleet health.
2. Heartbeat architecture is too weak: service up != heartbeat fresh != agent operational.
3. Operator tooling is brittle (`fleet-cli.js` crashes on missing optional state).
4. Task/queue layer is stale and does not reflect current live work.
5. Path/schema standardization is incomplete across agents (Builder path differs, fields/timestamps vary).

## Fix plan

### Phase 1 — Stop the bleeding (tonight)
1. Patch `fleet-cli.js` to be null-safe when `fleet.tasks` is absent.
2. Patch any fleet scripts assuming `fleet.tasks.*` exists.
3. Introduce canonical empty defaults for fleet state:
   - `tasks: { in_progress: [], blocked: [], backlog: [] }`
   - `alerts: []`
   - `decisions_needed: []`
4. Stop marking agents healthy purely from service/process state.
5. Mark Harper/Herald as degraded until they prove fresh heartbeat writes.

### Phase 2 — Rebuild health truth
Create one canonical fleet snapshot generated from:
- systemd/service state
- gateway/port reachable
- fresh `status.json`
- queue/current task freshness

Health must split into explicit dimensions:
- process_up
- gateway_reachable
- heartbeat_fresh
- work_state_fresh

Derived status labels:
- down
- up_unresponsive
- up_reporting
- up_reporting_with_work
- blocked

### Phase 3 — Standardize agent contract
Every agent must write the same schema:
```json
{
  "agent": "name",
  "status": "active|idle|blocked|stalled",
  "current_task": "string",
  "last_updated": "iso",
  "last_heartbeat": "iso",
  "progress": 1
}
```

Requirements:
- consistent timestamp format
- consistent file path handling per agent (Builder explicitly mapped)
- null-safe optional fields
- heartbeat write on every successful cycle

### Phase 4 — Rebuild queue discipline
- Refresh every `queue.json`
- Archive stale completed tasks
- Ensure every active agent has a live queue item or is explicitly idle
- Require `current_task` to map to a real queue item during business hours

## Work assignments

### Builder
1. Patch `fleet-cli.js` null safety.
2. Patch fleet state readers/writers to tolerate missing sections.
3. Implement canonical fleet-state generator from live sources.
4. Standardize agent status contract/path mapping.
5. Update any dashboard/doctor/CLI consumers to read the same source of truth.
6. Deliver a short changelog + proof of before/after behaviour.

### Sentinel
1. Prove service/process truth with logs/process checks.
2. Investigate why Harper and Herald can be up but fail heartbeat.
3. Confirm whether issue is init-path, heartbeat-path, or write-path.
4. Verify recovery only with fresh status writes, not service state.

### Cog
1. Continue manual reconciliation until tooling is fixed.
2. Maintain a list of false-green cases.
3. Escalate only proven mismatches with timestamps.

### Rivet
1. Refresh queues so the fleet has real live work.
2. Stop treating service-up as healthy.
3. Review outputs against the new standard.

## Success criteria
- `fleet-cli.js status` never crashes on incomplete state.
- `fleet-state.json` reflects live truth within one heartbeat cycle.
- Harper/Herald either produce fresh heartbeat or are explicitly degraded.
- Every active agent has a live queue item or explicit idle state.
- Dashboard/CLI/doctor agree on the same underlying health truth.

## Immediate priority
1. Null-safe CLI + fleet state defaults.
2. Harper/Herald heartbeat recovery diagnosis.
3. Canonical health model.
4. Queue rebuild.