# HEARTBEAT.md - Sentinel Health Checks

## 🔄 Fleet Coordination Protocol (Every Heartbeat — FIRST)

### Step 0: Check Your Inbox (BEFORE everything else)
```bash
node /home/ccuser/shared/scripts/inbox.js read --agent sentinel --unread
```
If you have unread task messages, **DO THEM FIRST** before your regular heartbeat work.
After completing a task, acknowledge it:
```bash
node /home/ccuser/shared/scripts/inbox.js ack --agent sentinel --id <msg-id>
```
Then check for the next unread task. Only proceed when inbox is empty.

### Step 1: Read Fleet State
Run: `node /home/ccuser/shared/scripts/fleet-cli.js briefing sentinel --brief`
If you need full context: `node /home/ccuser/shared/scripts/fleet-cli.js briefing sentinel`

### Step 2: Read Your Queue & Claim a Task
```bash
bash /home/ccuser/shared/scripts/queue-check.sh sentinel
```
Pick the highest-priority pending task, claim it, do it, complete it, write results to memory.

### Step 3: Do Your Work
Execute the claimed queue task OR one domain task below.

### Step 3a: Proactive Fleet Management (Every 3rd heartbeat)
1. Check agent memory preservation
2. Spot regression patterns
3. Force improvement pivots if an agent is stuck
4. Curate stale vs relevant
5. Update improvement baselines

### Step 4: Update Your Status
Run at the END of every heartbeat:
```bash
node /home/ccuser/shared/scripts/fleet-update.js sentinel --status active --task "what-you-did"
```
If idle: `--status idle`
If blocked: `--status blocked --blocked-on "reason"`

Also write `/home/ccuser/sentinel/status.json` with current timestamps.

### Step 5: Report Issues (if any)
- Alert: `node /home/ccuser/shared/scripts/fleet-cli.js alert sentinel "Something went wrong"`
- Decision needed: `node /home/ccuser/shared/scripts/fleet-cli.js decide sentinel "Need Michael to decide X"`

---

## Core System Health

### 1. System Resources
- Check CPU usage (`uptime`)
- Check memory usage (`free -h`)
- Check disk space (`df -h`)
- Alert if CPU >90% or disk >80%

### 2. Agent Health (Proof, not surface green)
For each agent, separate these checks:
- **process up** — systemd/service running
- **gateway reachable** — endpoint responds
- **heartbeat fresh** — status file updated recently
- **work state fresh** — current task / queue state makes sense

**Never call an agent healthy from service status alone.**
If process is up but heartbeat is stale, status is **up_unresponsive**, not healthy.

### 3. Critical Services
- RateRight app (port 3000) responding
- External site accessibility
- Database connectivity if applicable

## Automated Actions
- Restart failed services automatically
- Clear temp files if disk space >80%
- Kill runaway processes if memory >90%
- Log all actions to daily memory file

## Alert Conditions
Escalate when:
- Multiple agents down simultaneously
- Process is up but heartbeat remains stale after restart
- External services unreachable >15 minutes
- Unable to auto-restart critical services
- Dashboard/CLI health view contradicts live checks

Only report HEARTBEAT_OK if process/gateway/heartbeat/work-state all agree green.
Otherwise provide brief status report with proof.