# Sentinel Infrastructure Deep Dive — 2026-02-25

*Brutally honest assessment of DevOps & Infrastructure.*

---

## ✅ WORKING

### VPS Stability
- **8 days uptime**, no reboots. Load average consistently 0.1–0.5 on 2 vCPU. Well within capacity.
- **Disk steady at 48%** (37GB/77GB). No creep, no runaway logs. Journals at 269MB, /tmp at 137MB.
- **RAM comfortable at 42–46%** normal ops (3.3–3.6GB of 7.8GB). Temporary spikes during `next build` (hit 65%) resolve cleanly.
- **Swap healthy** — only 155MB used of 2GB. No OOM kills.

### Agent Fleet
- **8/8 agents active right now.** All systemd services green. Zero failed units.
- **RateRight app solid** — 200 responses in 5–7ms local, 49–69ms external. No degradation.
- **No zombie processes, no stale locks** blocking startup.

### SSL & Nginx
- **All 3 SSL certs valid** through May 2026 (3 months out). Plenty of runway.
- **Nginx config clean** — passes `nginx -t`, no errors in error log.

### Monitoring
- Heartbeat system works. I catch issues within 15 minutes. Buddy system with Cog provides cross-coverage.
- Fleet coordination (inbox, briefings, status updates) running smoothly every cycle.

### Agent Ports
- **All 8 agent ports bound to 127.0.0.1** — not exposed externally. Good security posture.

---

## ❌ NOT WORKING

### 1. UFW Firewall — INACTIVE (Critical)
**UFW is off.** Port 8765 (voice-assistant Python process) is bound to 0.0.0.0 with no authentication. Anyone on the internet can hit it. SSH (22), HTTP (80), HTTPS (443) are the only ports that *should* be externally accessible. This has been pending Michael's approval since the security scan. **Risk: unauthenticated service exposed to the internet.**

### 2. .env File Permissions — STILL BROKEN
Previous security scan fixed 9 files to 600. But **9 .env files are still world-readable (644/664)**:
- `/home/ccuser/the-50-dollar-app/.env.local` (644) — **active app, contains secrets**
- `/home/ccuser/rateright-v2/.env.local` (664)
- `/home/ccuser/workdir/.env.local` (664)
- `/home/ccuser/rateright-main-site/.env.local` (644)
- Multiple backup .env files at 644

The active app's .env.local being world-readable is the worst one. Any process on the box can read Supabase keys, API tokens, etc.

### 3. Agent Restart Counts — Excessive
In the last 7 days:
| Agent | Restarts | Per Day |
|-------|----------|---------|
| Cog | 100 | **14/day** |
| Susan | 82 | 12/day |
| Rivet | 74 | 11/day |
| Herald | 66 | 9/day |
| Radar | 49 | 7/day |
| Harper | 47 | 7/day |
| Sentinel | 45 | 6/day |
| Builder | 14 | 2/day |

**Cog at 100 restarts in 7 days is concerning.** Builder at 14 is the baseline (roughly matches expected Clawdbot session cycling). Everything else is 3–7x Builder's rate. These aren't crashes (no failed units), but they suggest aggressive session recycling or context-limit-triggered restarts. Each restart burns tokens on re-bootstrap. **We're paying for this in API costs.**

### 4. No Backup Strategy
- No `/home/ccuser/shared/backups/` directory
- No `/root/backups/` directory  
- Only backup mechanism: git sync every 30 min (crontab)
- **No database backups** (Supabase managed, but no local export)
- **No agent config backups** — if `/root/.clawdbot-*/clawdbot.json` gets corrupted, manual rebuild
- **No disaster recovery plan** — if this VPS dies, everything dies

### 5. Growth Engine DOWN (External)
Railway returning 404 for the Growth Engine CRM. External service — I can't fix it, Michael needs to check Railway dashboard. Susan's CRM operations are degraded.

### 6. MEMORY.md Stale
My own MEMORY.md says "4GB RAM" — the VPS is 8GB. Stale data in operational docs erodes trust in the monitoring system.

---

## 🔧 RECOMMENDATIONS

### Immediate (Do This Week)

1. **Enable UFW** — Allow 22/tcp, 80/tcp, 443/tcp. Deny everything else from external. All agent ports already on 127.0.0.1, safe. Risk of SSH lockout mitigated by DigitalOcean console access. **Needs Michael's approval — been waiting since security scan.**

2. **Fix .env permissions** — One command fixes it:
   ```bash
   find /home/ccuser -name ".env*" -type f ! -name "*.example" -exec chmod 600 {} \;
   ```
   I can do this right now if approved. Zero downtime risk.

3. **Fix Growth Engine** — Michael needs to check Railway dashboard. This is blocking Susan's CRM work.

### Short-Term (Next 2 Weeks)

4. **Investigate restart counts** — Cog at 14/day needs root cause analysis. Options:
   - Check if context limits are too aggressive
   - Check if heartbeat frequency is causing unnecessary cycling
   - Compare Cog's config against Builder (2/day baseline)
   
5. **Implement config backups** — Simple cron job:
   ```bash
   # Daily backup of all agent configs
   tar czf /home/ccuser/shared/backups/configs-$(date +%Y%m%d).tar.gz /root/.clawdbot-*/clawdbot.json
   ```
   Rotate weekly. 5 minutes to implement.

6. **Update MEMORY.md** — Fix the 4GB→8GB error and do a full accuracy sweep.

### Medium-Term (Next Month)

7. **Disaster recovery plan** — Document: "If this VPS dies, here's how to rebuild everything in <2 hours." Include: DigitalOcean snapshot schedule, config backup locations, git repos list, env var inventory, service startup order.

8. **Agent restart optimization** — If we can get all agents to Builder's 2/day rate, we save ~350 unnecessary restarts/week. At even $0.01/restart in bootstrap tokens, that's money.

9. **Monitoring dashboard** — Right now health data lives in my head (well, my status.json). A simple HTML page showing real-time fleet status would let Michael check health without messaging me.

---

## Bottom Line

**The system is stable but insecure.** Day-to-day reliability is solid — 8/8 agents up, good resource headroom, fast app response times. But the firewall being off and .env permissions being loose are real security gaps. The lack of backups is a "hope nothing goes wrong" strategy.

The restart count issue is the biggest hidden cost — we're burning tokens on unnecessary agent recycling. Worth investigating.

**Priority order:** UFW → .env permissions → Growth Engine → restart investigation → backups.

*— Sentinel, 2026-02-25T17:55 AEDT*