# Self-Healing Ops Brain - Implementation Plan > **Status:** PARKED > **Parked Date:** 2026-01-21 > **Reason:** User paused implementation --- > Problem detected → Diagnosed → Fixed → Verified → Learned **Estimated Effort:** 45-65 hours (4-week phased approach) **Risk Level:** Low (starts diagnosis-only, then gradual auto-fix enablement) --- ## Executive Summary Build a 0.1% self-healing operations system that: 1. **Detects** problems via existing monitoring (frequentAudit.js, criticalAlerts.js) 2. **Diagnoses** root cause using machine-executable SOP steps 3. **Decides** auto-fix vs escalate based on confidence scoring 4. **Executes** safe fixes with sandboxed operations 5. **Verifies** the fix worked by re-running health checks 6. **Learns** and improves SOPs based on outcomes **Safety First:** NEVER auto-fixes billing, security, schema, or user data. Starts in diagnosis-only mode. --- ## Architecture Flow ``` ┌─────────────────────────────────────────────────────────────────────┐ │ OPS BRAIN (24/7/365) │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ [MONITORS] [SOP MATCHER] [DECISION ENGINE] │ │ frequentAudit.js ──→ Find matching ──→ Confidence score │ │ criticalAlerts.js SOP 0-100% │ │ intelligence_alerts │ │ │ │ │ │ │ │ │ │ │ ▼ │ │ │ │ ┌────────────────────┐ │ │ │ │ │ HIGH (>90%): │ │ │ │ │ │ → Auto-fix │ │ │ │ │ │ MEDIUM (70-90%): │ │ │ │ │ │ → Await approval │ │ │ │ │ │ LOW (<70%): │ │ │ │ │ │ → Escalate │ │ │ │ │ └────────────────────┘ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ [DIAGNOSIS] [EXECUTION] [VERIFICATION] │ │ Run SOP diag ──→ Execute fix ──→ Re-run health check │ │ steps steps Compare before/after │ │ │ │ │ ▼ │ │ [LEARNING] │ │ Update SOP stats │ │ Improve on failures │ └─────────────────────────────────────────────────────────────────────┘ ``` --- ## Database Schema ### Table: `self_healing_sops` ```sql CREATE TABLE IF NOT EXISTS self_healing_sops ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), -- Identification sop_name VARCHAR(100) UNIQUE NOT NULL, category VARCHAR(50) NOT NULL, -- database, api, external, queue, cache description TEXT NOT NULL, version INTEGER DEFAULT 1, -- Trigger Conditions (when this SOP applies) trigger_conditions JSONB NOT NULL, -- { "alert_types": ["quality_audit_critical"], -- "title_patterns": ["Database Connection"], -- "severity_min": "high" } -- Diagnosis Steps (what to check) diagnosis_steps JSONB NOT NULL, -- [{ "step": 1, "name": "Check DB", "type": "db_query", -- "action": { "query": "SELECT 1" }, "success_condition": "no_error" }] -- Fix Operations (what actions to take) fix_operations JSONB NOT NULL, -- [{ "step": 1, "name": "Reset pool", "type": "function_call", -- "action": { "function": "resetConnectionPool" } }] -- Verification (how to confirm fix worked) verification_steps JSONB NOT NULL, -- [{ "step": 1, "name": "Re-run test", "type": "rerun_audit", -- "wait_before_ms": 5000 }] -- Safety Controls risk_level VARCHAR(20) NOT NULL, -- low, medium, high, critical auto_fix_enabled BOOLEAN DEFAULT FALSE, requires_approval BOOLEAN DEFAULT TRUE, max_executions_per_hour INTEGER DEFAULT 3, cooldown_minutes INTEGER DEFAULT 30, -- Learning Stats times_executed INTEGER DEFAULT 0, times_successful INTEGER DEFAULT 0, avg_resolution_time_ms INTEGER, last_executed_at TIMESTAMPTZ, created_at TIMESTAMPTZ DEFAULT NOW(), updated_at TIMESTAMPTZ DEFAULT NOW() ); ``` ### Table: `self_healing_executions` ```sql CREATE TABLE IF NOT EXISTS self_healing_executions ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), sop_id UUID REFERENCES self_healing_sops(id), alert_id UUID REFERENCES intelligence_alerts(id), -- Results diagnosis_results JSONB, fix_actions_taken JSONB, verification_results JSONB, -- Status: diagnosing → awaiting_approval → fixing → verifying → success/failed/escalated status VARCHAR(30) NOT NULL, confidence_score INTEGER, -- Timing started_at TIMESTAMPTZ DEFAULT NOW(), completed_at TIMESTAMPTZ, total_resolution_time_ms INTEGER, -- Human Interaction approved_by VARCHAR(100), escalation_reason TEXT, -- Learning was_effective_fix BOOLEAN, human_notes TEXT ); ``` --- ## Decision Engine ### Confidence Scoring (0-100) | Factor | Weight | Description | |--------|--------|-------------| | SOP Success Rate | 40% | Historical success rate (needs >5 executions for accuracy) | | Diagnosis Clarity | 30% | How clearly diagnosis matched expected patterns | | Pattern Match | 20% | How well alert matches SOP trigger conditions | | Environmental | 10% | Time of day, concurrent alerts, recent similar success | ### Decision Thresholds by Risk Level | Risk Level | Auto-Fix Threshold | Example SOPs | |------------|-------------------|--------------| | Low | ≥60% confidence | Cache clear, job retry, log rotation | | Medium | ≥75% confidence | Connection pool reset, rate limit adjust | | High | ≥90% confidence | Service restart, credential refresh | | Critical | Never auto-fix | Schema changes, security, billing | --- ## Safe Auto-Executable Operations | Operation | Risk | Description | |-----------|------|-------------| | `cache_clear` | Low | Clear specific cache keys | | `queue_retry` | Low | Retry failed queue items | | `job_restart` | Low | Restart stuck scheduled jobs | | `connection_refresh` | Low | Refresh external connections | | `log_rotation` | Low | Archive old logs | | `temp_cleanup` | Low | Clean temp files | | `metrics_backfill` | Low | Re-aggregate missing metrics | | `token_refresh` | Medium | Refresh API tokens | | `pool_reset` | Medium | Reset DB connection pool | | `rate_limit_reset` | Medium | Clear rate limit blocks | --- ## NEVER Auto-Fix (Always Escalate) - Schema changes (CREATE/ALTER/DROP) - User data deletion - Billing/payment operations - Security/authentication changes - Environment variable changes - API credential rotation - Production data modifications - Twilio number configuration - Supabase RLS policy changes --- ## Files to Create | File | Purpose | |------|---------| | `supabase/migrations/YYYYMMDD_self_healing_ops.sql` | Database schema | | `supabase/sop-seed.sql` | Initial 20 SOPs | | `src/services/selfHealing/sopMatcher.js` | Match alerts to SOPs | | `src/services/selfHealing/decisionEngine.js` | Confidence scoring, decisions | | `src/services/selfHealing/executor.js` | Safe fix execution | | `src/services/selfHealing/verifier.js` | Verify fixes worked | | `src/services/selfHealing/opsBrain.js` | Main orchestrator | | `src/jobs/dailyOpsReport.js` | 7am daily ops summary | | `src/routes/ops.js` | API for ops dashboard | --- ## Integration Points ### 1. frequentAudit.js (Primary Trigger) After creating intelligence_alert, call `opsBrain.evaluate(alertId)` ### 2. criticalAlerts.js (Secondary Trigger) After sending Slack alert, call `opsBrain.evaluate(alertId)` ### 3. slack.js (Notifications) Add `sendSelfHealingNotification()` with approval buttons ### 4. learning.js (Pattern Extension) Track SOP execution outcomes for improvement suggestions --- ## Phased Implementation ### Phase 1: Foundation (Week 1) - DIAGNOSIS ONLY - [ ] Create database schema - [ ] Create SOP matcher service - [ ] Create diagnosis engine - [ ] Seed 10 initial SOPs (diagnosis only) - [ ] Hook into frequentAudit.js - [ ] Slack notifications for diagnosis results - **NO auto-fixes** - just detection and reporting ### Phase 2: Approval Workflow (Week 2) - [ ] Create decision engine with confidence scoring - [ ] Create Slack approval buttons - [ ] Create executor service - [ ] Create verifier service - [ ] Audit logging - **Human approval required for ALL fixes** ### Phase 3: Cautious Auto-Fix (Week 3) - [ ] Enable auto-fix for 5 lowest-risk SOPs - [ ] Implement cooldowns and rate limits - [ ] Rollback capability - [ ] Learning stats tracking - [ ] Ops dashboard UI ### Phase 4: Full Self-Healing (Week 4+) - [ ] Enable auto-fix for remaining low-risk SOPs - [ ] SOP improvement from failures - [ ] Daily ops report job - [ ] Predictive issue detection --- ## Initial SOP Seed List (20 Runbooks) ### Infrastructure (5) 1. `database-connection-dropped` - DB connectivity issues 2. `server-memory-high` - Memory pressure 3. `ssl-cert-expiring` - Certificate renewal 4. `build-failed` - Deployment issues 5. `asset-404` - Missing frontend assets ### Integrations (5) 6. `twilio-sms-delivery-low` - SMS delivery problems 7. `openai-rate-limited` - GPT rate limiting 8. `deepgram-transcription-failed` - Transcription issues 9. `supabase-connection-dropped` - DB pool exhaustion 10. `slack-webhook-failed` - Notification failures ### Data (5) 11. `lead-sync-stuck` - Platform sync issues 12. `sequence-enrollment-stuck` - Sequence processor 13. `duplicate-leads-detected` - Data integrity 14. `conversion-not-tracked` - Conversion tracking 15. `orphaned-communications` - Data cleanup ### API/Business (5) 16. `endpoint-500-error` - API errors 17. `rate-limit-exceeded` - User rate limits 18. `no-calls-in-2-hours` - Business anomaly 19. `openai-spend-spike` - Cost alert 20. `twilio-spend-spike` - Cost alert --- ## Success Metrics | Metric | Target | |--------|--------| | Issues auto-fixed | 80%+ without human | | Fix time | <5 minutes average | | Fix success rate | 95%+ | | False escalations | <5% | | SOP coverage | 90% of common issues | --- ## Resume Instructions When resuming this plan: 1. Read this file for full context 2. Start with Phase 1 tasks (database schema first) 3. Follow the phased approach - don't skip ahead 4. Update this file with progress as you go