---
created: 2026-03-12
source: Growth-Engine
tags: [agent-archive, growth-engine]
---

# JOB CONSOLIDATION PLAN
**Date:** Feb 2, 2026  
**Status:** Phase 1 Complete (SMS duplication fixed)  
**Priority:** HIGH - Reduce costs, prevent duplicates, simplify maintenance

## 🚨 IMMEDIATE FIX (DONE)
### **Fixed: SMS Duplication Risk**
- **Problem:** `sequenceProcessor.js` AND `scheduledSms.js` both sending SMS
- **Risk:** Duplicate messages to leads, Twilio cost overruns
- **Solution:** Disabled `scheduledSms.js`, kept `sequenceProcessor.js`
- **Status:** ✅ Fixed in `src/jobs/index.js`

### **Added: Job Locking System**
- **Problem:** No coordination between parallel job executions
- **Risk:** Database contention, race conditions
- **Solution:** Created `jobLocking.js` service with database locks
- **Status:** ✅ Implemented, needs migration `20260202130000_create_job_locks_table.sql`

## 📊 PHASE 2: METRICS JOBS CONSOLIDATION (THIS WEEK)
### **Current State:**
1. `dailyMetricsAggregation.js` - Daily metrics
2. `activationMetrics.js` - Daily activation metrics  
3. `impactMeasurement.js` - Daily impact metrics

### **Problem:**
- All 3 jobs query same tables (communications, leads, users)
- Redundant calculations, database contention
- 3x API/CPU usage for same data

### **Solution: Unified Daily Metrics Job**
```javascript
// New file: src/jobs/unifiedDailyMetrics.js
async function runUnifiedDailyMetrics() {
  // 1. Calculate all metrics in single transaction
  // 2. Store in metrics_cache table
  // 3. Update dashboards from cache
}
```

### **Benefits:**
- 66% reduction in database queries
- Single source of truth for metrics
- Faster dashboard updates (cached)

## 🎯 PHASE 3: AUDIT SYSTEM SIMPLIFICATION (THIS WEEK)
### **Current State:**
1. `dailyQualityAudit.js` - Daily 6am audit
2. `frequentAudit.js` - Hourly + critical audits  
3. `weeklyExcellenceReport.js` - Weekly excellence report
4. `frictionDetection.js` - Daily friction detection

### **Problem:**
- 4 systems checking same things (call quality, user performance)
- Alert fatigue, confusing notifications
- CPU waste on redundant checks

### **Solution: Comprehensive Audit System**
```javascript
// New file: src/jobs/comprehensiveAudit.js
async function runComprehensiveAudit(mode = 'daily') {
  // Modes: 'real-time', 'daily', 'weekly', 'critical'
  // Single system, configurable checks
}
```

### **Benefits:**
- 75% reduction in audit code
- Unified alerting system
- Configurable check frequency

## 📈 PHASE 4: REPORT UNIFICATION (NEXT WEEK)
### **Current State:**
1. `weeklyReport.js` - Monday 7am report
2. `weeklyRepReport.js` - Friday 3pm rep report  
3. `weeklyExcellenceReport.js` - Wednesday excellence report
4. `monthlyDeepDive.js` - Monthly deep dive

### **Problem:**
- All query same data, different formats
- API quota waste (OpenAI, etc.)
- Storage bloat with similar reports

### **Solution: Template-Based Report Generator**
```javascript
// New file: src/jobs/reportGenerator.js
async function generateReport(template, timeframe) {
  // Single data query
  // Multiple output formats (Slack, Email, PDF)
  // Template-driven content
}
```

### **Benefits:**
- 75% reduction in report code
- Consistent data across reports
- Flexible output formats

## 🔧 TECHNICAL IMPLEMENTATION

### **Job Locking Migration:**
```sql
-- Run this migration first
-- File: supabase/migrations/20260202130000_create_job_locks_table.sql
```

### **Sequence Processor Update:**
```javascript
// Already updated to use job locking
runJobWithLock('sequence_processor', processSequences, 4 * 60 * 1000)
```

### **API Endpoint Changes:**
- `/api/jobs/run/scheduledSms` now returns error (use `sequences` instead)
- All job endpoints will support locking

## 📅 TIMELINE

### **Week 1 (Feb 2-8):**
- ✅ Fix SMS duplication (DONE)
- ✅ Implement job locking (DONE)
- Create unified daily metrics job
- Update dashboard to use cached metrics

### **Week 2 (Feb 9-15):**
- Create comprehensive audit system
- Migrate existing audit checks
- Update alerting system

### **Week 3 (Feb 16-22):**
- Create template-based report generator
- Migrate existing reports
- Add new report templates

### **Week 4 (Feb 23-29):**
- Performance testing
- Monitoring setup
- Documentation

## 📊 EXPECTED BENEFITS

### **Cost Reduction:**
- **Database:** 50-70% fewer queries
- **API:** 50% fewer OpenAI/Twilio calls  
- **CPU:** 60% less processing
- **Storage:** 40% less report data

### **Risk Reduction:**
- **No duplicate SMS** (critical fix)
- **No race conditions** (job locking)
- **Consistent metrics** (single source)
- **Reliable alerts** (unified system)

### **Maintenance Benefits:**
- **Code:** 70% less job code
- **Debugging:** Single system to monitor
- **Updates:** One place to modify
- **Testing:** Simplified test suite

## 🚀 NEXT STEPS

1. **Run migration:** `20260202130000_create_job_locks_table.sql`
2. **Restart Growth Engine** to apply SMS fix
3. **Monitor** for any sequence processing issues
4. **Begin Phase 2** (unified metrics job)

## 📝 NOTES

- Some jobs already disabled in code (commented out)
- Keep `weeklyBadges.js` (gamification works well)
- Keep `leaderboardSnapshot.js` (real-time tracking)
- Consider moving to proper job queue (Bull, Agenda) long-term

---
**Maintainer:** Autonomous System (Rivet + CC)  
**Last Updated:** Feb 2, 2026  
**Status:** Phase 1 Complete ✅