# RateRight Post-Launch Monitoring & Incident Response Plan

## 🚨 Emergency Contacts

**Critical Issues (App Down/Payments Broken):**
- Builder (Claude Code): Gateway bridge or direct Claude Code access
- Rivet: System coordination and monitoring
- Michael: Business decisions, customer communication
- Hosting: DigitalOcean support (if VPS issues)

---

## 1. Monitoring Dashboard

### Key Metrics to Watch

#### Application Health
- **Uptime**: Target 99.9% (8.77 hours downtime/year max)
- **Response Time**: <500ms average, <2s p95
- **Error Rate**: <1% of total requests
- **5xx Errors**: Zero tolerance - investigate immediately

#### Business Metrics
- **User Signups**: Track hourly/daily trends
- **Job Posts**: Monitor submission success rate
- **Applications**: Ensure delivery to employers
- **Payment Processing**: Zero failed transactions
- **SMS Delivery**: >95% success rate
- **Email Delivery**: >98% success rate

#### Infrastructure Metrics
- **CPU Usage**: <70% sustained
- **Memory Usage**: <80% sustained
- **Disk Space**: <85% used
- **Database Connections**: <80% of limit
- **SSL Certificate**: >30 days until expiry

### Alert Thresholds

| Metric | Warning | Critical | Action |
|--------|---------|----------|---------|
| Uptime | <99.5% | <99% | Immediate investigation |
| Response Time | >1s avg | >2s avg | Check logs, scale if needed |
| Error Rate | >0.5% | >1% | Code investigation |
| CPU/Memory | >70% | >85% | Scale/optimize |
| Disk Space | >80% | >90% | Clean up/expand |
| Failed Payments | Any | >1% | Emergency response |

---

## 2. Common Issues & Playbooks

### 🚨 APP IS DOWN
**Symptoms:** Users can't access site, 502/503 errors

**Immediate Actions:**
1. Check VPS status: `systemctl status rateright-app`
2. Check logs: `journalctl -u rateright-app -f`
3. Try restart: `systemctl restart rateright-app`
4. Check disk space: `df -h`
5. Check memory: `free -m`

**If restart fails:**
- Check recent deployments: `cd /home/ccuser/the-50-dollar-app && git log -5`
- Rollback if needed: `git revert HEAD && npm run build && systemctl restart rateright-app`
- Escalate to Builder for code issues

### 💳 PAYMENT FAILURES
**Symptoms:** Users report payment errors, failed transactions

**Immediate Actions:**
1. Check Stripe dashboard for errors
2. Verify webhook endpoints are responding
3. Check logs for payment-related errors
4. Test payment flow with $1 test transaction

**If widespread:**
- Enable maintenance mode temporarily
- Notify users via status page
- Escalate to Builder immediately

### 🗄️ DATABASE CONNECTION ISSUES
**Symptoms:** "Connection refused", "too many connections"

**Immediate Actions:**
1. Check Supabase status page
2. Verify connection pool settings
3. Check for long-running queries
4. Restart app to clear connection pool

**If persistent:**
- Scale database connections
- Check for connection leaks in code
- Escalate to Builder

### 📊 HIGH ERROR RATES
**Symptoms:** Error monitoring shows spike in 4xx/5xx

**Immediate Actions:**
1. Check error logs for patterns
2. Identify affected endpoints
3. Check recent deployments
4. Verify third-party services (SMS, email)

**Response:**
- If code-related: Escalate to Builder
- If third-party: Check status pages, implement fallback

### 📱 SMS/EMAIL DELIVERY FAILURES
**Symptoms:** Users not receiving notifications

**Immediate Actions:**
1. Check provider status (Twilio for SMS)
2. Verify API keys and quotas
3. Check logs for delivery errors
4. Test with your own number/email

**If provider down:**
- Switch to backup provider if configured
- Notify users of delays
- Document for post-incident review

### 🔒 SECURITY INCIDENTS
**Symptoms:** Suspicious logins, unauthorized access, data breaches

**Immediate Actions:**
1. Lock down affected accounts
2. Check access logs for IP addresses
3. Reset compromised credentials
4. Enable 2FA for all admin accounts

**Response:**
- Immediate password resets
- Review all recent admin activity
- Consider temporary site lockdown
- Full security audit with Builder

---

## 3. Incident Response Process

### Detection Methods
1. **Automated Alerts**: Uptime monitoring, error tracking
2. **User Reports**: Support tickets, direct messages
3. **Heartbeat Checks**: Regular health checks every 5 minutes
4. **Routine Monitoring**: Daily dashboard reviews

### Initial Assessment (First 5 minutes)
1. **Severity Level:**
   - 🔴 Critical: App down, payments broken, data loss
   - 🟡 High: Degraded performance, some features broken
   - 🟢 Low: Minor issues, workarounds available

2. **Who to Notify:**
   - Critical: Michael immediately (SMS/call)
   - High: Message Michael, start fixing
   - Low: Document, fix during business hours

### Triage & Quick Fixes

#### Critical Issues (11 PM Protocol)
1. Don't panic - follow the checklist
2. Try quick fixes first (restart, rollback)
3. If fixed in <30 min: Monitor closely
4. If not fixed: Wake Michael with:
   - What broke
   - What you've tried
   - Current status
   - Estimated fix time

#### Quick Fix Options
- **Restart services**: `systemctl restart rateright-app`
- **Rollback code**: `git revert HEAD && npm run build && systemctl restart rateright-app`
- **Clear cache**: Restart Redis if used
- **Scale up**: Increase VPS resources if needed
- **Enable maintenance**: Temporary downtime page

### Post-Incident Review
Within 24 hours of resolution:
1. Document timeline of events
2. Identify root cause
3. Implement preventive measures
4. Update runbooks if needed
5. Share learnings with team

---

## 4. Escalation Matrix

### Builder (Claude Code)
**Contact:** Gateway bridge or direct Claude Code
**Handles:**
- Code bugs and fixes
- Database migrations
- Deployment issues
- Complex technical problems

**When to escalate:**
- Quick fixes don't work
- Code changes needed
- Database issues
- Performance optimization needed

### Rivet (System Operations)
**Contact:** Direct message or system alerts
**Handles:**
- System monitoring
- Coordination between teams
- Status updates
- Routine maintenance

**When to escalate:**
- Multiple systems affected
- Need coordination help
- Status updates required
- Bridge between Builder and Michael

### Michael (Business Owner)
**Contact:** SMS +61 426 246 472, or call
**Handles:**
- Business decisions
- Customer communication
- Critical incident approval
- External vendor contact

**When to escalate:**
- Critical issues >30 min
- Payment/billing problems
- Customer data issues
- Legal/compliance concerns

### Hosting Provider (DigitalOcean)
**Contact:** Support ticket or call
**Handles:**
- Infrastructure issues
- Network problems
- Hardware failures
- Data center issues

**When to escalate:**
- VPS unreachable
- Network connectivity issues
- Disk/hardware failures
- Suspicious activity reports

---

## 5. Daily/Weekly Checks

### Daily (Automated where possible)
- [ ] Review overnight alerts
- [ ] Check payment processing
- [ ] Verify SMS/email delivery
- [ ] Monitor signup/application trends
- [ ] Review error logs for new issues
- [ ] Check SSL certificate status
- [ ] Verify backups completed

### Weekly
- [ ] Full system health check
- [ ] Database performance review
- [ ] Security log analysis
- [ ] Update dependencies check
- [ ] Review and update documentation
- [ ] Test incident response procedures
- [ ] Check domain renewal dates

### Monthly
- [ ] Disaster recovery test
- [ ] Security audit
- [ ] Performance optimization review
- [ ] Cost optimization check
- [ ] Update contact information
- [ ] Review and update this plan

---

## Quick Reference

### Emergency Commands
```bash
# Check app status
systemctl status rateright-app

# Restart app
systemctl restart rateright-app

# Check logs
journalctl -u rateright-app -f

# Check disk space
df -h

# Check memory
free -m

# Check recent deployments
cd /home/ccuser/the-50-dollar-app && git log -5
```

### Important URLs
- RateRight App: https://rivet.rateright.com.au
- Supabase Dashboard: [Check .env for URL]
- Stripe Dashboard: https://dashboard.stripe.com
- DigitalOcean: https://cloud.digitalocean.com

### Key Files
- App location: `/home/ccuser/the-50-dollar-app`
- Logs: `journalctl -u rateright-app`
- Environment: `.env.local` in app directory
- This plan: `/home/ccuser/rateright-growth/memory/plans/post-launch-monitoring-plan.md`

---

**Remember:** When something breaks at 11 PM:
1. Don't panic
2. Check this plan
3. Start with quick fixes
4. Escalate if needed
5. Document everything

**Last Updated:** $(date)
**Next Review:** $(date -d "+1 month")