# Phase 4 Nuclear Deployment Execution Guide

## Overview
This guide provides complete instructions for executing the Phase 4 Nuclear Deployment to replace the corrupted RateRight database cluster with a new, clean PostgreSQL 17.2 cluster.

## ⚠️ CRITICAL WARNINGS

**THIS IS A DESTRUCTIVE OPERATION** - The existing database will be detached and replaced. Ensure:
- Current database backup exists
- Application can tolerate downtime during switchover
- All team members are notified of the deployment window
- Recovery procedures are prepared

## Prerequisites

### 1. Environment Setup
```powershell
# Verify Fly.io CLI installation
fly version

# Authenticate with Fly.io
fly auth login

# Verify access to RateRight application
fly apps list | Where-Object {$_ -match "rateright-au"}
```

### 2. Pre-Deployment Checklist
- [ ] Fly.io CLI installed and authenticated
- [ ] Access to `rateright-au` application confirmed  
- [ ] Network connectivity to Fly.io infrastructure verified
- [ ] Team notification sent regarding deployment window
- [ ] Backup/recovery procedures reviewed
- [ ] Evidence directory permissions confirmed

## Execution Methods

### Method 1: Full Production Deployment (RECOMMENDED)
Execute the complete deployment with all safeguards:

```powershell
# Navigate to project directory
cd c:/Users/mclou/temp_compare_rateright/-RateRight

# Execute deployment script
.\PHASE4_NUCLEAR_DEPLOYMENT.ps1
```

### Method 2: Dry Run Execution (SAFE TESTING)
Test the deployment process without making changes:

```powershell
# Execute in WhatIf mode to preview operations
.\PHASE4_NUCLEAR_DEPLOYMENT.ps1 -WhatIf
```

### Method 3: Force Execution (EMERGENCY ONLY)
Continue deployment even if non-critical steps fail:

```powershell
# Force execution through minor failures
.\PHASE4_NUCLEAR_DEPLOYMENT.ps1 -Force
```

## Deployment Steps Breakdown

### Step 18: Database Backup Creation
**Command:** `fly postgres backup create rateright-db`
- **Purpose:** Create snapshot of current database state before destruction
- **Evidence:** `evidence/14_backup_creation.txt`
- **Critical:** YES - Deployment stops if this fails
- **Expected Duration:** 2-5 minutes

### Step 19: New Database Cluster Creation  
**Command:** `fly postgres create --name rateright-db-v2 --region syd --initial-cluster-size 3`
- **Purpose:** Create new clean PostgreSQL 17.2 cluster in Sydney
- **Evidence:** `evidence/15_new_db_creation.txt`
- **Critical:** YES - Deployment stops if this fails
- **Expected Duration:** 3-8 minutes
- **Configuration:**
  - Database Name: `rateright-db-v2`
  - Region: Sydney (syd)
  - Cluster Size: 3 instances (1 primary + 2 replicas)
  - PostgreSQL Version: 17.2

### Step 20: Old Database Detachment
**Command:** `fly postgres detach rateright-db -a rateright-au`
- **Purpose:** Disconnect corrupted database from application
- **Evidence:** `evidence/16_old_db_detach.txt`
- **Critical:** YES - Application will lose database connectivity
- **Expected Duration:** 30 seconds - 2 minutes
- **Impact:** **APPLICATION DOWNTIME BEGINS**

### Step 21: New Database Attachment
**Command:** `fly postgres attach rateright-db-v2 -a rateright-au`
- **Purpose:** Connect new clean database to application
- **Evidence:** `evidence/17_new_db_attach.txt`
- **Critical:** YES - Application needs database connectivity
- **Expected Duration:** 1-3 minutes
- **Impact:** **APPLICATION DOWNTIME ENDS** (pending schema migration)

### Step 22: DATABASE_URL Verification
**Command:** `fly secrets list -a rateright-au | Where-Object {$_ -match "DATABASE_URL"}`
- **Purpose:** Confirm new DATABASE_URL secret was created correctly
- **Evidence:** `evidence/18_new_database_url_confirmation.txt`
- **Critical:** YES - Application needs correct connection string
- **Expected Duration:** 10-30 seconds

### Step 23: Deployment Report Generation
- **Purpose:** Create comprehensive deployment documentation
- **Output:** `PHASE4_NUCLEAR_DEPLOYMENT_REPORT.txt`
- **Contains:** Timing, configuration, next steps, evidence files list

## Post-Deployment Actions Required

### Immediate (Within 30 minutes)
1. **Schema Migration:** Deploy complete database schema to new cluster
2. **Application Restart:** Restart application to pick up new DATABASE_URL
3. **Basic Validation:** Verify application starts and database connects

### Within 2 Hours  
1. **Data Migration:** If data preservation needed, restore from backup
2. **Full Testing:** Execute comprehensive application testing
3. **Performance Validation:** Verify new cluster performance
4. **Monitoring Setup:** Configure alerting for new database cluster

### Within 24 Hours
1. **Old Database Cleanup:** Consider destroying old corrupted database
2. **Documentation Update:** Update deployment documentation
3. **Team Notification:** Report deployment success/issues
4. **Backup Schedule:** Verify backup schedules are active on new cluster

## Expected Deployment Timeline

| Phase | Duration | Description |
|-------|----------|-------------|
| Backup Creation | 2-5 minutes | Snapshot current database |
| New Cluster Creation | 3-8 minutes | Provision new PostgreSQL cluster |
| Database Switchover | 1-3 minutes | Detach old, attach new |
| Verification | 1-2 minutes | Confirm new configuration |
| **Total Downtime** | **2-5 minutes** | Application unavailable during switchover |
| **Total Deployment** | **7-18 minutes** | Complete deployment process |

## Evidence Files Generated

All evidence files are stored in `./evidence/` directory:

- `14_backup_creation.txt` - Backup creation output
- `15_new_db_creation.txt` - New cluster creation output  
- `16_old_db_detach.txt` - Old database detachment output
- `17_new_db_attach.txt` - New database attachment output
- `18_new_database_url_confirmation.txt` - DATABASE_URL verification
- `phase4_deployment.log` - Complete deployment log

## Troubleshooting

### Common Issues

**Backup Creation Fails**
- Check database connectivity: `fly postgres connect -a rateright-db`
- Verify database health: `fly postgres list`
- Check Fly.io service status: `fly status`

**New Cluster Creation Fails**
- Verify region availability: `fly regions list`
- Check account limits: Contact Fly.io support
- Try different region if Sydney unavailable

**Database Attachment Fails**  
- Verify new cluster is ready: `fly postgres list`
- Check application exists: `fly apps list`
- Verify permissions: `fly auth whoami`

**APPLICATION STAYS DOWN**
- Check application logs: `fly logs -a rateright-au`
- Verify DATABASE_URL: `fly secrets list -a rateright-au`
- Restart application: `fly deploy -a rateright-au`

### Emergency Rollback

If deployment fails critically:

1. **Re-attach old database (if still available):**
   ```powershell
   fly postgres attach rateright-db -a rateright-au
   ```

2. **Check application status:**
   ```powershell
   fly status -a rateright-au
   ```

3. **Review evidence files for diagnostic information**

4. **Contact Fly.io support if infrastructure issues persist**

## Security Considerations

- All database credentials are managed by Fly.io secrets
- New DATABASE_URL is automatically generated and secured
- Old database remains available until explicitly destroyed
- Evidence files may contain sensitive configuration information

## Success Criteria

Deployment is considered successful when:
- [x] Database backup created successfully
- [x] New PostgreSQL 17.2 cluster operational
- [x] Old database detached without errors  
- [x] New database attached successfully
- [x] DATABASE_URL secret updated correctly
- [x] All evidence files generated
- [x] Deployment report created

## Next Steps After Successful Deployment

1. **Schema Migration:** Apply complete database schema
2. **Data Migration:** Restore critical data from backup if needed
3. **Application Testing:** Verify all functionality works
4. **Performance Monitoring:** Monitor new cluster performance
5. **Backup Validation:** Ensure backup systems work with new cluster

---

**IMPORTANT:** This deployment creates a completely new database cluster. All existing data will be lost unless restored from backup. Ensure data migration strategy is prepared before executing.