# YouTube Pipeline - Quick Start

## TL;DR

```bash
# 1. Export YouTube cookies (one-time setup)
#    Chrome: Install "Get cookies.txt LOCALLY" extension
#    Visit youtube.com, click extension, export cookies.txt
#    Save to ~/.youtube-cookies.txt

# 2. Run the pipeline
cd /home/ccuser/rateright-growth/rivet
./scripts/youtube-pipeline.sh "https://www.youtube.com/watch?v=VIDEO_ID" --cookies ~/.youtube-cookies.txt

# 3. Check output
ls -ltr memory/youtube/
cat memory/youtube/*_summary.md | tail -50
```

## Files Created

- **`youtube-pipeline.py`** - Main Python pipeline
  - Downloads audio from YouTube
  - Transcribes with OpenAI Whisper
  - Generates summaries and extracts key points
  - Saves to `memory/youtube/`

- **`youtube-pipeline.sh`** - Bash wrapper
  - Simple command-line interface
  - Auto-detects OpenAI API key from Clawdbot config
  - Color-coded output

## Features

✅ Audio-only download (smallest format)
✅ OpenAI Whisper transcription
✅ Smart summarization
✅ Topic and keyword extraction
✅ Local audio file support
✅ YouTube cookie support
✅ Structured output (TXT, MD, JSON)

## Output Structure

```
memory/youtube/
  ├── 20240205_143022_dQw4w9WgXcQ_Rick_Astley_Never_Gonna_Give_You_Up_transcript.txt
  ├── 20240205_143022_dQw4w9WgXcQ_Rick_Astley_Never_Gonna_Give_You_Up_summary.md
  └── 20240205_143022_dQw4w9WgXcQ_Rick_Astley_Never_Gonna_Give_You_Up_data.json
```

## Common Use Cases

### Research a competitor's video
```bash
./scripts/youtube-pipeline.sh "https://www.youtube.com/watch?v=COMPETITOR_VIDEO"
```

### Process multiple videos
```bash
for url in $(cat video_urls.txt); do
    ./scripts/youtube-pipeline.sh "$url" --cookies ~/.youtube-cookies.txt
    sleep 5  # Rate limiting
done
```

### Process a recorded call/meeting
```bash
python3 scripts/youtube-pipeline.py ~/recordings/meeting.mp3 --title "Team Standup 2024-02-05"
```

## Integration with Rivet

The pipeline outputs structured data that can be fed into:
- Content generation workflows
- Competitor analysis
- SEO research
- Knowledge base building
- Meeting transcription
- Voice call analysis

## Why Cookies Are Required

YouTube has aggressive bot detection. Even with:
- ✅ Latest yt-dlp version
- ✅ Different user agents
- ✅ Various client modes (android, tv, web)
- ✅ OAuth attempts

...YouTube still blocks automated downloads.

**Solution:** Use browser cookies (they prove you're logged in).

This is standard practice for all YouTube download tools in 2024.

## Current Limitations

1. **Requires YouTube cookies** - One-time setup per browser/account
2. **25MB audio limit** - OpenAI Whisper API restriction (2-3 hour videos)
3. **No playlist support yet** - Process videos one at a time
4. **Rate limiting** - YouTube may block if you download too many too fast

## Roadmap

- [ ] Automatic cookie refresh
- [ ] Playlist batch processing
- [ ] Local Whisper model option (unlimited size)
- [ ] Video chapter detection
- [ ] Sentiment analysis
- [ ] Speaker diarization
- [ ] Auto-tagging and categorization

## Cost Estimate

OpenAI Whisper API pricing:
- $0.006 per minute of audio
- Average 10-minute video = $0.06
- 30-minute video = $0.18
- 1-hour video = $0.36

Very affordable for intelligence gathering! 🎯

## Getting Help

1. Check `README-youtube-pipeline.md` for detailed docs
2. See `YOUTUBE-COOKIES-SETUP.md` for cookie setup guide
3. Test with local audio: `python3 scripts/youtube-pipeline.py test.mp3`

## Status

🟢 **READY TO USE** (with YouTube cookies setup)

The pipeline is fully functional. Only blocker is YouTube's bot detection,  
which is solved by exporting cookies from your browser (5-minute setup).