# YouTube Intelligence Pipeline for Rivet

Automated pipeline to download, transcribe, and summarize YouTube videos.

## Features

- 🎵 Downloads audio only (smallest format to save bandwidth)
- 🎤 Transcribes using OpenAI Whisper API
- 📝 Generates smart summaries and extracts key points
- 💾 Saves raw transcripts, summaries, and JSON data
- 🔄 Can process local audio files for testing
- 🍪 Supports YouTube cookies for bypassing restrictions

## Requirements

- `yt-dlp` (already installed at `/usr/local/bin/yt-dlp`)
- Python 3.6+
- OpenAI API key (configured in Clawdbot or as environment variable)
- YouTube cookies (optional, for restricted videos)

## Usage

### Process a YouTube Video

```bash
# Basic usage
./youtube-pipeline.sh https://www.youtube.com/watch?v=VIDEO_ID

# Or with Python directly
python3 youtube-pipeline.py https://www.youtube.com/watch?v=VIDEO_ID
```

### Process a Local Audio File (for testing)

```bash
python3 youtube-pipeline.py /path/to/audio.mp3 --title "My Audio Title"
```

### With YouTube Cookies (for restricted videos)

```bash
python3 youtube-pipeline.py https://www.youtube.com/watch?v=VIDEO_ID --cookies cookies.txt
```

## YouTube Cookie Setup (Required for Most Videos)

YouTube has aggressive bot detection. To download most videos, you'll need to export your browser cookies:

### Option 1: Export from Browser (Recommended)

1. Install a cookie export extension:
   - Chrome/Edge: [Get cookies.txt LOCALLY](https://chrome.google.com/webstore/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc)
   - Firefox: [cookies.txt](https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/)

2. Visit youtube.com while logged in
3. Export cookies using the extension
4. Save as `~/youtube-cookies.txt`
5. Use with: `--cookies ~/youtube-cookies.txt`

### Option 2: Use Browser's Cookie Database

```bash
# Chrome
python3 youtube-pipeline.py URL --cookies-from-browser chrome

# Firefox  
python3 youtube-pipeline.py URL --cookies-from-browser firefox
```

**Note:** The `--cookies-from-browser` option needs to be added to the yt-dlp commands in the script.

## Output

All files are saved to `/home/ccuser/rateright-growth/rivet/memory/youtube/`:

- `YYYYMMDD_HHMMSS_VIDEOID_Title_transcript.txt` - Raw transcript
- `YYYYMMDD_HHMMSS_VIDEOID_Title_summary.md` - Formatted summary with key points
- `YYYYMMDD_HHMMSS_VIDEOID_Title_data.json` - Complete JSON data

## Troubleshooting

### "Sign in to confirm you're not a bot"

YouTube is blocking the download. Solutions:

1. Export and use cookies (see above)
2. Try a different video (some are less restricted)
3. Use the local audio mode for testing

### "File size exceeds Whisper API limit"

The OpenAI Whisper API has a 25MB file size limit. For longer videos:

1. The script already uses smallest audio quality
2. Consider using longer videos split into segments
3. Or use a local Whisper model instead

### No audio file found

Check that yt-dlp successfully downloaded the audio:

```bash
yt-dlp --extract-audio --audio-format mp3 YOUR_URL
```

## Testing

Test with a local audio file first to verify transcription works:

```bash
# Create a test audio file (using TTS or any short audio)
python3 youtube-pipeline.py test_audio.mp3 --title "Test Audio"
```

## Examples

```bash
# Process a tutorial video
./youtube-pipeline.sh https://www.youtube.com/watch?v=dQw4w9WgXcQ

# Process with cookies
python3 youtube-pipeline.py https://youtu.be/ABC123 --cookies ~/youtube-cookies.txt

# Process local audio
python3 youtube-pipeline.py recording.mp3 --title "Meeting Recording"
```

## Integration

Use this pipeline in other scripts:

```python
from youtube_pipeline import YouTubePipeline

api_key = "your_openai_api_key"
pipeline = YouTubePipeline(api_key)

# Process a video
pipeline.process_video("https://www.youtube.com/watch?v=...")

# Process local audio
pipeline.process_local_audio("/path/to/audio.mp3", "Custom Title")
```

## Future Enhancements

- [ ] Support for playlists
- [ ] Batch processing
- [ ] Local Whisper model option (faster, unlimited)
- [ ] Video subtitle download fallback
- [ ] Automatic cookie refresh
- [ ] Integration with Rivet's knowledge base

## Related Files

- `youtube-intel.py` - Searches YouTube and analyzes multiple videos
- `youtube-api-scanner.py` - Uses YouTube Data API for metadata
- `test-yt-dlp.py` - Tests basic yt-dlp functionality