# YouTube Autonomous Intelligence System

**Never paste a URL again.** The system searches, ranks, and processes top-performing videos automatically.

## Overview

This autonomous system:
1. 🔍 **Searches YouTube** for configurable topics
2. 📊 **Ranks videos** by performance (views/age)
3. 🎯 **Picks top performers** (default: top 3)
4. 🎬 **Processes them** through the full pipeline (download → transcribe → summarize)
5. 📄 **Generates a report** with all findings

## Quick Start

### 1. Configure Search Topics

Edit `scripts/youtube-topics.json`:

```json
{
  "topics": [
    {
      "query": "Clawdbot tutorial",
      "category": "clawdbot",
      "enabled": true,
      "priority": 1
    }
  ]
}
```

### 2. Run Autonomous Scan

```bash
# Dry run (search only, don't process)
./scripts/youtube-auto-intel.sh --dry-run

# Full run (search + process top videos)
./scripts/youtube-auto-intel.sh --cookies ~/.youtube-cookies.txt

# Process top 5 instead of default 3
./scripts/youtube-auto-intel.sh --top 5 --cookies ~/.youtube-cookies.txt
```

### 3. Review Results

All output saved to `memory/youtube/`:
- Individual video transcripts and summaries
- Auto-generated intelligence report: `YYYYMMDD_HHMMSS_auto_intel_report.md`

## Configuration File

### `youtube-topics.json` Structure

```json
{
  "config": {
    "search_results_per_topic": 20,
    "top_videos_to_process": 3,
    "min_views": 100,
    "max_age_days": 365,
    "exclude_shorts": true,
    "min_duration_seconds": 180,
    "output_summary": true
  },
  "topics": [
    {
      "query": "your search query",
      "category": "category-name",
      "enabled": true,
      "priority": 1
    }
  ],
  "filters": {
    "exclude_channels": [],
    "exclude_keywords": ["scam", "fake"],
    "preferred_channels": ["Channel Name"],
    "language": "en"
  }
}
```

### Configuration Options

**Global Settings:**
- `search_results_per_topic` - How many results per search (default: 20)
- `top_videos_to_process` - How many top videos to process (default: 3)
- `min_views` - Minimum view count to consider (default: 100)
- `max_age_days` - Maximum video age in days (default: 365)
- `exclude_shorts` - Skip YouTube Shorts (default: true)
- `min_duration_seconds` - Minimum duration (default: 180 = 3 minutes)
- `output_summary` - Generate summary report (default: true)

**Topic Settings:**
- `query` - YouTube search query
- `category` - Internal category for organization
- `enabled` - Set to `false` to skip this topic
- `priority` - 1 (always), 2 (if quota available), 3 (manual only)

**Filters:**
- `exclude_channels` - Blacklist specific channels
- `exclude_keywords` - Skip videos with these words in title/description
- `preferred_channels` - Boost videos from these channels
- `language` - Content language (default: "en")

## Usage Examples

### Basic Usage

```bash
# Dry run - see what videos would be processed
./scripts/youtube-auto-intel.sh --dry-run

# Full run with cookies
./scripts/youtube-auto-intel.sh --cookies ~/.youtube-cookies.txt

# Process top 10 videos
./scripts/youtube-auto-intel.sh --top 10
```

### Python Script Directly

```bash
# With all options
python3 scripts/youtube-auto-intel.py \
  --config custom-topics.json \
  --cookies ~/.youtube-cookies.txt \
  --top 5

# Dry run
python3 scripts/youtube-auto-intel.py --dry-run
```

### Scheduled Automation

Run automatically via cron:

```bash
# Daily at 3am
0 3 * * * cd /home/ccuser/rateright-growth/rivet && ./scripts/youtube-auto-intel.sh --cookies ~/.youtube-cookies.txt >> /tmp/youtube-intel.log 2>&1

# Weekly on Monday at 9am
0 9 * * 1 cd /home/ccuser/rateright-growth/rivet && ./scripts/youtube-auto-intel.sh --top 10 --cookies ~/.youtube-cookies.txt
```

## How It Works

### 1. Search Phase

For each enabled topic in `youtube-topics.json`:
- Searches YouTube using yt-dlp: `ytsearch20:your query`
- Returns metadata: title, channel, views, upload date, duration
- Rate limits between searches (1 second delay)

### 2. Ranking Phase

For each video found:
- **Calculate performance score:** `views / days_since_upload`
- **Apply filters:**
  - Minimum views threshold
  - Maximum age limit
  - Duration requirements (exclude shorts)
  - Keyword blacklist
  - Channel blacklist
- **Sort by score** (highest first)

### 3. Processing Phase

For top N videos (default 3):
- Download audio only (smallest format)
- Transcribe with OpenAI Whisper
- Generate smart summary
- Extract key points and topics
- Save transcript, summary, and JSON data

### 4. Report Generation

Creates `YYYYMMDD_HHMMSS_auto_intel_report.md` with:
- Summary statistics
- Top 20 ranked videos
- Which videos were processed
- Any errors encountered
- Configuration used

## Output Structure

```
memory/youtube/
├── 20240205_150000_VideoID1_Title_transcript.txt
├── 20240205_150000_VideoID1_Title_summary.md
├── 20240205_150000_VideoID1_Title_data.json
├── 20240205_150500_VideoID2_Title_transcript.txt
├── 20240205_150500_VideoID2_Title_summary.md
├── 20240205_150500_VideoID2_Title_data.json
└── 20240205_151000_auto_intel_report.md  ← Summary report
```

## Real-World Example

**Goal:** Monitor Clawdbot tutorials and AI automation content

**Configuration:**
```json
{
  "config": {
    "top_videos_to_process": 3,
    "min_views": 1000
  },
  "topics": [
    {"query": "Clawdbot tutorial", "enabled": true},
    {"query": "AI agents automation", "enabled": true}
  ]
}
```

**Run:**
```bash
./scripts/youtube-auto-intel.sh --cookies ~/.youtube-cookies.txt
```

**Result:**
- Searches 2 topics (40 videos total)
- Filters to high-quality videos (1000+ views)
- Ranks by performance score
- Processes top 3 videos
- Generates report with all findings

**Time:** ~15-20 minutes for 3 videos (depends on video length)

**Cost:** ~$0.18 (3 videos × 10 min avg × $0.006/min)

## Use Cases

### Competitor Intelligence
```json
{"query": "construction marketplace app", "category": "competitors"}
{"query": "tradie hiring platform Australia", "category": "competitors"}
```

### Learning & Development
```json
{"query": "Clawdbot advanced features", "category": "learning"}
{"query": "AI agent best practices", "category": "learning"}
```

### Market Research
```json
{"query": "AI COO trends 2024", "category": "market"}
{"query": "business automation case studies", "category": "market"}
```

### Content Ideas
```json
{"query": "construction tech problems", "category": "content-ideas"}
{"query": "tradie pain points", "category": "content-ideas"}
```

## Performance Scoring

The system calculates: **score = views / days_since_upload**

**Why this metric?**
- Identifies **trending** videos (high recent engagement)
- Filters out **old viral** videos (may be outdated)
- Finds **rising stars** (new high-quality content)
- Balances **popularity** with **freshness**

**Example scores:**
- New viral video: 10,000 views / 2 days = **5,000 views/day** ⭐⭐⭐⭐⭐
- Steady popular video: 100,000 views / 365 days = **274 views/day** ⭐⭐⭐
- Old viral video: 1,000,000 views / 1,000 days = **1,000 views/day** ⭐⭐⭐⭐

## Filtering Strategy

### Recommended Filters

**High-quality content only:**
```json
{
  "min_views": 1000,
  "min_duration_seconds": 300,
  "exclude_shorts": true
}
```

**Recent content focus:**
```json
{
  "max_age_days": 90,
  "min_views": 500
}
```

**Trusted sources:**
```json
{
  "preferred_channels": ["freeCodeCamp.org", "Alex Finn Official"],
  "exclude_keywords": ["scam", "fake", "clickbait"]
}
```

## Troubleshooting

### "No videos passed filters"

**Cause:** Filters too strict  
**Solution:** Lower `min_views` or increase `max_age_days`

### "Sign in to confirm you're not a bot"

**Cause:** YouTube bot detection  
**Solution:** Use `--cookies ~/.youtube-cookies.txt` (see cookie setup guide)

### "All scores are 0.0"

**Cause:** Upload date not available in search results  
**Note:** This is a known limitation of flat-playlist mode. Videos are still ranked by raw view count.

### Script runs but processes 0 videos

**Cause:** Missing OpenAI API key  
**Solution:** Set `OPENAI_API_KEY` env var or configure in Clawdbot

## Integration with Rivet Workflows

### Scheduled Intelligence Gathering

```bash
# Add to crontab
0 2 * * * /home/ccuser/rateright-growth/rivet/scripts/youtube-auto-intel.sh --cookies ~/.youtube-cookies.txt
```

### Content Pipeline

1. **Autonomous scan** finds top videos
2. **Process transcripts** extract insights
3. **Generate content** based on findings
4. **Post to LinkedIn/Twitter** with unique angle

### Competitor Monitoring

1. Configure topics for competitor brands
2. Run weekly scans
3. Review summary reports
4. Extract competitive intelligence
5. Update strategy based on findings

### Learning System

1. Search for "Clawdbot tutorials"
2. Process top videos automatically
3. Extract tips and tricks
4. Update AGENTS.md and TOOLS.md with learnings

## Advanced Configuration

### Multiple Config Files

```bash
# Competitor intel
python3 scripts/youtube-auto-intel.py --config competitors.json

# Learning content
python3 scripts/youtube-auto-intel.py --config learning.json

# Market research
python3 scripts/youtube-auto-intel.py --config market-research.json
```

### Priority-Based Processing

Set topic priorities to control which videos get processed first:

- **Priority 1:** Always process (critical topics)
- **Priority 2:** Process if quota available
- **Priority 3:** Manual review only

### Channel Preferences

Boost videos from trusted sources:

```json
{
  "filters": {
    "preferred_channels": [
      "Alex Finn Official",
      "freeCodeCamp.org"
    ]
  }
}
```

Currently informational only (future: boost ranking score)

## Cost Optimization

**OpenAI Whisper API:** $0.006/minute

**Strategies:**
- Set `top_videos_to_process: 3` for daily scans
- Use `max_age_days: 30` to avoid reprocessing old content
- Set `min_views: 5000` to focus on popular content only
- Run weekly instead of daily for most topics

**Example costs:**
- Daily (3 videos × 10 min) = ~$0.18/day = $5.40/month
- Weekly (5 videos × 15 min) = ~$0.45/week = $2.00/month

## Roadmap

- [ ] Deduplication across runs (don't reprocess same videos)
- [ ] Better performance scoring (engagement rate, like/view ratio)
- [ ] Sentiment analysis of comments
- [ ] Automatic topic generation based on trends
- [ ] Integration with Rivet knowledge base
- [ ] Slack/Telegram notifications when done
- [ ] Comparison reports (this week vs last week)

## Related Scripts

- `youtube-pipeline.py` - Manual single-video processing
- `youtube-pipeline.sh` - Bash wrapper for single videos
- `youtube-intel.py` - Legacy multi-video analyzer
- `youtube-api-scanner.py` - YouTube Data API integration

## Summary

**Before:** Manually find videos, copy URLs, paste into pipeline  
**After:** Configure topics once, run automatically, get reports

The autonomous system eliminates manual work and ensures you never miss important content in your niche.

**It's like having a research assistant that works 24/7.** 🤖