# VPS Cron Script Traps — Lessons from Production

Documented from live fixes on Rocky's VPS (2026-06-12). These are Hermes-adjacent system scripts that run on the same host as hermes-gateway.

---

## hermes-health-snapshot.sh — 17-day hang root cause

**Script:** `/usr/local/bin/hermes-health-snapshot.sh`
**Trigger:** Runs via `/etc/cron.d/hermes-health` at 21:00 daily
**Output:** `/home/ccuser/opsman-work/health-log/YYYY-MM-DD.md`

### Bug 1: `set -a` + `.env` with base64 secrets = silent corruption

**Problem:** The script used `set -a` (export all sourced vars) + `source /root/.hermes/.env`. The `.env` contains `HERMES_DASHBOARD_BASIC_AUTH_SECRET=<base64 string ending in =>` and `HERMES_DASHBOARD_BASIC_AUTH_PASSWORD=...`. Under `set -a`, every variable is auto-exported. A malformed read (IFS==, whitespace, = in value) causes a partial var set that `set -u` then crashes on mid-script.

**Fix:** Replace blanket sourcing with targeted extraction:
```bash
# Only extract known vars — never blanket source .env with set -a
if [ -f /root/.hermes/.env ]; then
  while IFS='=' read -r key val; do
    case "$key" in
      MINIMAX_API_KEY|OPENROUTER_API_KEY|DEEPSEEK_API_KEY|OPENAI_API_KEY)
        export "$key"="$val"
        ;;
    esac
  done < <(grep -vE '^[[:space:]]*#' /root/.hermes/.env)
fi
```

### Bug 2: Wrong MiniMax API endpoint

**Problem:** Health probe called `https://api.minimaxi.chat/v1/text/chatcompletion_v2` — wrong host (should be `api.minimax.io`). Curl times out after 10s but doesn't fail hard. With `set -u` active, the timeout + subsequent undefined var reference causes the heredoc block to never close.

**Fix:**
```bash
# Correct endpoint:
code=$(curl -sS -o /dev/null -w "%{http_code}" \
  --max-time 10 --fail \
  -H "Authorization: Bearer ${MINIMAX_API_KEY}" \
  "https://api.minimax.io/v1/text/chatcompletion_v2" \
  -X POST -H "Content-Type: application/json" \
  -d '{"model":"MiniMax-M2.7","messages":[{"role":"user","content":"ok"}],"max_tokens":1}' \
  2>&1 || echo "curl_err")
```

### Bug 3: `pm2 list` hangs on non-interactive TTY

**Problem:** `pm2 list --no-color` hangs indefinitely when run from a cron script with no TTY attached. The command produces output but never returns.

**Fix:** Use `pm2 jlist` (JSON output) instead:
```bash
if timeout 10 pm2 jlist 2>/dev/null | python3 -c "
import sys, json
data = json.load(sys.stdin)
for p in data:
    s = p.get('pm2_env',{}).get('status','')
    if s in ('online','stopped','errored'):
        print(f'    {p[\"name\"]}: {s}')
" 2>/dev/null; then
  : # output was produced
else
  echo "    (pm2 unavailable or timed out)"
fi
```

### Bug 4: No flock guard — stacked instances

**Problem:** If the script hangs (as it did for 17 days), cron fires a new instance each night at 21:00. Without a guard, multiple instances stack.

**Fix:**
```bash
LOCKFILE="/run/hermes-health-snapshot.lock"
PIDFILE="/run/hermes-health-snapshot.pid"

exec 200>"$LOCKFILE"
if ! flock -n 200; then
  echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') already running, exiting" >&2
  exit 0
fi
echo $$ > "$PIDFILE"
# ... script body ...
rm -f "$PIDFILE"
```

### Bug 5: Heredoc partial file on crash

**Problem:** The script uses `} > "$OUT"` — if `set -u` fires mid-block (undefined var), the heredoc never closes and `chown ccuser:ccuser "$OUT"` never runs. Output file is empty or partial, owned by root.

**Fix:** Write to temp file then mv:
```bash
} > "${OUT}.tmp"
mv "${OUT}.tmp" "${OUT}"
chown ccuser:ccuser "$OUT"
```

---

## Google OAuth Token — Expiry and Re-auth

**Token path:** `~/.hermes/google_token.json`
**Token account field:** `account` — empty string means unverified/unauthenticated
**Token expiry:** `expiry` field — if in the past, token is expired

**Re-auth command:**
```bash
python3 /usr/local/lib/hermes-agent/setup.py --auth-url
```
This prints a URL to visit in a browser. Sign in to the correct Google account (mcloughlinmichael.r@gmail.com), approve scopes (gmail.readonly + calendar). Token auto-saves to `~/.hermes/google_token.json`.

**Scopes currently in token:** contacts.readonly, gmail.modify, documents.readonly, gmail.readonly, drive.readonly, gmail.send, spreadsheets, calendar, documents, drive.file. Expired May 11 2026.

**For morning-digest cron job:** If token is expired, the job sends a fallback message instructing re-auth. Do not fabricate emails when the token is invalid.

---

## Dashboard Auth — Verify Before Reporting Success

After starting the dashboard (via systemd or bare command), always verify auth is engaged:
```bash
curl -s -I http://<tailscale-ip>:9119
# Expected: HTTP/1.1 302 Found + location: /login?next=%2F
# Wrong: HTTP/1.1 200 with HTML body = dashboard is open
```

The 302 redirect is the auth gate confirming basic auth is active. Any other response means auth failed to initialize.