# VPS Cron Script Traps — Lessons from Production

Documented from live fixes on Rocky's VPS (2026-06-12). These are Hermes-adjacent system scripts that run on the same host as hermes-gateway.

---

## hermes-health-snapshot.sh — 17-day hang root cause

**Script:** `/usr/local/bin/hermes-health-snapshot.sh`
**Trigger:** Runs via `/etc/cron.d/hermes-health` at 21:00 daily
**Output:** `/home/ccuser/opsman-work/health-log/YYYY-MM-DD.md`

### Bug 1: `set -a` + `.env` with base64 secrets = silent corruption

**Problem:** The script used `set -a` (export all sourced vars) + `source /root/.hermes/.env`. The `.env` contains `HERMES_DASHBOARD_BASIC_AUTH_SECRET=<base64 string ending in =>` and `HERMES_DASHBOARD_BASIC_AUTH_PASSWORD=...`. Under `set -a`, every variable is auto-exported. A malformed read (IFS==, whitespace, = in value) causes a partial var set that `set -u` then crashes on mid-script.

**Fix:** Replace blanket sourcing with targeted extraction:
```bash
# Only extract known vars — never blanket source .env with set -a
if [ -f /root/.hermes/.env ]; then
  while IFS='=' read -r key val; do
    case "$key" in
      MINIMAX_API_KEY|OPENROUTER_API_KEY|DEEPSEEK_API_KEY|OPENAI_API_KEY)
        export "$key"="$val"
        ;;
    esac
  done < <(grep -vE '^[[:space:]]*#' /root/.hermes/.env)
fi
```

### Bug 2: Wrong MiniMax API endpoint

**Problem:** Health probe called `https://api.minimaxi.chat/v1/text/chatcompletion_v2` — wrong host (should be `api.minimax.io`). Curl times out after 10s but doesn't fail hard. With `set -u` active, the timeout + subsequent undefined var reference causes the heredoc block to never close.

**Fix:**
```bash
# Correct endpoint:
code=$(curl -sS -o /dev/null -w "%{http_code}" \
  --max-time 10 --fail \
  -H "Authorization: Bearer ${MINIMAX_API_KEY}" \
  "https://api.minimax.io/v1/text/chatcompletion_v2" \
  -X POST -H "Content-Type: application/json" \
  -d '{"model":"MiniMax-M2.7","messages":[{"role":"user","content":"ok"}],"max_tokens":1}' \
  2>&1 || echo "curl_err")
```

### Bug 3: `pm2 list` hangs on non-interactive TTY

**Problem:** `pm2 list --no-color` hangs indefinitely when run from a cron script with no TTY attached. The command produces output but never returns.

**Fix:** Use `pm2 jlist` (JSON output) instead:
```bash
if timeout 10 pm2 jlist 2>/dev/null | python3 -c "
import sys, json
data = json.load(sys.stdin)
for p in data:
    s = p.get('pm2_env',{}).get('status','')
    if s in ('online','stopped','errored'):
        print(f'    {p[\"name\"]}: {s}')
" 2>/dev/null; then
  : # output was produced
else
  echo "    (pm2 unavailable or timed out)"
fi
```

### Bug 4: No flock guard — stacked instances

**Problem:** If the script hangs (as it did for 17 days), cron fires a new instance each night at 21:00. Without a guard, multiple instances stack.

**Fix:**
```bash
LOCKFILE="/run/hermes-health-snapshot.lock"
PIDFILE="/run/hermes-health-snapshot.pid"

exec 200>"$LOCKFILE"
if ! flock -n 200; then
  echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') already running, exiting" >&2
  exit 0
fi
echo $$ > "$PIDFILE"
# ... script body ...
rm -f "$PIDFILE"
```

### Bug 5: Heredoc partial file on crash

**Problem:** The script uses `} > "$OUT"` — if `set -u` fires mid-block (undefined var), the heredoc never closes and `chown ccuser:ccuser "$OUT"` never runs. Output file is empty or partial, owned by root.

**Fix:** Write to temp file then mv:
```bash
} > "${OUT}.tmp"
mv "${OUT}.tmp" "${OUT}"
chown ccuser:ccuser "$OUT"
```

---

## Google OAuth Token — Programmatic Refresh (Preferred over Manual Re-auth)

**Token path:** `~/.hermes/google_token.json`

The token file contains both `token` (access token, short-lived) and `refresh_token` (long-lived). Cron jobs should attempt the API call first; on 401, refresh programmatically — no manual re-auth needed.

**Step 1 — Try the API call with stored access_token:**
```bash
TOKEN=$(python3 -c "import json; t=json.load(open('~/.hermes/google_token.json')); print(t['token'])")
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://gmail.googleapis.com/gmail/v1/users/me/messages?q=is:unread&maxResults=5&format=metadata"
```

**Step 2 — On 401: refresh using the stored refresh_token:**
```bash
TOKEN_JSON=$(python3 -c "import json; print(json.load(open('~/.hermes/google_token.json'))['refresh_token'])")
CLIENT_ID=$(python3 -c "import json; print(json.load(open('~/.hermes/google_token.json'))['client_id'])")
CLIENT_SECRET=$(python3 -c "import json; print(json.load(open('~/.hermes/google_token.json'))['client_secret'])")

NEW_TOKEN=$(curl -s -X POST https://oauth2.googleapis.com/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "client_id=${CLIENT_ID}&client_secret=${CLIENT_SECRET}&refresh_token=${TOKEN_JSON}&grant_type=refresh_token" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")
```

**Step 3 — Update the token file (replace old access_token, update expiry):**
```bash
python3 -c "
import json, time
path='/root/.hermes/google_token.json'
with open(path) as f: t=json.load(f)
t['token']='$NEW_TOKEN'
t['expiry']=time.time()+3600
with open(path,'w') as f: json.dump(t,f)
"
```

**Step 4 — Retry the original API call with the new token.**

**Only fall back to manual re-auth if the refresh fails** (e.g., refresh_token revoked or client deleted):
```bash
python3 /usr/local/lib/hermes-agent/setup.py --auth-url
```
Visit the URL, sign in to mcloughlinmichael.r@gmail.com, approve scopes (gmail.readonly + calendar). Token auto-saves.

**Important field notes:**
- `account` field in token file: empty string = unverified/unauthenticated
- `expiry` field: Unix timestamp. If `date +%s` > expiry, token is expired
- Never fabricate Gmail or Calendar data when token is invalid — send the fallback warning message instead

---

## Dashboard Auth — Verify Before Reporting Success

After starting the dashboard (via systemd or bare command), always verify auth is engaged:
```bash
curl -s -I http://<tailscale-ip>:9119
# Expected: HTTP/1.1 302 Found + location: /login?next=%2F
# Wrong: HTTP/1.1 200 with HTML body = dashboard is open
```

The 302 redirect is the auth gate confirming basic auth is active. Any other response means auth failed to initialize.