# Gmail Search & Send — Verified Patterns (LFCS)

> **Pitfall (2026-06-30):** `format=metadata&metadataHeaders=...` intermittently returns `payload.headers: []` on this VPS. For cron jobs and any "must work first time" path, use `format=full` and read `payload.headers` the same way. See "Critical: `from:` operator..." section below for the search caveat that still applies.

> **Pitfall (2026-07-06):** `google_token.json` has `"expiry": 1783281683.710308` (epoch float) instead of an ISO string. `Credentials.from_authorized_user_file()` raises `AttributeError: 'float' object has no attribute 'rstrip'`. **Workaround:** skip the `Credentials` object entirely, refresh the token manually with `urllib`, and use the access_token as a Bearer header. The access_token still works fine — only the auto-refresh path breaks. See "Bypass: manual refresh + Bearer" below. Fix the file shape when convenient (replace the float with its ISO 8601 equivalent).

## Bypass: manual refresh + Bearer (when google_token.json has bad expiry)

When the standard `Credentials.from_authorized_user_file()` path errors on a malformed `expiry` field, refresh the token by hand and use it directly. Confirmed working 2026-07-06 against `michael.mcloughlin@lfcs.com.au` (593 messages):

```python
import urllib.request, urllib.parse, json

token_data = json.load(open('/root/.hermes/google_token.json'))
req = urllib.request.Request('https://oauth2.googleapis.com/token',
    data=urllib.parse.urlencode({
        'client_id': token_data['client_id'],
        'client_secret': token_data['client_secret'],
        'refresh_token': token_data['refresh_token'],
        'grant_type': 'refresh_token',
    }).encode())
with urllib.request.urlopen(req) as r:
    tok = json.loads(r.read())['access_token']

# Then use `tok` as a Bearer header everywhere:
req = urllib.request.Request(
    'https://gmail.googleapis.com/gmail/v1/users/me/profile',
    headers={'Authorization': f'Bearer {tok}'}
)
```

This bypasses the google-auth library entirely. The same Bearer pattern works for search, message fetch, and send (raw payload, base64url-encoded). No `Credentials` object, no auto-refresh, no expiry parsing — just urllib + the token file's raw fields.

**Permanent fix:** rewrite `expiry` in `google_token.json` to its ISO 8601 form (`datetime.utcfromtimestamp(1783281683.710308).isoformat() + 'Z'`). Do this as a one-off shell command when next at the keyboard; not blocking for the manual-refresh path.

## Verified working: direct Python calls

When `google_api.py` fails or shows misleading errors, use direct REST calls via urllib:

```python
from google.oauth2.credentials import Credentials
from google.auth.transport.requests import Request
import urllib.request, json, urllib.parse
from email.mime.text import MIMEText
import base64

creds = Credentials.from_authorized_user_file('/root/.hermes/google_token.json',
    ['https://www.googleapis.com/auth/gmail.readonly',
     'https://www.googleapis.com/auth/gmail.send'])
creds.refresh(Request())  # idempotent even if token is still valid

# --- Search emails ---
query = urllib.parse.quote('from:Robert.Logan@decprojects.com.au after:2026/05/11')
url = f'https://gmail.googleapis.com/gmail/v1/users/me/messages?q={query}&maxResults=10'
req = urllib.request.Request(url, headers={'Authorization': f'Bearer {creds.token}'})
with urllib.request.urlopen(req) as r:
    data = json.loads(r.read())
    for m in data.get('messages', []):
        print(m['id'])

# --- Read message metadata ---
msg_url = f'https://gmail.googleapis.com/gmail/v1/users/me/messages/{msg_id}?format=metadata&metadataHeaders=From&metadataHeaders=Subject&metadataHeaders=Date'
# or include metadataHeaders=Body for snippet

# --- SAFER FALLBACK: format=full (2026-06-30) ---
# format=metadata intermittently returns empty headers arrays on this VPS.
# Default to format=full for cron-driven digests; pull the same payload.headers.
msg_url_full = f'https://gmail.googleapis.com/gmail/v1/users/me/messages/{msg_id}?format=full'

# --- Send email ---
msg = MIMEText("Email body text here")
msg['to'] = 'admin@lfcs.com.au'
msg['subject'] = 'Subject line'
raw = base64.urlsafe_b64encode(msg.as_bytes()).decode()
service = build('gmail', 'v1', credentials=creds)
result = service.users().messages().send(userId='me', body={'raw': raw}).execute()
```

## Critical: `from:` operator matches email address, not display name

**Gmail `from:tenders.dec` returned 0 results** even though emails existed from this sender. The display name was "DEC Projects" or similar, but the actual sender email was `Robert.Logan@decprojects.com.au`. Gmail's `from:` searches the envelope sender address.

**Rule: always search by the known email address, not the display name or domain fragment.**

Workaround for unknown exact sender: use `subject:tender` or broader search, then inspect message headers to find the actual from address, then re-search with the exact address.

## Confirm account identity before debugging search failures

```python
profile_url = 'https://gmail.googleapis.com/gmail/v1/users/me/profile'
# GET with Bearer token → returns emailAddress, messagesTotal, threadsTotal
```
Always check which account the token is connected to before assuming "no results" means no emails.

## google_api.py vs direct calls

`google_api.py` (gws CLI) prints errors to stderr even when the underlying API call succeeds. The `opsman-lfcs-27665` error from gws is a routing issue in the CLI wrapper — the actual Gmail API call via direct Python often works fine. When in doubt: ignore gws error output, run the direct Python pattern above.

## himalaya email client

Install: `curl -sSL https://raw.githubusercontent.com/pimalaya/himalaya/master/install.sh | PREFIX=~/.local sh`
Requires config at `~/.config/himalaya/config.toml` with IMAP/SMTP credentials. Not currently configured for LFCS — uses Google Workspace OAuth instead.

## Forward message as attachment (RFC822) — verified 2026-05-12

Use the raw endpoint to get original MIME bytes, then send via `messages/send` with a wrapped RFC822 payload:

```python
# Get raw message
url = f'https://gmail.googleapis.com/gmail/v1/users/me/messages/{msg_id}?format=raw'
req = urllib.request.Request(url, headers={'Authorization': f'Bearer {creds.token}'})
with urllib.request.urlopen(req) as r:
    data = json.loads(r.read())
    raw_b64 = data['raw']
    pad = len(raw_b64) % 4
    if pad: raw_b64 += '=' * (4 - pad)
    msg_bytes = base64.urlsafe_b64decode(raw_b64)

original = email.message_from_bytes(msg_bytes)
orig_subject = original.get('Subject', '(no subject)')

# Wrap and send as RFC822 forward
fwd_raw = (f"From: michael.mcloughlin@lfcs.com.au\r\n"
           f"To: admin@lfcs.com.au\r\n"
           f"Subject: Fwd: {orig_subject}\r\n"
           f"Content-Type: message/rfc822; charset=utf-8\r\n\r\n").encode('utf-8') + msg_bytes
b64_fwd = base64.urlsafe_b64encode(fwd_raw).decode('ascii').rstrip('=')

send_req = urllib.request.Request(
    'https://gmail.googleapis.com/gmail/v1/users/me/messages/send',
    data=json.dumps({'raw': b64_fwd}).encode(),
    headers={'Authorization': f'Bearer {creds.token}', 'Content-Type': 'application/json'}
)
with urllib.request.urlopen(send_req) as r:
    result = json.loads(r.read())
    print('Sent:', result.get('id'))
```

Send each message as a separate call — loops cause timeouts.

## Gmail triage workflow — what fires "what's happening with my emails"

Triggered when Rocky asks "what's going on with my emails", "inbox", "anything I missed", "what needs a reply". Produces a structured digest with action queues, not a raw list.

### Procedure (verified 2026-07-06)

1. **Refresh token manually** if `Credentials` lib chokes — see "Bypass" section above.
2. **Confirm account identity** via `users/me/profile` before reporting numbers. Surface `emailAddress`, `messagesTotal`, `threadsTotal`.
3. **Search inbox last 7 days** with `q=after:{epoch_ts} in:inbox&maxResults=25`. Use `format=full` for headers — `format=metadata` is intermittently empty.
4. **Parse per message**: extract `From`, `Subject`, `Date` from `payload.headers`. Split `From` into display name + email address (`name <addr@x.com>`).
5. **Sort newest first**. Format: `Weekday DD/MM HH:MM  Display Name <addr@x>` then indented subject line.
6. **Group in digest** by category, not chronologically:
   - **Money / claims / variations** — anything that affects revenue (VAR-XXX, day-works reprice, payment claims, RFI cost impacts)
   - **Tenders / RFQs out** — replies to bids you've submitted
   - **Active ops** — ongoing threads with multiple recent emails
   - **Out of office / noise** — auto-replies, marketing, security alerts
   - **Things needing a decision today** — explicit checklist with 2-4 items
7. **End with infrastructure notes** if any: bad token file, rate limits hit, scopes missing.
8. **Offer next actions**, not summaries. "Want me to draft X / chase Y / file Z?" beats "Here's everything that happened."

### Pitfalls

- **Don't dump 25 messages flat.** Group by category or the digest becomes wallpaper.
- **Search by exact address, not display name.** Same rule as `from:` operator pitfall above.
- **`in:inbox` only** — exclude Sent/Drafts/Spam unless Rocky asks.
- **Auto-replies are noise.** Filter out `Auto-Submitted: auto-replied` headers.
- **Security alerts (Google, Microsoft) are real but rare.** Surface once with date, don't dwell.
- **Token expiry:** if `Credentials` errors with `'float' object has no attribute 'rstrip'`, switch to the manual-refresh path. Don't waste a turn reporting "auth broken" — bypass and report what you actually found.
- **"Did I send X" / "did they reply" questions — check SENT, not just INBOX.** Rocky will ask "I sent these offers, can you not see it" — that's a SENT-folder question. Default Gmail queries are inbox-only and miss his Saturday/Sunday-afternoon sends. Always include `in:sent` (or `in:anywhere`) when Rocky's claim is "I sent it" or the question is "did I follow up". The first Bankstown investigation 2026-07-06 missed the $140,892.58 Rev E send because the search scoped to inbox only; corrected on Rocky's pushback.
- **When Rocky pushes back with "I sent it / I already did X" — believe him first, then re-prove with evidence.** Don't argue from your last query result. Re-run with broader scope (Sent + Drafts + Trash) before responding again. Faster and less embarrassing than a second correction.
- **Job-status briefing structure that worked for 2626 Peakhurst (2026-07-06):** Done / in the bank → 🔴 Waiting on them → 🟡 Your queue (assigned to Me) → 🟢 Watching (traps like battle-of-forms). End with one explicit next move, not a summary. Rocky accepted the dense state dump without pushback — this is the right shape for "what's happening on job X" investigations.

## Gmail sent-mail trigger — "did I send X" / "did they reply"

Triggered when Rocky asks "did I send that offer", "is that reply gone", "have I followed up with X", or pushes back with "I sent it, can't you see it". Default inbox searches will miss his outbox — SENT is a separate scope.

### Procedure (verified 2026-07-06 — Bankstown substation surrounds)

1. **When the question is "did I send X to <client>"** — search with `from:me to:<recipient-domain> after:<date>` to scope to outgoing only.
2. **When the question is about a specific thread** — fetch the thread directly via `users/me/threads/{threadId}` and inspect every message including the SENT ones (they show labels `SENT`, not `INBOX`).
3. **When in doubt, run `in:sent`** explicitly alongside `in:inbox`. Both return different message sets; both carry their own label.
4. **HTML-only emails show as "(no plain body — likely HTML only)"** when fetching via plain-text extraction. Decode the `text/html` part with `html.unescape` + strip tags if you need the actual body. Don't conclude "no body" — try HTML extraction.
5. **Attachments appear in `payload.parts[]`** with `filename`, `mimeType`, `attachmentId`. Note the `attachmentId` even if you don't download — useful for "did I attach the PDF" questions.

### Quick search recipes

```python
# All sent mail to a recipient (last N days)
q = f'from:me to:{recipient_domain} after:2026/06/25'
url = 'https://gmail.googleapis.com/gmail/v1/users/me/messages?q=' + urllib.parse.quote(q)

# Specific subject in sent
q = 'subject:"2652 Bankstown" in:sent'

# All threads (sent + received) involving a recipient
q = f'to:{recipient_domain} OR from:{recipient_domain}'
```

### Pitfalls

- **Inbox scope is the default trap.** Searching `q=in:inbox` returns Rocky's RECEIVED mail only. For "did I send it" questions, swap to `in:sent` or remove the scope entirely.
- **ThreadId ≠ messageId.** A thread contains N messages; some are SENT, some are INBOX. To reconstruct a full conversation including Rocky's outgoing replies, fetch the thread (`/threads/{id}`) not just one message.
- **HTML body vs plain body** — modern Outlook/Gmail sends HTML-only. The plain-text extraction returns empty for these. Decode the HTML part separately.
- **"Drafts" label matters.** A staged draft that was never sent has `DRAFT,TRASH` labels and never reaches `SENT`. Don't conflate "in the thread" with "actually sent".

## Job-status briefing shape — "what's happening on job X"

Triggered when Rocky asks "investigate job <num>" or "what's the state of <job>". Produces a dense, decision-ready briefing, not a chronological narrative.

### Procedure (verified 2026-07-06 — 2626 Mack Civil Peakhurst)

1. **Pull the Jobs record** from Airtable (base `appE43UvTyARe5oJs`, table `tblcL2EX2w34c1VPe`, filter on Job Number). Read the `Notes` field — it carries the rolling session-by-session log.
2. **Pull open Actions** from the Actions table (`tblwGQdNqZIPRiDSV`) filtered on `Job = <job_id>`. These are the live chase list.
3. **Structure the response in four sections:**
   - **🟢 Done / in the bank** — what is confirmed, dated, with amount/recipient if money
   - **🔴 Waiting on them** — what's outstanding where someone else owes Rocky a reply or artefact (chase line, due date, waiting-since date)
   - **🟡 Your queue (assigned to Me = Hermes)** — actions where I'm the owner, with due dates
   - **🟢 Watching (traps)** — risks that don't need action today but will bite if missed (battle-of-forms, blackout windows, scope creep signals)
4. **End with one explicit next move.** "Want me to: 1) surface the staged draft, 2) check the blackout schedule, 3) send the email?" — three concrete options, pick the highest-leverage one and lead with it.
5. **Total length: ~30 lines.** Long enough to be useful, short enough to read on a phone in one screen.

### Pitfalls

- **Don't recap the Notes field verbatim.** Synthesise into the four sections. The Notes is source material, not the deliverable.
- **Don't lead with "the bid is $X ex / $Y inc"** — that's the lowest-leverage fact. Lead with the bottleneck / unblock.
- **Don't list all Actions.** Show only what's still open and matters this week.
- **Don't apologise for state** ("lots going on") — Rocky knows. Just show it.