---
title: Diagnostic Investigation Protocol
created: 2026-05-03
status: active
type: protocol
---

# Diagnostic Investigation Protocol

## The rule

When investigating system state, always work outside-in:

1. **External monitoring first** — UptimeRobot, external probes, 
   nginx access logs (what did users actually experience?)
2. **User-facing data second** — Supabase queries, app logs, 
   actual transactions (what did the system actually do?)
3. **Infrastructure third** — systemd events, PM2 status, 
   journalctl restart counts (what was the underlying state?)

## Why this order matters

Infrastructure events are not user-visible reality. A service can 
flap repeatedly while still serving requests. systemd can show 
50,000 restart events without the site being unreachable to users. 
Cached pages, nginx fallbacks, fast restart cycles — all of these 
mean infrastructure logs overstate downtime.

External monitoring tests what users actually experience. nginx 
access logs show what users actually got. These are ground truth. 
Infrastructure logs are interpretation.

## The failure mode this prevents

2026-05-03: Claude Code reported a "49-day downtime" of 
rateright.com.au based on systemd restart counts (162,633 events 
between Mar 13 and May 3). This was incorrect. UptimeRobot showed 
94.344% uptime for April with 2 incidents totalling ~1.7 days. 
The service was flapping, not down.

Compounding errors:
- Claude Code interpreted infrastructure events as user reality
- Chat (Claude on phone) amplified the conclusion into a 
  business narrative without checking source
- Both could have been caught by checking external monitoring first

## Required elements in any state report

Any agent report claiming user-facing system state must include:

1. **Data source** — exact file path, log query, or API response
2. **Reasoning chain** — how the data led to the conclusion
3. **Falsification test** — what evidence would prove this wrong
4. **External corroboration** — does external monitoring agree?

If a conclusion contradicts Rocky's direct experience, the 
direct experience wins until external evidence resolves the 
contradiction.

## When this applies

- Site uptime / downtime claims
- "X is broken" / "Y is working" claims
- Customer-facing functionality claims
- Database state claims affecting user flows
- Any claim about what a real user would experience

## When this does NOT apply

- Pure infrastructure investigations (DB schema, agent skills, 
  vault structure) — internal data is appropriate ground truth
- Performance optimisation — infrastructure metrics matter directly
- Debugging the agents themselves — agent logs are ground truth 
  for agent behaviour