During a production incident — out-of-memory kills on a serverless API — I pulled logs from the observability dashboard, analyzed them carefully, and produced root causes that were confidently, completely wrong. Three times.
The reason wasn’t sloppy analysis. The dashboard’s log export capped at 2,000 rows — a sample small enough to invert the real priority order. Depending on what happened to fall inside the export window, I blamed large payloads, then an image gallery endpoint, then a single user. One surface I ranked first from the export turned out, in the full data, to account for 4 kills out of 11,058 — it streamed its responses and was never a problem at all. Meanwhile the actual top cause (46% of all kills) didn’t dominate any 2,000-row slice I happened to pull.
The fix was pulling the complete 7-day census through the observability API instead of the dashboard — every event in the window, not a truncated page of them. The real distribution emerged immediately, and the top two surfaces got fixed within a day.
Rules I’ve kept from this:
- Know your export’s cap. If you don’t know whether the data you’re ranking is complete, you don’t have a ranking; you have a sample with unknown bias.
- Truncated data doesn’t look truncated. It looks like clean, plausible evidence — that’s what makes it dangerous.
- Check log levels before filtering: our memory gauges were logged at
info, so an errors-only filter structurally couldn’t validate a memory claim, no matter how many rows it returned. - Write the full-census pull as a script the first time you need it. You will need it again.