Read the Failure Out of the Logs
Turn scattered log lines into one timeline, with evidence separated from guesses
You are an engineer debugging a production incident. Below is a slice of logs, possibly from several services. Do four things: 1) build a timeline of events and mark the first anomaly; 2) split what the logs prove from what you are inferring into two columns; 3) name the log line still missing that would settle the cause; 4) give the smallest next verification step. Do not start from the first ERROR line — find what happened before it. [log excerpt / time range / affected services]
How to use
Normalize every timestamp to one timezone before pasting. The most valuable clue is usually the first request that slowed down or timed out, not the first ERROR.
Good for
- Checkout latency jumps from 200 ms to 4 s, the dashboards stay green, and users have started reporting errors.
- Only one customer's requests are failing while the aggregate error rate barely moves, so you need the logs to show what is different.
Not for
- You have one stack trace and no request id, timestamp, or surrounding lines: add logging first, not analysis.
- You already know a config value is wrong and just need to change it back: fix it directly, no timeline required.
Example
Input
2026-03-14T02:11:03Z api ERROR upstream timeout path=/checkout uid=8821 2026-03-14T02:11:04Z api WARN pool wait 1980ms active=20 idle=0 2026-03-14T02:11:05Z api ERROR upstream timeout path=/checkout uid=8821 2026-03-14T02:11:07Z db slow query 4.2s sql=SELECT ... FOR UPDATE
What you get
The first anomaly is the timeout at 02:11:03, but the turning point in the timeline is the connection-pool wait at 02:11:04: active=20, idle=0, and a request already waiting 1,980 ms suggest that the pool was already exhausted well before that timeout, which is only the result. The 4.2 s slow query appears later and looks more like a consequence of connections being held; with no lock-wait target in the log, it does not prove the database is the cause. Ask for the pool config and the `lock wait` line to separate 'pool too small' from 'query got slower'.
Common mistakes
- Pasting only the ERROR lines and dropping the WARN and slow-query lines: the actual cause usually lives in those two.
- Leaving out the time range and instance and asking it to 'analyze the cause': it will stitch logs from different machines into a causal chain that never happened.
Try tweaking it like this
- Add 'group by service, then align timestamps' and a multi-service log stops being read as one serial machine.
- Require 'cite the source line after every conclusion' and you can see at a glance which claims are evidence and which are invention.