Yueseryueser.com

← All workflows

Production is down: how do I debug it step by step?

The slow part of an incident is rarely the fix. It is not knowing what you are looking at. Freeze the scene before you touch the code.

How to do it

  1. Turn the logs into one timeline

    Give the logs from the incident window, the services involved, and the time range to 'Read the Failure Out of the Logs'. Ask for the chain of events in time order, the first anomalous point, a two-column split between what the logs prove and what is inferred, and the specific log line still missing. Do not let it start from the first ERROR.

  2. Rank the causes, then verify each one

    Using the timeline and the still-missing log from step 1, hand the stack version, full trace, and everything you have tried to 'Locate the error'. It should rank causes by likelihood and give one command or minimal reproduction per cause. It should skip what you tried and say what information is missing rather than guessing further.

  3. Write the smallest test that fails before the fix

    Give the symptom and the relevant code to 'Write the Smallest Failing Test'. Keep only the input and state needed to trigger it, assert expected versus actual, and confirm it fails before the fix and passes after. If it cannot reproduce, have it list the missing environment conditions instead of inventing steps.

  4. Check what the fix can break before you ship it

    Once the fix is decided, give the diff and dependency map to 'Map the Blast Radius of a Change'. It should list affected callers, data, and scheduled jobs, rank the top three by people affected and recovery difficulty, and set rollback thresholds. A one-line fix can still invalidate a downstream assumption.

  5. Write a postmortem that can be audited

    Finally, give the timeline and response log to 'Write a Postmortem That Holds Up'. Keep only nodes that changed system state or a decision, all in one timezone; trace the root cause to the missing mechanism; split action items into done and pending with owner role, due date, and how completion will be verified. Add a separate section on what went right by luck.

Prompts used

FAQ

Should I restart first or read the logs first?

Preserve the scene first. Save the logs, metrics, and recent deploy records for the incident window, then decide about a restart. A restart wipes in-memory state and temporary logs, and after that much of the investigation becomes guesswork.

Can AI just tell me where the bug is?

It gives ranked hypotheses and ways to test them, not the answer. Verify them in step 2, then confirm with the failing test from step 3. Do not change code before you can reproduce the failure; otherwise you are just moving it somewhere else.

The incident is fixed. Do we still need a postmortem?

Fixing it only addresses this occurrence. The value of a postmortem is turning 'which assumption was wrong' and 'which mechanism was missing' into action items with an owner and a check. A postmortem that says 'improve monitoring' is not a postmortem.