Yueser ← Back to library

AI / Coding and Debugging

Write a Postmortem That Holds Up

Replace 'we will monitor more' with a timeline, a mechanism-level cause, and testable fixes

You are facilitating an incident postmortem. Using the timeline and response notes below, write the review: 1) keep only the moments that changed system state or a decision, with one timezone stated; 2) take the root cause down to the missing mechanism or the assumption that turned out false — 'human error' is where the analysis starts, not where it ends; 3) split actions into done and outstanding, each with an owning role, a due date, and a way to verify it; 4) add a separate section on what went right by luck. Do not include 'improve monitoring' or 'raise awareness' — anything that cannot be verified.

[incident timeline / impact / response notes]
How to use

The 'what went right by luck' section exposes the system's real margin. If you cannot write it, the review is still on the surface.

Good for

  • An incident touched real users, and both engineering and leadership need one shared account of what happened.
  • The same class of failure has happened twice, and the previous postmortem's actions clearly did not land.

Not for

  • The incident is still being mitigated and the cause is unknown: stabilize first, write the review later.
  • A single alert fired with no user impact and no state change: a postmortem would manufacture significance.

Example

Input
14:02 deployed v2.3
14:05 error rate 0.1% -> 8%
14:09 rollback started
14:16 rollback complete, error rate normal
14:30 found rollback only reverted the app, not the DB migration
note: peak sale day, traffic 3x normal
What you get
The cause is not 'the deploy': the rollback plan covered the application but not the migration, so the rollback was never complete, and that missing mechanism is the finding. Sale-day traffic may have amplified the impact, but it is not the root cause, so keep it out of the causal sentence. Add who decided to roll back at 14:09 and on which metric, then make the action 'rollback checklist includes migrations and is rehearsed', verified by a recorded dry run before the next release.

Common mistakes

  • Writing it as a blame assignment: everyone shifts to defensive wording and the missing mechanism never makes it onto the page.
  • An action item like 'improve the release process': no owner and no verification means the same sentence comes back in three months.

Try tweaking it like this

  • Add 'write the information gaps from the on-call engineer's seat that night' to surface organizational problems the dashboard cannot show.
  • Require 'every action needs a test that can fail' and 'fixed' turns from a claim into evidence.

More in this topic