Things that broke, written down properly

by @grep·5 jots

Failure writing I send to new hires. Two of these are not about software, which is deliberate.

Postmortem: 43 minutes of 502s and one very confident regex
Our gateway returned 502 for forty three minutes because of a regex I would have approved too. Full timeline, the pattern, and the dashboard that lied to me.
@grepstars: 79 min read
Mine. Included so nobody thinks I am only critical of other people's incidents.
Scraping a 1943 lathe bed flat
A 1943 South Bend that showed up 78 tenths hollow at the headstock end. What I measured, how I scraped it, and the three weeks I spent making it worse.
@rustbeltstars: 1010 min read
The closest thing to a good postmortem I have read this year, and it is about a lathe.
The cash was in the Philippines, allegedly
Wirecard as an accounting problem rather than a thriller: what a trustee account is, what a bank confirmation is meant to do, and the ordinary audit steps that settle it.
@ledgerstars: 119 min read
What happens when the control that would have caught it existed and nobody ran it.
It is always DNS, but here is the actual reason
Why DNS breaks things out of proportion to its size, with real dig output read line by line, plus the four commands I run to prove it is not DNS.
@grepstars: 36 min read
Read the ninety second section and skip the rest if you are busy.
I lost a won endgame and I want you to see it
A Lucena position I knew cold, and the one move on move 61 where I stopped asking which piece was supposed to move. Full annotation, clock times included.
@enpassantstars: 117 min read
Blameless retrospective on a single decision, under time pressure. I have sent this to more engineers than anything else here.