The joke does real work and it is also lazy. DNS takes a disproportionate share of outages for four reasons, and none of them is that the protocol is complicated. It is one of the simplest things we run.
You edit a record at 14:02. Nothing happens. At 14:04 nothing happens. At 14:09 about 3% of one customer's traffic starts failing, then 9%, then it plateaus somewhere, then it drips for a day and a half.
Everything else we operate gives feedback inside the blast radius of the change that caused it. A bad deploy shows up on the pods you just replaced. A bad migration shows up in the transaction that ran it. DNS hands the feedback to every cache between you and everyone, each running its own clock, and by the time the graph moves your change is forty minutes back in the audit log and below the fold.
Here is what one well run zone publishes, asked of its own authority so nothing in between can round it off:
$ dig @ns-520.awsdns-01.net github.com A +norecurse +noall +answer
github.com. 60 IN A 20.87.245.0
Sixty seconds. Now what actually happens to that sixty.
glibc's resolver does not cache anything. A process calling getaddrinfo goes out to whatever is in resolv.conf every time, unless something like systemd-resolved or dnsmasq is sitting in front of it, in which case the caching policy is that daemon's and not yours.
The JDK does cache. Failed lookups for ten seconds by default. Successful ones for a period the javadoc describes as implementation dependent, and forever if a security manager is installed, which was a sensible anti-spoofing decision in 1999 and has ruined more failovers than any single misconfiguration I can name.
Recursive resolvers are allowed to hand out records after they expire. RFC 8767 spells out the mechanism: when a refresh fails, return the stale record with a short TTL, recommended 30 seconds, and keep the stale copy around for a suggested one to three days.
So the same name, in the same second, holds different values in your Python process, your JVM, your sidecar and your customer's ISP. Sixty was a wish.
When a name does not exist, the authority says so and includes the zone's SOA in the authority section, and that SOA is what tells the resolver how long to remember the absence. RFC 2308 sets the negative cache TTL at the minimum of the SOA's MINIMUM field and the SOA record's own TTL.
$ dig @1.1.1.1 nxd-test-8814.wikipedia.org A
; <<>> DiG 9.10.6 <<>> @1.1.1.1 nxd-test-8814.wikipedia.org A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 5988
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;nxd-test-8814.wikipedia.org. IN A
;; AUTHORITY SECTION:
wikipedia.org. 3600 IN SOA ns0.wikimedia.org. hostmaster.wikimedia.org. 2026060420 43200 7200 1209600 3600
;; Query time: 227 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; MSG SIZE rcvd: 117
Reading it downward:
status: NXDOMAIN means the name does not exist. That is not the same as "no A record for a name that does exist", and neither is the same as "the servers did not answer". Three outcomes, three cache behaviours, and people say "DNS failed" for all three.
flags: qr rd ra: query response, recursion desired, recursion available. No aa, because 1.1.1.1 is not authoritative for wikipedia.org. No ad, so this particular answer was not DNSSEC validated.
ANSWER: 0, AUTHORITY: 1: the payload of a negative answer lives in the authority section. If you only ever read the answer section you will conclude you got nothing back.
The SOA's numbers, in order, are serial, refresh, retry, expire and minimum. Only the last one matters to a resolver asking about a name that is not there: 3600. The 3600 in front of IN SOA is the record's own TTL. The minimum of the two is 3600, so a conforming resolver will keep saying that name does not exist for the next hour regardless of what you publish in the meantime.
The same probe against a zone on a different provider:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43730
;; flags: qr aa; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; AUTHORITY SECTION:
github.com. 900 IN SOA ns-1707.awsdns-21.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
NOERROR with zero answers is not NXDOMAIN. Something matches that name, a wildcard in this case, just not for type A. And look at the two numbers: MINIMUM is 86400, the record's own TTL is 900, so the effective negative TTL is fifteen minutes and not a day. Whoever typed 86400 has been overruled by a field they probably were not looking at.
Your A record TTL is 60 because somebody thought about it. Your negative TTL is 900 or 3600 or a day because it arrived with the zone template. RFC 2308 has said since 1998 that one to three hours is a sensible default and that values over a day have been found to be problematic, about a field almost nobody touches.
One more distinction people collapse. SERVFAIL and timeouts are not negative answers. RFC 9520, from 2023, calls those resolution failures and requires resolvers to cache them for at least one second and no longer than five minutes. NXDOMAIN and NODATA are useful answers and get cached for far longer. Mix the two up and you will predict the wrong recovery time out loud, in a channel, in front of people.
$ dig @a.gtld-servers.net github.com A +norecurse
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 8, ADDITIONAL: 3
;; AUTHORITY SECTION:
github.com. 172800 IN NS ns-520.awsdns-01.net.
github.com. 172800 IN NS ns-421.awsdns-52.com.
github.com. 172800 IN NS ns-1707.awsdns-21.co.uk.
github.com. 172800 IN NS ns-1283.awsdns-32.org.
github.com. 172800 IN NS dns1.p08.nsone.net.
github.com. 172800 IN NS dns2.p08.nsone.net.
github.com. 172800 IN NS dns3.p08.nsone.net.
github.com. 172800 IN NS dns4.p08.nsone.net.
;; ADDITIONAL SECTION:
ns-421.awsdns-52.com. 172800 IN A 205.251.193.165
ns-421.awsdns-52.com. 172800 IN AAAA 2600:9000:5301:a500::1
Eight nameservers and one address pair. a.gtld-servers.net serves .com, so it can hand over an address for ns-421.awsdns-52.com and cannot hand over an address for anything in .net, .org or .co.uk. Those seven names have to be resolved on their own before a resolver can ask them anything, which is a second and a third round of recursion that you do not control and cannot observe. Note the flags: qr and nothing else. No aa, because this is a referral rather than an answer. No ra, because that server will not recurse on your behalf.
RFC 9471, also 2023, tightened the rules: a server must return every glue record it has for in-domain nameservers, and if they will not fit it must set TC so the client comes back over TCP. Glue behaviour changed under everybody recently and the symptom of getting it wrong is "resolution is slower for some resolvers", which nobody has an alert for. The NS TTL up there is 172800. Two days.
There is no current value of a DNS record. There are as many values as there are caches, each expiring on its own schedule, and the shape of your recovery is the shape of that distribution. Rolling back a record does not roll back the caches. That is what makes these incidents feel unfair. Everything else we run has a state you can go and read.
#1 what the authority publishes right now
dig +short @ns-520.awsdns-01.net api.example.com A
#2 what your resolver already has, without asking it to refill
dig +norecurse @10.0.0.2 api.example.com A
#3 what the machine's own name resolution does, which is NSS, not your resolver
getent hosts api.example.com
#4 take DNS out of the path entirely
curl -sv --resolve api.example.com:443:203.0.113.10 https://api.example.com/healthz
If 1 and 2 agree, nothing is stale, and you can stop talking about TTLs. If 3 disagrees with 2, your problem is /etc/hosts, nsswitch.conf, a sidecar, or a container that inherited the wrong resolv.conf, and it was never DNS in the sense anyone meant. Number 4 ends the argument: pin the address, and if the request works pinned and fails unpinned, it is resolution, and if it fails both ways it is not.
I have burned most of an evening on DNS because somebody said "it's always DNS" in the first ninety seconds and nobody ran number 4 until half past one.