Postmortem: The Flash Sale That Broke Our Load Balancer, Not Our Backend

1. 09:00 — Traffic Goes From a Trickle to a Flood

A marketing campaign goes live. Within minutes, traffic climbs from a steady ~100 requests per second to well over 2,000. The autoscaler does exactly what it's supposed to do: it detects the load and scales the backend fleet from 2 instances to 10.

Support tickets start anyway. Customers report intermittent 503 Service Unavailable, the occasional 504 Gateway Timeout, and checkout pages that hang far longer than they should.

The instinctive first move is to blame whatever just changed — the new instances, the autoscaler, a bad deploy. All three are innocent.

2. Rule Out the Fleet Before You Blame the Fleet

Before touching anything, confirm what's actually true instead of what seems likely:

• Every one of the ten backend instances answers its own /health endpoint correctly.
• The autoscaler's own status endpoint — which pings every instance directly, not just its internal bookkeeping — confirms all 10 are up.

That's an easy, uncomfortable conclusion to sit with: the fleet is healthy and fully scaled. It's tempting to treat that as a dead end and start digging into application code anyway. It isn't a dead end — it's the signal that tells you to stop looking at the backend entirely and start looking at whatever sits between the customer and the fleet.

3. The Metric That Actually Matters: Per-Instance Traffic Share

"Healthy" and "receiving traffic" are two different claims. A backend can pass every health check while receiving zero real requests. The signal that actually separates a backend problem from a routing problem is each instance's own request count — not what the load balancer claims to be doing, but what each backend independently reports it has received.

Pull that number per instance, and the pattern usually jumps out immediately: instead of ten instances sharing 2,000 RPS roughly evenly, you'll typically find requests piling onto a small handful of instances while the rest sit nearly idle — healthy, provisioned, paid for, and doing nothing.

4. Five Ways a Load Balancer Quietly Starves a Healthy Fleet

A reverse proxy makes several independent decisions every time it routes a request, and each one can silently go stale without ever throwing an error. In this incident (and in the wild), the usual suspects are:

• Stale upstream membership. Whatever process is supposed to keep the load balancer's server list in sync with the fleet doesn't always run on every scale-out event. The balancer keeps happily routing to the two instances it already knew about.
• Over-aggressive passive health checks. A single transient blip — completely normal at 2,000 RPS — can be enough to eject a perfectly good instance from rotation for minutes at a time, if the failure threshold and cooldown window were tuned for a much quieter service.
• Leftover session affinity. IP-hash or cookie-based stickiness made sense before session state moved into a shared store. Left on afterward, it actively concentrates traffic onto however many distinct client sources happen to be hitting the service — sometimes just one or two.
• Stale service-discovery caching. If the balancer resolves upstream membership via DNS, an hour-long cache TTL means even a correct fix to the DNS record won't take effect for a long time.
• A forgotten canary weight. A weight bump from an earlier gradual rollout, never reset once that rollout finished, quietly keeps sending one instance 50x the traffic of everyone else.

None of these show up as an error in the load balancer's own logs — from its point of view, it's making the routing decision it was configured to make. That's exactly what makes this class of bug slow to catch and easy to misdiagnose as an application problem.

5. This Pattern Shows Up in Production Constantly

Two of the fault categories above aren't hypothetical — variants of them have taken down very large, very real systems:

• Slack, January 4, 2021. The first Monday back from the holidays brought a traffic spike as everyone's stale local caches forced extra load. Underlying network degradation meant requests spent more time waiting rather than executing — which showed up as lower CPU utilization, not higher. Slack's autoscaler, watching CPU, read that as "we're over-provisioned" and scaled the web tier down in the middle of the surge. Slack's own postmortem specifically called out re-evaluating health-checking and autoscaling configuration afterward. (Source: Slack Engineering — Slack's Outage on January 4th, 2021)
• AWS us-east-1, October 20, 2025. A race condition in DynamoDB's own DNS automation wiped the DNS records backing the service endpoint, and the cascading failure was widely reported to include Network Load Balancer health checks flapping — new, otherwise-healthy capacity being pulled in and out of rotation as network state propagated slower than instances were coming online. (Source: ThousandEyes — AWS Outage Analysis: October 20, 2025)

The specific mistakes behind Incident #27 — stale upstream lists, over-tuned health checks, stale DNS caching, leftover canary weights — are the same small handful of ways a load balancer's view of the world can drift from reality, at any traffic volume, on any cloud. For the mechanics of weighted routing and why canary weights exist in the first place, Google's own SRE materials are the canonical reference: Load Balancing in the Datacenter and Canarying Releases.

6. Why This Is a Good Incident to Practice, Not Just Read About

Reading the list above is useful. Sitting in front of a live gateway config with a real ten-instance fleet behind it, a real traffic generator hammering the endpoint, and no idea in advance which of the five is actually live in your session — that's a different kind of learning. You have to form a hypothesis from the metrics, read the config to confirm it, make a production-safe edit, and watch traffic redistribute across the whole fleet in real time.

That's the exact incident behind this post — Incident #27: Flash Sale Traffic Surge on VibeInfra. Ten real Go backend instances, a real Nginx gateway in front of them, and a fault drawn at random each time you start it.