1. 09:00 — Traffic Goes From a Trickle to a Flood
A marketing campaign goes live. Within minutes, traffic climbs from a steady ~100 requests per second to well over 2,000. The autoscaler does exactly what it's supposed to do: it detects the load and scales the backend fleet from 2 instances to 10.
Support tickets start anyway. Customers report intermittent 503 Service Unavailable, the occasional 504 Gateway Timeout, and checkout pages that hang far longer than they should.
The instinctive first move is to blame whatever just changed — the new instances, the autoscaler, a bad deploy. All three are innocent.
2. Rule Out the Fleet Before You Blame the Fleet
Before touching anything, confirm what's actually true instead of what seems likely:
• Every one of the ten backend instances answers its own /health endpoint correctly.
• The autoscaler's own status endpoint — which pings every instance directly, not just its internal bookkeeping — confirms all 10 are up.
That's an easy, uncomfortable conclusion to sit with: the fleet is healthy and fully scaled. It's tempting to treat that as a dead end and start digging into application code anyway. It isn't a dead end — it's the signal that tells you to stop looking at the backend entirely and start looking at whatever sits between the customer and the fleet.
3. The Metric That Actually Matters: Per-Instance Traffic Share
"Healthy" and "receiving traffic" are two different claims. A backend can pass every health check while receiving zero real requests. The signal that actually separates a backend problem from a routing problem is each instance's own request count — not what the load balancer claims to be doing, but what each backend independently reports it has received.
Pull that number per instance, and the pattern usually jumps out immediately: instead of ten instances sharing 2,000 RPS roughly evenly, you'll typically find requests piling onto a small handful of instances while the rest sit nearly idle — healthy, provisioned, paid for, and doing nothing.
📐 Takeaway: when a fleet is confirmed healthy but still can't absorb load, the load balancer's routing decisions — not the fleet's capacity — are almost always the next place to look.
4. Five Ways a Load Balancer Quietly Starves a Healthy Fleet
A reverse proxy makes several independent decisions every time it routes a request, and each one can silently go stale without ever throwing an error. In this incident (and in the wild), the usual suspects are:
1. Stale upstream membership: Whatever process is supposed to keep the load balancer's server list in sync with the fleet doesn't run on every scale-out event. The balancer keeps happily routing only to the 2 initial instances.
2. Over-aggressive passive health checks: A single transient blip ejections a healthy instance from rotation for minutes at a time.
3. Leftover session affinity: IP-hash or cookie-based stickiness concentrates traffic onto single upstream nodes.
4. Stale service-discovery caching: Long DNS TTLs prevent newly provisioned backend IPs from entering rotation.
5. Forgotten canary weights: A legacy weight bump from a past gradual rollout quietly sends 50x traffic to one node.
5. Real-World Precedents: Slack & AWS
Slack (January 4, 2021): Traffic surged from the holiday return, while network waiting times disguised high latency as low CPU usage, causing autoscalers to downscale in the middle of peak surge.
AWS us-east-1 (October 20, 2025): DNS automation race conditions caused NLB health checks to flap as network state lagged instance provisioning.