A Rolling Update, an Autoscaler, and a Missing Environment Variable: Anatomy of a Production Outage

13 min read

Rolling updates are supposed to be the safe way to ship. Kubernetes brings up new pods, waits for them to become healthy, and only then removes the old ones. If the new version is broken, the rollout simply gets stuck and the old version keeps serving. That is the promise.

We shipped a broken release to production and the promise held for four days. Then every healthy pod disappeared anyway, and we had a full outage. Nothing in the cluster misbehaved. Every controller followed its own rules exactly. The outage lived in the gaps between them.

This post explains what happened, what we could prove, and what we could only partially reconstruct. It also covers the specific Kubernetes mechanics that most teams never see until they get burned by them, including one that we ourselves got wrong on the first pass and only understood after reading the controller source.

The impact

Zero healthy pods for about 27 hours, from the moment the last working pod was evicted until we rolled back. The service is a Kafka consumer, so the backlog grew the whole time the pods were down, and for the four days before that the broken pods were processing nothing new.

The setup

The service runs as a Kubernetes Deployment with three replicas. Scaling is handled by KEDA, which adjusts the replica count between 3 and 32. KEDA does this by creating a HorizontalPodAutoscaler behind the scenes [1]. Our ScaledObject has two kinds of triggers: consumer lag on the Kafka topics, and CPU at 75 percent. The HPA acts on the highest suggestion across all triggers.

Four details of the setup matter for this story:

The Deployment had no explicit update strategy, so it used the defaults.
There was no PodDisruptionBudget.
KEDA stayed active during deployments. Nothing paused it while a rollout was in progress.
The pods had a hard zone spreading rule: a topology spread constraint with whenUnsatisfiable: DoNotSchedule, meaning a pod that would break the zone balance is not scheduled at all.

The bad release

The application is a Spring Boot service. A few weeks before the incident, team added a feature flag to the configuration:

feature:
  enabled: ${FEATURE_X_ENABLED}

Note what is missing: a default value. In Spring, ${FEATURE_X_ENABLED:false} means “use false if the variable is not set”. Without the :false part, the variable becomes mandatory. If it is not set, the application throws an error at startup and exits.

In fact the flag had originally been written with a safe default. A later commit removed it, describing the default as unnecessary. The environment variable was then added to the staging and performance environments, but never to production.

The next production release contained that change. Every new pod started, failed to resolve the placeholder, and exited with code 1. The pods went into a crash loop and never became ready.

So far this is an ordinary bad deploy. Here is where it gets interesting.

Why the rolling update should have saved us

The default rolling update strategy allows 25 percent of pods to be unavailable during an update, and 25 percent extra pods as surge. The unavailable number is calculated by rounding down, the surge number by rounding up [2]. With three replicas:

maxUnavailable = floor(3 × 0.25) = 0
maxSurge       = ceil(3 × 0.25)  = 1

Zero unavailable. The rollout controller is not allowed to remove a single available pod until a new pod becomes ready. Our new pods never became ready. So the rollout did exactly what the documentation promises [3]: it stalled. One broken pod sat there crash looping, and the three healthy pods on the old version kept serving traffic.

We have direct proof of this. One of the old pods survived for 19 days with zero restarts, including four full days after the broken release went out. The rolling update mechanism worked perfectly.

And yet, by day five, the healthy pod count was zero. Something else was removing them.

Two counters that told the story

When we pulled the Deployment object from the cluster, two fields stood out:

metadata:
  annotations:
    deployment.kubernetes.io/revision: "40"
  generation: 1413

The revision annotation counts template changes, meaning actual new versions or restarts. Forty of those over the lifetime of the Deployment. But generation increments on every change to the spec, and that includes every change to the replica count. Roughly 1370 of the 1413 spec changes were pure replica changes. That is the autoscaler, rewriting the replica count over and over, hundreds of times during the incident window.

Why was it scaling at all? Because of a feedback loop. Crash looping pods consume nothing, so the Kafka backlog can only grow, so the lag trigger keeps asking for more pods. And unlike CPU metrics, where the HPA dampens scaling when pods are not ready, the lag metric is external. It is computed from the Kafka backlog and knows nothing about pod readiness. Dead consumers make the lag metric scream louder, not quieter.

Our monitoring showed the replica changes arriving in bursts roughly every six minutes. That is consistent with the HPA default scale down stabilization window of 300 seconds [4], and also with the KEDA cooldown period of 300 seconds in our configuration, so we treat the rhythm as corroborating heavy churn rather than as proof of which timer drove it.

The autoscaler was extremely busy. But the autoscaler only writes one integer, the desired replica count, through the scale subresource of the Deployment [5]. It never touches a pod. To understand what that integer did, we have to look at what the Deployment controller does with it.

Two ReplicaSets, two control loops

During a rolling update a Deployment briefly manages two ReplicaSets: the old one with the working pods, and the new one with the new template. Normally this state lasts a minute or two. For us it lasted days, because the rollout was stuck.

The Deployment controller runs two very different pieces of logic against those two ReplicaSets.

The rollout logic runs because the pod template changed. This is the careful one. It grows the new ReplicaSet within the surge budget and shrinks the old one only when the availability budget allows it. This is where the maxUnavailable guarantee lives.

The scale logic runs whenever the desired replica count changes, which for us was every few minutes. Its job is different: the total changed, so split the new total across the ReplicaSets. Kubernetes does this proportionally. Each ReplicaSet gets a share of the change proportional to its size, and bigger ReplicaSets get bigger shares [6]. The share is computed against the allowed total from the previous scaling point, rounding leftovers are handed to the largest ReplicaSet, and none of it checks pod health [7]. The maxUnavailable guarantee belongs to the rollout logic only. Proportional scaling is not a bug: if you are halfway through a canary and scale from 10 pods to 100, you want to keep the old to new ratio, not dump 90 pods onto the unproven version. It is a sensible design that gets uncomfortable when one of the two ReplicaSets is permanently broken.

One more thing the intro example hides: the budgets are percentages of the current desired count. At 3 replicas, maxUnavailable is 0 and you are fully protected. When the autoscaler pushes desired to 32, maxUnavailable becomes 8. The guarantee you tested at three replicas is not the guarantee you get at thirty.

What the controller actually does, and what we first got wrong

Our first internal writeup claimed that the rollout logic freely deletes not yet ready pods from the old ReplicaSet, so every scale up would be immediately confiscated and the old ReplicaSet could never grow. That claim is wrong, and the controller source says so.

What really happens on a scale up from 3 to 32 (allowed total 40):

The proportional split favors the old ReplicaSet, exactly as you would expect. Old goes from 3 toward roughly 30 on paper, new from 1 toward roughly 10. And the old template is the working image, so those pods would come up healthy.
The rollout logic then runs, and it may clean up unavailable old pods, but only within a budget:
```
maxScaledDown = allPodsCount − minAvailable − newRSUnavailablePodCount
```
With desired 32: 40 − 24 − 10 = 6. Six pods, not everything. Then the formula hits zero and cleanup stops. The controller source even documents our exact situation in a comment: when the new ReplicaSet pods crash loop and never become available, the arithmetic comes out to zero “so the oldRS won’t be scaled down” [7]. A permanently broken new ReplicaSet mostly freezes the old one rather than eating it.
So in the idealized model, the surviving fresh old pods pass their readiness probes, become available, and are then protected by the availability budget. The system should settle at roughly 24 working pods and 16 broken ones.

Read that again, because it is the uncomfortable part: by the controller’s own math, the cluster should have been mostly healthy at high replica counts. It was not. Our monitoring shows the total pod count peaking at 32, which is our configured maximum, while the service stayed unhealthy throughout.

Why we think it still fell over

This part is a reconstruction, not a proof. We know the start state, the end state, and the churn in between. We did not capture per ReplicaSet history during the incident, so the exact path of each deleted pod is unrecorded. With that said, three forces fit the evidence, and none of them require any controller to misbehave.

The idealized model assumes granted pods actually start. Ours often could not. The hard zone spreading rule (DoNotSchedule) was active while the cluster autoscaler was removing underused nodes. A pod that cannot be placed without breaking zone balance stays Pending. Pending pods are unavailable, and unavailable old pods never earn the protection of the availability budget. They stay inside the cleanup budget forever, a few deletions allowed on every reconcile, on a loop that ran every few minutes for days.

Scale downs cut both ReplicaSets without health checks. Every dip in the lag metric shrank the allowed total, and the proportional split assigned part of each cut to the old ReplicaSet. The deeper the dip, the harsher the clamp, and the rounding leftovers always favor the largest ReplicaSet, which by then was the broken one.

Nodes were being reclaimed underneath the healthy pods. This one is proven, not reconstructed. The last healthy pod’s termination record shows:

status:
  conditions:
    - type: DisruptionTarget
      status: "True"
      reason: EvictionByEvictionAPI
  containerStatuses:
    - state:
        terminated:
          exitCode: 143   # SIGTERM, a graceful shutdown request

EvictionByEvictionAPI [8] means something asked the eviction API to remove this pod. Rollouts and ReplicaSet scale downs never use that API; they delete pods directly. The eviction API is used by node drains and by the cluster autoscaler when it removes a node. Two minutes after this eviction, our monitoring recorded the pod’s node being terminated by autoscaling. Routine cost optimization removed the node under the last healthy pod, and its only possible replacement came from the broken template.

This is exactly the disruption a PodDisruptionBudget exists to prevent. A PDB with minAvailable set would have made that eviction request fail and the node drain wait. We had none. And note the boundary: PDBs only constrain the eviction API. Controllers deleting pods directly bypass them entirely; the documentation says “deleting deployments or pods bypasses Pod Disruption Budgets” [9]. A PDB would have saved the last pod from the node drain. It would have done nothing against the proportional scale downs.

For completeness: a similar symptom was reported against KEDA years ago, a Deployment with maxUnavailable zero losing nearly all pods during an update while a ScaledObject was attached [10]. That issue was closed as stale without a confirmed root cause, so we cite it as a symptom report from the field, not as a confirmed bug.

Recovery for us was mundane: roll back to the previous image. The service was healthy again within 90 seconds, which is itself evidence of where the fault was.

What we are changing

None of these had shipped when the incident happened, and at the time of writing they are in progress rather than done. Each one maps to a specific hole, and no single fix covers all of them.

Feature flags get defaults. ${FEATURE_X_ENABLED:false}. A missing flag should mean the feature is off, never that production crashes.
CI validates required configuration. A pipeline check that verifies every placeholder without a default is supplied by every environment. This catches the whole trigger class before deploy.
A PodDisruptionBudget. Closes the node drain and cluster autoscaler vector, the one we proved. Does not protect against proportional scaling.
Pause KEDA during rollouts. KEDA supports this directly with the autoscaling.keda.sh/paused-replicas annotation [11]. The deploy pipeline sets it, rolls out, waits for health, removes it. This is the only direct fix for mid rollout scale events.
An explicit update strategy. Spelling out maxUnavailable: 0 and maxSurge: 1 plus a sensible minReadySeconds, so the protection does not silently change shape as the replica count moves.
Softening the zone spreading rule to ScheduleAnyway. Zone balance is a preference, not something worth refusing to run at all for.
Alerts on stuck rollouts. Kubernetes marks a stuck Deployment with the condition reason ProgressDeadlineExceeded but takes no action on its own [12]. Everything described above needed days of exposure. An alert that pages within minutes shrinks the window to almost nothing.

There is also a structural option worth naming: progressive delivery controllers such as Argo Rollouts manage the stable and canary ReplicaSets explicitly and are autoscaler aware, which removes the two control loop conflict by design rather than by discipline.

Takeaways

The one sentence version: maxUnavailable protects you from the rollout, but nothing protects you from being scaled during a rollout. If an autoscaler and a broken rollout run at the same time for long enough, the availability guarantee you think you have is much weaker than it looks, and it changes shape with the replica count.

The slightly longer version is an AND gate. Several things had to be true at once: a release that could never become ready, a stuck rollout that nobody rolled back for days, an autoscaler oscillating the replica count, a hard scheduling rule in a shrinking cluster, no disruption budget, and no alert to break the loop. Remove any one of them and the story ends quietly. That is also why this class of outage is rare, confusing, and worth writing down.

And one meta lesson: our first explanation of the mechanism was confidently wrong until we checked the controller source. If your postmortem contains a paragraph about what a controller “must have done”, go read the code before you publish it.

References

KEDA, Scaling Deployments, KEDA creates and manages an HPA behind the scenes: https://keda.sh/docs/2.17/concepts/scaling-deployments/
Kubernetes, Deployments, Max Unavailable and Max Surge (rounding rules): https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#max-unavailable
Kubernetes, Deployments, Rolling Update Deployment (availability guarantee during updates): https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment
Kubernetes, Horizontal Pod Autoscaling, default behavior (“For scaling down the stabilization window is 300 seconds”): https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/#default-behavior
Kubernetes, Horizontal Pod Autoscaling, how it works (the scale subresource): https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/#how-does-a-horizontalpodautoscaler-work
Kubernetes, Deployments, Proportional scaling: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#proportional-scaling
Kubernetes source, deployment controller: rollout budget and cleanup in reconcileOldReplicaSets and cleanupUnhealthyReplicas (rolling.go), proportional split in scale (sync.go) and getReplicaSetFraction (util/deployment_util.go): https://github.com/kubernetes/kubernetes/blob/v1.33.0/pkg/controller/deployment/rolling.go and https://github.com/kubernetes/kubernetes/blob/v1.33.0/pkg/controller/deployment/sync.go and https://github.com/kubernetes/kubernetes/blob/v1.33.0/pkg/controller/deployment/util/deployment_util.go (formulas quoted above verified against master at the time of writing)
Kubernetes, Disruptions, Pod disruption conditions (DisruptionTarget, EvictionByEvictionAPI): https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-conditions
Kubernetes, Disruptions, Pod disruption budgets (“deleting deployments or pods bypasses Pod Disruption Budgets”): https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets
KEDA issue 2720, “RollingUpdate strategy is not respected with regards to maxUnavailable” (closed as stale, no confirmed root cause): https://github.com/kedacore/keda/issues/2720
KEDA, Scaling Deployments, Pausing autoscaling: https://keda.sh/docs/2.17/concepts/scaling-deployments/#pausing-autoscaling
Kubernetes, Deployments, Progress deadline seconds (ProgressDeadlineExceeded): https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#progress-deadline-seconds