What I Learned Studying EKS Cluster Upgrades (Beyond Just "Click Upgrade")

I'm fairly new to SRE/DevOps, and one of the topics I recently spent time studying properly was EKS cluster upgrades. My first instinct, like most people starting out, was: "it's just a version bump, click upgrade in the console, done." That's basically what most beginner blog posts say too.

But the more I read and the more I dug into real-world postmortems and discussions, the more I realized — the actual Kubernetes control plane upgrade is the easy part. Almost everything that can go wrong seems to happen around it, not because of it. Sharing what I learned here, mainly for my own notes, but hoping it's useful for anyone else early in their journey too.

Learning #1: There's No "Undo" Button

This was the first thing that surprised me. I assumed upgrades work like most software — if something breaks, you roll back. But with EKS, you cannot downgrade the control plane version once you upgrade it.

So the plan can't be "upgrade, and if it breaks, revert." It has to be "test enough beforehand that breaking isn't really an option," and if something does go wrong, the fix is always moving forward, not backward. That single fact changes how you're supposed to approach the whole thing — testing has to happen before the button is clicked, not after.

Learning #2: APIs Get Deprecated, and It's Usually Not Your Own Code That Breaks

Kubernetes removes old API versions on a schedule. I already knew this conceptually, but what I didn't realize is that the risk usually isn't your own YAML files — it's the Helm charts and third-party tools you installed a while back and forgot about, which might still be using an older API version internally.

There are tools built exactly for catching this before it becomes a problem:

pluto detect-helm -owide
pluto detect-files -d ./manifests

kubent (kube-no-trouble) does something similar. I hadn't heard of either tool before researching this, and it made me realize how much of "being good at Kubernetes" is really just knowing which small tools exist for which job.

Learning #3: Addons Have Their Own Version Compatibility Rules

This one genuinely surprised me. Things like CoreDNS, kube-proxy, the VPC CNI plugin, and the EBS/EFS CSI drivers each have their own version, and that version needs to be compatible with whatever Kubernetes version you're running — it's not automatic.

You can actually check this directly:

aws eks describe-addon-versions \
  --addon-name vpc-cni \
  --kubernetes-version 1.30

From what I read, a common mistake (and one that apparently causes confusing issues like pods failing to get IPs, or DNS acting weird) is upgrading the control plane and assuming the addons will "just work" because they were fine before. Apparently they need to be checked and upgraded as part of the same process, not treated as a separate afterthought.

Learning #4: Webhooks and Operators Can Break Things in Confusing Ways

This was the most interesting (and honestly a little intimidating) thing I came across. Tools like cert-manager, service meshes, or policy engines often register something called a ValidatingWebhookConfiguration or MutatingWebhookConfiguration. These sit directly in the path of every API request.

If one of these tools isn't fully compatible with the new Kubernetes version, the symptoms apparently don't look obviously related — you might just see random kubectl apply commands failing across the cluster, and it's not immediately obvious why.

The tip I picked up from reading around: before upgrading, it's worth just listing these out and knowing what owns each one:

kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

Then checking each tool's own documentation for explicit support of the version you're upgrading to.

Learning #5: Node Upgrades Have Their Own Gotchas Too

I learned about PodDisruptionBudgets (PDBs) in this context, which was new to me. Apparently, if a PDB is configured too strictly (like requiring 100% of pods to stay available on something that only has one replica), it can cause node drains to hang or fail during an upgrade, because Kubernetes literally can't evict anything without violating the rule.

A pattern that seems to be recommended over "in-place" node upgrades is creating a new node group on the new version alongside the old one, gradually moving workloads over, and then removing the old node group — instead of upgrading nodes in place and hoping the timing works out.

The Bigger Lesson

The thing that stuck with me most after all this reading wasn't really a technical detail — it was more of a mindset shift. A lot of the "scary" upgrade stories I came across seem to happen when teams upgrade rarely (like once a year, only when AWS forces them to), which means a huge number of these small issues pile up at once.

It seems like the safer approach — at least from what I've read — is treating upgrades as a routine, smaller, more frequent thing rather than a big scary annual event. Smaller version jumps apparently mean fewer surprises each time.

My Own Study Checklist (for next time I practice this hands-on)

• [ ] Run pluto or kubent against the target version before upgrading
• [ ] Check addon compatibility (VPC CNI, CoreDNS, kube-proxy, CSI drivers) for the target version
• [ ] List all webhook configurations and check if their owning tools support the new version
• [ ] Check PodDisruptionBudgets for anything that could block node draining
• [ ] Consider blue/green node groups instead of in-place upgrades
• [ ] Remember: no control plane rollback — plan accordingly

I'm planning to actually simulate an upgrade hands-on soon using a local KIND cluster, since reading about it only gets you so far. If anyone experienced has caught mistakes in how I've understood any of this, I'd genuinely love the correction — still very much learning here.