Argo CD Fixed My Drift, Then Deployed My Bad Release

This project started with a simple goal: run Kubernetes without keeping an EKS
cluster online every day.

In I Wanted Kubernetes Without an Always-On EKS Bill,
I built an always-on k3s lab on my home server and proved that I could deploy,
update, and roll back an application.

The rollback worked, but it exposed the next problem. Kubernetes restored
Version 2 while the saved YAML still declared Version 3. I corrected the file
manually, but the recovery depended on repairing the running cluster and its
saved instructions separately.

In The Rollback Worked. My Next Deploy Could Break It Again,
I designed a safer path. The automated build process would test and publish an
exact image, then stop at a Git pull request. Git would record the reviewed
version. Argo CD, running inside Kubernetes, would make the cluster follow that
record.

Now I needed to prove that the design worked outside a diagram.

I followed one release from source code to running Pods. Then I tested two
opposite failures:

• The cluster was wrong while Git was correct.
• Git contained a bad setting while the cluster followed it correctly.

Those experiments showed both the value and the limit of GitOps. Automation can
make the cluster match Git, but it cannot decide whether the human-approved
version in Git is a good one.

CI Built the Release but Did Not Deploy It

The GitHub Actions workflow—my continuous integration, or CI, worker—ran the
application tests and checked the Kubernetes package before building anything.
Its job was to prove and publish a release, not to change the cluster.

After validation, Buildx created a Linux AMD64 image with the full source commit
baked into /version:

docker buildx build \
  --platform linux/amd64 \
  --build-arg "APP_VERSION=$GITHUB_SHA" \
  --tag "$image_name:$GITHUB_SHA" \
  --provenance=mode=max \
  --sbom=true \
  --push \
  application

After publishing the image, CI read its registry digest. A digest is the image's
content fingerprint: if the image changes, the digest changes. CI updated one
field in gitops/environments/homelab/values.yaml. For the final measured
release, the automated pull request contained this one-line change:

-  digest: sha256:170bcc91b9c4bfcbdc194c004cb38d6c89094e9b49a1a128ed3a0507976fef97
+  digest: sha256:f8303e07e737f40fa5479b73a148261cf4221f4d4b52ebf095b74d2cb4ef5be5

The workflow opened the pull request and stopped. A human reviewed and merged
the digest. That merge—not a deployment command in CI—was the release
instruction.

Argo CD Made the Cluster Follow Git

Argo CD was the worker inside Kubernetes. It repeatedly compared the version
recorded in Git with the version running in the cluster and worked to make them
agree. This comparison-and-correction loop is called reconciliation.

I installed a pinned, resource-limited Argo CD Core deployment with no permanent
API server or public UI. Its Services remained internal ClusterIP Services.

The Application reads both the Helm chart and homelab values from the public
repository. Automated sync, pruning, and self-healing are enabled:

syncPolicy:
  automated:
    enabled: true
    prune: true
    selfHeal: true

At the healthy baseline, Argo CD reported Synced, Healthy, and operation
Succeeded. Two application Pods were ready with zero restarts.

I retained the complete artifact trace:

application commit
    -> registry digest
    -> Git desired state
    -> Deployment image
    -> both Pod runtime image IDs
    -> /version response

The Git declaration, Deployment, and both Pod image IDs agreed on the same
registry digest. /version returned the source commit baked into that image.

The four Argo CD components used a combined 19 millicores of CPU and 85 MiB of
memory in the retained steady-state sample. That measurement describes this
home lab; it is not production sizing guidance.

First, I Changed the Cluster Behind Git’s Back

Git declared two replicas. I deliberately changed the live Deployment to one:

kubectl -n delivery-api scale deployment/delivery-api --replicas=1

The first poll already showed one replica and Argo CD OutOfSync. Git remained
unchanged with replicaCount: 2.

Forty-two seconds after the mutation, the Deployment was back to two replicas
and the Application was Synced again.

This proved self-healing for that specific field. Git remained correct, so Argo
CD could restore the live cluster from the durable declaration.

The next failure reversed that relationship.

Then I Put a Bad Setting in Git

I merged a reviewed configuration change from:

readinessMode: ready

to:

readinessMode: fail

The change was valid YAML. It passed the Helm schema and lint checks. At runtime,
the new Pod stayed alive but /ready returned HTTP 503.

Forty-nine seconds after the merge, the evidence showed:

Argo CD sync:    Synced
Argo CD health:  Progressing
New Pod:         Running, not Ready
Old Pods:        Running, Ready

Synced meant the cluster matched Git. Progressing meant Kubernetes could not
complete the rollout.

GitOps had not malfunctioned. It had correctly delivered a harmful declaration.

The Deployment used maxUnavailable: 0, so the two old ready replicas remained
while the replacement failed readiness. One sampled request through the private
Service returned {"status":"ready"}. That proves availability at that moment,
not zero downtime across the experiment.

Recovery Had to Correct Git

Patching only the live Deployment would leave Git declaring the bad readiness
mode. Argo CD could then reapply it.

I reverted the bad merge in Git:

git revert --mainline 1 69ac4fe6de392aed4affc573ea20a2bb3f4ee59f
git push

Argo CD pulled the revert and restored readinessMode: ready. Forty-two seconds
after the recovery commit, the Application was Synced and Healthy with two
updated, ready replicas.

Argo CD did not choose the known-good version. I chose it by reverting Git; Argo
CD reconciled that decision.

I Timed a Fresh Release End to End

The first image promotion happened before Argo CD was installed, so I did not
use it for a continuous promotion-to-healthy claim. I ran another application
release with Argo CD already active and started the monitor before merging the
digest pull request.

The monitor sampled every two seconds, so the healthy timestamp has up to two
seconds of sampling uncertainty.

During the rolling update, the Deployment already declared the new digest while
a request still reached an old ready Pod and returned the previous commit. A few
seconds later, both updated replicas were ready and /version returned:

{"service":"delivery-api","version":"84db89f7f968c5860b07a7efd6a42e21895d9e5c"}

That was expected. A rolling update changes replicas gradually; the Deployment
declaration and every serving Pod do not switch at the same instant.

What the Implementation Proved

Application CI published the artifact and proposed the environment change
without receiving cluster authority. Git retained the reviewed digest. Argo CD
pulled and reconciled it. Kubernetes reported rollout health.

The failure tests also exposed the limits:

• Self-healing corrected live drift only because Git remained correct.
• Argo CD faithfully applied the bad readiness declaration from Git.
• Recovery required a human decision recorded as a Git revert.
• One successful request did not prove zero request loss.
• A single-node k3s cluster could not demonstrate high availability.

GitOps did not remove operational judgment. It moved release and recovery
decisions into a place where they were reviewable, durable, and reconcilable.

Conclusion

The first article created an affordable, always-on Kubernetes lab. The second
used its rollback problem to design one reviewable release path. This
implementation proved how that path behaved when either the cluster or Git was
wrong.

The common lesson across the three articles is that a working cluster is only
one part of a delivery platform. The approved release, permission to deploy,
health signals, and recovery decision also need clear owners.

The code, chart, GitOps configuration, runbooks, and sanitized evidence are in
the public
pull-based-kubernetes-delivery repository.