The Promise of GitOps: Automating Kubernetes Deployments
The journey began with a practical need: run Kubernetes without incurring the cost of an always-on EKS cluster. The author, Lalit Bagga, detailed in a prior post how a k3s lab on a home server could handle application deployment, updates, and crucially, rollbacks. This initial success, however, uncovered a subtle but critical flaw in the recovery process.
After a rollback to a previous application version, the Kubernetes cluster was restored to Version 2. The problem emerged when the saved YAML configuration, still pointing to Version 3, conflicted with the running state. Manually correcting the YAML was a workaround, but it revealed a disconnect: the cluster's state and its desired state, as defined in Git, were being managed in isolation during recovery. This separation is precisely what GitOps aims to eliminate.
The next step was to design a more robust GitOps workflow. The goal was to ensure that the automated build process would test and publish an exact application image. Crucially, the deployment would then pause, awaiting explicit approval before proceeding. This staged approach, where the build pipeline publishes an artifact and then halts for a Git commit, is a common pattern to prevent accidental deployments of unverified code. The expectation is that the Git repository accurately reflects the desired state of the Kubernetes cluster at all times.
When Drift Happens: The Unforeseen Consequence
The core principle of GitOps is that the Git repository is the single source of truth. Any deviation from this truth is termed "drift." Argo CD, a popular GitOps continuous delivery tool, excels at detecting and, in some configurations, automatically correcting this drift. It monitors the live Kubernetes cluster and compares its state against the desired state defined in Git. When a discrepancy is found, Argo CD can reconcile the cluster to match the Git repository.
In this specific scenario, the author had manually corrected a configuration drift in the running cluster. However, the saved YAML definition in Git still pointed to an older version of the application. When Argo CD was subsequently used to synchronize the cluster with the Git repository, it dutifully updated the cluster to match the YAML file. The problem was that the YAML file itself was outdated, still referencing an older application version (Version 2), while the author had intended to deploy a newer, tested version (Version 3).
Argo CD, functioning as designed, detected that the cluster's deployed state did not match the Git repository's declared state. The drift, in this case, was that the cluster was running Version 3 (after the manual correction) but the Git YAML declared Version 2. Argo CD's reconciliation process, aimed at enforcing the Git state, then rolled the application back to Version 2. This was not the intended outcome. The tool, designed to prevent drift and enforce Git as the source of truth, had inadvertently deployed an older version because the Git repository itself contained the outdated definition.
The Disconnect: Git as Source of Truth vs. Actual Truth
This experience highlights a critical nuance in GitOps: the system is only as good as the information committed to Git. If manual interventions or errors lead to a mismatch between the Git repository and the actual deployed artifact, GitOps tools can, paradoxically, enforce the incorrect state. Argo CD's strength lies in its ability to maintain configuration consistency. When that configuration is flawed, the tool faithfully applies the flaw.
The author's initial goal was to have a declarative system that would allow for reliable rollbacks. The drift issue arose because the cluster state and the Git state diverged. Argo CD's subsequent action, while technically correct in terms of synchronizing with Git, led to an unintended rollback. This underscores a common challenge: ensuring that the Git repository always accurately reflects the *latest tested and approved* artifact, not just a configuration file that might be out of sync with the actual deployed code.
Consider this less like a self-healing system and more like a meticulous librarian who will always retrieve the exact book you ask for, even if you accidentally ask for an old, outdated edition from the wrong shelf. The librarian (Argo CD) is performing its job perfectly by fetching what's requested, but the request itself was flawed. The core issue wasn't Argo CD's functionality but the state of the source of truth – the Git repository.
Refining the GitOps Workflow: Beyond Simple Synchronization
The solution involves strengthening the feedback loop between the build, the Git repository, and the deployment tool. Simply committing YAML to Git and letting Argo CD synchronize isn't enough if the YAML doesn't accurately represent the desired, *latest* artifact. The process needs to ensure that when a new artifact is built and tested, the Git repository is updated to point to that specific artifact version *before* Argo CD is triggered to deploy it.
This implies a more sophisticated CI/CD pipeline. The pipeline should:
- Build and test a specific application version.
- Publish the versioned artifact (e.g., a Docker image with a unique tag).
- Update the Kubernetes manifests (YAML files) in the Git repository to reference this new, specific artifact version. This step is crucial and often requires automation to prevent manual errors.
- Commit these updated manifests to Git, creating a new desired state.
- Argo CD, detecting the change in the Git repository, then synchronizes the cluster to this new, correct state.
This layered approach ensures that Git remains the single source of truth, but that truth is consistently and accurately updated with validated artifacts. The manual correction of drift, while a necessary immediate fix, should be a signal to improve the automated processes that prevent such divergence in the first place.
The Unanswered Question: Managing Manual Interventions
What remains unaddressed by many GitOps implementations is the robust handling of necessary manual interventions. While the goal is full automation, real-world scenarios often require manual adjustments or emergency fixes. The challenge lies in ensuring these manual changes are either immediately reflected in Git or that the GitOps tool is aware of and can reconcile these temporary deviations without enforcing an incorrect past state. Without a clear strategy for managing these exceptions, even the most sophisticated GitOps tools can become agents of unintended consequences.