Push or Pull: Two Ways to Hand Off a Deployment¶
I built the same handoff twice, in opposite directions. In three monolith pipelines, CI ends by committing an image tag into the deployment repository. For ten microservices, CI stops at the registry and a controller notices. Both are GitOps. Only one of them survived ten services.
Both models are public
The push model is the last stage of the monolith pipelines. The pull model is a workflow that never learns a cluster exists.
The Push Model, and Why It Is Appealing¶
CI builds an image, pushes it, then clones the deployment repository, writes the new tag into a file, and commits. ArgoCD sees the commit and syncs.
Everything about that is easy to reason about. The trigger is a commit, so git log on the deployment repository is a deployment history with authorship and timestamps. A rollback is a revert. There is exactly one thing that decides what runs, and it is a file you can read.
For a single application it is close to ideal, and I would still choose it there.
The Push Model Failing In My Own Repository¶
Before arguing about scale, the honest problem with push is smaller and more embarrassing: the write and the read are in two different files, and nothing checks that they agree.
In the Java monolith, CI writes an immutable tag into image.env on every build:
Meanwhile all three Kustomize files hardcode:
Nothing reads image.env. The tag is produced correctly, committed correctly, and consumed by nobody, so what actually deploys is whatever latest currently points at. I wrote this up as an open gap rather than quietly fixing it, because the failure mode is instructive: push does not fail loudly when it breaks. It keeps committing.
That is worth holding onto. The push model's guarantee is only as good as the link between the producer and the consumer, and that link is a convention rather than a mechanism.
What Broke at Ten Services¶
Scale exposed two problems the single-application case hides.
Volume. Every code push became a commit in the deployment repository. Ten services means ten times the traffic, in a repository whose history should read as intent and instead reads as a build log. Finding the commit where someone deliberately changed a replica count means scrolling past a hundred automated tag bumps.
Two writers. Once ArgoCD Image Updater is in the picture, there are two things that both believe they decide which image runs: the commit CI wrote, and the controller watching the registry. They will disagree eventually, and the first time you notice is during an incident when the running image matches neither.
Three Iterations to Get There¶
I did not arrive at pull by design. I arrived by two failures.
Iteration 1: tag images with the chart version¶
The idea was to give every image a fourth tag matching the Helm chart version, set tag: "" in the values so the chart template falls back to .Chart.AppVersion, and have a workflow write the chart version into the deployment repository after each build.
It failed on the first code-only push. I changed src/frontend/, CI built and pushed frontend:0.10.5, then the GitOps workflow tried to write chart_version: 0.10.5 into the deployment repository. It was already 0.10.5, because the chart had not changed:
No commit, no sync, and a new image sitting in the registry that nothing would ever deploy.
A no-op commit is a silent deployment failure
Every individual step succeeded. The build was green, the push was green, the GitOps job was green. The only signal that anything was wrong is a deployment that did not happen, which nothing was watching for. This design worked for release events (a chart version bump) and could never work for continuous delivery, and the distinction is invisible until the first code-only push.
Iteration 2: write a unique SHA tag every time¶
The obvious fix: make the written value unique per commit, so the diff is never empty. Every code push wrote sha-<commit> into the deployment repository, ArgoCD saw a real diff, and it worked.
It also produced exactly the noise described above, and it put CI on a collision course with Image Updater, which I already wanted for other reasons.
Iteration 3: stop writing¶
| A: SHA tags | B: Image Updater | C: both | |
|---|---|---|---|
| CI touches CD repo | Every push | Never | Every push |
| Deployment trigger | Git commit | Registry poll | Git commit |
| CD repo noise | High | Low | High |
| Conflicts with Image Updater | Yes | n/a | Yes, unless scoped |
I chose B. CI pushes three tags and stops. It has no credentials for the deployment repository, no knowledge that a cluster exists, and no step that can fail because of something on the CD side.
The controller watches one tag's digest and patches the application itself, which took three configuration attempts of its own before it worked.
The Write That Stayed¶
One thing still commits to the deployment repository: the chart release.
That looks like a contradiction and is not, because the two writes answer different questions.
- An image digest change means the container has new code inside it.
- A chart version change means the Deployment itself has a new field, a new environment variable, a different probe.
Image Updater owns the first. It has no opinion about the second and cannot express one, because a digest cannot add a ConfigMap. So chart-release.yaml packages the chart, pushes it to the registry as an OCI artifact, then updates the version in the deployment repository's kustomization.yaml.
They never contend for the same field. And chart changes are rare compared to code changes, so the commit volume that made iteration 2 unpleasant does not apply.
The test for whether two writers can coexist
Not "do they write to the same repository," which they do here. It is whether they write to the same field. Two mechanisms editing one value is a race. Two mechanisms editing adjacent values is just two mechanisms.
The Token That Only One Workflow Holds¶
Because exactly one workflow writes to a foreign repository, exactly one workflow needs a credential for it, and it can be scoped tightly.
GITHUB_TOKEN cannot push to another repository, so this needs a fine-grained personal access token limited to the deployment repository with Contents: Read and Write and nothing else. The handling matters as much as the scope:
git clone https://github.com/${CD_REPO}.git cd-repo
cd cd-repo
git remote set-url origin https://x-access-token:${GIT_TOKEN}@github.com/${CD_REPO}.git
# ... commit and push ...
The clone uses the public URL, so the token never appears in the process arguments where anything reading /proc or a CI log could pick it up. It is injected afterwards by rewriting the remote, and cleared from the remote after the push.
This is a side benefit of the pull model that I did not anticipate. In the push model every build job needs write access to the deployment repository, so a compromised build of any service can rewrite deployment intent for all of them. In the pull model that credential exists in one workflow that runs only when a chart changes.
What You Give Up¶
Pull is not free, and the cost is real.
The audit trail moves. There is no commit per deployment. What ran and when now lives in the controller's logs and the application's revision history, not in git log. If your compliance story is "every production change is a reviewed commit," pull does not give you that, and I would not pretend otherwise.
Rollback changes shape. With push, a revert is a rollback. With digest tracking, reverting means pinning a previous digest or pushing an older image forward, and the mechanism is the controller rather than Git.
A registry poll is not an event. There is a delay, roughly two minutes here. Push is as fast as ArgoCD's own sync interval.
The Line I Would Draw Now¶
Not a rule about GitOps. A question about ownership.
Push works when one producer owns one consumer and the link between them is short enough to see. It stops working when the number of producers grows, or when something else acquires an opinion about the same field, and the second one is the dangerous case because it does not announce itself.
The Java monolith is push and its link is broken today. The microservices are pull and their handoff has one moving part. That is not because pull is better. It is because ten producers writing to one file is a design that needs a reason, and I did not have one.
Source¶
- The CI runbook, including all three iterations with the logs
- The trigger workflow, which states its own scope in the header
- The chart release workflow, the one write that stayed
- Release identity, on what a build is called and what is allowed to ship
Series: Ten Services, No Hands (Part 2 of 3)