My Builds Were Reproducible, So My Deployments Stopped¶
Ten microservices on EKS, images landing in the registry on every push, and a controller whose entire job is to notice new images and roll the pods. It noticed nothing. No errors, no warnings, just a log line reporting that it had considered ten images and updated zero of them.
The cause turned out to be a feature working correctly in one tool making another tool's core assumption false.
Try It Yourself
The working configuration is public. The Image Updater CR and the build workflow that feeds it are both readable in full.
The Setup¶
CI builds each changed service and pushes three tags:
| Tag | Purpose |
|---|---|
sha-<40 char> | Immutable, traceable to the exact commit |
sha-<7 char> | Human readable in the ArgoCD UI and logs |
latest | The tag the controller watches |
ArgoCD Image Updater was configured with the newest-build strategy, filtering to the short SHA tags:
The intent reads correctly. Track tags matching that shape, pick the newest one, update the application. Nothing about it looks wrong.
The First Failure Is Not the Real One¶
The logs showed:
allowTags requires an explicit prefix declaring the match type. Without it, the value is not rejected, it is ignored, which is a materially different outcome:
A filter that is ignored is not a filter that fails
An ignored filter means every tag is a candidate rather than none, so the behaviour afterwards is still plausible and still wrong. The log line is there, at info level, in a controller that logs a great deal at info level.
With the prefix added, the controller found the tags. And still did nothing:
Zero errors. Ten images seen. Nothing skipped, which rules out a filter problem. Nothing updated, which rules out everything being current, because I had just pushed a new image.
Asking What "Newest" Means¶
The strategy is named newest-build. That name is a promise about ordering, and ordering requires a comparable field. The only per-image field that could carry age is the created timestamp in the image config.
The manifest itself has no created field. It lives in the config blob, and there it read:
Not the build time. The Unix epoch. Every image, every service, every push, identical.
Why BuildKit Does This¶
This is not a bug. It is the point.
A reproducible build means the same source produces a byte-identical image. If the build stamps the current time into the image config, then two builds of the same commit differ, which breaks the guarantee, breaks content-addressed caching, and makes digests useless as an identity for "this exact source".
So BuildKit zeroes the timestamp by default. The field still exists because the image spec requires it. It just carries no information.
The two designs are individually correct and jointly broken
Reproducible builds deliberately remove time from the image. The newest-build strategy requires time to be present in the image. Neither tool is misbehaving. The failure only exists where they meet, which is why nothing on either side logs an error.
Every image had the same age. There was no newest. The comparison ran, found a ten-way tie, and kept whatever it had.
The Fix, and Two More Failures Inside It¶
Ranking by age cannot work when age has been deliberately removed. The alternative is not to rank at all: watch one tag and compare its digest to the digest currently deployed.
This failed immediately, and usefully:
The digest strategy compares the digest of a specific tag. A bare repository path does not identify one. Every image entry needs the tag written explicitly:
That error message is a good one: it names the missing thing and refuses to guess.
The second failure was residue. The earlier newest-build attempt had written stale image overrides into the Application spec, which now competed with the digest pins:
kubectl patch application microservices-demo -n argocd --type json \
-p '[{"op": "remove", "path": "/spec/source/kustomize/images"}]'
This cleanup step is not part of the setup
On a fresh install it is never needed. It exists only because a previous strategy wrote state that the new strategy does not own. Worth separating in your head from the actual configuration, because copying it into a runbook as a required step teaches the wrong thing.
Then it worked:
Successfully updated image 'ghcr.io/.../frontend:latest' to 'ghcr.io/.../frontend:latest@sha256:...'
Committing 10 parameter update(s) for application microservices-demo
Processing results: applications=1 images_considered=10 images_updated=10 errors=0
The Consequence for the CD Side¶
Choosing digest tracking forces a decision upstream that looks unrelated. The chart's values had used an empty tag, falling back to the chart's appVersion. In chart/values-eks.yaml:
tag: "" cannot work here. CI does not push images tagged with the chart version, and adding such a tag would create a tag nothing else consumes purely to satisfy a default. The controller needs a tag it can watch, so the chart points at latest and the controller pins the digest behind it.
That produces the arrangement people find uncomfortable at first glance: the manifest says latest, and the running pod says something else entirely.
kubectl get pods -n boutique-app -l app=frontend \
-o jsonpath='{.items[0].spec.containers[0].image}'
# ghcr.io/ibtisam-iq/microservices-demo/frontend:latest@sha256:<digest>
Both are present. latest is the subscription, the digest is the delivery. What actually runs is pinned to an immutable content hash, which is a stronger guarantee than a mutable tag that happens to look immutable. The rule from release identity still holds: nothing floating reaches the cluster. It is just enforced by digest resolution rather than by writing a version string into a file.
Why Three Tags and Not One¶
The build pushes three tags on every run, which looks redundant until each one has a named consumer.
sha-<40 char> is the audit trail. It is the full commit hash, so it answers "which source produced this" without ambiguity and without a lookup. Nothing automated consumes it, and that is fine: its job is to exist when someone is reconstructing an incident six weeks later.
sha-<7 char> is the same answer, shortened, for humans reading the ArgoCD UI. Computing it needed a dedicated step, because GitHub Actions expressions have no string slicing:
latest is the only tag with an automated consumer, and its mutability is the feature rather than a compromise. The controller needs one stable name to watch. What it does with that name is resolve it to an immutable digest and pin that.
The redundancy is deliberate and cheap
All three tags point at the same image, so the registry stores one set of layers with three references. The cost is three docker push calls against layers that are already uploaded after the first. The benefit is that three different consumers each get a name shaped for them, instead of one name compromising for all three.
The Shape Worth Remembering¶
The tell was images_updated=0 with errors=0. A comparison that runs successfully and produces no action is either genuinely idempotent or comparing fields with no information in them, and those two look the same from outside.
When a tool ranks things, find the field it ranks on and look at the actual values. Not whether the field exists, which it did, and not whether the tool reads it, which it does. Whether the values differ.
The general version: reproducible builds strip variance on purpose. Anything downstream that was relying on that variance to order things is going to stop, quietly, and its logs will say everything is fine.
It is worth asking that question of any two tools you have adopted separately on the strength of good advice about each. Both being correct is not the same as both being compatible, and the seam between them is where nobody owns the error message.
Source¶
- Phase 4: GitOps with ArgoCD runbook, with the log output at each stage
- The working Image Updater configuration
- The build workflow that pushes the three tags
Series: Ten Services, No Hands (Part 1 of 3)