Ten Things That Failed Silently¶

I went back through a year of my own failures across five projects and found the same one ten times. Every case had the same shape: I set a value, the tool accepted it, the tool did not apply it, and nothing anywhere said so.

This is the list, grouped by the reason each tool stayed quiet, because the groups turn out to be more useful than the instances.

None of the ten is a bug report. Each default is individually defensible, several are the documented behaviour, and two are the entire point of the feature. The problem is not that tools misbehave. It is that "accepted" and "applied" are different states, and almost nothing distinguishes them for you.

Group 1: Nothing Was Listening¶

The value was written correctly to a place that has no reader.

Maven ignored a version override. Spring Boot's parent POM exposes a <properties> key for some managed dependencies and not others. Setting thymeleaf.version looks identical to setting spring-security.version, but the second is a key the BOM defines and the first is not, so the first is a property nobody reads. The build succeeded, the scanner kept reporting the CVE, and the POM looked correct. Full write-up.

Terraform dropped a cluster setting before the API call. With create_iam_role = false, version 21 of the community EKS module stops emitting bootstrap_cluster_creator_admin_permissions into the cluster's access configuration. The variable is still accepted at the module interface. The cluster came up ACTIVE and nobody, including the identity that created it, could authenticate to it. Full write-up.

CI wrote a deployment tag that nothing consumes. Every build writes an immutable image tag into image.env in the deployment repository. All three Kustomize files hardcode newTag: latest. The write is correct, the commit is correct, and the manifests have never once referred to the file. Full write-up.

The tell in all three: the producer is healthy and the consumer is somewhere else. No single file is wrong. The relationship between two files is wrong, and no tool validates relationships you did not declare.

Group 2: Something Read It and Moved On¶

The value was parsed, found unusable, and discarded at a log level nobody watches.

A tag filter was ignored rather than rejected. ArgoCD Image Updater's allowTags needs a prefix declaring the match type. Without regexp:, the value is not an error, it is skipped:

Invalid match option syntax '^sha-[a-f0-9]{7}$', ignoring

An ignored filter does not match nothing. It stops filtering, so every tag becomes a candidate, which is a plausible-looking wrong behaviour rather than an obvious one. That line is at info level in a controller that emits a lot of info. Full write-up.

An EKS cluster ignored its own aws-auth ConfigMap. authenticationMode has three values. Under API, which is now the default, the aws-auth ConfigMap is not consulted at all; access comes from access entries instead. Editing aws-auth on such a cluster is editing a file the control plane has stopped reading. It persists, it is valid YAML, and it does nothing. Full write-up.

The tell here: the rejection exists, and it is quieter than the thing it is rejecting. A message at info level inside a component that logs continuously is functionally invisible.

Group 3: The Default Is Itself a Decision¶

Absence did not mean "unset". It meant a real, different behaviour.

Spring did not trust X-Forwarded-Proto. Behind a load balancer terminating TLS, the application receives plain HTTP and a header saying the client used HTTPS. Without server.forward-headers-strategy=native, that header is ignored, so post-login redirects were generated as http://, the browser upgraded them, and the session established on the other scheme was lost. Not trusting forwarded headers by default is correct, because trusting them blindly is a spoofing vector. It is also exactly what breaks you. Full write-up.

S3 skipped every encrypted object during replication. Cross-Region Replication has a field, SourceSelectionCriteria.SseKmsEncryptedObjects. Leave it unset and KMS-encrypted objects are not replicated. The replication rule is valid, the role is correct, the destination bucket is empty, and the reason is a field whose absence means "skip these" rather than "no preference". Full write-up.

The tell: an unset field with a non-neutral default. These are the hardest of the ten to catch by reading your own configuration, because the thing you need to notice is not there.

Group 4: The Value Was Present and Meant Nothing¶

Something was there. It carried no usable information.

BuildKit zeroed the timestamp that the deployment controller ranks on. Image Updater's newest-build strategy orders tags by the created field in the image config. BuildKit sets that to 1970-01-01T00:00:00Z deliberately, because a real timestamp would make the same source produce a different image and break reproducibility. Every image was exactly as old as every other image. The comparison ran correctly on a ten-way tie and did nothing, reporting images_considered=10 images_updated=0 with zero errors. Full write-up.

Groovy would have rendered a null as the string "null". In a Jenkins pipeline, an unescaped ${IMAGE_TAG} is interpolated by Groovy at parse time, not by the shell at runtime. If the versioning stage had not run, that expression renders as the four-character string null, and the commit message and the deployment tag would both say null. ArgoCD would then deploy an image tagged null, which is a perfectly valid tag that does not exist. The fix is to escape it so the shell evaluates it and the guard can catch an empty value. Full write-up.

The Service I was inspecting was not the one serving traffic. NGINX Gateway Fabric provisions a separate nginx deployment and Service for each Gateway, in that Gateway's namespace. The controller Service stays ClusterIP forever, by design. I spent an afternoon reasoning about the wrong object, and every fact I gathered about it was accurate. Full write-up.

The tell: the check succeeded. A comparison that runs on uninformative data, a template that renders an absent variable, an inspection of a real object that is not the relevant one. Nothing failed, so nothing was reported.

What Actually Catches These¶

The habit that would have caught eight of the ten is boring and takes seconds: after setting something that matters, read it back from the thing that consumes it, not from the thing you wrote.

Concretely, per layer:

The pattern is the same each time. The producer's view is what you intended. The consumer's view is what happened. When those two disagree, only one of them is reality, and it is never the one in your editor.

Two of the ten resist this. The Maven property key and the unset S3 field are both absences, and you cannot read back a value you never set. For those the check is different: ask what the default is before assuming there is not one. mvn help:evaluate returning null object or invalid expression is a complete answer in under a second, and it arrives before you have edited anything.

Why This Keeps Happening¶

It is tempting to file this under carelessness, and I do not think that survives contact with the list.

Reproducible builds should strip timestamps. Frameworks should not trust forwarded headers by default. A BOM cannot expose a property for every transitive artifact. A module abstracting a resource will sometimes not pass a field through. Each of these is a reasonable decision made by someone competent, and the failure only exists where two of those decisions meet.

That is also why nothing logs it. The error is not inside either component. It is in the gap, and no component owns the gap.

What changed for me is smaller than a methodology. I stopped treating a successful apply as evidence that a setting took effect, because eight of these ten produced a successful apply. The green check mark was accurate about the thing it was checking and silent about the thing I cared about.

The Ten¶

Three of these are still open in my repositories rather than fixed, which is why they are written down. Number 3 in particular is live: the tag is produced correctly and consumed by nobody, and it is a better example than it is a bug.