I set app.messaging.sqs.topic in a values file. Helm accepted it. The chart rendered without error. The deployed pod did not have the environment variable, because the chart contains no reference to SQS anywhere.
I found that by reading the template, not by debugging the application. That is the skill this post is about, because most Helm charts you deploy are charts someone else wrote.
A freshly provisioned EKS cluster cannot provision a load balancer, cannot create a persistent volume on a modern disk type, cannot route Gateway API traffic, and cannot create a DNS record. It runs pods and that is roughly it.
Four add-ons close that gap, and three of them have a default that silently does the wrong thing.
Terraform applied cleanly. The cluster reached ACTIVE. Then kubectl returned 401, and it turned out that nobody, including the account that created it, had permission to talk to the thing that had just been created successfully.
The setting that grants that permission was in my configuration. The module took it, validated it, and did not pass it to AWS.
Provisioning EKS in a restricted AWS account produced seventeen distinct failures. What made it hard was not the count. It was that several of them only existed because of the fix for the previous one.
Three of the chains are worth walking through, because the pattern they share is the actual lesson: in a constrained environment, the correct move is often to back out rather than push forward.
The connection string was correct. I checked it several times. The application still could not reach MySQL, and the error said nothing useful about why.
The URL was being cut in half before Java ever saw it, by the shell, at a character I had not thought of as a character.
The Auto Scaling Group kept replacing instances. The application kept starting cleanly. Neither of those statements was wrong, and the gap between them took most of an afternoon to close. Then the moment the health check passed, a second failure appeared underneath it that had been invisible until the first one was gone.
Let's Encrypt could not reach my cluster to validate a domain. That part I expected to fix. What took the rest of the day was that the cluster could not reach itself either, and cert-manager will not ask Let's Encrypt for anything until it can.
A Kubernetes cluster on a private address cannot use the HTTP-01 challenge. Not because something is misconfigured, but because the challenge requires Let's Encrypt to open a connection to you, and nothing routable exists to connect to.
The mistake I made next was assuming that solving the certificate solved the problem. It does not. Getting a trusted certificate and being reachable from the internet are separate problems with separate solutions, and one of the three approaches below fixes only the first.
cert-manager has two HTTP-01 solvers. Which one you need is not a preference, it is determined entirely by what is routing traffic in your cluster. Pick the wrong one and the challenge never gets served.
Pick the right one and it still may not work, because on a manifest install the Gateway API solver is compiled in and switched off.
I ran the same routing problem three times: NGINX Gateway Fabric on a bare metal cluster, the AWS Load Balancer Controller on EKS, and an ALB Ingress Group sharing one load balancer across three services. Two used Gateway API and one used Ingress, and the differences between them were not the ones I expected.