The Auto Scaling Group kept replacing instances. The application kept starting cleanly. Neither of those statements was wrong, and the gap between them took most of an afternoon to close. Then the moment the health check passed, a second failure appeared underneath it that had been invisible until the first one was gone.
Let's Encrypt could not reach my cluster to validate a domain. That part I expected to fix. What took the rest of the day was that the cluster could not reach itself either, and cert-manager will not ask Let's Encrypt for anything until it can.
A Kubernetes cluster on a private address cannot use the HTTP-01 challenge. Not because something is misconfigured, but because the challenge requires Let's Encrypt to open a connection to you, and nothing routable exists to connect to.
The mistake I made next was assuming that solving the certificate solved the problem. It does not. Getting a trusted certificate and being reachable from the internet are separate problems with separate solutions, and one of the three approaches below fixes only the first.
cert-manager has two HTTP-01 solvers. Which one you need is not a preference, it is determined entirely by what is routing traffic in your cluster. Pick the wrong one and the challenge never gets served.
Pick the right one and it still may not work, because on a manifest install the Gateway API solver is compiled in and switched off.
I ran the same routing problem three times: NGINX Gateway Fabric on a bare metal cluster, the AWS Load Balancer Controller on EKS, and an ALB Ingress Group sharing one load balancer across three services. Two used Gateway API and one used Ingress, and the differences between them were not the ones I expected.
Splitting an Express app into three tiers took one line out of server.js. That line was the only thing serving the frontend, and removing it created a question the original architecture never had to answer: with the API no longer serving the UI, what does? I tried three answers before the right one, and each failure was a different lesson about what decoupling actually costs.
I ran curl ifconfig.me on four different servers and got the same address back four times. That single result invalidated the entire approach I had spent three days validating on EC2. This is the full path from that discovery to three services running on my own subdomains with valid certificates and zero inbound ports.