An ACME Challenge That Could Not Reach Itself¶

Let's Encrypt could not reach my cluster to validate a domain. That part I expected to fix. What took the rest of the day was that the cluster could not reach itself either, and cert-manager will not ask Let's Encrypt for anything until it can.

The Symptom, and Why It Is Not Obvious¶

A Certificate sat at READY=False. The Challenge underneath it stayed pending:

Reason: Waiting for HTTP-01 challenge propagation:
failed to perform self check GET request
'http://java-monolith.ibtisam-iq.com/.well-known/acme-challenge/<token>':
dial tcp <public-ec2-ip>:80: connect: connection refused
State: pending

Two details in that message matter more than they look.

"self check" means this is not Let's Encrypt failing. cert-manager runs its own probe first, and only proceeds to ask Let's Encrypt to validate once its own probe succeeds. The external CA had not been involved yet.

connection refused, not timed out. A timeout means packets vanished, which points at a firewall or a security group. A refusal means the packet arrived, the kernel found no socket listening on that port, and sent back a RST. The network was fine. There was simply nothing there.

The cascade from that one refusal runs the whole length of the object chain:

Port 80 not reachable
  -> cert-manager self-check fails
    -> Challenge stays pending
      -> Order stays pending
        -> CertificateRequest never issued
          -> Certificate READY=False
            -> Secret never created
              -> Gateway serves a self-signed cert on 443
                -> browser shows "Not Secure"

The visible symptom is at the bottom. The cause is at the top, seven objects away.

Nothing Was Listening, and That Is Normal¶

The first thing worth checking is the actual socket table on the node:

sudo ss -tlnp | grep -E ':80|:443'

Empty. Which is correct behaviour, and the part people skip past: a NodePort Service does not bind port 80 on the host. It binds a high port, usually somewhere above 30000, and kube-proxy routes from there.

Port 80    -> nothing            <- external traffic arrives here and is refused
Port 443   -> nothing
Port 32030 -> gateway nginx (HTTP)   <- the actual listener
Port 32315 -> gateway nginx (HTTPS)

The EC2 security group being open on 80 is irrelevant. The security group governs whether a packet reaches the instance. It has no opinion about whether anything is listening once it arrives.

The Wrong Service¶

Here is where I lost the most time, and it is specific to Gateway API implementations rather than to Ingress.

I had been inspecting the NGINX Gateway Fabric controller Service in the nginx-gateway namespace, found it was ClusterIP, and concluded I needed to convert it to a NodePort. That would not have helped.

NGF creates a separate nginx deployment and Service for each Gateway object, in the namespace of that Gateway. The controller watches Gateway resources and provisions data planes. It is not itself the data plane.

kubectl get svc -n bankapp

bankapp-gateway-nginx   NodePort    10.105.42.246   80:32030/TCP,443:32315/TCP
bankapp-service         NodePort    10.98.85.236    80:30082/TCP
cm-acme-http-solver-…   NodePort    10.104.173.56   8089:32563/TCP
mysql-service           ClusterIP   10.108.102.184  3306/TCP

bankapp-gateway-nginx is the thing serving traffic for bankapp-gateway. The controller Service stays ClusterIP permanently and that is by design.

Worth confirming the data path works before touching the host, so that a later failure cannot be blamed on the Gateway:

curl -I -H "Host: java-monolith.ibtisam-iq.com" http://172.31.86.199:32030
# HTTP/1.1 301 Moved Permanently
# Location: https://java-monolith.ibtisam-iq.com/

The Gateway, the listener and the HTTPRoute were all working. The only missing piece was that nothing connected host port 80 to port 32030.

Redirecting Port 80, Both Directions¶

iptables can intercept packets before they look for a socket:

HTTP_NODEPORT=32030
HTTPS_NODEPORT=32315

sudo iptables -t nat -A PREROUTING -p tcp --dport 80  -j REDIRECT --to-port $HTTP_NODEPORT
sudo iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port $HTTPS_NODEPORT

PREROUTING handles packets arriving from elsewhere. That covers Let's Encrypt, and it covers a browser.

It does not cover packets the node generates for itself. Locally originated traffic skips PREROUTING entirely and goes through OUTPUT, so a curl localhost test from the node still fails while external traffic works, which is a confusing pair of results to hold at once:

sudo iptables -t nat -A OUTPUT -p tcp --dport 80  -j REDIRECT --to-port $HTTP_NODEPORT
sudo iptables -t nat -A OUTPUT -p tcp --dport 443 -j REDIRECT --to-port $HTTPS_NODEPORT

The Self-Check Still Failed¶

External curl to the domain now worked. The challenge still sat pending with the same message.

The reason is where the self-check runs from. It is not the node. It is a pod, and pods resolve DNS through CoreDNS, which forwards to the upstream resolver, which returns the public A record. So the probe went: pod, out to the public IP of the instance it is already running on, and back. That hairpin does not reliably work on EC2, and the PREROUTING rules were written against traffic arriving at the private address.

The fix is to make the name resolve differently inside the cluster than outside it:

kubectl edit configmap coredns -n kube-system

Inside the .:53 block, above the forward stanza:

hosts {
  172.31.86.199 java-monolith.ibtisam-iq.com
  fallthrough
}

fallthrough matters. Without it, this block becomes authoritative for everything it does not match and other lookups break. With it, unmatched names carry on to forward as normal.

kubectl rollout restart deployment/coredns -n kube-system
kubectl exec -it <pod> -n bankapp -- getent hosts java-monolith.ibtisam-iq.com
# 172.31.86.199   java-monolith.ibtisam-iq.com

Now the self-check resolves to the node's private address, hits PREROUTING, gets redirected to the Gateway NodePort, and reaches the solver pod. Let's Encrypt still resolves the public address from outside, which is what it should do.

Stale Challenges Hide a Working Fix¶

The configuration was correct at this point and nothing changed, because ACME objects do not retry indefinitely and old ones do not re-evaluate:

kubectl delete challenge -n bankapp --all
kubectl delete order -n bankapp --all

kubectl annotate certificate java-monolith-tls -n bankapp \
  cert-manager.io/force-renewal="true" --overwrite

kubectl get challenges -n bankapp -w

Presented goes true, then State moves pending to valid.

The Alternative I Did Not Take¶

There is a second fix that removes the NodePort translation entirely: bind the proxy to the host network.

kubectl patch deployment ngf-nginx-gateway-fabric -n nginx-gateway \
  --type=json \
  -p='[
    {"op": "add", "path": "/spec/template/spec/hostNetwork", "value": true},
    {"op": "add", "path": "/spec/template/spec/dnsPolicy",  "value": "ClusterFirstWithHostNet"}
  ]'

Simpler, and it removes both the iptables rules and the reboot fragility. Two things to know before reaching for it.

dnsPolicy: ClusterFirstWithHostNet is not optional. With hostNetwork: true alone, the pod inherits the node's /etc/resolv.conf and in-cluster service DNS stops resolving, which breaks the proxy's ability to reach the backends it is proxying to.

And check the port is free first. If anything else is bound to 80, the pod crash-loops on bind: address already in use:

sudo ss -tlnp | grep -E ':80|:443'

I stayed with iptables because the redirect is reversible with one command and does not restart the data plane, which mattered while I was still changing things. On a node I intended to keep, hostNetwork is the better answer.

When None of This Applies¶

If you run ingress-nginx with the ingress solver rather than Gateway API with gatewayHTTPRoute, you will probably meet none of the above. Exposing the controller through a NodePort or LoadBalancer is usually enough, and neither the iptables hairpin nor the CoreDNS override is needed.

The difference is not that Gateway API is harder. It is that Gateway API separates the controller from the data plane, and that separation puts the thing you need to expose somewhere you were not looking. Every wrong turn in this chain came from checking the object that manages traffic instead of the object that carries it.

The transferable check is small. When something cannot be reached, look at the socket table before the Service, the Service before the Ingress, and always ask which process is expected to hold the port. connection refused is a complete answer to a question most of us skip: is anything actually listening.

Source¶

• The full troubleshooting runbook, with every verification command
• cert-manager overview, covering both solvers and both fixes
• Three approaches for a cluster with no public IP, where HTTP-01 is not viable at all