TLS Behind NAT Is Two Problems, Not One¶

A Kubernetes cluster on a private address cannot use the HTTP-01 challenge. Not because something is misconfigured, but because the challenge requires Let's Encrypt to open a connection to you, and nothing routable exists to connect to.

The mistake I made next was assuming that solving the certificate solved the problem. It does not. Getting a trusted certificate and being reachable from the internet are separate problems with separate solutions, and one of the three approaches below fixes only the first.

Why HTTP-01 Cannot Work Here¶

The cluster sits at 172.16.0.2, an RFC 1918 address. HTTP-01 requires three things:

1. A public DNS record pointing at your cluster
2. Let's Encrypt reaching http://your.domain/.well-known/acme-challenge/<token>
3. Port 80 open from the internet

The first is possible. The second cannot happen: a private address is not routable, so there is nowhere for the record to point and nothing for Let's Encrypt to connect to. The challenge times out or is refused, every time, forever.

This is worth stating plainly because the failure looks like a configuration problem for a long time. cert-manager retries, the Challenge object sits pending, and the message reads like something you could fix. On a cluster with a public IP that is often true. Here it is not.

The Split Nobody Mentions¶

Two questions get collapsed into "set up HTTPS":

• Issuance: can I obtain a certificate a browser will trust?
• Reachability: can anyone outside my network get to the service?

HTTP-01 answers both at once, which is why the split is invisible until HTTP-01 stops being available. Once it does, the three viable approaches divide along exactly that line.

That middle column is the one that costs people an afternoon. DNS-01 is the answer to "how do I get Let's Encrypt to issue a certificate behind NAT", and it is a complete answer to that question and no part of an answer to the other one.

Approach 1: DNS-01¶

Instead of proving you control the server, prove you control the domain.

Let's Encrypt asks you to create a TXT record under _acme-challenge.your.domain, then queries public DNS for it. It never contacts your cluster, so your cluster's address is irrelevant. Behind NAT, behind CGNAT, on a laptop, air-gapped from inbound traffic entirely: none of it matters.

The cost is that cert-manager now needs API credentials for your DNS provider, because it creates and removes that record automatically on every issuance and renewal.

solvers:
  - dns01:
      cloudflare:
        apiTokenSecretRef:
          name: cloudflare-api-token
          key: api-token

The token wants the narrowest scope the provider allows. For Cloudflare that is Zone:DNS:Edit plus Zone:Zone:Read, restricted to the single zone. It goes in a Secret in the cert-manager namespace, not in the application namespace, because the controller reads it rather than the workload.

Twenty-odd providers are supported and the pattern is identical for all of them. Route 53, Google Cloud DNS, DigitalOcean, Azure DNS all differ only in the solver block and the credential shape.

Approach 2: Cloudflare Tunnel¶

A daemon on the cluster opens an outbound connection to Cloudflare's edge and holds it open. Requests to your domain arrive at Cloudflare, get matched to that connection, and are pushed back down it.

Outbound works through NAT by default, because that is what NAT is for. No inbound route required, no port forwarding, no control of the gateway.

TLS terminates at Cloudflare's edge with their certificate, which means cert-manager, ACME and renewal leave the picture entirely. There is no certificate for you to obtain or rotate.

That is the trade. You get both problems solved in one move, and in exchange the certificate is not yours and the connection depends on a third party staying up. For a lab, a home server, or anything where a Cloudflare outage is survivable, it is the least total machinery of the three.

This is the approach I use across the SilverStack playground images, where the constraint is identical and the reasoning is written out at length.

Approach 3: Self-Signed¶

Generates a certificate nothing trusts. Every browser warns. It is not a production answer and it is not trying to be.

It is, however, the fastest way to answer a different question: is my TLS wiring correct at all?

A Gateway with an HTTPS listener, a certificateRefs pointing at a Secret, an HTTPRoute attached, redirects configured. All of that can be wrong independently of which authority signed the certificate. Debugging it while also debugging ACME means two unknowns at once, and ACME failures take minutes to surface.

Self-signed removes one of the unknowns in about five minutes. If HTTPS works with a self-signed certificate, the Gateway configuration is correct and every remaining failure is in issuance.

What I Actually Did¶

Self-signed first, deliberately, to prove the Gateway TLS path end to end. Then Cloudflare Tunnel for the public deployment.

Not because DNS-01 is worse. Because at the point of decision the cluster already needed to be publicly reachable, and DNS-01 does not do that, so choosing it would have meant standing up a tunnel anyway and then maintaining an ACME setup on top of a system that already terminates TLS for me.

If the cluster had been internal-only, on a VPN with private DNS, that calculus reverses completely. DNS-01 would be the right answer and a tunnel would be pointless.

Renewal Is Where the Approaches Really Diverge¶

Setup time is the number people compare and it is the least important one. A certificate is not a task, it is a recurring obligation every sixty days, and the three approaches distribute that obligation very differently.

DNS-01 renews unattended, and the credential is the liability. cert-manager creates the TXT record, waits for propagation, completes the challenge and deletes the record, roughly a month before expiry. Nothing to remember. What you now own is an API token with write access to your DNS zone, living in the cluster, valid until you rotate it. That token can change any record in the zone, not just _acme-challenge. Scope it to a single zone and treat its rotation as a real task, because a leaked DNS token is a domain takeover rather than a certificate problem.

A tunnel has no renewal at all. The certificate is Cloudflare's and lives at their edge. There is genuinely nothing to rotate on your side. In exchange, the connection itself becomes the thing that must stay healthy, so the failure mode moves from "expired in sixty days" to "the daemon died and nobody noticed". Those need different monitoring: one is a date, the other is a process.

Self-signed expires and nothing tells you. Whatever validity you chose at generation, a cron job or a calendar entry is the entire renewal mechanism. This is fine for a validation step measured in hours and quietly awful for anything that outlives your attention.

The Question That Picks For You¶

Not "which is most correct". This one:

Does anything outside my network need to reach this?

If no, DNS-01. You get a real certificate, cert-manager handles renewal, and there is no third party in the request path.

If yes, a tunnel, and DNS-01 becomes redundant because the edge already terminates TLS.

If you do not know yet, self-signed, because it validates everything except the authority and takes five minutes to undo.

The failure mode I want to name specifically: choosing DNS-01 because the article was titled "TLS behind NAT", getting a valid certificate, and then discovering the service is still unreachable. Nothing went wrong. The wrong problem was solved thoroughly.

Source¶

• The full guide, all three approaches with complete steps and a decision matrix
• The tunnel approach in depth, on a different platform with the same constraint
• When HTTP-01 can be made to work, on a public IP, and what it takes