Four Add-ons Between a Bare EKS Cluster and a Working One¶

A freshly provisioned EKS cluster cannot provision a load balancer, cannot create a persistent volume on a modern disk type, cannot route Gateway API traffic, and cannot create a DNS record. It runs pods and that is roughly it.

Four add-ons close that gap, and three of them have a default that silently does the wrong thing.

What Is Actually Missing¶

They install in that order because each depends on the previous one existing. The Gateway cannot be programmed without the controller; ExternalDNS has nothing to watch until HTTPRoutes exist.

Gateway API Support Is a Feature Gate¶

The ALB controller installs cleanly and, by default, ignores Gateway API entirely. It watches Ingress and Service objects and nothing else.

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  --namespace kube-system \
  --set clusterName="$CLUSTER_NAME" \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set controllerConfig.featureGates.ALBGatewayAPI=true \
  --set controllerConfig.featureGates.NLBGatewayAPI=true

Omit those two flags and the sequence that follows is: apply a GatewayClass, apply a Gateway, watch it stay Programmed: False indefinitely, and start debugging the Gateway. Nothing is wrong with the Gateway. The controller responsible for it is not watching that resource type.

This is the same shape as a pattern I kept hitting: a correct resource, a running controller, and no relationship between the two because a capability was left off.

The controller needs IRSA before any of this, and the OIDC provider association is a prerequisite for IRSA:

eksctl utils associate-iam-oidc-provider \
  --region $REGION --cluster "$CLUSTER_NAME" --approve

Skip that and every subsequent eksctl create iamserviceaccount produces a role whose trust policy references a provider that does not exist. The role is created. It cannot be assumed.

The Default StorageClass Is the Old One¶

EKS ships a gp2 StorageClass backed by kubernetes.io/aws-ebs, the in-tree provisioner. It works, which is why it survives.

gp3 is cheaper for the same volume, and it includes 3,000 IOPS and 125 MB/s of throughput at no additional cost, where gp2 ties IOPS to volume size. For a small Elasticsearch data volume, gp2 gives you a fraction of the IOPS that gp3 provides for free.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
allowVolumeExpansion: true
parameters:
  type: gp3

Two fields worth not skipping past.

volumeBindingMode: WaitForFirstConsumer delays volume creation until a pod is scheduled, so the EBS volume is created in the availability zone the pod actually landed in. With the default Immediate, the volume is created first and the pod is then constrained to that zone, which produces unschedulable pods on a multi-AZ cluster for reasons that look like a scheduling problem.

allowVolumeExpansion: true cannot be added retroactively to bound volumes in a useful way. Setting it at creation costs nothing and not setting it means recreating the class and the PVCs later.

The Add-on That Needed a Manual Annotation¶

The EBS CSI driver installs as an EKS managed add-on, which normally takes the IRSA role directly:

aws eks create-addon \
  --cluster-name $CLUSTER_NAME \
  --addon-name aws-ebs-csi-driver \
  --service-account-role-arn $ROLE_ARN     # blocked here

In the restricted account I was working in, --service-account-role-arn is not permitted. The add-on installs without it, creates its service account with no role annotation, and every volume provisioning attempt fails on permissions.

The workaround is to annotate afterwards and restart, because the credential is injected at pod admission:

kubectl annotate serviceaccount ebs-csi-controller-sa -n kube-system \
  eks.amazonaws.com/role-arn=$ROLE_ARN --overwrite

kubectl rollout restart deployment ebs-csi-controller -n kube-system

The restart is the part people miss. Annotating a service account does nothing to pods that are already running; the projected token and AWS_ROLE_ARN are set when the pod is admitted. Without the restart the annotation is correct, visible in kubectl get sa -o yaml, and having no effect.

ExternalDNS Watches Two Things By Default¶

This is the sharpest of the four.

ExternalDNS installs, runs, has Route 53 write access through IRSA, and reports healthy. You create an HTTPRoute with hostname: app.example.com. No DNS record appears.

The default sources list is service and ingress. Gateway API route types are not in it:

sources:
  - service
  - ingress
  - gateway-httproute
  - gateway-tlsroute
  - gateway-tcproute
  - gateway-udproute

Every symptom points at DNS or at permissions, and the actual cause is that the controller is not looking at the resource type you are using. Nothing logs an error, because from ExternalDNS's perspective there is simply nothing to reconcile.

Once the sources are right, the payoff is real: every HTTPRoute created afterwards gets its record automatically, and five subdomains across four namespaces populated themselves with no manual Route 53 work.

One Gateway, Five Hostnames¶

The routing model is worth stating because it is the argument for Gateway API over Ingress in a shared cluster.

One Gateway with two listeners, HTTP on 80 and HTTPS on 443, accepting routes from anywhere:

allowedRoutes:
  namespaces:
    from: All

The ALB certificate comes from a LoadBalancerConfiguration referencing an ACM wildcard ARN, so TLS is configured once rather than per route.

Then five HTTPRoutes in four different namespaces attach to it: the application, ArgoCD, Grafana, Prometheus and Kibana. One ALB serves all of them. Five ALBs would be five sets of hourly charges for what is architecturally one entry point.

The Gateway took about five minutes to reach Programmed: True while the ALB provisioned and passed its health checks. That delay is normal and looks identical to a misconfiguration for the first three of those minutes.

The One I Forgot¶

Metrics Server belongs in this phase and I installed it two phases later, when an HPA reported cpu: <unknown>/5% and refused to scale.

EKS does not include it. Without it the metrics.k8s.io API does not exist, so kubectl top fails and every HPA sits in ScalingActive: False with FailedGetResourceMetric.

It is worth listing alongside the other four rather than treating it as an autoscaling concern, because the thing that needs it is deployed much later than the thing that provides it, and that gap is exactly where it gets forgotten.

The Order, and Why¶

1. OIDC provider association, because IRSA depends on it and everything else depends on IRSA.
2. ALB Controller with both Gateway API feature gates on.
3. EBS CSI driver, then the gp3 class as default, before anything stateful.
4. Gateway API CRDs, GatewayClass, Gateway, which is when the ALB actually appears.
5. ExternalDNS with Gateway route types in sources.
6. Metrics Server, which I would now do here rather than later.

Three of those six have a default that produces a silent no-op: the feature gates, the ExternalDNS sources, and the EBS service account annotation. In every case the component is running, the resource is valid, and the two are simply not connected.

The check that catches all three is the same: after installing a controller, create the resource it is supposed to act on and confirm it acts. Not that the pod is Running, which it will be either way.

Add-ons Cost Capacity, and Two Kinds Scale Per Node¶

The six above are small. The platform tier that follows them is not, and the mistake I made was sizing the cluster for the application and then installing everything else into whatever was left.

The bill arrived later as a Pending pod:

FailedScheduling: 0/3 nodes are available: insufficient cpu/memory

That message says something narrower than it appears. insufficient cpu/memory is about requests, not usage. The scheduler places pods by summing requests against allocatable capacity, so a node can be twenty percent utilised and completely unschedulable, because requests are reservations rather than measurements. This is why "the nodes look idle in Grafana" and "nothing can schedule" are simultaneously true, and why looking at a CPU graph at that moment leads nowhere.

The second thing worth internalising is that DaemonSets do not behave like application workloads. A node exporter or a log shipper runs one pod per node, correctly, because node metrics and container logs are per-node facts. Their cost therefore scales with the cluster, not the traffic, so adding a node to fix a scheduling problem also adds more of them to the node you just added. The relief is real and smaller than the raw capacity suggests.

Both are knowable in advance. What the scheduler thinks is committed, per node:

kubectl describe node <node> | grep -A6 "Allocated resources"
# Requests are what matter. Limits are not a scheduling input.

And every request in the cluster, in one view:

kubectl get pods -A -o custom-columns=\
'NS:.metadata.namespace,NAME:.metadata.name,CPU:.spec.containers[*].resources.requests.cpu,MEM:.spec.containers[*].resources.requests.memory'

Run that after the application is installed and before the platform tier goes on. The gap between total requests and total allocatable is the budget, and whether the next component fits in it is a question you can answer before installing it rather than at the point a pod refuses to schedule.

One habit covers most of this: when adding a platform component, add its requests to the capacity model in the same change. Not afterwards, not as a follow-up. The install command and the sizing decision are the same decision, and separating them is how a cluster ends up ninety percent committed with nobody having decided that.

Verifying Each One Acted, Not Just Started¶

Because three of the six failure modes are "running but not connected", kubectl get pods is not a useful check for any of them. Each add-on needs a verification that exercises the thing it is supposed to do.

ALB Controller. Not the deployment being 2/2. Apply the Gateway and watch for the condition:

kubectl get gateway app-alb-gateway -o jsonpath='{.status.conditions}' | jq
# Programmed: True, and an ALB hostname in .status.addresses

An empty addresses after five minutes means the feature gates are off.

EBS CSI driver. Not the controller pods. Create a PVC and see whether it binds:

kubectl get pvc -A
# STATUS Bound, and STORAGECLASS gp3

Pending with WaitForFirstConsumer is normal until a pod claims it. Pending with a provisioning error in describe is the missing annotation.

ExternalDNS. Not the pod logs saying "running". Create an HTTPRoute with a hostname and check Route 53:

kubectl logs -n external-dns deploy/external-dns | grep -i "creating\|record"
aws route53 list-resource-record-sets --hosted-zone-id $ZONE_ID \
  --query "ResourceRecordSets[?contains(Name, 'app')]"

Silence in those logs when an HTTPRoute exists is the sources problem, and it is silence rather than an error.

Metrics Server. kubectl top nodes either returns numbers or it does not.

The pattern across all four: create the resource the controller is meant to reconcile, then look at the external system it is meant to change. Checking the controller tells you it is alive, which was never the question.

• Cluster add-ons runbook, with every command and the IRSA setup for each
• The manifests, GatewayClass through ExternalDNS patches
• What the Gateway model buys over Ingress, across three implementations