The EKS Module Accepted the Setting and Never Sent It¶

Terraform applied cleanly. The cluster reached ACTIVE. Then kubectl returned 401, and it turned out that nobody, including the account that created it, had permission to talk to the thing that had just been created successfully.

The setting that grants that permission was in my configuration. The module took it, validated it, and did not pass it to AWS.

The Constraint That Triggers It¶

This ran in an AWS account governed by an organisation policy that blocks a set of IAM actions. One of them shapes everything else:

iam:PassRole is denied unless the role name is exactly eksClusterRole.

That single restriction has a cascading consequence. The terraform-aws-modules/eks/aws module creates its own IAM roles by default, with generated names. Those names are not whitelisted, so the module's own role creation fails. The documented answer is to create the roles yourself and disable the module's:

create_iam_role = false
iam_role_arn    = aws_iam_role.eks_cluster.arn

That is a normal, supported configuration. It is also the exact flag that causes the silent drop.

The Symptom¶

After a clean apply:

aws eks update-kubeconfig --name silver-stack-eks --region us-east-1
kubectl get nodes
# error: You must be logged in to the server (the server has asked for credentials)

A 401 from the API server, from the identity that created the cluster. EKS grants the creating principal cluster admin through a field on the cluster's access configuration, and my configuration set it:

bootstrap_cluster_creator_admin_permissions = true

No Terraform error. No plan diff suggesting it was dropped. The value appears in the configuration and disappears somewhere between there and the API.

Confirming It Against the API Rather Than the Plan¶

Terraform state describes what Terraform believes. The API describes what exists, and when those two disagree the API wins:

aws eks describe-cluster --name silver-stack-eks \
  --query 'cluster.accessConfig'
# {"authenticationMode": "API_AND_CONFIG_MAP"}

The bootstrapClusterCreatorAdminPermissions field is simply absent from the response. Not false. Absent.

aws eks list-access-entries --cluster-name silver-stack-eks

One entry, for the EKS service role. Nothing for the identity that ran the apply.

The two requirements are individually reasonable and mutually exclusive in this account. The policy forces create_iam_role = false. That flag disables the admin bootstrap. So the module cannot produce a cluster you can reach.

The Fix Is to Stop Using the Module for the Cluster¶

Not for everything. The VPC module is fine, the EC2 module for the bastion is fine. Only the cluster resource needs to be raw, because only there does the wrapper drop a field. From eks.tf:

resource "aws_eks_cluster" "this" {
  name     = var.cluster_name
  role_arn = aws_iam_role.eks_cluster.arn
  version  = var.cluster_version

  access_config {
    authentication_mode                         = "API_AND_CONFIG_MAP"
    bootstrap_cluster_creator_admin_permissions = true
  }

  vpc_config {
    subnet_ids = module.vpc.private_subnets
  }
}

Passed directly to the provider, the setting reaches AWS. kubectl works on first try.

The general lesson is narrow and worth stating precisely: a module is a translation layer, and translation layers can drop things. When a setting is present in your configuration and absent from the API response, the module is where it went.

Five Denials Worth Knowing In Advance¶

Six actions are blocked in this account. Each one changes a design decision rather than just producing an error to retry.

The eks:CreateNodegroup denial is worth confirming rather than inferring, because "it did not work" and "it is not permitted" lead to different next steps:

EvalDecision: implicitDeny
AllowedByOrganizations: False

The IAM policy simulator answers this in seconds and removes an entire branch of debugging. AllowedByOrganizations: False means no amount of IAM policy on your user will change it.

Two Failures That Look Like Hangs¶

CoreDNS hangs forever on first apply. Declaring the CoreDNS addon in Terraform produces Still creating... until the timeout. CoreDNS is a Deployment; it needs a node to schedule onto; nodes do not exist yet because the node stack comes after the cluster. It waits for something that cannot happen until after it finishes.

The fix is to not declare it. EKS installs CoreDNS automatically and it activates once nodes join:

# CoreDNS is intentionally not declared here. EKS installs it automatically.
# It activates once self-managed nodes exist to schedule its pods.

Destroy fails on resources you cannot delete. With eks:DeleteAddon and logs:DeleteLogGroup blocked, destroy stops partway and leaves the rest standing. preserve = true handles the addons, and the log group simply should not be a Terraform resource in an account that will not let Terraform remove it.

Reading a Denial Correctly¶

Six of the seventeen failures were policy denials, and they are the ones most likely to be misdiagnosed, because a denial and a misconfiguration produce similar-looking errors.

The distinction that matters is where the denial comes from:

That last row is the one worth recognising quickly, because it ends the investigation. No IAM policy on your user, no role assumption, no different tool will change it. eksctl fails the same way terraform does, because both call the same blocked API.

The simulator answers this without waiting for another apply cycle:

aws iam simulate-principal-policy \
  --policy-source-arn "$(aws sts get-caller-identity --query Arn --output text)" \
  --action-names eks:CreateNodegroup \
  --query 'EvaluationResults[0].{Decision:EvalDecision,Org:OrganizationsDecisionDetail}'

What This Was Actually Worth¶

Seventeen distinct failures, most of them one line each once understood. The value is not the list. It is that a restricted account makes every default visible.

On an unrestricted account you never learn that the module creates roles, because it just works. You never learn that a cluster needs an explicit grant for its own creator, because the default handles it. You never learn that the addon list has an ordering dependency on nodes, because nothing waits long enough to notice.

Constraints turn defaults into decisions. That is an unpleasant way to learn a tool and a very effective one.

The module is still the right choice for most accounts, and I would reach for it again. What changed is that I now check the API response rather than the plan output whenever a setting matters enough that its absence would be silent.

Source¶

• The full runbook, phase by phase
• All seventeen errors with fixes, including the policy simulator output
• The Terraform configuration