401 and 403 Are Different Failures, and One of Them Is Unrecoverable¶

kubectl returned 403 on a cluster I had just created. Later, on the same cluster, it returned 401. Those two numbers look like degrees of the same problem and they are not: one meant my identity was known and unauthorised, the other meant it was not known at all, and the second one could not be fixed without destroying the cluster.

The reason both were possible is a setting that can only be chosen once.

Three Modes, and the Default Changed¶

EKS has two mechanisms for deciding who may talk to the API server. The aws-auth ConfigMap is the original one: a Kubernetes object mapping IAM ARNs to Kubernetes users and groups. Access entries are the newer one, managed through the EKS API rather than through an object inside the cluster.

Which of them is consulted is governed by authenticationMode:

API is the current default for new clusters. That single fact invalidates a large amount of still-accurate-looking advice, because "edit aws-auth to grant access" is the answer in most tutorials, most Stack Overflow posts, and a good deal of internal documentation written before the change.

On an API cluster, editing aws-auth succeeds. The ConfigMap accepts the edit, persists, and reads back exactly as you wrote it. Nothing consults it. It is a valid Kubernetes object that the control plane has stopped looking at, which is the same shape as several other failures I collected: the write works, the read never happens.

I chose API_AND_CONFIG_MAP deliberately. Not because I needed aws-auth, but because I wanted the recovery path that a mode with two mechanisms gives you when one of them is blocked.

403: Known, and Not Allowed¶

Error from server (Forbidden): nodes is forbidden:
User "arn:aws:iam::...:user/kk_labs_user_..." cannot list resource "nodes"
in API group "" at the cluster scope

The API server named my identity. That is the useful part of the message and it is easy to read past while looking at the word "forbidden". Authentication worked completely: the token was valid, the ARN was recognised, an access entry existed.

What was missing was authorisation. An access entry says who you are. It does not say what you may do. That comes from an associated access policy, and the association is a separate API call:

eks:AssociateAccessPolicy

In the restricted account I was working in, that call is blocked by an organisation policy. So the entry was created (that operation is permitted) and the policy could never be attached to it. Authenticated, permanently unauthorised.

403 means the fix is a permission. Something about your identity is fine and something about your rights is not.

401: Not Known At All¶

error: You must be logged in to the server
(the server has asked for the client to provide credentials)

No ARN in the message, because there is nothing to name. The API server has no record of this identity in any mechanism it consults.

This appeared after a terraform taint began replacing the cluster, successfully deleted the access entry, then aborted partway because a later deletion was also blocked. What remained was a live cluster with no access entry, no aws-auth entries, and bootstrapClusterCreatorAdminPermissions already set to false at creation.

401 means the fix is an identity. And on EKS, that is where the trap is.

The Setting That Exists Only at Creation¶

EKS grants the creating principal cluster-admin through a field in the cluster's access configuration:

access_config {
  authentication_mode                         = "API_AND_CONFIG_MAP"
  bootstrap_cluster_creator_admin_permissions = true
}

That field is create-time only. UpdateClusterConfig cannot change it. You can change the authentication mode later, you can add access entries later, but you cannot retroactively decide that the creator should have been an admin.

Which means the 401 state above is genuinely unrecoverable in that account:

• Cannot associate an access policy, that API is blocked
• Cannot update the existing access entry, that API is blocked too
• Cannot set the bootstrap field now, it is create-time only
• Cannot use aws-auth, because no identity has rights to edit it

Every door needs a key that is behind one of the other doors. The only exit is destroying the cluster and creating a new one with the field set correctly.

A Small Asymmetry Worth Knowing¶

While trying to escape the 403, I attempted to create an access entry granting the group that traditionally means "full admin":

aws eks create-access-entry --kubernetes-groups system:masters

InvalidParameterException: The kubernetes group name system:masters is invalid,
it cannot start with system:

That is not an organisation policy. EKS itself rejects the system: prefix for STANDARD access entries, because those groups are reserved for Kubernetes internals.

The asymmetry: system:masters is perfectly valid in the aws-auth ConfigMap and rejected by the access entry API. Two mechanisms for the same purpose with different rules about the same string. Advice written for one silently does not port to the other, and the error only appears at the point where you are already trying to recover from something else.

The Two-Call Problem¶

Underneath all of this is a structural detail that explains why the failure modes are so awkward: granting an identity access to EKS is two API calls, not one, and they can be permitted independently.

eks:CreateAccessEntry        <- who you are
eks:AssociateAccessPolicy    <- what you may do

In my case the first was allowed and the second was not, which produces a state that has no name in most documentation: an identity the cluster recognises and grants nothing to. Every tool that wraps this presents it as a single operation, so when only half succeeds the abstraction has nowhere to put the result.

The community Terraform module offers enable_cluster_creator_admin_permissions = true, which performs exactly those two calls in sequence. When the second is blocked, the apply fails after the first has already succeeded, leaving the cluster in the half-granted state and Terraform's state file believing the resource does not exist.

bootstrap_cluster_creator_admin_permissions is a different mechanism entirely and that is why it works. It is not a call you make against the cluster; it is a field the cluster is created with, and EKS establishes the admin entry internally. No AssociateAccessPolicy is ever invoked, so there is nothing for a policy to block.

That distinction is worth carrying: a field set at creation and an API call made afterwards are not interchangeable, even when they produce the same end state. One can be denied and the other cannot.

What Made This Visible¶

None of the above is a defect. The account I was in blocks four EKS API calls, and every one of those blocks converted an ordinary "fix it afterwards" into "you cannot fix it afterwards."

On an unrestricted account you would never learn any of this. enable_cluster_creator_admin_permissions = true in the community module works, the association call succeeds, and you never discover that it is two operations rather than one, or that only the second is blocked, or that the fallback the module is supposed to use gets silently dropped in a common configuration.

Constraints did not create these behaviours. They made them observable.

What To Check¶

Before creating a cluster:

# What mode will this cluster use, and does the creator get admin?
# Both are create-time decisions. Decide them deliberately.

After creating one, verify against the API rather than the plan:

aws eks describe-cluster --name <cluster> --query 'cluster.accessConfig'
# Expect authenticationMode AND bootstrapClusterCreatorAdminPermissions

aws eks list-access-entries --cluster-name <cluster>
# Expect an entry for the identity that ran the apply, not just the service role

And when kubectl fails, read the number before reading the text:

The two failures look adjacent and lead to opposite investigations. I spent time hunting for missing RBAC while holding a 401, which is a question about authentication asked in the authorisation layer, and nothing there was ever going to answer it.

Source¶

• The full challenge log, where these are errors 8 through 12 of seventeen
• The eksctl provisioning path, which carries the authentication mode reference table
• The Terraform configuration that ended up working