Seventeen Errors, and the Ones That Were Caused by the Fix¶

Provisioning EKS in a restricted AWS account produced seventeen distinct failures. What made it hard was not the count. It was that several of them only existed because of the fix for the previous one.

Three of the chains are worth walking through, because the pattern they share is the actual lesson: in a constrained environment, the correct move is often to back out rather than push forward.

Chain One: The Log Group¶

Error 1. terraform destroy fails:

Error: deleting CloudWatch Logs Log Group (/aws/eks/microservices-demo-eks/cluster):
api error AccessDeniedException: User: ... is not authorized to perform: logs:DeleteLogGroup

The EKS module creates a log group for control-plane logging. Deleting it is blocked by policy. The obvious fix is to make Terraform forget it exists:

terraform state rm 'module.eks.aws_cloudwatch_log_group.this[0]'
terraform destroy

Destroy completes. Problem solved.

Error 2, on the next apply:

Error: creating CloudWatch Logs Log Group: ResourceAlreadyExistsException

Of course. The log group was removed from state, not from AWS. Terraform now believes it does not exist and tries to create it, and AWS disagrees.

terraform import 'module.eks.aws_cloudwatch_log_group.this[0]' \
  '/aws/eks/microservices-demo-eks/cluster'

Two errors, one underlying fact: state rm changes what Terraform believes, not what exists. Every use of it creates a future divergence, and the second error is that divergence arriving.

The real fix is upstream of both: stop managing the log group at all. create_cloudwatch_log_group = false, and let EKS own a resource this account will not let Terraform delete.

Chain Two: Encryption, Three Errors Deep¶

This one is the clearest example of a fix producing its successor.

Error 4. The module enables KMS cluster encryption by default, and creating the accompanying IAM policy needs iam:TagPolicy, which is blocked:

api error AccessDenied: ... is not authorized to perform: iam:TagPolicy

Fine, disable encryption. So I set the variables I knew from the module's older documentation.

Error 5:

Error: Unsupported argument
  58:   cluster_encryption_config = {}
An argument named "cluster_encryption_config" is not expected here.

The variable names changed between major versions. cluster_encryption_config and attach_cluster_encryption_policy are v18 and v19 names; v21 uses encryption_config and attach_encryption_policy.

The fix is to stop trusting documentation and read the module that is actually downloaded:

grep -i 'variable.*kms\|variable.*encrypt' .terraform/modules/eks/variables.tf

That command is worth keeping. .terraform/modules/ holds the exact source Terraform will use, and it is the only authority on what the interface accepts.

Error 6, with the correct names:

Error: Missing required argument
The argument "encryption_config.0.provider.0.key_arn" is required,
but no definition was found.

I had written encryption_config = {}, reading an empty map as "nothing". The module reads it differently:

enable_encryption_config = var.encryption_config != null && ...

{} is not null. The condition is true, so the module emits an encryption_config block onto the cluster resource, and that block requires a key ARN it does not have.

encryption_config = null   # disables it
# encryption_config = {}   # enables it, empty, and therefore invalid

Chain Three: The Unrecoverable One¶

The access-control chain is five errors long and ends somewhere you cannot get back from.

Error 12 is the one worth understanding. The taint began a replacement, successfully deleted the access entry, then aborted because a later deletion was itself blocked. What remained was a live cluster with no access entry, no aws-auth entries, and bootstrapClusterCreatorAdminPermissions fixed at false since creation.

That field is create-time only. There is no path back, and the full reasoning is its own post.

The chain's lesson is not about EKS. It is that each attempted fix consumed a resource that the next attempt needed. Tainting to recreate is a reasonable escalation right up until the teardown half succeeds and the rebuild half does not, at which point you are worse off than when you started.

The Shape¶

Grouping all seventeen by what actually went wrong:

The fourth row is a quarter of the total, and it is the row that is invisible when you read a list of errors as independent items.

Two More Worth Knowing¶

Staged apply for a count that cannot be resolved. Error 3 is the module's managed node group submodule using a data source whose count depends on an attribute that does not exist until after apply:

The "count" value depends on resource attributes that cannot be determined until apply.

Terraform needs the count at plan time. The workaround is to apply in stages so the dependency exists by the time it is needed:

terraform apply -target=module.eks.aws_eks_cluster.this[0]
terraform apply -target=module.eks
terraform apply

-target is usually a smell. Here it is the documented escape hatch for a plan-time dependency, and the better fix removed the submodule entirely, since managed node groups were blocked anyway.

State outliving the account. Error 15 comes from a time-boxed lab, and the signal is unusually clear:

~ aws_account_id = "<previous-account-id>" -> "<current-account-id>"

A plan showing the account ID changing means the state file describes a different account than the credentials point at. Every resource in it is unreachable. Deleting terraform.tfstate is correct here specifically because the resources it references were destroyed with the previous session, which is exactly the condition under which deleting state is safe and never otherwise.

The Errors That Were Not Terraform's¶

Three more failures in the same account came from eksctl and CloudFormation. They are outside the seventeen because they are not Terraform errors, and they belong here anyway: two of the three are shapes already above, arriving through a different tool.

Switching tools to escape a policy denial is a category error. eksctl fails on a blocked action in exactly the words Terraform does:

User: arn:aws:iam::...:user/kk_labs_user_XXXXXX is not authorized to
perform: eks:CreateNodegroup on resource: arn:aws:eks:...:cluster/<name>

A managed node group is one API call and the policy denies that call, so eksctl, Terraform, the CLI and the Console are four clients of one refusal. I lost time to this anyway, on the reasonable-sounding theory that a different tool might do something different.

Bundling operations bundles their failure modes. eksctl create cluster creates a cluster, IAM roles, an OIDC provider, a VPC and the CloudFormation stacks that manage them. Every one is a call that can be blocked, and one blocked call aborts the command with several resources already created. The working cluster.yaml is therefore mostly a list of things turned off:

iam:
  serviceRoleARN: arn:aws:iam::<account-id>:role/eksClusterRole
  withOIDC: false          # association triggers blocked calls during creation

accessConfig:
  authenticationMode: API_AND_CONFIG_MAP

managedNodeGroups: []      # eks:CreateNodegroup is blocked

autoModeConfig:
  enabled: false

withOIDC: false is the one I would not have predicted, since OIDC association is exactly what IRSA needs. Doing it during creation triggers permission failures that abort the whole operation. Doing it afterwards, as its own command, works:

eksctl utils associate-iam-oidc-provider --cluster "$CLUSTER_NAME" --approve

Same end state, different failure surface. That is chains one and two in a different costume: the operation was not the problem, being welded to four others was.

A parameter that was accepted and meant nothing. With managed node groups blocked, the workers come from the AWS-published node template as a CloudFormation stack. Its AuthenticationMode parameter takes display strings rather than API enum values. The API says API_AND_CONFIG_MAP; the template wants the human-readable form. Passing the enum produces a stack that creates successfully and nodes that never join, with no error anywhere.

That is not a cascade, it is the silent-failure shape: a value that was present, accepted, and carried no usable information. Nodes reaching Ready about two minutes after the stack completes is the expected timing, so a healthy stack plus a healthy cluster plus no nodes is almost always the join path rather than the infrastructure.

That cluster.yaml line also deserves separating out, because it is create-time only in the same way the bootstrap admin field is. Newer clusters default to API, under which aws-auth is ignored entirely. API_AND_CONFIG_MAP keeps both join paths open, and in an account where the access-entry APIs are partly blocked, having the ConfigMap as a fallback is the difference between a recoverable cluster and chain three.

What I Would Do Differently¶

Simulate before configuring. Six of the seventeen are policy denials, and each one is answerable in seconds:

aws iam simulate-principal-policy \
  --policy-source-arn "$(aws sts get-caller-identity --query Arn --output text)" \
  --action-names eks:CreateNodegroup eks:AssociateAccessPolicy iam:TagPolicy \
  --query 'EvaluationResults[].{Action:EvalActionName,Decision:EvalDecision,Org:OrganizationsDecisionDetail}'

Running that against the full action list at the start would have converted six discovered failures into six known constraints before writing any HCL. The field to read is AllowedByOrganizations, and a False there ends the conversation: the denial sits above the account, so no IAM policy, no assumed role and no alternative client will move it.

Expect the tool boundary to be a runbook, and treat that as debt. This cluster ended up built by three tools. Terraform owns the VPC, the IAM roles and the control plane; CloudFormation owns the workers, because managed node groups are impossible here; eksctl owns the OIDC provider and the IRSA service accounts, because both are one-off operations against a cluster that already exists. That split was not a preference, it was what each tool could actually complete. The cost is real: terraform destroy removes one third of it, the state file describes a cluster and knows nothing about the nodes attached to that cluster, and the order that makes the whole thing work lives in a document rather than in a dependency graph.

Read the downloaded module, not the docs. Errors 5 and 6 both come from trusting a version's documentation over its source. .terraform/modules/ is right there.

Treat state rm and taint as one-way doors. Both were reasonable moves that created worse states. In an environment where teardown operations can be blocked, any command whose recovery path depends on being able to delete something is a gamble.

Prefer backing out to pushing through. The encryption chain resolved when I stopped trying to disable a feature through the module and started asking why it was enabled. The access chain resolved when I stopped fighting the module and used the raw resource. In both cases the fix was upstream of where I was working.

The Destroy Path Is Part of the Configuration¶

Three of the seventeen only appear at teardown, and they share a cause that is easy to miss while everything is being built: this account can create resources it cannot delete.

That asymmetry does not exist on a normal account, where anything you can create you can remove. Here, logs:DeleteLogGroup and eks:DeleteAddon are both blocked, so any resource of those types that Terraform manages becomes a permanent obstacle to terraform destroy.

The fix is structural rather than reactive:

resource "aws_eks_addon" "example" {
  # ...
  preserve = true    # leave it in AWS on destroy rather than failing
}

And for the log group, not managing it at all. Both amount to the same principle: do not let Terraform own a resource it will not be permitted to delete.

This matters more than it sounds in a time-boxed environment. A failed destroy leaves a state file describing resources that will vanish when the session expires, which is exactly the setup for error 15 on the next session. The teardown failure and the stale-state failure are the same chain viewed a day apart.

Worth generalising: if you are working anywhere with asymmetric permissions, walk the destroy path before building. It is the half of the lifecycle nobody rehearses and the half that determines whether the second attempt starts clean.

Source¶

• All seventeen with full error text and fixes
• The main provisioning runbook
• The eksctl path, including the node group decision matrix and the authentication mode reference
• The one that produces no error at all, which is error 16
• The Terraform that ended up working