Skip to content

A Pod With AWS Permissions and No Credentials

Two services in this cluster talk to AWS. Neither has an access key, a Secret, or anything in a values file that could leak one. They assume an IAM role through their service account, and the SDK inside them picks it up without being told.

That mechanism is IRSA, and the part worth writing down is not how to enable it. It is where the boundary sits: which object owns the binding, and why keeping it out of the Helm chart is what lets the same chart still run on a laptop.

The Problem IRSA Solves

A pod needs to call DynamoDB. The options before IRSA were all bad in the same way.

Put an access key in a Secret and you own a long-lived credential: rotation, blast radius, and something that appears in kubectl get secret -o yaml for anyone with read access to the namespace. Attach a policy to the node's instance role and every pod on that node inherits it, so a compromised sidecar gets your database.

IRSA replaces both. The service account carries an annotation naming an IAM role. A webhook injects a projected token into the pod, and the AWS SDK exchanges that token for temporary credentials via STS.

The result is a per-workload identity with no stored secret and automatic expiry. Nothing to rotate, because nothing persists.

Two Bindings, Created Outside the Chart

Both are one eksctl command:

eksctl create iamserviceaccount \
  --cluster $CLUSTER_NAME \
  --namespace cart \
  --name cart \
  --attach-policy-arn arn:aws:iam::$ACCOUNT_ID:policy/cart-dynamo \
  --role-name dynamo-table-access-for-cart \
  --approve \
  --override-existing-serviceaccounts

That does three things at once: creates the IAM role with a trust policy scoped to the cluster's OIDC provider, attaches the permission policy, and creates or annotates the Kubernetes service account.

The orders service gets the same treatment against an SQS policy. And then the Helm chart, in both cases, is told to keep its hands off:

serviceAccount:
  create: false
  name: "cart"

Why That Split Is the Whole Design

It would be tidier to put the IRSA annotation in the chart. One helm install and everything exists.

It would also make the chart un-runnable anywhere except EKS.

An IRSA annotation is an AWS concept: an account ID, a role ARN, an OIDC provider that exists only on that cluster. Put it in the chart and every bare metal deployment, every kind cluster, every laptop gets a service account annotated with a role that does not exist. That is harmless in the sense that nothing breaks, and it is a chart that now knows about one cloud provider.

Keeping it out preserves a clean contract:

  • The chart declares that the workload uses a service account of a given name.
  • The platform decides what that service account is allowed to do.

On EKS the platform grants it an IAM role. On bare metal the platform grants it nothing, and the same chart deploys against an in-cluster database instead, selected by a values overlay rather than a fork. Neither deployment needs a conditional.

--override-existing-serviceaccounts is what makes the ordering forgiving. Deploy the chart first and the service account already exists; the flag lets eksctl annotate it in place rather than failing on a conflict.

The Pipeline Past the Queue

The orders service publishes to SQS and stops. Everything downstream is outside Kubernetes entirely:

orders pod  --(IRSA)-->  SQS: orders-events
                            |
                            v
                        Lambda  (event source mapping)
                            |
                            v
                        SNS: order-notifications
                            |
                            v
                        email subscriber

The Lambda is nine lines:

import boto3
sns = boto3.client('sns')
TOPIC_ARN = "arn:aws:sns:...:order-notifications"

def lambda_handler(event, context):
    for record in event['Records']:
        sns.publish(TopicArn=TOPIC_ARN,
                    Message=f"Order confirmed: {record['body']}")

The architectural point is the boundary. The cluster's responsibility ends at "the event was published". It does not know that a Lambda consumes the queue, that SNS fans out, or that a human gets an email. Adding a second consumer, swapping email for Slack, or deleting the notification path entirely are all changes that never touch a manifest.

That is the thing a queue buys, and it is easy to state and easy to accidentally give away by having the producer know about the consumer.

Three Things I Would Not Ship

Being straight about the parts that are lab-shaped.

The SQS policy is over-permissioned. It grants sqs:CreateQueue alongside SendMessage, GetQueueAttributes and GetQueueUrl:

"Action": [
  "sqs:CreateQueue",
  "sqs:SendMessage",
  "sqs:GetQueueAttributes",
  "sqs:GetQueueUrl"
]

A service that publishes events has no business creating queues. That permission is there because some SDK paths create-if-absent on startup and it was easier to grant than to verify. It should be SendMessage plus GetQueueUrl, and the queue should be created by whatever provisions infrastructure.

The resource scoping is right, at least: a single queue ARN rather than *. The scope is correct and the verb list is lazy, which is the more common failure of the two.

The Lambda execution role has AdministratorAccess. In the account I was working in, iam:PassRole and iam:PutRolePolicy were blocked for the CLI user, so the function and its event source mapping were created through the Console and the role was given a policy broad enough to stop fighting it.

That is a lab workaround and nothing else. The correct policy is AWSLambdaBasicExecutionRole for logging plus sns:Publish scoped to the one topic ARN. I am recording it rather than quietly omitting it, because a blocked permission producing an over-broad grant is exactly how real environments end up with AdministratorAccess on things that need three actions.

The Lambda was created by hand. Everything else in this project is a command or a manifest. That one is console clicks, for the same permission reason. It is the least reproducible part of the system and the first thing I would move into IaC.

Verifying It Without Guessing

IRSA fails quietly when it fails, so it is worth confirming each layer rather than deploying and hoping.

The annotation reached the service account:

kubectl get sa orders -n orders -o yaml | grep eks.amazonaws.com/role-arn

The token was projected into the pod, which proves the webhook fired:

kubectl exec -it deploy/orders -n orders -- env | grep AWS
# AWS_ROLE_ARN=...
# AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

Those two variables are the whole mechanism. If they are absent, the pod started before the annotation existed and needs a restart; the webhook only injects at admission.

And the role can actually assume:

kubectl exec -it deploy/orders -n orders -- \
  aws sts get-caller-identity
# Arn should be the assumed role, not the node role

If that returns the node's instance role rather than the assumed role, IRSA is not working and the pod is quietly using node permissions instead. That is the failure worth catching, because it often works, by accident, for exactly as long as the node role is over-permissioned.

The Trust Policy Is the Part Worth Understanding

eksctl writes the trust policy for you, which is convenient and means most people never read it. It is the only piece that makes any of this safe:

{
  "Effect": "Allow",
  "Principal": {
    "Federated": "arn:aws:iam::<account>:oidc-provider/oidc.eks.<region>.amazonaws.com/id/<id>"
  },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "oidc.eks.<region>.amazonaws.com/id/<id>:sub":
        "system:serviceaccount:orders:orders"
    }
  }
}

Two things are doing the work.

The principal is the cluster's OIDC provider, not an account or a user. Only tokens signed by that specific cluster can be exchanged, so a token from any other cluster is rejected before permissions are considered.

The condition pins the exact namespace and service account name. Without it, any service account in the cluster could assume the role, which would give you node-role semantics with extra steps. system:serviceaccount:orders:orders means namespace orders, service account orders, and nothing else.

That is why a pod in a different namespace cannot borrow the identity even if it guesses the role ARN, and it is why the OIDC provider must exist before any of this works. In a restricted account where OIDC association is itself a blocked or deferred operation, IRSA silently does nothing: the annotation is present, the token is projected, and the exchange fails at STS.

What Transfers

The reusable idea is not IRSA specifically. It is that identity belongs to the platform layer and reference belongs to the application layer.

The chart says "I use a service account called orders". The platform decides whether that name maps to an IAM role, a bare service account, or something else entirely. Neither has to know how the other implements its half, which is why one chart covers a laptop and a cloud account without a conditional.

The same shape applies to EKS access control generally: what a workload is and what it may do are separate questions, and conflating them is what produces credentials in Secrets and roles on nodes.

Source

Related