Skip to content

Zero Compute, and Still a Dozen Decisions

Seven days of globally distributed static hosting on AWS cost me $1.45 across CloudFront, three S3 buckets, CloudTrail and KMS. There is no compute in it at all. No Lambda, no container, no instance.

"Static site on S3" gets treated as the trivial deployment, and the hosting genuinely is trivial. Everything around it is not, and the decisions turn out to be about who can read the bucket, which key encrypted the object, and what happens when a path does not exist.

The build this served

The payload is the production dist/ output of my portfolio site. The complete terminal session is the raw execution log, every command as it ran.

The Shape

Browser
  |
Cloudflare DNS (CNAME, DNS only)
  |
CloudFront  <- ACM certificate, us-east-1
  |          <- OAC, SigV4 signed requests
S3 primary (us-east-1, private, SSE-KMS, versioned)
  |-> CloudTrail + S3 access logs -> logging bucket
  |
  v  Cross-Region Replication
S3 replica (us-west-2, private, SSE-KMS with its own key)

Three buckets, two KMS keys, one distribution. The origin bucket has public access blocked at the account level and no bucket ACL that would let anyone in. The only principal that can read it is a specific CloudFront distribution.

Origin Access Control, Not a Public Bucket

The old pattern was a public bucket with a website endpoint. The slightly less old pattern was Origin Access Identity. Both are still widely published and neither is the current answer.

Origin Access Control signs CloudFront's requests to S3 with SigV4. That matters for a reason beyond being newer: OAI cannot read SSE-KMS encrypted objects. If you want customer-managed encryption at rest, OAC is not a preference, it is the only one of the two that works.

The bucket policy is where the actual security boundary lives:

{
  "Sid": "AllowCloudFrontServicePrincipal",
  "Effect": "Allow",
  "Principal": { "Service": "cloudfront.amazonaws.com" },
  "Action": "s3:GetObject",
  "Resource": "arn:aws:s3:::${PRIMARY_BUCKET}/*",
  "Condition": {
    "StringEquals": {
      "AWS:SourceArn": "arn:aws:cloudfront::${ACCOUNT_ID}:distribution/${CF_DISTRIBUTION_ID}"
    }
  }
}

The condition is the whole policy

Without the AWS:SourceArn condition, the principal is the CloudFront service, which means any CloudFront distribution in any AWS account could read the bucket. The policy would look correct, pass review, and be globally readable through a distribution someone else created. This is the single line in the build I would check first on someone else's setup.

There is also a field that exists purely to say it is not being used:

"S3OriginConfig": { "OriginAccessIdentity": "" }

It is required even with OAC, because it is how CloudFront distinguishes an S3 REST origin from a custom origin. Omit it and the distribution fails validation.

Two Keys, Because Keys Are Regional

A KMS key in us-east-1 cannot decrypt an object in us-west-2. That single fact determines the whole replication design.

So there are two customer-managed keys, one per region. During replication S3 decrypts at the source with the primary key and re-encrypts at the destination with the replica key. Each region ends up holding its own key material, which is the property you actually want from a disaster recovery copy: a compromised or deleted key in one region does not take the other region with it.

The replication IAM role therefore needs four permissions that are easy to under-scope: kms:Decrypt against the source key ARN, kms:Encrypt against the destination key ARN, and kms:DescribeKey on both.

The field whose absence means "skip"

One field in the replication rule does not fail loudly when it is missing:

SourceSelectionCriteria.SseKmsEncryptedObjects: Enabled

Leave it unset and S3 does not replicate KMS-encrypted objects at all. Not an error, not a partial copy. The rule is valid, the role is correct, the replica bucket is simply empty, and every object you have is encrypted so every object is skipped.

This is one of a pattern I kept hitting across five projects: an unset field whose default is a real behaviour rather than a neutral one. You cannot spot it by re-reading your configuration, because the thing to notice is not there.

Bucket Keys, and Where the $1.45 Comes From

SSE-KMS charges per GenerateDataKey call. Naively that is one KMS API call per object write, which for a site with a few hundred build artifacts is fine and for anything busier is the dominant line item.

S3 Bucket Keys change the unit. S3 requests a short-lived bucket-level key from KMS and derives per-object keys from it, cutting GenerateDataKey calls by up to 99 percent. It is one flag on the bucket's default encryption configuration and it is most of the reason the bill is small.

Worth knowing: enabling it is a change to the default encryption configuration, which applies to objects written afterwards. It does not retro-fit existing objects, so turning it on is something to do before the first upload rather than after noticing the cost.

The 403 That Is Actually a 404

This is the part that catches people, and it is a direct consequence of doing the security correctly.

A single-page application uses client-side routing. A request for /projects has no corresponding S3 key. On a public bucket with a website endpoint, S3 returns 404. On a private bucket, S3 returns 403, because the caller is not authorised to know whether the object exists. Hiding existence is the correct behaviour for a private bucket.

So the CloudFront custom error response has to map 403, not 404:

"CustomErrorResponses": {
  "Items": [{
    "ErrorCode": 403,
    "ResponsePagePath": "/index.html",
    "ResponseCode": "200",
    "ErrorCachingMinTTL": 0
  }]
}

Map 404 instead and the home page works perfectly while every deep link and every browser refresh on a sub-route returns an error page. It is a partial failure that looks like a routing bug in the application.

ErrorCachingMinTTL: 0 matters too. Without it CloudFront caches the error response, so a genuinely missing file stays missing at the edge after you upload it.

Four Small Distribution Choices

The rest of the distribution config is unremarkable individually and worth stating together:

Setting Value Why
Cache policy CachingOptimized (AWS-managed) 24h TTL, Gzip and Brotli, forwards no headers, cookies or query strings. Correct for a static origin
Price class PriceClass_100 North America and Europe only. Avoids edge charges in regions with negligible traffic
HTTP version http2and3 HTTP/3 over QUIC cuts connection latency on mobile and lossy networks
Minimum TLS TLSv1.2_2021 With sni-only, which is the standard pairing for a custom domain

The ACM certificate has a constraint that is not a choice: it must be issued in us-east-1 regardless of where anything else lives. CloudFront is a global service with a control plane in North Virginia. A certificate in the region your buckets are in will not attach.

DNS Only, Not Proxied

The domain is on Cloudflare, so the record is a CNAME to the distribution domain with the proxy turned off.

Proxying puts Cloudflare in front of CloudFront, which terminates TLS at Cloudflare and presents Cloudflare's IPs to CloudFront. That breaks SNI-based certificate matching and makes geo-restriction meaningless, since CloudFront now sees one origin rather than the client. Two CDNs stacked is not twice the CDN.

Audit Logging, and the Eleventh Silent Failure

Two log streams answer different questions. CloudTrail records API calls; S3 Server Access Logs record raw HTTP requests, which is what you want for cache-miss debugging.

CloudTrail also needs telling that object access is interesting. By default it logs management events only, so bucket creation is recorded and every GetObject is not. Data events are a separate put-event-selectors call.

And then the one I did not expect. Setting TargetBucket on the origin bucket is not sufficient:

{
  "Sid": "S3ServerAccessLogsWrite",
  "Principal": { "Service": "logging.s3.amazonaws.com" },
  "Action": "s3:PutObject",
  "Resource": "arn:aws:s3:::${LOG_BUCKET}/s3-access-logs/*",
  "Condition": { "StringEquals": { "aws:SourceAccount": "${ACCOUNT_ID}" } }
}

Without that statement on the logging bucket, put-bucket-logging succeeds, the configuration reads back correctly, and no logs are ever delivered. The permission is needed on the destination and the call that appears to configure it is made against the source.

That is the second instance of the same pattern in this one project, and I only noticed because I went looking for logs that should have existed. The ten I collected across five projects were never going to be exhaustive.

What Zero Compute Actually Means

It means there is no process to patch, no runtime to upgrade, no autoscaling policy, and no instance to be compromised. That is a genuine reduction in operational surface and it is why the bill is $1.45.

It does not mean there is nothing to design. The decisions moved rather than disappeared: from "how do I run this" to "who is allowed to read this, which key protects it, what happens on a path that does not exist, and how do I know any of it is working."

Those are better problems to have. They are also the ones that stay wrong quietly if you do not check them, which is why the next thing I did was write down six verifications rather than trust that a working home page meant a working deployment: HTTPS on the custom domain, direct S3 access returning 403, CloudTrail events actually arriving, replicated object count matching, Bucket Key enabled on both buckets, and a pre-signed URL proving the bucket really is private. Every one of those has failed for me at least once while the site itself looked fine.

Source


Related