A Site That Loads Proves Almost Nothing¶

The home page rendering proves CloudFront serves content. It does not prove the bucket is private, that replication runs, that audit logs are recorded, that encryption is uniform, or that the cache is working. All five can be wrong on a site that looks perfect, and four of the five fail without producing an error.

This is the verification pass for that architecture, starting with the status code that means four different things.

Four Different 403s, and Only One Is Good News¶

A private S3 origin behind CloudFront returns 403 Forbidden in four unrelated situations. The status code does not distinguish between them, and working out which one you have is most of the operational skill in this setup.

The first is a test you should run deliberately:

curl -I https://$PRIMARY_BUCKET.s3.us-east-1.amazonaws.com/index.html
# 403 Forbidden

If that returns 200, the bucket is public and the entire access-control design is not doing anything. A 403 here is a passing test, which makes it the only one of the four you want to see.

Telling the Other Three Apart¶

The distinguishing question is which URLs fail, not what the error says.

Root fails, deep links work. https://example.com returns 403 while https://example.com/about.html returns 200. CloudFront has no DefaultRootObject, so a request for / becomes a request for an empty key, which does not exist, which on a private bucket is 403 rather than 404.

aws cloudfront get-distribution-config --id $CF_DISTRIBUTION_ID > /tmp/cf.json
ETAG=$(jq -r '.ETag' /tmp/cf.json)
jq '.DistributionConfig.DefaultRootObject = "index.html" | .DistributionConfig' /tmp/cf.json > /tmp/upd.json
aws cloudfront update-distribution --id $CF_DISTRIBUTION_ID \
  --if-match $ETAG --distribution-config file:///tmp/upd.json

Everything fails, right after setting up OAC. The distribution deploys, the OAC exists, and every request is refused. The bucket policy was never applied, or was applied with the wrong distribution ID.

The policy grants the CloudFront service principal read access, conditioned on AWS:SourceArn matching one specific distribution. Creating the distribution and writing the policy are two operations, and the policy needs the distribution ID that only exists after the first one completes. Doing them out of order produces a policy scoped to nothing.

Worth checking the KMS key policy at the same time: with SSE-KMS, CloudFront also needs kms:Decrypt, and its absence produces the identical symptom.

A path that is not an object. For a single-page application this is every client-side route. It is handled by mapping 403 to index.html rather than by fixing anything, and it is the reason the custom error response maps 403 and not 404.

The Six Checks¶

Verification that a working home page does not give you.

1. HTTPS on the custom domain, and cache behaviour.

curl -I https://ibtisam.qzz.io
# HTTP/2 200
# x-cache: Hit from cloudfront     (after warm-up)

x-cache is the part worth reading. Miss from cloudfront on every request means the cache policy is not doing what you think, and the origin is taking load it should not.

2. Direct S3 access is refused, as above.

3. CloudTrail is actually receiving events.

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=$PRIMARY_BUCKET \
  --max-results 5

An empty result on a bucket you have been reading means the trail exists and data events were never enabled, since management events alone do not record object access.

4. Replication counts match.

echo "Primary: $(aws s3 ls s3://$PRIMARY_BUCKET --recursive | wc -l)"
echo "Replica: $(aws s3 ls s3://$REPLICA_BUCKET --recursive | wc -l)"

A count of zero on the replica has one dominant cause, and it gets its own section below because it is the quietest failure in the build.

5. Bucket Keys are on, on both buckets.

aws s3api get-bucket-encryption --bucket $PRIMARY_BUCKET \
  --query 'ServerSideEncryptionConfiguration.Rules[0].BucketKeyEnabled'
# true

Checking the replica separately matters because it is a different bucket with its own encryption configuration, and it is easy to set this on the primary only.

6. A pre-signed URL works.

aws s3 presign s3://$PRIMARY_BUCKET/index.html --expires-in 300

This is the positive proof that pairs with check 2. Check 2 shows nobody can read the bucket directly. Check 6 shows that authorised, time-limited access still works, which confirms the bucket is private rather than broken.

Check 4 in Full: the Replica That Stays Empty¶

Cross-Region Replication with SSE-KMS has one failure mode that dominates all the others. The replication rule is valid, the IAM role is correct, every object uploads successfully, and the replica bucket stays empty. No error, no partial copy, no metric that goes red.

Why there are two keys¶

KMS keys are regional. A key in us-east-1 cannot decrypt an object in us-west-2, and no amount of policy changes that. It is not a permissions question.

So replication across regions with customer-managed encryption is not a copy. It is a decrypt at the source and a re-encrypt at the destination, with different key material on each side.

S3 primary (us-east-1)          S3 replica (us-west-2)
  encrypted with key A            encrypted with key B
         |                                 ^
         |  decrypt with A                 |  encrypt with B
         +---------- replication role -----+

That is more work than a copy, and it is the property you want from a disaster recovery target. If both regions shared one key, deleting it, scheduling it for deletion, or having it compromised would take out the primary and the backup together.

The field that means "skip"¶

"SourceSelectionCriteria": {
  "SseKmsEncryptedObjects": { "Status": "Enabled" }
}

Without it, S3 does not replicate KMS-encrypted objects. Not "fails to", not "retries and gives up". It does not select them for replication in the first place.

If your bucket uses SSE-KMS by default, which is the entire reason you set up two keys, then every object you have is excluded. The rule is enabled, the role is assumable, the destination is writable, and the selection criteria quietly matched nothing.

There is a second, smaller version of the same trap in the same call. Any rule that includes Filter, even an empty {}, also requires Priority. Omit it and put-bucket-replication fails with MalformedXML, which at least has the decency to be loud.

The role is five statements, not two¶

The obvious two are read from the source and write to the destination. The KMS pair is where under-scoping usually happens:

{
  "Sid": "AllowKMSDecryptSource",
  "Effect": "Allow",
  "Action": ["kms:Decrypt", "kms:GenerateDataKey", "kms:DescribeKey"],
  "Resource": "${KMS_KEY_ARN1}"
},
{
  "Sid": "AllowKMSEncryptDestination",
  "Effect": "Allow",
  "Action": ["kms:Encrypt", "kms:GenerateDataKey", "kms:DescribeKey"],
  "Resource": "${KMS_KEY_ARN2}"
}

kms:DescribeKey on both is the one people leave out, because nothing about the operation suggests it is needed to move an object. kms:GenerateDataKey appears on both sides because the re-encrypt at the destination needs a fresh data key, not the source's.

The other three statements split by resource type in a way that is easy to get subtly wrong:

• s3:GetReplicationConfiguration and s3:ListBucket against the bucket ARN
• s3:GetObjectVersionForReplication, GetObjectVersionAcl, GetObjectVersionTagging against the object ARN with /*
• s3:ReplicateObject, ReplicateDelete, ReplicateTags against the destination object ARN

Everything on the source side is a version operation. Replication works on object versions, which is why versioning is a hard prerequisite on both buckets rather than a recommendation.

Two key policies nobody mentions¶

The IAM role is the part every guide covers. The key policies are the part that gets left out, and replication needs both of them changed.

A KMS key policy is not an IAM policy. IAM says what a principal may attempt; the key policy says who the key will answer to at all. Both must agree, and by default a newly created key answers only to the account root.

{
  "Sid": "AllowS3ReplicationUse",
  "Effect": "Allow",
  "Principal": { "Service": "s3.amazonaws.com" },
  "Action": ["kms:Decrypt", "kms:GenerateDataKey", "kms:DescribeKey"],
  "Resource": "*",
  "Condition": {
    "StringEquals": { "aws:SourceAccount": "${ACCOUNT_ID}" }
  }
}

The source key grants decrypt, the destination key grants encrypt, and the aws:SourceAccount condition on both is what stops the statement from being a standing invitation to any account whose S3 happens to point at your key ARN.

The primary key policy has a second consumer that is easy to forget. CloudFront needs kms:Decrypt on it to serve objects through Origin Access Control, which is a separate statement with a different principal, and omitting it produces the third kind of 403 on a bucket that replicates perfectly.

Order matters more than it should¶

Replication applies to objects written after the rule becomes active. It does not backfill.

The sequence is: create both buckets, enable versioning on both, create both keys, create the role, enable replication, and only then upload. Doing it in the intuitive order, upload the site and then set up disaster recovery, leaves the replica empty until the next deploy touches every file, which for a static site build may be never for the unchanged assets.

The upload flags matter for the same reason:

aws s3 sync dist/ s3://$PRIMARY_BUCKET \
  --sse aws:kms \
  --sse-kms-key-id $KMS_KEY_ID1 \
  --delete

--sse aws:kms enforces KMS per request rather than relying on the bucket default, so a caller that omits the header cannot silently land an SSE-S3 object in a bucket you believe is uniformly CMK-encrypted. --sse-kms-key-id pins each object to a specific key, and without it the per-request header may not carry the key ID, which surfaces much later as CloudFront failing kms:Decrypt. --delete creates delete markers on a versioned bucket rather than destroying anything, which is why DeleteMarkerReplication is enabled on the rule: without it the primary and replica diverge on deletions while agreeing on writes.

The direct answer¶

aws s3api head-object \
  --bucket $PRIMARY_BUCKET \
  --key index.html \
  --query ReplicationStatus
# COMPLETED

PENDING means it is in flight. FAILED means a permission is wrong and is at least honest. No ReplicationStatus field at all is the silent case: the object was never selected.

If the replica is empty, the diagnostic order that works:

1. Is SseKmsEncryptedObjects set to Enabled in the rule
2. Does the role have kms:Decrypt on the source key ARN and kms:Encrypt on the destination key ARN, both with kms:DescribeKey
3. Do both KMS key policies allow the s3.amazonaws.com service principal, with the aws:SourceAccount condition
4. Was the object uploaded after the rule became active

The order is deliberate. Step 1 is one line to read and explains complete emptiness. Steps 2 and 3 explain partial or failed replication. Step 4 explains why old objects are missing and new ones arrive.

The One That Takes Ten Minutes to Diagnose¶

ACM certificates stuck at PENDING_VALIDATION have two causes and one of them is subtle.

The obvious one is that the validation CNAME was never added. The subtle one is that it was added and Cloudflare is proxying it. An orange-cloud record is answered by Cloudflare rather than resolving to the value you set, which can prevent ACM's lookup from seeing what it needs.

dig _<acm-token>.ibtisam.qzz.io CNAME +short

If that returns the expected ACM value, the record is right and you wait. If it returns something else or nothing, the record is proxied or missing.

This is the same DNS-only requirement that applies to the distribution's own CNAME, for a related reason: Cloudflare proxying interferes with anything that needs to see the real record or the real client. Two different symptoms, one setting.

Teardown Has an Order¶

Not an afterthought, because several resources refuse to delete while something depends on them.

1. Disable the CloudFront distribution, wait, then delete it. A distribution cannot be deleted while enabled, and disabling propagates to every edge location before the delete is accepted. This is the slow step.
2. Empty the buckets, including all versions. Versioning is on, so a normal delete leaves every non-current version behind, and the bucket will not delete while they exist.
3. Delete the buckets.
4. Schedule the KMS keys for deletion.

The general shape: anything with a propagation delay or a mandatory waiting period has to go first, because the things that delete instantly are usually the things it depends on.

Running These After a Change, Not Just After a Build¶

The six checks are worth more as a routine than as a one-time gate, because three of the properties they test can be broken by changes that appear unrelated.

Uploading with the wrong flags breaks encryption uniformity. A sync without --sse aws:kms --sse-kms-key-id relies on the bucket default, which mostly works and can leave an object whose per-request header does not carry the key ID. Nothing in a bucket listing shows encryption per object, so check 5 is the only place it surfaces.

Recreating the distribution invalidates the bucket policy. The policy is scoped to a specific distribution ARN. Delete and recreate the distribution and the ID changes, so the policy now names something that does not exist and every request returns the second kind of 403.

Adding a new object type can silently miss replication. A rule with Filter: {} covers everything, but any narrowing of that filter later applies to new uploads only, and check 4 comparing counts is what catches the divergence.

The cheapest version of this is the two commands that cover the most ground:

curl -I https://ibtisam.qzz.io                                     # 200 plus x-cache
curl -I https://$PRIMARY_BUCKET.s3.us-east-1.amazonaws.com/index.html   # 403

One confirms the delivery path works. The other confirms the origin is still closed. Between them they catch the two failures that matter most, and they take four seconds.

Verify at the Destination¶

Two of the failures in this build have the same shape, and it is the shape worth taking away.

The empty replica: the rule is written on the source and the evidence is in the destination bucket. S3 Server Access Logs, which I found while writing this up, are the same trick in reverse: put-bucket-logging is called against the source and the permission is needed on the destination bucket's policy, so the call succeeds and no logs ever arrive.

Two instances in one small project, both found by looking for an output that should have existed rather than by re-reading configuration. It generalises past S3: for anything that produces a side effect somewhere else, verify at the destination. The source tells you what you asked for. Only the destination knows what happened.

That is the argument for a verification list rather than a smoke test. A smoke test asks whether the thing works. A verification list asks whether each property you deliberately built is actually present, and those properties are exactly the ones nobody notices the absence of.

Source¶

• Verification, troubleshooting and teardown, all six checks and the full teardown sequence
• Storage and encryption stage, both keys and their policies
• IAM and replication stage, the full role and rule
• The architecture being verified