Skip to content

Building a Multi-Arch Container CI Pipeline with Hard-Fail Security Gates

DebugBox ships 3 container image variants across 2 architectures to 2 registries, producing 22 tags per release. The pipeline that handles this has three properties I consider non-negotiable: no image ships without a security scan, no manual step is required to cut a release, and local development runs the same checks as CI.

This post covers the full pipeline architecture: how the base image lifecycle was separated from the variants, why SHA digest pinning replaced :latest, and how Trivy gates are configured to block releases on real vulnerabilities.

Pipeline Architecture

The build system consists of three GitHub Actions workflows:

  • base.yml: Builds, scans, and pushes the base image independently
  • ci.yml: Builds and tests all variant/architecture combinations on every push
  • release.yml: Validates, scans, and publishes a tagged release

Each workflow serves a distinct purpose. Merging them into one would create coupling between the base image lifecycle and variant releases, which have different trigger conditions and different failure modes.

The Base Image Lifecycle

The base image (debugbox-base) is the Alpine 3.21 foundation that all three variants build FROM. It contains CA certificates, a colored shell prompt, and the ll alias. Nothing else.

Before v1.1.0, this image was built and pushed manually. That created a gap: the base could change without being scanned, and variants would silently pick up whatever :latest pointed to.

Now base.yml triggers on any change to Dockerfile.base or the base shell profile:

on:
  push:
    branches: [main]
    paths:
      - 'dockerfiles/Dockerfile.base'
      - 'dockerfiles/profiles/debugbox-base-profile'
      - '.github/workflows/base.yml'

The workflow builds for both amd64 and arm64, runs Trivy on each, and only pushes if both scans pass. The push job depends on the scan job, so a vulnerability in either architecture blocks the entire push.

SHA Digest Pinning

With the base image automated, the next problem was determinism. If a variant Dockerfile says FROM ghcr.io/ibtisam-iq/debugbox-base:latest, then the variant build depends on whatever the base image happened to be at build time. Two builds of the same variant commit could produce different images.

The fix: pin by SHA digest.

ARG BASE_IMAGE=ghcr.io/ibtisam-iq/debugbox-base@sha256:56a6c7cc829a5b3eefbced578ffc097bdeedee0e0489c7a73604c409e501023f
FROM ${BASE_IMAGE}

This digest refers to a manifest list that covers both amd64 and arm64. Docker Buildx selects the correct platform automatically. The variant gets the exact base it was tested against, regardless of when the build runs.

The Makefile overrides BASE_IMAGE with debugbox:base-local for local development, so you can iterate on the base and variants together without pushing to a registry.

docker buildx build \
  --build-arg BASE_IMAGE=debugbox:base-local \
  -f dockerfiles/Dockerfile.$(variant) .

When the base image is updated and a new digest is available, the variant Dockerfiles are updated with the new SHA. This is an explicit, reviewable change in the git history, not a silent dependency shift.

The Release Pipeline

A release is triggered by pushing a semver git tag:

on:
  push:
    tags: ['v*.*.*']

The pipeline runs three jobs sequentially:

1. Validate

Extracts the version from the git tag and confirms it matches semver format. This catches typos (e.g., v1.1 instead of v1.1.0) before any images are built.

2. Build and Scan

Builds each of the three variants (lite, balanced, power) for amd64 only. The images are loaded locally (not pushed) and scanned with Trivy:

- name: Trivy security scan
  uses: aquasecurity/trivy-action@v0.36.0
  with:
    image-ref: debugbox-${{ matrix.variant }}:ci
    severity: HIGH,CRITICAL
    exit-code: '1'

exit-code: '1' is the critical setting. It means any HIGH or CRITICAL finding fails the job. The push job depends on this job, so a single vulnerability in any variant blocks the entire release.

The one override that does exist is a .trivyignore file, and it is used exactly once, for a specific reason recorded inline: yq v4.53.3 is compiled with Go v1.26.4, and CVE-2026-39822 (a symlink traversal in Go's os.Root) is fixed in Go v1.26.5, which no yq release uses yet. The entry names the CVE, states why it cannot be fixed upstream today, and says what condition removes it. That is the difference between a real override and the ones that erode a gate over time: it is one line, dated, and self-expiring rather than an accumulating list nobody remembers the reason for.

Why amd64 only for scanning? Building multi-arch images requires --push (they cannot be loaded locally when targeting multiple platforms). Scanning requires the image to be available locally. So the scan runs on amd64 builds, and the push job rebuilds for both architectures.

3. Push

If all scans pass, this job rebuilds all three variants for both amd64 and arm64, and pushes to both GHCR and Docker Hub. Each variant gets multiple tags:

# For the "balanced" variant at version 1.1.0:
ghcr.io/ibtisam-iq/debugbox:balanced-1.1.0     # Pinned
ghcr.io/ibtisam-iq/debugbox:balanced-latest    # Floating
ghcr.io/ibtisam-iq/debugbox:balanced           # Primary
ghcr.io/ibtisam-iq/debugbox:1.1.0              # Default alias
ghcr.io/ibtisam-iq/debugbox:latest             # Default alias

The same tags are mirrored to Docker Hub under mibtisam/debugbox. Total: 11 unique tag patterns across 2 registries = 22 tags.

CI: The Pre-Release Safety Net

The CI workflow (ci.yml) runs on every push to main and on pull requests. It builds a 3x2 matrix (3 variants, 2 architectures = 6 jobs) and runs smoke tests inside each built container:

- name: CI smoke test
  run: |
    docker run --rm \
      -v ${{ github.workspace }}/tests:/tests:ro \
      debugbox-${{ matrix.variant }}:ci \
      sh /tests/ci-smoke.sh

The smoke test verifies that every tool listed in the variant's verify script is actually present and callable. It catches cases where a package was removed from the install list but not from the verify list, or where an Alpine package was renamed upstream.

Dependency Hygiene

Two dependency upgrades in v1.1.0 illustrate why automated scanning matters:

yq (v4.50.1 to v4.53.3): The old version shipped with a Go runtime that Trivy flagged for multiple HIGH and CRITICAL CVEs. The new version resolves them. In the power variant, yq is installed from a pinned upstream binary rather than Alpine's apk, so the upgrade required updating the SHA checksum in scripts/install-yq-binary. Lite and balanced pull yq from apk and get the version apk resolves automatically.

kubectx/kubens (v0.9.5 to v0.11.0): These are built from source in a Go builder stage inside the Dockerfile. The build patches golang.org/x/net to v0.55.0 at build time to resolve a vulnerability in the upstream dependency:

RUN go get golang.org/x/net@v0.55.0 && go mod tidy

Both of these would have been caught by Trivy at release time. But catching them during development (through make scan locally) is faster and cheaper than discovering them during a release attempt. This is a different discipline from the tool-inclusion framework in Why I Removed Tools from My Container Image: that post is about whether a tool belongs in the image at all, this is about keeping a tool that does belong there patched.

Local Development Parity

The Makefile provides the same checks locally:

make check    # Runs: lint, build-all, test-all, scan

make lint runs Hadolint on all four Dockerfiles (including base). make build-all builds all variants using the local base image. make test-all runs smoke tests. make scan runs Trivy.

The goal: if make check passes locally, CI will pass. No surprises that only appear in the pipeline.

Shell Profile Externalization

One architectural decision worth noting: shell helper functions (like json(), yaml(), ports, sniff-http) used to be defined inline in each Dockerfile as heredocs. This had a practical problem. Any edit to a shell alias invalidated the Docker layer cache for every layer below it, triggering a full rebuild of packages that had not changed.

In v1.1.0, all shell helpers were extracted into standalone files under dockerfiles/profiles/:

dockerfiles/profiles/
  debugbox-base-profile       # ll alias
  debugbox-lite-profile       # json(), yaml()
  debugbox-balanced-profile   # ports, connections, routes, sniff, cert-check()
  debugbox-power-profile      # conntrack-watch (duplicates balanced intentionally)

Each profile is copied into the image via COPY and installed as a /etc/profile.d/ script. Editing a shell helper no longer invalidates the package installation layer. The power profile intentionally duplicates the balanced profile's content rather than sourcing it, avoiding load-order dependencies between profile scripts.

Series: DebugBox, From Variant Design to Release Pipeline (Part 3 of 4)