Building Only What Changed, and the Permissions Ceiling Nobody Warns You About¶

Ten services in one repository. Most commits touch one of them. Rebuilding all ten on every push is the default and it is wrong, so the pipeline detects which service directories changed and fans out one job per service.

Two things about that turned out to be harder than the idea: computing "what changed" correctly across every way a workflow can be triggered, and discovering that a reusable workflow cannot ask for permissions its caller did not grant.

The Detection Is Four Pipes¶

SERVICES=$(git diff --name-only "${BEFORE}" "${{ github.sha }}" \
  | grep '^src/' \
  | cut -d'/' -f2 \
  | sort -u \
  | grep -v '^loadgenerator$' \
  | jq -R -s -c 'split("\n") | map(select(length > 0))')

Filter to the source tree, take the directory name one level down, deduplicate, drop the one directory that is not a shippable service, and emit a compact JSON array.

That array feeds straight into a matrix:

strategy:
  fail-fast: false
  matrix:
    service: ${{ fromJson(needs.detect-changes.outputs.services) }}

fail-fast: false is the important line. The default aborts every sibling job when one fails, which for independent services is exactly backwards: a broken adservice should not cancel a clean frontend build that was going to succeed. Each service owns its own blast radius, and the default assumes a matrix of related variations rather than a matrix of independent things.

loadgenerator is excluded because it is a test harness, not a shippable service. That exclusion has a consequence on the deployment side: no image is ever pushed for it, so the chart must also set loadGenerator.create: false or the pod sits in ImagePullBackOff forever, looking like a registry problem.

Two Values of before That Are Not Commits¶

${{ github.event.before }} is the previous commit, which is the obvious diff base. It is also wrong in two cases that both happen regularly.

On workflow_dispatch, there is no push event, so the field is empty. On the first push to a new branch, there is no previous commit on that ref, so it is the all-zeros SHA.

BEFORE="${{ github.event.before }}"
if [[ -z "$BEFORE" || "$BEFORE" == "0000000000000000000000000000000000000000" ]]; then
  BEFORE="HEAD~1"
fi

Without that guard, git diff receives an invalid range. Depending on which case it is, the diff either errors or returns everything, and "returns everything" is the dangerous one: a manual re-run quietly rebuilds all ten services and looks like it worked.

There is a third prerequisite that is easy to omit and fatal:

- uses: actions/checkout@...
  with:
    fetch-depth: 0

The default checkout is shallow. A shallow clone does not contain the commit you are diffing against, so the range is unresolvable. This one at least fails loudly, but the error is about a missing object rather than about clone depth.

The Permissions Ceiling¶

The first run failed before any job started. Not a build error, a startup error.

A reusable workflow declares the permissions it needs:

# reusable-build.yaml
permissions:
  contents: read
  packages: write          # push to GHCR
  security-events: write   # upload scan results

The caller had only granted contents: read. And a called workflow can narrow the caller's permissions, never widen them. The token it receives is derived from the caller's grant, so requesting more is not a permission error at runtime, it is a validation failure at startup.

# ci-trigger.yaml
permissions:
  contents: read
  packages: write
  security-events: write

The caller is a ceiling, and every reusable workflow it invokes lives underneath it.

The consequence for design is not obvious at first: the caller has to know what its callees need. That leaks the worker's requirements upward into the orchestrator, which is a mild violation of the encapsulation reusable workflows are supposed to provide. There is no way around it, so the honest move is to comment the caller with why each permission is there:

permissions:
  contents: read           # checkout
  packages: write          # GHCR push, delegated to reusable-build.yaml
  security-events: write   # scan upload, delegated to reusable-build.yaml

Otherwise the next person removes security-events: write from a workflow that visibly does no scanning, and breaks a job two files away.

Keeping the Worker Generic¶

One service has its Dockerfile in a different place. cartservice keeps it at src/cartservice/src/, everything else at src/<service>/.

The wrong fix is a conditional inside the worker, which makes a generic component carry knowledge of one repository's layout. The right one is to resolve it at the caller and pass it in:

with:
  service: ${{ matrix.service }}
  docker_context: ${{ matrix.service == 'cartservice'
    && 'src/cartservice/src'
    || format('src/{0}', matrix.service) }}

The worker takes a context path and does not care how it was derived. The caller owns repository layout, which it already does by virtue of computing the service list.

That boundary is worth defending, because the pressure is always to put the special case wherever it is being handled at the time.

Two Details Worth Copying¶

Cache scoping per service. BuildKit's GitHub Actions cache is shared across the workflow unless scoped:

cache-from: type=gha,scope=${{ inputs.service }}
cache-to:   type=gha,mode=max,scope=${{ inputs.service }}

Without scope, ten parallel builds write to one cache key and evict each other's layers continuously. The result is not a failure, it is a cache that never hits, and a build that is slower than having no cache at all because it now also uploads.

Cleaning up the runner. Matrix jobs can share a runner pool, and image layers accumulate:

- name: Cleanup
  if: always()
  run: docker rmi ... || true

Runners are ephemeral, so this looks unnecessary. It matters because large images built in sequence on the same runner can exhaust its disk before it is recycled, and the failure surfaces as an unrelated build step running out of space.

Concurrency, Scoped to Pull Requests Only¶

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

Cancelling superseded pull request runs is free: they have no registry side effects, so killing one wastes nothing but compute.

Cancelling a run on main is not free. That run pushes images, and killing it mid-push can leave a service published while its siblings are not. The expression makes cancellation conditional on the event type rather than turning it off globally, which keeps the compute saving where it is safe.

Two Triggers, Two Flows, No Coordination¶

The repository has a second workflow that fires on chart changes rather than source changes, and the two never talk to each other.

# ci-trigger.yaml
on:
  push:
    paths: [ "src/**" ]

# chart-release.yaml
on:
  push:
    paths: [ "helm-chart/**" ]

Path filters are what makes that safe. A commit touching only src/frontend/ fires the first and not the second. A commit touching only helm-chart/ fires the second and not the first. A commit touching both fires both, in parallel, and they do not conflict because they write to different things: one pushes images, the other publishes a chart and bumps a version.

The alternative design is one workflow with conditional jobs deciding what to run. That centralises the logic and couples the two lifecycles, so a failure in chart packaging can block an image build that had nothing to do with it.

Two workflows with disjoint path filters is more files and less coupling, and the coupling is the thing that hurts later.

What Actually Made This Work¶

The pipeline logic is ordinary. The parts that took time were all edge conditions:

• two sentinel values for before that are not commits
• a checkout default that silently breaks the diff
• a permission model that fails at startup rather than at use
• a matrix default that aborts healthy siblings
• a cache that silently never hits without a scope

None of those appear when you build the happy path and push a normal commit to a branch that already exists. They appear on the first manual re-run, the first new branch, and the first parallel build, which is to say a day or two after you decide it works.

Source¶

• The trigger workflow, which documents its own design decisions in the header
• The reusable worker
• The CI runbook, including the permission failure in context
• Where the images go afterwards