Skip to content

Why I Built My Own CI/CD Stack (And What I Learned)

Most engineers use CI/CD tools. I wanted to understand them well enough to build the infrastructure underneath them: my own servers, my own domain, my own certificates, and eventually my own machine images. This is the reasoning, the honest cost, and an index to everything the project produced.

Launch Any of These Right Now

No install, no signup beyond an iximiuz Labs account. Click Start on any of them.

The Goal

A complete self-hosted CI/CD stack, with each service on its own machine:

  • Jenkins for pipeline automation
  • SonarQube for code quality analysis and quality gates
  • Nexus for artifact storage

Each reachable on its own subdomain with valid TLS. All of it reproducible from code, so that losing every machine costs an evening rather than a weekend.

Why Build It Yourself

Managed CI is genuinely better for shipping software. If the goal were to ship, this project would be a mistake.

The goal was different. Managed CI hides exactly the parts I wanted to be able to reason about: how a build agent is provisioned, what a reverse proxy is doing in front of an application, why services start in a particular order, where a certificate comes from, what happens when a machine is destroyed and rebuilt.

There is also a portfolio argument, and I will state it plainly rather than pretend otherwise. "I set up Jenkins" and "I built a reproducible Jenkins machine image with a build-time healthcheck, port-parameterised configuration, and a CI pipeline that publishes it" describe very different amounts of work. The second is checkable by anyone who opens the repository.

What It Actually Cost

Three days on EC2 to prove the architecture. Then a move to a microVM lab platform (iximiuz Labs), which broke the networking model completely and took another week to understand and solve properly. Then several weeks encoding everything into images so it would survive a reset.

Two things surprised me.

Most of the learning was in error messages. A PostgreSQL locale mismatch. A CREATE DATABASE statement that cannot run inside a DO block. Jenkins ignoring a configuration file it stopped reading in version 2.332. Nexus changing its download URL format. None of those were answerable from the first page of search results, and each required reading a changelog or source rather than a tutorial.

The constraints produced the good decisions. A hard resource ceiling forced me to justify every GiB, which produced JVM tuning I would otherwise have skipped. A platform with no public IP forced me to learn how outbound tunnelling actually works. An ephemeral filesystem forced a clean split between what belongs in an image and what has to be reconciled at every boot.

Three Things Worth Carrying Elsewhere

NAT is the normal case. A routable public IP per host is the exception, not the default. Home connections, corporate networks and most managed platforms all sit behind NAT, and the standard "point an A record at it and run Certbot" recipe silently assumes something you usually do not have.

A Dockerfile can describe a server, not just an application. Packages, users, service units, reverse proxy configuration, validation. Once the whole machine is a file in Git, rebuilding it stops being an event.

The build-time versus boot-time boundary is the hard part. SSH host keys, database provisioning and kernel parameters cannot be baked into an image. They belong in a boot-time unit ordered before everything that depends on them. Getting that split wrong produces failures that look like configuration problems and are actually lifecycle problems.

The Series

Everything the project produced, in reading order.

Post What it covers
Your server has no public IP Why DNS and Certbot fail behind NAT, the five approaches that do not work, and the one that does
Building an OCI image that boots as a microVM The image model: what gets removed on purpose, and the build-time versus boot-time split
The machine I actually work from The full DevOps workstation image: 40-plus tools, aliases wired to real completion, and the USER $USER reasoning in full
Six errors from building systemd rootfs images Specific failures with root causes: apt cache, DO blocks, locales, JVM preferences
Four machines, one playground Composition: node roles, capacity allocation against a fixed budget, JVM sizing
Publishing a custom playground on iximiuz Labs Custom rootfs requirements, the manifest, and labctl, so you can publish your own rather than copy mine
Three places to put provisioning Init tasks, startup files, or the image, and why I chose the image every time

The pipelines that run on this stack are a separate track:

Post What it covers
The vulnerability gate that blocked every build A scanning gate with no passing state, and the redesign that fixed it
The version override Maven silently ignored Clearing seven transitive criticals, and why <properties> works for some libraries and does nothing for others
Trivy could not find the image I had just built BuildKit's store is not the Docker daemon, and a scan that cannot resolve its target
The same pipeline on two CI engines Jenkins versus GitHub Actions, measured, and the gate drift only a side-by-side read found
Twelve fixes to a pipeline that was already passing Interpolation timing, swallowed errors, poisoned coverage trends, and four time bombs
One pipeline contract, three language ecosystems What stayed fixed across Java, Python and Node, and what had to diverge
Four ways to serve a frontend, three of them wrong What decoupling two tiers actually costs, and the reverse proxy it requires
Release identity What a build is called, and what is allowed to reach a cluster

The Repository

Everything is at github.com/ibtisam-iq/silver-stack: Dockerfiles, install scripts, systemd units, Nginx configuration, playground manifests and the workflows that build the images. MIT licensed.

All five environments are launchable in a browser from labs.iximiuz.com/a/ibtisam-iq.

Series: Building a Self-Hosted CI/CD Stack from Scratch (Hub)

This post is the hub, not a numbered part: the reasoning, the cost, and the index above. The numbered series itself starts here.