Building an OCI Image That Boots as a microVM, Not a Container¶
I write a Dockerfile and what comes out the other end is a bootable server. Not a container image that happens to run systemd, but a root filesystem that a hypervisor mounts as a block device and boots with its own kernel. Running any of these images with docker run fails immediately, and that is the correct behaviour.
Try It Yourself
See the model boot for real, not in theory: Launch Jenkins Playground ↗.
Click Start, then run the verification commands from this post against a live VM instead of a container.
The Delivery Model¶
The iximiuz Labs platform boots each playground machine as a Firecracker microVM. You supply the root filesystem as an OCI image and reference it in a manifest:
The platform pulls the image, mounts it as the VM's /, and boots it with its own kernel. systemd becomes PID 1 through that boot process, not through anything declared in the image.
The alternative on most lab platforms is init scripts that run on every start, which means every user waits through an install before they can begin. Baking the state into the image moves that cost to build time, once, and gives you a machine whose entire configuration is a Dockerfile in Git.
The catch is that a container image and a VM disk have opposite assumptions, and several Docker defaults are actively wrong in this context.
Four Things the Build Deliberately Removes¶
SSH host keys¶
Normal Docker practice is to bake in whatever the image needs. Here, baked-in host keys mean every VM ever launched from that image presents the same identity. Anyone who pulls the image has your host keys.
The base image also masks the units that would regenerate them at package-install time, so nothing quietly puts them back:
Keys are generated per machine at first boot instead. More on that below.
Machine IDs¶
/etc/machine-id and /var/lib/dbus/machine-id identify a host to systemd and D-Bus. If they are populated in the image, every VM built from it claims to be the same host, which breaks journald and anything that keys off machine identity.
Emptied rather than deleted. systemd regenerates the contents on first boot when the file exists but is empty; if the file is missing entirely, some tooling behaves differently.
/.dockerenv¶
systemd checks for container markers and changes its behaviour when it finds them, skipping units it considers inappropriate for a container. Since this filesystem boots on a real kernel, that detection would produce the wrong result.
This line is belt and braces, not load bearing
Establishing that took three attempts. Docker creates /.dockerenv in every container it starts, and never writes it into an image layer, so there is nothing persistent for the rm to delete.
You cannot confirm that with docker run, because running the test creates the file being tested for. You cannot confirm it with docker create plus docker export either: that reports the file present on every image, including a stock ubuntu:24.04, because the runtime injects it at create time. Only reading the layers settles it.
Nothing found, on any image built this way. The line stays because upstream keeps it and it costs nothing, but it is not what stops systemd misdetecting a container here. The hands-on version of this check walks through all three attempts.
CMD on the base image¶
The base image has no CMD and no ENTRYPOINT at all. The platform boots the filesystem with its own kernel and bootloader, so anything declared here is at best ignored and at worst fights the boot process.
Service images that build on the base do set CMD ["/lib/systemd/systemd"], but for a different reason, covered in the next section.
The USER Asymmetry¶
The workstation image ends with USER $USER. The service images end with USER root. That looks like an inconsistency and it took me a while to be confident it is not.
USER is an OCI image config field. Only docker run reads it. When the platform boots the image as a microVM, it mounts the filesystem and starts its own kernel; the image config is never consulted, so the directive has no effect on the running machine either way.
Which means the value only matters for the one thing it still influences: what happens when someone runs the image locally.
| Context | Who decides the starting user | Effect of USER |
|---|---|---|
docker run binary check | Docker daemon, from the image config | Real. Process starts as that user. |
| microVM boot | Platform kernel, systemd as PID 1 | None. Field is never read. |
For the workstation image, ending as the non-root user makes a local docker run binary check run as the user who will actually use those tools inside the VM:
docker run --rm ghcr.io/ibtisam-iq/dev-machine-rootfs:latest bash -c "
kubectl version --client
terraform version | head -1
trivy --version | head -1
"
If a tool were installed somewhere only root can reach, that check fails loudly. Had the Dockerfile ended with USER root, the same check would pass while the tool remained unusable for the actual user. The failure would surface later, inside a running VM, with no obvious cause.
Service images end as root because their CMD starts systemd, which must be PID 1 and must be root. That CMD exists purely so docker run produces an honest failure rather than dropping you into a shell that looks like it worked.
What Cannot Be Baked In¶
An immutable image gives you a reproducible filesystem. It does not give you a working machine. Three categories of state do not survive into a booted VM.
/run is tmpfs and is wiped on every boot. sshd requires /run/sshd to exist and will not create it. Nginx requires /run/nginx. PostgreSQL requires /run/postgresql owned by postgres. All three fail to start without them, and none of them can be created at build time because the directory does not persist.
Host keys were deliberately deleted, so something has to generate them before sshd starts.
Ownership resets on the mounted filesystem, so a service cannot necessarily write to its own data directory.
Each image handles this with a single systemd oneshot ordered ahead of everything else:
[Unit]
Description=Jenkins Lab Runtime Initialization
Before=ssh.service nginx.service jenkins.service
After=local-fs.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/opt/jenkins-scripts/lab-init.sh
[Install]
WantedBy=multi-user.target
The Jenkins version is short: ssh-keygen -A if keys are absent, mkdir -p /run/sshd /run/nginx, and a chown -R jenkins:jenkins /var/lib/jenkins. Then it exits and systemd starts the real services.
Why not the platform's init tasks feature¶
The platform provides an initTasks mechanism in the manifest that can run shell before the user lands, with dependency ordering across machines. It looks like a direct replacement and it is not, for two reasons.
Init tasks run once per playground instance and are not re-run after an in-session reboot. Since /run is tmpfs, a user who reboots the VM mid-session would come back to a machine where sshd and Nginx have no runtime directories and both fail to start. A systemd oneshot in multi-user.target.wants runs on every boot, including that one.
The second reason is packaging. Init tasks live in the manifest. The oneshot lives in the image. Anyone who forks the image and writes their own manifest gets working boot behaviour without having to know it was needed.
The SonarQube case¶
SonarQube's lab-init.sh is where this pattern earns its keep, because it has to provision a database at runtime.
PostgreSQL cannot be initialised at Docker build time. The cluster needs a live system with proper OS users, a real /run/postgresql, and a running postgres process. So the script does it on every boot, idempotently:
pg_ctlcluster 18 main start || true
for i in {1..30}; do
if sudo -u postgres psql -c '\q' 2>/dev/null; then break; fi
sleep 1
done
pg_ctlcluster rather than systemctl start postgresql because on Debian and Ubuntu the postgresql unit is a wrapper that does not behave predictably inside a oneshot. The readiness poll matters because the next step fails against a cluster that is still starting.
Role creation is idempotent with a DO block:
DO $$
BEGIN
IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'sonar') THEN
CREATE ROLE sonar WITH LOGIN ENCRYPTED PASSWORD 'sonar_password';
END IF;
END
$$;
Database creation cannot use the same pattern, because CREATE DATABASE is not permitted inside an anonymous block. That one needs a shell-level check:
DB_EXISTS=$(sudo -u postgres psql -tAc \
"SELECT 1 FROM pg_database WHERE datname='sonarqube'" 2>/dev/null || echo "0")
if [ "${DB_EXISTS}" != "1" ]; then
sudo -u postgres psql -c "CREATE DATABASE sonarqube OWNER sonar ..."
fi
Finally the Elasticsearch kernel parameters, applied live because the microVM may not read /etc/sysctl.conf during its boot:
These are also written to /etc/sysctl.conf at build time. The duplication is deliberate: the file is the declaration, the sysctl -w is the one that actually takes effect.
The database provisioning above did not work on the first attempt. The DO block versus shell-check split and the locale that had to change both came from real failures, covered with symptoms and fixes in Six Errors from Building systemd Rootfs Images.
The Model¶
Once I stopped treating this as a Docker problem, the design became mechanical.
The Core Principle
- Bake what is static: Packages, configuration files, systemd units, binaries, users.
- Reconcile what is per-machine or per-boot: Identity, runtime directories, ownership, data.
The first goes in the Dockerfile. The second goes in a oneshot ordered before everything that depends on it. That split holds for anything ephemeral, not just this platform.
Testing an Image That Cannot Run¶
The image cannot be started during its own build, so the usual smoke test is unavailable. systemd is not running, which means systemctl is-enabled returns nothing useful.
Each image runs a healthcheck.sh as a RUN step before the final CMD. It fails the build on any error, which keeps broken images out of the registry. The checks work around the absence of a running init system:
| What | How it is checked without systemd |
|---|---|
| Service enabled | Symlink present in /etc/systemd/system/multi-user.target.wants/ |
| Package installed | dpkg-query -W -f='${Status}' |
| Nginx config valid | nginx -t |
| Directory ownership | stat -c '%U' |
| Port substitution complete | grep for any remaining __PORT__ placeholder |
| SSH host keys | Asserted absent, because present would be the bug |
That last row is the one I like. The test asserts that something is missing, because in this context a populated /etc/ssh/ssh_host_* means the image would ship a shared identity.
Jenkins and Nexus check eight sections, SonarQube ten.
Build-Time Parameterisation¶
Every service port is a build argument, substituted with sed into every file that references it:
ARG JENKINS_PORT
COPY configs/nginx.conf /etc/nginx/sites-available/jenkins
RUN sed -i "s/__JENKINS_PORT__/${JENKINS_PORT}/g" /etc/nginx/sites-available/jenkins
COPY configs/jenkins.service /etc/systemd/system/jenkins.service
RUN sed -i "s/__JENKINS_PORT__/${JENKINS_PORT}/g" /etc/systemd/system/jenkins.service
The placeholder appears in the Nginx upstream, the systemd ExecStart, and the welcome banner. One build argument changes all three consistently, and the healthcheck fails the build if any placeholder survives.
Two Ordering Constraints That Bite¶
The welcome file must be copied last. customize-bashrc.sh appends logic to ~/.bashrc that displays ~/.welcome on first interactive login and then deletes it. If the COPY happens before that script runs, a non-interactive build step can consume the file and the banner never appears in the VM.
Some scripts must use COPY, not a bind mount. Most install scripts are bind-mounted, which avoids a layer:
But install-tools.sh ends with rm -rf /tmp/* as a cleanup step. Running that against a live bind mount is undefined behaviour, so that one script is copied into a real layer that can safely be deleted:
COPY scripts/install-tools.sh /tmp/scripts/install-tools.sh
RUN chmod +x /tmp/scripts/install-tools.sh && /tmp/scripts/install-tools.sh
The Inheritance Chain¶
One base, five children:
ubuntu-24-04-rootfs Ubuntu 24.04, unminimized, systemd, SSH, base toolset
├── dev-machine-rootfs Full workstation, ~40 tools
├── dev-cicd-rootfs Minimal jump host, SSH aliases only
├── jenkins-rootfs Java 21, Jenkins LTS, Nginx, cloudflared
├── sonarqube-rootfs Java 21, PostgreSQL 18, SonarQube CE, Nginx
└── nexus-rootfs Java 21, Nexus 3, Nginx
A child image may assume systemd is the init system, SSH is configured with host-key generation deferred to boot, the interactive user exists with a working shell, and the base toolset is on PATH.
The guidelines for writing one: start with USER root, enable services with systemctl enable during build, place the welcome COPY after all bashrc customisation, and end as root with CMD ["/lib/systemd/systemd"] unless the image is a pure workstation.
All six are built by GitHub Actions on every push and published to GHCR. The base builds multi-arch; the children are amd64 only, which is a gap rather than a decision.
What docker run Does, and Why That Is Fine¶
That is the expected result. docker run on these images is useful for exactly one thing: confirming that binaries and configuration files are present and readable by the right user. It cannot validate systemd, service startup, networking or anything else that requires a booted machine.
The only real runtime verification is booting the image:
Then, inside:
systemctl is-system-running # running
systemctl is-active lab-init nginx jenkins
curl -f http://localhost/health # healthy
Series: Building a Self-Hosted CI/CD Stack from Scratch (Part 2 of 6)
- Previous: Your Server Has No Public IP
- Next: The Machine I Actually Work From