Six Errors from Building systemd Rootfs Images, and How I Fixed Each¶
None of these appeared in documentation. Each one cost me somewhere between twenty minutes and an afternoon, and every fix turned out to be a one-line change once I understood the cause. Here they are with the symptom, the root cause and the resolution, in the order I hit them.
Try It Yourself
Every error below happened on a real boot, not in theory. Launch the CI/CD Stack ↗ to see the fixed versions running clean.
The context is a set of OCI images that boot as microVMs rather than run as containers, each carrying a full service stack: Java, Nginx, PostgreSQL, SonarQube, Nexus, Jenkins, all under systemd. The build model is covered separately; this post is only the things that broke.
1. Unable to locate package postgresql-common¶
Symptom
The package exists. apt-cache search from an interactive shell in the same base image finds it. The build still fails.
Root cause
A previous RUN layer ended with the conventional cleanup:
That rm -rf deletes the package index. The next RUN is a completely separate shell in a new layer, and it inherits a filesystem with no index at all. apt-get install has nothing to search.
The reason this is confusing is that the advice to clean apt lists is correct: it keeps the image smaller. It just has a consequence nobody mentions, which is that every subsequent install has to rebuild the index first.
Fix
Start every install script with an update, unconditionally:
The general rule. Each RUN is an isolated shell against the previous layer's filesystem. apt-get update and apt-get install must either be in the same RUN, or the install must begin with its own update. Assuming a warm cache across layers is the mistake.
2. CREATE DATABASE Cannot Run Inside DO $$¶
Symptom
Root cause
I was provisioning the database at boot and wanted it idempotent, so it could run on every start without failing the second time. The obvious approach is the same pattern that works for roles:
DO
$$
BEGIN
IF NOT EXISTS (SELECT 1 FROM pg_database WHERE datname = 'sonarqube') THEN
CREATE DATABASE sonarqube OWNER sonar;
END IF;
END
$$;
That works for CREATE ROLE and fails for CREATE DATABASE. DO blocks execute inside a transaction, and CREATE DATABASE is one of a small set of PostgreSQL statements that cannot run in one. The restriction is not about permissions or syntax.
Fix
Do the existence check in the shell, outside PostgreSQL, and issue a plain single command when it is needed:
DB_EXISTS=$(sudo -u postgres psql -tAc \
"SELECT 1 FROM pg_database WHERE datname='sonarqube'" 2>/dev/null || echo "0")
if [ "${DB_EXISTS}" != "1" ]; then
sudo -u postgres psql -v ON_ERROR_STOP=1 -c \
"CREATE DATABASE sonarqube OWNER sonar ENCODING 'UTF8' TEMPLATE template0;"
fi
The -tAc flags matter: tuples only, unaligned, single command. Without them the output carries formatting that breaks the comparison.
Roles keep the DO block, because that one genuinely works. Two different idempotency mechanisms in the same script, for a reason.
3. LC_COLLATE 'en_US.UTF-8' Is Not Recognised¶
Symptom
Immediately after fixing the previous error, on the very next line of the same statement.
Root cause
The base image is minimal and only generates the C.UTF-8 locale. en_US.UTF-8 is not compiled, so the libc collation provider cannot resolve it. PostgreSQL asks libc, libc does not know the name, the statement fails.
This is easy to misread as a PostgreSQL problem. It is a locale generation problem in the image underneath it.
Fix
Two options, and I took the second.
Generate the locale during the build:
Or use the locale that already exists:
CREATE DATABASE sonarqube OWNER sonar
ENCODING 'UTF8'
LC_COLLATE 'C.UTF-8'
LC_CTYPE 'C.UTF-8'
TEMPLATE template0;
C.UTF-8 sorts by byte value rather than by language rules. For a SonarQube backing store, which stores analysis results rather than user-facing sorted text, that difference does not matter and it avoids adding a locale to the image for one statement.
TEMPLATE template0 is required whenever you specify a locale that differs from the one template1 was created with. Omit it and PostgreSQL refuses the whole statement with a different and equally unhelpful error.
4. Nexus Download Returns 404¶
Symptom
Against a URL pattern that was correct in every tutorial I could find.
Root cause
Sonatype changed the download URL structure in Nexus 3.x. It is now architecture-specific, and the naming is not what you would guess:
Note aarch_64 with an underscore, not aarch64. That is Sonatype's convention, it does not match uname -m output, and nothing warns you.
Fix
Detect the architecture and map it explicitly:
ARCH=$(uname -m)
case "${ARCH}" in
x86_64) NEXUS_ARCH="linux-x86_64" ;;
aarch64) NEXUS_ARCH="linux-aarch_64" ;;
*) echo "Unsupported architecture: ${ARCH}"; exit 1 ;;
esac
curl -fsSL -o nexus.tar.gz \
"https://download.sonatype.com/nexus/3/nexus-${NEXUS_VERSION}-${NEXUS_ARCH}.tar.gz"
The *) branch is worth including. Without it, an unsupported architecture produces a 404 several steps later instead of a clear message at the point of failure.
5. Jenkins Ignores the Port I Configured¶
Symptom
Jenkins starts on 8080 regardless of what /etc/default/jenkins contains. Nginx proxies to the configured port and returns 502.
Root cause
/etc/default/jenkins is the classic configuration mechanism and Jenkins stopped reading it in 2.332. Configuration moved into the systemd unit. Every result in the top of a search still describes the old file, because it was correct for years.
Fix
Set the port directly in ExecStart:
[Service]
Type=notify
ExecStart=/usr/bin/jenkins --httpPort=__JENKINS_PORT__
User=jenkins
Group=jenkins
__JENKINS_PORT__ is a build-time placeholder substituted by sed, so the same value lands in the unit file, the Nginx upstream and the welcome banner from one build argument.
How to confirm which mechanism your version uses:
If the port is not in there, it is not being applied.
Two other settings in that unit are worth carrying over. Type=notify because Jenkins implements sd_notify, so systemd knows when it is genuinely ready rather than merely started. And OOMScoreAdjust=-900, so on a memory-constrained machine the kernel reaches for something else first.
6. java.util.prefs Warnings Every 30 Seconds¶
Symptom
Repeating in the Nexus log indefinitely. Nothing is broken. The log is unusable.
Root cause
The nexus user is created as a system account with no home directory:
That is correct for a daemon. But the JVM writes user preferences to ~/.java/.userPrefs, and with no home directory it cannot create the path, cannot take the lock, and warns on every flush attempt.
Fix
Point the JVM somewhere it can write, inside the directory the service already owns:
echo "-Djava.util.prefs.userRoot=/opt/sonatype-work/jvm-prefs" >> \
/opt/nexus/bin/nexus.vmoptions
mkdir -p /opt/sonatype-work/jvm-prefs
chown -R nexus:nexus /opt/sonatype-work/jvm-prefs
One extra step that is easy to miss: the boot-time init script recreates that directory and re-applies ownership on every start, because permissions on the mounted filesystem do not always survive into a fresh VM. Creating it once at build time is not enough.
What These Have in Common¶
Five of the six were caused by an assumption that was correct in a different context.
The apt cache is normally warm. DO blocks normally make things idempotent. en_US.UTF-8 normally exists. /etc/default/<service> is normally where configuration lives. A user normally has a home directory.
Every one of those is true on a full desktop or a conventional server. None of them is reliably true inside a minimal image built in layers, and the error messages point at the symptom rather than the assumption.
The one that does not fit the pattern is the Nexus URL, which was just an undocumented vendor change.
The habit that came out of this: when an error makes no sense, ask what the tool is assuming about the environment, rather than what it is doing wrong. The fix is usually in the environment.
Source¶
- All six images
- SonarQube boot script covers errors 2 and 3
- Nexus install script covers errors 4 and 6
- Jenkins service unit covers error 5
Series: Building a Self-Hosted CI/CD Stack from Scratch (Part 4 of 6)
- Previous: The Machine I Actually Work From
- Next: Four Machines, One Playground