Six Things That Break When You Actually Use a Debugging Container¶
Building the image is the easy part. What actually accumulates over months of running DebugBox against real clusters is a list of things that look like bugs the first time and turn out to be the correct behaviour of something underneath: musl, network namespaces, kube-proxy, ps column order. Six of them, with what actually caused each one.
1. ltrace prints "0 total" and nothing else¶
The symptom is quiet rather than loud. ltrace -c <command> runs to completion without error and reports zero library calls, on a command that obviously calls libraries.
The cause is the base image, and it is worth being explicit about the trade-off it represents. ltrace hooks shared library calls through the PLT, the Procedure Linkage Table, which is a glibc mechanism. DebugBox is built on Alpine, which uses musl libc, and musl does not implement glibc's PLT ABI. There is nothing to hook, so nothing fires, and the tool reports truthfully that it saw zero calls.
This is the same Alpine trade-off I wrote about from the other side in The Vulnerability Gate That Blocked Every Build, where a JRE runtime on Alpine produced a scanning gate with no passing state. Here the cost shows up differently: one specific tool loses a specific capability. ltrace still works, just only against processes that are themselves glibc-linked, which describes most application images (Debian, Ubuntu, Red Hat bases) even when the debugging container attached to them is not:
kubectl debug <pod> -it \
--image=ghcr.io/ibtisam-iq/debugbox:power \
--target=<pod>
APP_PID=$(ps aux | awk 'NR==2{print $2}')
ltrace -p "$APP_PID"
--target shares the process namespace of the named container, so ltrace attaches to the application's PID, not the debug container's own shell. Against a musl-linked target, strace is the tool that still works, because it hooks system calls at the kernel boundary rather than library calls at the PLT, and that boundary does not care which libc built the binary.
2. tcpdump -i eth0 counts zero packets while traffic is clearly flowing¶
This one costs people real time because nothing about it looks wrong. The interface exists, the command runs, and it sits at zero while the application pod is visibly serving traffic.
The cause is kubectl run. It creates a new pod with its own network namespace, complete with its own eth0. That interface is real and it is capturing correctly, it is just capturing a namespace nothing is talking to. A separate pod's traffic, on a separate virtual NIC, is invisible from here regardless of how long you wait.
kubectl debug solves this because it does something structurally different: it attaches an ephemeral container to an existing pod's namespaces rather than creating a new pod. The debug container's eth0 becomes the target's eth0.
kubectl debug <pod> -it --image=ghcr.io/ibtisam-iq/debugbox
tcpdump -i eth0 -n 'tcp port <service-port>'
With the capture running, generate traffic from a second terminal outside the pod:
kubectl port-forward service/<service-name> <service-port>:<service-port>
curl http://localhost:<service-port>/
The general lesson survives the specific tool: kubectl run gives you a new, isolated place to stand. kubectl debug --target gives you the target's own vantage point. Which one you want depends entirely on whether you are testing connectivity to a pod or inspecting what a pod itself sees, and picking the wrong one produces results that are accurate and useless at the same time.
3. nmap <service> hangs for minutes with no output¶
Not an error, just silence, for long enough that it looks hung.
A full nmap -p- sweep checks all 65,535 ports. Any port a Kubernetes Service does not expose is filtered at the cluster networking layer rather than actively refused: the packet is silently dropped, no RST is sent. nmap cannot distinguish "filtered, nothing here" from "slow to respond," so it waits out its per-port timeout on every one of them. Multiply a multi-second timeout by tens of thousands of ports and the sweep runs for hours.
The fix is to stop asking a question the cluster network was never going to answer usefully, and scope the scan to what you actually expect:
-Pn matters for a related reason: nmap's default host-discovery phase pings first, and Kubernetes ClusterIPs do not answer ICMP at all, so an unscoped scan can report "Host seems down" before it checks a single port. Skipping discovery and scoping the port range are two different fixes for two different consequences of the same fact, that ClusterIP networking does not behave like a normal host on a LAN.
4. iperf3 -c <pod-name> refuses the connection immediately¶
The instinct is to point one debug pod at another by name, and it fails as if nothing is listening, even though the server side is confirmed running.
Pod names alone carry no DNS entry in Kubernetes. DNS resolution requires a Service object; a bare pod, however healthy, is not addressable by name. The client is not failing to connect, it is failing to resolve, and the two errors can look identical from the client's side.
kubectl run iperf-server \
--image=ghcr.io/ibtisam-iq/debugbox:power \
--command -- iperf3 -s
kubectl expose pod iperf-server --port=5201
kubectl wait pod/iperf-server --for=condition=Ready --timeout=60s
kubectl run iperf-client --rm -it \
--image=ghcr.io/ibtisam-iq/debugbox:power \
--command -- iperf3 -c iperf-server -t 30
The expose step is what actually fixes it, not the retry. This generalises past iperf3: any tool that takes a hostname argument against a bare pod hits the same wall, because the problem was never the tool.
5. The shell helpers are missing even though the right variant is running¶
A correct power pod, ports and sniff and the rest of the helper functions simply not found.
The helpers live in /etc/profile.d/, and both bash and ash only source that directory for a login shell. kubectl exec without -l gives you a shell, just not a login one, so the profile scripts that register every helper never run.
kubectl exec -it <pod> -- bash -l # balanced, power
kubectl exec -it <pod> -- ash -l # lite
kubectl debug <pod> -it --image=ghcr.io/ibtisam-iq/debugbox -- bash -l
One flag, and it is the single most common way to lose functionality that was never actually missing.
6. nft list ruleset fails, iptables on the same node works fine¶
Two firewall tools, same host, one works and one reports the ruleset file does not exist.
nft talks to the kernel through the nf_tables subsystem specifically. A cluster running kube-proxy in iptables mode has no reason to ever load that kernel module, so the netlink socket nft needs is simply absent, not permission-denied, not misconfigured, absent. iptables talks to a different, older subsystem that is loaded regardless of proxy mode, which is why it keeps working on the exact same node where nft cannot even start.
On a Docker host, where nothing has decided the kernel's firewall backend on your behalf, nft works normally with the right capability:
Neither tool is broken. They are reading two different kernel subsystems, and only one of them is guaranteed to be loaded on a given cluster.
The shape of all six¶
None of these are bugs in DebugBox, and none of them are really about DebugBox at all. They are what the platform underneath actually does, made visible by a tool that does not paper over it: musl's PLT gap, network namespace isolation, silent packet drops on unexposed ports, DNS requiring a Service object, login shells versus exec shells, and a firewall backend that depends on how kube-proxy was configured before the debugging container ever started.
A debugging tool that hides these differences would be more comfortable to use and would teach nothing. The value in running into them once, with a clear enough symptom-cause-fix trail to not have to re-derive it, is that the second time one of these shows up on a different pod, in a different cluster, it takes thirty seconds instead of thirty minutes.
Source¶
The full guide, with all 24 entries including DNS resolution, image pull failures, and OpenSSL certificate handling, plus a placeholder-convention table used consistently throughout: Troubleshooting on the docs site.
- Pre-built example manifests, including the power variant with
NET_ADMINandNET_RAWalready declared - GitHub repository
Series: DebugBox, From Variant Design to Release Pipeline (Part 4 of 4)