Why rootfs Decides Whether Linux Can Enter User Space | IoT Worker

Why rootfs Decides Whether Linux Can Enter User Space

Reading time: 7 minute Word count: 1406
Embedded Linux Linux rootfs init initramfs

The serial log may show a long Linux kernel boot, then end in a panic. The kernel may appear to boot, but the product application never starts. The log may say No working init found or VFS: Unable to mount root fs.

These failures are often treated as application startup problems. But before any application can run, Linux has to cross a more basic boundary: the kernel must find and mount rootfs, then execute the first user-space process.

A running kernel is not the same as a running system. Embedded Linux has to complete at least this chain before normal user space exists:

kernel init
-> parse root= / rootfstype= / init=
-> mount root filesystem
-> open console
-> execute /sbin/init or /init
-> init mounts proc/sys/dev/run
-> udev/devtmpfs and system services start
-> device application starts

rootfs is not just a directory archive. It is the runtime contract between the kernel and user space: it must contain the first process, the dynamic linker, base libraries, configuration, device-node entry points, mount points, and service startup logic. If any part is missing, the system can stop in the state where the kernel is running but user space is not usable.

rootfs Is the First User-Space World

After Linux initializes the scheduler, interrupts, memory management, and the driver model, it tries to mount the root filesystem. That filesystem becomes / from the user-space point of view.

It usually has to provide:

  • the first user-space program, such as /sbin/init, /etc/init, /bin/init, /bin/sh, or /init inside initramfs
  • the dynamic linker and shared libraries
  • base directories such as /dev, /proc, /sys, /run, /tmp, and /etc
  • service scripts, systemd units, configuration files, and applications
  • a way to create device nodes, such as devtmpfs, mdev, or udev

So rootfs is not just “put BusyBox or systemd into an image”. It decides whether the kernel can hand control to user space, and whether that first user-space environment is complete enough to keep booting.

If rootfs Cannot Mount, init Never Runs

If the kernel cannot find the root filesystem, the system is still in the kernel stage. Common logs include:

VFS: Cannot open root device ...
VFS: Unable to mount root fs on unknown-block(...)
Kernel panic - not syncing: VFS: Unable to mount root fs

Application logs are not useful yet, because no application has had a chance to run. Start with the rootfs mount path:

  • whether root= points to the correct device, partition, UUID, or PARTUUID
  • whether rootfstype= matches the actual filesystem
  • whether the storage controller driver is already available
  • whether the partition table, device-tree partition description, and bootloader parameters agree
  • whether the filesystem image is corrupted
  • whether the rootfs device appears before the kernel tries to mount root

A frequent issue is this: drivers required to mount rootfs cannot be ordinary modules unless initramfs loads them first.

If rootfs lives on eMMC, NAND, NVMe, USB, or a filesystem that needs a specific driver, the kernel must be able to access it before mounting root. Those drivers must either be built into the kernel or loaded from initramfs. If the module only exists inside rootfs, the kernel cannot mount rootfs to read that module.

If init Cannot Run, rootfs Mounted but User Space Has No Entry Point

Another class of failure happens after rootfs is mounted, when the kernel cannot execute the first user-space process.

A common log is:

Kernel panic - not syncing: No working init found.

By default, the kernel tries paths such as /sbin/init, /etc/init, /bin/init, and /bin/sh. The init= command-line parameter can override this.

init execution can fail for more reasons than a missing file:

  • the file is not executable
  • the binary is for the wrong architecture
  • the dynamic linker path is missing, such as /lib/ld-linux-*.so.* referenced by the ELF file
  • shared libraries are missing or incompatible
  • a script shebang points to a missing interpreter, such as #!/bin/sh without /bin/sh
  • rootfs is mounted with noexec
  • console or standard I/O setup is broken enough that early init behavior is invisible

So when the log says No working init found, do not only check whether /sbin/init exists. Check whether the current kernel can execute it, whether its dependency chain exists, and whether permissions and interpreters are correct.

initramfs Is Not the Same Thing as the Final rootfs

Many systems enter initramfs before mounting the real rootfs. initramfs is an early user-space environment loaded into memory during kernel boot.

It is commonly used to:

  • load modules required to mount the real rootfs
  • wait for slow storage or complex buses to enumerate
  • decrypt, verify, or assemble the root filesystem
  • handle RAID, LVM, network root, or other complex boot paths
  • switch to the real rootfs using switch_root or a similar mechanism

Some small embedded systems use initramfs as the whole system. Many devices do not use initramfs at all and let the kernel mount rootfs directly from flash, eMMC, or NFS.

When debugging boot, first identify where the failure happens: before initramfs, inside initramfs, or after switching to the real rootfs. All three can look like “boot failed”, but the thing to inspect is different.

/dev, /proc, /sys, and /run Decide Whether Services Can Start

After the first user-space process starts, the system is still not fully ready. Init usually prepares the runtime environment:

  • mount procfs at /proc
  • mount sysfs at /sys
  • mount devtmpfs at /dev
  • prepare /run, /tmp, and other temporary directories
  • start udev, mdev, or systemd-udevd
  • start base services and the device application

If these mount points are missing, services often fail in scattered ways instead of reporting “rootfs is wrong”:

  • applications cannot find /dev/ttyS*, /dev/watchdog*, or /dev/input/*
  • systemd units wait on device or mount dependencies
  • scripts cannot read /proc/cmdline or /proc/mounts
  • a driver has probed successfully, but no user-space device node appears
  • temporary files, PID files, or sockets cannot be created

devtmpfs and udev are also different layers. devtmpfs is the kernel-provided base mechanism for device nodes. udev is a user-space rule system that can add permissions, symlinks, names, and actions. Applications that need device nodes should express real dependencies instead of relying on sleep 5.

Read-Only rootfs Needs Writable State Somewhere Else

Embedded devices often use a read-only rootfs, such as squashfs, for power-loss tolerance, verification, and recovery.

But Linux user space always creates writable state:

  • logs
  • PID files
  • sockets
  • temporary files
  • device runtime data
  • user configuration
  • update state

These should not be written casually into the root filesystem. A common design is:

  • keep rootfs read-only
  • use tmpfs for /run and /tmp
  • put configuration, logs, and product data on a data partition
  • use overlayfs when writable system directories are required

If writable paths are not planned, the system may work on a development board but fail in a production image, a read-only image, or a power-loss test.

A Practical Debugging Order

For boot problems, locating the failed stage is more important than guessing a cause.

Use this order:

bootloader parameters
-> kernel command line
-> storage and partitions appear
-> rootfs mounts
-> init exists and is executable
-> dynamic linker and shared libraries exist
-> /proc /sys /dev /run are mounted
-> udev/devtmpfs works
-> service dependencies and application logs

Useful checks include:

  • find Kernel command line in the serial log
  • inspect root=, rootfstype=, init=, and console=
  • read VFS, EXT4-fs, squashfs, mmc, ubi, and mtd logs
  • mount rootfs from a rescue system or initramfs and inspect /sbin/init
  • use file to check the architecture of init
  • use readelf -l to inspect the dynamic linker path
  • check that /dev/console, /proc, /sys, and /run exist and are mounted as expected
  • compare bootloader partition parameters with the device-tree partition description

If the kernel has already panicked in the rootfs or init stage, the user-space service manager has not taken over yet. systemd, application, network, and business logs may have no diagnostic value.

rootfs Is the Handoff From Kernel to Product System

Embedded Linux boot is not complete when the kernel runs. A product system has to cross from the kernel into user space, then init has to organize services, device nodes, mount points, configuration, and applications.

rootfs sits exactly at that handoff.

Before it are the bootloader, kernel, device tree, storage, and filesystem drivers. After it are init, udev, systemd, application services, and field operations. Once rootfs is understood, many “the kernel booted but the system did not” failures become easier to separate: did rootfs fail to mount, did init fail to execute, were runtime directories missing, or were service dependencies expressed incorrectly?