Embedded Linux

Some embedded Linux failures are easy to misread:

• the same rootfs boots with one kernel but not another
• a driver module exists in rootfs, but the device is unavailable during boot
• no serial log appears, so the system looks dead
• after changing bootloader variables, the system mounts the wrong partition
• the application is unchanged, but device nodes appear in a different order

These problems are not always application or rootfs problems. Device behavior is strongly shaped by three things:

An embedded Linux device that can boot is not necessarily ready for product deployment.

During development, putting the bootloader, kernel, dtb, rootfs, application, and data together may work. The problems appear during updates, factory reset, abnormal power loss, partition damage, and field repair:

• a rootfs update overwrites user data
• the device tree changes but the bootloader still loads the old dtb
• a full data partition breaks system services
• rootfs is damaged and there is no recovery path
• power loss during OTA leaves no bootable system
• factory reset removes too much or too little

The point of partition layout is not simply “more partitions”. It separates the boot chain from data lifetimes:

During development, many embedded Linux systems use a writable ext4 rootfs. It is convenient: copy missing files, edit configuration, and write logs under /var/log.

In a product, that convenience turns into risk:

• power loss can corrupt system files
• temporary files, logs, and databases get mixed into rootfs
• updates cannot easily tell user changes from system files
• factory reset has unclear boundaries
• the system partition becomes dirty, making field issues hard to reproduce

The point of a read-only rootfs is not simply to prevent changes. It is to make boundaries explicit: system files should be verifiable and recoverable; runtime state should be disposable; user data should have a clear home; updates and factory reset should not guess which files matter.

Many embedded Linux projects start by assembling a system by hand: download a cross compiler, build the kernel, copy BusyBox, add libraries, package rootfs, and drop in the application.

That may boot a board, but it quickly breaks down in a product:

• which cross toolchain built this image
• which kernel config and device tree were used
• where each library in rootfs came from
• why the same application links differently on another machine
• whether production images, debug images, and SDKs can be generated from one configuration
• whether the same system can be rebuilt six months later

Buildroot and Yocto do not mainly solve “how to install a few packages”. They solve this: how to put the toolchain, bootloader, kernel, rootfs, packages, image, and SDK into a repeatable build system.

On embedded Linux devices, applications often fail only during boot: a serial port has not appeared, the NIC has no address yet, the data partition is not mounted, the camera node is missing, or permissions have not been applied.

The common patch is:

ExecStartPre=/bin/sleep 5

It may appear to work, but it turns dependencies into a timing guess. If storage enumeration is slower, DHCP takes longer, or a driver is probed later after -EPROBE_DEFER, the failure comes back.

The serial log may show a long Linux kernel boot, then end in a panic. The kernel may appear to boot, but the product application never starts. The log may say No working init found or VFS: Unable to mount root fs.

These failures are often treated as application startup problems. But before any application can run, Linux has to cross a more basic boundary: the kernel must find and mount rootfs, then execute the first user-space process.

A common Linux device-debugging problem looks like this: the driver code did not change, the kernel boots, but the device never probes. The log may only say probe failed, or the driver may never be entered at all.

The problem is not always in C code.

On many embedded Linux platforms, whether a driver can find a device and obtain register ranges, interrupts, clocks, power supplies, GPIOs, and DMA channels first depends on whether the board description is correct. On ARM, RISC-V, and many embedded platforms, that description is usually Device Tree.

When an embedded Linux device boots slowly, an application does not start, a driver does not load, or a network service fails, people often jump straight to application logs.

But the application is only the last part of the boot chain. After power-on, the CPU does not directly jump to business logic. It starts from a fixed entry, initializes the minimal hardware environment, enters the bootloader, loads the Linux kernel, Device Tree, and rootfs-related information, waits for the kernel to reach user space, and only then reaches the application.

When application code calls read(), write(), or ioctl(), it can look like the program is directly operating a device. Reading a UART, writing to a network interface, controlling GPIO, or accessing a sensor may all appear to be simple function calls.

But that path is not the application touching hardware directly.

On systems with an operating system, applications, the kernel, and drivers are separated by several boundaries: permission boundaries, address-space boundaries, system-call boundaries, device abstraction boundaries, and blocking semantics. Many application-driver debugging problems come from mixing these boundaries together.