Why Kernel Configuration, Modules, and Boot Parameters Change Device Behavior

Some embedded Linux failures are easy to misread:

• the same rootfs boots with one kernel but not another
• a driver module exists in rootfs, but the device is unavailable during boot
• no serial log appears, so the system looks dead
• after changing bootloader variables, the system mounts the wrong partition
• the application is unchanged, but device nodes appear in a different order

These problems are not always application or rootfs problems. Device behavior is strongly shaped by three things:

kernel configuration: which capabilities are built into the kernel
kernel modules: which capabilities wait for user space or initramfs to load
boot parameters: which console, rootfs, init, and log policy this boot uses

The same rootfs can behave like a different device when paired with another kernel config, module layout, or kernel command line.

built-in and module Are About Timing

Linux kernel functionality can be built in or built as a module.

built-in: linked into the kernel image and available during kernel boot
module: stored in rootfs or initramfs and loaded later

This is not merely a code organization preference. It changes when the feature becomes available.

If a capability is needed before rootfs is mounted, it usually cannot be only an ordinary module. Examples include:

• the storage controller driver for rootfs
• the filesystem driver for rootfs
• base support required to access initramfs
• early console
• some buses, clocks, or power controls needed during boot

If eMMC, UBI/UBIFS, ext4, squashfs, or NVMe support exists only in /lib/modules inside rootfs, the kernel cannot mount rootfs to read those modules.

The usual fix is:

• build drivers required for mounting rootfs into the kernel
• or use initramfs to load those modules before switching to the real rootfs

So do not only check whether a .ko file exists. Check when it must be available.

Modules Must Match the Running Kernel

Kernel modules are not ordinary shared libraries. They depend on the running kernel version, configuration, exported symbols, and build options.

Common failures include:

invalid module format
Unknown symbol ...
version magic ...
disagrees about version of symbol ...

Typical causes are:

• /lib/modules/$(uname -r) in rootfs does not match the running kernel
• modules were built with another kernel tree or configuration
• CONFIG_MODVERSIONS is enabled and symbol versions do not match
• device tree or platform resources changed, so the module loads but probe fails
• OTA updated the kernel but did not update modules

In a product image, kernel, dtb, and rootfs module directories should be treated as one consistent artifact set. Replacing only one piece easily creates field failures.

Boot Parameters Control This Boot

The kernel command line is passed by the bootloader. It affects console, rootfs, init, logging, memory, and driver behavior.

Common parameters include:

console=ttyS0,115200
root=/dev/mmcblk0p2
rootfstype=ext4
rootwait
init=/sbin/init
loglevel=7
earlycon
panic=5

These are not minor debug switches. They decide which boot path the system follows.

When debugging boot, the Kernel command line printed in the serial log is important. Do not only trust what the bootloader configuration was supposed to pass. Check what the kernel actually received.

console= Decides Where Logs and Login Go

console= selects the kernel console. No serial output does not necessarily mean the system is not booting. The console may be wrong.

Common issues include:

• UART numbering changes, such as ttyS0 becoming ttyS2
• device-tree aliases change console names
• baud rate mismatch
• multiple console= entries, where the last one controls /dev/console
• early boot needs earlycon, but only normal console is configured
• the cable is connected to different pins or pinmux is wrong

console= affects both kernel log output and user-space /dev/console. If it is wrong, a running device can look dead.

root= Decides Which rootfs Is Mounted

root= tells the kernel which root filesystem to mount. It can be a device node, UUID, PARTUUID, NFS path, or another form.

Common issues include:

• partition numbers change, making /dev/mmcblk0p2 unstable
• after A/B switch, root= still points to the old slot
• bootloader variables and device-tree partition descriptions disagree
• the rootfs device enumerates slowly, but rootwait is missing
• rootfstype= does not match the filesystem
• initramfs switches to the real rootfs using stale configuration

On product devices, raw paths such as /dev/mmcblk0p2 should be used carefully. PARTUUID, labels, or a value passed by the bootloader/OTA state machine are usually more robust.

init= Changes the First User-Space Process

init= tells the kernel which first user-space program to execute.

By default, the kernel tries paths such as /sbin/init, /etc/init, /bin/init, and /bin/sh. init=/bin/sh is useful for rescue, but it may bypass the normal init system in production.

A wrong init= can mean:

• systemd or init scripts never run
• /proc, /sys, /dev, and /run are not mounted properly
• udev does not start
• services do not start
• permissions, device nodes, and mount dependencies are wrong

Rescue parameters should be managed separately from production parameters. Temporary debug settings should not leak into release images.

loglevel= and Early Logs Decide What You Can See

loglevel= affects kernel message verbosity. Fewer logs do not mean nothing happened; they may simply be filtered.

Some logs occur before the normal console is ready. Seeing them may require:

• earlycon
• earlyprintk
• correct UART address and baud rate
• bootloader preserving serial setup
• the related early console options enabled in the kernel

When a board looks stuck with no output, first verify the log channel. Otherwise “no logs” is easily mistaken for “not running”.

Module Loading Order Changes Device Timing

Modules are loaded through several paths:

• initramfs scripts
• systemd-modules-load
• udev automatic loading from modalias
• application or scripts calling modprobe
• kernel subsystems calling request_module

Loading order affects when device nodes appear, when drivers probe, and when services can start.

For example:

• if udev has not started, modules that rely on autoloading may appear late
• a driver may return -EPROBE_DEFER while waiting for another provider
• a service may start before the module loads and fail to find /dev/xxx
• OTA may update modules but leave stale depmod data
• blacklist or module parameters may change driver behavior

Applications should not assume that a module file existing means the device is ready. Express the real dependency through systemd and udev.

initramfs Changes the Relationship Between Modules and rootfs

Without initramfs, the kernel must already contain everything needed to mount rootfs.

With initramfs, early user space can run first:

• load storage or filesystem modules
• wait for devices
• decrypt or verify rootfs
• select an A/B slot
• inspect or repair data
• mount the real rootfs and switch_root

This makes the system more flexible, but it adds another boot stage. A failure may occur:

before kernel starts
inside initramfs
while switching to real rootfs
after init in the real rootfs

Debugging starts by identifying the stage. Otherwise initramfs scripts, real rootfs, and kernel config problems get mixed together.

Treat kernel, dtb, modules, and cmdline as One Set

Embedded Linux boot artifacts should not be upgraded in isolation.

At minimum, keep these consistent:

• kernel image
• dtb / dtbo
• /lib/modules/$(uname -r)
• initramfs
• bootloader command line
• rootfs fstab, udev rules, and systemd units
• OTA slot and partition configuration

A typical accident is updating rootfs without updating the kernel in boot, or updating the kernel while leaving the old module directory in rootfs. The system may boot, but drivers, device nodes, and services become inconsistent.

Version them as one artifact set.

Debugging Order

When behavior changes after a kernel update, inspect this chain:

actual kernel version
-> kernel config
-> command line
-> matching dtb
-> correct root= and init=
-> required drivers built-in or modules
-> /lib/modules/$(uname -r) matches
-> initramfs loads the right modules
-> udev/systemd loads modules as expected
-> when application device nodes appear

Useful commands include:

uname -a
cat /proc/cmdline
zcat /proc/config.gz
ls /lib/modules
modinfo module_name
lsmod
dmesg
journalctl -b
findmnt /

If the device cannot boot and only bootloader and serial access remain, check console=, root=, rootwait, init=, and the real partition layout first.

Kernel Configuration and Boot Parameters Are Product Interfaces

Kernel configuration, modules, and boot parameters are often treated as low-level implementation details. In embedded Linux products, they are part of the system interface.

They decide which drivers are available before rootfs, whether modules match the kernel, where logs appear, which rootfs is mounted, which process becomes the first user-space process, and when services can see devices.

The same application and rootfs can behave differently when these boundaries change. Managing kernel, dtb, modules, command line, and rootfs as one artifact set is a basic requirement for stable embedded Linux boot.