On embedded Linux devices, applications often fail only during boot: a serial port has not appeared, the NIC has no address yet, the data partition is not mounted, the camera node is missing, or permissions have not been applied.
The common patch is:
ExecStartPre=/bin/sleep 5
It may appear to work, but it turns dependencies into a timing guess. If storage enumeration is slower, DHCP takes longer, or a driver is probed later after -EPROBE_DEFER, the failure comes back.
systemd and udev affect device application startup because they sit on the critical user-space path:
kernel probe
-> devtmpfs creates device nodes
-> udev applies rules, permissions, and symlinks
-> systemd handles mount / device / network / service dependencies
-> device application starts
Reliable startup depends not only on whether the binary exists, but also on whether its devices, filesystems, network, and permissions are ready.
A Device Node Existing Is Not Always Enough
After a Linux driver probes successfully, user space often accesses the device through /dev/xxx, such as:
/dev/ttyS0/dev/video0/dev/watchdog0/dev/input/event0/dev/i2c-1
Those nodes may be created by devtmpfs and then processed by udev rules. devtmpfs is the kernel-provided mechanism for basic device nodes. udev is a user-space rule engine that can change permissions, create symlinks, tag devices, and trigger systemd units.
At least three states should be separated:
driver probed
-> /dev node appeared
-> udev rule finished
Applications often depend on more than a node name. They may need permissions, stable names, and related initialization actions.
For example, if a camera application fails to open /dev/video0, the cause may be:
- the node has not appeared yet
- the node exists but permissions are wrong
- the stable symlink created by udev is not ready
- the application starts before firmware, clocks, or subdevices are ready
- multiple similar devices changed order, so
/dev/video0is another device
If all of these are reduced to “the device is not ready”, the only available tool becomes a delay.
systemd Startup Order Is Not Filename Order
systemd does not run scripts by filename order. It builds a startup graph from dependencies and ordering constraints.
Common relationships include:
Requires=: this unit requires another unit; if the other one fails, this one fails tooWants=: this unit wants another unit, but failure is not necessarily fatalAfter=: ordering only; this unit starts after the other unit’s start jobBefore=: ordering only; this unit starts before the other unitBindsTo=: bind lifecycle to another unit, commonly a device
A frequent mistake is:
After=network.target
This does not mean the network is usable. After= is ordering, not a health check. network.target usually means the network management stack has reached a point, not that IP, routes, and DNS are ready.
Similarly, After=dev-ttyS0.device only helps if that .device unit matches the real device and if ordering is the dependency the service needs.
device Units Express “Wait for This Device”
systemd exposes many devices as .device units. A node such as /dev/ttyS0 often maps to a unit like:
dev-ttyS0.device
If an application really depends on that device, the unit can say:
[Unit]
Requires=dev-ttyS0.device
After=dev-ttyS0.device
If the service should stop when the device disappears, use a lifecycle binding:
BindsTo=dev-ttyS0.device
After=dev-ttyS0.device
This is more stable than sleep 5 because it waits for an event, not a guessed duration.
.device units are not universal solutions. Many applications depend on udev-created symlinks, permissions, or a compound device being ready. In those cases, udev rules, systemd units, and application dependencies should be designed together.
Mount Dependencies Decide Whether Data and Config Are Usable
Many device applications need a data partition, configuration partition, certificate directory, or log directory during startup.
If those paths are not mounted yet, an application may:
- create directories in the wrong rootfs location
- read factory default configuration
- fail to write logs
- write state into tmpfs and lose it after reboot
- exit because a database or certificate is missing
systemd can express these relationships with mount units and path dependencies. If a service needs /data:
[Unit]
RequiresMountsFor=/data
After=local-fs.target
If the application needs a real mount point, depend on that mount point instead of looping inside the application waiting for files to appear.
Embedded devices must be especially clear about read-only rootfs, overlayfs, and data partitions. A writable path on a development board may not be writable in the production image.
network.target Does Not Mean the Network Is Usable
Network startup is easy to misread.
Separate these states:
network.target: the network management stack has reached a targetnetwork-online.target: the network manager believes the network is online- interface presence: whether
eth0orwlan0exists - address configuration: whether DHCP or static IP setup has completed
- routing and DNS: whether business traffic can actually connect
An MQTT, HTTP, or OTA service with only After=network.target does not reliably wait for a usable network.
A better design is:
- enable the wait-online service provided by the network manager
- depend on
network-online.targetwhen startup really needs it - still handle connection failures, retries, and DNS failures inside the application
- avoid treating “server unreachable at boot” as a permanent service startup failure
Unit dependencies can express boot order, but business networking needs runtime retry. Field networks, APs, SIM cards, DHCP, DNS, and servers can change after boot.
udev Rules Are Mainly for Names, Permissions, and Triggers
udev commonly helps device applications in three ways.
First, stable naming. Multiple USB serial adapters, cameras, or storage devices can appear in different orders. Hardcoding /dev/ttyUSB0 or /dev/video0 is fragile. udev can create stable symlinks based on vendor, product, serial, or physical path:
/dev/serial/by-id/...
/dev/camera/front
/dev/modem/main
Second, permissions. If an application runs as a non-root user, device nodes need the right owner, group, or mode. Otherwise the service may start, then fail when it opens the device.
Third, triggers. A udev event can tag a device for systemd, start a service, or instantiate a template service for that device.
udev rules should not contain heavy business logic. They are good at device events, names, and permissions. Longer initialization belongs in a systemd service where lifecycle, logs, and restarts can be managed.
sleep 5 Hides the Real Dependency
Adding a delay can diagnose a race, but it does not describe what the service needs.
Its problems are:
- fast devices waste boot time
- slow devices still fail
- different hardware batches have different timing
-EPROBE_DEFER, DHCP, and filesystem repair take unpredictable time- services do not react when a dependency disappears later
- logs do not show what the service was actually waiting for
Prefer explicit dependencies:
- device nodes:
.deviceunits, udev tags, stable symlinks - mount points:
RequiresMountsFor= - online network:
network-online.targetplus application retry - permissions: udev rules, groups, and service
User=/Group= - other services:
Requires=,Wants=, andAfter=
Delays are useful temporary probes. They should not become the product startup design.
A Debugging Order for Boot Failures
When a device application fails during boot, split the path:
driver probed
-> /dev node appeared
-> udev rule finished
-> permissions and symlinks are correct
-> mount completed
-> network meets business requirements
-> systemd unit dependencies are correct
-> application has retry and fallback behavior
Useful commands include:
systemctl status your.service
systemctl list-dependencies your.service
systemctl list-units '*.device'
systemd-analyze critical-chain your.service
udevadm info /dev/xxx
udevadm monitor --kernel --udev
journalctl -u your.service -b
findmnt /data
ip addr
ip route
resolvectl status
For intermittent boot failures, inspect the relative order of device events, mounts, network state, and unit logs from the failed boot. Looking only at the final state after a successful boot often hides the race.
Reliable Startup Comes From Explicit Dependencies
systemd and udev are not incidental details around a device application. They decide how user space sees devices, how permissions are applied, how services are ordered, and how mounts and network state affect startup.
Much of embedded Linux startup reliability comes from turning hidden assumptions into explicit dependencies.
Wait for device events instead of time. Depend on mount points instead of paths that may or may not be backed by the right filesystem. Treat network as a runtime condition with retries. Let udev handle names and permissions. Express service relationships in units. The boot sequence becomes understandable because it says what it is waiting for.