During development, many embedded Linux systems use a writable ext4 rootfs. It is convenient: copy missing files, edit configuration, and write logs under /var/log.
In a product, that convenience turns into risk:
- power loss can corrupt system files
- temporary files, logs, and databases get mixed into rootfs
- updates cannot easily tell user changes from system files
- factory reset has unclear boundaries
- the system partition becomes dirty, making field issues hard to reproduce
The point of a read-only rootfs is not simply to prevent changes. It is to make boundaries explicit: system files should be verifiable and recoverable; runtime state should be disposable; user data should have a clear home; updates and factory reset should not guess which files matter.
A common model is:
read-only rootfs: programs, libraries, default config, units, rules
tmpfs: /run, /tmp, PIDs, sockets, temporary files
data partition: user config, logs, databases, business data
overlayfs: writable compatibility layer for paths that expect writes
This is much more suitable for long-running devices than making the whole / writable.
rootfs Should Be the System Baseline
rootfs contains the system baseline:
- programs under
/bin,/sbin, and/usr/bin - dynamic linker and shared libraries
- systemd units or init scripts
- udev rules
- default configuration
- baseline certificates
- directory layout and default permissions
These should be generated by the build system and managed by firmware versions, not casually changed on field devices.
A read-only rootfs gives immediate benefits:
- power loss is less likely to damage system files
- firmware versions are easier to verify
- field issues can be reproduced against a known baseline
- updates can replace the whole system file set
- factory reset does not risk deleting the operating system itself
This is why many devices use squashfs, read-only ext4, dm-verity, or signed verification schemes.
Writable State Needs an Owner
Linux user space always writes something. A read-only rootfs does not mean there are no writable files. It means each kind of write needs an owner.
A common classification is:
/run: runtime state for this boot
/tmp: temporary files
/var/log: logs
/var/lib/app: persistent application state
/etc: configuration
/home or /data: user and business data
Embedded devices may choose different paths, but the categories must not be mixed.
Disposable runtime state fits tmpfs:
- PID files
- Unix sockets
- lock files
- temporary caches
- one-shot intermediate files
Persistent state should live on a data partition or another clearly defined persistent path:
- user configuration
- device identity
- certificates and keys
- business databases
- collected data
- update state
- field logs
If applications write casually to /etc, /usr, or /var, they will eventually break under read-only rootfs, factory reset, or OTA updates.
tmpfs Handles Temporary Runtime State
tmpfs is an in-memory filesystem commonly used for /run and /tmp.
It fits data that is valid only during the current boot:
- service PIDs
- sockets
- lock files
- IPC temporary files
- small caches
It avoids flash wear and does not leave half-written files after power loss. The tradeoff is memory usage, and content disappears after reboot.
So tmpfs should not contain data that must persist. Putting databases, configuration, or critical logs into tmpfs may work briefly, then become “data lost after reboot” in the field.
The data Partition Holds Persistent State
The data partition usually stores information that must survive reboots.
Its boundary with rootfs should be clear:
rootfs: part of the system version, replaceable by firmware update
data: part of the device state, usually preserved across updates
Typical content includes:
- user configuration
- business databases
- device identity and keys
- logs
- downloaded update packages
- runtime statistics
- calibration data
The data partition needs its own power-loss and lifetime design: filesystem choice, write frequency, log rotation, fsync policy, database commit strategy, and behavior when space runs out.
A read-only rootfs does not remove risk. It concentrates state in the place designed to hold it and gives that place a recovery strategy.
overlayfs Provides Writable Compatibility
Some software expects to write /etc, /var, or other system paths. Rewriting all of it may not be practical, so overlayfs is often used.
overlayfs combines a read-only lower layer and a writable upper layer into one writable view:
lowerdir: read-only rootfs
upperdir: writable layer on data partition
workdir: overlayfs work directory
merged: merged directory seen by applications
Applications appear to modify /etc or /var, but writes land in upperdir. lowerdir remains read-only.
overlayfs is valuable for compatibility, but it adds new rules:
- upper files hide lower files with the same name
- deleting lower files creates whiteouts
- after a rootfs update, old upper files may hide new default files
- factory reset must clean the correct upperdir
- running out of upperdir space appears as system-path write failures
So overlayfs is not something to enable and forget. Decide which directories may be overlaid, how configuration is migrated on update, and what factory reset removes.
Separate Default Configuration From User Configuration
Configuration is where read-only rootfs designs often get messy.
A robust model is:
rootfs: default configuration
data: user or field configuration
boot: default configuration plus user overrides
Default configuration follows the firmware version. User configuration follows the device state. Updating the system can update defaults; factory reset can remove user overrides; field debugging can see what changed.
If an application directly edits /etc/app.conf in the system partition, several problems appear:
- it cannot write under a read-only rootfs
- OTA updates may overwrite user changes
- factory reset boundaries are unclear
- new defaults and old user values are hard to migrate
Applications should support a clear configuration precedence instead of hardcoding one mutable file in the system partition.
Logs and Caches Need Limits
After rootfs becomes read-only, logs usually move to the data partition or memory. Two boundaries matter.
First, space. Logs need rotation, quotas, and expiration. If data fills up, databases, configuration saves, and update downloads may all fail.
Second, flash lifetime. High-frequency logs, debug logs, and sampled data cannot be flushed forever without a policy. Production images need controlled log levels, write rates, and persistence rules.
Caches are similar. A cache can improve performance, but it must not become uncontrolled persistent state. Cache directories should be removable, and the system should rebuild them.
Power-Loss Recovery Comes From Filesystem Boundaries
Many field failures happen during power loss: configuration half-written, database transaction interrupted, update package partially downloaded, logs being flushed.
A read-only rootfs reduces system-file corruption, but it does not automatically make business data consistent. You still need:
- temporary files plus atomic rename for persistent writes
- version numbers, checksums, or duplicated records for critical state
- appropriate database transactions and sync policy
- an update state machine that can resume or roll back
- a recovery path if the data partition is damaged
- logs and caches that can be discarded without blocking boot
System partition read-only, data partition recoverable, temporary directories disposable. With these boundaries, devices are more likely to return to a runnable state after abnormal power loss.
Debugging Read-Only rootfs Issues
When something works on a development board but fails in the production image, inspect this chain:
is rootfs read-only
-> which paths does the application write
-> are /run and /tmp tmpfs
-> is the data partition mounted
-> is overlayfs upperdir correct
-> how are defaults and user config merged
-> do logs and caches have space limits
-> can data recover after power loss
Useful commands include:
mount
findmnt /
findmnt /run /tmp /data
findmnt -t overlay
df -h
df -i
journalctl -b
ls -la /etc /var /data
If a service fails only on the read-only image, check which paths it writes during startup. Often the program can run, but its default state path points into the system partition.
A Read-Only rootfs Is a Product Boundary
The essence of a read-only rootfs is not merely writing less to flash. It makes product boundaries clear.
System files belong to the firmware version. Runtime temporary state belongs to tmpfs. User and business state belongs to data. Compatibility writes to system paths can be handled by overlayfs. Updates, factory reset, power-loss recovery, and field debugging all depend on those boundaries.
With those boundaries in place, one configuration write, one log spike, or one abnormal power loss is far less likely to drag the whole system partition into an uncontrolled state.