Architecture problems are hard to debug by intuition.
A program reports Exec format error, and someone suspects permissions. A file exists but execution says not found, and someone suspects the path. A program hits an illegal instruction, and the compiler is blamed. A device faults immediately after reset, and application code is changed. Much later, the actual cause turns out to be a wrong ELF machine, missing dynamic linker, floating-point ABI mismatch, unsupported ISA extension, or a PC that was never inside the expected code section.
These problems should not be guessed. Start with evidence.
The most useful first model is this: first identify what the binary is, then identify which runtime it requests, then place PC, cause, and fault address from exception state back into the corresponding ISA, ABI, address-space, and entry-path context.
file / readelf: what this binary is
readelf -l / ldd: what runtime it needs
objdump: what instructions and symbols it contains
exception state: where the CPU was, why it entered, which address it touched
These tools do not fix bugs for you, but they put the problem into the right layer.
Start With file
file is the fastest first check.
file ./app
Look for:
- ELF or script
- 32-bit or 64-bit
- target architecture
- LSB or MSB
- dynamically or statically linked
- stripped or not
- rough interpreter information
Example:
ELF 64-bit LSB executable, ARM aarch64, dynamically linked, interpreter /lib/ld-linux-aarch64.so.1
This immediately answers common questions: did you copy an x86 program to an ARM board? Is this a 64-bit binary on a 32-bit user space? Is it dynamically linked? Is it a script with a missing interpreter?
When you see Exec format error, file is usually the first command.
Use readelf -h for ELF Header and Machine
readelf -h is more explicit than file.
readelf -h ./app
Important fields include:
Class: ELF32 or ELF64Data: little-endian or big-endianType: EXEC, DYN, and so onMachine: target architectureEntry point address: entry addressFlags: ABI or extension-related information on some architectures
If Machine does not match the target device, stop guessing. The kernel may refuse execution, or the CPU may later fault on unsupported instructions.
The entry address is also useful. For bare metal, bootloaders, or manually loaded programs, an entry address outside the expected code region points back to linker script, load address, or image format.
Use readelf -l for the Dynamic Linker
When a file exists but execution says not found, inspect INTERP.
readelf -l ./app
Look for:
[Requesting program interpreter: /lib/ld-linux-xxx.so.x]
A dynamically linked ELF is not loaded entirely by the kernel alone. The kernel first loads the interpreter named in INTERP. If that path does not exist in the target rootfs, the error may look like a missing executable.
Common causes include:
- app built with a glibc toolchain, rootfs uses musl
- dynamic linker path came from another SDK
- rootfs lacks the loader
- app copied without dependent libraries
- container or chroot paths differ from the target board
So not found does not always mean ./app is missing. The runtime requested by the ELF may be missing.
Use readelf -A for Architecture Attributes and ABI Clues
Many architectures store ABI or extension information in attributes.
readelf -A ./app
For ARM, common clues include:
- CPU architecture
- Thumb / ARM state attributes
- VFP or Advanced SIMD attributes
- hard-float / soft-float ABI clues
- EABI version
For RISC-V, common clues include:
Tag_RISCV_arch- RV32 or RV64
- extensions such as
m,a,f,d,c,v - ABI-related markers
These explain many “same ARM/RISC-V, still does not run” cases.
For example, the target CPU may not support a RISC-V extension used by the binary. The rootfs may use soft-float ABI while the app or library uses hard-float ABI. User-space bit width may not match kernel support.
readelf -A output varies by architecture and toolchain. Do not memorize every field. Know that this is where ISA, extension, and ABI evidence can appear.
ldd Shows Shared Libraries, but Mind Where It Runs
ldd can help inspect dynamic library dependencies.
ldd ./app
In cross environments, be careful: running host ldd on a target binary may be invalid or misleading. Prefer running it on the target board, or inspect dependencies through the target sysroot or cross-toolchain facilities.
At a lower level:
readelf -d ./app
Look for NEEDED entries.
Check:
- required libraries exist in rootfs
- library search paths allow the dynamic linker to find them
- C library matches the build environment
- libstdc++, libgcc_s, pthread, dl, rt, and related libraries are compatible
- symbol versions are available
Dynamic library debugging is not only about filenames. Libraries with the same name may still have different ABI, version, or C-library ecosystems.
objdump Shows Actual Instructions and Symbol Locations
objdump answers what is actually inside the binary.
Common commands:
objdump -d ./app
objdump -t ./app
objdump -h ./app
It can help determine:
- which instruction is near an address
- whether a fault PC is inside the expected function
- whether the binary contains unsupported instructions
- whether section layout matches the linker script
- whether entry symbols, exception vectors, and handlers are in expected locations
- whether handwritten assembly follows the calling convention
If runtime reports an illegal instruction, use the PC from exception state and find that instruction in objdump -d. This is usually better than guessing compiler flags.
For bare-metal or kernel images, you may need the right architecture option or an ELF with symbols. For raw binary images, you need the load address, or addresses will not line up.
Exception State Starts With PC, Cause, and Fault Address
A binary that loads can still enter exceptions at runtime.
For faults, oopses, panics, illegal instructions, page faults, bus faults, or alignment faults, first find:
- PC or EPC: which instruction caused the exception
- LR / return address: where it was called from
- cause / exception code: why the exception happened
- badaddr / tval / fault address: which address was accessed
- status / flags: privilege, interrupt state, return state
- sp: whether the stack looks reasonable
Names vary by architecture, but the questions are similar.
PC tells where the CPU was. Cause tells why it entered. Fault address tells what address triggered it. Status tells whether it happened in user mode, kernel mode, machine mode, handler mode, or another exception level.
Without these, it is easy to mistake permission problems for pointer bugs, illegal instructions for bad memory, or address-translation failures for driver logic bugs.
Map PC Back to Disassembly and Mappings
After obtaining PC, map it back to code.
For a user-space program with symbols:
addr2line -e ./app 0x...
objdump -d ./app
For kernels or bare metal, account for relocation:
- link address
- actual load address
- whether MMU is enabled at runtime
- whether PC is virtual or physical
- whether the symbol file matches the running image
Many “PC is not in code” cases are really address-base mismatches. A bootloader may load an image at one address while the ELF is linked for another. After the kernel enables the MMU, PC may be a virtual address. A stripped production binary may not match the local symbol file.
Classify by Symptom
Symptoms can guide the first branch.
Exec format error:
filereadelf -h- check ELF class, machine, endianness, and target kernel support
File exists but says not found:
readelf -l- inspect
INTERP - confirm dynamic linker path exists in rootfs
Illegal instruction after start:
- inspect exception PC
- find instruction with
objdump -d - check ISA extensions and ABI with
readelf -A - confirm target CPU supports them
Crash during library call:
- inspect stack trace and PC
- check ABI, floating-point ABI, stack alignment
- confirm library and app come from matching sysroot/rootfs
User-space segmentation fault:
- inspect fault address
- check permissions, mmap, null pointer, out-of-bounds access
- distinguish user virtual address from physical address
Bare-metal fault immediately after reset:
- check whether PC is in reset handler or expected code
- check vector table, stack pointer, link address, load address
- inspect fault cause and accessed address
DMA or driver data corruption:
- do not start with objdump
- check address type, DMA mapping, cache sync, barrier, and buffer ownership
Tool Output Must Be Interpreted With the Target System
Tools provide evidence, but evidence must be interpreted with the target system.
The same Machine: AArch64 may run on a glibc or musl rootfs. The same riscv64 may require different ISA extensions. The same dynamic-linker path may exist but point to incompatible libraries. The same PC may belong to a different address space before and after MMU is enabled.
Keep three sets of information together:
- binary information:
file,readelf,objdump - runtime information: rootfs, dynamic linker, shared libraries, kernel configuration
- exception state: PC, cause, fault address, privilege state, stack
When these line up, architecture problems narrow quickly.
What to Remember
file, readelf, objdump, and exception state are not afterthoughts. They are the first evidence for architecture-level bugs.
A practical order is:
- use
fileandreadelf -hto identify the binary - use
readelf -lto inspect the dynamic linker - use
readelf -Aand compiler flags to check ISA, extensions, and ABI - use
readelf -dor target-sidelddfor shared libraries - use
objdumpandaddr2lineto map PC back to code - use exception state to decide whether this is illegal instruction, permission, address translation, unaligned access, bus error, or wrong entry
Classify the problem as ISA, ABI, loader, rootfs, address-space, exception-entry, or DMA/cache before changing code. This is more reliable than guessing compiler flags, swapping toolchains, or adding logs.