When an RTOS device resets in the field, hits a random HardFault, jumps into invalid code, or corrupts queue data, engineers often suspect pointers, concurrency, peripherals, or power.
Those are valid suspects, but one common cause is simpler: a task stack is too small.
RTOS tasks usually do not have a large process address space or strong isolation like Linux processes. Each task owns a stack region whose size is often fixed at task creation. Make it too large and RAM is wasted. Make it too small and the failure may not happen immediately; the stack may silently overwrite nearby memory.
The first model is:
each task has its own stack
-> function calls, local variables, and saved context consume stack
-> interrupts and exceptions may use task stacks or a separate interrupt stack
-> insufficient stack overflows
-> without strong isolation, overflow can corrupt other objects
RTOS stack bugs are hard because they often do not crash at the moment of overflow. They corrupt something else and fail later in another module.
A Task Stack Is a Fixed Budget
When creating an RTOS task, a stack size is usually specified.
That stack has to hold:
- function call frames
- return addresses
- saved registers
- local variables
- function arguments
- compiler temporaries
- context-switch save area
- extra exception or interrupt context on some systems
So stack usage cannot be estimated only from how small the task source code looks. It depends on call depth, local object size, library functions, compiler optimization, logging paths, and exception paths.
A task that usually uses 300 bytes may use much more on an error path. The dangerous paths are often rare: error handling, logging, protocol parse failure, reconnect, factory reset, or OTA rollback.
Large Local Variables Kill Small Stacks
Large local variables are a common way to destroy an RTOS stack.
Dangerous examples:
void task(void *arg)
{
uint8_t packet[1500];
char logbuf[512];
char json[1024];
}
These arrays live on the task stack. A few nested calls can exceed the stack budget.
Safer options include:
- move large buffers to static storage, global objects, memory pools, or heap
- reuse fixed-size message buffers
- parse protocols in chunks
- avoid repeated temporary arrays deep in the call chain
- stress-test third-party libraries for stack usage
Moving large objects away from the stack is not free. Global buffers need concurrency control, heap allocation needs fragmentation and failure handling, and pools need exhaustion policy. The point is to avoid invisible large locals consuming every task stack.
printf and Logging Consume Stack
Many stack overflows appear only after logs are enabled.
Logging paths can add stack usage:
- formatting functions have deep call chains
- floating-point formatting can be expensive
- timestamps, task names, colors, and prefixes add processing
- log backends may use queues, locks, drivers, or filesystems
- assertions and error logs often run when the stack is already low
So both “adding logs makes the bug disappear” and “adding logs makes it crash” are plausible. Logging changes timing and stack usage.
A production logging design should know:
- maximum stack usage per log path
- whether floating-point printing is allowed
- whether printing is allowed in interrupts
- whether a dedicated logging task is used
- whether formatting buffers are on stack or static storage
- whether small-stack high-priority tasks restrict log levels
Do not casually print complex formats from tiny high-priority tasks.
Separate Interrupt Stack From Task Stack
Different RTOSes and CPU architectures handle interrupt stacks differently.
Some systems use the current task stack for interrupts. Some have a separate interrupt stack. Some exception entries first push a hardware frame and then switch to an interrupt stack.
This changes stack sizing:
interrupts use current task stack
-> every task stack needs worst-case interrupt nesting margin
interrupts use a separate interrupt stack
-> task stacks are lighter, but interrupt stack must be sized
If interrupt nesting is allowed, or ISR code calls heavy functions, the interrupt stack can overflow too.
Interrupt paths should avoid complex logging, protocol parsing, large local arrays, and blocking waits. They affect real-time behavior and can overflow the interrupt stack or current task stack.
High-Water Mark Means “What Happened So Far”
Many RTOSes provide a stack high-water mark showing the minimum remaining stack observed so far.
It is useful, but easy to misread.
Limitations include:
- only paths that have executed are reflected
- untested error paths are not counted
- compiler optimization, log switches, and library versions change usage
- whether interrupts count depends on the stack model
- fill-pattern detection can be damaged by out-of-bounds writes
If a task shows 200 bytes remaining, that does not mean it is always safe. One error-path printf, deep JSON parser, or exception handler may consume that margin.
Use high-water marks under stress tests and fault injection, then keep safety margin.
Stack Overflow May Not Crash Immediately
With MPU or stack guards, overflow may trigger a fault. Without protection, overflow may corrupt nearby memory:
- another task stack
- TCB or task control block
- queue control structures
- heap metadata
- global variables
- driver state
This is why RTOS stack issues are hard. Task A overflows, but task B crashes. Queue metadata is corrupted, but the visible symptom is that the communication task stops receiving messages.
If the system supports stack guards, MPU regions, canaries, or overflow hooks, enable them where possible. Without hardware protection, use fill patterns, high-water marks, assertions, and periodic checks to reduce blind spots.
Static vs Dynamic Allocation
RTOS task stacks may be statically or dynamically allocated.
Static allocation:
- makes memory layout known at boot
- avoids runtime heap fragmentation
- is easier for worst-case analysis
- fits safety-critical tasks
The cost is fixed RAM reservation and possible waste.
Dynamic allocation is flexible and allows tasks to be created on demand. The cost is:
- dependence on heap availability
- fragmentation over long runtime
- task creation failure paths
- correct cleanup on task deletion
- harder field reproduction
In product devices, critical tasks often use static allocation. Temporary tasks can be dynamic only with clear limits and failure behavior.
Stack Size Is Not a Guess
Sizing a task stack requires checking:
- normal main loop
- deepest call chain
- largest local variables
- heaviest logging path
- error handling paths
- callbacks and library functions
- interrupt or exception frames
- compiler optimization options
- differences between debug and production logging
A practical flow:
estimate from call chain and locals
-> enable stack fill and high-water mark
-> run stress tests, error paths, and long-duration tests
-> inspect minimum remaining stack
-> add safety margin
-> confirm again on production build
Do not treat a high-water mark from a short normal-path test as final evidence.
Field Debugging Order
When RTOS task stack problems are suspected, inspect:
which tasks still had heartbeat
-> which task ran last before crash
-> high-water marks for all tasks
-> large local arrays
-> complex logging or floating-point printf
-> whether interrupts use task stack or separate stack
-> whether error paths and rare paths were tested
-> whether canary, MPU, or overflow hooks captured evidence
Also check whether the crash happens in an unrelated module. If queues, heap, or global state are randomly corrupted, do not only inspect the damaged object. Check nearby task stacks for overflow.
Task Stack Is a Reliability Budget
An RTOS task stack is not a number casually filled in at task creation. It is a reliability budget for every execution path.
More tasks mean more total stack memory. Stacks that are too small may fail much later in another module. Logging, error paths, interrupt nesting, third-party libraries, and compiler options all change the budget.
A stable RTOS system does not only ask whether tasks can start. It keeps checking how much stack remains under worst-case paths, whether overflow can be detected, and whether the field device can preserve evidence when it happens.