Many concurrency bugs trigger the same first reaction: add a lock. That is only half right.
A lock can protect shared state, but it does not make concurrency problems disappear. It turns “multiple execution flows modify data at the same time” into rules about who enters first, who waits, and who releases. If those rules are poorly designed, the system may stop corrupting data and instead start hanging, stalling, timing out, or delaying high-priority work.
The safest first model is this: a lock does not protect code. It protects shared state so that only a specific execution flow modifies it during a critical section. Deadlock is a cycle in the wait relationship. Priority inversion is a high-priority execution flow indirectly blocked by a low-priority lock holder.
Shared state
-> take lock before entering critical section
-> read or modify state
-> release lock after leaving critical section
-> other execution flows may enter
The important question is not “is there a lock.” It is what the lock protects, how long it is held, and whether the holder waits for anything else.
A Lock Protects Shared State, Not Lines of Code
Locks are written around code, so it is easy to think they protect those lines. More accurately, they protect the shared state accessed by those lines.
For example, two threads update a counter:
read counter
counter = counter + 1
write counter
This looks like one increment, but it can contain separate read, compute, and write steps. If two threads interleave, both may read the same old value and only one increment is effectively recorded.
A lock turns that read-modify-write sequence into a critical section:
lock
read counter
counter = counter + 1
write counter
unlock
The point is not that these lines are sacred. The point is that counter will not be modified by another execution flow while this critical section is active.
So when designing locks, do not start with “should this code be locked.” Start with:
- which data can be accessed by multiple execution flows
- which accesses are reads and which modify state
- whether multiple fields must stay consistent together
- which paths must observe one consistent snapshot
- whether interrupts, threads, or tasks can access the same state
If the state boundary is unclear, locks are easily added too broadly, too narrowly, or in the wrong place.
A Mutex Solves Simultaneous Entry, Not Every Ordering Problem
The basic guarantee of a mutex is: only one execution flow enters the protected critical section at a time.
It does not automatically guarantee:
- waiting time is short
- lock ordering is correct
- code inside the lock never blocks
- object lifetime is correct
- multiple locks cannot wait on each other
For example, Thread A takes a lock and then waits for I/O. Thread B needs the same lock. Even if the mutual exclusion is correct, B’s response is now delayed by A’s I/O time.
Critical sections should usually be short and limited to work directly required for shared-state consistency.
Operations that usually do not belong inside a lock include:
- network requests
- file writes
- waiting for queue messages
- long computation
- calling unknown callbacks
- complex functions that may take other locks
These operations are not absolutely forbidden while holding a lock, but each one needs a clear reason. Many occasional stalls are not caused by the lock itself, but by doing too much serialized work under it.
Deadlock Comes From Cyclic Waiting
Deadlock is not “the program is slow.” It is a set of execution flows waiting on each other so that none can advance the condition needed by the others.
The classic example uses two locks:
Thread A: holds lock1, waits for lock2
Thread B: holds lock2, waits for lock1
Each side is waiting for the other to release the needed lock. Without timeout, cancellation, or external intervention, the wait will not end naturally.
The key is not “two locks.” The key is a cycle in the wait relationship:
A waits for a resource held by B
B waits for a resource held by C
C waits for a resource held by A
The resource does not have to be a mutex. It can be:
- a lock
- a semaphore
- queue space
- thread exit
- callback completion
- I/O completion
- state that can only be advanced by a blocked task
So when debugging deadlocks, do not only search for mutex. Draw who waits for whom, and who can release each condition.
Fixed Lock Ordering Is the Common Deadlock Prevention
If a system must take multiple locks at the same time, a fixed order is the most common prevention.
For example, all paths follow:
take device_lock first
then queue_lock
then stats_lock
If every path follows the same order, it becomes much harder to form a cycle where A holds device_lock and waits for queue_lock while B holds queue_lock and waits for device_lock.
The rule is simple. Maintaining it is the hard part:
- new paths must know the global lock order
- callbacks must not secretly take locks in reverse order
- code inside locks must not call complex functions that indirectly take locks
- error paths must also follow the order
Lock order should be an explicit constraint, not tribal memory. In complex modules, reducing the need to hold multiple locks at once is often more reliable than explaining deadlocks later.
Why Priority Inversion Happens
Priority inversion is common in RTOS and real-time systems, but it is not exclusive to RTOSes.
A typical case has three tasks:
Low-priority L takes a lock
High-priority H needs that lock and blocks
Medium-priority M does not need the lock but keeps running
L cannot get CPU and cannot release the lock
H keeps waiting for L
At first glance, high-priority H is blocked by low-priority L. The subtle part is M: it does not hold the lock, but it keeps preempting L and prevents L from reaching the release point.
That is priority inversion.
It shows that priority only chooses among ready tasks. If a high-priority task is blocked on a lock, it is not competing in the ready queue. Its response time is affected by both the lock holder and unrelated medium-priority work.
In real-time systems, a lock is not just a synchronization tool. It directly changes the worst-case response time of high-priority paths.
What Priority Inheritance Solves
Priority inheritance is a common mechanism for priority inversion.
When high-priority H waits for a lock held by low-priority L, the system temporarily raises L’s effective priority to H’s level, so L can run and release the lock sooner.
L holds lock
H waits for L's lock
-> L temporarily inherits H's priority
-> L preempts medium-priority M and releases lock
-> H gets lock and continues
-> L returns to original priority
It does not solve lock contention itself. It prevents unrelated medium-priority tasks from indefinitely delaying the lock holder.
But priority inheritance is not magic:
- long critical sections still delay H
- nested locks complicate analysis
- blocking I/O while holding a lock is still dangerous
- some synchronization primitives do not support priority inheritance
- wrong spinlock or interrupt masking use can still break real-time behavior
Priority inheritance reduces one class of inversion risk. It does not replace short critical sections and clear lock design.
Spinlocks and Mutexes Have Different Costs
Locks also differ in what the waiter does.
If a mutex cannot be acquired, the thread usually blocks and lets the scheduler run something else. It fits waits that may be longer.
If a spinlock cannot be acquired, the execution flow repeatedly checks in place and keeps occupying CPU. It fits very short waits where sleeping is not allowed or scheduling overhead is more expensive than the wait.
Mutex: cannot acquire -> sleep/block -> wake later
Spinlock: cannot acquire -> busy wait -> keep using CPU
On a single-core system, incorrect spinlock use is especially dangerous. If the lock holder must run on the same CPU to release the lock, while the waiter spins and occupies that CPU, the system can deadlock or degrade badly.
Interrupt context also requires care. Whether code may sleep, whether interrupts must be disabled, and whether interrupt and thread context can compete for the same state are system-specific rules. Treating all locks as the same thing causes many kernel and driver bugs.
Lock Granularity Can Be Too Coarse or Too Fine
A lock that is too coarse makes unrelated operations wait for each other.
For example, one global lock protects the entire device state: reading configuration, writing data, updating statistics, and handling callbacks all need it. This is simple, but any slow path delays every other path.
A lock that is too fine also creates problems:
- more locks
- more complex lock ordering
- higher deadlock risk
- harder consistency rules
- harder debugging of wait relationships
Good granularity usually comes from state boundaries, not source-code directories.
Useful questions include:
- which state must stay consistent together
- which paths are high frequency
- which paths may block or take long
- which paths are real-time response paths
- whether read-heavy state can use read-write locks, RCU, snapshots, or queues
The goal is not “fewer locks” or “finer locks.” The goal is clear access rules, bounded waiting, and analyzable failure paths.
Debug Lock Problems by Drawing Wait Relationships
When a system hangs, latency explodes, or a high-priority task stops responding, do not only look at the line taking the lock.
A better order is:
First, identify the lock holder. Which thread or task owns the lock, and where is it executing?
Second, identify waiters. What are their priorities, how long have they waited, and can they time out?
Third, inspect what happens while holding the lock. Is there I/O, queue waiting, callbacks, allocation, logging, or another lock acquisition?
Fourth, draw the wait graph. A waits for B, B waits for C, and does C wait for A?
Fifth, check priorities. Is a high-priority task waiting for a low-priority holder? Is medium-priority work delaying the release path?
Sixth, check interrupts and critical sections. Are interrupts disabled too long? Can interrupt context and thread context compete for the same state?
This information is much more useful than “add another lock.” A wrong lock can turn data corruption into a harder-to-reproduce hang.
What to Remember in Practice
The core of locking is not wrapping code. It is defining access rules for shared state.
A mutex solves simultaneous entry into a critical section. Deadlock comes from cyclic waiting. Priority inversion comes from a high-priority execution flow being indirectly blocked by a low-priority lock holder.
When writing concurrent code, the important questions are:
- what state does this lock protect
- how long is the critical section
- can the code block while holding the lock
- is there a fixed order for multiple locks
- can a high-priority path wait for a low-priority holder
- can interrupt context touch the same state
If these questions have clear answers, locks reduce risk. If they do not, locks often turn concurrency bugs into stalls and deadlocks that are harder to reproduce.