Mutexes in the Linux Kernel

Mutexes in the Linux Kernel

A mutex is the kernel’s ordinary sleeping mutual-exclusion lock. It allows exactly one task to hold it at a time, it records which task that is, and it puts contending tasks to sleep instead of keeping a CPU busy. In Linux, the concrete type is struct mutex; the word “mutex” is also used generically for any lock that provides mutual exclusion, so the distinction matters when reading code and documentation.

The ownership record is the defining feature. A semaphore initialized to one also provides mutual exclusion, but it has no idea who acquired it. A mutex does. That single fact enables ownership validation on unlock, lockdep’s ownership-aware diagnostics, owner-directed handoff to a waiter, and — on AArch64 — optimistic spinning, where a contender briefly watches the owner run before deciding to sleep.

This document describes the non-PREEMPT_RT implementation in Linux v7.3-rc7, with particular attention to SMP AArch64. Other architectures and the RT implementation appear only where they clarify a design choice.

---

Table of Contents

1. The Problem: Why a Mutex Exists
   • 1.1 A Mutex Is Not a Binary Semaphore
2. Rules, Guarantees, and Failure Modes
   • 2.1 The Rules as a Checklist
   • 2.2 What the Debugging Options Actually Enforce
   • 2.3 Failure Modes in Practice
3. Mutex, Semaphore, and Spinlock
   • 3.1 Mutex Versus Semaphore
   • 3.2 Mutex Versus Spinlock
   • 3.3 The AArch64 Spinlock Comparison
4. The struct mutex Data Structure
   • 4.1 The owner Field
   • 4.2 Ownership State Machine
   • 4.3 The wait_lock Spinlock
   • 4.4 The OSQ Optimistic-Spin Queue
   • 4.5 The Waiter Object and first_waiter
   • 4.6 magic and lockdep’s dep_map
   • 4.7 blocked_on and blocked_lock
5. Initialization and Lifetime
   • 5.1 Static Initialization
   • 5.2 Dynamic Initialization
   • 5.3 Device-Managed Initialization
   • 5.4 mutex_destroy()
   • 5.5 Lifetime Rules
   • 5.6 A Minimal Heap-Allocated Example
6. The Public API
   • 6.1 Lock and Unlock
   • 6.2 mutex_trylock()
   • 6.3 mutex_lock_interruptible() and mutex_lock_killable()
   • 6.4 mutex_lock_io()
   • 6.5 mutex_lock_nested() and Nest Locks
   • 6.6 mutex_is_locked() and mutex_get_owner()
   • 6.7 atomic_dec_and_mutex_lock()
   • 6.8 Cleanup Guards
   • 6.9 What CONFIG_DEBUG_LOCK_ALLOC Changes
7. The Lock Path: Fast, Optimistic, and Slow
   • 7.1 Fast Path
   • 7.2 Mid Path: Adaptive Optimistic Spinning
   • 7.3 Slow Path: Enqueue, Set Blocked-On State, Sleep
   • 7.4 The Error Path
   • 7.5 Handoff Is Requested by a Waiter, Not by the Owner
   • 7.6 Fairness Is Eventual, Not Strict FIFO
8. The Unlock Path and Handoff
   • 8.1 Fast Unlock
   • 8.2 Slow Unlock: Clearing the Owner or Preserving Handoff
   • 8.3 Selecting the Next Task
   • 8.4 The Handoff Operation
   • 8.5 Handoff, Fairness, and Latency
   • 8.6 Lock Lifetime During Unlock
9. Ownership, Debugging, and lockdep
   • 9.1 What the Fast Path Checks
   • 9.2 The Magic Pointer
   • 9.3 A Safe “Unlock Without Lock” Demonstration
   • 9.4 Recursive Locking
   • 9.5 lockdep Annotations
   • 9.6 Context Diagnostics
   • 9.7 Hung-Task and Tracepoint Visibility
   • 9.8 What lockdep Does Not Do
10. AArch64 Hardware Behavior
    • 10.1 LL/SC Versus LSE
    • 10.2 Acquire and Release, Precisely
    • 10.3 Why the Owner Word Is a Single Cache Line
    • 10.4 YIELD, WFE, and Power
    • 10.5 The Internal wait_lock
    • 10.6 AArch64-Specific Practical Implications
11. Scheduling, Fairness, and Contention
    • 11.1 The Three Contention Regimes
    • 11.2 What Bounds the Spin
    • 11.3 A Fairness Timeline
    • 11.4 Sleeping and Wakeup
    • 11.5 Priority Inversion
    • 11.6 Starvation and Latency
    • 11.7 How Contention Affects Throughput
12. Worked Examples
    • 12.1 Two Kernel Threads Contending on One Mutex
    • 12.2 Non-Blocking Acquisition
    • 12.3 A Character-Device Read Path
    • 12.4 Observing the Lock State
    • 12.5 Device-Managed Mutex
    • 12.6 A Reference-Counted Release Pattern
13. WW-Mutex: Deadlock Avoidance
    • 13.1 The Problem
    • 13.2 Wait-Die and Wound-Wait
    • 13.3 The Transaction State Machine
    • 13.4 Data Structures
    • 13.5 How the Conflict Detection Works
    • 13.6 The Fast Path and the Context Publication Barrier
    • 13.7 API
    • 13.8 A Correct Multi-Lock Retry Pattern
    • 13.9 WW-Mutex Rules
14. PREEMPT_RT Contrast
15. Performance and Alternatives
    • 15.1 Cost Model
    • 15.2 Choosing an Alternative
    • 15.3 Reducing Contention
    • 15.4 When Not to Use a Mutex
16. Summary
17. References

---

1. The Problem: Why a Mutex Exists

Semaphores arrived first, and they are still the right tool for counting resources and for producer/consumer signaling. For many years, however, they were also the kernel’s only general sleeping lock. The common pattern was to initialize a semaphore to one and use it as a sleeping mutual-exclusion lock. That worked mechanically, but it left the kernel unable to reason about ownership.

A semaphore has no owner field. Any task — or an interrupt handler calling up() — can release a semaphore that a different task acquired. The kernel cannot detect the mistake, lockdep cannot validate release ownership, and the scheduler cannot identify a holder to boost when priority inversion occurs.

That gap creates several concrete problems:

• Priority inversion without a remedy. A low-priority task holds the semaphore; a high-priority task blocks on it; a medium-priority task preempts the holder. The high-priority task waits for an unbounded time because the kernel does not know which task should inherit its priority.
• Wrong-unlocker and double-unlock bugs. Error paths can call up() without a matching down(), or release from a different task. The semaphore silently accepts it, and the next waiter may enter a critical section with no protection.
• Interrupt-context ambiguity. A blocking semaphore acquisition cannot be used in interrupt context because it may sleep, but nothing in the semaphore object records or enforces that rule. The failure appears later as a hang.
• No ownership-aware diagnostics. Lockdep can model the dependency graph, but it cannot tell whether the task releasing a semaphore is the task that acquired it.

The mutex was introduced to make these mistakes either impossible or detectable. The central mechanism is the owner field: struct mutex stores a pointer to the owning task_struct, with a few low bits used as state flags. Ownership enables:

• an implicit check in the fast unlock path and an explicit diagnostic in the slow path;
• priority inheritance when the mutex is backed by an rtmutex under PREEMPT_RT;
• lockdep call-site and ownership tracking;
• optimistic spinning while the owner is actually running; and
• a direct handoff to the next waiter.

1.1 A Mutex Is Not a Binary Semaphore

It is tempting to describe a mutex as “a binary semaphore with an owner.” The first half is misleading. A mutex is binary in the sense that it has one logical holder, but its interface and semantics are deliberately narrower than a semaphore’s:

• only the owner may unlock it;
• it cannot be used for signaling between tasks;
• it cannot represent a pool of resources;
• it cannot be acquired from interrupt context; and
• recursive acquisition is forbidden.

Those restrictions are the feature. They turn a class of silent coordination bugs into warnings, lockdep reports, or explicit API failures.

For the complementary primitive — counting, signaling, and cross-task release — see the general semaphore documentation. In this document, “mutex” means struct mutex unless stated otherwise.

---

2. Rules, Guarantees, and Failure Modes

The non-RT contract is stated compactly in include/linux/mutex_types.h:

/*
 * Simple, straightforward mutexes with strict semantics:
 *
 * - only one task can hold the mutex at a time
 * - only the owner can unlock the mutex
 * - multiple unlocks are not permitted
 * - recursive locking is not permitted
 * - a mutex object must be initialized via the API
 * - a mutex object must not be initialized via memset or copying
 * - task may not exit with mutex held
 * - memory areas where held locks reside must not be freed
 * - held mutexes must not be reinitialized
 * - mutexes may not be used in hardware or software interrupt
 *   contexts such as tasklets and timers
 */

There is one additional lifetime rule that is easy to miss: the mutex must remain valid until mutex_unlock() returns. The unlock path can touch the lock after it has logically released ownership — it may wake a waiter, inspect flags, or perform a handoff. The source spells this out in kernel/locking/mutex.c:

/*
 * The caller must ensure that the mutex stays alive until this function has
 * returned - mutex_unlock() can NOT directly be used to release an object such
 * that another concurrent task can free it.
 * Mutexes are different from spinlocks & refcounts in this aspect.
 */

2.1 The Rules as a Checklist

Rule	Consequence of violation
One holder at a time	Mutual exclusion is lost; data corruption follows
Only the owner unlocks	Fast-path CAS fails; slow path warns only with debugging enabled, then behavior is undefined
No recursive locking	The task waits for itself, typically until a soft-lockup or hung-task report
Initialize through DEFINE_MUTEX or mutex_init	Zeroed or copied state is not a valid mutex
Do not memset or copy an initialized mutex	The owner, wait list, and lockdep state become inconsistent
Do not reinitialize a held mutex	Waiters and ownership state are destroyed
Do not exit while holding the mutex	The lock is never released; waiters hang
Do not free memory containing a held or waited-on mutex	Use-after-free in the wake or handoff path
Do not free the object until mutex_unlock() returns	The unlock path may still touch the lock
No interrupt, softirq, tasklet, timer, or NMI use	Sleeping in atomic context; might_sleep() may warn, then the kernel hangs or crashes
Do not use mutex_trylock() in interrupt context either	The mutex infrastructure assumes process context even when it does not sleep

2.2 What the Debugging Options Actually Enforce

The options are complementary, not interchangeable:

Configuration	What it adds	What it does not guarantee
CONFIG_DEBUG_MUTEXES	Magic-pointer checks, waiter poisoning, slow-path owner warning, mutex_destroy() checks	Does not by itself build a full lock dependency graph; recursive locking can still hang
CONFIG_PROVE_LOCKING / lockdep	Dependency graph, recursive-lock detection, ABBA detection, context and call-site tracking	Does not replace correct runtime ownership checks
CONFIG_DEBUG_ATOMIC_SLEEP	might_sleep() checks for atomic/interrupt context	Does not prove that the lock is held for a safe duration
CONFIG_DEBUG_LOCK_ALLOC	Lockdep maps and ownership annotations; API wrappers route through lockdep	Adds overhead; the non-lockdep fast path is not compiled in this configuration
CONFIG_DEBUG_WW_MUTEX_SLOWPATH	Fault injection for ww-mutex wound/die paths	Intended for testing, not production

A common misconception is that CONFIG_DEBUG_MUTEXES alone detects recursive locking. The current debug code checks the magic pointer, poisons waiter objects, verifies the task is not already blocked, and warns on an ownership mismatch during slow unlock; it does not perform a complete recursive-lock analysis. Lockdep is the tool that catches recursion before the atomic operation. A recursive mutex_lock() in a kernel with only CONFIG_DEBUG_MUTEXES can spin or sleep until the soft-lockup or hung-task detector fires.

When DEBUG_LOCKS_WARN_ON() does fire, it prints the condition text and a stack trace, and turns off further lock debugging. It does not automatically print a symbolic lock name or both task names; lockdep reports may contain more context, but the bare mutex warning should not be expected to.

2.3 Failure Modes in Practice

• Atomic-context call. might_sleep() reports it in debug kernels. In production, the slow path may schedule with preemption disabled or interrupts disabled, producing a hang or a scheduler assertion.
• Signaling misuse. A mutex cannot replace a completion or semaphore. If task A must wake task B and transfer ownership, use the appropriate signaling primitive.
• Counting misuse. A mutex protects one object or one serialized state transition. A pool of N resources needs a counting semaphore, a bitmap plus a lock, or another explicit allocator.
• Recursive locking. The owner waits for a lock it already holds. Optimistic spinning can make this worse by keeping the CPU busy; lockdep is the reliable detector.
• ABBA deadlock. Two code paths take the same pair of mutexes in opposite orders. Use a documented global order, or a ww-mutex when the order is genuinely data-dependent.
• Priority inversion. The non-RT mutex has no priority inheritance. A high-priority waiter can be delayed by a medium-priority task that preempts the holder.
• Wrong unlocker. The fast path fails its compare-and-swap and falls to the slow path. Without debugging, the slow path does not repair the situation; it may corrupt the owner or wait-list state.
• Unbounded hold time. A mutex held across I/O, page faults, or another sleeping lock can block all waiters for an unbounded period. Keep critical sections short and do not acquire a mutex while holding a spinlock.
• Free-after-unlock race. mutex_unlock() may still access the lock. A concurrent free before unlock returns is a use-after-free, even if the lock word already looks unlocked.

---

3. Mutex, Semaphore, and Spinlock

3.1 Mutex Versus Semaphore

Property	Semaphore	Mutex
Ownership	None	Strict: only the acquiring task may unlock
Count	Any non-negative integer	Binary: one logical holder
Release from another task	Allowed	Forbidden
Acquisition in interrupt context	Forbidden (may sleep)	Forbidden
Release in interrupt context	Possible for signaling (up()) with care	Forbidden
Priority inheritance basis	No owner to boost	Owner known; full PI under PREEMPT_RT
Adaptive spinning	No	Yes, with CONFIG_MUTEX_SPIN_ON_OWNER
Recursive acquisition	Usually deadlocks; not diagnosed	Forbidden; lockdep diagnoses it
lockdep integration	Basic	Ownership-aware, with call sites
Counting resource pool	Yes	No
Producer/consumer signaling	Yes	No
Wrong-unlocker detection	None	Debug-time warning; lockdep may diagnose

A semaphore’s up() can be called from interrupt context when it is being used purely as a wakeup/signaling mechanism, provided the surrounding code obeys the semaphore’s own rules. down() cannot. A mutex forbids both acquisition and release in interrupt context because its implementation, ownership model, and wakeup path all assume process context.

Choose a mutex for ordinary mutual exclusion. Choose a semaphore only when the release side is genuinely different from the acquire side, or when a count greater than one is required. Do not choose a semaphore merely because it looks simpler; the mutex’s ownership checks and debugging support are usually worth more than the small amount of flexibility.

3.2 Mutex Versus Spinlock

The difference is what a contender does while waiting. A spinlock keeps the CPU busy; a mutex yields it. Everything else follows from that.

Property	Spinlock	Mutex
Wait behavior	Busy-waits	Sleeps and context-switches
Interrupt context	Yes (raw spinlock for NMI/low-level use)	No
Process context	Yes	Yes
Preemption while held	Disabled	Allowed
IRQ disabling	Optional (spin_lock_irqsave)	Not applicable
Priority inheritance	No	No on non-RT; yes on PREEMPT_RT via rtmutex
Ownership enforcement	None	Strict
Optimistic spinning	N/A	Yes, as a bounded pre-sleep phase
Long hold times	Waste CPU cycles	Appropriate
Very short hold times	Appropriate	Works, but heavier
Sleeping while held	Illegal	Allowed
Recursive lock	Silent deadlock	Forbidden; lockdep diagnoses
lockdep support	Yes	Yes, with ownership tracking

Decision guide
=============

  Must this code run in interrupt, softirq, NMI, or with preemption
  already disabled?
       │
      YES ──► use a spinlock (raw_spinlock_t where required)
       │
       NO
       ▼
  May the critical section sleep (copy_to_user, kmalloc(GFP_KERNEL),
  I/O, another mutex, wait_event, ...)?
       │
      YES ──► use a mutex
       │
       NO
       ▼
  Is the protected region only a few instructions, with no cache miss,
  and is contention expected to be rare?
       │
      YES ──► a spinlock may be cheaper
       │
       NO
       ▼
  Is the data also accessed from interrupt context?
       │
      YES ──► spinlock with irqsave/irqrestore, or split the design
       │
       NO ──► a mutex is usually the safer default

The key rule is simple: a sleeping lock must not be used in interrupt context, and a spinlock must not be held across a sleep. When a design needs both properties, split the state into an interrupt-safe part protected by a spinlock and a process-context part protected by a mutex. Do not hold a spinlock while acquiring a mutex, and do not hold a mutex while acquiring a spinlock unless the lock order is documented and unavoidable.

3.3 The AArch64 Spinlock Comparison

On AArch64, a normal spinlock_t is implemented on top of the generic queued spinlock (qspinlock) infrastructure. The architecture selects queued spinlocks in arch/arm64/Kconfig, and the raw lock word is manipulated with LL/SC or LSE atomics depending on the CPU. A mutex uses the same atomic primitives for its owner word, but it adds sleeping, ownership, wait queues, and optimistic spinning. The mutex’s wait_lock is itself a raw spinlock, so a contended mutex slow path briefly uses the spinlock machinery to protect its waiter list.

A uniprocessor kernel compiles spinlocks down to preemption control rather than a busy-wait loop, because there is no other CPU to make progress. A mutex still sleeps on a uniprocessor build when the owner is blocked, because sleeping is about yielding the CPU, not about waiting for another CPU.

---

4. The struct mutex Data Structure

The non-RT definition is in include/linux/mutex_types.h:

context_lock_struct(mutex) {
	atomic_long_t		owner;
	raw_spinlock_t		wait_lock;
#ifdef CONFIG_MUTEX_SPIN_ON_OWNER
	struct optimistic_spin_queue osq;	/* Spinner MCS lock */
#endif
	struct mutex_waiter	*first_waiter __guarded_by(&wait_lock);
#ifdef CONFIG_DEBUG_MUTEXES
	void			*magic;
#endif
#ifdef CONFIG_DEBUG_LOCK_ALLOC
	struct lockdep_map	dep_map;
#endif
};

context_lock_struct(mutex) is an internal declaration macro. In the non-RT build it expands to struct mutex; under PREEMPT_RT it expands to a different layout backed by struct rt_mutex_base (include/linux/mutex_types.h). Callers should never depend on the macro or on the field layout — use the public API.

The layout divides naturally into a hot path and a cold path:

                    struct mutex
┌─────────────────────────────────────────────────────────────┐
│ owner        (atomic_long_t, 8 bytes on AArch64)            │  fast lock/unlock
├─────────────────────────────────────────────────────────────┤
│ wait_lock    (raw_spinlock_t / qspinlock)                   │  slow path only
├─────────────────────────────────────────────────────────────┤
│ osq          (optimistic_spin_queue, optional)              │  contended spin
├─────────────────────────────────────────────────────────────┤
│ first_waiter (struct mutex_waiter *, guarded by wait_lock)  │  sleeping waiters
├─────────────────────────────────────────────────────────────┤
│ magic        (CONFIG_DEBUG_MUTEXES)                         │  debug only
├─────────────────────────────────────────────────────────────┤
│ dep_map      (CONFIG_DEBUG_LOCK_ALLOC)                      │  lockdep only
└─────────────────────────────────────────────────────────────┘

The exact byte offsets and cache-line placement depend on the configuration and architecture. What matters is that the uncontended fast path touches owner, while the contended path brings in wait_lock, the wait list, and (when enabled) the OSQ.

4.1 The owner Field

owner is an atomic_long_t: 64 bits on AArch64. In the uncontended case it contains the address of the owning task_struct. The low bits of that pointer are free because task_struct is aligned to at least L1_CACHE_BYTES; on AArch64 that is 64 bytes, and the current kernel declares the structure with 64-byte alignment (include/linux/sched.h). Six low bits are therefore free, but the mutex reserves only the lowest three for state.

The flags are defined in kernel/locking/mutex.h:

#define MUTEX_FLAG_WAITERS	0x01
#define MUTEX_FLAG_HANDOFF	0x02
#define MUTEX_FLAG_PICKUP	0x04

#define MUTEX_FLAGS		0x07

Bit	Name	Meaning
0	MUTEX_FLAG_WAITERS	The wait list is non-empty; unlock must wake at least one waiter.
1	MUTEX_FLAG_HANDOFF	Unlock must transfer the lock directly to the top waiter instead of simply clearing it.
2	MUTEX_FLAG_PICKUP	A handoff has been performed; the designated task still has to pick up the lock.

The bit layout is:

 63                                                    3   2   1   0
┌─────────────────────────────────────────────────────┬───┬───┬───┬───┐
│              task_struct pointer (bits 63:3)        │ 0 │ P │ H │ W │
└─────────────────────────────────────────────────────┴───┴───┴───┴───┘
                                                          │   │   │
                                             PICKUP ──────┘   │   │
                                             HANDOFF ─────────┘   │
                                             WAITERS ─────────────┘

  0x0000...0000  = unlocked, no waiters
  task|0         = owned, no waiters
  task|W         = owned, waiters present
  task|W|H       = owned, top waiter requested handoff
  task|W|P       = handoff completed; task is the designated next owner
  flags-only|W   = transient state after owner clearing, waiters still present

The helper that strips the flags is __mutex_owner():

static inline struct task_struct *__mutex_owner(struct mutex *lock)
{
	if (!lock)
		return NULL;
	return (struct task_struct *)(atomic_long_read(&lock->owner) & ~MUTEX_FLAGS);
}

The transient flags-only|W state is why mutex_is_locked() is not a synchronization primitive. A mutex can have waiters while the owner word no longer names a task; mutex_is_locked() reports only whether a task pointer is present, not whether the wait list is empty or a handoff is in progress.

4.2 Ownership State Machine

The owner word is easiest to understand as a small state machine:

                 CAS(0 -> current)
  ┌──────────┐ ───────────────────► ┌──────────────┐
  │ UNLOCKED │                      │    OWNED     │
  │  owner=0 │ ◄─────────────────── │    task|0    │
  └──────────┘   unlock, no waiters └──────────────┘
                                           │
                                           │ waiter queued
                                           ▼
                                     ┌──────────────┐
                                     │  OWNED+WAIT  │
                                     │    task|W    │
                                     └──────────────┘
                                           │
                                           │ first waiter sets H
                                           ▼
                                     ┌──────────────┐
                                     │ OWNED+WAIT+H │
                                     │   task|W|H   │
                                     └──────────────┘
                                           │
                                           │ unlock hands off
                                           ▼
                                     ┌──────────────┐
                                     │   HANDOFF    │
                                     │   next|W|P   │
                                     └──────────────┘
                                           │
                                           │ next picks up
                                           ▼
                                     next becomes owner
                                     (PICKUP cleared)

The important transitions are:

1. Acquire: CAS(0 -> current) succeeds only from the fully unlocked state.
2. First waiter: the waiter is appended and MUTEX_FLAG_WAITERS is ORed into the owner word.
3. Handoff request: the first waiter cannot acquire immediately and sets MUTEX_FLAG_HANDOFF on the current owner.
4. Handoff: unlock replaces the owner task with the designated next task and sets PICKUP; it also preserves WAITERS and clears HANDOFF.
5. Pickup: the designated task’s __mutex_trylock() sees PICKUP, clears it, and acquires the lock without contending on a zero owner word.

The flags are manipulated with atomic operations because a fast-path contender can race with the slow path. The __mutex_trylock_common() loop (kernel/locking/mutex.c) is the authoritative implementation of the state transitions.

4.3 The wait_lock Spinlock

wait_lock is a raw spinlock that protects three things:

• the first_waiter list and the MUTEX_FLAG_WAITERS bit;
• the HANDOFF/PICKUP handoff protocol; and
• the synchronization between a woken waiter and the unlock path.

It is taken with IRQs disabled (raw_spin_lock_irqsave) because the unlock path may run in contexts where interrupts are already disabled, and because the wakeup/handoff protocol must not be interrupted by a lock-taking interrupt handler on the same CPU.

wait_lock is a raw spinlock, not a sleeping lock. It is held only for short bookkeeping operations: adding or removing a waiter, selecting the next owner, and clearing blocked-on state. The lock is released before waking the next task.

        mutex->wait_lock                         task->blocked_lock
        ─────────────────                        ───────────────────
        first_waiter list                        blocked_on
        WAITERS / HANDOFF / PICKUP               blocked_donor
        wait-list ordering                       scheduler-visible donor link
        handoff vs. wakeup serialization         unlock vs. scheduler serialization

The two locks protect different state and are never interchangeable. wait_lock serializes every task that manipulates this mutex’s wait list; blocked_lock serializes the task’s own blocked-on relationship with the scheduler and hung-task detector.

4.4 The OSQ Optimistic-Spin Queue

When CONFIG_MUTEX_SPIN_ON_OWNER is enabled, the mutex contains an optimistic spin queue (osq). The OSQ is an MCS-style queue tailored for sleeping locks. It prevents a stampede of optimistic spinners from all polling the same owner cache line.

Without OSQ: N spinners poll one hot cache line

  CPU0 ──┐
  CPU1 ──┤
  CPU2 ──┼──► mutex->owner       (coherence traffic grows with N)
  CPU3 ──┤
  CPU4 ──┘


With OSQ: each non-waiter spinner polls its own per-CPU node

  CPU0 (head) ──► node0.locked ──► watches mutex->owner
  CPU1         ──► node1.locked ◄── CPU0 signals when it leaves
  CPU2         ──► node2.locked ◄── CPU1 signals when it leaves
  CPU3         ──► node3.locked ◄── CPU2 signals when it leaves

  Only the head of the OSQ polls the owner cache line.

There is an important exception: a task that is already on the mutex wait list does not take the OSQ. When it becomes the first waiter, it may spin directly on the owner concurrently with the OSQ head. The kernel source calls this the “waiter-spinner” case (kernel/locking/mutex.c):

/*
 * The waiter flag is set to true if the spinner is a waiter in the wait
 * queue. The waiter-spinner will spin on the lock directly and concurrently
 * with the spinner at the head of the OSQ, if present, until the owner is
 * changed to itself.
 */

So the OSQ does not guarantee that exactly one CPU ever polls owner; it guarantees that the non-waiter optimistic spinners are serialized. The wait-list mechanism independently limits the number of queued waiter-spinners.

The OSQ node is per-CPU and lives in a shared-aligned per-CPU area (kernel/locking/osq_lock.c):

struct optimistic_spin_node {
	struct optimistic_spin_node *next, *prev;
	int locked; /* 1 if lock acquired */
	int cpu;    /* encoded CPU # + 1 value */
};

static DEFINE_PER_CPU_SHARED_ALIGNED(struct optimistic_spin_node, osq_node);

The queue tail is an atomic_t initialized to OSQ_UNLOCKED_VAL (include/linux/osq_lock.h). osq_lock() is at kernel/locking/osq_lock.c; osq_unlock() is at kernel/locking/osq_lock.c.

4.5 The Waiter Object and first_waiter

first_waiter points to the head of the wait list. The waiter objects are not allocated from a slab; they live on the blocked task’s kernel stack (kernel/locking/mutex.h):

struct mutex_waiter {
	struct list_head	list;
	struct task_struct	*task;
	struct ww_acquire_ctx	*ww_ctx;
#ifdef CONFIG_DEBUG_MUTEXES
	void			*magic;
#endif
};

The list is a circular list_head list. first_waiter is the head; list_add_tail(&waiter->list, &first->list) appends behind the head, and list_next_entry() walks toward the tail. The __guarded_by(&wait_lock) annotation on the pointer is a static-analysis hint that all accesses must hold wait_lock.

The stack allocation is safe because a waiter cannot return from mutex_lock() until it has removed itself from the list and debug_mutex_free_waiter() has poisoned the object. It also means a waiter must not be moved or freed while blocked.

4.6 magic and lockdep’s dep_map

With CONFIG_DEBUG_MUTEXES, magic points back to the mutex at initialization (kernel/locking/mutex-debug.c). The key entry points check lock->magic == lock; a mismatch means the object was freed, never initialized, or overwritten. mutex_destroy() clears the magic and warns if the mutex is still locked (kernel/locking/mutex-debug.c).

With CONFIG_DEBUG_LOCK_ALLOC, dep_map connects the mutex to lockdep. It records the lock class, the acquisition call site, and the wait type (LD_WAIT_SLEEP). The fast path is omitted when lockdep maps are enabled, because every acquisition and release must be reported to the dependency checker; this is why a lockdep kernel routes mutex_lock() through mutex_lock_nested() and ultimately the slow path.

4.7 blocked_on and blocked_lock

The mutex also participates in the scheduler’s blocked-on graph. Those fields are stored in task_struct, not in the mutex (include/linux/sched.h):

struct mutex			*blocked_on;	/* lock we're blocked on */
raw_spinlock_t			blocked_lock;

/*
 * The task that is boosting this task; a back link for the current
 * donor stack. Set in schedule() -> find_proxy_task() and only stable
 * under preempt_disable().
 */
struct task_struct		*blocked_donor;

blocked_on lets the hung-task detector identify the lock blocking a task. When CONFIG_SCHED_PROXY_EXEC is enabled, it also lets the scheduler find a runnable owner and temporarily execute it in the waiter’s scheduling context; this is proxy execution, a form of priority donation. The mutex slow path sets and clears blocked_on under the task’s blocked_lock, and the unlock path may hand the lock to a designated donor. These fields are not part of the locking algorithm’s mutual-exclusion guarantee, but they are part of the current implementation’s scheduling and diagnostics behavior.

---

5. Initialization and Lifetime

A mutex has exactly two supported initialization paths: static definition with DEFINE_MUTEX and dynamic initialization with mutex_init() (or its keyed variant). Both paths initialize the same state; there is no valid “zero it and use it” shortcut.

5.1 Static Initialization

The macros are defined in include/linux/mutex.h:

#ifndef CONFIG_PREEMPT_RT
#define __MUTEX_INITIALIZER(lockname) \
		{ .owner = ATOMIC_LONG_INIT(0) \
		, .wait_lock = __RAW_SPIN_LOCK_UNLOCKED(lockname.wait_lock) \
		, .first_waiter = NULL \
		__DEBUG_MUTEX_INITIALIZER(lockname) \
		__DEP_MAP_MUTEX_INITIALIZER(lockname) }

#define DEFINE_MUTEX(mutexname) \
	struct mutex mutexname = __MUTEX_INITIALIZER(mutexname)
#endif

DEFINE_MUTEX(my_lock) declares a mutex object and initializes it at compile time. It is suitable for module globals and static file-scope objects:

static DEFINE_MUTEX(device_lock);

The initializer sets owner to zero, initializes the raw spinlock, sets first_waiter to NULL, and (when configured) installs the debug magic and lockdep map. The OSQ’s zero-initialized tail is the unlocked value, so no separate OSQ initializer is needed in the static literal.

5.2 Dynamic Initialization

The public macro is also in include/linux/mutex.h:

#define mutex_init(mutex)						\
do {									\
	static struct lock_class_key __key;				\
									\
	__mutex_init((mutex), #mutex, &__key);				\
} while (0)

mutex_init() creates a static lock_class_key at the call site, which gives lockdep a stable class identity even when the mutex is embedded in many dynamically allocated objects. The internal initialization path depends on the configuration:

mutex_init(lock)
      │
      ▼
__mutex_init(lock, "lock", &__key)
      │
      ├── CONFIG_DEBUG_LOCK_ALLOC=y ──► mutex_init_lockdep()
      │                                      ├─ __mutex_init_generic()
      │                                      └─ lockdep_init_map_wait()
      │
      └── CONFIG_DEBUG_LOCK_ALLOC=n ──► mutex_init_generic()
                                             │
                                             ▼
                                       __mutex_init_generic()
                                             ├─ owner = 0
                                             ├─ raw_spinlock_init(wait_lock)
                                             ├─ first_waiter = NULL
                                             ├─ osq_lock_init(osq)       [if spinning]
                                             └─ debug_mutex_init()      [if DEBUG_MUTEXES]

__mutex_init_generic() is at kernel/locking/mutex.c:

static void __mutex_init_generic(struct mutex *lock)
{
	atomic_long_set(&lock->owner, 0);
	scoped_guard (raw_spinlock_init, &lock->wait_lock) {
		lock->first_waiter = NULL;
	}
#ifdef CONFIG_MUTEX_SPIN_ON_OWNER
	osq_lock_init(&lock->osq);
#endif
	debug_mutex_init(lock);
}

mutex_init_with_key() is available when the caller needs to supply the lock_class_key explicitly. The keyed form is useful for initialization wrappers that want all instances to share a class.

5.3 Device-Managed Initialization

devm_mutex_init(dev, lock) ties destruction to a device’s lifetime. It is a macro around __devm_mutex_init() (include/linux/mutex.h):

#define devm_mutex_init(dev, mutex) \
	__devm_mutex_init(dev, __mutex_init_ret(mutex))

When CONFIG_DEBUG_MUTEXES is disabled, mutex_destroy() is a no-op, so __devm_mutex_init() simply returns zero; there is nothing to register with devres. With debugging enabled, it registers a devres action that calls mutex_destroy() when the device is removed. The helper does not allocate the mutex; the caller still supplies storage.

5.4 mutex_destroy()

mutex_destroy() marks a mutex unusable. It is a no-op without CONFIG_DEBUG_MUTEXES; with debugging, it warns if the mutex is locked and clears magic (kernel/locking/mutex-debug.c):

void mutex_destroy(struct mutex *lock)
{
	DEBUG_LOCKS_WARN_ON(mutex_is_locked(lock));
	lock->magic = NULL;
}

Destroying a locked mutex is a bug even if the debug warning is compiled out: waiters may still reference the object, and the memory is about to become invalid. The check uses mutex_is_locked(), which reports owner presence but not every transient waiter-only state, so it is a diagnostic aid rather than a complete lifetime proof. Call mutex_destroy() only after the last unlock has returned and no waiter can still access the mutex. It does not free the memory; the owner of the containing object does that.

5.5 Lifetime Rules

The safe lifecycle is:

allocate storage
      │
      ▼
mutex_init()  or  DEFINE_MUTEX()
      │
      ▼
mutex_lock() / mutex_unlock()  (repeatedly; same task for each pair)
      │
      ▼
ensure no waiter is blocked and no unlock is in flight
      │
      ▼
mutex_destroy()        (debug builds: marks magic invalid)
      │
      ▼
free storage

The ordering at the end is not optional. mutex_unlock() can wake a task, perform a handoff, or touch the owner word after the logical release. Freeing the storage as soon as the lock word reads zero is a use-after-free. The same rule applies to a mutex embedded in a reference-counted object: drop the final reference only after unlock has returned and all waiters have been accounted for.

5.6 A Minimal Heap-Allocated Example

#include <linux/mutex.h>
#include <linux/slab.h>

struct my_state {
	struct mutex lock;
	int value;
};

static int __init my_init(void)
{
	struct my_state *state;

	state = kmalloc(sizeof(*state), GFP_KERNEL);
	if (!state)
		return -ENOMEM;

	mutex_init(&state->lock);
	mutex_lock(&state->lock);
	state->value = 42;
	mutex_unlock(&state->lock);

	mutex_destroy(&state->lock);
	kfree(state);
	return 0;
}

This example intentionally does all its work in init and then frees the state. In real code the state would live as long as the device or subsystem that owns it, and mutex_destroy() would run from the corresponding teardown path. The important detail is that mutex_destroy() and kfree() happen only after the final mutex_unlock() has returned.

---

6. The Public API

The public declarations are in include/linux/mutex.h. The table below summarizes the common entry points; the sections that follow explain the less obvious semantics.

API	Behavior	Context	Return
mutex_lock(lock)	Acquire, sleeping uninterruptibly	Process	void
mutex_unlock(lock)	Release; must be same task	Process	void
mutex_trylock(lock)	Non-blocking attempt	Process	1 success, 0 contention
mutex_lock_interruptible(lock)	Acquire, interruptible by any signal	Process	0 or -EINTR
mutex_lock_killable(lock)	Acquire, interruptible by fatal signal	Process	0 or -EINTR
mutex_lock_io(lock)	Acquire; account wait as I/O wait	Process	void
mutex_lock_nested(lock, subclass)	Acquire with lockdep subclass	Process	void
mutex_lock_nest_lock(lock, nest_lock)	Acquire with an explicit nest lock	Process	void
mutex_trylock_nest_lock(lock, nest_lock)	Non-blocking, with nest annotation	Process	1/0
mutex_is_locked(lock)	Snapshot of owner presence	Any (diagnostic)	bool
mutex_get_owner(lock)	Return owner pointer as unsigned long	Any (diagnostic)	unsigned long
mutex_destroy(lock)	Mark unusable	Process/teardown	void
devm_mutex_init(dev, lock)	Initialize and register teardown	Process	int
atomic_dec_and_mutex_lock(cnt, lock)	Decrement; lock only if it reaches zero	Process	int (1/0)

6.1 Lock and Unlock

mutex_lock() is at kernel/locking/mutex.c. It calls might_sleep(), attempts the single-CAS fast path, and falls back to __mutex_lock_slowpath():

void __sched mutex_lock(struct mutex *lock)
{
	might_sleep();

	if (!__mutex_trylock_fast(lock))
		__mutex_lock_slowpath(lock);
}

mutex_unlock() is at kernel/locking/mutex.c. It has its own fast path and then calls __mutex_unlock_slowpath():

void __sched mutex_unlock(struct mutex *lock)
{
#ifndef CONFIG_DEBUG_LOCK_ALLOC
	if (__mutex_unlock_fast(lock))
		return;
#endif
	__mutex_unlock_slowpath(lock, _RET_IP_);
}

With CONFIG_DEBUG_LOCK_ALLOC, the fast unlock is omitted so that every release is reported to lockdep. In a non-lockdep build, the fast path handles the common case: owner == current and no flags set.

6.2 mutex_trylock()

mutex_trylock() is at kernel/locking/mutex.c. It returns 1 on success and 0 on contention, following the spin_trylock() convention, not the down_trylock() convention:

if (mutex_trylock(&dev->lock)) {
	/* acquired */
	mutex_unlock(&dev->lock);
} else {
	/* busy: do not touch the protected state */
}

The non-blocking property does not make it interrupt-safe. The mutex implementation still assumes process context, and the lock must be released by the same task. Use a raw spinlock or a dedicated atomic protocol for interrupt-context trylock behavior.

6.3 mutex_lock_interruptible() and mutex_lock_killable()

mutex_lock_interruptible() (kernel/locking/mutex.c) sleeps in TASK_INTERRUPTIBLE and returns -EINTR if any signal is pending:

if (mutex_lock_interruptible(&dev->lock))
	return -ERESTARTSYS;
/* critical section */
mutex_unlock(&dev->lock);

mutex_lock_killable() (kernel/locking/mutex.c) uses TASK_KILLABLE: only a fatal signal interrupts the wait. Regular signals are ignored for the purpose of breaking the lock wait. Both return -EINTR on interruption and must be paired with an unlock only on success.

mutex_lock_killable() is common in filesystem paths that must remain killable but should not abort on ordinary job-control signals. mutex_lock_interruptible() is common in character-device read()/write()/ioctl() handlers where a user process should be able to abort a long wait.

6.4 mutex_lock_io()

mutex_lock_io() (kernel/locking/mutex.c) marks the time spent waiting for the mutex as I/O wait:

void __sched mutex_lock_io(struct mutex *lock)
{
	int token;

	token = io_schedule_prepare();
	mutex_lock(lock);
	io_schedule_finish(token);
}

The sleep state is still uninterruptible; the difference is scheduler accounting. The wait contributes to the task’s I/O-wait statistics rather than ordinary sleep time. It does not make the critical section asynchronous, and it does not change the ownership rules.

6.5 mutex_lock_nested() and Nest Locks

With CONFIG_DEBUG_LOCK_ALLOC, mutex_lock() is a macro for mutex_lock_nested(lock, 0) (include/linux/mutex.h). The subclass tells lockdep that two locks of the same class may nest in a known order:

mutex_lock_nested(&child->lock, SINGLE_DEPTH_NESTING);

Without subclasses, lockdep sees two acquisitions of the same class and may report a false recursive-locking or ABBA problem. The subclass is a statement about the locking hierarchy, not a runtime lock ordering mechanism; it does not make an unsafe order safe. Without CONFIG_DEBUG_LOCK_ALLOC, the nested and nest-lock macros collapse to the corresponding plain lock calls, so the annotations have no runtime effect.

mutex_lock_nest_lock(lock, nest_lock) uses another lock’s dep_map as the nesting context (include/linux/mutex.h). It is for cases where the held lock, rather than a numeric subclass, defines the nesting relationship. mutex_trylock_nest_lock() is the non-blocking counterpart.

6.6 mutex_is_locked() and mutex_get_owner()

mutex_is_locked() is a diagnostic snapshot (kernel/locking/mutex.c):

bool mutex_is_locked(struct mutex *lock)
{
	return __mutex_owner(lock) != NULL;
}

It returns true only when the owner word contains a task pointer. It does not report a wait list, a handoff, or the transient owner-cleared state. Never use it to decide whether it is safe to access protected data; that decision must come from actually holding the lock.

mutex_get_owner() returns the owner pointer as an unsigned long (kernel/locking/mutex.c). The source explicitly warns: “Do not use the return value as a pointer directly.” It is intended for diagnostics and debugging, not for dereferencing without additional lifetime guarantees.

6.7 atomic_dec_and_mutex_lock()

atomic_dec_and_mutex_lock() is a convenience for reference-counted objects that only need the mutex on the final reference (kernel/locking/mutex.c):

int atomic_dec_and_mutex_lock(atomic_t *cnt, struct mutex *lock)
{
	/* dec if we can't possibly hit 0 */
	if (atomic_add_unless(cnt, -1, 1))
		return 0;
	/* we might hit 0, so take the lock */
	mutex_lock(lock);
	if (!atomic_dec_and_test(cnt)) {
		/* when we actually did the dec, we didn't hit 0 */
		mutex_unlock(lock);
		return 0;
	}
	/* we hit 0, and we hold the lock */
	return 1;
}

It returns 1 with the mutex held only when the decrement reaches zero. The caller is responsible for unlocking in that case. The fast decrement path does not take the mutex, which is the point of the helper.

6.8 Cleanup Guards

The header defines scope guards for the mutex API (include/linux/mutex.h):

DEFINE_LOCK_GUARD_1(mutex, struct mutex, mutex_lock(_T->lock), mutex_unlock(_T->lock))
DEFINE_LOCK_GUARD_1_COND(mutex, _try, mutex_trylock(_T->lock))
DEFINE_LOCK_GUARD_1_COND(mutex, _intr, mutex_lock_interruptible(_T->lock), _RET == 0)
DEFINE_LOCK_GUARD_1_COND(mutex, _kill, mutex_lock_killable(_T->lock), _RET == 0)

A guarded lock is released automatically at scope exit:

guard(mutex)(&dev->lock);
/* critical section; unlock happens when the scope ends */

The conditional guards are useful for error paths: the guard knows whether the acquisition succeeded and only unlocks on success. Automatic cleanup reduces unlock-on-error bugs, but it does not change the rule that the lock must be held only in process context.

6.9 What CONFIG_DEBUG_LOCK_ALLOC Changes

The fast-path functions are compiled only when CONFIG_DEBUG_LOCK_ALLOC is disabled (kernel/locking/mutex.c). With lockdep maps enabled:

• mutex_lock() is a macro for mutex_lock_nested(lock, 0);
• mutex_trylock() is a macro for _mutex_trylock_nest_lock(lock, NULL);
• mutex_lock_interruptible(), mutex_lock_killable(), and mutex_lock_io() have nested variants;
• the non-lockdep __mutex_trylock_fast() and __mutex_unlock_fast() are not part of the build; and
• every lock/unlock reports to lockdep, which is why lockdep kernels are slower but far more diagnostic.

This is a configuration-level branch, not a runtime branch. It is one reason a bug that reproduces on a production kernel may behave differently on a lockdep kernel: the fast path itself is different.

---

7. The Lock Path: Fast, Optimistic, and Slow

The lock path is layered so that the common case stays small. A task tries the single-instruction fast path, then a bounded optimistic-spin path, then the sleeping slow path. The layers are not separate APIs; they are successive attempts inside mutex_lock() and __mutex_lock_common().

mutex_lock(lock)
      │
      ▼
might_sleep()                         ← process-context assertion
      │
      ▼
__mutex_trylock_fast()                ← CAS(owner, 0 → current)
      │
   success ───────────────────────────────────────────────► return
      │
    fail
      ▼
__mutex_lock_slowpath()
      │
      ▼
__mutex_lock_common()
      │
      ├─ preempt_disable()
      ├─ mutex_acquire_nest()          ← lockdep bookkeeping
      │
      ▼
__mutex_trylock()                     ← retry; also picks up a handoff
      │
   success ───────────────────────────────────────────────► return
      │
    fail
      ▼
mutex_optimistic_spin(waiter=NULL)    ← bounded spinning; may schedule
      │
   success ───────────────────────────────────────────────► return
      │
    fail
      ▼
raw_spin_lock_irqsave(&lock->wait_lock)
      │
      ▼
__mutex_trylock()                     ← try once under wait_lock
      │
   success ───────────────────────────────────────────────► skip_wait
      │
    fail
      ▼
enqueue waiter on the task's kernel stack
      │
      ▼
set blocked_on, task state, and wait loop
      │
      ├─ trylock again
      ├─ check signals / ww-mutex kill conditions
      ├─ release wait_lock, schedule, reacquire
      ├─ first waiter: trylock_or_handoff, then optimistic spin
      └─ repeat until acquired or interrupted

7.1 Fast Path

The public entry point is mutex_lock():

void __sched mutex_lock(struct mutex *lock)
{
	might_sleep();

	if (!__mutex_trylock_fast(lock))
		__mutex_lock_slowpath(lock);
}

The fast path is __mutex_trylock_fast():

static __always_inline bool __mutex_trylock_fast(struct mutex *lock)
{
	unsigned long curr = (unsigned long)current;
	unsigned long zero = 0UL;

	MUTEX_WARN_ON(lock->magic != lock);

	if (atomic_long_try_cmpxchg_acquire(&lock->owner, &zero, curr))
		return true;

	return false;
}

The compare-and-swap succeeds only if owner is exactly zero. A held mutex, a wait list, a handoff in progress, or any flag bit causes the fast path to fail. The operation has acquire semantics: loads and stores inside the critical section cannot be reordered before the successful CAS.

On AArch64, the implementation depends on whether the CPU implements the Large System Extensions (LSE):

CPU capability	Instruction sequence for atomic_long_try_cmpxchg_acquire()
Pre-ARMv8.1, LL/SC	LDAXR (load-acquire exclusive) → compare → STXR → retry on failure
ARMv8.1+ with LSE	CASA (compare-and-swap acquire), a single instruction
ARMv8.1+ without LSE	The LL/SC sequence remains available

The kernel selects the implementation at runtime through ARM64_HAS_LSE_ATOMICS (arch/arm64/include/asm/lse.h). The acquire cmpxchg template emits casa, not casal (arch/arm64/include/asm/atomic_lse.h). CASAL is the acquire-release form; using it here would add an unnecessary release barrier to the lock acquisition.

If the fast path fails, the task enters the slow path even if the failure was only because another task won the race by a few cycles. The slow path then retries, spins optimistically, or sleeps.

7.2 Mid Path: Adaptive Optimistic Spinning

Optimistic spinning is enabled by CONFIG_MUTEX_SPIN_ON_OWNER, which is def_bool y on SMP systems that provide ARCH_SUPPORTS_ATOMIC_RMW (kernel/Kconfig.locks). AArch64 selects the required support (arch/arm64/Kconfig), so the feature is on by default for normal SMP AArch64 kernels.

The premise is simple: if the owner is currently running on another CPU, it may release the lock within microseconds. Sleeping and waking can cost much more than a short spin, especially when the runqueue is loaded. The spinning is deliberately bounded by scheduler state rather than by a fixed iteration count.

mutex_optimistic_spin(lock, ww_ctx, waiter)
      │
      ├─ waiter == NULL?
      │     ├─ yes: mutex_can_spin_on_owner()
      │     │        ├─ need_resched()? ── yes ──► give up
      │     │        ├─ no owner? ──────────────► trylock
      │     │        ├─ owner not on CPU? ──────► give up
      │     │        └─ osq_lock() ── fail ─────► give up
      │     │
      │     └─ no: waiter-spinner; skip OSQ and spin directly
      │
      ▼
  loop:
      owner = __mutex_trylock_or_owner(lock)
      if (!owner) ──────────────────────────────► acquired
      if (!mutex_spin_on_owner(lock, owner, ...)) ──► give up
      cpu_relax()
      goto loop

mutex_can_spin_on_owner() is at kernel/locking/mutex.c. It checks need_resched() first, then reads the owner and asks whether the owner is on a CPU. If the lock has no owner, it returns true so the caller can try to acquire the now-free lock without entering the sleeping path.

mutex_spin_on_owner() is at kernel/locking/mutex.c:

static noinline
bool mutex_spin_on_owner(struct mutex *lock, struct task_struct *owner,
			 struct ww_acquire_ctx *ww_ctx, struct mutex_waiter *waiter)
{
	bool ret = true;

	lockdep_assert_preemption_disabled();

	while (__mutex_owner(lock) == owner) {
		barrier();

		if (!owner_on_cpu(owner) || need_resched()) {
			ret = false;
			break;
		}

		if (ww_ctx && !ww_mutex_spin_on_owner(lock, ww_ctx, waiter)) {
			ret = false;
			break;
		}

		cpu_relax();
	}

	return ret;
}

The owner pointer is a speculative read. It is safe because the spinner has disabled preemption, which is also an RCU read-side critical section in this context, so the task_struct cannot disappear while it is being observed. The barrier() ensures the owner->on_cpu load happens after the owner check; otherwise the spinner could read a stale on_cpu value for a task that has just been replaced.

The loop stops when:

• the owner changes (the lock may have been released or stolen);
• the owner is no longer on a CPU (it was preempted or migrated);
• need_resched() becomes true (the scheduler wants this CPU); or
• a ww-mutex deadlock-detection condition makes spinning unsafe.

cpu_relax() on AArch64 is a YIELD instruction (arch/arm64/include/asm/vdso/processor.h):

static inline void cpu_relax(void)
{
	asm volatile("yield" ::: "memory");
}

YIELD is a hint, not a sleep. On an out-of-order core it can reduce pressure on the store pipeline and improve fairness among hardware threads; on a simple in-order core it may be close to a no-op. The compiler barrier in the inline assembly also forces the loop to reload the relevant values.

The full spinner is mutex_optimistic_spin():

static __always_inline bool
mutex_optimistic_spin(struct mutex *lock, struct ww_acquire_ctx *ww_ctx,
		      struct mutex_waiter *waiter)
{
	if (!waiter) {
		if (!mutex_can_spin_on_owner(lock))
			goto fail;

		if (!osq_lock(&lock->osq))
			goto fail;
	}

	for (;;) {
		struct task_struct *owner;

		owner = __mutex_trylock_or_owner(lock);
		if (!owner)
			break;

		if (!mutex_spin_on_owner(lock, owner, ww_ctx, waiter))
			goto fail_unlock;

		cpu_relax();
	}

	if (!waiter)
		osq_unlock(&lock->osq);

	return true;

fail_unlock:
	if (!waiter)
		osq_unlock(&lock->osq);

fail:
	if (need_resched()) {
		__set_current_state(TASK_RUNNING);
		schedule_preempt_disabled();
	}

	return false;
}

Two details are easy to miss:

1. The waiter-spinner is special. A task already on the wait list does not join the OSQ. The head waiter may spin on the lock directly, concurrently with the OSQ head. This is intentional: the waiter has already paid the cost of enqueueing and must not be forced behind unrelated optimistic spinners.
2. A failed spin may schedule. If need_resched() is set, the failure path calls schedule_preempt_disabled() before returning to the slow path. This avoids acquiring the mutex and then immediately being preempted.

The OSQ itself is an MCS queue: osq_lock() adds the current CPU’s per-CPU node to the tail, then waits on its predecessor’s locked field. osq_unlock() marks the node unlocked and releases the next spinner by setting that spinner’s locked field; it does not call the scheduler. Because each waiter polls a different cache line, the coherence traffic is bounded. On AArch64, osq_lock() uses smp_cond_load_relaxed(), which can compile to a WFE-based wait on CPUs that support it (arch/arm64/include/asm/cmpxchg.h). The WFE path is a power optimization, not a correctness requirement.

7.3 Slow Path: Enqueue, Set Blocked-On State, Sleep

When optimistic spinning fails, the task takes the wait lock and becomes a real waiter. The entry point is __mutex_lock_slowpath() (kernel/locking/mutex.c), which calls __mutex_lock_common() (kernel/locking/mutex.c) with a task state such as TASK_UNINTERRUPTIBLE, TASK_INTERRUPTIBLE, or TASK_KILLABLE.

The first part of the common path retries under preemption-disabled protection:

preempt_disable();
mutex_acquire_nest(&lock->dep_map, subclass, 0, nest_lock, ip);

if (__mutex_trylock(lock) ||
    mutex_optimistic_spin(lock, ww_ctx, NULL)) {
	lock_acquired(&lock->dep_map, ip);
	preempt_enable();
	return 0;
}

raw_spin_lock_irqsave(&lock->wait_lock, flags);

if (__mutex_trylock(lock)) {
	if (ww_ctx)
		__ww_mutex_check_waiters(lock, ww_ctx, &wake_q);
	goto skip_wait;
}

The __mutex_trylock() call is not just a duplicate of the fast path. It handles the full owner-word state machine, including a pending PICKUP intended for the current task. That is why the slow path can succeed even though the owner field is non-zero.

If the retry fails, the task initializes a stack waiter and enqueues it:

debug_mutex_lock_common(lock, &waiter);
waiter.task = current;
if (use_ww_ctx)
	waiter.ww_ctx = ww_ctx;

if (!use_ww_ctx) {
	/* add waiting tasks to the end of the waitqueue (FIFO): */
	__mutex_add_waiter(lock, &waiter, NULL);
} else {
	/*
	 * Add in stamp order, waking up waiters that must kill
	 * themselves.
	 */
	ret = __ww_mutex_add_waiter(&waiter, lock, ww_ctx, &wake_q);
	if (ret)
		goto err_early_kill;
}

A normal mutex is FIFO at the wait-list level. A ww-mutex uses stamp order instead, so that the oldest transaction is first; that ordering is part of the deadlock-avoidance protocol, not ordinary mutex fairness.

The waiter lifecycle is:

                    ┌────────────────┐
                    │    RUNNING     │
                    └───────┬────────┘
                            │ __mutex_add_waiter()
                            ▼
                    ┌────────────────┐
                    │    QUEUED      │
                    │ blocked_on set │
                    └───────┬────────┘
              schedule()    │
                            ▼
                    ┌────────────────┐
                    │    SLEEPING    │
                    └───────┬────────┘
          wake_up_process() │
                            ▼
                    ┌────────────────┐
                    │ WOKEN / SPIN   │
                    │ first waiter?  │
                    └───┬────────┬───┘
          retry / PICKUP│        │ spin fails
                        ▼        ▼
                   ┌────────┐  ┌────────────┐
                   │ACQUIRED│  │  SLEEPING  │
                   └────────┘  └────────────┘

A woken waiter first retries the owner-word protocol while holding wait_lock; only the first waiter may subsequently spin outside the lock. PICKUP makes the acquisition step deterministic after a handoff, while an ordinary wakeup is only a hint to retry.

Before sleeping, the task records what it is blocked on and publishes its state:

raw_spin_lock(&current->blocked_lock);
__set_task_blocked_on(current, lock);
set_current_state(state);
trace_contention_begin(lock, LCB_F_MUTEX);

for (;;) {
	if (__mutex_trylock(lock))
		break;

	raw_spin_unlock(&current->blocked_lock);

	if (signal_pending_state(state, current)) {
		ret = -EINTR;
		goto err;
	}

	if (ww_ctx) {
		ret = __ww_mutex_check_kill(lock, &waiter, ww_ctx);
		if (ret)
			goto err;
	}

	raw_spin_unlock_irqrestore_wake(&lock->wait_lock, flags, &wake_q);
	schedule_preempt_disabled();

	first = lock->first_waiter == &waiter;

	raw_spin_lock_irqsave(&lock->wait_lock, flags);
	raw_spin_lock(&current->blocked_lock);
	__set_task_blocked_on(current, lock);
	set_current_state(state);

	if (__mutex_trylock_or_handoff(lock, first))
		break;

	if (first) {
		__clear_task_blocked_on(current, lock);
		raw_spin_unlock(&current->blocked_lock);
		raw_spin_unlock_irqrestore(&lock->wait_lock, flags);

		trace_contention_begin(lock, LCB_F_MUTEX | LCB_F_SPIN);
		opt_acquired = mutex_optimistic_spin(lock, ww_ctx, &waiter);

		raw_spin_lock_irqsave(&lock->wait_lock, flags);
		raw_spin_lock(&current->blocked_lock);
		__set_task_blocked_on(current, lock);
		set_current_state(state);

		if (opt_acquired)
			break;
		trace_contention_begin(lock, LCB_F_MUTEX);
	}
}

The code above is a compact version of the actual loop; the real function includes tracepoints and lockdep annotations. The synchronization protocol is what matters:

• wait_lock serializes the handoff. The waiter checks __mutex_trylock() while holding wait_lock, so an unlock cannot hand the lock off and then have the waiter miss the change.
• blocked_on is protected by the task’s blocked_lock. The scheduler and hung-task detector can inspect it without taking wait_lock.
• The task releases both locks before scheduling. It cannot sleep while holding wait_lock or blocked_lock.
• After wakeup, the waiter re-establishes blocked_on and its state before retrying. This closes the race with an unlock that cleared blocked_on while waking it.
• The first flag is captured after scheduling but before reacquiring wait_lock. It is a hint for handoff and optimistic spinning, not a substitute for the eventual trylock_or_handoff().

On successful acquisition, the task clears blocked_on, sets TASK_RUNNING, and removes itself from the wait list:

__clear_task_blocked_on(current, lock);
__set_current_state(TASK_RUNNING);
raw_spin_unlock(&current->blocked_lock);

if (ww_ctx) {
	if (!ww_ctx->is_wait_die && lock->first_waiter != &waiter)
		__ww_mutex_check_waiters(lock, ww_ctx, &wake_q);
}

__mutex_remove_waiter(lock, &waiter);
debug_mutex_free_waiter(&waiter);

skip_wait:
lock_acquired(&lock->dep_map, ip);
trace_contention_end(lock, 0);

if (ww_ctx)
	ww_mutex_lock_acquired(ww, ww_ctx);

raw_spin_unlock_irqrestore_wake(&lock->wait_lock, flags, &wake_q);
preempt_enable();
return 0;

__mutex_remove_waiter() has a subtle fast case. If the waiter’s list is empty, it was the only waiter; the function clears all flags and sets first_waiter = NULL (kernel/locking/mutex.c). If other waiters remain, it advances first_waiter to the next entry and deletes the current one. This is why an owner can acquire the lock and still have MUTEX_FLAG_WAITERS set until it removes its own waiter entry.

7.4 The Error Path

The err and err_early_kill labels undo the waiter state without acquiring the lock:

err:
	clear_task_blocked_on(current, lock);
	__set_current_state(TASK_RUNNING);
	__mutex_remove_waiter(lock, &waiter);

err_early_kill:
	WARN_ON(get_task_blocked_on(current));
	trace_contention_end(lock, ret);
	raw_spin_unlock_irqrestore_wake(&lock->wait_lock, flags, &wake_q);
	debug_mutex_free_waiter(&waiter);
	mutex_release(&lock->dep_map, ip);
	preempt_enable();
	return ret;

This path is used for signal interruption and for ww-mutex -EDEADLK. It must remove the waiter before releasing wait_lock; otherwise unlock could wake a task that is no longer waiting. The WARN_ON(get_task_blocked_on(current)) asserts that the cleanup actually cleared the blocked-on relation.

7.5 Handoff Is Requested by a Waiter, Not by the Owner

The handoff protocol is often described as “the unlock path sets HANDOFF.” That is backwards for the normal path. The first waiter sets MUTEX_FLAG_HANDOFF when it wakes and still cannot acquire the lock. The unlock path later observes the flag and performs the actual transfer. The one exception is the proxy-execution case described in Section 8.3, where unlock deliberately forces a handoff to a scheduler-selected donor.

The relevant helper is __mutex_trylock_or_handoff(), which passes handoff = first into the common owner-word loop:

static inline bool __mutex_trylock_or_handoff(struct mutex *lock, bool handoff)
{
	return !__mutex_trylock_common(lock, handoff);
}

Inside __mutex_trylock_common(), if the lock is held and handoff is true and HANDOFF is not already set, the helper ORs the flag into the owner word. The waiter then either spins or sleeps again. When the current owner eventually unlocks, it sees MUTEX_FLAG_HANDOFF and transfers the lock directly to the designated next task. That transfer is what prevents a newly arriving fast-path contender from stealing the lock from an already-woken head waiter.

The full sequence is shown in Section 8.

7.6 Fairness Is Eventual, Not Strict FIFO

The wait list is FIFO, but acquisition is not strictly FIFO. A newly arriving task can win the fast path or optimistic spin before an existing waiter has set HANDOFF. This is deliberate: allowing a brief spin to steal a lock can be cheaper than waking a task, and the lock is likely to be released soon.

Once the head waiter sets HANDOFF, the protocol becomes much stronger. The unlock path cannot simply clear the owner word; it must name the next owner. A newcomer’s CAS(0 -> current) cannot succeed because the owner field contains the designated task and PICKUP. The designated task then clears PICKUP and acquires. This gives the head waiter an eventual path to the lock rather than an absolute guarantee about arrival order.

The practical consequences are:

• do not rely on mutex acquisition order for correctness;
• do not assume that a task blocked longer will always acquire before a new arrival;
• do assume that a head waiter which has requested handoff will not be starved indefinitely by fast-path stealers; and
• keep critical sections short enough that the difference between “eventual” and “strict” does not become a latency problem.

---

8. The Unlock Path and Handoff

The unlock path has a fast case and a slow case. The fast case is a single release-CAS from current to zero. The slow case handles waiters, handoff, lockdep, and optional proxy-execution donation.

mutex_unlock(lock)
      │
      ▼
__mutex_unlock_fast()
  CAS(owner, current → 0) with RELEASE
      │
   success ───────────────────────────────────────────────► return
   (owner was exactly current; no flags, no waiters)
      │
    fail (flags, waiters, or wrong owner)
      ▼
__mutex_unlock_slowpath()
      │
      ├─ preempt guard
      ├─ read owner
      ├─ proxy donor present? ── yes ──► force HANDOFF
      ├─ HANDOFF already set? ── yes ──► skip clearing
      ├─ CAS(owner → owner flags) with RELEASE
      │       │
      │    no WAITERS ───────────────────────────────────► return
      │       │
      │    WAITERS
      ▼
raw_spin_lock_irqsave(&wait_lock)
raw_spin_lock(&current->blocked_lock)
      │
      ├─ choose proxy donor or first_waiter
      ├─ clear blocked_on for the chosen task
      ├─ if HANDOFF: __mutex_handoff(lock, next)
      ▼
unlock wait_lock and blocked_lock
wake_up_process(next)

8.1 Fast Unlock

The public function is mutex_unlock(). In a non-lockdep build it first tries __mutex_unlock_fast():

static __always_inline bool __mutex_unlock_fast(struct mutex *lock)
{
	unsigned long curr = (unsigned long)current;

	return atomic_long_try_cmpxchg_release(&lock->owner, &curr, 0UL);
}

The CAS succeeds only if owner is exactly current. If the owner word contains WAITERS, HANDOFF, or PICKUP, the comparison fails and the slow path runs. The operation has release semantics: all stores inside the critical section must be visible before the owner word becomes zero.

On AArch64, the release CAS is CASL on LSE hardware or a LDXR/STLXR loop on LL/SC hardware. The release barrier is the mirror image of the acquire barrier on lock: without it, another CPU could observe the lock as free while still seeing stale data from inside the critical section.

8.2 Slow Unlock: Clearing the Owner or Preserving Handoff

The slow path is __mutex_unlock_slowpath(). It begins by releasing lockdep state and disabling preemption:

mutex_release(&lock->dep_map, ip);
__release(lock);

guard(preempt)();

owner = atomic_long_read(&lock->owner);
for (;;) {
	MUTEX_WARN_ON(__owner_task(owner) != current);
	MUTEX_WARN_ON(owner & MUTEX_FLAG_PICKUP);

	if (sched_proxy_exec() && current->blocked_donor) {
		/* force handoff if we have a blocked_donor */
		owner = MUTEX_FLAG_HANDOFF;
		break;
	}

	if (owner & MUTEX_FLAG_HANDOFF)
		break;

	if (atomic_long_try_cmpxchg_release(&lock->owner, &owner,
					    __owner_flags(owner))) {
		if (owner & MUTEX_FLAG_WAITERS)
			break;

		return;
	}
}

There are three outcomes:

1. No handoff, no waiters. The CAS changes current to zero and returns immediately.
2. No handoff, but waiters. The CAS changes current to the flag bits (normally WAITERS), then the slow path takes wait_lock, selects a waiter, and wakes it. The waiter will retry the lock normally.
3. Handoff requested. The owner word is left intact for the moment. The slow path takes wait_lock, selects the next task, and calls __mutex_handoff().

The PICKUP warning is important: PICKUP is set only on a task that has been designated as the next owner. The current owner must never see it in its own owner word. HANDOFF is the request; PICKUP is the completed transfer.

8.3 Selecting the Next Task

After taking wait_lock and the current task’s blocked_lock, the slow path chooses between a proxy-execution donor and the first waiter:

struct task_struct *donor, *next = NULL;
struct mutex_waiter *waiter;
struct mutex *next_lock;

if (sched_proxy_exec()) {
	donor = current->blocked_donor;
	if (donor) {
		raw_spin_lock_nested(&donor->blocked_lock, SINGLE_DEPTH_NESTING);
		next_lock = __get_task_blocked_on(donor);
		if (next_lock == lock) {
			next = get_task_struct(donor);
			__clear_task_blocked_on(next, lock);
			current->blocked_donor = NULL;
		}
		raw_spin_unlock(&donor->blocked_lock);
	}
}

waiter = lock->first_waiter;
if (!next && waiter) {
	next = get_task_struct(waiter->task);

	raw_spin_lock_nested(&next->blocked_lock, SINGLE_DEPTH_NESTING);
	debug_mutex_wake_waiter(lock, waiter);
	__clear_task_blocked_on(next, lock);
	raw_spin_unlock(&next->blocked_lock);
}

The normal path picks first_waiter, clears its blocked_on relation, and takes a reference so the task cannot disappear before the wakeup. The proxy-execution path is optional and only active when CONFIG_SCHED_PROXY_EXEC is enabled; it lets the scheduler choose a higher-priority task that is blocked through this mutex, rather than blindly choosing the FIFO head. The blocked_donor field is set by the scheduler’s proxy-execution logic, not by the mutex itself.

After the choice, the handoff is performed if the owner had requested it:

if (owner & MUTEX_FLAG_HANDOFF)
	__mutex_handoff(lock, next);

raw_spin_unlock(&current->blocked_lock);
raw_spin_unlock_irqrestore(&lock->wait_lock, flags);
if (next) {
	wake_up_process(next);
	put_task_struct(next);
}

The wakeup happens after both spinlocks are released. Waking a task can take scheduler locks, so doing it under wait_lock would create an unnecessary lock-order constraint and extend the critical section.

8.4 The Handoff Operation

__mutex_handoff() is at kernel/locking/mutex.c:

static void __mutex_handoff(struct mutex *lock, struct task_struct *task)
{
	unsigned long owner = atomic_long_read(&lock->owner);

	for (;;) {
		unsigned long new;

		MUTEX_WARN_ON(__owner_task(owner) != current);
		MUTEX_WARN_ON(owner & MUTEX_FLAG_PICKUP);

		new = (owner & MUTEX_FLAG_WAITERS);
		new |= (unsigned long)task;
		if (task)
			new |= MUTEX_FLAG_PICKUP;

		if (atomic_long_try_cmpxchg_release(&lock->owner, &owner, new))
			break;
	}
}

It preserves WAITERS, clears HANDOFF, sets the next task pointer, and adds PICKUP when there is a next task. The release semantics match the acquire semantics of the designated task’s pickup.

The owner-word transition is:

Before handoff:       current | WAITERS | HANDOFF
                                   │
                    __mutex_handoff(next)
                                   │
                                   ▼
After handoff:          next | WAITERS | PICKUP
                                   │
                    next's __mutex_trylock()
                    (clears PICKUP, CAS to next | flags)
                                   │
                                   ▼
                        next | WAITERS  (next is owner)

If next is NULL, the new value is just the preserved WAITERS bits (normally zero), and no PICKUP is set. There is no designated task to wake; a later fast-path contender can acquire the owner word if it is zero. The invariant that WAITERS implies a non-empty wait list is what keeps this from becoming a stuck flag-only state.

A newly arriving fast-path contender cannot steal during this window. Its fast path requires owner == 0, but the owner field is next | PICKUP. The designated task’s __mutex_trylock_common() sees PICKUP, verifies that the task pointer equals current, clears PICKUP, and acquires the lock. If a different task somehow reads the handoff state, it fails the task != curr check and does not steal.

8.5 Handoff, Fairness, and Latency

Handoff solves the “wakeup race” in which a woken waiter loses the lock to a fresh fast-path caller before it can run. It does not make the waiter run immediately. If the designated task is not scheduled promptly, the mutex remains reserved in its name and other contenders wait. This is a deliberate tradeoff: fairness and predictable wakeup progress are more important than letting a running contender steal the lock.

The handoff flag is per-mutex, not per-waiter. The first waiter is the target because it is the head of the FIFO list (or the oldest ww transaction). The unlock path wakes that task, and the task’s pickup path clears PICKUP; the wakeup and the reservation are paired so that the designated task has a lock to pick up. The flag is not a general-purpose “lock reservation” usable by application code.

8.6 Lock Lifetime During Unlock

The unlock path can access the mutex after it has logically released ownership. It reads the owner word, may take wait_lock, may inspect first_waiter, and may call __mutex_handoff(). The lock object must therefore remain allocated until mutex_unlock() returns. This is stricter than a spinlock or a reference count, where the release operation may be the last access. A common driver pattern is to protect the free operation with a separate lifetime mechanism (reference count, RCU, or device teardown ordering) so that no unlock can still be in flight.

---

9. Ownership, Debugging, and lockdep

9.1 What the Fast Path Checks

The fast unlock path checks ownership implicitly. It attempts CAS(owner, current -> 0). If the current task is not the owner, the comparison fails. There is no branch that says “wrong owner”; the code simply falls into the slow path. In a production kernel, the slow path may then perform an undefined operation on a lock it does not own.

The slow path has explicit diagnostics (kernel/locking/mutex.c). The essential loop, with the proxy-execution and handoff branches omitted for readability, is:

owner = atomic_long_read(&lock->owner);
for (;;) {
	MUTEX_WARN_ON(__owner_task(owner) != current);
	MUTEX_WARN_ON(owner & MUTEX_FLAG_PICKUP);

	/* Clear the owner, hand off, or retry the compare-and-swap. */
	if (atomic_long_try_cmpxchg_release(&lock->owner, &owner,
					    __owner_flags(owner))) {
		if (owner & MUTEX_FLAG_WAITERS)
			break;
		return;
	}
}

MUTEX_WARN_ON() is defined only when CONFIG_DEBUG_MUTEXES is enabled (kernel/locking/mutex.c):

#ifdef CONFIG_DEBUG_MUTEXES
# define MUTEX_WARN_ON(cond) DEBUG_LOCKS_WARN_ON(cond)
#else
# define MUTEX_WARN_ON(cond)
#endif

DEBUG_LOCKS_WARN_ON() prints the condition and a stack trace, disables further lock debugging, and continues. It does not repair the lock. A warning therefore identifies the bug, but the kernel may still be in an inconsistent state; a debug kernel should be treated as compromised after the warning.

9.2 The Magic Pointer

With CONFIG_DEBUG_MUTEXES, mutex_init() sets magic = lock and key entry points check magic == lock (kernel/locking/mutex-debug.c):

void debug_mutex_unlock(struct mutex *lock)
{
	if (likely(debug_locks)) {
		DEBUG_LOCKS_WARN_ON(lock->magic != lock);
	}
}

void debug_mutex_init(struct mutex *lock)
{
	lock->magic = lock;
}

A freed, zeroed, or never-initialized mutex has the wrong magic. The check catches some use-after-free and initialization bugs, but it cannot catch every lifetime race: a freed object whose memory has been reused could happen to contain the right pointer. Debug checks reduce the search space; they do not replace correct lifetime management.

mutex_destroy() clears the magic and warns if the mutex is still locked (kernel/locking/mutex-debug.c). It does not free memory and does not release lockdep state; it marks the object unusable.

9.3 A Safe “Unlock Without Lock” Demonstration

The following demonstrates a mutex-protocol violation without intentionally corrupting a held lock. It unlocks a mutex that is already unlocked:

#include <linux/err.h>
#include <linux/kthread.h>
#include <linux/mutex.h>
#include <linux/printk.h>

static DEFINE_MUTEX(demo_lock);
static struct task_struct *bad_thread;

static int bad_unlock(void *data)
{
	(void)data;
	pr_info("wrong-unlock demo: unlocking an unlocked mutex\n");
	mutex_unlock(&demo_lock);	/* protocol violation */
	return 0;
}

static int __init demo_init(void)
{
	bad_thread = kthread_run(bad_unlock, NULL, "mutex-bad-unlock");
	if (IS_ERR(bad_thread))
		return PTR_ERR(bad_thread);
	kthread_stop(bad_thread);
	return 0;
}

With CONFIG_DEBUG_MUTEXES, the slow unlock path reports an ownership mismatch and prints a stack trace. Without it, the slow path may simply CAS the zero owner to zero and return; the bug is silent. This example is safe-ish because there is no real owner or waiter to corrupt, but it still turns off lock debugging after the warning. Do not use it as a template.

A wrong unlock while another task does own the mutex is worse. The warning may be followed by the slow path treating the real owner’s word as its own, clearing ownership, or waking a waiter incorrectly. In a production kernel, that is undefined behavior, not a recoverable error.

9.4 Recursive Locking

A recursive acquisition is a self-deadlock:

Task A holds M (owner = A)
      │
      ▼
Task A calls mutex_lock(M) again
      │
      ▼
fast CAS(0 -> A) fails
      │
      ▼
slow path / optimistic spin
      │
      ├─ owner is A, still on CPU
      ├─ A spins watching itself, or
      └─ A sleeps waiting for A to unlock
      │
      ▼
no progress; soft-lockup or hung-task detector eventually reports

Lockdep catches the recursion before the atomic operation, because it sees a second acquisition of the same lock class by the same task. The report includes both call sites. CONFIG_DEBUG_MUTEXES alone does not reliably catch this: its waiter checks are about blocked-on state and object validity, not a complete same-task ownership analysis. In a production kernel with neither lockdep nor lockup/hung-task detection, the task may spin or sleep forever.

An intentionally broken function is enough to illustrate the pattern:

static void recursive_bug(struct mutex *lock)
{
	mutex_lock(lock);
	mutex_lock(lock);	/* self-deadlock; lockdep should report this */
	mutex_unlock(lock);
	mutex_unlock(lock);
}

Do not load a module that calls this into a shared machine. Use a disposable VM and a kernel with CONFIG_PROVE_LOCKING, or rely on lockdep’s early report.

9.5 lockdep Annotations

Lockdep builds a graph of lock classes and acquisition contexts. The mutex API exposes several annotations:

• mutex_acquire(map, subclass, trylock, ip) — internal notification on acquisition; normally called by mutex_lock().
• mutex_release(map, ip) — internal notification on release; called by mutex_unlock().
• lockdep_assert_held(&lock) — runtime assertion that the current task holds the lock.
• mutex_lock_nested(lock, subclass) — acquire with a subclass that tells lockdep the nesting order.
• mutex_lock_nest_lock(lock, nest_lock) — use another lock’s dep_map as the nesting context.
• mutex_trylock_nest_lock(lock, nest_lock) — non-blocking form with the same annotation.

Example of a documented locking precondition:

static void update_state(struct my_device *dev)
	__must_hold(&dev->lock)
{
	lockdep_assert_held(&dev->lock);
	/* update dev->state */
}

__must_hold() is a sparse annotation; lockdep_assert_held() is the runtime check. Use both when the function is internal and the locking contract is part of its interface.

9.6 Context Diagnostics

might_sleep() is the first line of defense against atomic-context misuse. It is called by mutex_lock() (kernel/locking/mutex.c), mutex_lock_interruptible() (kernel/locking/mutex.c), mutex_lock_killable() (kernel/locking/mutex.c), and __mutex_lock_common() (kernel/locking/mutex.c). With CONFIG_DEBUG_ATOMIC_SLEEP, it checks whether the caller is in an atomic section or has interrupts disabled and reports a warning. Without that option, the check is compiled out or reduced, and the eventual failure is a hang or scheduler assertion.

9.7 Hung-Task and Tracepoint Visibility

The mutex slow path calls hung_task_set_blocker(lock, BLOCKER_TYPE_MUTEX) when a waiter is enqueued, and clears it on removal (kernel/locking/mutex.c). The hung-task detector can use this to report which mutex is blocking a task. The blocked_on field provides the same information to the scheduler and to diagnostics.

Contention tracepoints (trace_contention_begin, trace_contention_end) distinguish spinning from sleeping. They are useful for latency analysis: a high rate of LCB_F_SPIN entries suggests that optimistic spinning is common, while a high rate of sleeping entries suggests that the owner is repeatedly preempted or blocked. The tracepoints are implementation instrumentation, not part of the API contract.

9.8 What lockdep Does Not Do

Lockdep is a runtime dependency checker, not a proof of correctness. It detects cycles it has observed in the lock-class graph, and it cannot see orderings that no execution has exercised. It also cannot make a wrong-unlocker bug safe; it can report an ownership violation but cannot repair the lock. Keep runtime checks, code review, and lock-order documentation in the design.

---

10. AArch64 Hardware Behavior

The mutex algorithm is portable C, but its correctness and performance depend on AArch64’s weak memory model and on which atomic instructions the CPU implements. The kernel hides the choice behind runtime alternatives, so the same binary can use LSE instructions on a newer core and LL/SC sequences on an older one.

10.1 LL/SC Versus LSE

Before ARMv8.1, AArch64 provides load-exclusive and store-exclusive instructions. A compare-and-swap is a loop:

retry:
    LDAXR  x0, [owner]       // load-acquire exclusive
    CMP    x0, expected
    B.NE   fail
    STXR   w1, new, [owner]  // store-exclusive; w1 = 0 on success
    CBNZ   w1, retry
fail:

ARMv8.1 adds the Large System Extensions, including single-instruction atomics such as CASA (compare-and-swap acquire), CASL (compare-and-swap release), CASAL (acquire-release), LDADD, LDSET, and LDCLR. The kernel uses runtime patching so that a kernel built with both paths executes the best one for the current CPU (arch/arm64/include/asm/lse.h):

#define __lse_ll_sc_body(op, ...)					\
({									\
	alternative_has_cap_likely(ARM64_HAS_LSE_ATOMICS) ?		\
		__lse_##op(__VA_ARGS__) :				\
		__ll_sc_##op(__VA_ARGS__);				\
})

The LSE templates are in arch/arm64/include/asm/atomic_lse.h. The acquire variant emits casa; the release variant emits casl; the fully ordered variant emits casal.

10.2 Acquire and Release, Precisely

The mutex fast path uses:

• acquire on lock, so that operations inside the critical section cannot be reordered before the lock is acquired; and
• release on unlock, so that operations inside the critical section cannot be reordered after the lock is released.

CPU 0 (lock)                                  CPU 1 (unlock)
─────────────                                 ──────────────
CASA owner, 0 -> current
      │  acquire
      ▼
read/write protected data
      │
      │                                      CASL owner, current -> 0
      │                                            │  release
      │                                            ▼
      │                                      other CPUs may now acquire
      ▼
critical section

In the AArch64 memory model, LDAXR/STXR and CASA provide the acquire ordering, while LDXR/STLXR and CASL provide the release ordering. A CASAL would provide both, but the lock and unlock paths do not need both at each site. Using the weaker operation where it is sufficient avoids an unnecessary barrier.

The mutex’s correctness does not rely on a separate DMB on the fast path. The acquire/release instructions themselves order the critical-section accesses. The DMB instructions become relevant for explicit barriers, for some atomic RMW operations that do not carry the required ordering, and for the ww-mutex context publication path.

Operation	LL/SC form	LSE form
Lock CAS(owner, 0 → current)	LDAXR / STXR loop	CASA
Unlock CAS(owner, current → 0)	LDXR / STLXR loop	CASL
OR flags into owner	LDXR / STXR loop, followed by the helper’s barrier	LDSET (or LDCLR/LDADD as appropriate)
Full SMP barrier (smp_mb())	DMB ISH	DMB ISH
Read barrier (smp_rmb())	DMB ISHLD	DMB ISHLD
Write barrier (smp_wmb())	DMB ISHST	DMB ISHST
Busy-wait hint	YIELD	YIELD
OSQ wait	polling or WFE	polling or WFE

The flag helpers use atomic_long_or() and atomic_long_andnot(), which are fully ordered atomics rather than acquire/release operations. On the LL/SC path that means the generated sequence includes a full DMB ISH; on the LSE path the single instruction carries the ordering. This is one reason the fast paths use the explicitly acquire/release cmpxchg helpers, while the slower flag updates can use the simpler fully ordered form.

The barrier definitions are in arch/arm64/include/asm/barrier.h:

#define __smp_mb()	dmb(ish)
#define __smp_rmb()	dmb(ishld)
#define __smp_wmb()	dmb(ishst)

ISH means “inner shareable”; it covers the CPUs that share the same inner-shareable domain, which is the normal scope for kernel SMP synchronization. ISHLD and ISHST are the read and write variants, respectively.

10.3 Why the Owner Word Is a Single Cache Line

The fast path’s cost is dominated by cache-line ownership. A successful CAS must obtain the line in the exclusive state, and an unlock must transfer it. If the owner word shared a line with frequently read data, every lock operation would invalidate that data on other CPUs. The mutex structure itself is not declared cache-line aligned; the important design choice is to keep the hot owner word away from the protected data when false sharing is a concern.

   False sharing (avoid)                         Separate lines (prefer)
   ┌──────────────┬──────────────────┐           ┌──────────────┐   ┌──────────────────┐
   │ owner / CAS  │ frequently read  │           │ owner / CAS  │   │ frequently read  │
   │              │ protected data   │           │              │   │ protected data   │
   └──────────────┴──────────────────┘           └──────────────┘   └──────────────────┘
        one cache line                                two cache lines

The exact byte offsets and cache-line placement depend on the configuration, the compiler, and the embedding object. Measure before adding padding; unconditional alignment can waste memory and does not by itself remove contention on the owner word.

Handoff makes the cache-line transfer explicit: the unlock path writes next | PICKUP, which sends the line to the next owner’s cache when that task reads it. A fast-path contender that polls the same line can slow the transfer, which is why the OSQ limits polling.

10.4 YIELD, WFE, and Power

cpu_relax() on AArch64 emits YIELD. It is a hint that the current thread is in a spin loop. It does not guarantee that the hardware will do anything, but on SMT-capable or out-of-order cores it can reduce contention for execution resources.

The OSQ’s wait primitive can use WFE (Wait For Event) on CPUs that support the compare-wait sequence. The generated sequence is approximately:

    SEVL
    WFE
    LDXR   tmp, [monitored_address]
    EOR    tmp, tmp, expected
    CBNZ   tmp, done
    WFE
done:
    /* the outer smp_cond_load_relaxed() loop retries the load */

This is a power optimization: the core can enter a low-power state until a cache event occurs. It is not a replacement for the memory-ordering instructions; the LDXR and the surrounding barriers still provide the required visibility.

10.5 The Internal wait_lock

wait_lock is a raw spinlock, which on AArch64 is implemented by the queued spinlock code. A contended slow path may spin briefly on the qspinlock before sleeping on the mutex itself. The qspinlock’s own MCS queue is separate from the mutex’s OSQ: the qspinlock protects the short wait-list critical section, while the OSQ serializes optimistic mutex spinners. Do not confuse the two.

The mutex’s wait_lock is taken with IRQs disabled. That does not make the mutex interrupt-safe: it only means the short wait-list bookkeeping is safe against interrupt handlers on the same CPU. The mutex’s sleeping path still requires process context.

10.6 AArch64-Specific Practical Implications

• Use mutex_lock() for process-context critical sections that may sleep; do not try to emulate it with atomics plus a busy-wait.
• Do not assume a particular instruction sequence when profiling; the same source may run CASA or an LL/SC loop depending on the CPU.
• Keep the protected data off the same cache line as the mutex when false sharing is a concern, but do not add padding blindly; measure first.
• Treat YIELD as a hint, not a delay. It does not bound the spin; need_resched() and owner_on_cpu() do.
• Remember that a mutex can sleep even on a uniprocessor; the absence of another CPU does not make it a spinlock.

---

11. Scheduling, Fairness, and Contention

11.1 The Three Contention Regimes

 Uncontended                 Briefly contended              Sleepy / long hold
 ───────────                 ─────────────────              ──────────────────
 CAS succeeds                fast path fails                wait list grows
 no scheduler                OSQ spin                       task sleeps
 no wait list                owner likely running           wakeup + handoff
 few cycles                  bounded by need_resched         scheduler latency

The fast path is the common case. The mid path exists for the window in which the owner is running and likely to release soon. The slow path exists because no amount of spinning can guarantee progress if the owner is blocked, preempted, or waiting for I/O.

11.2 What Bounds the Spin

There is no fixed microsecond timeout in mutex_optimistic_spin(). The loop exits when:

• the owner changes or the lock becomes free;
• owner_on_cpu() becomes false;
• need_resched() becomes true; or
• a ww-mutex-specific spin condition fails.

need_resched() is the scheduler’s signal that another task should run on this CPU. It is checked both before entering the spin and inside the owner-watch loop. This is why a heavily loaded system naturally stops optimistic spinning: the spinner yields to the scheduler instead of monopolizing the CPU.

owner_on_cpu() is not a perfect predictor. The owner may be running but about to block; it may be in a long I/O wait; it may be preempted immediately after the check. Optimistic spinning is a heuristic, not a guarantee. The fallback to sleeping is what makes the lock correct under arbitrary scheduling.

11.3 A Fairness Timeline

Task A: owner
Task B: first waiter, sleeping
Task C: new arrival, fast path fails, starts spinning

time ──────────────────────────────────────────────────────────────►

A:  [ critical section ]
                         │
                         │ unlock: owner becomes W, or remains held
                         ▼
B:  wake ──► trylock ──► fails (C acquired first, or owner word still changing)
                         │
                         │ B sets HANDOFF on the current owner word
                         ▼
current owner (A if still unlocking, otherwise C):
                         │
                         │ unlock sees HANDOFF
                         ▼
                    __mutex_handoff(next = B)
                         │
                         ▼
B:  sees PICKUP, clears it, acquires
C:  loses; retries, spins, or sleeps according to OSQ state

Without handoff, C could repeatedly win the CAS after the current owner clears the word, even though B was woken first. Handoff closes that window. It does not prevent C from winning before B sets HANDOFF; that is why fairness is described as eventual rather than strict.

11.4 Sleeping and Wakeup

A sleeping waiter is removed from the runqueue. The unlock path calls wake_up_process(next) after releasing the mutex’s spinlocks. The scheduler places the task on a runqueue, and the task eventually runs, retries the lock, and either acquires it or sleeps again.

The wakeup path is not a direct function call into the waiter’s critical section. It merely makes the waiter runnable. The waiter still has to acquire the lock, which is why PICKUP and HANDOFF matter: they reserve the lock for the designated task and avoid a pointless wakeup race.

11.5 Priority Inversion

On a non-RT kernel, the mutex has no priority inheritance. Suppose:

low-priority task L holds mutex M
high-priority task H blocks on M
medium-priority task P becomes runnable and preempts L

H waits until L runs again. If P is CPU-bound, the delay can be arbitrarily long. The mutex cannot boost L because the non-RT implementation does not have the rtmutex priority-inheritance machinery.

Under PREEMPT_RT, struct mutex is backed by an rtmutex and the holder inherits the waiter’s priority. The owner field still exists conceptually, but the implementation and scheduling behavior are different; see Section 14.

11.6 Starvation and Latency

The combination of FIFO wait list and handoff gives the head waiter a strong eventual guarantee, but it does not guarantee bounded latency:

• the owner may hold the lock for a long time;
• the designated next task may not be scheduled promptly;
• a higher-priority task may preempt the owner or the waiter;
• a ww-mutex may force a transaction to restart, adding work rather than reducing lock wait;
• a wrong-unlocker or missed wakeup is a bug, not a fairness property.

For latency-sensitive code, keep the critical section short, avoid sleeping while holding the mutex, and measure contention with tracepoints before adding complexity. If the workload is extremely read-heavy or requires cross-task release, a mutex may be the wrong primitive.

11.7 How Contention Affects Throughput

A mutex serializes the critical section, so throughput is bounded by the section’s duration, not by the number of CPUs. Optimistic spinning can reduce the wakeup latency when the owner is running, but it consumes CPU time that could be used by another task. Sleeping frees the CPU but adds scheduler latency. The best choice depends on the section length and the probability that the owner is running:

Critical section	Likely best behavior
A few instructions, low contention	Fast CAS; a mutex is fine, though a spinlock may be cheaper
Microseconds, owner usually running	Optimistic spin reduces wakeup latency
Milliseconds or may block	Sleep; spinning would waste CPU
High contention, many waiters	Handoff and OSQ keep the system stable; consider reducing the section or sharding the lock
Read-mostly	Consider rw_semaphore or RCU instead of a mutex

---

12. Worked Examples

The examples below are deliberately small. They show the ownership rules, the non-blocking return convention, and the lifetime discipline. They are not a substitute for a real driver’s error handling.

12.1 Two Kernel Threads Contending on One Mutex

The 2-second msleep() inside the critical section makes the serialization visible in dmesg: the second thread prints that it is waiting immediately, but its “entered critical section” message appears only after the first thread prints “leaving critical section”.

#include <linux/delay.h>
#include <linux/err.h>
#include <linux/kthread.h>
#include <linux/mutex.h>
#include <linux/printk.h>

static DEFINE_MUTEX(demo_lock);
static struct task_struct *thread_a;
static struct task_struct *thread_b;

static int worker(void *arg)
{
	const char *name = arg;

	pr_info("%s: waiting for the mutex\n", name);
	mutex_lock(&demo_lock);
	pr_info("%s: entered critical section\n", name);
	msleep(2000);
	pr_info("%s: leaving critical section\n", name);
	mutex_unlock(&demo_lock);
	return 0;
}

static int __init demo_init(void)
{
	thread_a = kthread_run(worker, "thread-a", "mutex-a");
	if (IS_ERR(thread_a))
		return PTR_ERR(thread_a);

	thread_b = kthread_run(worker, "thread-b", "mutex-b");
	if (IS_ERR(thread_b)) {
		kthread_stop(thread_a);
		return PTR_ERR(thread_b);
	}
	return 0;
}

static void __exit demo_exit(void)
{
	kthread_stop(thread_a);
	kthread_stop(thread_b);
}

The kernel thread function must not return while another thread may still use module text or static data. The exit path waits for both threads with kthread_stop(), so the module cannot be unloaded while a worker is inside msleep() or waiting on the mutex.

12.2 Non-Blocking Acquisition

mutex_trylock() is useful when the caller has other work to do rather than waiting:

static int try_worker(void *arg)
{
	bool acquired = mutex_trylock(&demo_lock);

	if (!acquired) {
		pr_info("lock busy; doing fallback work\n");
		return 0;
	}

	pr_info("lock acquired; entering critical section\n");
	msleep(2000);
	mutex_unlock(&demo_lock);
	return 0;
}

A loser must not touch the protected data and must not call mutex_unlock(). The 1/0 convention is the opposite of down_trylock(), which is a frequent source of conversion bugs.

12.3 A Character-Device Read Path

Do not acquire a mutex in open() and release it in release(). File descriptors can be passed between processes, and release() may run in a different task context than open() (for example, through file-descriptor passing or deferred fput()). A mutex is strict about ownership, so such a pairing is a latent wrong-unlocker bug.

The safe pattern is to take and release the mutex inside each callback that needs serialization:

#include <linux/fs.h>
#include <linux/minmax.h>
#include <linux/mutex.h>
#include <linux/string.h>
#include <linux/uaccess.h>

struct demo_device {
	struct mutex lock;	/* initialized by the driver's probe path */
	char buffer[128];
};

static ssize_t demo_read(struct file *file, char __user *user_buf,
			 size_t count, loff_t *ppos)
{
	struct demo_device *dev = file->private_data;
	char snapshot[sizeof(dev->buffer)];
	size_t n;

	if (mutex_lock_interruptible(&dev->lock))
		return -ERESTARTSYS;

	n = min(count, sizeof(dev->buffer));
	memcpy(snapshot, dev->buffer, n);
	mutex_unlock(&dev->lock);

	if (copy_to_user(user_buf, snapshot, n))
		return -EFAULT;

	return n;
}

The mutex protects the device buffer while it is copied into a local snapshot. The potentially faulting copy_to_user() runs after the unlock, so a user page fault cannot hold the mutex for an unbounded time. The open() callback can initialize per-file state and return; it does not need to hold the device mutex across the file’s lifetime.

If a design genuinely needs a lock held across open and close, a mutex is the wrong tool unless the same task is guaranteed to perform both operations. A semaphore or a reference-counted object with explicit acquire/release semantics is usually more appropriate.

12.4 Observing the Lock State

mutex_is_locked() is diagnostic, not a synchronization primitive:

#include <linux/mutex.h>
#include <linux/printk.h>
#include <linux/slab.h>

static int __init state_demo_init(void)
{
	struct demo_device *dev;
	bool locked;

	dev = kzalloc(sizeof(*dev), GFP_KERNEL);
	if (!dev)
		return -ENOMEM;

	mutex_init(&dev->lock);
	locked = mutex_is_locked(&dev->lock);
	pr_info("before lock: %s\n", locked ? "locked" : "unlocked");

	mutex_lock(&dev->lock);
	locked = mutex_is_locked(&dev->lock);
	pr_info("inside lock: %s\n", locked ? "locked" : "unlocked");
	mutex_unlock(&dev->lock);

	locked = mutex_is_locked(&dev->lock);
	pr_info("after unlock: %s\n", locked ? "locked" : "unlocked");

	mutex_destroy(&dev->lock);
	kfree(dev);
	return 0;
}

The result is a snapshot. Another task can change the state immediately after the read. The function is useful for assertions and debugging, not for deciding whether to access protected data.

12.5 Device-Managed Mutex

For a mutex embedded in a device-managed structure, devm_mutex_init() can simplify teardown when CONFIG_DEBUG_MUTEXES is enabled:

#include <linux/device.h>
#include <linux/mutex.h>
#include <linux/platform_device.h>
#include <linux/slab.h>

struct demo_pdev {
	struct mutex lock;
};

static int demo_probe(struct platform_device *pdev)
{
	struct demo_pdev *dev;
	int ret;

	dev = devm_kzalloc(&pdev->dev, sizeof(*dev), GFP_KERNEL);
	if (!dev)
		return -ENOMEM;

	ret = devm_mutex_init(&pdev->dev, &dev->lock);
	if (ret)
		return ret;

	platform_set_drvdata(pdev, dev);
	return 0;
}

The helper does not allocate the mutex, and without CONFIG_DEBUG_MUTEXES it is effectively a no-op because mutex_destroy() is a no-op. The mutex itself still has the same ownership and context rules; devres only manages the debug-destroy timing.

12.6 A Reference-Counted Release Pattern

atomic_dec_and_mutex_lock() avoids taking the mutex on every reference drop:

#include <linux/atomic.h>
#include <linux/mutex.h>
#include <linux/slab.h>

struct object {
	struct mutex lock;
	atomic_t refcount;
};

void object_put(struct object *obj)
{
	if (atomic_dec_and_mutex_lock(&obj->refcount, &obj->lock)) {
		/* Last reference: the mutex is held. */
		object_cleanup_locked(obj);
		mutex_unlock(&obj->lock);
		mutex_destroy(&obj->lock);
		kfree(obj);
	}
}

The helper takes the mutex only when the decrement might reach zero. The caller must unlock and free only on the true return, and only after ensuring no other task can still reach the object through a stale reference. The mutex does not provide the object lifetime; the reference count does.

---

13. WW-Mutex: Deadlock Avoidance

13.1 The Problem

A mutex protects a single object. Many subsystems need to lock several objects at once. If different tasks acquire the same objects in different orders, an ABBA deadlock is possible:

Task A:  lock(obj1) ──► waiting for lock(obj2)
Task B:  lock(obj2) ──► waiting for lock(obj1)
                    ▲
                    └── circular wait

A global lock order solves the problem when the order can be defined. GPU drivers, DRM, and similar subsystems often cannot impose one: userspace command buffers and buffer-object lifetimes determine which objects are needed, and a fixed order across thousands of objects is impractical. A ww-mutex (wound/wait mutex) assigns each transaction a timestamp and uses that timestamp to decide which transaction must yield when a conflict occurs.

13.2 Wait-Die and Wound-Wait

A transaction is a set of locks acquired under one acquire context. Each context gets a monotonically increasing stamp; a larger stamp means a younger transaction. The conflict rules are easiest to state in terms of the requester and the current holder:

Algorithm	Requester older than holder	Requester younger than holder
Wait-Die	Requester waits	Requester dies with -EDEADLK and retries later
Wound-Wait	Requester wounds the holder; holder restarts	Requester waits

Wait-Die:  older waits, younger dies
           fewer wounds, more restarts
           often good for short transactions

Wound-Wait: older wounds younger, younger waits
            more wounds, fewer restarts
            often good for long GPU transactions

The kernel supports both algorithms per lock class. DEFINE_WD_CLASS() selects Wait-Die; DEFINE_WW_CLASS() selects Wound-Wait (include/linux/ww_mutex.h):

#define DEFINE_WD_CLASS(classname) \
	struct ww_class classname = __WW_CLASS_INITIALIZER(classname, 1)

#define DEFINE_WW_CLASS(classname) \
	struct ww_class classname = __WW_CLASS_INITIALIZER(classname, 0)

The is_wait_die field is per class, not per lock. Mixing Wait-Die and Wound-Wait on the same wait list is forbidden; the class property is what keeps the ordering coherent.

13.3 The Transaction State Machine

ww_acquire_init(&ctx, &class)
      │  ctx->task = current
      │  ctx->stamp = atomic_long_inc_return_relaxed(&class->stamp)
      │  ctx->acquired = 0
      ▼
acquire each object
      │
      ├─ success ─────────────────────────────────► retry next object
      │
      ├─ -EALREADY: already held by this context ──► skip and continue
      │
      ├─ -EDEADLK: back off (die or wounded)
      │       ├─ release every object acquired so far
      │       ├─ ww_mutex_lock_slow(contended_lock, &ctx)
      │       └─ retry the transaction from the beginning
      │
      └─ -EINTR (interruptible variants)
              release acquired objects, abort

success path:

ww_acquire_done(&ctx)
      ▼
use protected objects
      ▼
ww_mutex_unlock() for each object
      ▼
ww_acquire_fini(&ctx)

ww_acquire_done() is optional but recommended: it documents the end of the acquisition phase, and debug builds use it to catch attempts to acquire another lock with the same context. ww_acquire_fini() must be called only after every ww-mutex acquired with the context has been unlocked, by the same task, and before the context memory is freed.

13.4 Data Structures

The public structures are in include/linux/ww_mutex.h:

struct ww_class {
	atomic_long_t stamp;
	struct lock_class_key acquire_key;
	struct lock_class_key mutex_key;
	const char *acquire_name;
	const char *mutex_name;
	unsigned int is_wait_die;
};

context_lock_struct(ww_mutex) {
	struct WW_MUTEX_BASE base;	/* struct mutex on non-RT */
	struct ww_acquire_ctx *ctx;
#ifdef DEBUG_WW_MUTEXES
	struct ww_class *ww_class;
#endif
};

context_lock_struct(ww_acquire_ctx) {
	struct task_struct *task;
	unsigned long stamp;
	unsigned int acquired;
	unsigned short wounded;
	unsigned short is_wait_die;
#ifdef DEBUG_WW_MUTEXES
	unsigned int done_acquire;
	struct ww_class *ww_class;
	void *contending_lock;
#endif
#ifdef CONFIG_DEBUG_LOCK_ALLOC
	struct lockdep_map dep_map;
	struct lockdep_map first_lock_dep_map;
#endif
};

On the non-RT path, WW_MUTEX_BASE is struct mutex; under PREEMPT_RT it is struct rt_mutex. ww_mutex therefore reuses the base mutex’s fast path, wait list, and ownership machinery. The extra fields implement the transaction protocol.

13.5 How the Conflict Detection Works

ww_acquire_init() assigns the stamp with atomic_long_inc_return_relaxed() (include/linux/ww_mutex.h):

ctx->task = current;
ctx->stamp = atomic_long_inc_return_relaxed(&ww_class->stamp);
ctx->acquired = 0;
ctx->wounded = false;
ctx->is_wait_die = ww_class->is_wait_die;

A larger stamp means a younger transaction. The comparison helper is __ww_ctx_less():

static inline bool
__ww_ctx_less(struct ww_acquire_ctx *a, struct ww_acquire_ctx *b)
{
#ifdef WW_RT
	/* RT priority/deadline comparison omitted here. */
#endif

	/* FIFO order tie break -- bigger is younger */
	return (signed long)(a->stamp - b->stamp) > 0;
}

On the non-RT mutex path, the comparison is stamp-only. On the RT path, the WW_RT branch can give priority to a higher-priority waiter before falling back to the stamp. That is necessary because rtmutex priority inheritance can change effective priorities; the wait-list ordering has to remain consistent with the scheduler.

The wait list is kept in stamp order, smallest (oldest) first (kernel/locking/ww_mutex.h). Waiters without a context are interspersed in FIFO order so they are not starved. When a waiter is added:

• Wait-Die checks whether an older context is already waiting; if so, it kills itself immediately instead of queueing behind a transaction it cannot beat.
• Wound-Wait checks whether the current holder is younger; if so, it marks the holder’s context wounded and wakes it.

The wound path is __ww_mutex_wound():

if (!hold_ctx)
	return false;

if (ww_ctx->acquired > 0 && __ww_ctx_less(hold_ctx, ww_ctx)) {
	hold_ctx->wounded = 1;

	if (owner != current) {
		clear_task_blocked_on(owner, NULL);
		wake_q_add(wake_q, owner);
	}
	return true;
}

The holder notices the wound when it next checks for kill conditions (__ww_mutex_check_kill())—typically after it blocks on another ww-mutex—and returns -EDEADLK. If it holds other locks, it must release them before retrying the contended lock with ww_mutex_lock_slow().

The Wait-Die body is __ww_mutex_die(). It wakes a younger waiter that has already acquired locks so that the waiter can unwind. The distinction between “die” and “wound” is not cosmetic: in Wait-Die, the requester backs off; in Wound-Wait, the holder is forced to back off.

13.6 The Fast Path and the Context Publication Barrier

A ww-mutex acquisition can succeed on the base mutex’s fast path, without holding wait_lock. In that case, the context must be published and any waiters that arrived concurrently must be checked. ww_mutex_set_context_fastpath() does this:

ww_mutex_lock_acquired(lock, ctx);

smp_mb(); /* publish ww->ctx before checking WAITERS */

has_waiters = data_race(__ww_mutex_has_waiters(&lock->base));
if (likely(!has_waiters))
	return;

lock_wait_lock(&lock->base, &flags);
__ww_mutex_check_waiters(&lock->base, ctx, &wake_q);
preempt_disable();
unlock_wait_lock(&lock->base, &flags);
wake_up_q(&wake_q);
preempt_enable();

The smp_mb() pairs with the barrier in __ww_mutex_add_waiter():

        fast-path acquirer                         contended waiter
        ───────────────────                        ─────────────────
        ww->ctx = ctx                              add waiter + WAITERS
              │                                          │
              │ smp_mb()                                 │ smp_mb()
              ▼                                          ▼
        read WAITERS                               read ww->ctx
              │                                          │
              └──────────── at least one sees ──────────┘
                         the other's update

Either the fast-path acquirer sees WAITERS, or the waiter sees the published ww->ctx; without the barrier, both could miss each other and a wound/die decision could be delayed indefinitely. This is one of the few places where an explicit DMB ISH on AArch64 is essential to the algorithm rather than incidental to an atomic primitive.

13.7 API

The transaction API is in include/linux/ww_mutex.h:

DEFINE_WW_CLASS(reservation_ww_class);	/* Wound-Wait */
DEFINE_WD_CLASS(wait_die_class);	/* Wait-Die */

struct ww_mutex obj_lock;
ww_mutex_init(&obj_lock, &reservation_ww_class);

struct ww_acquire_ctx ctx;
int ret;
ww_acquire_init(&ctx, &reservation_ww_class);

/* Acquire one or more locks, in any order. */
ret = ww_mutex_lock(&obj_lock, &ctx);
if (ret)
	goto out_fini;		/* -EDEADLK, -EALREADY, -EINTR, ... */

/* Optional: mark the end of the acquisition phase. */
ww_acquire_done(&ctx);

/* Use the protected object. */
ww_mutex_unlock(&obj_lock);

out_fini:
/*
 * This single-lock outline has nothing to release on the error path.
 * A multi-lock transaction must release every acquired lock first.
 */
ww_acquire_fini(&ctx);
ww_mutex_destroy(&obj_lock);

ww_mutex_lock_slow() is not a substitute for retrying after -EDEADLK. Its contract requires the caller to release every other ww-mutex acquired under the context first, then wait on the single contended lock and restart the transaction. Section 13.8 shows the complete retry loop. Calling ww_mutex_destroy() is valid only after the final unlock has returned and no task can still reach the object.

ww_mutex_init() is the only supported initializer; there is no static DEFINE_WW_MUTEX. The lock must belong to the same ww class as the context. ww_mutex_lock() can also be called with a NULL context, in which case it behaves like a normal mutex and performs no deadlock detection.

ww_mutex_trylock() is non-blocking and does not perform deadlock detection. Its success convention is 1/0, like mutex_trylock(), not the 0/-E* convention of the blocking ww-mutex calls. A supplied context is used for bookkeeping, not for a full wound/die decision; callers that need deadlock handling must use ww_mutex_lock() or ww_mutex_lock_interruptible().

13.8 A Correct Multi-Lock Retry Pattern

The essential rule is: on -EDEADLK, release all locks acquired under the context, then wait on the contended lock with ww_mutex_lock_slow(), then restart the transaction. The following helper makes the control flow explicit:

DEFINE_WW_CLASS(reservation_ww_class);

static void release_all(struct ww_mutex **objs, bool *held, int n)
{
	int i;

	for (i = 0; i < n; i++) {
		if (held[i]) {
			ww_mutex_unlock(objs[i]);
			held[i] = false;
		}
	}
}

static int lock_pair(struct ww_mutex *a, struct ww_mutex *b,
		     struct ww_acquire_ctx *ctx)
{
	struct ww_mutex *objs[2] = { a, b };
	bool held[2] = { false, false };
	int i, ret;

	ww_acquire_init(ctx, &reservation_ww_class);

retry:
	for (i = 0; i < ARRAY_SIZE(objs); i++) {
		if (held[i])
			continue;

		ret = ww_mutex_lock(objs[i], ctx);
		if (ret == -EALREADY) {
			held[i] = true;	/* already held by this context */
			continue;
		}
		if (ret == -EDEADLK) {
			/*
			 * The transaction must not hold any context-stamped
			 * lock while waiting on the contended lock.
			 */
			release_all(objs, held, ARRAY_SIZE(objs));
			ww_mutex_lock_slow(objs[i], ctx);
			held[i] = true;
			goto retry;
		}
		if (ret) {		/* -EINTR or another error */
			release_all(objs, held, ARRAY_SIZE(objs));
			ww_acquire_fini(ctx);
			return ret;
		}
		held[i] = true;
	}

	ww_acquire_done(ctx);
	return 0;
}

The held[] array is important. A simple “release the first i entries” loop is not enough: after a slow-path acquisition, a later deadlock can occur on a lock with a lower index while a higher-index lock is still held. The transaction must release every lock it currently owns, not just a prefix of the acquisition loop. The two pointers must also refer to distinct locks; passing the same object twice would make the second acquisition return -EALREADY and the caller’s later unlock loop would unlock it twice.

After this helper returns success, the caller owns both locks and must release them with ww_mutex_unlock() before ww_acquire_fini(). On error, the helper has already released everything and finalized the context.

The -EALREADY check is not optional. After ww_mutex_lock_slow() acquires the contended lock, the retry loop will encounter that lock again; -EALREADY tells the transaction to skip it rather than treating it as an error.

ww_mutex_lock_slow() is a readability wrapper around ww_mutex_lock() with a context that has no other locks held. It is not a separate algorithm, and it must be called only on the contended lock, only after releasing all other context locks. The interruptible variant is ww_mutex_lock_slow_interruptible(), which returns -EINTR if a signal arrives.

13.9 WW-Mutex Rules

• Every lock in a transaction must belong to the same ww class as the context.
• Do not mix ww-mutex locking with ordinary mutex_lock() on the same object.
• Do not use two different contexts for the same ww class in one transaction.
• Release every acquired ww-mutex before ww_acquire_fini().
• Use ww_mutex_unlock(), never mutex_unlock(), for a ww-mutex.
• Call ww_mutex_destroy() only after the final unlock has returned and the object is no longer reachable.
• A -EDEADLK means “unwind and retry”, not “the lock is permanently unavailable”.
• -EALREADY means the context already holds the lock; it is safe to skip that acquisition.
• Do not use a ww-mutex from interrupt context; it is still a sleeping lock.

WW-mutexes are a specialized tool. For ordinary two-lock ordering, a documented lock order and lockdep are simpler and faster. Use ww-mutexes when the acquisition order is genuinely determined by data and a fixed order cannot be enforced.

---

14. PREEMPT_RT Contrast

Under CONFIG_PREEMPT_RT, struct mutex is no longer the owner-word implementation described above. The same public API is backed by an rtmutex (include/linux/mutex_types.h):

        non-RT struct mutex                    PREEMPT_RT struct mutex
        ───────────────────                    ───────────────────────
        atomic_long_t owner                    struct rt_mutex_base
        raw_spinlock_t wait_lock                 wait_lock + wait tree
        first_waiter / mutex_waiter              rt_mutex_waiter
        optimistic_spin_queue osq                no mutex OSQ
        HANDOFF / PICKUP flags                   PI state in the task and tree
        magic / dep_map                          dep_map

#else /* CONFIG_PREEMPT_RT */
context_lock_struct(mutex) {
	struct rt_mutex_base rtmutex;
#ifdef CONFIG_DEBUG_LOCK_ALLOC
	struct lockdep_map dep_map;
#endif
};
#endif

The RT mutex supports priority inheritance. When a high-priority task blocks on a mutex held by a low-priority task, the holder temporarily inherits the waiter’s priority. That prevents the classic unbounded inversion in which a medium-priority task keeps preempting the holder. The implementation uses the rtmutex wait tree, priority queues, and the scheduler’s PI machinery rather than the mutex owner word.

The public behavior remains recognizable:

• only one task holds the mutex;
• only the owner unlocks;
• the lock may sleep;
• recursive locking is forbidden;
• the APIs are the same, so most callers do not need conditional compilation; and
• lockdep still tracks the lock class and ownership.

The internals differ enough that AArch64-specific statements in this document — CAS on owner, the OSQ, the HANDOFF/PICKUP bits, and the mutex’s own optimistic spinning — apply to the non-RT implementation. The RT path uses the rtmutex fast cmpxchg, the PI wait tree, and the rtmutex slow path (kernel/locking/rtmutex.c); it does not use the mutex OSQ or the non-RT handoff protocol. CONFIG_DEBUG_MUTEXES is also unavailable under PREEMPT_RT; RT has its own rtmutex debugging options.

The ww-mutex layer adapts as well: WW_MUTEX_BASE becomes struct rt_mutex, and __ww_ctx_less() can consider RT priorities before the stamp. The transaction rules and the -EDEADLK/-EALREADY contract remain the same.

For this document’s scope — non-RT SMP AArch64 — the owner-word path, the OSQ, and the HANDOFF protocol are the relevant implementation. The RT contrast is included so that statements about priority inheritance and data layout are not accidentally applied to an RT kernel.

---

15. Performance and Alternatives

15.1 Cost Model

The cost of a mutex operation depends on the contention regime:

Regime	Main cost	Mutex behavior
Uncontended	One acquire CAS, one release CAS	Fast path; no scheduler involvement
Brief contention, owner running	Cache-line polling; possible OSQ enqueue	Optimistic spin; may acquire without sleeping
Owner blocked or preempted	Scheduler wakeup and context switch	Sleep; handoff protects the head waiter
High contention	Queueing, cache-line transfer, wakeups	OSQ + wait list + handoff; throughput limited by the critical section
Lockdep enabled	Graph updates on every acquisition/release	Much slower, much more diagnostic

The fast path is not free: even an uncontended mutex performs an atomic read-modify-write with acquire semantics. A spinlock’s trylock can be cheaper for a critical section of a few instructions, but a mutex is usually the better default because it is safe to sleep, it has ownership checks, and it avoids burning CPU under contention.

15.2 Choosing an Alternative

Primitive	Best for	Avoid when
mutex	Process-context mutual exclusion, may sleep, medium/long sections	Interrupt context; signaling; counting; very short uncontended sections where a spinlock is measurably better
spinlock_t / raw_spinlock_t	Interrupt context, very short sections, low-level state	The critical section may sleep; long hold times
rw_semaphore	Read-mostly sleeping workloads with multiple readers	Writers are frequent; the read-side overhead is not justified
semaphore	Counting resources, producer/consumer signaling, cross-task release	Ordinary mutual exclusion where ownership matters
completion	One-shot or repeated signaling	Mutual exclusion
RCU	Read-mostly data with safe replacement and grace periods	Readers need to sleep or mutate the structure
Per-CPU data	Avoiding shared-cache-line contention entirely	The state must be globally consistent

A mutex is the right default for “protect this object in process context.” If profiling shows that the fast path or the wakeup latency is the bottleneck, measure before changing primitives; a different lock can move the cost rather than remove it.

15.3 Reducing Contention

• Shorten the critical section; move I/O, allocation, and copy operations outside the lock where possible.
• Avoid calling another sleeping lock while holding the mutex unless the order is documented.
• Split a global lock into per-object or per-shard locks when the protected state is independent.
• Use a read-mostly primitive when the workload is genuinely read-dominated.
• Keep the mutex and the protected data on separate cache lines if false sharing is measurable.
• Use mutex_trylock() for optional work rather than making every caller wait.
• Use lockdep and contention tracepoints to find the actual hot lock, not the one that looks hot in the source.

15.4 When Not to Use a Mutex

Do not use a mutex when:

• the code may run in interrupt, softirq, NMI, or with preemption disabled;
• the release must happen in a different task;
• the protected resource is a count or pool;
• the operation is a one-way wakeup or event;
• the critical section is so short and uncontended that a spinlock is measurably better; or
• the design requires non-blocking progress guarantees that a sleeping lock cannot provide.

Those are not mutex failures; they are signs that a different synchronization primitive matches the problem.

---

16. Summary

• A mutex is a sleeping, single-owner mutual-exclusion lock. It is not a counting semaphore and not a signaling primitive.
• Ownership is encoded in owner: a task_struct * with three low-bit flags, WAITERS, HANDOFF, and PICKUP.
• The fast path is one acquire CAS from zero to the current task. The fast unlock is one release CAS from the current task to zero.
• On AArch64, the fast path uses CASA on LSE hardware or an LDAXR/STXR loop on older cores. Unlock uses CASL or an LDXR/STLXR loop. Acquire and release ordering come from those instructions; no separate DMB is needed on the fast path.
• Contention first enters optimistic spinning. mutex_can_spin_on_owner() checks need_resched() and owner_on_cpu(); the OSQ serializes non-waiter spinners; a queued waiter may spin directly.
• If spinning fails, the task enqueues a stack-allocated waiter, records blocked_on, sleeps, and is woken by unlock.
• The wait list is FIFO for a normal mutex and stamp-ordered for a ww-mutex. Fairness is eventual, not strict: a newcomer can steal before handoff is requested, but a head waiter that sets HANDOFF gets a reserved transfer.
• __mutex_handoff() replaces the owner with next | PICKUP, preserving WAITERS; the designated task clears PICKUP on pickup. A fresh fast-path contender cannot steal during this window.
• CONFIG_DEBUG_MUTEXES adds magic-pointer checks, waiter poisoning, and slow-path owner warnings. CONFIG_PROVE_LOCKING/lockdep adds recursive-lock, ABBA, and context diagnostics. They are complementary; neither makes a broken design correct.
• A recursive lock is a self-deadlock. Lockdep is the reliable detector; CONFIG_DEBUG_MUTEXES alone is not a complete recursive-lock checker.
• Do not hold a mutex across open()/release() unless the same task is guaranteed to perform both. File descriptors can outlive the opening task’s context.
• PREEMPT_RT replaces the owner-word mutex with an rtmutex-backed mutex and adds priority inheritance. The API is the same; the internals and AArch64 fast-path details differ.
• Use a mutex for ordinary process-context mutual exclusion, a spinlock for interrupt/atomic context, and a semaphore or completion for counting and signaling.

The shortest correct mental model is:

mutex = owner + sleep + optional optimistic spin + eventual handoff

Everything else in the implementation follows from those four ideas.

---

17. References

All references are against Linux v7.3-rc7 unless noted otherwise.

File	Purpose
include/linux/mutex.h	Public API, initializers, lockdep wrappers, cleanup guards
include/linux/mutex_types.h	struct mutex, RT layout, semantic rules
kernel/locking/mutex.c	Fast/slow lock, unlock, handoff, optimistic spinning, ww-mutex entry points
kernel/locking/mutex.h	Internal flags, struct mutex_waiter, __mutex_owner()
kernel/locking/mutex-debug.c	Magic pointers, waiter poisoning, mutex_destroy()
kernel/locking/osq_lock.c	MCS/OSQ optimistic-spin queue
include/linux/osq_lock.h	OSQ type and initialization
include/linux/ww_mutex.h	WW-mutex public structures and API
kernel/locking/ww_mutex.h	WW-mutex stamp ordering, wound/die logic, wait-list insertion
kernel/locking/rtmutex.c	PREEMPT_RT fast path, PI wait tree, and slow path
include/linux/sched.h	task_struct alignment, blocked_on, and blocked_donor
arch/arm64/include/asm/atomic_lse.h	LSE CASA/CASL/CASAL templates
arch/arm64/include/asm/atomic_ll_sc.h	LL/SC cmpxchg and atomic sequences
arch/arm64/include/asm/lse.h	Runtime LSE/LL-SC selection
arch/arm64/include/asm/barrier.h	DMB variants and AArch64 barriers
arch/arm64/include/asm/cmpxchg.h	Acquire/release cmpxchg wrappers and compare-wait
arch/arm64/include/asm/vdso/processor.h	cpu_relax() / YIELD
arch/arm64/Kconfig	Queued spinlock/rwlock selection
kernel/Kconfig.locks	MUTEX_SPIN_ON_OWNER dependencies
Documentation/locking/mutex-design.rst	Design rationale and semantics
Documentation/locking/ww-mutex-design.rst	WW-mutex design and usage guidance
Documentation/locking/locktypes.rst	Sleeping versus spinning lock classes