Post

Spinlocks in the Linux Kernel

Posted
By Mukesh Pilaniya
108 min read
Spinlocks in the Linux Kernel

Spinlocks in the Linux Kernel

Concurrent execution is a fundamental reality of modern operating systems. On a multiprocessor system, multiple CPUs run code simultaneously, and any shared data structure that is read and modified by more than one CPU at a time is a candidate for corruption. The Linux kernel uses a family of synchronization primitives to prevent this, and the spinlock is the most fundamental of them all.

Unlike a mutex, which puts a waiting thread to sleep, a spinlock keeps the CPU busy in a tight polling loop until the lock becomes available. This characteristic makes spinlocks uniquely appropriate for short, atomic critical sections. As a practical rule of thumb, a spinlock should be held for less than the duration of two context switches – the overhead of sleeping and waking a thread – because beyond that point a sleeping lock becomes more efficient. The real criterion is stricter: a spinlock must never be held across a sleep, a blocking allocation, a user access that may fault, or any operation whose duration is unbounded.

On AArch64, the kernel’s spinlock implementation is not a naive test-and-set loop. It is a sophisticated queued spinlock (qspinlock) built on top of MCS nodes, designed to avoid having every waiter poll the same cache line. The queue tail is encoded into the lock word itself, while the actual MCS nodes live in per-CPU storage. The result is a 32-bit lock with MCS-like scalability and a fast path that is a single atomic compare-and-swap.

Understanding spinlocks from first principles – their data structures, memory-ordering requirements, interaction with preemption and interrupts, and the hardware instructions that make them work – is essential for anyone writing or reviewing kernel code.

This document describes spinlock_t as implemented in Linux v7.3-rc7 on AArch64, in a non-PREEMPT_RT SMP configuration. PREEMPT_RT changes the semantics of spinlock_t substantially and is covered in Section 14. Source links point at the v7.3-rc7 tag; kernel code evolves, so verify against the tree you are building.

Diagram conventions. Timelines run from top to bottom. Side-by-side columns represent concurrent CPUs or contexts, not a strict wall-clock ordering. In qspinlock diagrams, L, P, and T denote the locked, pending, and tail fields; lock-word values are hexadecimal unless noted.

Table of Contents

  1. Why Spinlocks Exist
  2. The spinlock_t Type on AArch64
  3. Initialization
  4. Core API and Call Chains
  5. Preemption, Recursion, and Scope
  6. Interrupt and Bottom-Half Contexts
  7. Sleeping and Uniprocessor Semantics
  8. AArch64 Implementation
  9. Memory Ordering on AArch64
  10. Writing Correct Spinlock Code
  11. Debugging and Observability
  12. Performance and Scalability
  13. Problems and Pitfalls
  14. PREEMPT_RT
  15. Summary: Rules of Thumb
  16. References

1. Why Spinlocks Exist

Atomic variables and spinlocks both protect shared data from concurrent modification, but they solve different problems at different granularities. Choosing between them is one of the first practical decisions a kernel developer makes.

1.1 The Atomic Alternative

Atomic variables (atomic_t, atomic64_t, and the bitops family) wrap a single integer value and expose operations – read, write, add, subtract, compare-and-swap – that are indivisible at the hardware level. On AArch64 without LSE, these compile to LDAXR/STLXR (Load-Acquire Exclusive / Store-Release Exclusive) loops. With LSE they become single instructions such as LDADD, STADD, or CASA.

The critical constraint is scope: an atomic operation works on exactly one word at a time. It cannot protect a multi-field data structure as a unit.

1
2
3
4
5
6
7
8
9
10

Atomic variable:                     Spinlock:
----------------                     ---------
One word, one operation              Any amount of data, any number of
                                     operations -- all serialized together

atomic_inc(&counter);                spin_lock(&lock);
                                     cache.hits++;
  ^ indivisible, no lock needed      cache.misses--;
                                     cache.last_update = jiffies;
                                     spin_unlock(&lock);

Use an atomic when the invariant you need to preserve fits inside a single read-modify-write on one value:

  • A reference count (atomic_inc, atomic_dec_and_test).
  • A global enable/disable flag (atomic_set, atomic_read).
  • A statistics counter where a slightly stale read is acceptable.
  • A bitmask where individual bit transitions must not overlap (set_bit, clear_bit, test_and_set_bit).
PropertyAtomic variableSpinlock
ScopeOne word (32 or 64 bits)Any amount of data
Operations per critical sectionExactly one read-modify-writeUnlimited
WaitingLock-free; retries on contentionBusy-spins or waits on an MCS node
Uncontended overheadOne LSE instruction or an LL/SC loopOne atomic RMW plus preemption-count updates
Contended overheadRetry traffic on one cache lineQueue keeps most polling local, but serialization remains
Disables preemptionNoYes (non-RT)
Disables interruptsNo, unless done explicitlyOnly with the _irq/_irqsave/_irq_disable variants
Protects a multi-field structureNoYes
Recursive acquisitionNot applicable; there is no ownershipDeadlocks
Usable in interrupt contextYesYes, with the correct IRQ-safe variant

A reference-counted open path is a typical atomic-only pattern:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

static atomic_t open_count = ATOMIC_INIT(0);

static int device_open(struct inode *inode, struct file *file)
{
    if (atomic_inc_return(&open_count) > 1) {
        atomic_dec(&open_count);
        return -EBUSY;
    }
    return 0;
}

static int device_release(struct inode *inode, struct file *file)
{
    atomic_dec(&open_count);
    return 0;
}

1.2 When a Spinlock Is Required

Use a spinlock when multiple fields must remain consistent with each other, or when the read-modify-write spans more than one memory location:

1
2
3
4
5
6
7
8
9
10

/* Wrong: two separate atomics do not protect the pair as a unit. */
atomic_inc(&stats.hits);
atomic_set(&stats.last_cpu, smp_processor_id());
/* A reader can see hits incremented but last_cpu not yet updated. */

/* Correct: the spinlock makes the pair atomic. */
spin_lock(&stats_lock);
stats.hits++;
stats.last_cpu = smp_processor_id();
spin_unlock(&stats_lock);

A spinlock also serializes operations that are not expressible as a single atomic instruction: inserting into a linked list, updating a red-black tree, moving an object between queues, or checking a multi-field state machine.

Key decision rule: if the entire critical section is a single read-modify-write on one variable, prefer an atomic. The moment two or more memory locations must stay consistent with each other, use a spinlock.

1.3 Spinlock vs. Sleeping Lock

The dividing line between a spinlock and a mutex is what the waiter is allowed to do. A spinlock waiter cannot sleep; a mutex waiter can.

PropertySpinlockMutex
Wait behaviorBusy-spins, never sleepsSleeps until woken
ContextProcess, softirq, hard IRQ (with the right suffix)Process context only
PreemptionDisabled (non-RT)Enabled while blocked
Typical hold timeA few instructions to a few microsecondsLonger, may include blocking operations
May sleep insideNeverYes
Priority inversionBounded by hold time; no PI on non-RTNo PI for a regular mutex on non-RT; RT-mutex PI on RT
PREEMPT_RTspinlock_t becomes an RT mutex; raw_spinlock_t still spinsRemains a sleeping lock

The practical decision is:

1
2
3
4
5
6
7
8
9

Can the critical section sleep or block?
        |
        +-- YES --> mutex, semaphore, rw_semaphore, or completion
        |
        +-- NO --> Is the protected state a single word?
                     |
                     +-- YES --> atomic_t / atomic64_t / bitops
                     |
                     +-- NO --> spinlock_t (with the correct _irq/_bh suffix)

1.4 The Cost Model

An uncontended spinlock is cheap: one atomic compare-and-swap plus a preemption-count update. A contended spinlock is expensive because every waiter burns CPU cycles that could otherwise be used for useful work.

1
2
3
4
5
6
7
8
9

Uncontended:
  spin_lock   -> CASA success (one atomic instruction) + preempt_disable
  critical    -> a few instructions
  spin_unlock -> STLRB + preempt_enable

Contended:
  holder      -> keeps running
  waiters     -> poll (or WFE) until the lock is free
  cost        -> proportional to the hold time and the number of waiters

The hold-time rule follows directly from this cost model:

  • Keep the critical section short. Prefer a handful of instructions. Move logging, allocation, copying, and wakeups outside the lock.
  • Do not hold a spinlock across a sleep. This is a correctness rule, not just a performance rule; see Section 7.
  • If the expected wait is longer than the cost of two context switches, use a mutex. The common “two context switches” formulation is a heuristic; the hard rule is that the section must be atomic and bounded.
  • Do not add a spinlock to a path that does not need mutual exclusion. Per-CPU data, RCU, atomics, and seqlocks are often better.

2. The spinlock_t Type on AArch64

Every user-facing spinlock_t is a layered abstraction. On a non-PREEMPT_RT SMP kernel targeting AArch64, the types nest as follows:

1
2
3
4
5

spinlock_t                          [include/linux/spinlock_types.h]
+-- raw_spinlock_t  (rlock)         [include/linux/spinlock_types_raw.h]
    +-- arch_spinlock_t (raw_lock)  [arch/arm64/include/asm/spinlock_types.h]
        +-- struct qspinlock (val)  [include/asm-generic/qspinlock_types.h]
            +-- atomic_t  (u32)     <- single 32-bit word in memory

2.1 Type Hierarchy

spinlock_t is defined in include/linux/spinlock_types.h:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

#ifndef CONFIG_PREEMPT_RT

/* Non PREEMPT_RT kernels map spinlock to raw_spinlock */
context_lock_struct(spinlock) {
    union {
        struct raw_spinlock rlock;

#ifdef CONFIG_DEBUG_LOCK_ALLOC
# define LOCK_PADSIZE (offsetof(struct raw_spinlock, dep_map))
        struct {
            u8 __padding[LOCK_PADSIZE];
            struct lockdep_map dep_map;
        };
#endif
    };
};
typedef struct spinlock spinlock_t;

The context_lock_struct() wrapper is part of the kernel’s optional context-analysis support. In a normal build it expands to a plain struct; when WARN_CONTEXT_ANALYSIS is enabled, it adds Clang context-analysis annotations so the compiler can check that data guarded by the lock is accessed only while the lock is held. It does not change the runtime layout in a normal build (see include/linux/compiler-context-analysis.h).

On PREEMPT_RT, the same spinlock_t name is instead a wrapper around struct rt_mutex_base; that variant is described in Section 14.

2.2 Raw Spinlock and Debug Fields

raw_spinlock_t is defined in include/linux/spinlock_types_raw.h:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

context_lock_struct(raw_spinlock) {
    arch_spinlock_t raw_lock;
#ifdef CONFIG_DEBUG_SPINLOCK
    unsigned int magic, owner_cpu;
    void *owner;
#endif
#ifdef CONFIG_DEBUG_LOCK_ALLOC
    struct lockdep_map dep_map;
#endif
};
typedef struct raw_spinlock raw_spinlock_t;

#define SPINLOCK_MAGIC      0xdead4ead
#define SPINLOCK_OWNER_INIT ((void *)-1L)

The debug fields are deliberately parallel to those of other kernel locks:

  • magic catches corruption and use-after-free.
  • owner_cpu records the CPU that holds the lock.
  • owner records the task that holds the lock.
  • dep_map connects the lock instance to lockdep’s class graph.

In a production kernel without CONFIG_DEBUG_SPINLOCK or CONFIG_DEBUG_LOCK_ALLOC, raw_spinlock_t is just the architecture lock word.

2.3 The qspinlock Structure

On AArch64, arch/arm64/include/asm/spinlock_types.h delegates to the generic queued spinlock:

1
2

#include <asm-generic/qspinlock_types.h>
#include <asm-generic/qrwlock_types.h>

The qspinlock itself, from include/asm-generic/qspinlock_types.h, is a single 32-bit atomic word with overlapping views:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

typedef struct qspinlock {
    union {
        atomic_t val;

        /*
         * By using the whole 2nd least significant byte for the
         * pending bit, we can allow better optimization of the lock
         * acquisition for the pending bit holder.
         */
#ifdef __LITTLE_ENDIAN
        struct {
            u8  locked;
            u8  pending;
        };
        struct {
            u16 locked_pending;
            u16 tail;
        };
#else
        struct {
            u16 tail;
            u16 locked_pending;
        };
        struct {
            u8 reserved[2];
            u8 pending;
            u8 locked;
        };
#endif
    };
} arch_spinlock_t;

#define __ARCH_SPIN_LOCK_UNLOCKED { { .val = ATOMIC_INIT(0) } }

The union is the entire trick: the fast path can compare and swap the whole word, the pending waiter can poll the locked byte and claim through locked_pending, and the slow path can exchange the tail half-word without a separate pointer.

2.4 Lock Word Bit Layout

On little-endian AArch64, with the default CONFIG_NR_CPUS range, the word is laid out as follows. tail encodes both the CPU number and the per-CPU MCS-node index; the CPU field stores cpu + 1 so that an all-zero tail means “no queue”.

1
2
3
4
5
6
7
8

Bit 31                  16  15       8  7       0
+--------------------------+----------+----------+
|   tail  (CPU+1 | index)  | pending  |  locked  |
+--------------------------+----------+----------+
       ^                          ^           ^
  MCS queue tail            fast waiter    lock bit
  encoded as                slot (1 CPU)   0 = free
  (cpu+1, per-cpu-idx)                     1 = held
FieldBitsWidthMeaning
locked0-78 bLock is held (1) or free (0)
pending8-158 bOne CPU is waiting in the fast-path slot
tail16-3116 bIdentity of the last CPU in the MCS queue

The source also defines a compact layout for machines with CONFIG_NR_CPUS >= 16384, where the pending field shrinks to one bit and the tail grows:

FieldNR_CPUS < 16KNR_CPUS >= 16K
lockedbits 0-7bits 0-7
pendingbits 8-15bit 8
tail indexbits 16-17bits 9-10
tail CPU (cpu + 1)bits 18-31bits 11-31

See the bitfield comments in qspinlock_types.h. The rest of this document assumes the common < 16K layout; the algorithm is the same.

The unlocked state is the all-zero word. The qspinlock source explicitly notes that any non-zero value counts as locked for queued_spin_is_locked(), because the lock word may contain a queued tail even while the locked byte is temporarily zero during a handoff (include/asm-generic/qspinlock.h).

2.5 Size, Alignment, and Cache-Line Behavior

The non-debug spinlock_t is 4 bytes: exactly one qspinlock word. That compactness is why qspinlock replaced the older ticket lock without increasing the size of the many spinlocks embedded in kernel structures.

1
2
3
4
5
6
7
8
9

Non-debug SMP build:
  spinlock_t          = 4 bytes
  raw_spinlock_t      = 4 bytes
  arch_spinlock_t     = 4 bytes
  qspinlock.val       = 4 bytes

Debug build (conceptually):
  raw_spinlock_t      = raw_lock + magic + owner_cpu + owner (+ padding)
  spinlock_t          = raw_spinlock_t (+ lockdep map with CONFIG_DEBUG_LOCK_ALLOC)

The qspinlock’s per-CPU queue nodes are separate. A struct qnode contains a struct mcs_spinlock; on a native 64-bit build the MCS node is 16 bytes, so four nodes fit in one 64-byte cache line. CONFIG_PARAVIRT_SPINLOCKS pads the qnode to 32 bytes, so only two fit in that case. The kernel allocates four nested nodes per CPU and falls back to direct spinning if the nesting depth exceeds four (kernel/locking/qspinlock.h, kernel/locking/qspinlock.c).

The lock word itself is a shared cache line. False sharing matters: two hot spinlocks in the same cache line will invalidate each other even when they protect unrelated data. For frequently contended locks, place them on separate cache lines (for example with ____cacheline_aligned) and keep read-mostly data away from the same line.

3. Initialization

3.1 Static Initialization

DEFINE_SPINLOCK is the normal way to initialize a lock with static storage duration:

1

static DEFINE_SPINLOCK(my_lock);

Its expansion chain is:

1
2
3
4
5
6
7
8
9

DEFINE_SPINLOCK(x)
  -> spinlock_t x = __SPIN_LOCK_UNLOCKED(x)
     -> __SPIN_LOCK_INITIALIZER(x)
        -> ___SPIN_LOCK_INITIALIZER(x)
           -> .raw_lock = __ARCH_SPIN_LOCK_UNLOCKED
              .magic    = SPINLOCK_MAGIC          (if DEBUG_SPINLOCK)
              .owner_cpu = -1                     (if DEBUG_SPINLOCK)
              .owner    = SPINLOCK_OWNER_INIT     (if DEBUG_SPINLOCK)
              .dep_map  = { .name = "x", ... }    (if DEBUG_LOCK_ALLOC)

A complete module using a static lock:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/spinlock.h>

MODULE_LICENSE("GPL");

static DEFINE_SPINLOCK(my_lock);

static int __init test_hello_init(void)
{
    spin_lock(&my_lock);
    pr_info("Starting critical region\n");
    pr_info("Ending critical region\n");
    spin_unlock(&my_lock);
    return 0;
}

static void __exit test_hello_exit(void) {}

module_init(test_hello_init);
module_exit(test_hello_exit);

This example is deliberately trivial: the lock protects no shared data. In real code the lock must be associated with a specific data structure, as described in Section 10.

3.2 Dynamic Initialization

Use spin_lock_init() when the lock is embedded in dynamically allocated memory or is initialized at runtime:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/slab.h>
#include <linux/spinlock.h>

MODULE_LICENSE("GPL");

struct my_state {
    spinlock_t lock;
    unsigned int counter;
};

static struct my_state *state;

static int __init test_hello_init(void)
{
    state = kmalloc(sizeof(*state), GFP_KERNEL);
    if (!state)
        return -ENOMEM;

    spin_lock_init(&state->lock);
    state->counter = 0;

    spin_lock(&state->lock);
    state->counter++;
    spin_unlock(&state->lock);

    pr_info("counter = %u\n", state->counter);
    kfree(state);
    state = NULL;
    return 0;
}

static void __exit test_hello_exit(void) {}

module_init(test_hello_init);
module_exit(test_hello_exit);

The lock is initialized before the object is published to any other context. That ordering is not optional: another CPU that sees the object must also see a fully initialized lock.

3.3 What Initialization Actually Does

spin_lock_init() is a macro. In a debug build it calls __raw_spin_lock_init() with a static lock_class_key tied to the call site; in a non-debug build it simply assigns the unlocked initializer:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#ifdef CONFIG_DEBUG_SPINLOCK

# define spin_lock_init(lock)                    \
do {                                            \
    static struct lock_class_key __key;        \
                                                \
    __raw_spin_lock_init(spinlock_check(lock),  \
                         #lock, &__key, LD_WAIT_CONFIG); \
} while (0)

#else

# define spin_lock_init(_lock)                  \
do {                                            \
    spinlock_check(_lock);                      \
    *(_lock) = __SPIN_LOCK_UNLOCKED(_lock);     \
} while (0)

#endif

See include/linux/spinlock.h. __raw_spin_lock_init() sets the raw lock to the unlocked state, initializes magic and the owner fields, and registers the lockdep map when lockdep is enabled (kernel/locking/spinlock_debug.c).

Without CONFIG_DEBUG_SPINLOCK and CONFIG_DEBUG_LOCK_ALLOC, the lock word is zero and there is nothing else to initialize. The important invariant is the same in both cases: never hand-initialize a lock by writing fields directly, and never reinitialize a lock that another CPU may still be using.

3.4 Lifetime Rules

  • Initialize before publication. A lock must be fully initialized before the object containing it becomes visible to another CPU.
  • Keep the address stable. A spinlock must not move while it can be used. Do not memcpy a structure containing a lock, and do not reallocate it with krealloc without quiescing all users.
  • Do not free a lock while it is held or awaited. A waiter holds a pointer to the lock word; freeing or reusing the memory corrupts the queue.
  • Do not reinitialize a held lock. Lockdep’s debug_check_no_locks_freed() checks catch some cases, but production kernels may not.
  • Use the matching initializer. DEFINE_SPINLOCK for static storage, spin_lock_init() for dynamic storage. For per-CPU locks, use the per-CPU initialization helpers where appropriate.

4. Core API and Call Chains

4.1 Acquire Call Chain

The public spin_lock() is a thin wrapper around the raw spinlock layer. On a non-PREEMPT_RT kernel the chain is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

spin_lock(spinlock_t *lock)                    [include/linux/spinlock.h]
  -> raw_spin_lock(&lock->rlock)
       -> _raw_spin_lock(raw_spinlock_t *)     [include/linux/spinlock_api_smp.h]
            -> __raw_spin_lock()
                 preempt_disable()
                 spin_acquire(&lock->dep_map, ...)
                 LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock)
                   |
                   +-- do_raw_spin_trylock()  [only when CONFIG_LOCK_STAT]
                   |     -> arch_spin_trylock()
                   |          -> queued_spin_trylock()
                   |
                   +-- do_raw_spin_lock()
                         -> arch_spin_lock()
                              -> queued_spin_lock()
                                   -> atomic_try_cmpxchg_acquire()
                                   -> queued_spin_lock_slowpath() on failure

The corresponding source is include/linux/spinlock_api_smp.h:

1
2
3
4
5
6

static inline void __raw_spin_lock(raw_spinlock_t *lock)
{
    preempt_disable();
    spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
    LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
}

LOCK_CONTENDED() has two forms. With CONFIG_LOCK_STAT, it first tries do_raw_spin_trylock() and reports contention if that fails; otherwise it calls do_raw_spin_lock() directly (include/linux/lockdep.h). Either way, the architectural acquisition is either queued_spin_trylock() (trylock success) or queued_spin_lock() (contended path).

The fast path in include/asm-generic/qspinlock.h is a single compare-and-swap:

1
2
3
4
5
6
7
8
9

static __always_inline void queued_spin_lock(struct qspinlock *lock)
{
    int val = 0;

    if (likely(atomic_try_cmpxchg_acquire(&lock->val, &val, _Q_LOCKED_VAL)))
        return;

    queued_spin_lock_slowpath(lock, val);
}

If the word is zero, the compare-and-swap changes it to _Q_LOCKED_VAL and the caller owns the lock. If the word is non-zero, the slow path decides whether to use the pending slot or queue on an MCS node.

4.2 Release Call Chain

The release path is shorter, but the ordering is just as important:

1
2
3
4
5
6
7
8
9
10
11
12
13

spin_unlock(spinlock_t *lock)                  [include/linux/spinlock.h]
  -> raw_spin_unlock(&lock->rlock)
       -> __raw_spin_unlock()
            spin_release(&lock->dep_map, ...)
            do_raw_spin_unlock()
              mmiowb_spin_unlock()              [when configured]
              debug_spin_unlock()               [CONFIG_DEBUG_SPINLOCK]
              arch_spin_unlock()
                -> queued_spin_unlock()
                     -> queued_spin_release()
                          -> smp_store_release(&lock->locked, 0)
                               -> STLRB
            preempt_enable()

The key instruction is the store-release in include/asm-generic/qspinlock.h:

1
2
3
4
5
6
7

static __always_inline void queued_spin_release(struct qspinlock *lock)
{
    /*
     * unlock() needs release semantics:
     */
    smp_store_release(&lock->locked, 0);
}

On AArch64, a byte-sized smp_store_release() compiles to STLRB (Store-Release Byte), defined in arch/arm64/include/asm/barrier.h. The release barrier ensures that all prior memory accesses inside the critical section are visible before the lock is published as free.

The source order in __raw_spin_unlock() is deliberate: spin_release() runs before the lock is released, do_raw_spin_unlock() performs the architectural release, and only then does preempt_enable() run. Reversing those steps would allow a new lock holder to run while lockdep still believes the old holder owns the lock, and it would publish the lock-free state before the preemption bookkeeping is consistent.

4.3 spin_trylock

spin_trylock() attempts to acquire the lock without waiting. It returns 1 on success and 0 on failure. A failed trylock leaves the lock state and the preemption count exactly as they were.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

spin_trylock(&lock)
      |
      +-- preempt_disable()
      |
      +-- atomic_read(&lock->val) -- non-zero? -- YES --> preempt_enable()
      |                                                       |
      |                                                  return 0
      |
      +-- val == 0: atomic_try_cmpxchg_acquire(&val, 0, _Q_LOCKED_VAL)
                |
          +-----+-----+
        SUCCESS     FAILURE
          |            |
      return 1    preempt_enable()
                       |
                  return 0

The qspinlock implementation is in include/asm-generic/qspinlock.h:

1
2
3
4
5
6
7
8
9

static __always_inline int queued_spin_trylock(struct qspinlock *lock)
{
    int val = atomic_read(&lock->val);

    if (unlikely(val))
        return 0;

    return likely(atomic_try_cmpxchg_acquire(&lock->val, &val, _Q_LOCKED_VAL));
}

On AArch64 with LSE atomics, atomic_try_cmpxchg_acquire() is a single CASA instruction. Without LSE it is an LDAXR/STXR loop. spin_trylock() is appropriate when the caller has a meaningful fallback – for example, a timer callback that can reschedule itself, or a best-effort path that can drop work instead of waiting.

On a uniprocessor non-debug kernel, _raw_spin_trylock() is implemented as an unconditional success. On a uniprocessor debug kernel, a failed trylock triggers a SPIN_BUG_ON() because it should be impossible (include/linux/spinlock_up.h, kernel/locking/spinlock_debug.c).

A character device that wants to reject a second opener can use spin_trylock() to guard a short busy flag update. The lock is held only while the flag is read and changed; it is never held for the lifetime of the open file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

static DEFINE_SPINLOCK(open_lock);
static bool device_busy;

static int device_open(struct inode *inode, struct file *file)
{
    if (!spin_trylock(&open_lock))
        return -EBUSY;

    if (device_busy) {
        spin_unlock(&open_lock);
        return -EBUSY;
    }

    device_busy = true;
    spin_unlock(&open_lock);
    return 0;
}

static int device_release(struct inode *inode, struct file *file)
{
    spin_lock(&open_lock);
    device_busy = false;
    spin_unlock(&open_lock);
    return 0;
}

Releasing a spinlock from a different task than the one that acquired it is not a supported ownership model. Holding this lock from device_open() to device_release() would span arbitrary time and a different task; that is exactly the kind of long, cross-context ownership a spinlock must not have.

4.4 spin_is_locked, spin_is_contended, and spin_needbreak

spin_is_locked() returns non-zero if the lock is observed to be held:

1
2

if (spin_is_locked(&my_lock))
    pr_warn("lock is held during probe!\n");

It is a hint, not a synchronization operation. It provides no memory ordering guarantee, it does not acquire the lock, and the answer can change immediately after it returns. On a uniprocessor build without CONFIG_DEBUG_SPINLOCK it always returns zero (include/linux/spinlock.h).

spin_is_contended() returns non-zero when the lock word has a pending waiter or a queued tail. It is used by spin_needbreak() to decide whether a long-running critical section should be broken voluntarily on a preemptible kernel:

1
2
3
4
5
6
7

static inline int spin_needbreak(spinlock_t *lock)
{
    if (!preempt_model_preemptible())
        return 0;

    return spin_is_contended(lock);
}

See include/linux/spinlock.h. spin_needbreak() is a latency optimization, not a correctness mechanism. It is meaningful only on kernels where preemption is possible; on a non-preemptible kernel it returns zero.

assert_spin_locked() is a diagnostic assertion that the lock word is currently locked. On SMP it expands to BUG_ON(!raw_spin_is_locked()); on UP it is a no-op. It does not verify ownership, so it is not a substitute for correct locking and must not be used to protect data (include/linux/spinlock_api_smp.h).

4.5 Complete API Table

APIAcquires?DisablesTypical use
DEFINE_SPINLOCK(lock)––Static initialization
spin_lock_init(lock)––Dynamic initialization
spin_lock(lock)YesPreemptionProcess context, or interrupt context with local IRQs already off
spin_unlock(lock)ReleasesRe-enables preemptionMatches spin_lock
spin_lock_bh(lock)YesPreemption + softirqsProcess context vs. BH
spin_unlock_bh(lock)ReleasesRe-enables BH + preemptionMatches spin_lock_bh
spin_lock_irq(lock)YesPreemption + maskable IRQsIRQ state known enabled
spin_unlock_irq(lock)ReleasesEnables IRQs + preemptionMatches spin_lock_irq
spin_lock_irqsave(lock, flags)YesPreemption + IRQs, saves stateIRQ state unknown or nested
spin_unlock_irqrestore(lock, flags)ReleasesRestores saved IRQ state + preemptionMatches spin_lock_irqsave
spin_lock_irq_disable(lock)YesPreemption + counted IRQ disableNested IRQ-disabling sections
spin_unlock_irq_enable(lock)ReleasesCounted IRQ enable + preemptionMatches spin_lock_irq_disable
spin_trylock(lock)MaybePreemption only on successNon-blocking fallback
spin_trylock_bh(lock)MaybePreemption + BH on successNon-blocking BH-protected acquire
spin_trylock_irq(lock)MaybePreemption + IRQs on successNon-blocking IRQ-protected acquire
spin_trylock_irqsave(lock, flags)MaybeSaves IRQ state; disables on successNon-blocking, state unknown
spin_trylock_irq_disable(lock)MaybeCounted IRQ disable on successNon-blocking nested IRQ disable
spin_is_locked(lock)NoNoDebug hint only
spin_is_contended(lock)NoNoContention hint
spin_needbreak(lock)NoNoVoluntary preemption point
spin_lock_nested(lock, subclass)YesPreemptionLockdep subclass for nesting
spin_lock_nest_lock(lock, nest_lock)YesPreemptionLockdep nesting annotation
assert_spin_locked(lock)NoNoLocked-state assertion

4.6 A Complete kthread Example

Two kernel threads share a counter. The writer updates it under the lock; the reader copies it under the lock and prints outside. Both sleep outside the critical section.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

#include <linux/delay.h>
#include <linux/kernel.h>
#include <linux/kthread.h>
#include <linux/module.h>
#include <linux/spinlock.h>

MODULE_LICENSE("GPL");

static unsigned int counter;
static DEFINE_SPINLOCK(counter_lock);
static struct task_struct *read_thread;
static struct task_struct *write_thread;

static int writer_function(void *data)
{
    while (!kthread_should_stop()) {
        spin_lock(&counter_lock);
        counter++;
        spin_unlock(&counter_lock);
        msleep(500);
    }
    return 0;
}

static int read_function(void *data)
{
    while (!kthread_should_stop()) {
        unsigned int value;

        spin_lock(&counter_lock);
        value = counter;
        spin_unlock(&counter_lock);

        pr_info("counter: %u\n", value);
        msleep(500);
    }
    return 0;
}

static int __init my_mod_init(void)
{
    read_thread = kthread_run(read_function, NULL, "read-thread");
    if (IS_ERR(read_thread))
        return PTR_ERR(read_thread);

    write_thread = kthread_run(writer_function, NULL, "write-thread");
    if (IS_ERR(write_thread)) {
        kthread_stop(read_thread);
        read_thread = NULL;
        return PTR_ERR(write_thread);
    }

    return 0;
}

static void __exit my_mod_exit(void)
{
    if (write_thread)
        kthread_stop(write_thread);
    if (read_thread)
        kthread_stop(read_thread);
}

module_init(my_mod_init);
module_exit(my_mod_exit);

The lock protects the counter, not the pr_info() call. Logging inside the lock would work in many configurations, but it lengthens the critical section and can interact with console paths; copy the value out and log after unlocking.

5. Preemption, Recursion, and Scope

5.1 Preemption Is Disabled

On a non-PREEMPT_RT kernel, acquiring a spinlock disables kernel preemption for the duration of the critical section. preempt_disable() increments the preemption count stored in the current task’s thread_info:

1
2
3
4
5
6

static inline void __preempt_count_add(int val)
{
    u32 pc = READ_ONCE(current_thread_info()->preempt.count);
    pc += val;
    WRITE_ONCE(current_thread_info()->preempt.count, pc);
}

See arch/arm64/include/asm/preempt.h. The AArch64 thread_info packs the 32-bit preemption count and the 32-bit need_resched field into one 64-bit union on little-endian systems (arch/arm64/include/asm/thread_info.h):

1
2
3
4
5
6
7
8

thread_info->preempt_count (64 bits on little-endian AArch64):

  +--------------------------------+--------------------------------+
  |  need_resched (bits 32-63)     |  preempt count  (bits 0-31)    |
  +--------------------------------+--------------------------------+
                                              |
                              low 32 bits non-zero -> preemption disabled
                              low 32 bits zero     -> preemption allowed

PREEMPT_NEED_RESCHED is defined as BIT(32) in arch/arm64/include/asm/preempt.h. The count portion is incremented by preempt_disable() and decremented by preempt_enable(). If a reschedule became pending while the count was non-zero, preempt_enable() calls preempt_schedule() on the way back to zero.

Why disable preemption? Consider a task that holds a spinlock and is preempted by a higher-priority task on the same CPU. Another CPU may spin on the lock, but the holder cannot run until the higher-priority task yields. On a non-RT kernel, disabling preemption bounds that delay to the critical section itself. It does not eliminate contention, and it does not provide priority inheritance; it simply prevents the holder from being involuntarily descheduled mid-section.

Two qualifications matter:

  • On a CONFIG_PREEMPT_NONE build, the scheduler does not involuntarily preempt kernel code anyway. If CONFIG_PREEMPT_COUNT is enabled (for example for debug checks or dynamic preemption), the count still tracks atomic context, but the “prevents preemption” effect is less visible; otherwise preempt_disable() is only a compiler barrier.
  • On PREEMPT_RT, spinlock_t does not disable preemption; it becomes a sleeping RT mutex. See Section 14.

5.2 Spinlocks Are Not Recursive

A non-debug spinlock has no notion of ownership. The hardware state is a single lock word: there is no CPU identity, no task pointer, and no nesting depth. If a CPU that already holds a spinlock tries to acquire it again, it spins forever on a lock that only it can release.

1
2
3
4
5
6
7
8
9
10
11
12
13

CPU 0 holds lock_X
      |
      +-- calls spin_lock(&lock_X) again
                |
                v
         spins on the locked byte
                |
         locked byte stays 1
         (only CPU 0 can clear it,
          but CPU 0 is stuck spinning)
                |
                v
           DEADLOCK

The API contract says that the context that acquired the lock must release it, and must not acquire it recursively. In a production kernel, that contract is not enforced by the hardware.

CONFIG_DEBUG_SPINLOCK adds magic, owner_cpu, and owner fields, checks them before acquisition and on unlock, and records the owner after acquisition. When recursion is detected, spin_bug() calls debug_locks_off(), prints a report such as BUG: spinlock recursion, and dumps the stack. It does not abort the acquisition: after the report, the CPU proceeds to arch_spin_lock() and still deadlocks. The report is a diagnostic, not a recovery mechanism (kernel/locking/spinlock_debug.c). Lockdep also detects recursive acquisition when it is enabled.

A module that hangs at load time because it tries to acquire a lock it already holds:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/spinlock.h>

MODULE_LICENSE("GPL");

static DEFINE_SPINLOCK(my_lock);

static int __init test_hello_init(void)
{
    spin_lock(&my_lock);
    pr_info("Starting critical region\n");
    spin_lock(&my_lock);      /* Deadlock: spins forever waiting for itself. */
    pr_info("Ending critical region\n");
    spin_unlock(&my_lock);
    return 0;
}

static void __exit test_hello_exit(void) {}

module_init(test_hello_init);
module_exit(test_hello_exit);

The same rule applies to helper functions: if foo() takes lock_A, do not call foo() from a context that already holds lock_A. Refactor the helper into a locked core and an unlocked wrapper, or pass a flag that tells the helper the lock is already held.

5.3 Spinlocks Are Global Objects

A spinlock lives at one fixed address in memory. Every CPU that uses that lock contends for the same word. The qspinlock design ensures that only one CPU holds the lock at a time while other CPUs either wait in the pending slot or queue on their private MCS node.

1
2
3
4
5
6
7
8
9

System memory
+-----------------------+
|  arch_spinlock_t      |  <- one instance, visible to all CPUs
| [ tail | pend | lock ]|
+-----------------------+
        ^         ^         ^
        |         |         |
      CPU 0     CPU 1     CPU 2
     (holds)   (pending)  (queued on MCS node)

This is different from per-CPU data, where each CPU has its own copy and no cross-CPU serialization is required. If the data is naturally per-CPU, use get_cpu_var() / put_cpu_var() or the per-CPU accessors with preemption disabled instead of adding a global lock. A global spinlock serializes every CPU; per-CPU data does not.

A lock embedded in a structure protects that structure only if every access uses the same lock. Two different locks protecting the same field do not provide mutual exclusion, and one lock protecting two unrelated structures creates false contention. Section 10 develops this point.

6. Interrupt and Bottom-Half Contexts

Spinlocks protect data shared between process context and interrupt context. The danger is not another CPU; it is the same CPU interrupting itself. Unless stated otherwise, this section describes non-PREEMPT_RT kernels; Section 14 covers the RT changes.

6.1 The Same-CPU IRQ Deadlock

1
2
3
4
5
6
7
8
9
10
11
12

CPU 0 -- process context
  +-- spin_lock(&lock)          <- locked = 1
  +-- ... critical section ...
  |
  |   <- hardware interrupt fires on CPU 0
  |
  +-- interrupt handler begins
        +-- spin_lock(&lock)    <- locked == 1, spins forever
                                     (the holder is CPU 0, suspended at IRQ entry)
                                     DEADLOCK
  v
  (never reached)

The interrupt handler cannot make progress because the lock holder is the very context that the interrupt interrupted. The holder cannot resume until the handler returns, and the handler cannot return until it acquires the lock. Disabling interrupts on the local CPU before acquiring the lock breaks the cycle.

6.2 spin_lock_irq on AArch64

On a non-PREEMPT_RT kernel, spin_lock_irq() disables maskable interrupts on the local CPU and then acquires the lock. On AArch64, the normal path sets the I (IRQ) and F (FIQ) bits of the DAIF register with msr daifset, #3:

1
2
3
4
5
6

static __always_inline void __daif_local_irq_disable(void)
{
    barrier();
    asm volatile("msr daifset, #3");
    barrier();
}

See arch/arm64/include/asm/irqflags.h. DAIF stands for Debug, SError, IRQ, and FIQ. #3 sets the two low-order mask bits corresponding to IRQ and FIQ; debug and SError are not masked by this operation.

On systems with GIC priority masking (CONFIG_ARM64_PSEUDO_NMI), the kernel does not use DAIF for normal IRQ masking. It writes GIC_PRIO_IRQOFF to ICC_PMR_EL1 instead. The enable and restore paths call pmr_sync() so the new priority is visible to the GIC:

1
2
3
4
5
6

static __always_inline void __pmr_local_irq_disable(void)
{
    barrier();
    write_sysreg_s(GIC_PRIO_IRQOFF, SYS_ICC_PMR_EL1);
    barrier();
}

See arch/arm64/include/asm/irqflags.h. The _irqsave/_irqrestore path checks system_uses_irq_prio_masking() and saves/restores either DAIF or the PMR value; spin_unlock_irq() enables interrupts unconditionally.

spin_lock_irq() is safe only when interrupts are known to be enabled at the call site. If they were already disabled, spin_unlock_irq() will enable them, producing the wrong final state.

6.3 spin_lock_irqsave and the flags Argument

spin_lock_irqsave() saves the current interrupt state into flags before disabling interrupts, and spin_unlock_irqrestore() restores that exact state:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

spin_lock_irqsave(&lock, flags)
      |
      +-- local_irq_save(flags)
      |     +-- flags = read_sysreg(daif)       [normal path]
      |     |   or flags = read_sysreg_s(ICC_PMR_EL1) [priority masking]
      |     +-- disable IRQ+FIQ or lower PMR
      |
      +-- preempt_disable()
      +-- spin_acquire()
      +-- LOCK_CONTENDED(...)

... critical section ...

spin_unlock_irqrestore(&lock, flags)
      +-- spin_release()
      +-- do_raw_spin_unlock()       (STLRB locked=0)
      +-- local_irq_restore(flags)   (restore DAIF or PMR)
      +-- preempt_enable()

The ordering of the last two steps is important: interrupts are restored before preemption is re-enabled. The kernel source is explicit:

1
2
3
4
5
6
7
8

static inline void __raw_spin_unlock_irqrestore(raw_spinlock_t *lock,
                                                unsigned long flags)
{
    spin_release(&lock->dep_map, _RET_IP_);
    do_raw_spin_unlock(lock);
    local_irq_restore(flags);
    preempt_enable();
}

See include/linux/spinlock_api_smp.h. The acquire side saves and disables interrupts before preempt_disable(), so the critical section is protected against both interrupt-context reentry and task preemption.

flags is not a boolean. It is the literal hardware register value captured at the call site – a raw DAIF value on the normal path, or a PMR value when priority masking is active. The API enforces unsigned long with typecheck():

1
2
3
4
5

#define raw_spin_lock_irqsave(lock, flags)          \
do {                                                \
    typecheck(unsigned long, flags);                \
    flags = _raw_spin_lock_irqsave(lock);           \
} while (0)

See include/linux/spinlock.h. Never store the flags in an int, a bool, or a bitfield.

6.4 The _irq, _irqsave, and _irq_disable Variants

On a non-PREEMPT_RT kernel there are three interrupt-disabling lock families, and they solve different nesting problems:

VariantSaves state?Nested-safe?Use when
spin_lock_irq() / spin_unlock_irq()NoNoIRQs are known enabled; no nesting
spin_lock_irqsave() / spin_unlock_irqrestore()Yes, in flagsOnly if unlocks are properly nestedGeneral process-context use
spin_lock_irq_disable() / spin_unlock_irq_enable()Per-CPU counted stateYes, outside NMIMultiple nested IRQ-disabling locks

The first form is the oldest and the easiest to misuse. The second form is the usual safe choice. The third form was added because _irqsave is not actually safe for arbitrary hand-over-hand locking:

1
2
3
4
5
6
7

/* BUG if l2 is IRQ-protected: interrupts are enabled while l2 is
 * still held, and the final restore leaves them disabled.
 */
spin_lock_irqsave(&l1, flags1);
spin_lock_irqsave(&l2, flags2);
spin_unlock_irqrestore(&l1, flags1);  /* restores IRQs (flags1) */
spin_unlock_irqrestore(&l2, flags2);  /* restores IRQs off (flags2) */

Each _irqrestore restores the state saved by its matching acquire. If the outer lock is released first, the inner lock remains held with interrupts enabled; the final restore then restores the inner (already-disabled) state and leaves interrupts disabled after all locks are gone. The counted API avoids both problems by keeping a per-CPU nesting depth: local_interrupt_disable() saves the interrupt state only on the outermost entry, and local_interrupt_enable() restores it only on the outermost exit. See include/linux/interrupt_rc.h and the introducing commit e901c1510e24 (“irq,spin_lock: Add counted interrupt disabling/enabling”).

1
2
3
4

spin_lock_irq_disable(&l1);     depth 0 -> 1, save IRQ state
spin_lock_irq_disable(&l2);     depth 1 -> 2
spin_unlock_irq_enable(&l1);    depth 2 -> 1, interrupts stay off
spin_unlock_irq_enable(&l2);    depth 1 -> 0, restore IRQ state

The trade-off is that _irq_disable() has no flags argument and relies on per-CPU state. It is the better choice for nested or hand-over-hand interrupt-disabling lock sequences; _irqsave() remains common and is correct when each critical section is properly nested and unlocked in reverse order.

6.5 spin_lock_bh and Bottom Halves

Bottom halves (softirqs and tasklets) run in interrupt context but are less time-critical than hard IRQ handlers. If a spinlock protects data shared between process context and a bottom half only – never with a hard IRQ handler – it is unnecessary to disable hardware interrupts. It is enough to prevent bottom-half execution on the local CPU.

On a non-PREEMPT_RT kernel, spin_lock_bh() calls __local_bh_disable_ip() with SOFTIRQ_LOCK_OFFSET, which disables both preemption and softirq processing by advancing the preemption count:

1
2
3
4
5
6

static inline void __raw_spin_lock_bh(raw_spinlock_t *lock)
{
    __local_bh_disable_ip(_RET_IP_, SOFTIRQ_LOCK_OFFSET);
    spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
    LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
}

See include/linux/spinlock_api_smp.h. SOFTIRQ_LOCK_OFFSET is defined as SOFTIRQ_DISABLE_OFFSET + PREEMPT_LOCK_OFFSET in include/linux/preempt.h, so one increment accounts for both.

Hardware interrupts still fire normally. Only softirq processing on the local CPU is deferred. This keeps interrupt latency low while still preventing a softirq or tasklet from preempting the process-context critical section on the same CPU.

On unlock, the order is:

1
2
3
4

spin_unlock_bh()
  +-- spin_release()
  +-- do_raw_spin_unlock()       (lock is free)
  +-- __local_bh_enable_ip()     (decrement BH count; may run pending softirqs)

Pending softirqs may run during spin_unlock_bh(), after the lock is released. That is intentional: the data is no longer protected, and the BH handler must take its own lock if it shares the data.

spin_lock_bh() is a process-context tool. Calling it from inside a BH handler is unnecessary and usually a sign that the lock variant was copied from process context: BH processing is already disabled on that CPU, and spin_lock_bh() does not protect against hard IRQs. Use plain spin_lock() from a BH handler, or spin_lock_irqsave() if a hard IRQ handler also accesses the data.

6.6 Execution Context Hierarchy

Before choosing a lock variant, understand who can preempt whom:

1
2
3
4
5
6
7

Priority (high to low)              Can be preempted by
----------------------              -------------------
[5] NMI                             Nothing (not maskable by DAIF/PMR)
[4] Hard IRQ handler                NMI (and higher-priority NMI-like exceptions)
[3] Softirq / tasklet               Hard IRQ
[2] Kernel process context          Hard IRQ; softirq if BH is enabled
[1] User space                      Everything above

Two scheduling rules drive most locking decisions:

  • A softirq does not preempt another softirq on the same CPU. Softirqs of the same type can, however, run simultaneously on different CPUs.
  • A specific tasklet instance is serialized globally. The tasklet core uses TASKLET_STATE_RUN on that struct tasklet_struct, so two CPUs cannot run the same tasklet instance at the same time. Different tasklet instances – even ones that use the same callback function – can run concurrently on different CPUs.

The second rule is often stated as “same-type tasklets are serialized,” but that wording is imprecise. Serialization is per tasklet instance, not per callback or per subsystem.

6.7 Tasklet and Softirq Serialization

The kernel’s tasklet execution path takes a per-CPU list, then calls tasklet_trylock(t) before invoking the callback:

1
2
3
4
5
6
7
8
9
10

if (tasklet_trylock(t)) {
    if (!atomic_read(&t->count)) {
        if (tasklet_clear_sched(t)) {
            t->callback(t);
        }
        tasklet_unlock(t);
        continue;
    }
    tasklet_unlock(t);
}

See kernel/softirq.c. tasklet_trylock() is an atomic test-and-set of TASKLET_STATE_RUN on the tasklet’s own state word (include/linux/interrupt.h). If the same tasklet is already running on another CPU, the second CPU defers it.

What this means for locking:

SituationSerializationLock needed?
One tasklet instance, data used only by that instanceTasklet core serializes the instanceNo lock for that data
Two tasklet instances, even with the same callbackThey can run on different CPUsspin_lock() in each
Same tasklet instance plus process contextProcess uses spin_lock_bh(); tasklet uses spin_lock()Yes, both sides
Same tasklet instance plus hard IRQHard IRQ can preempt the taskletTasklet spin_lock_irqsave(); IRQ spin_lock()
Softirq plus softirq, or different tasklet instancesDifferent CPUs may run concurrentlyspin_lock() in each

“Data used only by that instance” is the key qualifier. If a process, another tasklet, or a hard IRQ can touch the data, the tasklet must take a lock even though the tasklet core serializes the instance against itself.

6.8 Comprehensive Locking Matrix

The table below covers non-PREEMPT_RT kernels and data that is genuinely shared between two contexts. If the data is private to one context, no lock is needed for that context; the matrix would not apply.

Context AContext BLock from ALock from BReason
ProcessProcessspin_lockspin_lockCross-CPU mutual exclusion
ProcessOne tasklet instancespin_lock_bhspin_lockBH disable on A’s CPU; lock for other CPUs
ProcessMultiple tasklets or any softirqspin_lock_bhspin_lockMultiple instances can run on different CPUs
ProcessHard IRQspin_lock_irqsave (or _irq_disable)spin_lockIRQ can interrupt A on the same CPU
BH (tasklet/softirq)Hard IRQspin_lock_irqsavespin_lockIRQ can interrupt the BH on the same CPU
BHBH (different instances/types)spin_lockspin_lockDifferent CPUs may run concurrently
Same tasklet instanceItselfNo lockNo lockTasklet core serializes the instance
Hard IRQHard IRQ (different CPUs)spin_lockspin_lockLocal IRQs already off; lock for other CPUs
NMIAny contextNot NMI-safeNot NMI-safe_irqsave cannot mask NMI; see Section 6.11

A few important details:

  • spin_lock_bh() from process context protects against softirq reentry on the same CPU. The actual lock word still provides cross-CPU exclusion.
  • A BH handler normally uses plain spin_lock(), not spin_lock_bh(), because BH processing is already disabled in that context.
  • A BH handler that shares data with a hard IRQ handler must use spin_lock_irqsave(), because the hard IRQ can interrupt the BH handler on the same CPU.
  • A hard IRQ handler normally uses plain spin_lock(), because local IRQs are already disabled. It normally should not use spin_lock_irqsave() unless there is a specific reason; the extra save/restore is unnecessary.
  • NMI is the exception that breaks all of these rules; _irqsave does not mask NMI.

6.9 Variant Selection Decision Tree

On a non-PREEMPT_RT kernel, the decision tree is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

Does a hard IRQ handler access the data?
|
+-- YES
|     |
|     +-- Are you in a hard IRQ handler?
|     |     -> spin_lock()
|     |
|     +-- Are you in a BH handler?
|     |     -> spin_lock_irqsave()
|     |
|     +-- Are you in process context?
|           |
|           +-- IRQ state known enabled?
|           |     -> spin_lock_irq()
|           |
|           +-- Need nested or hand-over-hand IRQ disabling?
|           |     -> spin_lock_irq_disable()
|           |
|           +-- Otherwise
|                 -> spin_lock_irqsave(&lock, flags)
|
+-- NO
      |
      +-- Does a softirq or tasklet access the data?
      |     |
      |     +-- Are you in a BH handler?
      |     |     |
      |     |     +-- Is this the same tasklet instance and the
      |     |     |   only accessor? -> no lock needed
      |     |     |
      |     |     +-- Otherwise -> spin_lock()
      |     |
      |     +-- Are you in process context?
      |           -> spin_lock_bh()
      |
      +-- Only process contexts access the data
            -> spin_lock()

This tree assumes the data is truly shared. If the data is per-CPU or private to one thread, use the per-CPU or private-data mechanisms instead of a global lock.

6.10 Summary Tables

The following tables apply to non-PREEMPT_RT kernels; Section 14 describes how the suffixes change on RT.

From process context:

VariantDisablesUse when the data is also accessed by
spin_lockPreemptionOther process contexts only
spin_lock_bhPreemption + softirqsSoftirqs or tasklets, but not hard IRQs
spin_lock_irqPreemption + maskable IRQsHard IRQs; IRQ state known enabled
spin_lock_irqsavePreemption + IRQs, saves stateHard IRQs; IRQ state unknown or nested
spin_lock_irq_disablePreemption + counted IRQ disableHard IRQs; nested/hand-over-hand locking

From BH (softirq/tasklet) context:

VariantUse when the data is also accessed by
No lockOnly the same tasklet instance, which the tasklet core serializes
spin_lockDifferent BH contexts (tasklets or softirqs) on other CPUs
spin_lock_irqsaveA hard IRQ handler on the same CPU
spin_lock_irq_disableA hard IRQ handler, with nested IRQ-disabling sections

From hard IRQ context:

VariantUse when the data is also accessed by
spin_lockAny other context; local IRQs are already disabled, and the lock handles other CPUs
raw_spin_trylockNMI context, with a lockless fallback if the trylock fails

6.11 NMI Context

An NMI cannot be masked by local_irq_save(), by msr daifset, #3, or by lowering the GIC priority mask. Therefore spin_lock_irqsave() does not protect a lock against an NMI that interrupts the holder on the same CPU.

1
2
3
4
5
6
7

CPU 0 holds lock L
      |
      +-- NMI arrives (not maskable)
            |
            +-- NMI handler tries spin_lock(&L)
                  -> spins forever
                     (the holder is the interrupted context)

An ordinary spinlock shared with an NMI handler is therefore not NMI-safe. NMI handlers that need to coordinate with the rest of the kernel generally use one of these patterns:

  • A lockless algorithm or per-CPU data that the NMI can read without blocking.
  • A raw_spin_trylock() attempt, with a fallback that drops, defers, or records the event if the lock is busy.
  • A lock whose holders are guaranteed never to be interruptible by the NMI in question, which is a very strong and fragile requirement.

When using raw_spin_trylock() from NMI context, the fallback must not sleep or block. The NMI handler cannot wait for the holder to finish, because the holder may be the context the NMI interrupted.

The DMA and tracing subsystems contain carefully designed NMI-safe rings and per-CPU buffers for exactly this reason. Do not assume that _irqsave makes a lock NMI-safe.

7. Sleeping and Uniprocessor Semantics

7.1 Sleeping Inside a Spinlock

Calling a sleeping primitive while holding a spinlock is always a bug. This includes msleep(), schedule(), wait_event(), mutex_lock(), down(), kmalloc(GFP_KERNEL), copy_to_user() / copy_from_user() when a page fault is possible, blocking I/O, synchronize_rcu(), and flush_work().

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

CPU 0
  +-- spin_lock(&lock)      <- preempt_disable(); locked = 1
  +-- msleep(100)
  |     +-- schedule()
  |           |
  |     preemption is disabled
  |     -> schedule_debug() reports "scheduling while atomic"
  |     -> preempt_count is forced to PREEMPT_DISABLED
  |     -> the task may still block while holding the lock
  |
  |                     CPU 1
  |                       +-- spin_lock(&lock) -> spins
  |                       +-- ... spins while CPU 0 is blocked ...
  v
DEADLOCK / KERNEL BUG

The scheduler does not silently “refuse” to switch. schedule_debug() detects that the caller is in an atomic section, prints a BUG: scheduling while atomic report, calls check_panic_on_warn(), taints the kernel, and sets the preemption count to PREEMPT_DISABLED; then the scheduler continues in an invalid state (kernel/sched/core.c). If panic_on_warn is enabled, the report becomes a panic. If it is not, the task may block while other CPUs spin on the lock, which is still a hang or a severe correctness failure.

CONFIG_DEBUG_ATOMIC_SLEEP instruments sleeping primitives with might_sleep(), which warns before the actual schedule:

1
2
3
4
5
6
7
8

void __might_sleep(const char *file, int line)
{
    unsigned int state = get_current_state();

    WARN_ONCE(state != TASK_RUNNING && current->task_state_change,
              "do not call blocking ops when !TASK_RUNNING; ...");
    __might_resched(file, line, 0);
}

See kernel/sched/core.c. The warning is not a guarantee that every sleeping path is instrumented, and it is not a substitute for understanding the context rules.

A module that sleeps in the critical section:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

#include <linux/delay.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/slab.h>
#include <linux/spinlock.h>

MODULE_LICENSE("GPL");

struct my_state {
    spinlock_t lock;
    unsigned int value;
};

static struct my_state *state;

static int __init test_hello_init(void)
{
    state = kmalloc(sizeof(*state), GFP_KERNEL);
    if (!state)
        return -ENOMEM;

    spin_lock_init(&state->lock);
    state->value = 0;

    spin_lock(&state->lock);
    pr_info("Starting critical region\n");
    msleep(1000);             /* BUG: sleeps with preemption disabled. */
    pr_info("Ending critical region\n");
    spin_unlock(&state->lock);

    kfree(state);
    state = NULL;
    return 0;
}

static void __exit test_hello_exit(void) {}

module_init(test_hello_init);
module_exit(test_hello_exit);

The pr_info() calls are not the problem; the msleep() is. Logging under a spinlock is usually undesirable for latency reasons, but it does not sleep in the normal console path. Sleeping does.

On PREEMPT_RT, spinlock_t is a sleeping RT mutex, so the “preemption count” argument no longer applies. The rule still applies: a spinlock_t critical section must not call a sleeping function. Sleeping while holding the lock can deadlock or destroy the latency guarantees the lock is supposed to provide.

7.2 Debugging Atomic Sleep

The relevant debug options are:

  • CONFIG_DEBUG_ATOMIC_SLEEP – makes might_sleep() and scheduler checks noisy in atomic context. It selects PREEMPT_COUNT.
  • CONFIG_DEBUG_PREEMPT – reports the instruction that disabled preemption, which helps locate the lock that was held.
  • CONFIG_PROVE_LOCKING – lockdep also tracks context and can report “inconsistent lock state” for locks used from the wrong context.
  • panic_on_warn – turns warnings into panics, which is useful in test farms but hostile on a developer workstation.

A typical report looks like:

1
2
3
4
5
6
7
8
9

BUG: sleeping function called from invalid context at kernel/locking/mutex.c:...
in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 1234, name: test
preempt_count: 1, expected: 0
...
Call trace:
  __might_resched
  ...
  msleep
  test_hello_init

The preempt_count: 1 is the smoking gun: the spinlock’s preempt_disable() is still active.

7.3 Uniprocessor and CONFIG_PREEMPT

On a uniprocessor (UP) machine there is only one CPU, so two CPUs cannot race to acquire the same lock. The implementation changes with the preemption model:

1
2
3
4
5
6
7
8
9
10
11
12

Uniprocessor (UP) kernel
         |
    +----+----------------------------+
    |                                 |
CONFIG_PREEMPT=y                CONFIG_PREEMPT=n
    |                                 |
preempt_disable() still runs     spin_lock() is a compiler
arch lock word op is elided      barrier; no hardware atomic
    |                            operation at all
still prevents another task
from preempting the holder
mid-critical-section

In the non-debug UP build, arch_spin_lock() and arch_spin_unlock() are compiler barriers only:

1
2
3
4
5

#define arch_spin_is_locked(lock) ((void)(lock), 0)

#define arch_spin_lock(lock)   do { barrier(); (void)(lock); } while (0)
#define arch_spin_unlock(lock) do { barrier(); (void)(lock); } while (0)
#define arch_spin_trylock(lock) ({ barrier(); (void)(lock); 1; })

See include/linux/spinlock_up.h. The barriers are still required: the compiler must not move memory accesses across the lock boundary, particularly accesses that may fault.

With CONFIG_PREEMPT=y and CONFIG_PREEMPT_COUNT=y, spin_lock() still calls preempt_disable() and spin_unlock() still calls preempt_enable(). The lock word operation is elided, but a second task cannot preempt the holder in the middle of the critical section. With CONFIG_PREEMPT=n, preempt_disable() may compile to a barrier as well; the lock is then a pure compiler barrier.

UP debug builds are different. CONFIG_DEBUG_SPINLOCK uses an inverted slock value to catch initialization bugs and provides real recursion/owner checks:

1
2
3
4
5
6
7
8
9
10
11
12
13

#define arch_spin_is_locked(x) ((x)->slock == 0)

static inline void arch_spin_lock(arch_spinlock_t *lock)
{
    lock->slock = 0;
    barrier();
}

static inline void arch_spin_unlock(arch_spinlock_t *lock)
{
    barrier();
    lock->slock = 1;
}

See include/linux/spinlock_up.h. Even on UP, the API rules still matter: the code may later be built for SMP, and _irq/_bh variants still control interrupt and softirq state.

8. AArch64 Implementation

8.1 Evolution: Ticket Locks, MCS, and qspinlock

AArch64’s spinlock implementation evolved through three designs:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

+-----------------+     cache-line stampede     +------------------+
|  Ticket lock    | --------------------------> |  MCS lock        |
|  (FIFO, shared  |    all CPUs poll the same   |  (FIFO, each CPU |
|   poll point)   |    memory word on unlock    |   polls its own  |
+-----------------+                             |   node)          |
                                                +------------------+
                                                         |
                                               lock structure grows
                                               (pointer to queue tail)
                                                         |
                                                         v
                                                +------------------+
                                                |  qspinlock       |
                                                |  (MCS scalability|
                                                |   in a 32-bit    |
                                                |   lock word)     |
                                                +------------------+

AArch64 used a ticket lock until commit c11090474d70 (“arm64: locking: Replace ticket lock implementation with qspinlock”). The qspinlock is the generic Linux implementation, selected on AArch64 through ARCH_USE_QUEUED_SPINLOCKS (arch/arm64/Kconfig, kernel/Kconfig.locks). The asm/qspinlock.h header is generated from the generic version via arch/arm64/include/asm/Kbuild.

The qspinlock implementation is credited to Waiman Long and Peter Zijlstra in kernel/locking/qspinlock.c. It is based on the MCS lock but keeps the lock word at 32 bits by encoding the queue tail rather than storing a full pointer.

8.2 Ticket Spinlock (Historical)

The original AArch64 spinlock was a ticket lock – a FIFO mechanism inspired by numbered-ticket service counters. The lock held two counters:

1
2
3
4
5
6

  +----------------+----------------+
  |   next (tail)  |  owner (head)  |
  +----------------+----------------+
         ^                  ^
  CPU reads next       CPU spins until
  and increments it     its ticket == owner

To acquire, a CPU atomically read-and-incremented next to claim a ticket, then spun comparing its ticket against owner. To release, the holder incremented owner by one. FIFO ordering was guaranteed: CPUs acquired the lock in the order they claimed their tickets.

The fundamental flaw was the shared polling point. Every waiting CPU spun on the same owner word. When the lock was released, the write to owner invalidated that cache line on every other CPU simultaneously. They all fetched the new value at once – a cache-line stampede whose cost grew with the number of CPUs. The lock was fair, but it did not scale.

8.3 MCS Lock

The MCS lock (Mellor-Crummey and Scott, 1991) removes the stampede by giving every waiting CPU its own node to spin on. The generic node is defined in include/asm-generic/mcs_spinlock.h:

1
2
3
4
5

struct mcs_spinlock {
    struct mcs_spinlock *next;
    int locked;   /* 1 if lock acquired */
    int count;    /* nesting count */
};

The MCS API in kernel/locking/mcs_spinlock.h implements the wait and wake operations with acquire/release semantics:

1
2
3
4
5

#define arch_mcs_spin_lock_contended(l) \
    smp_cond_load_acquire(l, VAL)

#define arch_mcs_spin_unlock_contended(l) \
    smp_store_release((l), 1)

A queue with three waiters looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

MCS queue (3 waiters after the holder):

 holder node       first waiter
      |                 |
      v                 v
 +-----------+     +-----------+     +-----------+     +-----------+
 |  holder   |---->|  CPU 1    |---->|  CPU 2    |---->|  CPU 3    |
 |  node     |     |  node     |     |  node     |     |  node     |
 | locked=-- |     | locked=0  |     | locked=0  |     | locked=0  |
 +-----------+     +-----------+     +-----------+     +-----------+
                                                       ^
                                                lock->tail
                                          (last waiter, not the head)

  CPU 1 spins on its own node's locked field
  (local cache line; no cross-CPU invalidation)

On handoff, the holder writes only to CPU 1's node:
  CPU 1 node.locked = 1  -> one targeted cache-line write, one wakeup

The drawback is size. The lock variable must hold a pointer to the queue tail, which is wider than 32 bits. Embedding an 8-byte pointer in every spinlock in the kernel would significantly increase memory consumption. The qspinlock solves this by encoding the tail in the lock word and keeping the actual nodes in per-CPU storage.

8.4 qspinlock Design

The qspinlock preserves the compact 32-bit footprint of the ticket lock while delivering MCS-like scalability. It encodes the MCS queue tail into the upper bits of the lock word and stores the actual MCS nodes in statically allocated per-CPU arrays:

1

static DEFINE_PER_CPU_ALIGNED(struct qnode, qnodes[_Q_MAX_NODES]);

See kernel/locking/qspinlock.c. _Q_MAX_NODES is 4, enough for four nested contexts (task, softirq, hardirq, NMI); deeper nesting is assumed to be extremely unlikely (kernel/locking/qspinlock.h). If a CPU nests deeper than four, the slow path falls back to spinning directly on the lock word without an MCS node.

The tail encoding is deliberately chosen so that a zero tail means “no queue”:

1
2
3
4
5
6
7
8
9

static inline __pure u32 encode_tail(int cpu, int idx)
{
    u32 tail;

    tail  = (cpu + 1) << _Q_TAIL_CPU_OFFSET;
    tail |= idx << _Q_TAIL_IDX_OFFSET; /* assume < 4 */

    return tail;
}

See kernel/locking/qspinlock.h. decode_tail() reverses the encoding and returns a pointer to the previous CPU’s struct mcs_spinlock.

A struct qnode contains the MCS node and, optionally, paravirt padding. On a native 64-bit build it is 16 bytes, so four nodes fit in a 64-byte cache line; with CONFIG_PARAVIRT_SPINLOCKS, the qnode is padded to 32 bytes and only two fit. The per-CPU array is cache-line aligned because each CPU’s nodes are written by other CPUs during handoff.

8.5 Fast Path

The uncontended fast path is a single compare-and-swap on the whole lock word:

1
2
3
4
5
6
7
8
9
10

CPU arrives, lock word == 0
         |
         v
atomic_try_cmpxchg_acquire(&lock->val, 0 -> _Q_LOCKED_VAL)
         |
    +----+----+
  SUCCESS   FAILURE (another CPU beat us)
    |             |
  locked=1   queued_spin_lock_slowpath(lock, observed_val)
  return

1
2
3
4
5
6
7
8
9

static __always_inline void queued_spin_lock(struct qspinlock *lock)
{
    int val = 0;

    if (likely(atomic_try_cmpxchg_acquire(&lock->val, &val, _Q_LOCKED_VAL)))
        return;

    queued_spin_lock_slowpath(lock, val);
}

See include/asm-generic/qspinlock.h. On AArch64 with LSE, atomic_try_cmpxchg_acquire() is a single CASA instruction. Without LSE it is an LDAXR/STXR loop. The observed val is passed to the slow path so it can decide whether to use the pending slot or queue immediately.

The qspinlock depends on its atomic operations having at least RCsc ordering (or no weaker than RCtso on Power). The header comments call this out explicitly, because qspinlock combines atomics of different widths and cannot tolerate a weaker implementation (include/asm-generic/qspinlock.h).

8.6 Pending Bit: Second Waiter

When exactly one CPU holds the lock and the queue is empty, a second CPU uses the pending slot instead of allocating an MCS node. This handles the common two-CPU contention case cheaply: the second CPU sets the pending bit and waits on the byte-sized locked field.

Source: kernel/locking/qspinlock.c.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

/*
 * If the lock word already has a pending bit or a queue tail, this
 * CPU must queue; only a bare locked word is eligible for the
 * pending slot.
 */
if (val & ~_Q_LOCKED_MASK)
    goto queue;

/*
 * trylock || pending
 *
 * 0,0,* -> 0,1,* -> 0,0,1 pending, trylock
 */
val = queued_fetch_set_pending_acquire(lock);

/*
 * If we observe contention, there is a concurrent locker.
 */
if (unlikely(val & ~_Q_LOCKED_MASK)) {
    if (!(val & _Q_PENDING_MASK))
        clear_pending(lock);
    goto queue;
}

/*
 * We're pending, wait for the owner to go away.
 *
 * 0,1,1 -> *,1,0
 */
if (val & _Q_LOCKED_MASK)
    smp_cond_load_acquire(&lock->locked, !VAL);

/*
 * take ownership and clear the pending bit.
 *
 * 0,1,0 -> 0,0,1
 */
clear_pending_set_locked(lock);
return;

queued_fetch_set_pending_acquire() is an atomic fetch-or that sets the pending bit and returns the old word (kernel/locking/qspinlock.h). If the old word already had a queue tail or a pending bit, this CPU abandons the pending slot and goes to the queue.

The successful pending sequence is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

State notation: T = tail, P = pending, L = locked

Initial:
  T=0, P=0, L=0          UNLOCKED

CPU A acquires:
  T=0, P=0, L=1          A holds

CPU B sets pending:
  T=0, P=1, L=1          B waits on the locked byte

CPU A unlocks:
  T=0, P=1, L=0          B's conditional load wakes

CPU B claims:
  T=0, P=0, L=1          B holds; no MCS node was allocated

CPU B unlocks:
  T=0, P=0, L=0          UNLOCKED

The pending waiter waits on lock->locked, not on the whole word. The smp_cond_load_acquire() is an acquire load so that the critical section that follows is ordered after the store-release performed by the previous holder. clear_pending_set_locked() writes the 16-bit locked_pending field to _Q_LOCKED_VAL, which both clears the pending bit and sets the locked bit.

If a third CPU arrives while B is pending, its val & ~_Q_LOCKED_MASK test sees the pending bit set and it enters the MCS queue. It does not disturb B’s pending slot. The third CPU becomes the queue head if the tail was zero, and waits directly on the lock word.

8.7 Slow Path: Third Waiter and Beyond

Once the pending slot is occupied, additional CPUs queue on MCS nodes. The full slow path is in kernel/locking/qspinlock.c. The important steps are:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

CPU C enters slow path:
  |
  +-- node = this_cpu_ptr(&qnodes[0].mcs)
  +-- idx = node->count++
  +-- tail = encode_tail(cpu, idx)
  |
  +-- if idx >= _Q_MAX_NODES:
  |     fall back to direct spinning; no MCS node
  |
  +-- node = grab_mcs_node(node, idx)
  +-- node->locked = 0
  +-- node->next = NULL
  |
  +-- one more trylock attempt
  |     (the holder may have released while the cache line was fetched)
  |
  +-- smp_wmb()                   (publish node init before linking)
  +-- old = xchg_tail(lock, tail) (publish new tail, fetch previous)
  |
  +-- if old tail != 0:
  |     prev = decode_tail(old)
  |     WRITE_ONCE(prev->next, node)
  |     arch_mcs_spin_lock_contended(&node->locked)
  |     next = READ_ONCE(node->next)
  |
  +-- else:
  |     this CPU is the queue head
  |
  +-- wait until both locked and pending are clear:
        atomic_cond_read_acquire(&lock->val,
                                 !(VAL & (LOCKED | PENDING)))
  |
  +-- lock acquisition:
        if ((val & TAIL_MASK) == tail)
            if CAS(val -> LOCKED): no one behind; done
        set_locked(lock)           (tail changed or pending appeared)
        ensure next is known
        wake next

Several details are worth emphasizing:

  • The per-CPU node is reused. node->count is a nesting counter. A CPU can enter the slow path recursively (for example, from an interrupt that interrupts a slow-path waiter), so the kernel has four nodes per CPU. Beyond four, it falls back to direct spinning.
  • The tail exchange is the queue insertion. xchg_tail() atomically installs the new tail and returns the old one. If the old tail was zero, there was no previous waiter and this CPU is the head. Otherwise, the old tail identifies the previous waiter’s node, and the new CPU links itself with WRITE_ONCE(prev->next, node).
  • The smp_wmb() orders node initialization before publication. Without it, another CPU could observe the new tail and follow prev->next to a node whose locked or next fields were not yet initialized.
  • Waiters behind the head spin on their own node. arch_mcs_spin_lock_contended(&node->locked) calls smp_cond_load_acquire(), which on AArch64 uses WFE. This is the MCS scalability property: among queued waiters, only the head polls the shared lock word; the separate pending waiter polls only the locked byte.
  • The head waits on the lock word, not on its node. It must wait until both the locked and pending bits are clear, because a pending waiter (B) is entitled to claim before the queue head (C).
  • The claim path distinguishes “alone” from “someone behind.” If the tail still points to this CPU, a CAS from the observed word to LOCKED succeeds only if no pending bit appeared and the tail did not change; that clears the tail and takes the lock. If the CAS fails, either another CPU queued behind or a pending waiter appeared. The holder then calls set_locked(), preserves the tail, and wakes the next node (waiting for node->next if necessary).

The wake operation is:

1

arch_mcs_spin_unlock_contended(&next->locked);

which expands to smp_store_release(&next->locked, 1) and compiles to a 32-bit STLR on AArch64. This is the targeted handoff that replaces the ticket lock’s cache-line stampede.

8.8 Full qspinlock State Machine

The following state machine shows the common three-CPU sequence: A holds, B uses the pending slot, C queues. It deliberately omits the fourth-CPU case; if a fourth CPU D queues behind C, the tail becomes D and C wakes D when C acquires.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

State notation: T = tail, P = pending, L = locked

[T=0, P=0, L=0]  UNLOCKED
      |
      |  CPU A: CASA(0 -> LOCKED)
      v
[T=0, P=0, L=1]  A holds
      |
      |  CPU B: fetch_or(PENDING); old word has no queue/pending
      v
[T=0, P=1, L=1]  B in pending slot; B waits on locked byte
      |
      |  CPU C: sees pending/queue, enters slow path
      |  old tail = 0, xchg_tail(C) -> C is queue head
      v
[T=C, P=1, L=1]  C queued as head; C waits on lock word
      |
      |  A unlocks: STLRB locked = 0
      v
[T=C, P=1, L=0]  B's pending wait ends
      |
      |  B: clear_pending_set_locked()
      v
[T=C, P=0, L=1]  B holds; C still waiting because L=1
      |
      |  B unlocks: STLRB locked = 0
      v
[T=C, P=0, L=0]  C sees locked=0 and pending=0
      |
      |  C: tail == C, CAS(word -> LOCKED), clears tail
      v
[T=0, P=0, L=1]  C holds
      |
      |  C unlocks: STLRB locked = 0
      v
[T=0, P=0, L=0]  UNLOCKED

Two points are easy to get wrong:

  • B claims before C even though C is in the queue. B occupies the pending slot, which is the first waiter position. C is the queue head and waits for B to release. This is why the state after A unlocks has P=1 and L=0, not “C immediately takes the lock.”
  • C does not need a targeted node wake from A’s unlock in this three-CPU case. C is the queue head with no predecessor, so it polls the lock word directly; A’s unlock and B’s pending-claim/unlock writes are the events that wake it, and it proceeds only after B releases.

If there are four CPUs, D queues behind C and the tail becomes D. When C acquires, it sees that the tail is not its own, sets its locked bit, and wakes D’s node with arch_mcs_spin_unlock_contended(). C’s later unlock is again just a STLRB; D is already awake and polling the lock word.

8.9 WFE and the Conditional Wait Loop

The MCS wait and the pending-slot wait both use smp_cond_load_acquire(). On AArch64 this is defined in arch/arm64/include/asm/barrier.h:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#define smp_cond_load_relaxed(ptr, cond_expr)               \
({                                                          \
    typeof(ptr) __PTR = (ptr);                              \
    __unqual_scalar_typeof(*ptr) VAL;                       \
    for (;;) {                                              \
        VAL = READ_ONCE(*__PTR);                            \
        if (cond_expr)                                      \
            break;                                          \
        __cmpwait_relaxed(__PTR, VAL);                      \
    }                                                       \
    (typeof(*ptr))VAL;                                      \
})

#define smp_cond_load_acquire(ptr, cond_expr)               \
({                                                          \
    typeof(ptr) __PTR = (ptr);                              \
    __unqual_scalar_typeof(*ptr) VAL;                       \
    for (;;) {                                              \
        VAL = smp_load_acquire(__PTR);                      \
        if (cond_expr)                                      \
            break;                                          \
        __cmpwait_relaxed(__PTR, VAL);                      \
    }                                                       \
    (typeof(*ptr))VAL;                                      \
})

__cmpwait_relaxed() is a WFE-based wait in arch/arm64/include/asm/cmpxchg.h:

1
2
3
4
5
6
7
8

__cmpwait_case_32(ptr, val):
    sevl                 ; send an event locally
    wfe                  ; consume it and continue immediately
    ldxr  tmp, [ptr]     ; load the watched word
    eor   tmp, tmp, val  ; did it change?
    cbnz  tmp, 1f        ; yes -> return
    wfe                  ; no -> sleep until a store event
1:

The loop structure matters: smp_cond_load_acquire() loads with acquire semantics, checks the condition, and only then calls __cmpwait_relaxed() with the last observed value. The first WFE consumes the locally sent event so the initial load happens immediately; the second WFE performs the actual wait. The WFE wait is an optimization, and the outer loop rechecks the condition after every wake, so spurious wakeups are harmless.

There is an important correction to a common piece of folklore: qspinlock does not use arch_timer_evtstrm_available() directly, and it does not use arch/arm64/include/asm/rqspinlock.h. The rqspinlock.h header defines a timed conditional wait for the separate rqspinlock implementation, using smp_cond_load_acquire_timewait(). The qspinlock path uses the ordinary smp_cond_load_acquire() above, which is implemented with sevl/wfe in __cmpwait_relaxed(). The event-stream check in rqspinlock is a different mechanism for a different lock.

The acquire semantics in smp_cond_load_acquire() are required: the wait must pair with the previous holder’s store-release that cleared the locked bit. A relaxed load would allow later critical-section accesses to be observed before the lock acquisition.

8.10 Unlock and Handoff

In the native (non-paravirt) implementation, a qspinlock unlock is deliberately minimal:

1
2
3
4

static __always_inline void queued_spin_release(struct qspinlock *lock)
{
    smp_store_release(&lock->locked, 0);
}

It does not walk the queue, and it does not write to the next waiter’s node. The handoff is cooperative and happens in two places:

  1. Pending waiter. If the next waiter is in the pending slot, the STLRB that clears locked makes that waiter’s smp_cond_load_acquire(&lock->locked, !VAL) succeed. The pending waiter then claims the lock with clear_pending_set_locked().
  2. Queued waiter. If the waiter is queued behind this holder’s MCS node, the new holder performs the node wake after it acquires the lock. In the slow path, when the holder has a next node, it calls arch_mcs_spin_unlock_contended(&next->locked) after set_locked(), which is a targeted STLR to the next CPU’s node; a successful CAS means there is no one behind, so no wake is needed. A queue head with no predecessor does not need a node wake; it observes the locked byte directly. This is also why the pending waiter B does not wake the queued head C: C is already the head and polls the lock word.

This is why the statement “unlock wakes the next CPU” is a useful shorthand but not literally what the unlock store does. The unlock publishes locked = 0; the next holder (or pending waiter) performs the queue bookkeeping. The design keeps the unlock fast path to a single store-release, which is exactly what a hot lock needs.

9. Memory Ordering on AArch64

9.1 Weak Ordering, Acquire, and Release

AArch64 implements a weakly ordered memory model. The processor may reorder independent loads and stores beyond program order unless an explicit barrier or an acquire/release instruction constrains it. The kernel does not rely on ordering by accident; it uses dedicated load-acquire and store-release instructions.

The contract for a spinlock is:

1
2
3
4
5
6
7

spin_lock():                         spin_unlock():

  all later memory accesses            all earlier memory accesses
  stay AFTER the lock acquisition      stay BEFORE the lock release
          |                                      |
          v                                      v
  [ acquire ]                            [ release ]

Together, a successful lock/unlock pair gives the critical section both directions of ordering. Inside the critical section, ordinary loads and stores can be reordered with each other; only the boundary operations are ordered.

9.2 Instructions You Will See

InstructionMeaningTypical use
LDAR, LDARB, LDARHLoad-acquire (word, byte, halfword)Acquire lock state, load a published pointer
STLR, STLRB, STLRHStore-release (word, byte, halfword)Release lock, publish a pointer
LDAXR / STLXRLoad-acquire exclusive / store-release exclusiveLL/SC atomic loops without LSE
CASA / CASAB / CASHCompare-and-swap with acquire semantics (LSE)Fast-path lock acquisition, cmpxchg
DMB ISHData memory barrier, inner shareableFull smp_mb()
DMB ISHLDDMB for loadssmp_rmb()
DMB ISHSTDMB for storessmp_wmb()
WFEWait for eventConditional wait loops
SEVLSend event locallyPrime a WFE loop

DMB orders memory accesses; DSB is stronger and also waits for completion, which is usually not needed for ordinary lock ordering. ISB is an instruction synchronization barrier and does not order data accesses.

9.3 Kernel Wrappers

The kernel exposes the architecture operations through smp_load_acquire(), smp_store_release(), smp_mb(), smp_rmb(), and smp_wmb(). The AArch64 definitions are in arch/arm64/include/asm/barrier.h:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

#define __smp_mb()  dmb(ish)
#define __smp_rmb() dmb(ishld)
#define __smp_wmb() dmb(ishst)

#define __smp_store_release(p, v)                  \
do {                                               \
    switch (sizeof(*p)) {                          \
    case 1:                                        \
        asm volatile("stlrb %w1, %0" ...);         \
        break;                                     \
    case 2:                                        \
        asm volatile("stlrh %w1, %0" ...);         \
        break;                                     \
    case 4:                                        \
        asm volatile("stlr %w1, %0" ...);          \
        break;                                     \
    case 8:                                        \
        asm volatile("stlr %x1, %0" ...);          \
        break;                                     \
    }                                              \
} while (0)

The byte-sized case is what qspinlock uses for locked: STLRB. The 32-bit case is what MCS node handoff uses for node->locked: STLR. On the common < 16K layout, the 16-bit locked_pending write in clear_pending_set_locked() is an ordinary WRITE_ONCE(); the surrounding atomic operations provide the required ordering. The compact layout uses an atomic add instead.

9.4 smp_mb__after_spinlock and RCsc

A plain spin_lock() on AArch64 is an acquire operation, but it is not necessarily a sequentially consistent operation across CPUs. The kernel’s scheduler and wakeup paths need a stronger property: if CPU 0 executes a critical section before CPU 1, a third CPU must not observe CPU 1’s later store without also observing CPU 0’s earlier store. The kernel calls this an RCsc lock.

AArch64 provides the missing barrier through:

1
2

/* See include/linux/spinlock.h */
#define smp_mb__after_spinlock() smp_mb()

See arch/arm64/include/asm/spinlock.h. The generic comment in include/linux/spinlock.h explains the two properties it guarantees. The short version:

  • It prevents the store-buffer litmus where two CPUs each acquire a lock and fail to observe the other’s earlier store.
  • It provides the RCsc property for code that needs the scheduler’s ordering guarantees.

Most driver code does not call smp_mb__after_spinlock() explicitly. The scheduler calls it in __schedule() and try_to_wake_up() where the RCsc property is required. Adding it to an ordinary lock acquisition is a performance cost and should be justified by a specific ordering requirement.

9.5 LSE Atomics

ARMv8.1 Large System Extensions (LSE) add single-instruction atomic operations that complete inside the memory subsystem without holding an exclusive reservation. The kernel selects between LL/SC and LSE at runtime via binary patching:

1
2
3
4
5
6

#define __lse_ll_sc_body(op, ...)                       \
({                                                      \
    alternative_has_cap_likely(ARM64_HAS_LSE_ATOMICS) ? \
        __lse_##op(__VA_ARGS__) :                       \
        __ll_sc_##op(__VA_ARGS__);                      \
})

See arch/arm64/include/asm/lse.h. On an LSE-capable CPU the uncontended qspinlock path is:

1
2

spin_lock()   -> CASA     (one atomic instruction, acquire)
spin_unlock() -> STLRB    (one store-release byte)

On a CPU without LSE:

1
2

spin_lock()   -> LDAXR / STXR loop
spin_unlock() -> STLRB

LSE is common on ARMv8.1+ server and mobile cores, but the kernel still supports pre-LSE hardware (for example Cortex-A53 and Cortex-A72) and virtual CPUs that do not expose the feature.

9.6 Common Misconceptions

  • “spin_lock() is a full memory barrier.” It is an acquire operation. It orders later accesses after the acquisition, not earlier accesses before it. spin_unlock() is the matching release. The pair gives ordering across the critical section, but each half is one-directional.
  • “barrier() is a CPU memory barrier.” barrier() is a compiler barrier only. It prevents compiler reordering but emits no AArch64 instruction.
  • “WRITE_ONCE()/READ_ONCE() provide ordering.” They prevent compiler tricks and torn accesses; they do not provide inter-CPU ordering. Use acquire/release or smp_mb() as needed.
  • “smp_wmb() is a full barrier.” It orders stores, not loads. smp_mb() is the full barrier.
  • “spin_is_locked() can be used to synchronize.” It cannot; it has no ordering guarantee and the value may be stale immediately.
  • “smp_mb__after_spinlock() is part of every lock.” It is a separate operation used where RCsc is required, notably by the scheduler.
  • “dmb ish is always required after an acquire.” A load-acquire already orders later accesses. The full barrier is needed only for the stronger RCsc property.

10. Writing Correct Spinlock Code

10.1 Lock Data, Not Code

A common mistake is to think of a spinlock as protecting a code region – “this function is the critical section.” The correct mental model is the opposite: lock data, not code. Every lock should be clearly associated with the specific shared data it protects, not with a function or a code path.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Wrong mental model:               Correct mental model:
------------------                --------------------
"I need to protect                "struct foo is protected
 this function."                  by foo->lock."

spin_lock(&some_lock);            struct foo {
call_foo();                           spinlock_t lock;
spin_unlock(&some_lock);              int count;
                                      struct list_head list;
                                  };

                                  /* At every access site: */
                                  spin_lock(&obj->lock);
                                  obj->count++;
                                  spin_unlock(&obj->lock);

Locks that wrap arbitrary code regions are hard to audit and prone to races because it is never clear which data they protect. When a lock is named and co-located with a structure – the inode’s i_lock protects i_blocks and i_bytes (and sometimes i_size), for example – reviewers can immediately verify whether every access to that data is correctly guarded.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

struct my_cache {
    spinlock_t lock;          /* protects everything below */
    struct list_head entries;
    unsigned int count;
    unsigned long last_update;
};

static void my_cache_add(struct my_cache *cache, struct entry *e)
{
    spin_lock(&cache->lock);
    list_add_tail(&e->node, &cache->entries);
    cache->count++;
    cache->last_update = jiffies;
    spin_unlock(&cache->lock);
}

The comment on the lock is part of the design. If the lock protects only some fields, say which ones. If a field has a different lock, say so. Ambiguous ownership is where races hide.

10.2 Keep Critical Sections Short

The critical section should contain only the operations that must be atomic with respect to the protected state. Move everything else outside:

  • Allocation and freeing. kmalloc(GFP_KERNEL), kfree of a large object, and vfree can be expensive. Allocate before the lock, or remove the object under the lock and free it after.
  • Copying and formatting. memcpy, snprintf, seq_printf, and logging can be slow. Copy the necessary values under the lock and process them afterward.
  • Wakeups. wake_up() can take runqueue locks and invoke scheduler code. It is common to update state under the spinlock and wake outside it.
  • I/O. MMIO reads and writes can be slow and may have ordering requirements of their own. Keep them out of a hot spinlock unless the hardware state must be updated atomically with the software state.
  • User access. copy_to_user() and copy_from_user() can fault and sleep. They must never run under a spinlock.
  • Loops with data-dependent bounds. A loop that can iterate thousands of times is not a short critical section.

A typical pattern is “remove under the lock, free outside”:

1
2
3
4
5
6
7
8
9
10
11
12

struct entry *e;

spin_lock(&cache->lock);
e = list_first_entry_or_null(&cache->entries, struct entry, node);
if (e) {
    list_del(&e->node);
    cache->count--;
}
spin_unlock(&cache->lock);

if (e)
    kfree(e);

The lock protects the list and count. The free happens after the lock is released, where it can take its time.

10.3 Lock Ordering

When a code path acquires more than one lock, every path must acquire them in the same global order. Otherwise, a circular wait can form:

1
2
3
4
5
6

CPU 0: spin_lock(A) -> spin_lock(B)
CPU 1: spin_lock(B) -> spin_lock(A)

  CPU 0 holds A, waits for B
  CPU 1 holds B, waits for A
  -> deadlock

The problem is invisible in testing until both paths execute concurrently at exactly the wrong moment. The defenses are:

  • Define and document a lock order. For example: parent->lock before child->lock; inode->i_lock before dentry->d_lock.
  • Use lockdep. CONFIG_PROVE_LOCKING tracks acquisition-order edges and reports cycles before they occur. See Section 11.2.
  • Use lockdep_assert_held() to document which locks a function expects.
  • Do not call unknown code while holding a lock. A callback may take locks in an order you do not control. Release the lock, call the callback, then reacquire if needed.
  • Use spin_lock_nested() only for lockdep. The subclass argument tells lockdep that two acquisitions of the same class are intentionally nested; it does not change the hardware lock. Do not use it to paper over a real ordering bug.

10.4 Lifetime and Initialization

A spinlock must remain at a stable address for its entire usable lifetime. The rules are:

  • Initialize before publication. If another CPU can find the object, the lock must already be initialized.
  • Never memcpy a structure containing a lock. The copy creates a second lock word and a duplicate lockdep map, and the original object’s waiters are still attached to the original address.
  • Never free or reuse a lock while waiters exist. A waiter has a pointer to the lock word and to an MCS node that may be linked into the queue.
  • Do not reinitialize a held lock. spin_lock_init() on a live lock resets the lock word and debug state; lockdep may catch it, but production kernels may not.
  • Keep the lock with the data. Embedding the lock in the structure makes lifetime management natural: allocate, initialize, publish; unpublish, quiesce, destroy.
  • Be careful with stack locks. A spinlock_t on the stack is safe only if it is used exclusively by the current context and never escapes. A stack lock shared with an interrupt or another thread is a lifetime bug.

10.5 Ownership and Release Context

The spinlock API contract says that the same context that acquired the lock must release it. Non-debug hardware does not enforce this: spin_unlock() simply clears the lock word. Debug builds and lockdep do enforce it:

  • CONFIG_DEBUG_SPINLOCK checks owner and owner_cpu and reports “wrong owner” or “wrong CPU” on unlock.
  • Lockdep tracks the task and context that acquired the lock and reports an unlock from the wrong context.

Do not use a spinlock for hand-off locking, where one context acquires and another releases. That is what completions, waitqueues, and explicit state machines are for. A hand-off may appear to work on non-debug hardware, but it breaks the ownership model and confuses every debugging tool.

Interrupt and BH contexts are distinct contexts for this purpose. A hard IRQ handler may acquire and release its own lock; it must not release a lock that process context acquired. If data must cross from process to IRQ context, use a flag, a completion, or a properly designed producer/consumer queue, not lock ownership transfer.

10.6 Do Not Use a Lock as a Completion

A spinlock is a mutual-exclusion primitive, not a condition variable. Polling it to wait for another context is almost always wrong:

1
2
3

/* BAD: burns CPU and may never terminate. */
while (spin_is_locked(&ready_lock))
    cpu_relax();

If the waiter can sleep, use a completion or a waitqueue. If the waiter is in atomic context, redesign the state machine so the event is recorded in a flag or a queue and the waiter is invoked when the event occurs. A lock’s state is not an event: it can be held for unrelated reasons, and it can be released before the waiter observes it.

11. Debugging and Observability

11.1 CONFIG_DEBUG_SPINLOCK

CONFIG_DEBUG_SPINLOCK adds the magic, owner_cpu, and owner fields to raw_spinlock_t and checks them on the normal acquire and unlock paths. The relevant checks are in kernel/locking/spinlock_debug.c:

CheckWhenWhat it detects
magic != SPINLOCK_MAGICdo_raw_spin_lock() and unlockCorruption, use-after-free, overwritten lock
owner == currentbefore do_raw_spin_lock()Recursive acquisition by the same task
owner_cpu == current CPUbefore do_raw_spin_lock()Recursive acquisition by the same CPU
!raw_spin_is_locked()unlockUnlock of an already unlocked lock
owner != currentunlockUnlock by the wrong task
owner_cpu != current CPUunlockUnlock on the wrong CPU

do_raw_spin_trylock() is different: it records the owner on success but does not run the pre-acquire checks. A failed trylock returns zero without a recursion report.

The layout is:

1
2
3
4
5
6
7

raw_spinlock_t (with CONFIG_DEBUG_SPINLOCK)
+------------------------------------------+
|  arch_spinlock_t  raw_lock               |  <- hardware lock word
|  unsigned int     magic      0xdead4ead  |  <- corruption detector
|  unsigned int     owner_cpu              |  <- which CPU holds it
|  void            *owner                  |  <- which task holds it
+------------------------------------------+

When a check fails, spin_bug() calls debug_locks_off() and prints a BUG: spinlock ... report with the lock address, magic, owner, owner CPU, and a stack dump. It does not call BUG() or panic by itself. The acquisition or unlock then continues, which means a recursive lock still deadlocks; the report is there to tell you why before the hard lockup detector fires (kernel/locking/spinlock_debug.c).

CONFIG_DEBUG_SPINLOCK also changes the UP implementation. The UP debug lock uses an inverted slock value: 1 means unlocked and 0 means locked, so a zero-initialized lock is immediately detected as corrupt (include/linux/spinlock_up.h).

The cost is real: extra fields, extra loads and stores, and much worse cache behavior. Use it on development and test kernels, not on production performance measurements.

11.2 Lockdep

Lockdep is the kernel’s lock-dependency validator. It builds a directed graph in which:

  • Nodes are lock classes, not individual lock instances. A class is identified by the initialization site or an explicit key for dynamic locks; for statically defined locks, lockdep uses the lock object itself as the key.
  • Edges represent observed acquisition order: if a CPU holds lock X and acquires lock Y, lockdep records X -> Y.
  • Cycles represent potential deadlocks. If X -> ... -> Y -> X exists, lockdep reports the cycle with the stack traces that created each edge.
1
2
3
4
5
6
7
8
9

lockdep dependency graph:

  lock_A ---> lock_B ---> lock_C
                |
                +------> lock_D ---> lock_A

  Cycle: A -> B -> D -> A
  lockdep reports a potential deadlock, even if the timing has
  never occurred in practice.

Lockdep also tracks whether a lock is used from hard IRQ, softirq, or process context, and whether interrupts are enabled. That lets it catch context bugs such as acquiring a lock in process context with interrupts enabled when the same class is acquired in a hard IRQ handler.

CONFIG_PROVE_LOCKING selects LOCKDEP, DEBUG_SPINLOCK, DEBUG_LOCK_ALLOC, and TRACE_IRQFLAGS (lib/Kconfig.debug). It is the single most valuable debugging option for spinlock code.

Lockdep identifies dynamic lock classes by initialization site. spin_lock_init() creates a static struct lock_class_key in the macro expansion, so two locks initialized at the same source line share a class. A statically defined DEFINE_SPINLOCK uses the lock object itself as its key. That is usually what you want: the class represents “this kind of lock,” and the graph reasons about all instances of that kind. If two instances need different ordering rules, use lockdep_set_class() or the _nested() annotations to distinguish them.

CONFIG_LOCK_STAT adds contention statistics and the lock_contended / lock_acquired tracepoints. With CONFIG_LOCK_STAT, LOCK_CONTENDED() first tries do_raw_spin_trylock() and records contention if the trylock fails (include/linux/lockdep.h). Without it, the macro calls do_raw_spin_lock() directly and the fast path is unchanged.

Lockdep is not a proof for every possible timing, but it is remarkably effective at finding ordering bugs before they hang a machine. It also has false positives when code intentionally uses unusual lock orders; annotate those cases explicitly rather than disabling lockdep.

11.3 CONFIG_DEBUG_ATOMIC_SLEEP

CONFIG_DEBUG_ATOMIC_SLEEP turns “sleeping in atomic context” into a visible report. It selects PREEMPT_COUNT and instruments sleeping primitives with might_sleep():

  • might_sleep() warns if the caller is in an atomic section, has interrupts disabled, or is otherwise not allowed to block.
  • schedule_debug() checks in_atomic_preempt_off() and reports BUG: scheduling while atomic if a schedule occurs with preemption disabled.
  • The report includes the preemption count, IRQ state, task name, held locks, and a stack trace.
  • panic_on_warn turns the warning into a panic if the test farm is configured that way.

The scheduler report is generated in kernel/sched/core.c, and the might_sleep() implementation is at kernel/sched/core.c. The warning is not a guarantee that every possible sleep path is covered, but it catches the common cases: msleep(), mutex_lock(), wait_event(), GFP_KERNEL allocation, and user copies that fault.

11.4 KCSAN

KCSAN (Kernel Concurrency Sanitizer) detects data races by instrumenting memory accesses and watching for conflicting accesses that are not ordered by a recognized synchronization operation. It is enabled with CONFIG_KCSAN and is documented in Documentation/dev-tools/kcsan.rst.

KCSAN complements lockdep rather than replacing it:

  • Lockdep checks whether the locking protocol can deadlock or is used from the wrong context.
  • KCSAN checks whether individual memory accesses are actually protected or marked with READ_ONCE() / WRITE_ONCE() / data_race().

A typical KCSAN report shows two conflicting accesses, the tasks and CPUs involved, and the stack traces. KCSAN understands many kernel synchronization primitives, including spinlocks, but it can produce false positives for accesses that are intentionally racy (statistics, heuristics, lockless fast paths). Use data_race() for those, and READ_ONCE() / WRITE_ONCE() for legitimate lockless accesses.

KCSAN is a runtime detector: it only reports races that actually occur during the test. Run it with a workload that exercises the code; a boot-only test will miss most races.

11.5 Contention Tracing

The qspinlock slow path emits contention tracepoints:

1
2
3

trace_contention_begin(lock, LCB_F_SPIN);
...
trace_contention_end(lock, 0);

See kernel/locking/qspinlock.c and include/trace/events/lock.h. The LCB_F_SPIN flag distinguishes spinning locks from mutexes, RT mutexes, and read/write locks. These events let you see which locks are contended, how long waiters spin, and which call sites are responsible.

There is also an optional release-side tracepoint for queued spinlocks, controlled by CONFIG_QUEUED_SPINLOCKS_TRACE_CONTENDED_RELEASE (kernel/Kconfig.locks). It fires lock:contended_release when a contended queued spinlock is released, allowing a trace to correlate contention begin, acquisition, and release.

Other useful tools:

  • CONFIG_LOCK_STAT – lock contention statistics and lock:lock_contended / lock:lock_acquired tracepoints.
  • perf lock – record and report lock contention from perf events, when supported by the architecture and kernel configuration.
  • lockdep stats – /proc/lockdep_stats shows class counts, chain lookup depth, and other internal costs.
  • tracefs – enable lock:contention_begin and lock:contention_end and correlate with scheduler and IRQ events.

Measure hold time, wait time, and contention frequency separately. A lock with high contention but a tiny hold time may be fine; a lock with low contention but a millisecond hold time is a latency disaster.

11.6 Debugging Checklist

When a spinlock bug is suspected:

  1. Reproduce with CONFIG_DEBUG_SPINLOCK, CONFIG_PROVE_LOCKING, and CONFIG_DEBUG_ATOMIC_SLEEP. These options turn silent hangs into actionable reports.
  2. Read the first report, not the last. A recursive lock or a sleep-in-atomic bug often produces a cascade of secondary warnings.
  3. Check the lock’s context rules. Is it acquired from a hard IRQ? From a tasklet? From process context? Does every path use the same suffix?
  4. Check the lock order. Lockdep’s cycle report names the classes and the stack traces that created the edges.
  5. Check the lifetime. Was the lock freed, reinitialized, copied, or allocated on the stack while another context could use it?
  6. Check the release path. Every acquire must have exactly one release on every path, including error paths.
  7. Run KCSAN on a workload that exercises the code. A missing lock or a missing READ_ONCE() may not deadlock, but it is still a data race.
  8. Measure before optimizing. Use contention tracepoints and hold-time measurements; do not guess which lock is hot.

12. Performance and Scalability

12.1 The Cost of Busy-Waiting

A contended spinlock keeps the waiting CPU executing a polling loop or a WFE wait. The waiting CPU produces no useful work, and on an oversubscribed system it may be consuming a vCPU that the lock holder needs.

1
2
3
4

CPU 0 holds lock (10 us critical section)
CPU 1 ------------ spinning ------------> waits 10 us, burns ~30,000 cycles
CPU 2 ------------ spinning ------------> waits 10 us, burns ~30,000 cycles
CPU 3 ------------ spinning ------------> waits 10 us, burns ~30,000 cycles

The cycle count assumes a 3 GHz CPU and a tight loop; the exact number depends on the wait instruction, cache behavior, and whether the CPU enters a low-power WFE state. The important point is that the cost scales with the number of waiters and the hold time.

A mutex or semaphore would put the waiters to sleep and let them do other work, at the cost of two context switches. Spinlocks are justified when the expected wait is shorter than that overhead, or when the waiter cannot sleep because it is in interrupt context.

12.2 Cache-Line Contention

The lock word is a shared cache line. Every acquisition, release, and handoff transfers ownership of that line between CPUs. The qspinlock reduces the polling traffic – waiters behind the head spin on their own nodes – but it cannot eliminate the cache-line transfer at handoff.

Two distinct problems are often conflated:

  • True sharing. Multiple CPUs genuinely need the same lock word. The serialization is inherent; the best fix is to reduce hold time or shard the lock.
  • False sharing. Two unrelated locks or variables share a cache line, so accessing one invalidates the other. The fix is alignment and padding.
1
2
3
4
5
6

/* Hot lock: keep it away from unrelated read-mostly data. */
struct hot_state {
    spinlock_t lock ____cacheline_aligned;
    unsigned int count;
    ...
};

For a global lock that is heavily contended, consider whether the data can be sharded. An array of locks indexed by a hash of the object address can turn one hot lock into many warm locks, at the cost of more memory and more complex lookup.

12.3 NUMA

On a NUMA system, memory accesses to remote nodes are significantly slower than local accesses. A spinlock’s lock word resides on exactly one node. Under high contention from remote nodes, the cache line must travel across the interconnect.

1
2
3
4
5
6
7
8
9
10

NUMA topology -- 2-node system

  Node 0                     Node 1
  +------------------+        +------------------+
  | CPU 0  CPU 1     |        | CPU 2  CPU 3     |
  |  +------+        |        |  +------+        |
  |  Local memory    |        |  Local memory    |
  |  lock word       |<------>|  remote access   |
  |  lives here      | fabric |  crosses fabric  |
  +------------------+        +------------------+

The behavior differs by lock design:

  • Ticket lock: every waiter polls the same owner word. On unlock, the cache line crosses the interconnect to invalidate every waiting CPU simultaneously – O(N) interconnect traffic per handoff.
  • qspinlock: waiters behind the queue head spin on their own per-CPU MCS nodes, which are local to their CPUs. The queue head still polls the shared lock word, which may be remote, and the handoff writes to the next CPU’s node. This is much better than a broadcast, but it is not “zero cross-node traffic.”

Even with qspinlock, the serialization point remains. Under sustained contention from many NUMA nodes, architectural responses include:

  • Lock sharding – replace one global lock with an array indexed by a hash, so different objects contend on different locks.
  • RCU – concurrent readers with no lock on the read side; writers pay a higher cost but readers scale.
  • Per-CPU counters – each CPU increments its own counter; a global sum is computed only when needed.
  • Data placement – allocate the protected data and its lock on the node that uses them most, when the workload has locality.

12.4 Alternatives

A spinlock is not the only tool, and it is often not the best one:

PrimitiveBest forCost / caveat
atomic_t / atomic64_tSingle-word counters, flags, reference countsNo multi-field atomicity
spinlock_tShort, atomic, multi-field critical sectionsBusy-waits; no sleeping
mutexProcess-context critical sections that may sleepContext-switch cost; cannot use in IRQ
rwlock_tRead-mostly data with short read sectionsReader/writer fairness; cache-line contention
rw_semaphoreRead-mostly data where readers may sleepProcess context only
seqlock_tRead-mostly data with retryable readersReaders may retry after a concurrent write; writers serialize with other writers
RCURead-mostly, pointer-based dataWriters must wait for grace periods; classic RCU readers cannot sleep
Per-CPU dataNaturally CPU-local stateNo cross-CPU consistency without extra work
local_lock_tPer-CPU data with preemption/IRQ protectionNot a cross-CPU lock
Completion / waitqueueEvent notification and sleeping waitsNot a mutual-exclusion primitive

A useful rule: if the data is read far more often than written and readers can tolerate a retry, RCU or a seqlock may remove the lock from the read path entirely. If the data is naturally partitioned by CPU, per-CPU data is better than a global lock. If the critical section can sleep, a mutex is simpler and more efficient than a spinlock.

12.5 Virtualization and vcpu_is_preempted

In a virtualized environment, a vCPU holding a spinlock may be descheduled by the hypervisor. Other vCPUs spinning on the lock waste real CPU time waiting for a vCPU that is not running. Some architectures expose a vcpu_is_preempted(cpu) hook so optimistic spinning loops can detect this and stop spinning.

On AArch64, arch/arm64/include/asm/spinlock.h defines it to unconditionally return false:

1
2
3
4
5

#define vcpu_is_preempted vcpu_is_preempted
static inline bool vcpu_is_preempted(int cpu)
{
    return false;
}

The comment explains why: changing it would break osq_lock(), which calls vcpu_is_preempted() inside smp_cond_load_relaxed() (kernel/locking/osq_lock.c). On AArch64, smp_cond_load_relaxed() is implemented with a WFE-based wait. A polling condition like vcpu_is_preempted() needs a wakeup mechanism; WFE may sleep until a store event, so a condition that becomes true without a store would not be noticed. Wiring in a real paravirt yield would require a kick or another event source, and that optimization has not been implemented for AArch64.

Consequences:

  • Optimistic spinning in mutexes and rwsems may spin longer than necessary when a vCPU holder is preempted.
  • qspinlock waiters on a preempted vCPU holder can wait for a long time; the lock has no timeout and relies on the holder eventually running.
  • Overcommitting vCPUs makes this worse. If the workload is latency-sensitive, avoid oversubscription or use a hypervisor that provides lock-holder preemption information.

12.6 Measuring Contention

Do not optimize a lock without data. Useful measurements include:

  • Contention frequency – how often a waiter enters the slow path. The lock:contention_begin tracepoint and CONFIG_LOCK_STAT counters provide this.
  • Wait time – how long waiters spend before acquiring. The tracepoint timestamps and scheduler events can reconstruct it.
  • Hold time – how long the critical section runs. Instrument the entry and exit, or use lock:contention_end together with a release-side tracepoint; contention tracepoints alone only cover the wait.
  • Cache behavior – perf counters for cache misses and interconnect traffic, if the platform supports them.
  • CPU time – a high-contention lock shows up as system time and idle-spin time; do not mistake it for useful work.

A lock with high contention and a short hold time may be a candidate for sharding. A lock with low contention and a long hold time is a latency problem even if the CPU cost is low. Measure both.

13. Problems and Pitfalls

Spinlocks are the kernel’s workhorse synchronization primitive, but they carry a set of inherent problems. Most spinlock bugs fall into one of the categories below.

13.1 Busy-Waiting Wastes Cycles

A contended spinlock keeps the waiting CPU executing a polling loop or a WFE wait for the entire duration of the wait. Those cycles produce no useful work. On a system with many CPUs contending for the same lock, the waste multiplies.

1
2
3
4

CPU 0 holds lock (10 us critical section)
CPU 1 ------------ spinning ------------> waits 10 us
CPU 2 ------------ spinning ------------> waits 10 us
CPU 3 ------------ spinning ------------> waits 10 us

A mutex or semaphore would put CPUs 1-3 to sleep and let them do productive work while waiting. Spinlocks are justified only when the expected wait is shorter than the overhead of two context switches, or when the waiter cannot sleep because it is in interrupt context. WFE reduces power and bus traffic, but it does not make the waiting CPU available for other work.

13.2 Self-Deadlock

A CPU that holds a spinlock and attempts to acquire it again will spin forever. The holder is the only entity that can release the lock, but it is now stuck spinning.

1
2
3
4

spin_lock(&A);          <- CPU 0 holds A, preemption disabled
    ...
    spin_lock(&A);      <- CPU 0 spins on A; only CPU 0 can release it
                                           DEADLOCK

This is a special case of the general rule that spinlocks are not recursive. The same problem appears when a locked function calls another function that tries to take the same lock. Structure the code so the locked core and the unlocked wrapper are separate.

13.3 Sleeping Under the Lock

Any function that may schedule is forbidden inside a spinlock critical section:

1
2
3
4
5

Forbidden under a spinlock:
  msleep()              schedule()             wait_event()
  mutex_lock()          down()                 down_read()
  kmalloc(GFP_KERNEL)   copy_to_user()         copy_from_user()
  blocking I/O          synchronize_rcu()      flush_work()

Some qualifications:

  • GFP_ATOMIC allocation does not sleep, but it can fail and should still be used sparingly in a critical section.
  • printk() is generally safe in atomic context, but it is slow and can interact with console and NMI paths. It is not forbidden because it sleeps; it is discouraged because it lengthens the critical section.
  • copy_to_user() / copy_from_user() can fault and sleep. They must be outside the lock. If you need a best-effort user access in atomic context, use the appropriate _nofault variant with a clear fallback, not a normal copy.
  • kfree() is generally safe in atomic context, but freeing a large object or a complex subsystem object may still be expensive. Remove the object under the lock and free it afterward.

On a non-RT kernel, the immediate symptom is a BUG: scheduling while atomic report or a deadlock. On PREEMPT_RT, spinlock_t is a sleeping lock, so a task can block during acquisition. The critical section itself must still be non-blocking: a blocking section destroys the latency guarantees and can create deadlocks with locks that are not PI-aware. Treat the rule as configuration-independent.

13.4 Priority Inversion

Priority inversion occurs when a high-priority task waits for a resource held by a low-priority task, and a medium-priority task prevents the low-priority holder from running. On a non-RT kernel, a spinlock holder cannot be involuntarily preempted, so the classic unbounded inversion is largely prevented for the duration of the critical section:

1
2
3
4
5
6
7
8
9
10
11

Low-priority task L             High-priority task H
-------------------             --------------------
spin_lock(&lock)
  preemption disabled
  ... critical section ...
                                becomes runnable
                                spin_lock(&lock) -> spins
                                (or waits for the CPU)
spin_unlock(&lock)
  preemption enabled
                                eventually acquires

On the same CPU, H cannot run until L unlocks, because L has preemption disabled. On another CPU, H spins until L unlocks. Either way, H’s wait is bounded by the critical section, assuming L does not block or get delayed by interrupts. There is no priority inheritance: L is not boosted, so a long critical section delays H for its full duration.

Being preempted just before spin_lock() does not create an inversion, because L does not hold the lock yet. If L is preempted before acquiring, H can acquire the lock itself. The dangerous case is a holder that blocks or sleeps while holding the lock, which is a bug, or a critical section that is simply too long.

On PREEMPT_RT, spinlock_t becomes an RT mutex with priority inheritance. If H blocks on the lock, the kernel boosts L’s priority so it can finish and release the lock. raw_spinlock_t remains a spinning lock and does not provide this protection; use it only where true atomicity is required. See Section 14.

13.5 AB-BA Deadlock

When two or more locks are acquired in inconsistent order across different code paths, a circular wait can form:

1
2
3
4
5
6

CPU 0: spin_lock(A) -> spin_lock(B)
CPU 1: spin_lock(B) -> spin_lock(A)

  CPU 0 holds A, waits for B
  CPU 1 holds B, waits for A
  -> deadlock

The problem is invisible in testing until both paths execute concurrently at exactly the wrong moment. lockdep (CONFIG_PROVE_LOCKING) detects the cycle by tracking acquisition-order graphs and reports the stack traces that created each edge. The fix is a consistent global lock order, documented and enforced by code review and lockdep annotations.

13.6 IRQ Context Deadlock

If a spinlock is shared between process context and a hard IRQ handler, and process context acquires it with plain spin_lock() (without disabling interrupts), a hard IRQ on the same CPU can fire while the lock is held and attempt to acquire it:

1
2
3
4

CPU 0 (process context): spin_lock(&lock)   <- locked = 1
                         <- IRQ fires on CPU 0
CPU 0 (IRQ handler):     spin_lock(&lock)   <- spins on locked = 1
                                               holder cannot run

The fix is to use spin_lock_irqsave() or spin_lock_irq_disable() from process context. The hard IRQ handler itself uses plain spin_lock(), because local IRQs are already disabled in that context. If a BH handler shares the data with the hard IRQ handler, the BH handler must use spin_lock_irqsave().

13.7 Cache-Line and NUMA Contention

On systems with many CPUs or multiple NUMA nodes, a heavily contended spinlock creates cache-coherence traffic. Although the qspinlock MCS design limits polling traffic, the serialization bottleneck remains: only one CPU holds the lock at a time, and the lock word’s cache line must move between CPUs at handoff.

1
2
3
4
5

Node 0                     Node 1
+------------------+       +------------------+
| CPU 0  CPU 1     |       | CPU 2  CPU 3     |
|  lock word       |<----->|  remote access   |
+------------------+       +------------------+

Mitigations include lock sharding, RCU for read-mostly data, per-CPU counters, and better data placement. See Section 12.3.

13.8 Hold-Time Constraint

Spinlocks impose a strict discipline: keep critical sections short. There is no mechanism that enforces this at compile time or runtime. A developer who puts a slow path – a loop, an allocation, a conditional that sometimes takes a long time – inside a spinlock will silently degrade system throughput and latency under load.

The practical test is: can you explain why every instruction in the critical section must be atomic with respect to the protected data? If not, move it out.

13.9 Summary

ProblemConsequenceMitigation
Busy-waitingWastes CPU cycles under contentionKeep sections short; use a mutex for long waits
Not recursiveSelf-deadlock, silent hangNever re-acquire; use CONFIG_DEBUG_SPINLOCK and lockdep
Sleeping under the lockScheduler bug / deadlockCONFIG_DEBUG_ATOMIC_SLEEP; use a mutex instead
Priority inversionHigh-priority task delayed by the holderKeep sections short; PREEMPT_RT for PI
AB-BA deadlockSystem hang under concurrent loadConsistent lock order; lockdep
IRQ context deadlockIRQ handler spins forever_irqsave / _irq_disable from process context
Cache-line / NUMA contentionMemory traffic at scaleLock sharding, RCU, per-CPU data
Long critical sectionsThroughput and latency collapseProfile; refactor to finer granularity

14. PREEMPT_RT

PREEMPT_RT changes the meaning of spinlock_t. The name stays the same, but the implementation becomes a sleeping RT mutex. Code that assumes a spinlock disables preemption or interrupts will be wrong on an RT kernel.

14.1 What Changes

PropertyNon-RTPREEMPT_RT
spinlock_t implementationraw_spinlock_t / qspinlockstruct rt_mutex_base
Acquisition can sleepNoYes, when contended
Disables preemptionYesNo (disables migration)
_irq / _irqsave / _irq_disable disable hard IRQsYesNo (_irqsave sets flags = 0)
_bh disables softirqsYesYes
raw_spinlock_tSpinning qspinlockStill a spinning qspinlock
Priority inheritanceNoYes, for spinlock_t
Usable in hard IRQ contextYes, with the right suffixspinlock_t is for threaded/preemptible contexts; use raw_spinlock_t in true hard IRQ
Sleeping inside the critical sectionForbidden; may deadlock or warnTechnically possible, but defeats the latency and PI design; keep it non-blocking

The documentation summarizes the model in Documentation/locking/locktypes.rst:

  • raw_spinlock_t is a strict spinning lock in all kernels.
  • spinlock_t is a spinning lock on non-RT and an RT mutex on RT.
  • On RT, spinlock_t does not disable preemption; tasks holding it do not migrate, but they can be preempted.
  • _irq and _irqsave do not affect the CPU’s interrupt state on RT; _bh still disables softirq handlers.

14.2 spinlock_t Becomes an RT Mutex

The type definition changes under CONFIG_PREEMPT_RT. The excerpt below is the #else branch of #ifndef CONFIG_PREEMPT_RT, so it shows the RT definition:

1
2
3
4
5
6
7
8
9
10
11
12

#else /* !CONFIG_PREEMPT_RT */

/* PREEMPT_RT kernels map spinlock to rt_mutex */
#include <linux/rtmutex.h>

context_lock_struct(spinlock) {
    struct rt_mutex_base lock;
#ifdef CONFIG_DEBUG_LOCK_ALLOC
    struct lockdep_map dep_map;
#endif
};
typedef struct spinlock spinlock_t;

See include/linux/spinlock_types.h. The API wrappers call rt_spin_lock() and rt_spin_unlock() instead of the raw qspinlock path:

1
2
3
4

static __always_inline void spin_lock(spinlock_t *lock)
{
    rt_spin_lock(lock);
}

See include/linux/spinlock_rt.h. The RT mutex can block if the lock is contended, so the caller must be in a context where sleeping is allowed. A spinlock_t held in a hard IRQ handler on RT is wrong; the hard IRQ path must use a raw spinlock or be threaded.

14.3 raw_spinlock_t Remains a Spinning Lock

raw_spinlock_t is not remapped on RT. It remains the qspinlock-based spinning lock, and its _irq / _irqsave / _irq_disable variants still disable interrupts. It is intended for:

  • Low-level interrupt and exception handling.
  • Scheduler and runqueue internals.
  • Code that must not sleep even on RT.
  • Very small critical sections where the RT mutex overhead is not justified.

The rule of thumb is: use spinlock_t by default, and switch to raw_spinlock_t only when the code genuinely cannot sleep and cannot tolerate the RT mutex’s blocking behavior. Do not use raw_spinlock_t to “make it faster” without understanding the RT latency consequences.

14.4 IRQ Suffixes on RT

On RT, the interrupt-disabling suffixes do not do what their names suggest:

1
2
3
4

/* On PREEMPT_RT, flags is set to 0 and no hard IRQ is disabled. */
spin_lock_irqsave(&lock, flags);
...
spin_unlock_irqrestore(&lock, flags);

See include/linux/spinlock_rt.h. spin_lock_irq() / spin_unlock_irq() and spin_lock_irq_disable() / spin_unlock_irq_enable() likewise do not change the hard IRQ state. spin_lock_bh() still disables softirqs, so it remains meaningful for BH-protected data.

Consequences for porting:

  • If the code requires hard IRQs to be disabled for a hardware access, prefer a raw_spinlock_t for the whole critical section. If the spinlock_t protects only state and the IRQ-off window is genuinely short and local, acquire the sleeping lock first, then use local_irq_save() / local_irq_restore() around the access itself. Do not sleep with IRQs disabled.
  • Do not use spin_lock_irqsave() as a substitute for local_irq_save() on RT.
  • Do not assume that a spinlock_t critical section runs with preemption disabled. Code that calls preempt_disable() or relies on per-CPU stability must use the appropriate local_lock_t or raw_spinlock_t.

14.5 Priority Inheritance

RT mutexes implement priority inheritance. If a high-priority task blocks on a spinlock_t held by a low-priority task, the kernel temporarily boosts the holder’s priority so it can finish and release the lock. This addresses the unbounded priority inversion that a sleeping lock would otherwise suffer.

Priority inheritance has limits:

  • It cannot boost a task that is not runnable because it is blocked on a non-PI resource such as I/O or a semaphore.
  • It cannot preempt a preemption-disabled or interrupt-disabled region; RT tries to convert most such regions to preemptible contexts, but raw_spinlock_t sections remain non-preemptible.
  • It does not make a long critical section short. The holder still determines the worst-case latency.

14.6 Porting Checklist

When moving code to PREEMPT_RT:

  1. Audit every spinlock_t. Does the critical section sleep? Does it rely on preemption being disabled? Does it run in hard IRQ context?
  2. Use raw_spinlock_t only where required. True hardware access, scheduler internals, and NMI/exception paths may need it. Ordinary driver state usually does not.
  3. Replace spin_lock_irqsave() assumptions. On RT, it does not disable hard IRQs. For a hardware access, prefer a raw lock for the whole critical section; if the sleeping lock protects only state and the IRQ-off window is short, use local_irq_save() around the access itself. Do not sleep with IRQs disabled.
  4. Check spin_is_locked() and spin_is_contended(). On RT, spin_is_locked() reads the RT mutex state; spin_is_contended() is always zero. Polling either is still wrong.
  5. Do not sleep inside the critical section. Acquisition may sleep, but the section itself must remain atomic in the API’s sense.
  6. Use local_lock_t for per-CPU data. On RT, local_lock_t provides the per-CPU serialization that preempt_disable() used to provide.
  7. Test with lockdep and RT. Run the RT-specific lockdep checks and latency tests; a non-RT boot test will not catch these issues.

15. Summary: Rules of Thumb

  1. Use a spinlock only for short, atomic, multi-field critical sections. If the section can sleep, use a mutex or another sleeping lock. If the state is one word, use an atomic.

  2. Initialize with DEFINE_SPINLOCK or spin_lock_init(). Never memset, copy, or hand-initialize a lock. Keep it at a stable address for its lifetime.

  3. Lock data, not code. Put the lock next to the data it protects, name it for that data, and acquire it at every access site.

  4. Never sleep under the lock. No msleep, schedule, mutex_lock, GFP_KERNEL allocation, user copy that may fault, or blocking I/O. The rule applies on RT as well, even though acquisition itself may sleep.

  5. Choose the suffix by context (non-RT). _irqsave when a hard IRQ handler shares the lock; _irq_disable for nested/hand-over-hand IRQ-disabling sections; _bh when softirqs or tasklets share it; plain spin_lock when the lock is never taken from interrupt or BH context. When in doubt, _irqsave is the safe default. On RT, see Section 14.

  6. On a non-RT kernel, a hard IRQ handler uses plain spin_lock(). Local IRQs are already disabled. A BH handler uses spin_lock() unless a hard IRQ also shares the data, in which case it uses _irqsave.

  7. spin_lock_bh() is a process-context tool. It prevents a softirq/tasklet from preempting the local CPU. It is not needed inside a BH handler, and it does not protect against hard IRQs.

  8. Never recurse. On a non-RT kernel, a spinlock has no ownership in production hardware, and the same CPU trying to acquire it again deadlocks. On RT the implementation is an RT mutex, but recursive acquisition is still forbidden. CONFIG_DEBUG_SPINLOCK reports the recursion but does not prevent the hang.

  9. Release in the same context that acquired. Do not hand a spinlock from one task to another. Use completions, waitqueues, or explicit state for cross-context handoff. Locking in one function and unlocking in another is legal but fragile; prefer a helper that keeps the pair together.

  10. Maintain a global lock order. Lockdep is the tool that proves it. Do not call unknown callbacks or acquire unrelated locks while holding a lock unless the order is documented.

  11. qspinlock is fair, but it is not a ticket lock. Queued waiters are served in queue order, and the pending slot is reserved for the first waiter. The fast path and pending path are optimizations; the lock does not provide the strict total arrival order of a ticket lock, but it does not allow unbounded barging.

  12. Use the debug tools. CONFIG_DEBUG_SPINLOCK, CONFIG_PROVE_LOCKING, CONFIG_DEBUG_ATOMIC_SLEEP, KCSAN, and contention tracepoints exist to catch the mistakes that otherwise produce silent hangs.

  13. Measure before optimizing. Contention frequency, wait time, and hold time are different metrics. A lock can be hot but short, or cold but disastrous for latency.

  14. Remember the RT caveat. On PREEMPT_RT, spinlock_t is a sleeping RT mutex. Code that relies on preemption or interrupt disabling must be audited and often converted to raw_spinlock_t or explicit IRQ control.

The spinlock is a precision tool. It buys mutual exclusion at the cost of busy-waiting, preemption-disabled sections, context rules, and a strict hold-time discipline. Used for the right shape of data – short, atomic, multi-field critical sections – it is the fastest and simplest lock in the kernel. Used outside that shape, it is a source of hangs, latency spikes, and difficult races. The source makes the trade-off visible: one 32-bit word for the fast path, a pending byte for the second waiter, per-CPU MCS nodes for the queue, and acquire/release ordering to make it all work on AArch64.

16. References

16.1 Core Types and API

16.2 Queued Spinlock Implementation

16.3 AArch64 Memory Ordering and Atomics

16.4 Lockdep and Debugging

16.5 Documentation and Real-World Usage

This document describes spinlock_t as implemented in Linux v7.3-rc7 on AArch64, in a non-PREEMPT_RT SMP configuration unless otherwise noted. Source line numbers refer to that tag; kernel code evolves, so verify against the tree you are building when using this as a reference.

Synchronization
spinlock qspinlock mcs interrupt preempt_rt
This post is licensed under CC BY 4.0 by the author.