Debugging lock contention

Contention looks like waiting: throughput that stops scaling with threads, CPU utilisation that stays low while latency rises, many context switches, and time spent in futex system calls. A CPU profile shows little, because waiting threads are not running. Sampling profiles see on-CPU time; contention is off-CPU time.

• Off-CPU profiling. Record where threads block and for how long, with their stacks. BCC's offcputime, perf sched record, or bpftrace scripts on scheduler events give an off-CPU flame graph; the widest stacks blocked inside pthread_mutex_lock name the lock's callers.
• futex tracing. perf trace -e futex or strace -f -e futex shows the waits and wakes, with the address of each futex. Repeated waits on one address identify one mutex.
• perf lock on kernels and builds that support it reports contended locks with wait times.
• Instrumented mutexes. Wrapping the mutex to record the time between attempting and acquiring, and how long it was held, with the call site, gives the most direct answer and can stay in production at low cost:

class TimedMutex {
    std::mutex m_;
public:
    void lock(const char* site) {
        auto t0 = tsc();
        if (!m_.try_lock()) { m_.lock(); record_wait(site, tsc() - t0); }   // only contended locks pay
        held_since_ = tsc();
    }
    void unlock(const char* site) { record_hold(site, tsc() - held_since_); m_.unlock(); }
private:
    std::uint64_t held_since_ = 0;
};

• Valgrind's DRD and Helgrind, and ThreadSanitizer, find lock-order problems and races, not contention, but are worth running for the deadlock side of lock bugs.

Contention is a product of two numbers: how often threads want the lock, and how long each holds it. The wait histogram tells you contention exists; the hold-time histogram per call site tells you who causes it. A lock held for 50 µs by one call site while others want it for 200 ns is a hold-time problem at that site. A lock held briefly but wanted by every thread for every request is a frequency problem: a single global serialisation point.

1. Shorten the critical section: move logging, allocation, I/O and computation on copies outside it (Module 4).
2. Split the lock: separate locks for independent data, or striped locks (a lock per hash bucket or per instrument).
3. Read-mostly data: a reader-writer lock, or better, publish immutable snapshots that readers access with no lock at all (an atomic pointer to the current version, or RCU).
4. Per-thread state: per-thread counters, caches and buffers, merged occasionally.
5. Ownership and messages: give the data a single owning thread and send it requests through queues.

Then measure again: wait and hold histograms per site, throughput against thread count, and off-CPU time.