Allocators, pmr and object pools

Low-latency code uses three patterns to get memory without a general-purpose allocator on the hot path. Each is built on the same move: take one large block at start-up, then hand out pieces of it with an operation that is a few instructions.

An object pool holds a fixed number of objects of one type. Free slots form a linked list threaded through their own storage, so acquiring an object pops the head and releasing pushes it back.

template <class T, std::size_t N>
class Pool {
    union Slot { Slot* next; alignas(T) std::byte obj[sizeof(T)]; };
    std::array<Slot, N> slots_;
    Slot* free_ = nullptr;
public:
    Pool() { for (auto& s : slots_) { s.next = free_; free_ = &s; } }
    template <class... A> T* acquire(A&&... a) {
        if (!free_) return nullptr;                   // full: the caller decides what to do
        Slot* s = free_; free_ = s->next;
        return ::new (s->obj) T(std::forward<A>(a)...);
    }
    void release(T* p) {
        p->~T();
        Slot* s = reinterpret_cast<Slot*>(p); s->next = free_; free_ = s;
    }
};

Acquire and release are constant time and never touch the heap. All objects sit in one array, so they share cache lines and pages that stay warm. The cost is a fixed capacity, chosen up front, and a policy for running out: reject the order, fall back to the heap and raise an alert, or size it so it cannot happen.

An arena (also called a bump or monotonic allocator) hands out memory by advancing a pointer through a block, and frees everything at once by resetting the pointer. It suits objects that all die together: everything allocated while handling one message, or one batch.

class Arena {
    std::byte* base_; std::size_t cap_, used_ = 0;
public:
    void* alloc(std::size_t n, std::size_t align) {
        std::size_t p = (used_ + align - 1) & ~(align - 1);
        if (p + n > cap_) return nullptr;
        used_ = p + n;
        return base_ + p;
    }
    void reset() { used_ = 0; }        // frees everything; destructors are your job
};

A size-class allocator keeps a free list per size range, like a general allocator but without locks and owned by one thread. Use it when object types vary but stay on one thread.

Standard containers take an allocator as a template parameter, which historically meant a different container type for every allocator. C++17's polymorphic memory resources fix that: std::pmr::vector<T> is a std::vector whose allocator calls a std::pmr::memory_resource through a virtual function, chosen at run time.

std::array<std::byte, 64 * 1024> buf;
std::pmr::monotonic_buffer_resource arena(buf.data(), buf.size(),
                                          std::pmr::null_memory_resource());   // never touches the heap
std::pmr::vector<Fill> fills(&arena);
std::pmr::string note("partial fill", &arena);

The library provides monotonic_buffer_resource (an arena), unsynchronized_pool_resource (size-class pools for one thread), synchronized_pool_resource (the same with locking), and null_memory_resource as an upstream that throws instead of falling back to the heap, so an overflow fails loudly in testing.

The virtual call per allocation is a small price; the benefit is that existing code using standard containers can be moved onto an arena by changing the resource, not the types.

• Fixed capacity, for pools and arenas, which turns running out into a design decision.
• Per-object freeing with an arena: nothing is reclaimed until reset, so a long-lived object allocated from a per-message arena dangles after the reset.
• Thread confinement. These allocators are fast because one thread owns them. An object released on a different thread from the one that acquired it needs a handoff, usually by sending it back through a queue.
• Destructors. An arena reset frees memory without running destructors. That is fine for trivially destructible types and a leak for anything that owns other memory.