Skip to content
Work Practice Products Coding interviews Machine learning coding Quant developer HFT course Blog Method Why me About Enquire Book a call

Quant developer HFT course · Module 7: The network

The market-data feed handler

Lesson · 12 min

The question

Design the component that receives an exchange's market data and keeps order books up to date for the strategies. What happens when a packet is lost?

What it does

A feedmulticastarbitratefirst copy winssequence checkgap?decodebinary → structbook updatepublishring to strategies
A feed handler. Two identical feeds arrive on separate lines; the handler keeps whichever copy of each packet arrives first, checks sequence numbers for gaps, decodes, updates the book and publishes.
  1. Receive packets from the A and B multicast feeds, often on different network cards, by kernel bypass.
  2. Arbitrate: keep the first copy of each sequence number and discard the duplicate from the other feed.
  3. Check sequence: compare each packet's sequence number with the next expected one; detect gaps.
  4. Decode each message in the packet from the exchange's binary format.
  5. Apply it to the book for its instrument: add, modify or delete an order, or a level update, or a trade.
  6. Publish the result to strategies: the changed book, a top-of-book update, or the raw normalised event, through shared-memory rings.

Sequence numbers and arbitration

Every packet on a feed carries a sequence number that increases by one (per channel; large exchanges split instruments across many channels, each with its own sequence and multicast group). The handler keeps next_expected per channel:

void on_packet(Channel& ch, const Packet& p) {
    if (p.seq < ch.next) return;                    // duplicate: the other feed already delivered it
    if (p.seq > ch.next) { ch.gap(ch.next, p.seq); }  // missing ch.next .. p.seq-1
    process(p);                                       // or buffer it while recovering (below)
    ch.next = p.seq + 1;
}

With two feeds, a packet lost on A is usually delivered by B, so the handler never sees a gap.

Arbitration between two feeds. Packet 103 never arrived on A, but B delivered it, so the handler saw no gap. Only a packet missing from both feeds forces recovery.

  1. Step 1 / 4
    feed Afeed Bhandler101101101

    101 arrives on feed A first; the handler applies it and discards B's copy.

  2. Step 2 / 4
    feed Afeed Bhandler101102101102101102A lost 103: B fills the gap

    102: the same.

  3. Step 3 / 4
    feed Afeed Bhandler101102101102103101102103A lost 103: B fills the gap

    103 never arrives on A. B delivers it, the handler applies it, and there is no gap.

  4. Step 4 / 4
    feed Afeed Bhandler101102104101102103104101102103104A lost 103: B fills the gap

    104 arrives on A first again.

Arbitration is a race: whichever feed delivers a packet first wins. Feeds have different path latencies, and the faster one is not always the same one, so handlers arbitrate per packet rather than preferring one feed.

Handling a real gap

When both feeds miss a packet, the book is now wrong: an order may have been added, changed or deleted without the handler knowing. Options, in rough order of preference:

  • Wait briefly for the missing packet: it may be reordered rather than lost. Buffer the packets after the gap meanwhile.
  • Request retransmission of the missing range from the exchange's recovery service, usually over TCP, then apply the buffered packets after it.
  • Rebuild from a snapshot. Exchanges publish periodic full-book snapshots, on a separate feed or on request, tagged with the sequence number they correspond to. The handler clears the affected books, applies the snapshot, then replays buffered incremental packets with higher sequence numbers.

Throughout recovery, the affected books must be marked stale and strategies told, so that nothing trades on a book known to be wrong. Getting this logic right, including the edge cases (gaps during recovery, snapshots that are older than the buffer), matters more than shaving nanoseconds, and interviewers often probe it.

Decoding

Exchange protocols are binary: fixed-layout messages with a type byte and a length, little- or big-endian integers, prices as scaled integers (Module 10). Decoding is a switch on the type and memcpy of fixed-offset fields (Module 2), and it is one of the places where every branch and copy is visible in latency. Common tricks: generate the decoders from the exchange's specification; handle the most frequent message types first; keep instruments' books in an array indexed by a dense id mapped from the exchange's instrument id at start-up.

Order-by-order and price-level books

Feeds come in two styles. Market-by-order feeds (in the style of Nasdaq's ITCH) report every individual order: add order 123 at price p, execute 50 of order 123, delete order 123. The handler keeps every order and aggregates levels itself, which needs a hash map from order id to order (Module 3). Market-by-price feeds report changes to aggregated levels: the quantity at price p is now q. They are simpler to process and carry less information. Module 9 builds the book.

Publishing to strategies

The handler is a single writer serving many readers. Common designs: a broadcast ring of normalised events (Module 6), plus a seqlock-protected top of book per instrument (Module 4) that strategies can read at any time. Strategies that need only some instruments filter early, so they do not pay for everything.

Check yourself

Question 1 of 3

Packet 205 arrives on feed B after packet 205 was already processed from feed A. What should the handler do?

Question 2 of 3

Both feeds miss packet 310. What must happen to the affected books?

Question 3 of 3

Why do handlers arbitrate per packet rather than always preferring feed A?

In the interview

Say your own answer out loud first, then open the one we would give.

Describe a market data feed handler.

It receives the A and B multicast feeds, keeps the first copy of each sequence number, checks for gaps, decodes the exchange's binary messages, applies them to per-instrument books, and publishes updates to strategies through shared-memory rings. Recovery from gaps uses retransmission requests or snapshots, with affected books marked stale until they are rebuilt.

How do you handle a sequence gap?

Buffer later packets and wait briefly in case the missing one is only reordered or arrives on the other feed. If not, request the range from the exchange's retransmission service, or rebuild the affected books from a snapshot and replay the buffered packets after the snapshot's sequence number. While recovering, the books are flagged stale so nothing trades on them.

What is the difference between market-by-order and market-by-price feeds?

Market-by-order feeds report every individual order event, so the handler maintains each order and builds levels itself; that gives queue positions and requires an order-id map. Market-by-price feeds report aggregated level quantities; they are simpler and lighter but carry less information.