Socket programming for trading systems

A trading process talks to an exchange in two ways:

• Market data is usually UDP multicast: the exchange sends each packet once to a multicast group, and the network delivers it to every subscriber. Many exchanges publish the same data on two groups over separate network paths (the A and B feeds) for redundancy.
• Order entry is usually a TCP session, using a binary protocol of the exchange's own design or FIX: the client logs in, sends orders and receives acknowledgements and fills on the same connection.

int fd = socket(AF_INET, SOCK_DGRAM, 0);
int one = 1;
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof one);   // several processes may bind the port
sockaddr_in addr{}; addr.sin_family = AF_INET; addr.sin_port = htons(port);
addr.sin_addr.s_addr = inet_addr(group);                      // bind to the group, not INADDR_ANY
bind(fd, reinterpret_cast<sockaddr*>(&addr), sizeof addr);
ip_mreq mreq{}; mreq.imr_multiaddr.s_addr = inet_addr(group);
mreq.imr_interface.s_addr = inet_addr(local_if);              // the interface facing the exchange
setsockopt(fd, IPPROTO_IP, IP_ADD_MEMBERSHIP, &mreq, sizeof mreq);
int rcvbuf = 64 << 20;
setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &rcvbuf, sizeof rcvbuf); // capped by net.core.rmem_max
fcntl(fd, F_SETFL, O_NONBLOCK);

Points worth explaining in an interview:

• Joining the group (IP_ADD_MEMBERSHIP) makes the host send an IGMP report so the switches forward the group to it. Choosing the interface matters on machines with several networks.
• The receive buffer absorbs bursts while the reader is busy. If it fills, the kernel drops packets silently (lesson 4). The requested size is capped by net.core.rmem_max, which must be raised as well.
• Non-blocking mode lets a polling thread call recv without sleeping; recvmmsg reads many datagrams per system call.
• Timestamps: SO_TIMESTAMPING asks for software or hardware receive timestamps with each packet, which Module 8 and Module 10 depend on.

int fd = socket(AF_INET, SOCK_STREAM, 0);
connect(fd, /* exchange gateway address */ ...);
int one = 1;
setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &one, sizeof one);   // send small writes immediately

Nagle's algorithm, on by default, holds back small writes while earlier data is unacknowledged, so that it can combine them into fewer packets. It saves bandwidth for interactive terminals and is exactly wrong for orders, which are small and urgent. Combined with the receiver's delayed acknowledgements, it can hold an order back for tens of milliseconds. TCP_NODELAY disables it. Write each message in one send call so it leaves as one segment.

Other TCP options that come up: TCP_QUICKACK to avoid delayed acknowledgements on the receive side, SO_SNDBUF for the send buffer, and keepalive settings, though the exchange protocol's own heartbeats usually do that job.

TCP is a byte stream, not a message stream. One recv can return half a message, or three messages and part of a fourth. The reader must keep a buffer, append whatever arrives, and parse complete messages using the length field in each message header, keeping any partial message for the next read. Assuming one recv returns one message is one of the most common bugs in interview code.

A hot thread polls its sockets in a loop with non-blocking reads, or uses kernel bypass so there are no system calls at all (lesson 6). For many sockets that are not latency-critical, epoll reports which are ready without polling each one. Blocking reads have no place on a hot thread: the thread sleeps and must be woken (Module 4).