Time, timestamps and clocks

• Wall-clock time (CLOCK_REALTIME, std::chrono::system_clock) answers "what time is it?" in UTC. It can jump: NTP or PTP corrections, an administrator setting the clock, leap-second handling. Needed for anything compared with the outside world: regulatory records, exchange timestamps, logs read by people.
• Monotonic time (CLOCK_MONOTONIC, std::chrono::steady_clock) answers "how much time has passed?". It never goes backwards and is not set, though it may be slewed slightly by synchronisation. Needed for measuring intervals, timeouts and rate limits.

Measuring a latency with the wall clock can give negative or absurd durations when the clock is adjusted mid-measurement. That is the first point of the answer.

On Linux, clock_gettime for the common clocks is served by the vDSO: a small piece of kernel code mapped into the process, which reads the time-stamp counter and converts it using parameters the kernel publishes, with no system call. That costs a few tens of nanoseconds. On some virtual machines, or with an unsuitable clock source, it falls back to a system call costing much more; cat /sys/devices/system/clocksource/clocksource0/current_clocksource should say tsc.

Tens of nanoseconds per read is acceptable for most purposes and too much for timestamping every stage of every message on the hot path, which is where the TSC comes in.

The time-stamp counter is a per-core 64-bit counter read by the rdtsc instruction. On modern x86 processors with an "invariant TSC" (the constant_tsc and nonstop_tsc flags in /proc/cpuinfo), it ticks at a constant rate regardless of frequency changes and power states, and is synchronised across the cores of a machine. Reading it takes on the order of ten to twenty cycles.

Using it correctly:

• Calibrate. It counts cycles of a nominal frequency, not nanoseconds. Measure its rate against CLOCK_MONOTONIC at start-up (and periodically), and convert with a multiply and shift.
• Ordering. rdtsc is not a serialising instruction: the CPU may execute it before earlier instructions finish or after later ones start. For measuring a short region precisely, rdtscp waits for earlier instructions, and a fence (lfence) before rdtsc stops it being executed early. For timestamping events in a pipeline, plain rdtsc is usually fine.
• Cross-machine comparisons need conversion to a common clock: TSCs on different machines are unrelated.

inline std::uint64_t ticks() { return __rdtsc(); }       // stage timestamps on the hot path

struct TscClock {                                          // calibrated at start-up
    std::uint64_t t0; std::int64_t ns0; double ns_per_tick;
    std::int64_t to_ns(std::uint64_t t) const { return ns0 + static_cast<std::int64_t>((t - t0) * ns_per_tick); }
};

Hot-path code records raw TSC values; conversion to nanoseconds and to UTC happens later, off the hot path.

The network card can timestamp packets as they arrive and leave, with its own clock (the PTP hardware clock, PHC). These are the most accurate timestamps of when an event reached or left the machine, unaffected by kernel processing, and the basis of tick-to-trade measurement (Modules 7 and 8).

• NTP synchronises over the network to within milliseconds, sometimes better, in software.
• PTP (Precision Time Protocol, IEEE 1588) uses hardware timestamping in network cards and PTP-aware switches to synchronise to within well under a microsecond on a good network, from a grandmaster clock usually disciplined by GPS.
• On the host, ptp4l keeps the card's PHC in sync with the grandmaster, and phc2sys steers the system clock to the PHC.

Regulators require trading firms to record order events with timestamps traceable to UTC. In the European Union, MiFID II's clock-synchronisation rules (RTS 25) require firms engaged in high-frequency algorithmic trading to keep their clocks within 100 microseconds of UTC and to timestamp with a granularity of one microsecond or better, with lower requirements for other activity. US rules (the Consolidated Audit Trail) also set clock tolerances and timestamp granularity for reportable events. Meeting these means PTP-synchronised clocks, monitoring of the offset, and records of that monitoring, which is why the timing chain is a production system in its own right.

For latency inside the process, the TSC: cheap, monotonic and synchronised across cores on modern x86, read on the hot path and converted to nanoseconds later using a calibration against the monotonic clock. For anything that leaves the process or must be compared with the outside world, UTC from the system clock, which PTP keeps within the regulatory tolerance via the NIC's hardware clock. Network events are best timestamped by the card itself. I would never measure durations with the wall clock, because it can jump.