Quant developer HFT course · Module 8: Measuring latency
Telling a debugging story in an interview
The question
Tell me about the hardest performance problem you have debugged.
What the question tests
This is a behavioural question with a technical core. The interviewer wants to know whether you investigate methodically or guess, whether you measure, whether you understand the mechanism you found, and whether you made sure it did not come back. They will interrupt with follow-ups, so every claim in your story should be one you can go one level deeper on.
A structure that works
- The symptom, concretely: what was slow, how you knew, how big, how often, and why it mattered. "p99 of order acknowledgement handling went from 6 to 40 µs after a release, on busy days only."
- Measurement first. What you measured to locate it, and what it showed. "Stage timestamps showed the time was in the risk check; off-CPU profiling showed threads blocked in a mutex."
- A hypothesis with a mechanism. "The new position report took the risk lock and formatted a report under it."
- A test of the hypothesis, ideally one that predicted a number. "If that was it, lock wait time at the risk call site should match the report's duration and occur only when reports ran. It did."
- The fix and its trade-off. "We took a snapshot of positions under the lock and formatted outside it. It cost a copy of a few kilobytes per report."
- Verification: the same measurement that found it, now healthy. "p99 back to 6 µs; wait time at that site near zero."
- Prevention: what changed so it would not recur. "A test that fails if any lock on the order path is held longer than a threshold, and the hold-time histogram on the dashboard."
Good and bad versions
Weak: "The system was slow, so I profiled it, found a hot function and optimised it, and it was three times faster." No numbers you can defend, no mechanism, no explanation of why the profile pointed there, and the interviewer's first follow-up ("what made it hot?") has no answer.
Strong stories have a few features in common:
- Numbers with units and percentiles, and how they were measured.
- At least one wrong hypothesis, and how evidence ruled it out. Real investigations have them, and saying so is more credible than a straight line.
- A mechanism at the right level: cache misses because of a layout, contention because of a hold time, faults because of lazy allocation.
- Your role, clearly: what you did versus what the team did.
If you have no story from a trading system
Use one from anywhere: a slow web service, a batch job, a game, a university project, a benchmark you investigated out of curiosity. The method is what is being tested. A small problem told with real measurements beats a large one told vaguely. If you truly have none, investigate something now: take a program, profile it, find a cause, fix it, and tell that story honestly as a self-directed investigation.
Preparing
Write the story down in the seven steps, with the numbers. Then list the follow-ups an interviewer could ask at each step and make sure you can answer them: how the tool works, why the mechanism costs what it does, what else you considered, what you would do differently now. Rehearse it aloud to about three minutes, leaving room for questions.
Check yourself
Question 1 of 3
What should come before the fix in a strong debugging story?
The interviewer is testing method. Evidence leading to a tested hypothesis is the core of the answer.
Question 2 of 3
Why mention a hypothesis that turned out wrong?
Real investigations involve ruling things out. Showing how evidence eliminated a plausible cause is persuasive.
Question 3 of 3
What makes a fix credible in the story?
Closing the loop with the original measurement, plus a test or alert, shows the problem is understood and controlled.
In the interview
Say your own answer out loud first, then open the one we would give.
Tell me about a hard performance problem you solved.
I would give the symptom with numbers and why it mattered, how I measured to locate it, the hypothesis with its mechanism and how I tested it, including anything I ruled out, the fix and its cost, the verification with the same measurement, and what I added to prevent a recurrence, in about three minutes, ready to go deeper on any step.
What if your first idea had been wrong?
I would say so. In the story, the first suspect was the allocator, but allocation counts on the path were zero after warm-up, which ruled it out and led to the lock measurements. Showing how evidence redirected the investigation is part of the method.
How do you make sure a performance fix stays fixed?
Add a test or benchmark that would fail on a regression of that specific mechanism, such as an allocation-count check, a lock hold-time limit or a percentile threshold on a replay, and put the relevant metric on a dashboard with an alert.