Skip to content
Work Practice Products Coding interviews Machine learning coding Quant developer HFT course Blog Method Why me About Enquire Book a call

Quant developer HFT course · Module 8: Measuring latency

Telling a debugging story in an interview

Lesson · 10 min

The question

Tell me about the hardest performance problem you have debugged.

What the question tests

This is a behavioural question with a technical core. The interviewer wants to know whether you investigate methodically or guess, whether you measure, whether you understand the mechanism you found, and whether you made sure it did not come back. They will interrupt with follow-ups, so every claim in your story should be one you can go one level deeper on.

A structure that works

symptomwhat, when, how bigmeasurewhere the time goeshypothesisone mechanismtest itpredict a counterfixpreventtest, alert
The shape of a good debugging story. Each step has evidence, and the fix is confirmed by the measurement that found the problem.
  1. The symptom, concretely: what was slow, how you knew, how big, how often, and why it mattered. "p99 of order acknowledgement handling went from 6 to 40 µs after a release, on busy days only."
  2. Measurement first. What you measured to locate it, and what it showed. "Stage timestamps showed the time was in the risk check; off-CPU profiling showed threads blocked in a mutex."
  3. A hypothesis with a mechanism. "The new position report took the risk lock and formatted a report under it."
  4. A test of the hypothesis, ideally one that predicted a number. "If that was it, lock wait time at the risk call site should match the report's duration and occur only when reports ran. It did."
  5. The fix and its trade-off. "We took a snapshot of positions under the lock and formatted outside it. It cost a copy of a few kilobytes per report."
  6. Verification: the same measurement that found it, now healthy. "p99 back to 6 µs; wait time at that site near zero."
  7. Prevention: what changed so it would not recur. "A test that fails if any lock on the order path is held longer than a threshold, and the hold-time histogram on the dashboard."

Good and bad versions

Weak: "The system was slow, so I profiled it, found a hot function and optimised it, and it was three times faster." No numbers you can defend, no mechanism, no explanation of why the profile pointed there, and the interviewer's first follow-up ("what made it hot?") has no answer.

Strong stories have a few features in common:

  • Numbers with units and percentiles, and how they were measured.
  • At least one wrong hypothesis, and how evidence ruled it out. Real investigations have them, and saying so is more credible than a straight line.
  • A mechanism at the right level: cache misses because of a layout, contention because of a hold time, faults because of lazy allocation.
  • Your role, clearly: what you did versus what the team did.

If you have no story from a trading system

Use one from anywhere: a slow web service, a batch job, a game, a university project, a benchmark you investigated out of curiosity. The method is what is being tested. A small problem told with real measurements beats a large one told vaguely. If you truly have none, investigate something now: take a program, profile it, find a cause, fix it, and tell that story honestly as a self-directed investigation.

Preparing

Write the story down in the seven steps, with the numbers. Then list the follow-ups an interviewer could ask at each step and make sure you can answer them: how the tool works, why the mechanism costs what it does, what else you considered, what you would do differently now. Rehearse it aloud to about three minutes, leaving room for questions.

Check yourself

Question 1 of 3

What should come before the fix in a strong debugging story?

Question 2 of 3

Why mention a hypothesis that turned out wrong?

Question 3 of 3

What makes a fix credible in the story?

In the interview

Say your own answer out loud first, then open the one we would give.

Tell me about a hard performance problem you solved.

I would give the symptom with numbers and why it mattered, how I measured to locate it, the hypothesis with its mechanism and how I tested it, including anything I ruled out, the fix and its cost, the verification with the same measurement, and what I added to prevent a recurrence, in about three minutes, ready to go deeper on any step.

What if your first idea had been wrong?

I would say so. In the story, the first suspect was the allocator, but allocation counts on the path were zero after warm-up, which ruled it out and led to the lock measurements. Showing how evidence redirected the investigation is part of the method.

How do you make sure a performance fix stays fixed?

Add a test or benchmark that would fail on a regression of that specific mechanism, such as an allocation-count check, a lock hold-time limit or a percentile threshold on a replay, and put the relevant metric on a dashboard with an alert.