Statistics coding questions: variance, AUC and rank correlation

Why the textbook formula fails

The variance is E[x²] − E[x]². As algebra it is exact. As floating point it is a subtraction of two large, nearly equal numbers, and the difference is where all the information lives. Take four prices near 10⁹: their squares are near 10¹⁸, where adjacent float64 values are 128 apart. The true variance is 30. The one-pass textbook formula returns whatever is left after the rounding, which can be 0, negative, or several hundred.

The two-pass method, mean first and then the mean of squared deviations, is accurate but needs the data twice. When the data arrives as a stream, from a market feed or a training loop, there is no second pass.

Welford's algorithm

Keep the count n, the running mean, and M₂, the sum of squared deviations from the current mean. For each new value x:

n += 1
delta = x - mean
mean += delta / n
m2 += delta * (x - mean)     # old deviation times new deviation

The sample variance is M₂/(n − 1). Every quantity stays on the scale of the deviations rather than the raw values, so nothing cancels catastrophically. The same idea merges two partial results, which is how variance is computed in parallel across machines or GPU threads.

ROC AUC is a ranking statistic

The area under the ROC curve equals the probability that a randomly chosen positive is scored higher than a randomly chosen negative, counting ties as one half. That is the Mann–Whitney U statistic, divided by the number of pairs. Read that way, the AUC needs no curve at all:

• Rank all scores together, giving tied scores the average of the ranks they span.
• Add up the positives' ranks. Subtract the smallest that sum could be, n₊(n₊ + 1)/2.
• Divide by n₊n₋.

That is O(n log n) for the sort, against O(n₊n₋) for comparing every pair. The tie handling is not a detail: classifiers that output rounded scores, or a handful of distinct values, have ties everywhere, and an implementation that ignores them is biased by the order of the input.

Rank correlation

Spearman's correlation is Pearson's correlation applied to ranks. It measures whether two variables move together monotonically, without caring how. In quant research this is the information coefficient: the rank correlation between a signal today and returns tomorrow, the number most alpha research is judged on. It is preferred to Pearson because a single extreme return cannot dominate it.

The shortcut 1 − 6Σd²/(n(n² − 1)) is exact only when there are no ties. With ties, use average ranks and compute Pearson on them directly.

What interviewers listen for

• Knowing that E[x²] − E[x]² is unstable, and why.
• The probabilistic meaning of AUC, said before any code.
• Average ranks for ties, in both AUC and Spearman.
• For quant roles: the link between Spearman and the information coefficient.