Statistical inference in code: standard errors, permutation tests, the Kalman filter

Standard errors are the point

Fitting a regression is the easy half. The question a research interviewer actually asks is whether the coefficient means anything, and that needs its standard error. Under the usual assumptions (errors independent, with constant variance σ²) the least-squares estimate β̂ = (XᵀX)⁻¹Xᵀy has covariance σ²(XᵀX)⁻¹. σ² is unknown, so it is estimated from the residuals:

sigma2 = resid @ resid / (n - p)      # p coefficients used up p degrees of freedom
se = np.sqrt(sigma2 * np.diag(XtX_inv))
t = beta / se

A t-statistic above about 2 in absolute value is the conventional bar, but in quant research it is a low one: try twenty signals and one will clear it by chance. Researchers who have been burned ask for 3 or more.

Computing (XᵀX)⁻¹ explicitly squares the condition number of X. A QR factorisation, X = QR, gives the same quantities without that loss: β̂ solves Rβ = Qᵀy, and the diagonal of (XᵀX)⁻¹ is the squared row norms of R⁻¹.

A test without a table

Two groups of numbers, and the question whether the difference in their means is real. A t-test needs assumptions about the distribution. A permutation test needs only one: if there is no difference, the group labels are arbitrary. So try every way of relabelling the pooled values into groups of the same sizes, compute the difference for each, and count how often it is at least as extreme as the one observed. That fraction is the p-value, exactly.

There are C(n, k) relabellings, 924 for two groups of 6, which is nothing for a computer. Beyond about 20 values, sample random relabellings instead. Compare differences in integer arithmetic when the data are integers: two relabellings with the same true difference must count the same, and floating-point rounding can split them.

The Kalman filter is Bayes' rule, repeated

A level drifts at random, and you see it only through noisy measurements: a fair price under a noisy quote, a position under a noisy sensor. Model the level as a random walk with step variance q, and each measurement as the level plus noise of variance r. The filter keeps a Gaussian belief, mean m and variance P, and alternates two steps:

• Predict. The level may have moved: P grows by q. The mean stays where it was.
• Update. A measurement y arrives. The gain K = P / (P + r) says how far to move towards it: m ← m + K(y − m), and the uncertainty shrinks, P ← (1 − K)P.

When the measurements are noisy (r large) the gain is small and the estimate barely moves. When the level is volatile (q large) the gain is large and the filter follows the data. The answer at every step is exactly the posterior you would get by conditioning on all measurements at once, which is how the problem below is checked.

Autocorrelation: the first thing to check

Before modelling a return series, look at how each value relates to the ones before it. The sample autocorrelation at lag k is

d = x - x.mean()
r_k = (d[:n - k] @ d[k:]) / (d @ d)

Both sums use the mean of the whole series, and the denominator is the same at every lag. That is the standard estimator; it is slightly biased towards zero but guarantees a valid correlation structure. For white noise, about 95% of the r_k fall within ±1.96/√n. Returns usually pass; squared returns usually do not, which is volatility clustering, and the reason GARCH models exist.

What interviewers listen for

• That σ² is estimated with n − p in the denominator, and why.
• What a t-statistic of 2 means after testing many signals, and what you would do about it.
• The assumption behind a permutation test, stated in one sentence.
• The Kalman gain explained in words, before any matrices.
• The difference between autocorrelation in returns and in squared returns.