NDCG, calibration error and macro-F1 from scratch

Ranking: NDCG

For a ranked list, items near the top matter more. Discounted cumulative gain adds each item's gain, 2^rel − 1, divided by log₂ of its position plus one. Dividing by the gain of the ideal ordering gives NDCG, between 0 and 1, comparable across queries with different numbers of relevant items. Two choices change the number: the gain (2^rel − 1 or rel itself) and whether the ideal ranking is built from the top k returned items or from all items. State both.

Calibration: expected calibration error

A model is calibrated if, among the cases where it says 0.8, about 80% are positive. Expected calibration error groups predictions into bins by predicted probability and averages, weighted by bin size, the gap between each bin's mean prediction and its observed frequency. Accuracy and AUC say nothing about this; a model can rank perfectly and still be badly calibrated, which matters whenever its probabilities are used to price, size or threshold.

ECE depends on the binning. Equal-width bins leave some bins nearly empty when predictions cluster; equal-mass bins avoid that. Whatever the choice, the edges must be stated precisely, including which bin a prediction of exactly 1.0 falls into.

Many classes: macro-F1

F1 for one class is 2TP / (2TP + FP + FN). Macro-F1 averages it over classes, so a rare class counts as much as a common one; micro-F1 pools the counts first and, for single-label problems, equals accuracy. Which classes enter the average (those in the labels, the predictions, or both) is a real decision: a class that is predicted but never true has F1 of zero and pulls the average down.

What interviewers listen for

• Each metric's convention stated before it is computed.
• The zero-denominator cases handled: no relevant items, an empty bin, a class never predicted.
• An explanation of when macro and micro averages disagree, and which one the business cares about.