Expected Calibration Error (ECE) From Scratch in NumPy

Given predicted probabilities p (each in [0, 1]), binary labels y and a number of bins m, put each prediction in bin b = min(⌊p·m⌋, m − 1), so the bins are [0, 1/m), [1/m, 2/m), …, with the last bin also containing 1. For every non-empty bin, take the absolute difference between the mean label and the mean prediction, weight it by the bin's share of all predictions, and return the sum.

Examples

Input p = [0.1, 0.2, 0.8, 0.9], y = [0, 0, 1, 1], m = 2

Output 0.15

Two bins of two predictions each.

Input p = [1, 1, 0], y = [1, 1, 0], m = 10

Output 0

Perfectly confident and correct: ECE 0. The value 1.0 goes in the last bin.

Constraints

1 ≤ n ≤ 10⁶, 1 ≤ m ≤ 100
Checked to a relative tolerance of 10⁻¹²

Hints

Hint 1

Compute the bin index for every prediction at once with np.floor(p * m), then clip to m − 1.

Hint 2

np.bincount(b, minlength=m) gives counts; with weights= it gives the sums of labels and of predictions per bin.

Hint 3

Skip bins with a count of zero; their weight is zero anyway, but their means are undefined.

Where candidates go wrong

Letting p = 1.0 create bin m, one past the end.
Averaging the per-bin gaps without weighting by bin size.
Dividing by zero for empty bins.

The idea

Assign bins with one vectorised floor and a clip. Accumulate counts, label sums and prediction sums per bin with np.bincount. For non-empty bins, the contribution is count/n × |label sum − prediction sum| / count, which simplifies to |label sum − prediction sum| / n.

Time O(n + m) · Space O(m)

Reference solution

import numpy as np

def expected_calibration_error(p, y, m):
    p = np.asarray(p, dtype=float)
    y = np.asarray(y, dtype=float)
    b = np.minimum(np.floor(p * m).astype(int), m - 1)       # 1.0 goes in the last bin
    count = np.bincount(b, minlength=m)
    label_sum = np.bincount(b, weights=y, minlength=m)
    pred_sum = np.bincount(b, weights=p, minlength=m)
    used = count > 0
    return float(np.abs(label_sum[used] - pred_sum[used]).sum() / p.size)

Console⌘/Ctrl + Enter runs

Python runs in your browser with NumPy. The first run downloads the runtime, which takes a few seconds.

Why interviewers ask it

Asked wherever probabilities are used rather than just ranks: credit, insurance, ad bidding, trading signals. The implementation is short; getting the bin edges and the weighting right is the test.

The theory behind it is in the lesson NDCG, calibration error and macro-F1 from scratch.

Keep going

Easy10 min

Expected calibration error