ML coding · Numerical stability
Binary cross-entropy from logits
Given logits z and labels y (each 0 or 1) of the same length,
return the mean binary cross-entropy
L = −(1/n) Σ [ yi log σ(zi) + (1 − yi) log(1 − σ(zi)) ], σ(t) = 1 / (1 + e−t)
as a float. It must be accurate for |z| up to 10⁴, where σ(z) rounds to exactly 0 or 1.
Examples
Input z = [0], y = [1]
Output 0.6931
σ(0) = ½ and −log ½ = log 2.
Input z = [2, -1], y = [1, 0]
Output 0.2201
Both predictions lean the right way.
Input z = [-800], y = [1]
Output 800
σ(−800) underflows to 0. The loss is still exactly 800.
Constraints
- 1 ≤ n ≤ 10⁵
- |z[i]| ≤ 10⁴
- y[i] ∈ {0, 1}
- Checked to a relative tolerance of 10⁻⁹
Hints
Hint 1
Write log σ(z) and log(1 − σ(z)) in terms of log(1 + e^(−z)) and log(1 + e^(z)). Then combine the two terms of the loss into one expression.
Hint 2
You should reach ℓ = log(1 + eᶻ) − y·z. That is still unsafe for large z. How does log(1 + eᶻ) behave when z is large?
Hint 3
log(1 + eᶻ) = max(z, 0) + log(1 + e^(−|z|)). Every exponent is now ≤ 0. Use np.log1p for the last term.
Where candidates go wrong
- Computing σ(z) first and then the log: σ(−800) is exactly 0 and log 0 = −inf.
- Clipping probabilities to [1e−7, 1 − 1e−7]. The loss of a badly wrong prediction is then capped at about 16, and the gradient stops pushing.
- Using np.log(1 + np.exp(-abs(z))) instead of np.log1p: for |z| above about 37, 1 + e^(−|z|) rounds to 1 and the term is lost.
The idea
Since 1 − σ(z) = σ(−z) and log σ(z) = −log(1 + e−z), the per-example loss is
ℓ = y log(1 + e−z) + (1 − y) log(1 + ez) = log(1 + ez) − y z,
using log(1 + e−z) = log(1 + ez) − z. The function log(1 + ez) is softplus, and it overflows for large z. Splitting on the sign of z,
log(1 + ez) = max(z, 0) + log(1 + e−|z|),
so ℓ = max(z, 0) − z y + log1p(e−|z|). The exponent is never positive, log1p keeps full precision when e−|z| is tiny, and no branch is needed. This is the formula behind PyTorch's BCEWithLogitsLoss and TensorFlow's sigmoid_cross_entropy_with_logits.
Time O(n) · Space O(n)
Reference solution
import numpy as np
def bce_with_logits(z, y):
z = np.asarray(z, dtype=float)
y = np.asarray(y, dtype=float)
# log(1 + e^z) - y z, with softplus written so that no exponent is positive
return float(np.mean(np.maximum(z, 0) - z * y + np.log1p(np.exp(-np.abs(z)))))Follow-ups an interviewer asks
- The gradient with respect to z is σ(z) − y. Write a sigmoid that never overflows to compute it.
- Add a positive-class weight w: ℓ = (1 − y) z + (1 + (w − 1) y) · softplus(−z). Derive it.
- Focal loss: multiply each term by (1 − p_t)^γ. Keep it in log space.
Python runs in your browser with NumPy. The first run downloads the runtime, which takes a few seconds.
Why interviewers ask it
Logistic regression, every binary classifier and every GAN discriminator is trained on this loss. Writing it as sigmoid followed by log is the obvious answer and it is wrong in floating point. The interviewer wants to see the algebra that removes the problem, not a clip.
The theory behind it is in the lesson Numerically stable softmax and log-sum-exp.
Keep going
Softmax that never overflows
Write a softmax that returns the right probabilities for logits of 1000 or −10000.
Open the problem → Medium15 minCross-entropy loss and its gradient from logits
Mean cross-entropy over a batch of logits, computed in log space, with the gradient p − y.
Open the problem → Medium15 minBinary cross-entropy from logits
Derive the overflow-free form of binary cross-entropy, max(z, 0) − zy + log(1 + e^−|z|), and implement it in NumPy.
Open the problem →