Dropout, gradient clipping and learning-rate schedules

Dropout, and why it is inverted

Dropout zeroes each activation with probability p during training. If nothing else changed, the expected size of every activation would fall to (1 − p) of its value, and the network would see larger inputs at test time, when dropout is off, than it was trained on. The original paper fixed this by scaling the weights down at test time. Every modern library instead scales the surviving activations up by 1/(1 − p) during training, so that inference needs no change at all. That is inverted dropout, and it is what torch.nn.Dropout does.

The backward pass is the same mask and the same scale applied to the incoming gradient: a dropped unit passes no gradient back, and a kept one passes it scaled by 1/(1 − p).

Gradient clipping by global norm

A single bad batch, or a recurrent network unrolled over many steps, can produce a gradient thousands of times larger than usual, and one update along it can undo days of training. Clipping caps the step. The standard version treats every parameter's gradient as one long vector, measures its Euclidean norm, and if that exceeds a limit, scales all the gradients by the same factor:

total = np.sqrt(sum((g ** 2).sum() for g in grads))
coef = max_norm / (total + 1e-6)
if coef < 1:
    grads = [g * coef for g in grads]

Scaling every tensor by one factor keeps the direction of the update and changes only its length. Clipping each tensor separately, or each element, changes the direction, which is a different and usually worse algorithm.

Learning-rate schedules

Large models are trained with a learning rate that rises from near zero over a warm-up period and then decays. The warm-up matters for Adam in particular: in the first steps its second-moment estimate rests on very few gradients, and a full-size step can push the weights somewhere they never recover from. The decay matters at the end, when smaller steps settle into a minimum instead of bouncing around it. Cosine decay is the common choice because it falls slowly at first, quickly in the middle and slowly again at the end:

lr(t) = lr_min + ½(lr_max − lr_min)(1 + cos(π · progress))

with progress running from 0 at the end of warm-up to 1 at the last step.

What interviewers listen for

• Why dropout is scaled at training time and not at test time.
• That clipping by global norm preserves the direction of the gradient.
• Why warm-up helps Adam, in one sentence about the second moment.
• Off-by-one errors at the boundaries of a schedule, which are where most bugs are.