Inverted Dropout From Scratch: Forward and Backward in NumPy

Given activations X, a 0/1 mask of the same shape (1 keeps a unit), the drop probability p and the upstream gradient dout, return [out, dX] for inverted dropout in training mode: out = X ⊙ mask / (1 − p), and dX the gradient of the loss with respect to X.

Examples

Input X = [[1, 2], [3, 4]], mask = [[1, 0], [1, 1]], p = 0.5, dout = [[1, 1], [1, 1]]

Output [[[2, 0], [6, 8]], [[2, 0], [2, 2]]]

Kept units are doubled; the dropped one passes nothing forward or back.

Input X = [[5, -5, 5]], mask = [[1, 1, 1]], p = 0.2, dout = [[0.1, 0.2, 0.3]]

Output [[[6.25, -6.25, 6.25]], [[0.125, 0.25, 0.375]]]

Nothing dropped: everything is still scaled by 1/0.8.

Input X = [[2]], mask = [[0]], p = 0.9, dout = [[7]]

Output [[[0]], [[0]]]

A dropped unit: zero out, zero gradient.

Constraints

Any shape up to 10⁶ elements
0 ≤ p < 1
Checked to a relative tolerance of 10⁻¹²

Hints

Hint 1

The forward pass is one line: multiply by the mask, divide by 1 − p.

Hint 2

The operation is linear in X. Its derivative with respect to each element is mask / (1 − p).

Hint 3

So dX = dout ⊙ mask / (1 − p): the same mask, the same scale.

Where candidates go wrong

Scaling by (1 − p) instead of dividing by it.
Leaving out the scale in the backward pass.
Generating a fresh random mask for the backward pass instead of reusing the forward one.

The idea

Dividing the kept activations by 1 − p keeps each activation's expected value equal to its value without dropout, so the network sees the same scale in training and at inference, where dropout is simply switched off. The layer is elementwise and linear in X, so its Jacobian is diagonal with entries mask/(1 − p), and the backward pass multiplies the upstream gradient by exactly that.

Time O(size of X) · Space O(size of X)

Reference solution

import numpy as np

def dropout(X, mask, p, dout):
    keep = np.asarray(mask, dtype=float) / (1.0 - p)      # kept units are scaled up, dropped ones are 0
    return [np.asarray(X, dtype=float) * keep, np.asarray(dout, dtype=float) * keep]

Console⌘/Ctrl + Enter runs

Python runs in your browser with NumPy. The first run downloads the runtime, which takes a few seconds.

Why interviewers ask it

A short question with a single idea in it: where the scaling goes. Candidates who scale at test time, or forget the backward pass uses the same mask, reveal that they have used dropout without thinking about it.

The theory behind it is in the lesson Dropout, gradient clipping and learning-rate schedules.

Keep going

Easy8 min

Inverted dropout, forward and backward