Deep Learning Glossary

The Neuron

• Neuron (unit)

The atom of a network. It multiplies each input by a weight, sums them, adds a bias, then applies a nonlinearity. Stack many in parallel to make a layer; stack layers to make a deep network.

Everything else is bookkeeping around this one operation, repeated billions of times.

w, b Weights & bias

The learnable parameters. A weight scales how much an input matters (large = influential, negative = inverting); the bias shifts the output up or down independently of the inputs.

Training is the search for good values of w and b — nothing else changes.

z Pre-activation

The weighted sum plus bias, before the nonlinearity. A single number (per unit) that packs all the inputs into one linear score. Sometimes called the "logit" or "net input".

Everything up to z is linear; the activation that follows is what makes networks expressive.

z Logit

The output layer's pre-activation: the raw, unbounded score for each class, before softmax (or sigmoid) turns it into a probability. Same kind of object as any z — the name is simply reserved for the final layer, the numbers a classification loss is computed from.

The name comes from log-odds: the logit function log(p/(1−p)) is the inverse of the sigmoid, mapping a probability in (0,1) back to an unbounded score.
In PyTorch, CrossEntropyLoss and BCEWithLogitsLoss expect logits, not probabilities — applying softmax yourself first is a classic bug.

a Activation

The unit's output after the nonlinearity, a = φ(z). This value is what feeds the next layer.

In the output layer, a is the prediction; in hidden layers, it is a learned feature.

Activation Functions

φ Activation function

A nonlinear function applied elementwise to z. It is what lets a network approximate curved, complex functions — without it, stacking layers collapses to a single linear map.

Common choices: ReLU (hidden layers), sigmoid/softmax (outputs), tanh, GELU.

σ Sigmoid

Squashes any real number into (0, 1) — an S-shaped curve. Handy for probabilities, with an especially clean derivative used in backprop.

It saturates: for large |z| the slope → 0, so gradients vanish — the reason ReLU took over hidden layers.

R ReLU

Rectified Linear Unit: pass positives through unchanged, clamp negatives to zero. Trivially cheap, and its gradient is either 0 or 1 — so it doesn't shrink gradients the way sigmoid does.

Units stuck at z < 0 output 0 with zero gradient — "dead" ReLUs; leaky variants fix this.

S Softmax

Turns a vector of raw scores (logits) into a probability distribution: exponentiate each, then divide by the total. Outputs are all positive and sum to 1. The multi-class generalisation of the sigmoid, used at the output of a classifier.

Exponentiating exaggerates differences — the largest logit dominates. Pairs naturally with cross-entropy loss, whose gradient then simplifies to (prediction − target).

Training — Forward & Backward

→ Forward pass

Run the network input → output, left to right: at each layer compute z then a, until you reach the prediction and the loss. Pure evaluation — no learning happens yet.

Each unit caches the values it computed; the backward pass reuses them.

L Loss function

A single number measuring how wrong the prediction is versus the target. Training minimises it. Squared error for regression, cross-entropy for classification.

The loss is the only thing the whole network is optimised against.

H Cross-entropy loss

The classification loss: how surprised the model is by the correct answer. With one-hot labels the sum collapses to just the negative log-probability assigned to the true class — so a confident mistake costs −log(p) → ∞.

Paired with softmax/sigmoid the gradient is simply ∂L/∂z = p − y: the a(1−a) factor cancels exactly, so gradients stay large when the model is confidently wrong — the reason classification uses this instead of squared error.
In PyTorch, CrossEntropyLoss applies softmax itself — feed it logits, not probabilities.

∇L Gradient

The vector of partial derivatives of L with respect to every parameter — the direction of steepest increase of the loss. Step the opposite way to reduce it.

∂L/∂w tells each weight which way, and how strongly, to move.

← Backward pass (backpropagation)

Compute the gradient efficiently by walking the graph right to left: each node multiplies the incoming (upstream) gradient by its own local derivative and passes it back. One local rule per node, reusing the forward values.

Same cost as one forward pass — the reason training huge models is feasible.

∘ Chain rule

The calculus that powers backprop: the derivative of a composition is the product of local derivatives along the path. Backprop is just the chain rule applied over a graph.

"Upstream gradient × local derivative", chained from the loss back to each weight.

↓ Gradient descent

The update step: nudge each parameter a little in the direction that lowers the loss, i.e. opposite the gradient. Repeat over many batches and the loss rolls downhill.

"Stochastic" GD estimates the gradient from a mini-batch rather than the whole dataset.

η Learning rate

The size of each gradient-descent step. Too large and the loss overshoots or diverges; too small and training crawls. The single most important hyperparameter to tune.

Schedules and adaptive optimisers (Adam) adjust it over the course of training.

The Training Loop

B Batch (mini-batch)

A small subset of the training data used to estimate the gradient for one update. Averaging over a batch is cheaper than using the whole dataset and less noisy than using a single example.

Typical sizes: 32–512. Bigger batches give a smoother gradient estimate but cost more memory per step.

t Iteration (step)

One parameter update: forward pass on a batch, backward pass, then nudge the weights once. The atomic unit of training.

One iteration ≠ one epoch. An iteration consumes a single batch.

↻ Epoch

One complete pass through the entire training set — every example seen exactly once. Purely a bookkeeping unit: nothing special happens at the boundary. The data is reshuffled each epoch so the batches differ.

50,000 examples with batch size 100 → 500 iterations = 1 epoch.
Too few epochs underfits; too many overfits — what early stopping watches for.

SGD Stochastic gradient descent

Gradient descent where each step's gradient is estimated from a random mini-batch rather than the full dataset. The steps are noisy, but each is far cheaper — and the noise itself helps escape poor minima.

The "stochastic" is the randomness of which examples land in each batch. Adam and friends build on this with per-parameter step sizes.

Convolutional Networks

∗ Convolution

A small kernel slides across the input and takes a dot product at each position, writing one number into the output. It replaces the dense "every output to every pixel" layer with a local, reusable operation that respects spatial structure.

Strictly this is cross-correlation (no kernel flip), which is what deep-learning libraries actually compute — the name "convolution" stuck anyway.

K Kernel (filter)

The small grid of learned weights that does the detecting. A kernel spans the full input depth, so a 3×3 kernel on a C-channel input is really 3×3×C weights (plus a bias) — and it produces exactly one output channel.

The nine (or 3×3×C) weights are not hand-designed — gradient descent learns which little patterns are worth detecting.

C Channel

The depth axis of the tensor — how many 2D feature maps are stacked together. An RGB image has 3 input channels (R, G, B); a hidden layer has as many channels as it has filters. One channel is one 2D slice of that stack.

Key asymmetry: a filter spans channels (covers all of them at once) but slides over space. Output channels = number of filters — an input axis you receive, an output axis you choose.

▦ Feature map

The 2D output of a single filter — one channel of a conv layer's output. Each cell records how strongly that filter's pattern matched the input at that location. A layer with K filters produces K feature maps.

"Feature map", "activation map", and "output channel" are the same object. Bright where the feature is present, dark where it is not.

↓ Pooling

Downsampling a feature map by summarising each small window with one number — usually its maximum. It halves the resolution and, by reporting whether a feature was present rather than exactly where, adds a little translation invariance.

Convolution is equivariant (a feature moves with the input); pooling adds invariance (small shifts stop mattering). Pooling has no learnable weights.

▣ Receptive field

The region of the input a given unit can "see". A single 3×3 conv sees a 3×3 patch; stack another and each unit now depends on a 5×5 patch. Stride and pooling enlarge it faster.

Layer by layer the receptive field grows, so deep units respond to large, abstract structure — edges become textures become parts become objects.