How Backpropagation Works in Neural Networks

Neural networks learn by trial and error. When a network receives an input, it makes an initial prediction, checks how far off that prediction was from the correct answer, and updates its internal parameters to do better next time.

The engine that makes this update possible is backpropagation. This article explains how backpropagation works, walks through a complete numerical calculation of a loss derivative, covers batch processing and checkpoint management, and discusses what happens when you stack too many layers.

1. The Core Idea of Backpropagation

A neural network consists of layers of interconnected neurons. Each connection has a weight, and each neuron has a bias.

Learning happens in two phases:

1. The Forward Pass: The input data moves forward through the network. Each layer multiplies the inputs by weights, adds biases, and applies an activation function. At the end, the network produces a prediction.
2. The Backward Pass (Backpropagation): The network compares its prediction against the actual target using a loss function, which returns a single number showing the error. Backpropagation takes this error and runs backwards from the output layer to the input layer.

Using the chain rule from calculus, backpropagation calculates the partial derivative of the loss function with respect to every weight and bias. This derivative—often called the gradient—tells us two things:

• Which direction we should adjust the weight (increase or decrease).
• How sensitive the total error is to a change in that weight.

Once we have the gradients, an optimization algorithm like gradient descent updates the weights:

In this equation, is the learning rate, a small positive number that controls how big of a step we take during the update.

2. Calculating the Derivative of a Loss Function: A Step-by-Step Example

To understand the math, let us calculate the numbers by hand on a single neuron without an activation function.

The Initial Values

• Input value ():
• Target output ():
• Initial weight ():
• Initial bias ():
• Learning rate ():

Step 1: Forward Pass

First, we compute the output of the neuron ():

Since there is no activation function in this simple example, our prediction is:

Step 2: Compute the Loss

We will measure the error using Mean Squared Error (MSE), scaled by half to make the derivative simpler:

Substitute our values:

The loss is .

Step 3: Backward Pass Using the Chain Rule

We need to calculate , which represents how much a change in affects the loss . By the chain rule:

Let us break down each part:

1. Derivative of the loss with respect to prediction:

2. Derivative of prediction with respect to :

3. Derivative of with respect to weight :

Now, multiply all three terms together:

The derivative is . Because this value is positive, increasing the weight would increase the error. Therefore, the weight must decrease.

Step 4: Update the Weight

Now we apply gradient descent:

Verification

Let us run the forward pass again with the new weight ():

The error dropped from to . The single backpropagation step moved the parameter in the right direction.

3. Batch Processing and Checkpoint Retention

In practice, we do not train on one sample at a time, nor do we load an entire dataset of millions of rows into memory at once.

Mini-Batch Processing

Training data is divided into smaller groups called mini-batches (commonly 32, 64, or 128 samples).

Using mini-batches provides two practical advantages:

• Hardware efficiency: GPUs process parallel matrices much faster when handling batches of data rather than individual items.
• Stable weight updates: Updating after every single sample makes training erratic because individual noisy samples pull the weights in random directions. Averaging gradients across a batch provides a more reliable update step.

Checkpoint Retention: Training Elements and Keeping the Last

Training large models takes hours, days, or weeks. During this process, you need to save snapshots of the model’s weights (checkpoints) to disk.

A common pattern is:

1. Train on a set number of elements or steps ().
2. Save the current model state to disk.
3. Keep only the last snapshots in storage, deleting older ones automatically.

Here is a straightforward example in Python showing how this retention policy works:

import os
 
def train_and_checkpoint(dataset, batch_size=32, save_every_steps=500, max_to_keep=3):
    saved_checkpoints = []
    
    for step, batch in enumerate(dataset):
        loss = model.train_on_batch(batch)
        
        # Save a checkpoint every X steps
        if step > 0 and step % save_every_steps == 0:
            checkpoint_file = f"checkpoint_step_{step}.pt"
            model.save_weights(checkpoint_file)
            saved_checkpoints.append(checkpoint_file)
            
            # If we exceed the retention limit, remove the oldest file
            if len(saved_checkpoints) > max_to_keep:
                oldest_file = saved_checkpoints.pop(0)
                if os.path.exists(oldest_file):
                    os.remove(oldest_file)

This sliding-window retention is important for two reasons:

• Storage management: Deep learning checkpoints are large (often hundreds of megabytes or several gigabytes each). Keeping every checkpoint would quickly fill your hard drive.
• Fault recovery and selection: If training crashes or the machine restarts unexpectedly, you only lose work done since the most recent checkpoint. It also allows you to test the last few versions to see which one performed best on validation data before overfitting began.

4. The Purpose of Layers and the Cost of Excessive Depth

Why Networks Need Multiple Layers

A single-layer network can only solve linear problems. If you plot data points that cannot be separated by a straight line, a network without hidden layers will fail.

Hidden layers allow the network to learn representations at different levels of abstraction:

• First layers: Capture simple, low-level features such as lines, color borders, or basic audio frequencies.
• Middle layers: Combine low-level features into patterns, such as textures, corners, or phonemes.
• Deeper layers: Combine patterns into complex concepts, such as entire objects, faces, or words.

Every added layer with a non-linear activation function expands the mathematical shapes the network can model.

The Problems of Adding Too Many Layers

Making a network deeper does not always make it better. Stacking too many layers introduces several well-documented issues:

1. Vanishing Gradients

Backpropagation calculates gradients by multiplying derivatives across every layer using the chain rule. If these derivatives are fractions smaller than 1 (which happens frequently with older activation functions like Sigmoid or Tanh), multiplying them repeatedly through dozens of layers causes the gradient to shrink exponentially. By the time the signal reaches the earliest layers, the gradient is practically zero, and those layers stop learning entirely.

2. Exploding Gradients

The opposite happens if the weights or derivatives are consistently greater than 1. Repeated multiplication causes gradients to grow exponentially as they travel backward. The updates become huge, parameters overflow into NaN (Not a Number) values, and training destabilizes.

3. Overfitting

When a network has more layers and parameters than the task requires, it stops finding general rules. Instead, it starts memorizing the specific training examples, including their background noise. The model performs exceptionally well on training data but fails on real-world inputs.

4. Inference Latency and Hardware Costs

Every layer added requires memory to store its intermediate activations during the forward pass and additional compute operations during inference. In real-time applications (such as robotics, audio processing, or mobile apps), deeper models can be too slow to deliver timely results.

5. Optimization Degradation

Past a certain depth, standard feedforward networks become harder to train simply because the optimization landscape becomes too rough. Even without overfitting, training accuracy itself can begin to drop. Modern architectures overcome this specific issue using residual connections (skip connections) that let data bypass layers, but the architectural complexity remains a tradeoff.

Summary

Backpropagation provides the mathematical mechanism that lets a neural network learn from its errors. By calculating how sensitive the loss is to every weight, the network can systematically adjust itself.

However, good machine learning engineering requires balance: using batches for computational stability, managing checkpoints to save disk space, and keeping layer depth aligned with the complexity of your problem.