Machine learning · A worked visual guide

Gradient descent:
finding the way down.

I want to understand how a model improves, not just memorize an equation. So I’ll start with five observations, build a line that predicts them, measure how wrong it is, and update that line one careful step at a time.

Gradient descent
A little journey downhill
Conceptual view · different starting guesses move downhill
Worked numerical example Batch gradient descent Interactive experiment
What I should be able to do by the end: calculate a loss, derive both gradients, perform a real parameter update, explain the learning rate, and read the same update in the graphs.

01 / Start with actual data

What line are we trying to learn?

Imagine I collected five observations. Each one records how many hours a student studied and the score they received. The points do not sit perfectly on one line, so I need a model that captures the overall pattern rather than passing through every point.

ObservationStudy hoursObserved score
1135
2242
3355
4461
5572

For each study-hour value, the model produces a prediction. I’ll use a straight-line model with two adjustable parameters:

prediction = b + m x

To keep the numbers easy to work with, I centre and scale study hours:

x = (study hours − 3) / 2

So the five input values become −1, −0.5, 0, 0.5, 1. In this version of the model, b is the predicted score at 3 study hours, and m is how much the prediction changes when study time increases by 2 hours. This is still an ordinary straight line; centring the input just makes the arithmetic cleaner.

I’ll begin with a deliberately poor guess: b = 20 and m = 5. That gives a low, fairly flat line. The model is underpredicting every observation.

HoursScaled xActual yInitial predictionError (prediction − actual)
1−13515−20
2−0.54217.5−24.5
305520−35
40.56122.5−38.5
517225−47

Important: these five values are a small teaching dataset, not evidence that studying a particular number of hours causes a particular score. We are using them to understand the optimization mechanics.

02 / Give “wrong” a number

How do I measure whether the line is bad?

Looking at a graph is useful, but an algorithm needs a numerical score. I’ll compare every prediction with its observed score, square each error so positive and negative errors do not cancel, and average the result.

L(b,m) = (1 / 2n) Σ [predictionᵢ − yᵢ]²

Here n = 5, and the prediction for observation i is b + m xᵢ. The factor of 1/2 is included because it cancels the 2 that appears when we differentiate a squared error. It does not change which parameters give the smallest loss.

For my starting line, the errors are −20, −24.5, −35, −38.5 and −47. Squaring and adding them gives:

400 + 600.25 + 1,225 + 1,482.25 + 2,209 = 5,916.5
L(20, 5) = 5,916.5 / (2 × 5) = 591.65

That number is our starting point. Gradient descent does not know what a “good-looking line” is. It only knows the loss function I gave it, and it tries to make that number smaller.

03 / Find the direction

What does the gradient actually tell me?

The loss depends on two parameters, b and m. So there are two questions: what happens to the loss if I nudge b, and what happens if I nudge m? Those two rates of change make the gradient.

Write the error for one observation as eᵢ = b + m xᵢ − yᵢ. Our loss is the average of the squared errors, with the factor of 1/2:

L(b,m) = (1 / 2n) Σ eᵢ²

Differentiate with respect to b. The derivative of eᵢ with respect to b is 1, so the 2 from the square cancels the 1/2:

∂L / ∂b = (1 / n) Σ eᵢ

Now differentiate with respect to m. Because m is multiplied by xᵢ, the derivative of eᵢ with respect to m is xᵢ:

∂L / ∂m = (1 / n) Σ (eᵢ xᵢ)

At the starting guess, the errors are the five values from the table and the scaled inputs are −1, −0.5, 0, 0.5, 1. Therefore:

∂L / ∂b = (−20 − 24.5 − 35 − 38.5 − 47) / 5 = −33
∂L / ∂m = [20 + 12.25 + 0 − 19.25 − 47] / 5 = −6.8

So the gradient is (−33, −6.8). It points in the direction where the loss increases fastest. Both components are negative, which tells me that increasing b and increasing m will reduce the loss from this starting point. The downhill direction is the opposite of the gradient.

Gradient · uphill(−33, −6.8)Direction of steepest increase in loss.
Negative gradient · downhill(+33, +6.8)Direction that locally decreases the loss fastest.

04 / Make one real update

Turn the gradient into a better line

The learning rate, usually written as α (alpha), controls how much of the gradient I follow. I’ll choose α = 0.2 for this first worked step.

new parameter = old parameter − α × gradient

Update b and m using the gradients we just calculated. Both updates use the same old parameter values; they happen together, not one after the other.

b new = 20 − 0.2(−33) = 26.6
m new = 5 − 0.2(−6.8) = 6.36

Now the model is prediction = 26.6 + 6.36x. Its predictions are 20.24, 23.42, 26.6, 29.78 and 32.96. The new loss is 386.8444, down from 591.65. One update has made the line better according to our chosen loss function.

Notice what happened: we did not tell the model “raise the line by 6.6” or “make the slope 6.36”. The gradients and learning rate calculated those changes from the data.

Starting loss591.65
After one update386.84
Loss reduction34.61%

05 / Repeat, don't guess

What happens over several updates?

Every step repeats the same routine: calculate predictions, calculate errors, calculate both gradients, update both parameters, and measure the new loss. With α = 0.2, the first few updates are:

StepbmLoss
0 · Start20.0005.000591.650
126.6006.360386.844
231.8807.584254.275
336.1048.686168.221
439.4839.677112.167
542.18710.56975.498

The values in this table are calculated from the same five observations and the same gradient formulas shown above. The interactive lab below runs these actual updates; it does not move along a precomputed path to the answer.

06 / Now make the idea visible

Watch both parameters learn

The left plot shows the loss landscape over the two parameters b and m. Each contour is a set of parameter pairs with the same loss. The right plot shows the line those parameters produce against the five observations. When I take a step, both views update from the same calculation.

Batch gradient descent · live experiment Choose a point on the contour map, then update b and m using the real gradients.
Step 0
Current loss591.65
Baseline b20.00
Slope m5.00
Gradient (db, dm)−33.00, −6.80

1. Loss landscape · L(b,m)Click to choose b and m

The dot is the current parameter pair. Contours closer to the centre have lower loss. The teal arrow shows the negative-gradient direction: the way to step downhill.

2. Prediction line · data fitb = 20.0 · m = 5.0

Teal dots are the observations. The orange line is the model's current prediction. Dashed vertical gaps are prediction errors.

For this scaled example, values below 2 converge; values near 2 can oscillate, and values at 2 do not converge and values above 2 diverge.
Start at b = 20 and m = 5. Both gradients are negative, so subtracting them moves both parameters upward and reduces the loss.
StepbmLoss
020.0005.000591.650

The contour plot is calculated from the same loss function as the line plot. It is not a decorative valley or a separate animation: each point represents a real pair of model parameters.

07 / Why the learning rate matters

Why not take the biggest step possible?

The gradient tells me which way is downhill, but the learning rate decides how far I move in that direction. A direction can be correct while the step is too large.

Too small: the updates reduce the loss, but progress is slow. I may need many iterations.

Reasonable: the loss falls steadily and the parameter pair approaches the minimum.

Too large: an update can cross the bottom of the valley. It may bounce from one side to the other. If it is large enough, the loss increases and the process diverges.

Because we centred and scaled x, the two curvature values for this particular loss surface are 1 and 0.5. That makes a constant learning rate between 0 and 2 stable for this example. Near 2, one parameter direction can oscillate strongly. At exactly 2, the b update does not shrink; above 2, it diverges. This threshold is specific to this model and scaling—not a universal learning-rate rule.

08 / Put the steps into code

The algorithm is a small loop

This Python code performs exactly the calculation used by the interactive lab. It uses every observation to calculate each gradient, so this is batch gradient descent.

import numpy as np

hours = np.array([1, 2, 3, 4, 5], dtype=float)
y = np.array([35, 42, 55, 61, 72], dtype=float)

# Centre and scale the input: [-1, -0.5, 0, 0.5, 1]
x = (hours - 3) / 2

b = 20.0
m = 5.0
learning_rate = 0.20

for step in range(100):
    predictions = b + m * x
    errors = predictions - y

    loss = np.mean(errors ** 2) / 2
    db = np.mean(errors)
    dm = np.mean(errors * x)

    # Both updates use gradients from the same old parameters.
    b = b - learning_rate * db
    m = m - learning_rate * dm

print("b:", b)
print("m:", m)

The order matters: first calculate both gradients from the current predictions, then update both parameters. If I update b before calculating the gradient for m, I would accidentally mix two different parameter states.

09 / Know what the result means

Where does the algorithm end up?

For these five observations, the minimum-loss line is approximately b = 53 and m = 18.6. Remember what those parameters mean: 53 is the predicted score at 3 study hours, and 18.6 is the predicted change across 2 additional study hours.

In the original, unscaled study-hours variable, this same line is approximately score = 25.1 + 9.3 × hours. The minimum loss is about 0.91, not zero. That is expected: the five observations do not lie perfectly on a straight line. The goal is not to force every point onto the line; it is to find the parameters that minimize the chosen squared-error loss.

This example is linear regression, whose squared-error loss is a convex bowl with one global minimum. Neural networks can have much more complicated loss surfaces, but the central operation is the same: calculate a gradient and adjust parameters in the negative-gradient direction.

One final caution: reducing training loss only means the model fits these five examples better. It does not prove that study hours cause higher scores or that the model will predict new students accurately. For that, we would need more data and a separate test set.

Best baseline b53.0
Best slope m18.6
Minimum loss0.91

10 / Continue learning

Good next explanations

These are worth opening after working through the numbers above. The first develops the mathematical treatment; the second builds a visual intuition for gradient descent in machine learning.

Rohit Sahera · Machine learning notes
All interactive calculations run in your browser. No data is sent anywhere.