Gradient Descent

Gradient descent is the optimization algorithm that trains most machine learning models. It repeatedly adjusts the model's parameters in the direction that reduces the loss (error), step by step.

Data & AI

At each step it computes the gradient — the slope of the loss with respect to each parameter — and moves a little 'downhill' against that slope. The size of each step is controlled by the learning rate.

Imagine standing on a foggy hillside trying to reach the lowest valley. You can't see far, so you feel the ground for the steepest downhill direction and take a small step that way, over and over. That's gradient descent minimizing the loss.

Examples

Python

# Minimize f(x) = x^2 (gradient is 2x)
x, lr = 5.0, 0.1
for _ in range(3):
    x = x - lr * (2 * x)
    print(round(x, 2))

Output:

4.0
3.2
2.56

Learn it in a lesson

See also

Back to the Dictionary