Gradient Descent
Gradient descent is the optimization algorithm that trains most machine learning models. It repeatedly adjusts the model's parameters in the direction that reduces the loss (error), step by step.
Data & AI
At each step it computes the gradient — the slope of the loss with respect to each parameter — and moves a little 'downhill' against that slope. The size of each step is controlled by the learning rate.
Imagine standing on a foggy hillside trying to reach the lowest valley. You can't see far, so you feel the ground for the steepest downhill direction and take a small step that way, over and over. That's gradient descent minimizing the loss.
Examples
Python
# Minimize f(x) = x^2 (gradient is 2x)
x, lr = 5.0, 0.1
for _ in range(3):
x = x - lr * (2 * x)
print(round(x, 2))Output:
4.0
3.2
2.56