Learning Rate

The learning rate is a setting that controls how big a step a model takes when updating its parameters during training. It is one of the most important hyperparameters to tune.

Data & AI

A learning rate that's too high makes training overshoot and bounce around without settling; too low and training crawls or gets stuck. The right value lets the model converge smoothly toward a good solution.

Picture walking downhill in fog: huge strides might leap right over the valley, while tiny shuffles take forever. The learning rate is the size of your stride in gradient descent.

Examples

Python

import torch

optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
print("learning rate:", optimizer.defaults["lr"])

Output:

learning rate: 0.01

Learn it in a lesson

See also

Back to the Dictionary