Learning Rate
The learning rate is a setting that controls how big a step a model takes when updating its parameters during training. It is one of the most important hyperparameters to tune.
Data & AI
A learning rate that's too high makes training overshoot and bounce around without settling; too low and training crawls or gets stuck. The right value lets the model converge smoothly toward a good solution.
Picture walking downhill in fog: huge strides might leap right over the valley, while tiny shuffles take forever. The learning rate is the size of your stride in gradient descent.
Examples
Python
import torch
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
print("learning rate:", optimizer.defaults["lr"])Output:
learning rate: 0.01