Activation Function

An activation function is a small math function applied to a neuron's output in a neural network. It introduces non-linearity, allowing the network to learn complex patterns rather than just straight lines.

Data & AI

Without activation functions, stacking layers would be no more powerful than a single linear model. The activation decides how strongly a neuron 'fires' based on its input. Common ones include ReLU (keeps positives, zeros out negatives), sigmoid (squashes to 0-1), and tanh.

ReLU is the most popular in deep learning because it's simple and trains fast. The activation is what lets neural networks bend and curve to fit messy, real-world data.

Examples

Python

import numpy as np

def relu(x):
    return np.maximum(0, x)

print(relu(np.array([-2.0, -0.5, 1.0, 3.0])))

Output:

[0.  0.  1.  3.]

Learn it in a lesson

See also

Back to the Dictionary