Activation Function
An activation function is a small math function applied to a neuron's output in a neural network. It introduces non-linearity, allowing the network to learn complex patterns rather than just straight lines.
Data & AI
Without activation functions, stacking layers would be no more powerful than a single linear model. The activation decides how strongly a neuron 'fires' based on its input. Common ones include ReLU (keeps positives, zeros out negatives), sigmoid (squashes to 0-1), and tanh.
ReLU is the most popular in deep learning because it's simple and trains fast. The activation is what lets neural networks bend and curve to fit messy, real-world data.
Examples
Python
import numpy as np
def relu(x):
return np.maximum(0, x)
print(relu(np.array([-2.0, -0.5, 1.0, 3.0])))Output:
[0. 0. 1. 3.]