Advanced Neural Network Techniques

Reviewed & published by Brayan K

Learn the practical knobs that turn a network that barely trains into one that trains fast and generalizes well — activations, initialization, normalization, regularization, optimizers, and learning-rate schedules.

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

• Pick the right activation: ReLU, LeakyReLU, or GELU
• Initialize weights with Xavier or He so signals stay healthy
• Stabilize training with BatchNorm and LayerNorm
• Fight overfitting with dropout, L2, and early stopping
• Choose an optimizer: SGD with momentum vs Adam
• Schedule the learning rate with decay, cosine, and warm-up

🔧 Real-World Analogy: Tuning an Engine

A raw neural network is like an engine straight off the assembly line: the parts are all there, but it sputters, stalls, and burns fuel. The techniques in this lesson are the tuning that turns it into a smooth, powerful machine.

• Activation functions are the spark plugs — they decide which signals fire.
• Weight initialization is the cold-start setting — get it wrong and the engine floods before it even runs.
• Normalization is the cooling system — it keeps internal values in a safe range so nothing overheats.
• Regularization is the rev limiter — it stops the model over-revving and memorizing instead of learning.
• The optimizer is the throttle response — how the model converts feedback into movement.
• The learning-rate schedule is the gearbox — big steps to get going, smaller steps to cruise precisely.

Each part matters on its own, but the magic is in tuning them together.

1 Activation Functions — ReLU, LeakyReLU, GELU

An activation function sits after each layer and decides what the neuron passes on. Without one, stacking layers would just be one big linear function — the network could not learn curves or anything interesting. The activation adds the non-linearity that lets a network model complex patterns.

• ReLU (Rectified Linear Unit): outputs the value if positive, else 0. Fast and the most common default.
• LeakyReLU: like ReLU, but negatives leak through with a small slope so neurons cannot get permanently stuck at 0.
• GELU (Gaussian Error Linear Unit): a smooth curve used in transformers and most modern large language models.

Here is each one built from scratch in plain Python so you can watch what they do. Run it:

# Activation functions decide what a neuron passes on.
# We build them from scratch so you can SEE what they do —
# no libraries, just lists and simple math.

def relu(values):
    # ReLU: keep positives, clamp negatives to 0
    return [max(0.0, x) for x in values]

def leaky_relu(values, slope=0.01):
    # LeakyReLU: like ReLU, but negatives leak through a little
    # so the neuron never fully "dies"
    return [x if x > 0 else slope * x for x in values]

scores = [3.0, -2.0, 0.5, -10.0]

print("input       :", scores)
print("ReLU        :", relu(scores))
print("LeakyReLU   :", leaky_relu(scores))

# Expected output:
# input       : [3.0, -2.0, 0.5, -10.0]
# ReLU        : [3.0, 0.0, 0.5, 0.0]
# LeakyReLU   : [3.0, -0.02, 0.5, -0.1]

Notice ReLU flattens every negative to 0.0, while LeakyReLU keeps a faint negative signal (-2.0 becomes -0.02). That tiny leak is what keeps the neuron alive during training.

In Keras, you do not write the math yourself — you name the activation:

from tensorflow import keras
from tensorflow.keras import layers

model = keras.Sequential([
    layers.Dense(64, activation="relu"),    # ReLU hidden layer
    layers.Dense(64, activation="gelu"),    # GELU hidden layer
    layers.Dense(10, activation="softmax"), # softmax OUTPUT for 10 classes
])

# Expected output: a model whose hidden layers use ReLU and GELU,
# and whose output layer turns scores into class probabilities.

# 🎯 YOUR TURN — fill in the blanks marked with ___

def relu(values):
    # 1) Return each value if it is positive, else 0.0
    return [___ for x in values]   # 👉 replace ___ with: max(0.0, x)

def leaky_relu(values, slope=0.01):
    # 2) Let negatives leak through scaled by 'slope'
    return [x if x > 0 else ___ for x in values]  # 👉 replace ___ with: slope * x

data = [1.0, -4.0, 2.5, -0.5]
print("ReLU      :", relu(data))
print("LeakyReLU :", leaky_relu(data))

# ✅ Expected output:
# ReLU      : [1.0, 0.0, 2.5, 0.0]
# LeakyReLU : [1.0, -0.04, 2.5, -0.005]

2 Weight Initialization — Xavier & He

Before training starts, every weight needs a starting value. Set them all to zero and every neuron learns the same thing (useless). Set them too large and the signal explodes; too small and it vanishes. Smart initialization scales the random starting weights by the layer size so signals stay a healthy size on the very first forward pass.

• Xavier (Glorot): tuned for symmetric activations like tanh and sigmoid.
• He: scales variance up by 2 to account for ReLU discarding the negative half — use it with ReLU.

Choosing an initializer in Keras is a one-word argument:

from tensorflow.keras import layers

# He init pairs with ReLU
layers.Dense(64, activation="relu",
             kernel_initializer="he_normal")

# Xavier/Glorot init pairs with tanh
layers.Dense(64, activation="tanh",
             kernel_initializer="glorot_uniform")

# Expected output: layers whose initial weights are scaled to the
# right size for their activation, so early training stays stable.

3 Normalization — BatchNorm & LayerNorm

As data flows through a deep network, the scale of values can drift wildly between layers, which slows training. Normalization re-centers and re-scales these values so each layer sees inputs in a stable range — like a cooling system keeping the engine from overheating.

• BatchNorm: normalizes each feature across the examples in a mini-batch. Great for CNNs; needs a decent batch size and behaves differently at inference.
• LayerNorm: normalizes across the features of a single example. Independent of batch size, identical in training and inference — the default in transformers.

Both are just layers you drop into the model:

from tensorflow.keras import layers

# Typical CNN pattern: Dense -> BatchNorm -> activation
layers.Dense(64, use_bias=False)
layers.BatchNormalization()
layers.Activation("relu")

# Transformer pattern uses LayerNorm instead
layers.LayerNormalization()

# Expected output: activations kept in a stable range layer to layer,
# letting you train deeper networks faster.

4 Regularization — Dropout, L2, Early Stopping

Overfitting is when a model memorizes the training data instead of learning the general pattern — it scores great on data it has seen and badly on anything new. Regularization is the set of tools that prevent this.

• Dropout: randomly switch off a fraction of neurons each training step, so the network cannot lean on any single unit.
• L2 (weight decay): add a penalty for large weights, nudging the model toward simpler, smoother solutions.
• Early stopping: watch validation loss and stop training the moment it stops improving.

All three together in Keras:

from tensorflow import keras
from tensorflow.keras import layers, regularizers

model = keras.Sequential([
    layers.Dense(128, activation="relu",
                 kernel_regularizer=regularizers.l2(1e-4)),  # L2
    layers.Dropout(0.3),                                     # drop 30%
    layers.Dense(10, activation="softmax"),
])

early_stop = keras.callbacks.EarlyStopping(
    monitor="val_loss", patience=3, restore_best_weights=True
)
# model.fit(..., callbacks=[early_stop])

# Expected output: training halts automatically once validation loss
# stops improving, and the best (not the last) weights are kept.

5 Optimizers — SGD with Momentum & Adam

The optimizer is the rule that turns gradients into weight updates. Plain SGD takes a step straight downhill. Momentum adds memory of the previous direction so the optimizer keeps rolling through small bumps and noisy gradients — like a ball gaining speed downhill. Adam goes further, giving every weight its own adaptive step size.

Below is a single momentum update step in plain Python. The formula is just two lines: velocity = beta * velocity - lr * gradient, then weight = weight + velocity. Run it:

# One step of SGD WITH MOMENTUM, in plain Python.
# Momentum remembers the previous update direction so the
# optimizer keeps rolling and smooths out noisy gradients.
#
#   velocity = beta * velocity - lr * gradient
#   weight   = weight + velocity

lr = 0.1        # learning rate (step size)
beta = 0.9      # momentum factor (how much past direction to keep)

weight = 2.0    # the parameter we are training
velocity = 0.0  # momentum starts at rest
gradient = 0.5  # slope of the loss w.r.t. weight (pretend it was computed)

# --- one update step ---
velocity = beta * velocity - lr * gradient
weight = weight + velocity

print("velocity after step:", round(velocity, 4))
print("weight   after step:", round(weight, 4))

# Expected output:
# velocity after step: -0.05
# weight   after step: 1.95

In Keras you pick the optimizer by name when you compile:

from tensorflow import keras

# SGD with momentum
sgd = keras.optimizers.SGD(learning_rate=0.01, momentum=0.9)

# Adam — adaptive per-parameter step sizes (the safe default)
adam = keras.optimizers.Adam(learning_rate=1e-3)

model.compile(optimizer=adam, loss="categorical_crossentropy")

# Expected output: a compiled model that updates its weights with Adam,
# adapting the step size for every parameter automatically.

# 🎯 YOUR TURN — fill in the blanks marked with ___
# Do ONE SGD-with-momentum update step.

lr = 0.2
beta = 0.9
weight = 5.0
velocity = 0.0
gradient = 1.0

# 1) Update the velocity: beta * velocity - lr * gradient
velocity = ___          # 👉 replace ___ with: beta * velocity - lr * gradient

# 2) Apply the velocity to the weight
weight = ___            # 👉 replace ___ with: weight + velocity

print("velocity:", round(velocity, 4))
print("weight  :", round(weight, 4))

# ✅ Expected output:
# velocity: -0.2
# weight  : 4.8

6 Learning-Rate Schedules — Decay, Cosine, Warm-up

The learning rate is the size of each step. A fixed rate is a compromise: too big and the model bounces around the target; too small and training crawls. A schedule changes the rate over time — big early to cover ground, small later to settle precisely. It is the gearbox of training.

• Step decay: drop the rate by a factor every few epochs.
• Cosine annealing: smoothly curve the rate down to near zero.
• Warm-up: start tiny and ramp up over the first few hundred steps so an untrained net is not destabilized.

A cosine schedule in Keras:

from tensorflow import keras

lr_schedule = keras.optimizers.schedules.CosineDecay(
    initial_learning_rate=1e-3,
    decay_steps=10000,   # curve down to ~0 over 10k steps
)

opt = keras.optimizers.Adam(learning_rate=lr_schedule)

# Expected output: the learning rate starts at 1e-3 and smoothly
# decreases toward 0 as training proceeds, helping the model settle.

! Common Errors (And How to Fix Them)

These five mistakes trip up almost everyone at first. Here is how to spot and fix each one:

Many neurons output 0 forever and stop learning — usually caused by a too-high learning rate or large negative bias pushing inputs permanently negative.

✅ Fix: switch to LeakyReLU or GELU, lower the learning rate, and use He initialization.

Loss is nan on the first step, or it barely moves. Weights started too large (exploding) or too small (vanishing).

✅ Fix: use he_normal with ReLU or glorot_uniform with tanh — never all-zeros.

❌ No normalization in a deep net

A deep network trains painfully slowly or diverges because activations drift to extreme scales between layers.

✅ Fix: add BatchNorm (CNNs) or LayerNorm (transformers) between layers.

Loss explodes to nan (rate too high) or hardly changes over many epochs (rate too low).

✅ Fix: start around 1e-3 for Adam, add warm-up, and use a decay schedule.

❌ Dropout left on at inference

Predictions are randomly different each run because dropout is still zeroing out neurons during evaluation.

✅ Fix: use model.predict() (which disables it), or pass training=False in a custom loop.

📋 Quick Reference

🎯 Mini Challenge: A Two-Step Momentum Optimizer

Time to fade the scaffolding. You get only a comment outline — write the optimizer loop yourself. Run two momentum update steps and print the weight after each.

# 🎯 MINI-CHALLENGE: a two-step momentum optimizer
# 1. Start: lr = 0.1, beta = 0.9, weight = 10.0, velocity = 0.0
# 2. Pretend the gradient is 2.0 on BOTH steps
# 3. Run the update TWICE in a loop:
#       velocity = beta * velocity - lr * gradient
#       weight   = weight + velocity
# 4. After each step, print the rounded weight (4 decimals)
#
# ✅ Expected output:
# step 1 weight: 9.8
# step 2 weight: 9.42

# your code here

Lesson complete — you can now tune a network like an engine!

You can choose an activation (ReLU, LeakyReLU, GELU), initialize weights with He or Xavier, stabilize training with BatchNorm or LayerNorm, fight overfitting with dropout, L2, and early stopping, pick between SGD-with-momentum and Adam, and shape the learning rate with a schedule. These are the everyday levers professionals reach for on every serious model.

Practice quiz

Continue this course

• Previous: Computer Vision Basics
• Next: Transformers & LLMs — How attention mechanisms power GPT, BERT, and large language models
• Quick reference: AI & Machine Learning cheat sheet

Frequently asked questions

Which activation function should I use by default?

Start with ReLU for hidden layers — it is fast, simple, and works well most of the time. If you see many 'dead' neurons (units stuck at zero that never recover), switch to LeakyReLU or GELU. GELU is the modern default inside transformers and most large language models because its smooth curve tends to train a little more reliably. Keep the OUTPUT activation matched to the task: softmax for multi-class classification, sigmoid for binary, and none (a plain linear output) for regression.

What is the difference between Xavier and He initialization?

Both pick the starting weights from a distribution scaled by the layer size so signals neither explode nor vanish on the first forward pass — but they assume different activations. Xavier (also called Glorot) is tuned for activations that are symmetric around zero, like tanh and sigmoid. He initialization scales the variance up by a factor of 2 to account for ReLU throwing away the negative half, so use He with ReLU and its variants. Frameworks pick a sensible default, but choosing the right one speeds up early training.

When should I use BatchNorm versus LayerNorm?

BatchNorm normalizes each feature across the examples in a mini-batch, so it needs a reasonably large batch and behaves differently in training versus inference (it tracks running averages). It shines in convolutional vision networks. LayerNorm normalizes across the features of a single example, so it does not depend on batch size and behaves identically in training and inference — that is why transformers and recurrent networks use it. Rule of thumb: CNNs lean on BatchNorm, sequence and transformer models lean on LayerNorm.

How do dropout, L2, and early stopping fit together?

They are three independent tools against overfitting, and you can stack them. Dropout randomly switches off a fraction of neurons each training step so the network cannot rely on any single unit. L2 (weight decay) adds a penalty for large weights, nudging the model toward simpler solutions. Early stopping watches the validation loss and halts training the moment it stops improving, before the model memorizes the training set. A common recipe is light L2 plus a bit of dropout plus early stopping.

Why is Adam usually preferred over plain SGD?

Adam keeps a per-parameter learning rate by tracking both the recent average of gradients (momentum) and their recent variance, so it adapts the step size for every weight automatically. That makes it forgiving about the initial learning rate and quick to get a model training. Plain SGD with momentum can generalize slightly better and is still popular for vision models trained with a careful learning-rate schedule, but Adam (or AdamW, which fixes how weight decay is applied) is the safe default to reach for first.

What does a learning-rate schedule actually do?

It changes the learning rate as training progresses instead of holding it fixed. A large rate early on covers ground fast; shrinking it later lets the model settle precisely into a good minimum instead of bouncing around it. Common schedules are step decay (drop the rate by a factor every few epochs), cosine annealing (smoothly curve down to near zero), and warm-up (start tiny and ramp up over the first few hundred steps so an untrained network does not get destabilized).

Related lessons

• Neural Networks
• Deep Learning