Reinforcement Learning Basics

Reviewed & published by Brayan K

Master the core mechanics every RL algorithm is built on — the discounted return, value functions V(s) and Q(s,a), the Bellman equation, and the words used to describe how agents learn.

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

• Compute the discounted return G_t from a list of rewards
• Explain what the discount factor gamma does and why it exists
• Tell apart the value function V(s) and the action-value Q(s,a)
• Read the Bellman equation as plain English, not algebra
• Distinguish episodic tasks from continuing tasks
• Describe on-policy vs off-policy learning at a high level

💰 Real-World Analogy: Money Today vs Money Later

Imagine someone offers you £100 today or £100 in a year. You take it today — money now is worth more than money later, because you could spend or invest it. RL agents feel exactly the same about reward.

The discount factor gamma (a number between 0 and 1) is the "interest rate" the agent applies to the future. A reward two steps away is multiplied by gamma², three steps away by gamma³, and so on. With gamma close to 1 the agent is patient and plans far ahead; with gamma close to 0 it is impulsive and grabs whatever reward is nearest. The return is just the total of all those discounted rewards — and maximising it is the agent's entire job.

1 The Return G_t — What RL Actually Maximises

A reward is the single number the environment hands you on one step. But a greedy agent that only chased the next reward would walk straight into traps. What you really care about is the return — the sum of all future rewards, with later ones discounted.

Read it left to right: take this step's reward in full, then add next step's reward shrunk by gamma, then the one after shrunk by gamma², and keep going. Because gamma is below 1, far-future rewards barely register — which also keeps the sum finite even if it never ends. The worked example below computes this for a real reward list and then does the one update step that turns an observed return into a learned value.

# The two numbers at the heart of RL: the return and the value
# Plain Python — no libraries needed.

# A reward is the single number the environment hands back each step.
# The RETURN G_t is the TOTAL future reward from a point in time,
# but with future rewards shrunk by a discount factor gamma (0..1).
#
#   G_t = r_t + gamma*r_{t+1} + gamma^2*r_{t+2} + ...
#
# gamma < 1 means "a reward now is worth more than the same reward later".

rewards = [1, 0, 2, 5]   # rewards collected over 4 time steps
gamma   = 0.9            # discount factor: how much the future matters

# Add them up, discounting each step one more power of gamma.
G = 0.0
for step, r in enumerate(rewards):
    G += (gamma ** step) * r        # gamma**0=1, gamma**1=0.9, gamma**2=0.81 ...
    print(f"step {step}: reward={r}, gamma^{step}={gamma**step:.3f}, running G={G:.3f}")

print()
print(f"Discounted return G = {G:.3f}")     # 1 + 0 + 1.62 + 3.645 = 6.265

# A VALUE V(s) is just the return you EXPECT starting from state s.
# We estimate it by nudging an old guess toward what we actually saw.
# This single line is the engine inside almost every RL algorithm:
#
#   V(s) <- V(s) + alpha * (target - V(s))
#
# alpha (learning rate) controls how big each nudge is.
V_s   = 5.0     # our current guess for the value of this state
alpha = 0.1     # learning rate
target = G      # the return we just observed is our learning target

V_s = V_s + alpha * (target - V_s)            # move 10% of the way toward G
print(f"Updated value V(s) = {V_s:.3f}")      # 5 + 0.1*(6.265-5) = 5.127

2 Value Functions: V(s) and Q(s,a)

A single return is what happened in one run. A value function is the return you expect on average. There are two of them, and the only difference is how much they pin down:

"How good is it to be in state s?" The expected return if you start in s and act well from there.

"How good is it to do action a in state s?" The expected return if you take a now, then act well.

Q is the more useful of the two because it scores every action — so the agent can simply pick the action with the highest Q. And the two connect neatly: the value of a state is the value of its best action, written V(s) = max over actions Q(s, a). The example computes both for a small two-action state.

# V(s) vs Q(s,a): two flavours of "how good is this?"
#
#   V(s)    = expected return if you START in state s and act well after.
#   Q(s,a)  = expected return if you START in state s, take action a NOW,
#             then act well after.
#
# Q is more detailed: it scores each ACTION, so you can pick the best one.
# Relationship: V(s) = the Q of the best action = max over a of Q(s, a).

gamma = 0.9

def discounted_return(rewards):
    """Total reward in one episode, discounting later steps."""
    G = 0.0
    for step, r in enumerate(rewards):
        G += (gamma ** step) * r
    return G

# Two episodes that started in state "S" but took different first actions.
episode_left  = [-1, -1, 10]    # took action "Left" first, reached goal
episode_right = [-1, 5]         # took action "Right" first, smaller payoff

Q_S_left  = discounted_return(episode_left)
Q_S_right = discounted_return(episode_right)

print(f"Q(S, Left)  = {Q_S_left:.3f}")   # -1 -0.9 +8.1 = 6.200
print(f"Q(S, Right) = {Q_S_right:.3f}")  # -1 +4.5      = 3.500

# V(s) is the value of the BEST action available in that state.
V_S = max(Q_S_left, Q_S_right)
print(f"V(S) = max over actions = {V_S:.3f}")   # 6.200

best_action = "Left" if Q_S_left >= Q_S_right else "Right"
print(f"Best action from S: {best_action}")      # Left

3 The Bellman Equation — The Big Idea

The return G_t is an infinite sum, which sounds impossible to compute. The Bellman equation is the trick that makes it easy. It notices that the future, after one step, is just another value — so the whole sum folds into a tiny recursive rule:

In plain English: the value of where you are equals the reward you get now, plus the discounted value of wherever you land next (s′). You don't need to look at the entire future — you only need this step's reward and an estimate of the next state. That's the insight that turns RL from "sum over infinity" into a loop you can actually run.

4 Episodes vs Continuing Tasks

RL problems come in two shapes, and they change how you think about the return:

Have a natural end — a chess game finishes, a robot reaches the goal, the player dies. The run is an episode; afterwards everything resets. The return is a finite sum over the steps of that one episode.

Never stop — a thermostat, a stock-trading bot, a server tuning itself run forever. There's no reset and no final step.

This is the second reason discounting exists: for a continuing task the reward sum would run to infinity, but multiplying each future reward by gamma below 1 makes the total settle on a finite number. A terminal state (the end of an episode) is treated as having value 0 — there's no future left to discount.

5 On-Policy vs Off-Policy (High Level)

A policy is the agent's strategy — its rule for choosing an action in each state. Algorithms differ in whether the policy they learn about is the same one they act with:

• On-policy — you learn about the very policy you're following, exploration and all. Honest but cautious; it can't easily reuse old data. (SARSA is the classic example.)
• Off-policy — you act with one policy (often a curious, exploratory one) but learn the value of a different policy (usually the best greedy one). More flexible, and it lets you reuse stored experience — at the cost of being harder to keep stable. (Q-learning is off-policy.)

🎯 Your Turn 1: Compute a Discounted Return

Fill in the two blanks so the program adds up the rewards with a discount factor of 0.5. Use the expected output to check yourself.

# 🎯 YOUR TURN: compute a discounted return from a reward list
# Fill in each ___ then run it.

rewards = [2, 3, 0, 4]      # rewards over 4 steps (given)
gamma   = ___               # 👉 use 0.5 as the discount factor

G = 0.0
for step, r in enumerate(rewards):
    # 👉 add this step's reward, discounted by gamma**step
    G += ___ * r            # 👉 replace ___ with  gamma ** step

print(f"Discounted return G = {G:.3f}")
# ✅ Expected output: Discounted return G = 4.000
#    (2 + 0.5*3 + 0.25*0 + 0.125*4 = 2 + 1.5 + 0 + 0.5 = 4.0)

🎯 Your Turn 2: One Value-Update Step

This is the learning rule at the heart of value-based RL. Fill in the blank to move the value estimate part of the way toward the observed return.

# 🎯 YOUR TURN: do ONE value-update step toward an observed return
# This is the core learning rule of value-based RL.
#
#   V(s) <- V(s) + alpha * (target - V(s))

V_s    = 10.0      # current value estimate for this state (given)
alpha  = 0.2       # learning rate (given)
target = 20.0      # the return we actually observed (given)

# 👉 nudge V_s 20% of the way from its old value toward the target
V_s = V_s + ___ * (target - V_s)      # 👉 replace ___ with  alpha

print(f"Updated V(s) = {V_s:.3f}")
# ✅ Expected output: Updated V(s) = 12.000
#    (10 + 0.2*(20-10) = 10 + 2 = 12)

Common Errors (And How to Fix Them)

❌ Off-by-one on the discount power

Discounting the first reward as well — every term ends up too small.

G += (gamma ** (step + 1)) * r   # ❌ first reward shouldn't be discounted

✅ Fix: step starts at 0, so the first reward uses gamma**0 = 1.

G += (gamma ** step) * r          # ✅ step = 0, 1, 2, ...

Setting gamma = 1 on a continuing (never-ending) task makes the return grow without bound.

gamma = 1.0   # ❌ on a task with no terminal state, G never converges

✅ Fix: use gamma below 1 (0.9 or 0.99) for continuing tasks.

gamma = 0.99  # ✅ future rewards fade, return stays finite

❌ Wrong sign in the value update

Writing (V_s - target) pushes the estimate away from the return — it diverges.

V_s = V_s + alpha * (V_s - target)   # ❌ moves the wrong way

✅ Fix: the error is target minus current estimate.

V_s = V_s + alpha * (target - V_s)   # ✅ moves toward the target

📋 Quick Reference

🎯 Mini-Challenge: Average Return Over Episodes

Support is faded here — only the outline is given. Write the function and the averaging logic yourself, then check against the expected output in the comments.

# 🎯 MINI-CHALLENGE: average return over several episodes
# 1. You have 3 episodes, each a list of rewards (given below).
# 2. Write a function discounted_return(rewards) using gamma = 0.9
#    that returns r0 + 0.9*r1 + 0.9^2*r2 + ...
# 3. Compute the return for each episode, then print the AVERAGE return.
#
# ✅ Expected (gamma=0.9): Average return = 6.660

gamma = 0.9
episodes = [
    [1, 2, 3],      # return = 1 + 1.8 + 2.43 = 5.23
    [0, 0, 10],     # return = 0 + 0   + 8.1  = 8.10
    [5, 1, 1],      # return = 5 + 0.9 + 0.81 = 6.71
]

# your code here

Lesson complete — you now know the machinery of RL!

You can compute a discounted return, tell V(s) from Q(s,a), read the Bellman equation as plain English, separate episodic from continuing tasks, and explain on-policy vs off-policy. These are the exact pieces every RL algorithm assembles.

Practice quiz

Continue this course

• Previous: Fine-Tuning LLMs: LoRA, QLoRA & PEFT Techniques
• Next: Q-Learning & Deep Q-Networks (DQN) — Implement Q-learning and DQN with experience replay and target networks
• Quick reference: AI & Machine Learning cheat sheet
• From the blog: Reinforcement Learning: Q-Learning Explained

Frequently asked questions

What is the return G_t in reinforcement learning?

The return is the total reward an agent collects from a moment onward, with each future reward shrunk by the discount factor gamma: G_t = r_t + gamma*r_{t+1} + gamma^2*r_{t+2} + .... It is the quantity RL actually tries to maximise — not the single next reward.

What is the difference between V(s) and Q(s,a)?

V(s) is the expected return when you start in state s and then act well. Q(s,a) is the expected return when you start in state s, take action a now, and act well afterwards. Q scores each action, so you can pick the best one, and V(s) equals the Q value of that best action.

What does the Bellman equation actually say?

It says the value of a state equals the immediate reward plus the discounted value of where you land next: V(s) = r + gamma*V(s'). It turns one giant sum over the whole future into a short recursive rule, which is what makes RL computable.

What is the difference between an episode and a continuing task?

An episodic task has a natural end — a game finishes, a robot reaches the goal — and then resets. A continuing task never terminates, like a server that runs forever. Discounting (gamma < 1) keeps the return finite even for continuing tasks.

What does on-policy versus off-policy mean?

On-policy methods learn about the same policy they use to choose actions. Off-policy methods can learn about one policy (for example the best greedy one) while exploring with a different policy. Off-policy is more flexible and lets you reuse old experience, but is trickier to make stable.

Related lessons

• Reinforcement Learning