Working with Queues

Reviewed & published by Brayan K

By the end of this lesson you'll be able to push slow work — emails, image processing, API calls — onto a background queue so your app responds instantly, and run a worker that retries, backs off, and never loses a job.

Part of the free PHP course at LearnCodingFast — hands-on lessons with worked examples and the output they print, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

• Explain why offloading slow work keeps requests fast
• Identify producers, consumers, workers, and the broker
• Compare Redis, RabbitMQ, Beanstalkd, and a database queue
• Write a job class and a worker loop that processes it
• Add retries with backoff and a dead-letter queue
• Make jobs idempotent and keep long-lived workers healthy

1️⃣ Why Offload Work to a Queue?

When a request comes in, your code runs top to bottom and only then sends a response. If that code blocks — sits and waits — on something slow like sending email (2–3 seconds talking to a mail server), resizing an image, or calling a third-party API, the user stares at a spinner the whole time. A queue fixes this: instead of doing the slow work now, you write a tiny note describing the work (a job), drop it on the queue, and reply immediately. Something else does the work later. Here is the difference side by side.

<?php
// WHY queues exist. Imagine a signup that sends a welcome email.
// Sending email talks to a remote mail server, which can take 2-3 seconds.

// === The SLOW way: do the work DURING the request ===
function handleSignupBlocking(string $email): string
{
    // sendWelcomeEmail($email);  // <-- user waits here for 2-3 seconds
    return "Account created (after a slow wait)";
}

// === The FAST way: hand the slow work to a queue, respond instantly ===
$queue = [];                                  // pretend this is Redis/RabbitMQ

function handleSignupQueued(string $email, array &$queue): string
{
    // "Enqueue" = drop a small job onto the queue and move on immediately.
    $queue[] = ['type' => 'send_welcome_email', 'to' => $email];
    return "Account created (instantly)";     // user never waits for email
}

echo handleSignupBlocking('[email protected]') . "\n";
echo handleSignupQueued('[email protected]', $queue) . "\n";

// The job is now waiting in the queue for a worker to pick up later.
echo "Jobs waiting in queue: " . count($queue) . "\n";
?>

The blocking version makes the user wait for the email; the queued version returns the moment the job is on the queue. The email still gets sent — just a moment later, by a separate process. The user's experience goes from slow to instant.

2️⃣ Producers, Consumers, Workers & Brokers

Four words describe every queue system. The producer is the code that creates a job and pushes it (usually your web app). The broker is the storage that holds jobs in the middle — it's the ticket rail. The consumer (also called a worker) is a long-running process that pulls jobs off the broker and runs them. Producers and workers don't talk to each other directly; they only ever touch the broker, which is what lets them scale and fail independently.

You have several choices of broker, and they trade simplicity for power:

Most PHP apps start with Redis or a database queue and only move to RabbitMQ when they genuinely need its routing. The concepts below are identical whichever you pick.

3️⃣ The Job: a Unit of Deferred Work

A job is a small object that describes work to do later. Because the producer and worker run in different processes — often on different machines — the job has to be serialisable: turned into plain text (usually JSON) to store in the broker, then rebuilt on the worker side. The golden rule is keep the payload tiny: store a user's id, not the whole user object. The worker reloads fresh data when it runs, so the job stays small and never goes stale.

<?php
// A "job" is a small, serialisable description of work to do LATER.
// Keep the PAYLOAD tiny — an id, not the whole object — so it survives being
// written to Redis as JSON and read back by a separate worker process.
class SendWelcomeEmail
{
    public function __construct(public int $userId) {}

    // handle() is where the actual work happens, run by the worker.
    public function handle(): void
    {
        // In real code you'd load the user and call your mailer here.
        echo "  -> emailing user #{$this->userId}\n";
    }

    // Turn the job into a string the broker can store...
    public function toJson(): string
    {
        return json_encode(['job' => static::class, 'userId' => $this->userId]);
    }

    // ...and back into a job object on the worker side.
    public static function fromJson(string $raw): self
    {
        $data = json_decode($raw, true);
        return new self($data['userId']);
    }
}

$job = new SendWelcomeEmail(42);
$line = $job->toJson();               // what gets pushed to the queue
echo "On the wire: {$line}\n";

$revived = SendWelcomeEmail::fromJson($line);   // worker rebuilds it
echo "Worker runs the job:\n";
$revived->handle();
?>

Notice the round trip: toJson() produces the string that lives in the broker, and fromJson() rebuilds the job on the worker so it can call handle(). That handle method is where your real work goes.

4️⃣ Your Turn: Make a Producer Respond Instantly

Now you try. The upload handler below should queue the resize work and return straight away, never making the user wait for the image to be processed. Fill in each ___ using the 👉 hint, then run it and check it against the Output panel.

<?php
// 🎯 YOUR TURN — make the producer return INSTANTLY.
// Fill in each blank marked ___ , then run it and check the Output panel.

$queue = [];   // pretend this is Redis

function uploadAvatar(string $file, array &$queue): string
{
    // 1) Push a 'resize_image' job onto the queue instead of resizing now.
    $queue[] = ___;   // 👉 e.g.  ['type' => 'resize_image', 'file' => $file]

    // 2) Return straight away — the user shouldn't wait for the resize.
    return ___;       // 👉 e.g.  "Upload received"
}

echo uploadAvatar('avatar.jpg', $queue) . "\n";
echo "Jobs queued: " . count($queue) . "\n";

// ✅ Expected output:
//    Upload received
//    Jobs queued: 1
?>

5️⃣ The Worker Loop: Retries, Backoff & Dead-Letter

A worker is a loop: pull a job, run it, and decide what to do with the result. Real work fails for boring, temporary reasons — a network blip, a rate-limited API — so a worker retries. To avoid hammering a struggling service, each retry waits a little longer than the last; this is exponential backoff (1s, then 2s, then 4s…). When a job has used up all its attempts, you don't throw it away — you move it to a dead-letter queue, a separate list where a human can inspect what went wrong. And because a worker can crash after doing the work but before recording it, jobs must be idempotent: running one twice has the same effect as running it once. The example below shows all four working together — fully deterministic, so you can match the output exactly.

<?php
// A complete producer/worker pipeline you can run anywhere. A real system swaps
// the array for Redis, but the LIFECYCLE is identical: push -> pop -> handle.

class Job
{
    public int $attempts = 0;
    public function __construct(
        public string $id,            // a STABLE id, used for idempotency
        public string $type,
        public array $payload,
        public int $maxAttempts = 3,
    ) {}
}

$queue   = [];   // pending jobs (the broker)
$dead    = [];   // dead-letter queue: jobs that exhausted all retries
$done    = [];   // ids we've already completed (idempotency guard)

// === PRODUCER: push jobs, then return to the user instantly ===
function enqueue(array &$queue, Job $job): void {
    $queue[] = $job;
    echo "queued {$job->type} (#{$job->id})\n";
}

enqueue($queue, new Job('e1', 'send_email',   ['to' => 'alice']));
enqueue($queue, new Job('i1', 'resize_image', ['file' => 'avatar.jpg']));
enqueue($queue, new Job('p1', 'charge_card',  ['amount' => 999]));

// Pretend charge_card fails twice (flaky network) then succeeds.
$failsLeft = ['p1' => 2];

// The worker's "do the work" step. Returns true on success.
function handle(Job $job, array &$done, array &$failsLeft): bool {
    // IDEMPOTENCY: if we've already done this exact job id, skip it.
    if (in_array($job->id, $done, true)) {
        echo "  skip {$job->id} (already done)\n";
        return true;
    }
    if (($failsLeft[$job->id] ?? 0) > 0) {
        $failsLeft[$job->id]--;
        return false;                 // simulate a transient failure
    }
    $done[] = $job->id;               // record success so retries are safe
    return true;
}

// === WORKER LOOP: pop a job, run it, retry or dead-letter on failure ===
echo "\nworker starting...\n";
while (!empty($queue)) {
    $job = array_shift($queue);       // brPop in Redis; pop here
    $job->attempts++;

    if (handle($job, $done, $failsLeft)) {
        echo "  ok {$job->type} (attempt {$job->attempts})\n";
    } elseif ($job->attempts >= $job->maxAttempts) {
        $dead[] = $job;               // give up -> dead-letter for a human
        echo "  DEAD {$job->type} after {$job->attempts} attempts\n";
    } else {
        // BACKOFF: wait longer before each retry (1s, 2s, 4s...). We just
        // print it here; a real worker would re-queue with a delay.
        $backoff = 2 ** ($job->attempts - 1);
        echo "  retry {$job->type} in {$backoff}s\n";
        $queue[] = $job;              // put it back for another go
    }
}

echo "\ndead-letter queue: " . count($dead) . " job(s)\n";
?>

Trace the charge_card job: it fails twice, so the worker re-queues it with growing backoff (1s, then 2s), and succeeds on the third attempt. The idempotency guard means that even if that job had been delivered twice, the second delivery would print skip rather than charging the card again. Nothing reaches the dead-letter queue here because every job eventually succeeds.

6️⃣ Your Turn: Finish the Retry Rule

Here's the heart of a worker loop with one piece missing: the rule that decides when to stop retrying and send a job to the dead-letter queue. Fill in the ___ with the right comparison operator, then run it.

<?php
// 🎯 YOUR TURN — finish the retry rule in the worker loop.
// A job should retry while it has attempts left, and dead-letter once it runs out.

$job  = ['type' => 'charge_card', 'attempts' => 0, 'maxAttempts' => 3];
$dead = [];
$fails = 5;   // this job always fails, so it must end up dead-lettered

while (true) {
    $job['attempts']++;
    $succeeded = $fails-- <= 0;          // stays false here — always fails
    if ($succeeded) {
        echo "ok\n";
        break;
    }

    // 👉 Stop retrying once attempts reach maxAttempts: send it to dead-letter.
    if ($job['attempts'] ___ $job['maxAttempts']) {   // 👉 use the >= operator
        $dead[] = $job;
        echo "DEAD after {$job['attempts']} attempts\n";
        break;
    }
    echo "retry {$job['attempts']}\n";
}

echo "dead-letter: " . count($dead) . "\n";

// ✅ Expected output:
//    retry 1
//    retry 2
//    DEAD after 3 attempts
//    dead-letter: 1
?>

7️⃣ A Real Broker: Redis + a Worker Process

Swap the in-memory array for Redis and the same lifecycle becomes durable across processes and restarts. The producer uses LPUSH to add a job; the worker uses BRPOP, which blocks (waits efficiently, using no CPU) until a job arrives or it times out. The idempotency guard here is SETNX — "set if not exists" — which returns false if this job id was already handled, so duplicate deliveries are skipped safely.

<?php
// The SAME lifecycle, now durable across processes. Needs a Redis server and
// the phpredis extension. The producer runs in your web app; the worker runs
// as a separate long-lived process (see Supervisor below).
$redis = new Redis();
$redis->connect('127.0.0.1', 6379);

// === Producer (inside your web request): push, then respond instantly ===
$redis->lPush('queue:emails', json_encode([
    'id'    => 'welcome-42',          // stable id for idempotency
    'job'   => 'send_welcome_email',
    'to'    => '[email protected]',
]));

// === Consumer (worker.php — its own process) ===
while (true) {
    // brPop BLOCKS up to 30s waiting for a job, so the worker doesn't busy-loop
    // burning CPU. It returns [list_name, value] or null on timeout.
    $popped = $redis->brPop(['queue:emails'], 30);
    if (empty($popped)) {
        continue;                     // timed out — loop and wait again
    }

    $data = json_decode($popped[1], true);

    // IDEMPOTENCY: setnx returns false if this id was already processed.
    if (!$redis->setnx("done:{$data['id']}", 1)) {
        continue;                     // duplicate delivery — skip safely
    }

    try {
        sendWelcomeEmail($data['to']);            // your real work
    } catch (Throwable $e) {
        $redis->del("done:{$data['id']}");        // allow a retry
        // DEAD-LETTER: park the raw job for a human to inspect later.
        $redis->lPush('queue:emails:dead', $popped[1]);
        error_log("job failed: {$e->getMessage()}");
    }
}
?>

That while (true) loop must stay alive forever, restart if it crashes, and not be killed mid-job. You don't babysit it by hand — a process manager like Supervisor does. This config runs three parallel workers and restarts any that die.

; /etc/supervisor/conf.d/email-worker.conf
[program:email-worker]
command=php /var/www/worker.php
numprocs=3            ; run 3 parallel workers
process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true     ; restart a worker if it crashes
stopwaitsecs=30      ; let an in-flight job finish before stopping
stderr_logfile=/var/log/email-worker.err.log
stdout_logfile=/var/log/email-worker.out.log

8️⃣ In Practice: Laravel Queues & Symfony Messenger

You've now built every piece by hand so you understand it — but in production you let your framework do the plumbing. In Laravel, a job is a class implementing ShouldQueue; you dispatch it with SendWelcomeEmail::dispatch($user->id) and run workers with php artisan queue:work. Retries, backoff, delays, rate limiting, batching and a failed_jobs table all come built in across Redis, SQS, Beanstalkd and database drivers. Symfony Messenger follows the same shape: a message class, a handler, configurable transports (the broker), and retry/failure strategies in config. The vocabulary you learned — producer, broker, worker, retry, dead-letter, idempotency — maps straight onto both.

Common Errors (and the fix)

• The request is still slow even though "there's a queue" — you're doing the slow work before enqueuing, so the user still waits. The producer must do almost nothing: build a tiny job, push it, return. All the heavy lifting belongs in the worker's handle().
• A flaky job fails forever or charges a customer twice — you have no retry, or no idempotency. Wrap the work in retries with backoff and give each job a stable id you record on success (Redis SETNX, or a unique column) so a re-delivery is skipped instead of repeated.
• Jobs vanish when a worker crashes — you popped the job and lost it before finishing. Use a broker that holds the job until you acknowledge it (Redis reliable-queue pattern, or your framework's driver), and send exhausted jobs to a dead-letter queue rather than letting them disappear.
• The worker's memory grows until it's killed (OOM) — long-lived PHP processes accumulate leaks that a normal request would never hit. Restart workers regularly: queue:work --max-jobs=1000 or --max-time=3600, then let Supervisor start a fresh process.
• New code never runs in the worker — workers load your code once at boot and keep it in memory, so they don't see changes until restarted. After every deploy run php artisan queue:restart (or restart the Supervisor program).

Pro Tips

• 💡 Keep payloads tiny. Push an id, not a whole object — the worker reloads fresh data and the job never goes stale.
• 💡 Make every job idempotent by default. Assume "at least once" delivery; a stable id plus a "have I done this?" check makes duplicates harmless.
• 💡 Use separate queues per priority. Put fast, urgent jobs (password resets) on one queue and slow batch jobs (reports) on another so one never starves the other.
• 💡 Let the framework do it in production. Hand-rolling is for learning; Laravel queues and Symfony Messenger ship retries, backoff and monitoring for free.

📋 Quick Reference — Queues

Mini-Challenge: An Idempotent Worker

No code is filled in this time — just a brief and an outline. Write it yourself, run it on onecompiler.com/php or your own machine, then check your result against the expected output in the comments. This is the same write-run-check loop you'll use on every real worker.

<?php
// 🎯 MINI-CHALLENGE: a tiny worker with an idempotency guard.
// No code is filled in — work from the steps below, then run it.
//
// 1. Make an array $queue holding three jobs. Give TWO of them the SAME id,
//    e.g. ['id' => 'n1', 'type' => 'notify'] appears twice, plus ['id' => 'n2', ...].
// 2. Make an empty array $done to remember finished ids.
// 3. Loop with array_shift($queue) until the queue is empty.
// 4. For each job: if in_array($job['id'], $done) -> echo "skip <id>" and continue.
//    Otherwise add the id to $done and echo "ran <id>".
// 5. After the loop, echo "completed: " . count($done) . " job(s)".
//
// ✅ Expected output (with ids n1, n1, n2):
//    ran n1
//    skip n1
//    ran n2
//    completed: 2 job(s)

// your code here
?>

🎉 Lesson Complete!

• ✅ Queues let you respond instantly by deferring slow work — email, image processing, API calls
• ✅ A producer pushes jobs to a broker (Redis, RabbitMQ, Beanstalkd, or a database); a worker pulls and runs them
• ✅ A job is a small, serialisable object — keep the payload tiny (an id, not the whole object)
• ✅ Workers retry with backoff and send exhausted jobs to a dead-letter queue instead of losing them
• ✅ Make jobs idempotent (a stable id you record on success) so "at least once" delivery is safe
• ✅ Keep long-lived workers healthy: restart them periodically and manage them with Supervisor
• ✅ In production, Laravel queues and Symfony Messenger give you all of this out of the box

Practice quiz

Continue this course

• Previous: Rate Limiting, Throttling & IP-Based Controls
• Next: Building Email Systems: SMTP, PHPMailer, DKIM, SPF — Send transactional emails with PHPMailer and configure deliverability
• Quick reference: PHP cheat sheet

Frequently asked questions

When should I use a queue instead of doing the work in the request?

Use a queue whenever the work is slow, can fail and be retried, or doesn't have to finish before you reply to the user. Classic cases are sending email, processing or resizing images and video, calling slow third-party APIs, generating PDFs or reports, and sending push notifications. If the user needs the result on screen right now (for example, validating a password), keep it in the request. The rule of thumb: if it takes more than a fraction of a second and the user doesn't need the answer immediately, queue it.

What's the difference between Redis, RabbitMQ, Beanstalkd and a database queue?

They are all brokers — the storage that holds jobs between the producer and the worker. Redis is the simplest and fastest; a list with LPUSH/BRPOP is a perfectly good queue and most PHP apps start here. RabbitMQ is a full message broker with advanced routing, exchanges and strong delivery guarantees — reach for it when you need fan-out or complex topologies. Beanstalkd is a small, purpose-built work queue with built-in delays and retries. A database queue (just a jobs table) needs no extra service and is great for low volume, but it doesn't scale as well under heavy load. Start with Redis or the database, move to RabbitMQ when routing demands it.

What does idempotency mean and why do queues need it?

Idempotent means running the same job twice has the same effect as running it once. Queues need this because most brokers guarantee 'at least once' delivery — a worker can crash after doing the work but before acknowledging the job, so the job gets delivered again. If your job charges a card or sends an email, a duplicate is a real problem. You make a job idempotent by giving it a stable id and recording that id when it completes (for example, with Redis SETNX or a unique column), then skipping any job whose id you've already processed.

Why do long-running PHP workers leak memory, and how do I fix it?

A normal PHP request starts fresh and is torn down at the end, so leaks rarely matter. A worker runs the same script for hours or days, so small leaks — static caches, growing arrays, accumulated entity state — add up until the process is killed. The standard fix is to restart the worker periodically: process a fixed number of jobs (Laravel's --max-jobs) or run for a set time (--max-time), then exit cleanly and let Supervisor start a fresh process. Also avoid holding references to processed jobs and free large variables when you're done with them.

Do I need to write all this queue plumbing myself?

No. The examples here are deliberately hand-rolled so you understand the moving parts, but in production you use your framework's queue layer. Laravel's queues give you job classes, ShouldQueue, automatic retries, backoff, delays, rate limiting, batching and a failed_jobs table out of the box across Redis, SQS, Beanstalkd and database drivers. Symfony Messenger does the same with message classes and handlers, configurable transports, and retry/failure strategies. You still need to understand producers, workers, retries and idempotency — but you won't write the loop by hand.

Related lessons

• classes & objects