Background Jobs

Reviewed & published by Brayan K

By the end of this lesson you'll be able to move slow work off the web request and onto a background worker — enqueuing jobs, processing them in a loop, running things on a recurring schedule, and reaching for BackgroundService, Hangfire, or Quartz.NET when you need persistence, retries, and idempotency.

Part of the free C# course at LearnCodingFast — hands-on lessons with worked examples and the output they print, plus practice exercises and a quick quiz.

What You'll Learn

• Why you offload slow work off the request thread instead of blocking the user
• The three job shapes: fire-and-forget, delayed, and recurring
• Build a queue and a worker loop — the core of every job system
• Run long-lived workers with IHostedService / BackgroundService
• Use Hangfire and Quartz.NET for persistent, scheduled, retrying jobs
• Make jobs idempotent and resilient with retries, poison handling, and persistence

💡 Real-World Analogy

Picture a busy restaurant. When you order, the waiter doesn't stand at your table cooking your meal — that would block them from serving anyone else. Instead they clip your ticket onto a rail in the kitchen and walk away. The chefs work that rail of tickets in order, one after another, at their own pace. A web request is the waiter: it should take the order (enqueue a job) and return to the customer instantly. The background worker is the chef: it pulls tickets off the queue and does the slow cooking. Some tickets are "make this now" (fire-and-forget), some say "fire the dessert in 10 minutes" (delayed), and the daily prep list runs every morning whether anyone ordered it or not (recurring). The whole point is that the dining room never waits on the stove.

Why offload work at all?

A web request should finish in milliseconds. The moment you do something slow inside the request — send an email, resize an image, call a flaky third-party API, build a PDF — the user sits there watching a spinner, your thread is tied up, and a timeout or a crash loses the work entirely.

The fix is to offload: the request records "this needs doing" and returns immediately; a separate worker does the slow part later. That gives you three big wins — fast responses, work that survives a slow downstream service, and the ability to retry failures without the user ever knowing.

Background work comes in three shapes you'll use again and again:

• 🔥 Fire-and-forget — do this once, soon, in the background (send a welcome email).
• ⏳ Delayed — do this once, after a wait (a follow-up email in 3 days).
• 🔁 Recurring — do this on a schedule, forever (a nightly report at 02:00).

📊 The three job shapes

📊 Choosing a tool: HostedService vs Hangfire vs Quartz

Rule of thumb: start with BackgroundService for in-process work, reach for Hangfire the moment jobs must survive a restart or retry on failure, and pick Quartz.NET only when you need its advanced scheduling or clustering.

1. Queues and Workers — the core idea

Strip away the frameworks and every job system is the same two pieces: a queue of pending work and a worker that drains it. A Queue<string> is a first-in, first-out line — Enqueue adds to the back, Dequeue removes from the front. The request side enqueues and returns instantly; the worker side loops, pulling one job at a time. Read this worked example, run it, then you'll build the same loop yourself.

using System;
using System.Collections.Generic;

class Program
{
    static void Main()
    {
        // A background job system is, at heart, a QUEUE and a WORKER.
        // The web request just ENQUEUES work and returns immediately;
        // a separate worker drains the queue and does the slow part.

        // Queue<string> = a first-in, first-out (FIFO) line of tickets.
        Queue<string> jobs = new Queue<string>();

        // --- The "request" side: enqueue work and return fast ---
        jobs.Enqueue("send-welcome-email:[email protected]");
        jobs.Enqueue("resize-image:avatar-42.png");
        jobs.Enqueue("send-welcome-email:[email protected]");
        Console.WriteLine($"Request done. {jobs.Count} job(s) queued."); // Request done. 3 job(s) queued.

        // --- The "worker" side: drain the queue one job at a time ---
        // Dequeue() removes and returns the OLDEST item (front of the line).
        while (jobs.Count > 0)
        {
            string job = jobs.Dequeue();           // take the next ticket
            Console.WriteLine($"Processing: {job}"); // pretend this is slow work
        }

        Console.WriteLine($"All done. {jobs.Count} job(s) left.");        // All done. 0 job(s) left.

        // ✅ Expected output:
        // Request done. 3 job(s) queued.
        // Processing: send-welcome-email:[email protected]
        // Processing: resize-image:avatar-42.png
        // Processing: send-welcome-email:[email protected]
        // All done. 0 job(s) left.
    }
}

Your turn. The program below queues three jobs and processes them — but three pieces are missing. Fill in the ___ blanks using the hints, then run it.

using System;
using System.Collections.Generic;

class Program
{
    static void Main()
    {
        // 🎯 YOUR TURN — fill in the blanks marked with ___ then run it.

        Queue<string> jobs = new Queue<string>();

        // 1) Add three jobs to the queue (the request side).
        jobs.Enqueue("backup-database");
        jobs.___("send-report");        // 👉 same method as the line above: Enqueue
        jobs.Enqueue("clear-cache");

        // 2) Drain the queue: keep going while there is work left.
        while (jobs.Count ___ 0)        // 👉 a comparison so the loop runs while jobs remain: >
        {
            // 3) Take the next job off the FRONT of the queue.
            string job = jobs.___();    // 👉 the FIFO "remove and return" method: Dequeue
            Console.WriteLine($"Running: {job}");
        }

        Console.WriteLine("Queue empty.");

        // ✅ Expected output:
        // Running: backup-database
        // Running: send-report
        // Running: clear-cache
        // Queue empty.
    }
}

2. Recurring Schedules — running only when due

A recurring job shouldn't run every time the worker wakes up — only when its interval has elapsed. The pattern is a small piece of bookkeeping: count how long it's been since the last run, and when that count reaches the interval, run the job and reset the counter. Real schedulers do this with CRON expressions and timestamps, but the logic is identical. Fill in the two ___ blanks below to make a job that fires every third tick.

using System;

class Program
{
    static void Main()
    {
        // 🎯 YOUR TURN — a recurring job runs only when its INTERVAL is due.
        // We simulate "ticks" of a clock and run the job every 3rd tick.

        int interval = 3;     // run the job every 3 ticks
        int ticksSinceRun = 0;

        for (int tick = 1; tick <= 9; tick++)
        {
            ticksSinceRun++;

            // 1) The job is DUE only when enough ticks have passed.
            if (ticksSinceRun ___ interval)   // 👉 due when count reaches the interval: >=
            {
                Console.WriteLine($"Tick {tick}: running recurring job");

                // 2) Reset the counter so we wait a full interval again.
                ticksSinceRun = ___;          // 👉 start counting from zero again: 0
            }
            else
            {
                Console.WriteLine($"Tick {tick}: not due yet");
            }
        }

        // ✅ Expected output:
        // Tick 1: not due yet
        // Tick 2: not due yet
        // Tick 3: running recurring job
        // Tick 4: not due yet
        // Tick 5: not due yet
        // Tick 6: running recurring job
        // Tick 7: not due yet
        // Tick 8: not due yet
        // Tick 9: running recurring job
    }
}

3. IHostedService & BackgroundService

.NET has the worker loop built in. IHostedService is the interface for "something that starts when the app starts and stops when it stops"; BackgroundService is the convenient base class — you just override ExecuteAsync and write your loop. Pair it with a Channel<T> (a thread-safe async queue) and a controller can enqueue work that the worker drains in the background. Note the cancellation token and the try/catch: a real worker must shut down gracefully and must never die because one job threw.

// ══════════════════════════════════════════════
// IHostedService / BackgroundService — built into .NET
// ══════════════════════════════════════════════
// A BackgroundService is a long-running task the .NET host starts at
// boot and stops on shutdown. No external dependencies needed — perfect
// for an in-process worker loop draining a queue.

using System;
using System.Threading;
using System.Threading.Tasks;
using System.Threading.Channels;
using Microsoft.Extensions.Hosting;

public record EmailJob(string To, string Subject);

public class EmailWorker : BackgroundService
{
    private readonly Channel<EmailJob> _queue;   // a thread-safe async queue

    public EmailWorker(Channel<EmailJob> queue) => _queue = queue;

    // ExecuteAsync runs once when the app starts and loops until shutdown.
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        // ReadAllAsync waits for items and yields them as they arrive.
        await foreach (var job in _queue.Reader.ReadAllAsync(stoppingToken))
        {
            try
            {
                Console.WriteLine($"Sending email to {job.To}: {job.Subject}");
                // await emailSender.SendAsync(job.To, job.Subject);  // the slow bit
            }
            catch (Exception ex)
            {
                // Log and CONTINUE — one bad job must not kill the worker.
                Console.WriteLine($"Failed: {ex.Message}");
            }
        }
    }
}

// Program.cs — register the queue and the worker
// builder.Services.AddSingleton(Channel.CreateUnbounded<EmailJob>());
// builder.Services.AddHostedService<EmailWorker>();

// A controller just enqueues and returns immediately (fire-and-forget):
// await _queue.Writer.WriteAsync(new EmailJob("[email protected]", "Welcome!"));

// ✅ Expected output (when two jobs are enqueued):
// Sending email to [email protected]: Welcome!
// Sending email to [email protected]: Your receipt

4. Hangfire — persistent jobs with retries

A plain BackgroundService keeps its queue in memory, so a restart loses every pending job. Hangfire fixes that by storing jobs in a database: they survive restarts, retry automatically when they throw, and show up in a built-in dashboard at /hangfire. All three job shapes have a one-liner — BackgroundJob.Enqueue for fire-and-forget, BackgroundJob.Schedule for delayed, and RecurringJob.AddOrUpdate (with a stable id) for recurring.

// ══════════════════════════════════════════════
// Hangfire — persistent jobs with automatic retries + dashboard
// ══════════════════════════════════════════════
// Hangfire stores every job in a database (SQL Server, PostgreSQL,
// Redis...). Jobs SURVIVE a restart, retry automatically on failure,
// and you get a /hangfire dashboard showing history.

using System;
using Hangfire;

// Program.cs — setup
// builder.Services.AddHangfire(cfg => cfg
//     .UseSqlServerStorage(connectionString));
// builder.Services.AddHangfireServer();
// app.UseHangfireDashboard("/hangfire");

// ── Fire-and-forget: run once, now, in the background ──
BackgroundJob.Enqueue<IEmailSender>(
    sender => sender.Send("[email protected]", "Welcome!"));

// ── Delayed: run once, after a delay ──
BackgroundJob.Schedule<IEmailSender>(
    sender => sender.Send("[email protected]", "How are you finding it?"),
    TimeSpan.FromDays(3));

// ── Recurring: run on a schedule (CRON), identified by a stable id ──
// The id ("nightly-report") means re-deploying does NOT create a duplicate.
RecurringJob.AddOrUpdate<IReportService>(
    "nightly-report",
    svc => svc.GenerateDailyReport(),
    Cron.Daily(hour: 2, minute: 0));   // 02:00 every day

public interface IEmailSender { void Send(string to, string subject); }
public interface IReportService { void GenerateDailyReport(); }

// ✅ Expected behaviour:
// - The welcome email is sent within seconds (fire-and-forget).
// - The follow-up email is queued and sent 3 days later (delayed).
// - GenerateDailyReport runs at 02:00 daily, retrying if it throws.
// - All three appear in the /hangfire dashboard with their state.

5. Quartz.NET — enterprise scheduling

Quartz.NET is the heavyweight option. It splits the job (a class implementing IJob) from the trigger (when it fires), so one job can have many triggers, each with a full CRON expression and misfire handling for runs missed during downtime. It also supports clustering, so a job runs on exactly one node in a farm. Reach for it when Hangfire's scheduling isn't expressive enough — otherwise Hangfire is simpler.

// ══════════════════════════════════════════════
// Quartz.NET — enterprise scheduling (rich CRON, clustering)
// ══════════════════════════════════════════════
// Quartz separates the JOB (what to do) from the TRIGGER (when to do it).
// One job can have several triggers; triggers support full CRON and
// "misfire" handling for missed runs.

using System.Threading.Tasks;
using Quartz;

// A job: implement IJob and put the work in Execute.
public class InvoiceJob : IJob
{
    public Task Execute(IJobExecutionContext context)
    {
        // Make jobs idempotent: safe to run again if Quartz refires it.
        System.Console.WriteLine("Generating invoices...");
        return Task.CompletedTask;
    }
}

// Program.cs — register the job and a CRON trigger
// builder.Services.AddQuartz(q =>
// {
//     var jobKey = new JobKey("invoice-job");
//     q.AddJob<InvoiceJob>(opts => opts.WithIdentity(jobKey));
//     q.AddTrigger(opts => opts
//         .ForJob(jobKey)
//         .WithIdentity("invoice-trigger")
//         .WithCronSchedule("0 0 9 1 * ?"));  // 09:00 on the 1st of each month
// });
// builder.Services.AddQuartzHostedService(o => o.WaitForJobsToComplete = true);

// ✅ Expected behaviour:
// On the 1st of every month at 09:00, Quartz fires the trigger, which
// runs InvoiceJob.Execute and prints "Generating invoices...".
// WaitForJobsToComplete = true lets in-flight jobs finish on shutdown.

🔎 Deep Dive: idempotency, retries & persistence

Background jobs will run more than once. A worker can crash after doing the work but before marking it done, so the job runs again on retry. That means every job must be idempotent — running it twice has the same effect as running it once. "Charge the card" is dangerous; "charge the card if this order isn't already paid" is safe.

Retries are your safety net for transient failures (a network blip, a locked row). Hangfire and Quartz retry automatically; in a hand-rolled worker you add it yourself, usually with exponential backoff (wait 1s, then 2s, then 4s…). After a few failures a job is "poison" — move it to a dead-letter store and alert, rather than retrying forever.

// Idempotent: a guard makes a second run a no-op.
if (order.IsPaid) return;          // already done — nothing to do
order.Charge();
order.IsPaid = true;

// Poison handling: stop after N attempts, don't loop forever.
if (attempt >= maxAttempts) {
    MoveToDeadLetter(job);          // park it, alert a human
    return;
}

Persistence is what makes all this survive a restart. A queue in memory vanishes when the process dies; a queue in a database (Hangfire) or on a broker (RabbitMQ, Azure Service Bus) is still there when the app comes back. If losing a job would matter, the queue must be persistent.

Pro Tips

• 💡 Never do slow work in the request: enqueue it and return. The user gets a fast response and the work retries on its own if it fails.
• 💡 Write every job idempotent: assume it can run twice. Guard with a check (if (order.IsPaid) return;) so a retry is harmless.
• 💡 Pass IDs, not objects: Hangfire and Quartz serialise arguments. Pass an orderId and reload the order inside the job, not a fat object graph.
• 💡 Give recurring jobs a stable id: RecurringJob.AddOrUpdate("nightly-report", ...) — the id stops re-deploys creating duplicate schedules.
• 💡 Honour the cancellation token: pass stoppingToken to your awaits so the worker stops promptly and cleanly on shutdown.
• 💡 Keep jobs small and DI-friendly: create a scope inside the job for scoped services like a DbContext; don't capture them across loop iterations.

Common Errors (and the fix)

• Doing long work on the request thread: sending an email or building a PDF inside a controller blocks the response and risks a timeout. Enqueue a job and return immediately — let the worker do the slow part.
• Non-idempotent jobs: a job that charges a card or sends an email unconditionally will double-charge or double-send when it retries. Guard it (if (order.IsPaid) return;) so a second run is a no-op.
• No retry or poison handling: letting a job die on the first transient error loses work; retrying forever on a permanent error spins the worker and floods logs. Retry with backoff, then dead-letter after N attempts.
• Losing jobs on restart: an in-memory queue (a plain Queue<T> or Channel<T>) is gone the moment the process stops. If the work matters, use a persistent store — Hangfire's database or a message broker.
• "Cannot resolve scoped service from root provider": you injected a scoped service (like DbContext) into a singleton BackgroundService. Inject IServiceProvider and call provider.CreateScope() inside the loop instead.

📋 Quick Reference

Frequently Asked Questions

Q: Can't I just use Task.Run to do work in the background?

For a quick, fire-and-forget bit of work it sometimes seems fine, but it's risky: the work isn't persisted, won't retry, and can be killed mid-flight when the app recycles. Use a real job system (BackgroundService + a queue, or Hangfire) so work survives and retries.

Q: What does "idempotent" actually mean here?

A job is idempotent if running it twice has the same effect as running it once. Because retries can re-run a job, design each one so a repeat is harmless — usually by checking whether the work is already done before doing it.

Q: BackgroundService or Hangfire — how do I choose?

Use BackgroundService for simple, in-process work where losing a job on restart is acceptable. The moment jobs must survive restarts, retry automatically, or be visible in a dashboard, switch to Hangfire.

Q: Why pass an ID into a job instead of the whole object?

Hangfire and Quartz serialise job arguments to the database. A big object can fail to serialise or go stale by the time the job runs. Pass a small ID and reload the current data inside the job.

Q: What's a "poison" message?

A job that fails every time it runs — bad data, a permanent error. Retrying it forever wastes resources, so after a few attempts you move it to a dead-letter store and alert a human, instead of looping.

Mini-Challenge: a tiny scheduler

No blanks this time — just a brief and an outline. Build a tiny scheduler that holds a few jobs, each with its own interval, and on every tick runs the ones that are due while skipping the rest. This is exactly the bookkeeping a real recurring scheduler does. Run it and check your output against the expected lines in the comments.

using System;
using System.Collections.Generic;

// 🎯 MINI-CHALLENGE: a tiny scheduler
// Each job has a name, an interval (how often it should run) and a
// "ticksSinceRun" counter. On every tick:
//   - add 1 to every job's ticksSinceRun
//   - if ticksSinceRun >= interval: run it (print "Running NAME") and reset to 0
//   - otherwise skip it
//
// Set up two jobs:  "heartbeat" every 2 ticks,  "report" every 5 ticks.
// Loop ticks 1..5 and let the scheduler decide what runs each tick.
//
// ✅ Expected output:
// -- tick 1 --
// -- tick 2 --
// Running heartbeat
// -- tick 3 --
// -- tick 4 --
// Running heartbeat
// -- tick 5 --
// Running report

class Job
{
    public string Name;
    public int Interval;
    public int TicksSinceRun;
    // hint: a constructor that sets Name and Interval and starts TicksSinceRun at 0
}

class Program
{
    static void Main()
    {
        // 1. Build a List<Job> with the two jobs described above.
        // 2. for (int tick = 1; tick <= 5; tick++) { print the tick header,
        //    then for each job: bump the counter, run-and-reset if due. }

        // your code here
    }
}

🎉 Lesson Complete

• ✅ Offload slow work off the request thread — enqueue and return immediately
• ✅ Three shapes: fire-and-forget, delayed, and recurring
• ✅ Every job system is a queue plus a worker that drains it (FIFO)
• ✅ IHostedService/BackgroundService give you a built-in long-running worker
• ✅ Hangfire adds persistence, automatic retries, and a dashboard; Quartz.NET adds rich scheduling
• ✅ Make jobs idempotent, add retries with poison handling, and persist anything you can't lose

Practice quiz

Continue this course

• Previous: Microservices in .NET (gRPC, Messaging, Resilience)
• Next: Performance Profiling & Benchmarking (BenchmarkDotNet) — Measure and optimise .NET performance with BenchmarkDotNet and dotTrace
• Quick reference: C# cheat sheet