Worker Pools & Pipelines

A worker pool is a fixed set of goroutines that pull jobs from a shared channel and push results to another, letting you process a large workload concurrently while capping how many run at once — and a pipeline chains stages together so data flows through them one step at a time.

Learn Worker Pools & Pipelines in our free Go course — an interactive lesson with runnable examples, a practice exercise and a quick reference.

Part of the free Go course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Learn in This Lesson

1️⃣ The classic worker pool

The shape is always the same: a jobs channel, N worker goroutines that range over it, and a results channel they write to. A WaitGroup tracks the workers. Note the channel directions in the worker signature: <-chan int (receive-only jobs) and chan<- int (send-only results) document intent and let the compiler catch misuse.

2️⃣ Bounding concurrency with a semaphore

Sometimes you don't want a pool of pre-spawned workers — you want to launch a goroutine per task but cap how many run at once (say, to respect an API rate limit). A buffered channel works as a counting semaphore : its capacity is the limit. Send an empty struct{}{} to acquire a slot before working, receive to release it. Once full, further acquires block.

3️⃣ The pipeline pattern

A pipeline composes stages . Each stage is a function that takes a receive-only input channel, launches a goroutine, and returns a receive-only output channel — so you wire them up by nesting calls: double(keepEven(gen(...))) . Data flows one direction; closing each output channel propagates "no more values" downstream. No shared state, no mutex.

🎯 Your Turn

The pool is almost complete — but the workers will block forever waiting for more jobs. Add the one line that tells them no more are coming. Fill in the blank marked ___ .

❌ Never closing jobs — every worker's range blocks forever; deadlock.

❌ Closing results too early (before workers finish) — "panic: send on closed channel".

✅ Close it after wg.Wait() , in a goroutine so main keeps draining.

❌ Unbuffered results with no reader — workers block on send and never call Done() .

✅ Drain results concurrently, or buffer it to numJobs .

❌ Expecting ordered output from concurrent workers.

✅ Collect into a slice and sort , or key results by job ID and reassemble.

{'sem := make(chan struct , 3) // what does 3 mean?'}

At most 3 goroutines hold a slot at once. The fourth sem <- struct{}{} blocks until someone receives (releases) a slot.

A goroutine running wg.Wait(); close(results) — i.e. once every worker has called Done() . Never close it from a worker.

Build a 4-worker pool where each worker sums the decimal digits of its number. Feed in 12, 305, 99, 7, 4444 , then collect and sort the results.

Practice quiz

What is a worker pool in Go?

A single goroutine that loops over tasks
A pool of database connections
A fixed set of goroutines reading from a shared jobs channel
A buffered channel of results

Answer: A fixed set of goroutines reading from a shared jobs channel. N workers all range over one jobs channel and write to results.

Why bound the number of workers?

To get bounded concurrency that protects shared resources
To make output ordered
To avoid using channels
To reduce code size

Answer: To get bounded concurrency that protects shared resources. However many jobs arrive, only N run at once.

Who closes the jobs channel?

Each worker when it finishes
The runtime automatically
The results goroutine
The producer, after sending the last job

Answer: The producer, after sending the last job. Closing jobs lets each worker's for range end naturally.

When is the results channel closed?

As soon as jobs is closed
After wg.Wait(), once every worker has finished
By the first worker to finish
Never; it stays open

Answer: After wg.Wait(), once every worker has finished. Use go func(){ wg.Wait(); close(results) }() so main can keep draining.

Why run wg.Wait() then close(results) in a separate goroutine?

So main can range over results without deadlocking
To speed up the workers
Because Wait blocks forever otherwise
To avoid a data race on score

Answer: So main can range over results without deadlocking. main drains results concurrently while the goroutine waits and closes.

What does make(chan struct{}, 3) used as a semaphore allow?

Exactly 3 total goroutines ever
3 results buffered
At most 3 goroutines holding a slot at once
3 retries per job

Answer: At most 3 goroutines holding a slot at once. The buffer capacity is the concurrency limit; a 4th acquire blocks.

How do you acquire a slot from a semaphore channel sem?

<-sem
sem <- struct{}{}
close(sem)
sem.Lock()

Answer: sem <- struct{}{}. Send to acquire (blocks if full); receive <-sem to release.

Why use chan struct{} rather than chan bool for a semaphore?

bool is slower to compare
bool channels cannot be buffered
struct{} is required by the compiler
struct{} occupies zero bytes; only the act of sending matters

Answer: struct{} occupies zero bytes; only the act of sending matters. An empty struct carries no data and wastes no memory.

What is a pipeline in Go?

A pool of identical workers
A chain of stages, each a goroutine connected by channels
A single buffered channel
A mutex-protected slice

Answer: A chain of stages, each a goroutine connected by channels. Stages like gen then filter then transform pass data one direction.

How do you get deterministic output from a worker pool?

Use an unbuffered results channel
Add more workers
Collect results into a slice and sort them
Close results before wg.Wait()

Answer: Collect results into a slice and sort them. You cannot control finish order, so collect-then-sort.