Benchmarking with Criterion
Guessing at performance is how regressions sneak in. Criterion gives Rust statistically rigorous benchmarks — on stable — so you measure real timings, catch slowdowns, and prove your optimizations actually worked.
Learn Benchmarking with Criterion in our free Rust course — a beginner-friendly interactive lesson with worked examples, a practice exercise and a quick…
Part of the free Rust course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
In this lesson you'll set up a benches/ target, write a Criterion benchmark with criterion_group! and criterion_main!, use b.iter and black_box correctly, and read the statistical reports.
What You'll Learn in This Lesson
1️⃣ Setting Up a benches/ Target
Criterion is a dev-dependency , and benchmark files live in a benches/ directory. A [[bench]] section with harness = false lets Criterion supply its own main .
With that in place, cargo bench compiles in release mode and runs every benchmark target.
2️⃣ Writing a Benchmark
A benchmark function takes &mut Criterion , registers work with bench_function , and times the closure passed to b.iter . The criterion_group! and criterion_main! macros build the harness.
Criterion reports a low/median/high interval, not a single number, because it samples many runs and analyzes the distribution statistically.
3️⃣ black_box and Avoiding Pitfalls
The biggest benchmarking trap is the optimizer deleting work whose result is unused, producing impossibly fast times. black_box is an opaque barrier: wrap the input so it can't be precomputed, and let the output escape so it can't be removed.
When the html_reports feature is on, Criterion also writes plots and a comparison against the previous run under target/criterion .
Write a small function and a Criterion benchmark that times it with black_box .
📋 Quick Reference — Criterion
Practice quiz
What is Criterion used for in Rust?
- Statistics-driven benchmarking on stable Rust
- Unit testing
- Code formatting
- Async scheduling
Answer: Statistics-driven benchmarking on stable Rust. Criterion is a benchmarking library that runs on stable Rust and analyzes timings statistically.
Where do Criterion benchmark files conventionally live?
- In src/
- In a tests/ directory
- In a benches/ directory
- In examples/
Answer: In a benches/ directory. Cargo looks for benchmark targets in the benches/ directory at the crate root.
Which command runs your benchmarks?
- cargo test
- cargo bench
- cargo run --bench
- cargo check
Answer: cargo bench. cargo bench compiles and runs the benchmark targets, including Criterion ones.
What does black_box do in a benchmark?
- Logs results
- Hides output
- Stops the timer
- Prevents the optimizer from eliminating the code under test
Answer: Prevents the optimizer from eliminating the code under test. black_box is an opaque barrier that stops the compiler from optimizing away the work being measured.
What does Bencher::iter (the closure passed to b.iter) do?
- Repeatedly runs the routine so Criterion can time it
- Sorts the input
- Spawns threads
- Runs the routine once
Answer: Repeatedly runs the routine so Criterion can time it. b.iter runs the supplied closure many times so Criterion can collect a reliable timing sample.
What pair of macros wires up a Criterion benchmark binary?
- main! and group!
- criterion_group! and criterion_main!
- bench_start! and bench_end!
- setup! and teardown!
Answer: criterion_group! and criterion_main!. criterion_group! collects benchmark functions and criterion_main! generates the harness main.
Why is Criterion's built-in alternative, #[bench], not commonly used?
- It needs the internet
- It is deprecated and removed
- It only works on Windows
- It requires unstable nightly Rust
Answer: It requires unstable nightly Rust. The built-in #[bench] attribute and test::Bencher are unstable and only available on nightly.
What kind of report can Criterion generate after a run?
- A SQL dump
- A PDF invoice
- HTML reports with plots and statistics
- A Dockerfile
Answer: HTML reports with plots and statistics. Criterion produces detailed HTML reports including plots, distributions, and regression analysis.
How does Criterion help detect performance changes between runs?
- It deletes old results
- It compares against saved previous results and reports regressions/improvements
- It only prints the latest time
- It requires a paid service
Answer: It compares against saved previous results and reports regressions/improvements. Criterion stores baseline results and statistically compares new runs to flag regressions or speedups.
A common pitfall when benchmarking pure functions is...
- The compiler optimizing the work away because the result is unused
- Forgetting to import std
- Running on a laptop
- Using too many threads
Answer: The compiler optimizing the work away because the result is unused. If a result is never used, the optimizer can delete the computation; wrapping inputs/outputs in black_box prevents this.