Streams, Buffers & Pipelines

Reviewed & published by Brayan K

By the end of this lesson you'll be able to read and write data with C#'s stream types — StreamReader, StreamWriter, BufferedStream and MemoryStream — dispose them cleanly with using, and understand why flushing and rewinding matter. These are the tools behind every file, network and pipeline read in .NET.

Part of the free C# course at LearnCodingFast — hands-on lessons with worked examples and the output they print, plus practice exercises and a quick quiz.

What You'll Learn

• Explain what a Stream is — a one-way flow of bytes from a source
• Write and read text with StreamWriter and StreamReader
• Use File.ReadAllText / WriteAllText for quick whole-file work
• Wrap a stream in a BufferedStream to batch small operations
• Use MemoryStream to read and write entirely in memory
• Dispose streams safely with using, and know when to Flush and rewind

💡 Real-World Analogy

A stream is a conveyor belt in a factory. Data flows past you piece by piece — you never need to see the whole shipment at once, which is how a program can process a file far bigger than its memory. A buffer is the loading dock beside the belt: items pile up there and get moved in one bulk trip instead of one box at a time, which is far more efficient. A StreamWriter is the worker placing boxes on the belt; a StreamReader is the worker taking them off. And Flush is shouting "send what's on the dock right now!" — until you do, the last few boxes may still be sitting there, not yet on the belt.

The Stream Family at a Glance

Every I/O type in .NET is built on the abstract Stream class — a flow of bytes you can read, write, and (sometimes) seek within. Different streams have different sources, and helper classes layer text or buffering on top. Here's the cast you'll meet in this lesson.

The clever part: because they all share the Stream base type, a StreamReader doesn't care whether it's reading a file, a network socket, or memory. Learn the pattern once and it works everywhere.

1. What a Stream Is — Write, Flush, Rewind, Read

A stream is a one-way flow of bytes with an internal cursor marking your current position. You write bytes in or read bytes out, and the cursor moves along as you go. A StreamWriter sits on top and converts your text into bytes; a StreamReader converts bytes back into text. To keep this lesson runnable without a real file, we use a MemoryStream — a stream whose source is just an array in memory. Read this worked example carefully: it shows the full round trip and the two steps beginners forget — Flush and rewinding the cursor.

using System;
using System.IO;
using System.Text;

class Program
{
    static void Main()
    {
        // A STREAM is a one-way flow of bytes you read from or write to.
        // The SOURCE can be a file, the network, or — here — memory.
        // We use a MemoryStream so this runs anywhere, with no real file.

        using var ms = new MemoryStream();   // an empty, growable byte buffer

        // === WRITE ===
        // A StreamWriter turns your text into bytes and pushes them in.
        // 'leaveOpen: true' keeps the MemoryStream usable after the writer
        // is disposed, so we can rewind and read it back below.
        using (var writer = new StreamWriter(ms, Encoding.UTF8, leaveOpen: true))
        {
            writer.WriteLine("Hello from a StreamWriter!");
            writer.WriteLine("Streams move data piece by piece.");
            writer.Flush();   // PUSH buffered text into the stream NOW
        }

        // === REWIND ===
        // After writing, the stream's cursor sits at the END.
        // Reading from there gives you nothing, so move it back to 0.
        ms.Position = 0;   // 👈 the step beginners forget the most

        // === READ ===
        // A StreamReader turns the bytes back into text for you.
        using (var reader = new StreamReader(ms, Encoding.UTF8))
        {
            string text = reader.ReadToEnd();   // pull every remaining byte
            Console.Write(text);
        }

        // ✅ Expected output:
        //    Hello from a StreamWriter!
        //    Streams move data piece by piece.
    }
}

Your turn. The program below does the same round trip but is missing two pieces. Fill in the two ___ blanks using the hints, then run it. If you see nothing printed, you forgot to rewind.

using System;
using System.IO;
using System.Text;

class Program
{
    static void Main()
    {
        // 🎯 YOUR TURN — fill in the blanks marked with ___, then run it.
        // Goal: write one line into memory, rewind, and read it back.

        using var ms = new MemoryStream();

        // 1) Write a line of text into the stream.
        using (var writer = new StreamWriter(ms, Encoding.UTF8, leaveOpen: true))
        {
            writer.WriteLine("Saved without touching the disk!");
            writer.___();          // 👉 push buffered text out now:  Flush
        }

        // 2) Move the cursor back to the start so the read sees your text.
        ms.Position = ___;         // 👉 the very first byte is at position  0

        // 3) Read everything back and print it.
        using (var reader = new StreamReader(ms, Encoding.UTF8))
        {
            Console.Write(reader.ReadToEnd());
        }

        // ✅ Expected output:
        //    Saved without touching the disk!
    }
}

2. Reading & Writing Files — Helpers and StreamReader

For small files, .NET gives you one-line helpers: File.WriteAllText replaces a file's contents, and File.ReadAllText / File.ReadAllLines pull it all back. They're perfect for config files and quick scripts. But they load the entire file into memory at once, so for large files you read line by line with a StreamReader inside a using block. The using is not optional politeness — it guarantees the operating-system file handle is closed even if an exception is thrown halfway through.

using System;
using System.IO;

class Program
{
    static void Main()
    {
        string path = "notes.txt";

        // === Whole-file helpers — shortest path for SMALL files ===
        // WriteAllText replaces the file's entire contents in one call.
        File.WriteAllText(path, "First line\nSecond line\nThird line");

        // ReadAllText pulls the WHOLE file into one string.
        string everything = File.ReadAllText(path);
        Console.WriteLine($"Characters in file: {everything.Length}");  // Characters in file: 33

        // ReadAllLines splits on newlines into a string[].
        string[] lines = File.ReadAllLines(path);
        Console.WriteLine($"Line count: {lines.Length}");               // Line count: 3

        // === Line-by-line with a StreamReader — scales to HUGE files ===
        // 'using' guarantees the file handle is released even if an error
        // is thrown — no leaked handles, no "file in use" surprises.
        Console.WriteLine("--- reading with StreamReader ---");
        using (var reader = new StreamReader(path))
        {
            string? line;
            int n = 1;
            while ((line = reader.ReadLine()) != null)   // null = end of file
            {
                Console.WriteLine($"  {n++}: {line}");
            }
        }
        // ✅ Expected output:
        //    Characters in file: 33
        //    Line count: 3
        //    --- reading with StreamReader ---
        //      1: First line
        //      2: Second line
        //      3: Third line

        File.Delete(path);   // tidy up the demo file
    }
}

Now you try. The buffer below already has three lines written into it; your job is to read them back and process them by counting how many there are. Fill in the two ___ blanks:

using System;
using System.IO;
using System.Text;

class Program
{
    static void Main()
    {
        // 🎯 YOUR TURN — count how many lines are in an in-memory buffer.
        // The data is already written for you; you just read and count.

        using var ms = new MemoryStream();
        using (var writer = new StreamWriter(ms, Encoding.UTF8, leaveOpen: true))
        {
            writer.WriteLine("apple");
            writer.WriteLine("banana");
            writer.WriteLine("cherry");
            writer.Flush();
        }
        ms.Position = 0;   // rewind before reading

        int count = 0;
        using (var reader = new StreamReader(ms, Encoding.UTF8))
        {
            string? line;
            // 1) Loop until ReadLine() returns null (end of the stream).
            while ((line = reader.ReadLine()) != ___)   // 👉 the end marker:  null
            {
                count++;          // one more line seen
            }
        }

        // 2) Print the total. Use the count variable inside the braces.
        Console.WriteLine($"Read {___} lines");   // 👉 the variable:  count

        // ✅ Expected output:
        //    Read 3 lines
    }
}

3. Buffering — Doing Fewer, Bigger Trips

Every individual read or write to a raw stream can cost a trip to the disk or network — and those trips have fixed overhead, so a thousand tiny writes are far slower than a few big ones. A BufferedStream wraps another stream and accumulates small operations in memory, flushing them out in large chunks. You don't change what gets written — only how efficiently. (Note: StreamWriter and FileStream already buffer internally, so you mainly add a BufferedStream around streams that don't, such as a raw network stream.)

using System;
using System.IO;
using System.Text;

class Program
{
    static void Main()
    {
        // === BufferedStream — fewer, bigger trips to the source ===
        // Every read/write to a raw stream can be an expensive system call.
        // BufferedStream wraps another stream and batches small operations
        // into large ones, so 1,000 tiny writes become a handful of big ones.
        using var ms = new MemoryStream();

        // Careful with ownership: disposing a BufferedStream also disposes the
        // stream it wraps, and there is no leaveOpen option for it. So this one
        // is flushed rather than disposed — the MemoryStream has to survive to
        // be read back below. (Dispose it in a using and the next line throws
        // ObjectDisposedException: Cannot access a closed Stream.)
        var buffered = new BufferedStream(ms, bufferSize: 8192);
        using (var writer = new StreamWriter(buffered, Encoding.UTF8, leaveOpen: true))
        {
            for (int i = 1; i <= 3; i++)
                writer.WriteLine($"Row {i}");
        }
        buffered.Flush();   // push the batched bytes down into the MemoryStream

        // Read it back to prove the bytes really landed in the MemoryStream.
        ms.Position = 0;
        using (var reader = new StreamReader(ms, Encoding.UTF8))
        {
            Console.Write(reader.ReadToEnd());
        }

        // ✅ Expected output:
        //    Row 1
        //    Row 2
        //    Row 3
        //
        // Buffering doesn't change the RESULT — only how fast you get it.
    }
}

🔎 Deep Dive: why using, Flush, and rewind matter

Streams hold unmanaged resources — OS file handles, network sockets, locked buffers. The garbage collector cleans up normal C# objects, but it can't promptly release these. That's what IDisposable and using are for: the moment a using block ends, Dispose() runs, which flushes any buffered data and frees the handle — even if an exception was thrown.

Disposing a StreamWriter normally flushes and then closes the underlying stream too. That's why the examples pass leaveOpen: true — so disposing the writer flushes the text but leaves the MemoryStream open for us to rewind and read.

using var ms = new MemoryStream();          // disposed at end of method
using (var w = new StreamWriter(ms, leaveOpen: true))
{
    w.WriteLine("data");
    w.Flush();        // push text out NOW; Dispose would also do this
}                     // 👈 writer disposed here — ms stays open
ms.Position = 0;      // rewind — without this, the read returns nothing
// ... read ms ...

Two failure modes to remember: forget to Flush/dispose and your last writes never reach the stream; forget to set Position = 0 and the reader starts at the end, reading nothing.

Putting It Together: a Mini CSV Reader

Here's a small but real program that uses everything from this lesson at once — a MemoryStream built straight from text, a StreamReader in a using, line-by-line reading, and parsing each line into a total. This is exactly how you'd process a real CSV file; only the source would change from memory to a FileStream.

using System;
using System.Globalization;
using System.IO;
using System.Text;

class Program
{
    static void Main()
    {
        // === Mini "CSV" reader — total an order from an in-memory buffer ===
        // Each line is  product,quantity,unitPrice
        string data = "Keyboard,2,29.99\nMouse,1,14.50\nMonitor,1,189.00";

        // Build a stream straight from the text's UTF-8 bytes.
        using var ms = new MemoryStream(Encoding.UTF8.GetBytes(data));
        using var reader = new StreamReader(ms, Encoding.UTF8);

        decimal grandTotal = 0m;
        string? line;
        while ((line = reader.ReadLine()) != null)
        {
            string[] parts = line.Split(',');               // ["Keyboard","2","29.99"]
            int qty = int.Parse(parts[1]);
            decimal price = decimal.Parse(parts[2], CultureInfo.InvariantCulture);
            decimal lineTotal = qty * price;
            grandTotal += lineTotal;
            Console.WriteLine($"{parts[0],-10} {qty} x {price,7:F2} = {lineTotal,8:F2}");
        }

        Console.WriteLine(new string('-', 34));
        Console.WriteLine($"{"TOTAL",-10} {grandTotal,24:F2}");

        // ✅ Expected output:
        //    Keyboard   2 x   29.99 =    59.98
        //    Mouse      1 x   14.50 =    14.50
        //    Monitor    1 x  189.00 =   189.00
        //    ----------------------------------
        //    TOTAL                        263.48
    }
}

Swapping new MemoryStream(...) for File.OpenRead("orders.csv") turns this into a real file reader — the StreamReader loop doesn't change a single line. That's the power of programming against the Stream abstraction.

Beyond Streams: System.IO.Pipelines

When you need the absolute highest throughput — a web server parsing millions of request bodies, say — even buffered streams leave performance on the table. System.IO.Pipelines is a newer API built for exactly that. A Pipe has a PipeWriter and a PipeReader that share a pool of memory buffers, giving you zero-copy reads (no shuffling bytes between arrays), automatic back-pressure (the writer slows down if the reader falls behind), and far less garbage-collector pressure.

It's the engine inside Kestrel, ASP.NET Core's web server. You won't reach for it in everyday code — a StreamReader is simpler and plenty fast — but it's the tool to know about the day a profiler says stream parsing is your bottleneck.

// Sketch only — Pipelines trades simplicity for raw speed:
var pipe = new Pipe();
PipeWriter writer = pipe.Writer;   // fills shared buffers
PipeReader reader = pipe.Reader;   // reads them with zero copying
// reader.ReadAsync() hands you a ReadOnlySequence<byte> to parse in place

Pro Tips

• 💡 Always wrap streams in using: it disposes them — flushing and closing the handle — even when an exception is thrown.
• 💡 Rewind before you read: after writing to a MemoryStream, set Position = 0 or the reader starts at the end and gets nothing.
• 💡 Use leaveOpen: true when you want to keep a stream alive after disposing a StreamWriter/StreamReader wrapped around it.
• 💡 Read big files line by line with StreamReader.ReadLine() instead of File.ReadAllText to avoid loading gigabytes into memory.
• 💡 Prefer the async versions (ReadLineAsync, WriteLineAsync) for files in server apps so you don't block a thread on slow I/O.

Common Errors (and the fix)

• "System.IO.FileNotFoundException: Could not find file '...'": you tried to open a file for reading that doesn't exist (wrong path or wrong working directory). Check the path, and guard reads with if (File.Exists(path)) before opening.
• Nothing comes out / empty read: you wrote to a MemoryStream but forgot ms.Position = 0; before reading. The cursor is at the end, so there's nothing left to read. Rewind first.
• Last lines missing from the file: you didn't dispose or Flush the StreamWriter, so its buffer never reached the stream. Wrap it in using (or call Flush()) so the buffered text is written.
• "System.ObjectDisposedException: Cannot access a closed Stream": you used a stream after its using block ended, or a StreamWriter closed the underlying stream. Keep usage inside the block, or pass leaveOpen: true.
• "System.IO.IOException: The process cannot access the file ... because it is being used by another process": a previous handle wasn't released. This is exactly what using prevents — make sure every stream is disposed.

📋 Quick Reference

Frequently Asked Questions

Q: When should I use File.ReadAllText instead of a StreamReader?

Use File.ReadAllText for small files where loading everything at once is fine — config, small data files, quick scripts. Switch to a StreamReader with ReadLine() when the file is large, so you process one line at a time instead of pulling gigabytes into memory.

Q: Why does my MemoryStream read return nothing?

Almost always because you didn't rewind. Writing leaves the cursor at the end of the stream; reading from there yields nothing. Set ms.Position = 0; after writing and before reading.

Q: Do I really need Flush() if I'm using using?

Disposing (which using does at the end of the block) flushes for you, so an explicit Flush() is only needed when you want the data pushed out before the block ends — for example, to rewind and read the same stream straight away.

Q: What's the difference between a Stream and a StreamReader?

A Stream deals in raw bytes. A StreamReader/StreamWriter wraps a stream and handles the text encoding for you, so you work with strings and lines instead of byte arrays.

Q: When would I use System.IO.Pipelines over streams?

Only in high-throughput, performance-critical code — parsing huge volumes of network data, for instance. For everyday file work a StreamReader is simpler and fast enough; Pipelines trades that simplicity for zero-copy reads and back-pressure.

Mini-Challenge: Average a Column

No blanks this time — just a brief and an outline to keep you on track. Read the in-memory buffer of numbers line by line, total them, count them, and print the average. Run it and check your output against the expected line in the comments. This is the same loop you'd use over a real file of readings or scores.

using System;
using System.IO;
using System.Text;

class Program
{
    static void Main()
    {
        // 🎯 MINI-CHALLENGE: average a column of numbers from a buffer
        // The data below is one number per line. Compute the AVERAGE.
        //
        // 1. Make a MemoryStream from these UTF-8 bytes (done for you).
        // 2. Wrap it in a StreamReader.
        // 3. Loop with ReadLine() until it returns null.
        //    - Parse each line with int.Parse(line)
        //    - Add it to a running 'sum' and increase a 'count'
        // 4. Print:  "Average: {sum / (double)count:F2}"
        //    (cast to double so you get a decimal, not integer division)
        //
        // ✅ Expected output:
        //    Average: 30.00

        string numbers = "10\n20\n30\n40\n50";
        using var ms = new MemoryStream(Encoding.UTF8.GetBytes(numbers));

        // your code here
    }
}

🎉 Lesson Complete

• ✅ A stream is a one-way flow of bytes with a moving cursor
• ✅ StreamWriter writes text as bytes; StreamReader reads bytes back as text
• ✅ File.ReadAllText / WriteAllText are the quick path for small files
• ✅ BufferedStream batches small operations into big ones for speed
• ✅ MemoryStream gives you a stream with no disk — ideal for tests and buffers
• ✅ using disposes streams safely; Flush and Position = 0 are the round-trip essentials
• ✅ System.IO.Pipelines is the zero-copy, back-pressure-aware option for extreme throughput

Practice quiz

Continue this course

• Previous: Logging Architecture (ILogger, Providers, Serilog)
• Next: Advanced JSON Processing with System.Text.Json — Serialise, deserialise, and customise JSON processing without Newtonsoft
• Quick reference: C# cheat sheet