Files, Streams & Large Datasets
Published by Brayan K
Master efficient file handling, streaming patterns, and processing massive datasets that don't fit in memory.
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
Reading a file without holding all of it
A stream hands out data a piece at a time: a file read line by line, a compressed file unpacked as it's read, the text one program pipes into another. A program that works through a stream needs only the piece in front of it, so the memory it needs doesn't grow with its input. This lesson shows which ways of reading do that and which quietly keep everything, how to read files that have no lines, what text mode does to the bytes underneath, how to move around in a file, when written text really reaches it, and how compressed files, zip archives, CSV and JSON Lines fit in.
File Handling covered the basics this builds on: open() and its modes, encoding=, with, readline() and readlines(), and binary mode. Each example here writes the file it reads, since the editor starts every Run in an empty folder. Sizes are shown by counts the programs make themselves (lines kept, lines waiting, calls to read()), not by times or memory figures, which change from one computer and Python version to the next.
What holds the whole file, and what reads it a line at a time
A for loop over a file hands you one line, runs the loop body, and asks the file for the next line only when the body has finished. read() and readlines() return the whole file at once. On a small file you can't tell the difference. On a big one it decides whether the program fits in memory, and whether it can stop early.
This program writes a 100,000-line log with errors on lines 37 and 61,000, then looks for the first error both ways. Python ignores the underscores in numbers such as 100_001 and 61_000; they're only there to make long numbers easier to read:
with open("app.log", "w", encoding="utf-8") as f:
for n in range(1, 100_001):
level = "ERROR" if n in (37, 61_000) else "INFO"
f.write(f"{n} {level} request\n")
with open("app.log", encoding="utf-8") as f:
lines = f.readlines()
print("readlines() kept", len(lines), "lines")
with open("app.log", encoding="utf-8") as f:
for number, line in enumerate(f, start=1):
if " ERROR " in line:
break
print("the loop stopped at line", number)
print(line, end="")
# ✅ Expected output:
# readlines() kept 100000 lines
# the loop stopped at line 37
# 37 ERROR requestreadlines() read all 100,000 lines and kept them in a list, so a search through them could only start once the whole file was in memory. The loop read 37 lines and stopped: break ends it, and the rest of the file is never read. After the loop, number and line still hold the values from the turn that stopped it, which is why they can be printed.
Reading a line at a time isn't enough on its own, though. Some code takes the lines one by one and still keeps them all: sorted() has to see every line before it can hand back the first. This program counts how many lines have been read from the file but not yet handled by the loop body. lines_from() is a generator, as in Generators & Iterators Mastery, that counts each line it hands out:
with open("app.log", "w", encoding="utf-8") as f:
for n in range(1, 100_001):
level = "ERROR" if n in (37, 61_000) else "INFO"
f.write(f"{n} {level} request\n")
stats = {"read": 0, "done": 0, "most": 0}
def lines_from(f):
for line in f:
stats["read"] += 1
yield line
def handle(line):
stats["most"] = max(stats["most"], stats["read"] - stats["done"])
stats["done"] += 1
with open("app.log", encoding="utf-8") as f:
for line in lines_from(f):
handle(line)
print("streamed: at most", stats["most"], "line waiting")
stats = {"read": 0, "done": 0, "most": 0}
with open("app.log", encoding="utf-8") as f:
for line in sorted(lines_from(f)):
handle(line)
print("sorted: at most", stats["most"], "lines waiting")
# ✅ Expected output:
# streamed: at most 1 line waiting
# sorted: at most 100000 lines waitingStreamed, each line was handled as soon as it was read, so no more than one was ever waiting. Through sorted(), all 100,000 were read before the first was handled. These keep the whole file in memory in the same way: read(), readlines(), list(f), sorted(f), json.load(f) and list() of a csv reader. These take a line at a time and keep only what they need: a for loop, sum(1 for line in f), max(f, key=len), any() over a generator expression, and next(f).
To work through a big file a batch of lines at a time, use the islice() loop from itertools. An open file already remembers its place, so list(islice(f, 1000)) takes the next 1,000 lines each time, and an empty batch means the file is used up.
Memory Management & Garbage Collection Internals measured memory itself with tracemalloc. Measured that way on the same 100,000-line log, the peak while counting with readlines() was many times the peak while counting in a loop: more than ten times in every run made for this lesson, on Python 3.10 to 3.14 and in this site's editor. How many times depends on the Python version and the computer, so compare the figures, as that lesson did, rather than print them.
A file object hands out lines, but it isn't a list. It has no length:
with open("todo.txt", "w", encoding="utf-8") as f:
f.write("Buy milk\nWalk the dog\n")
with open("todo.txt", encoding="utf-8") as f:
print(len(f))
# ✅ Expected error:
# TypeError: object of type '_io.TextIOWrapper' has no len()It has no index or reverse order either: f[0] stops with TypeError: '_io.TextIOWrapper' object is not subscriptable, and reversed(f) with TypeError: '_io.TextIOWrapper' object is not reversible. Count lines as they go past, with sum(1 for line in f), and reach the end of a big file with seek(), shown further down.
A line at a time is only as small as the file's lines. A minified JSON file, its data squeezed onto one line, has no line breaks at all, so the loop gets the whole file as its first and only line:
with open("minified.json", "w", encoding="utf-8") as f:
f.write("[" + ",".join(str(n) for n in range(200_000)) + "]")
with open("minified.json", encoding="utf-8") as f:
for line in f:
print("one line of", len(line), "characters")
# ✅ Expected output:
# one line of 1288891 charactersThat's 1,288,891 characters in a single line. A file like that, or one that isn't text at all, is read in pieces of a size you choose.
read(n) for files that aren't lines
f.read(n) returns at most n characters in text mode, or n bytes in binary mode, and an empty string, or b"", once the file is used up. In a while True loop that stops on that empty result, it reads a file of any size in pieces no bigger than n. Images, archives and other binary files have no lines, so this is how they're read. The CountingReader class below, written as in Object-Oriented Programming, wraps a file, passes each read() on to it, and counts the calls and the biggest piece returned:
class CountingReader:
def __init__(self, f):
self.f = f
self.calls = 0
self.biggest = 0
def read(self, size=-1):
data = self.f.read(size)
self.calls += 1
self.biggest = max(self.biggest, len(data))
return data
with open("photo.bin", "wb") as f:
f.write(b"x" * 1_000_000)
with open("photo.bin", "rb") as f:
reader = CountingReader(f)
data = reader.read()
print("read():", reader.calls, "call, biggest piece", reader.biggest)
with open("photo.bin", "rb") as f:
reader = CountingReader(f)
while True:
chunk = reader.read(65_536)
if not chunk:
break
print("read(65_536):", reader.calls, "calls, biggest piece", reader.biggest)
# ✅ Expected output:
# read(): 1 call, biggest piece 1000000
# read(65_536): 17 calls, biggest piece 65536A single read() returned all 1,000,000 bytes at once. The loop made 17 calls: 15 full pieces of 65,536 bytes, one of the 16,960 bytes left over, and a last call that returned b"" and ended the loop. However big the file, the loop never holds more than 65,536 bytes of it. The same loop reads the one-line JSON file in text mode, counting the commas between its numbers as it goes:
with open("minified.json", "w", encoding="utf-8") as f:
f.write("[" + ",".join(str(n) for n in range(200_000)) + "]")
commas = 0
biggest = 0
with open("minified.json", encoding="utf-8") as f:
while True:
chunk = f.read(65_536)
if not chunk:
break
commas += chunk.count(",")
biggest = max(biggest, len(chunk))
print(commas + 1, "numbers, at most", biggest, "characters at once")
# ✅ Expected output:
# 200000 numbers, at most 65536 characters at onceA shorter way to write the loop hides a trap. iter() with two arguments, iter(func, stop), calls func() with no arguments, again and again, until it returns stop. Given f.read, it calls read() with no size, which reads everything:
class CountingReader:
def __init__(self, f):
self.f = f
self.calls = 0
self.biggest = 0
def read(self, size=-1):
data = self.f.read(size)
self.calls += 1
self.biggest = max(self.biggest, len(data))
return data
with open("photo.bin", "wb") as f:
f.write(b"x" * 1_000_000)
with open("photo.bin", "rb") as f:
reader = CountingReader(f)
for chunk in iter(reader.read, b""):
pass
print("iter(f.read, b''):", reader.calls, "calls, biggest piece", reader.biggest)
with open("photo.bin", "rb") as f:
reader = CountingReader(f)
for chunk in iter(lambda: reader.read(65_536), b""):
pass
print("iter(lambda: ...):", reader.calls, "calls, biggest piece", reader.biggest)
# ✅ Expected output:
# iter(f.read, b''): 2 calls, biggest piece 1000000
# iter(lambda: ...): 17 calls, biggest piece 65536The first call returned the whole file. A lambda, as in Lambda Functions, passes the size, and makes the same 17 calls as the while loop. The Walrus Operator (:=) has a third way to write it, while chunk := f.read(65_536):.
A piece ends where the count runs out, not where a line or a word does. Counting words piece by piece counts a word cut in two as two words:
with open("words.txt", "w", encoding="utf-8") as f:
f.write("apple banana cherry\n" * 3)
count = 0
with open("words.txt", encoding="utf-8") as f:
while True:
chunk = f.read(16)
if not chunk:
break
count += len(chunk.split())
print("in chunks of 16:", count)
with open("words.txt", encoding="utf-8") as f:
print("line by line:", sum(len(line.split()) for line in f))
# ✅ Expected output:
# in chunks of 16: 11
# line by line: 9Pieces of 16 characters cut cherry into che and rry, and later banana into ba and nana, so two words were counted twice. Characters can be cut too. Text mode counts characters and binary mode counts bytes, and in UTF-8 é is two bytes:
with open("menu.txt", "w", encoding="utf-8") as f:
f.write("café crème\n")
with open("menu.txt", encoding="utf-8") as f:
print(repr(f.read(4)), repr(f.read(4)))
with open("menu.txt", "rb") as f:
print(f.read(4), f.read(4))
# ✅ Expected output:
# 'café' ' crè'
# b'caf\xc3' b'\xa9 cr'In text mode read(4) returned café, four characters. In binary mode the four bytes end halfway through é: \xc3 is its first byte, and \xa9 starts the next piece. A bytes value's decode() method turns bytes into text, the step text mode takes for you, and Strings, Bytes & Unicode Encodings covers it in depth. Decoding a piece that ends inside a character fails:
with open("menu.txt", "w", encoding="utf-8") as f:
f.write("café\n")
with open("menu.txt", "rb") as f:
first = f.read(4)
print(first)
print(first.decode("utf-8"))
# ✅ Expected output:
# b'caf\xc3'
# ✅ Expected error:
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 3: unexpected end of dataSo cut binary data into pieces for work on bytes: copying them, or counting the byte b"\n". In UTF-8 that byte only ever means a line break, never part of another character, so counting it piece by piece counts newline terminators. For the newline-terminated input in the practice exercise below, that is also the number of lines; a final unterminated line needs separate handling. For text, read in text mode, where read(n) counts whole characters.
Text mode decodes and translates; binary mode doesn't
In text mode, open() builds three objects, each wrapping the one below it. A FileIO reads raw bytes from the operating system. A BufferedReader reads those in large blocks and hands them out in the sizes asked for. A TextIOWrapper decodes the bytes into text and translates the line ends. Binary mode stops at the BufferedReader:
with open("todo.txt", "w", encoding="utf-8") as f:
f.write("Buy milk\n")
with open("todo.txt", encoding="utf-8") as f:
print(type(f).__name__, type(f.buffer).__name__, type(f.buffer.raw).__name__)
with open("todo.txt", "rb") as f:
print(type(f).__name__, type(f.raw).__name__)
# ✅ Expected output:
# TextIOWrapper BufferedReader FileIO
# BufferedReader FileIOA text file's buffer attribute is the layer below it, and raw the one below that. The example prints only the type names: printing a file object also shows its encoding, and for a file opened without encoding= that depends on the computer.
File Handling showed text mode reading Windows line ends, \r\n, as \n. It reads a lone \r, which some older files use, as \n too, and records each kind it met in f.newlines. newline="" turns the translation off but still ends lines at all three; binary mode ends them at \n only:
with open("mixed.txt", "wb") as f:
f.write(b"a\r\nb\rc\nd")
with open("mixed.txt", encoding="utf-8") as f:
print(list(f))
print(f.newlines)
with open("mixed.txt", encoding="utf-8", newline="") as f:
print(list(f))
with open("mixed.txt", "rb") as f:
print(list(f))
# ✅ Expected output:
# ['a\n', 'b\n', 'c\n', 'd']
# ('\r', '\n', '\r\n')
# ['a\r\n', 'b\r', 'c\n', 'd']
# [b'a\r\n', b'b\rc\n', b'd']In binary mode, b\rc\n is one line, because only \n ends a line there. With newline="" each line keeps its own ending exactly as it was, which is what the csv module needs to read rows correctly.
Some programs start a UTF-8 file with three extra bytes, \xef\xbb\xbf, called a byte order mark or BOM. Python's codecs documentation calls this variant utf-8-sig, says Microsoft invented it for Notepad, and adds that in UTF-8 "the use of the BOM is discouraged". Read as plain utf-8, the BOM stays in the text as the character \ufeff, and in a CSV file it sticks to the first column's name:
import csv
with open("export.csv", "w", encoding="utf-8-sig", newline="") as f:
f.write("id,name\r\n1,Ada\r\n")
with open("export.csv", "rb") as f:
print(f.read(5))
with open("export.csv", encoding="utf-8", newline="") as f:
print(next(csv.DictReader(f)))
with open("export.csv", encoding="utf-8-sig", newline="") as f:
print(next(csv.DictReader(f)))
# ✅ Expected output:
# b'\xef\xbb\xbfid'
# {'\ufeffid': '1', 'name': 'Ada'}
# {'id': '1', 'name': 'Ada'}Read with utf-8, the first key is '\ufeffid', so row["id"] stops with KeyError: 'id'. utf-8-sig removes the BOM, and a file without one reads the same with it as with utf-8. So read with utf-8-sig when a file may come with a BOM, and write with utf-8.
Text mode decodes as it reads, so a file in another encoding fails partway through. This one has 2,000 ordinary lines, then Zoé written in Latin-1, where é is the single byte \xe9, which isn't valid UTF-8 there. To find the bad line, read in binary mode and decode each line yourself, catching the error with try. The fixture uses newline="\n" so its bytes are identical on Windows and Unix:
with open("names.txt", "w", encoding="latin-1", newline="\n") as f:
for n in range(2000):
f.write(f"name{n}\n")
f.write("Zoé\n")
for n in range(10):
f.write(f"tail{n}\n")
with open("names.txt", "rb") as f:
for number, raw in enumerate(f, start=1):
try:
raw.decode("utf-8")
except UnicodeDecodeError:
print("line", number, "is not UTF-8:", raw)
# ✅ Expected output:
# line 2001 is not UTF-8: b'Zo\xe9\n'The same file read in text mode stops the loop with a UnicodeDecodeError before line 2,001 comes round. Measured on Python 3.10 to 3.14 and in this site's editor, the loop had counted 1,943 lines when it stopped, and the error's message was 'utf-8' codec can't decode byte 0xe9 in position 508: invalid continuation byte. Text mode decodes a block of the file ahead of the loop, so the error comes early, and its position counts from the start of that block, not of the file. Once you know the encoding, pass it: encoding="latin-1" here. Reading on past bytes that won't decode, with errors=, is covered in Strings, Bytes & Unicode Encodings.
seek() and tell(): byte positions, and bookmarks in text mode
A file object keeps a position, where the next read starts. File Handling went back to the start with seek(0). seek(offset, whence) goes anywhere: a whence of 0 counts from the start, 1 from the current position and 2 from the end, and os.SEEK_SET, os.SEEK_CUR and os.SEEK_END are names for 0, 1 and 2. seek() returns the new position, and tell() reports it. In binary mode a position is a count of bytes:
with open("digits.bin", "wb") as f:
f.write(b"0123456789")
with open("digits.bin", "rb") as f:
print(f.seek(3))
print(f.read(2), f.tell())
print(f.seek(-2, 2), f.read())
print(f.seek(-5, 1), f.read(1))
# ✅ Expected output:
# 3
# b'34' 5
# 8 b'89'
# 5 b'5'seek(-2, 2) went to 2 bytes before the end, and seek(-5, 1) to 5 bytes before where the last read had stopped, at 10. Seeking close to the end reads the last lines of a big log without reading the rest. The bytes you land on rarely start a line, so take more than you need and keep the last whole lines:
with open("app.log", "w", encoding="utf-8", newline="\n") as f:
for n in range(1, 100_001):
f.write(f"{n} INFO request\n")
with open("app.log", "rb") as f:
f.seek(-60, 2)
tail = f.read()
print(tail)
for raw in tail.splitlines()[-3:]:
print(raw.decode("utf-8"))
# ✅ Expected output:
# b't\n99998 INFO request\n99999 INFO request\n100000 INFO request\n'
# 99998 INFO request
# 99999 INFO request
# 100000 INFO requestWith the fixture's explicit LF line endings, the last 60 bytes started with the end of line 99,997, t\n, so the code keeps the last three lines and decodes each. However long the log, read() returns only those 60 bytes.
Text mode is stricter, because a position there isn't a count of characters. Counted from the start, seek() takes a number that tell() returned, or 0. Counted from the end or from the current position, the only move it takes is 0, so seek(0, 2) goes to the end. Python's documentation calls any other position counted from the start "undefined behaviour", and any other move from the end or the current position fails:
with open("todo.txt", "w", encoding="utf-8") as f:
f.write("Buy milk\nWalk the dog\n")
with open("todo.txt", encoding="utf-8") as f:
f.seek(-3, 2)
# ✅ Expected error:
# io.UnsupportedOperation: can't do nonzero end-relative seeksA number from tell() works as a bookmark: save it, read on, and seek() back to it:
with open("menu.txt", "w", encoding="utf-8") as f:
f.write("café\nthé\ncrème\n")
with open("menu.txt", encoding="utf-8") as f:
f.readline()
mark = f.tell()
print(repr(f.readline()), repr(f.readline()))
f.seek(mark)
print(repr(f.readline()))
# ✅ Expected output:
# 'thé\n' 'crème\n'
# 'thé\n'The documentation calls what tell() returns in text mode "an opaque number" that "does not usually represent a number of bytes", so use it only to seek() back, and don't do arithmetic with it. It isn't available inside a for loop over the file. The loop asks for one line at a time, but underneath, the file reads a block ahead of the line it hands you, and while the loop runs Python skips the record-keeping tell() would need to work back from there:
with open("todo.txt", "w", encoding="utf-8") as f:
f.write("Buy milk\nWalk the dog\n")
with open("todo.txt", encoding="utf-8") as f:
for line in f:
print(f.tell())
# ✅ Expected error:
# OSError: telling position disabled by next() callWhen you need a bookmark at each line, call tell() and then readline() in a while loop, stopping when readline() returns an empty string. A number that tell() didn't give you can land in the middle of a character. In café, byte 4 is the second byte of é:
with open("menu.txt", "w", encoding="utf-8") as f:
f.write("café\n")
with open("menu.txt", encoding="utf-8") as f:
f.seek(4)
print(f.read())
# ✅ Expected error:
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 0: invalid start bytewrite() isn't in the file yet: buffers and flush()
File Handling showed that what you write may not be in the file until it's closed. A file object keeps written text in a buffer, in memory, and sends it to the file in large blocks. flush() sends whatever is waiting now, without closing the file. Each check below reads the file afresh with read_text() from Modern File Paths with pathlib, so it shows only what has reached the file:
from pathlib import Path
log = open("log.txt", "w", encoding="utf-8")
log.write("09:00 start\n")
print(repr(Path("log.txt").read_text(encoding="utf-8")))
log.flush()
print(repr(Path("log.txt").read_text(encoding="utf-8")))
log.write("09:05 login\n")
print(repr(Path("log.txt").read_text(encoding="utf-8")))
log.close()
print(repr(Path("log.txt").read_text(encoding="utf-8")))
# ✅ Expected output:
# ''
# '09:00 start\n'
# '09:00 start\n'
# '09:00 start\n09:05 login\n'That matters for a program that runs for a long time and writes a log: anyone reading the log meanwhile sees only what has been flushed. When the program ends, Python flushes and closes the files it still has open, even when an uncaught exception stops it. In runs made for this lesson on Python 3.10 to 3.14, a program that wrote a log line and then stopped with a RuntimeError, without flush() or close(), left the line in the file.
A program that is killed gets no such clean-up, and neither does one that calls os._exit(), which ends it on the spot: in the same runs, both left the file empty. Python's tutorial also warns that text written without with or close() "might" not all reach the file, so don't rely on the clean-up. And because the lines written before a crash do reach the file, a crash halfway through leaves it half written. To make sure that can't happen, write a new file and swap it in, as Context Managers & the with Statement in Depth does with its SafeWriter.
buffering=1 makes a text file line-buffered, flushing each time a \n is written:
from pathlib import Path
log = open("log.txt", "w", encoding="utf-8", buffering=1)
log.write("09:00 start\n")
print(repr(Path("log.txt").read_text(encoding="utf-8")))
log.write("09:05 ")
print(repr(Path("log.txt").read_text(encoding="utf-8")))
log.write("login\n")
print(repr(Path("log.txt").read_text(encoding="utf-8")))
log.close()
# ✅ Expected output:
# '09:00 start\n'
# '09:00 start\n'
# '09:00 start\n09:05 login\n'The half line 09:05 stayed in the buffer until its \n arrived. print() takes flush=True for the same job, one call at a time:
from pathlib import Path
log = open("log.txt", "w", encoding="utf-8")
print("09:00 start", file=log)
print(repr(Path("log.txt").read_text(encoding="utf-8")))
print("09:05 login", file=log, flush=True)
print(repr(Path("log.txt").read_text(encoding="utf-8")))
log.close()
# ✅ Expected output:
# ''
# '09:00 start\n09:05 login\n'No buffer at all, buffering=0, is allowed in binary mode only:
log = open("log.txt", "w", encoding="utf-8", buffering=0)
# ✅ Expected error:
# ValueError: can't have unbuffered text I/OHow much a buffer holds before it's sent on its own depends on the Python version and on the disk. help(open) on Python 3.10 to 3.13 says a file's buffer "will typically be 4096 or 8192 bytes long", and on 3.14 "128 kilobytes long". In runs made for this lesson, after an unflushed write() of 10,000 characters, a second reader found all 10,000 in the file on 3.10 to 3.13 and in this site's editor, and none on 3.14. So don't count on a buffer filling: call flush() when the text must be in the file, or close it.
A program's standard output has a buffer too. Logging, Debugging & Error Handling at Scale covered standard output and standard error, the two streams a program writes text to. Measured on Python 3.10 to 3.14, sys.stdout.line_buffering was True when the output went to a terminal, so each printed line appears at once, and False when it was piped into another program, so lines wait in the buffer. Running python3 -u made standard output write straight through, and print(..., flush=True) does that for one line.
Streams that aren't files: StringIO, BytesIO and TextIOWrapper
Code that reads a stream doesn't need a file name. Building Command-Line Tools with argparse & Typer wrote count_lines() to take a stream as a parameter, so a program could pass it sys.stdin and a test an io.StringIO, the text file in memory from Reading CSV & Excel Files. That version called readlines(), which keeps every line. Counting the lines as they pass streams instead, and works with anything a for loop can take lines from:
import io
def count_lines(stream):
return sum(1 for line in stream)
with open("todo.txt", "w", encoding="utf-8") as f:
f.write("Buy milk\nWalk the dog\n")
with open("todo.txt", encoding="utf-8") as f:
print(count_lines(f))
print(count_lines(io.StringIO("one\ntwo\nthree\n")))
print(count_lines(["a\n", "b\n"]))
# ✅ Expected output:
# 2
# 3
# 2A StringIO behaves like a file opened for reading and writing, position included. Writing leaves the position at the end, so a read() straight afterwards finds nothing, while getvalue() returns all the text wherever the position is:
import io
buffer = io.StringIO()
buffer.write("first\n")
buffer.write("second\n")
print(repr(buffer.read()))
buffer.seek(0)
print(repr(buffer.readline()))
print(repr(buffer.getvalue()))
# ✅ Expected output:
# ''
# 'first\n'
# 'first\nsecond\n'io.BytesIO is the same for bytes: a binary file in memory. io.TextIOWrapper is the layer text mode adds, and wrapped round any binary stream it reads that stream as text. In this example, encode(), the opposite of decode(), turns text into the bytes a file would hold:
import io
raw = io.BytesIO("café\nthé\n".encode("utf-8"))
print(raw.read(5))
raw.seek(0)
for line in io.TextIOWrapper(raw, encoding="utf-8"):
print(repr(line))
# ✅ Expected output:
# b'caf\xc3\xa9'
# 'café\n'
# 'thé\n'A TextIOWrapper owns the stream it wraps. Closing it, which the end of a with block does, closes the BytesIO as well, and what was written can't be read any more:
import io
raw = io.BytesIO()
with io.TextIOWrapper(raw, encoding="utf-8") as text:
text.write("café\n")
print(raw.getvalue())
# ✅ Expected error:
# ValueError: I/O operation on closed file.detach() hands the stream back instead, still open. It flushes the wrapper's buffer first, so the text written through the wrapper is in the BytesIO:
import io
raw = io.BytesIO()
text = io.TextIOWrapper(raw, encoding="utf-8", newline="\n")
text.write("café\n")
text.detach()
print(raw.getvalue(), raw.closed)
# ✅ Expected output:
# b'caf\xc3\xa9\n' FalseStandard input is a stream too
sys.stdin, a program's standard input, is a TextIOWrapper over a BufferedReader, like a file opened in text mode. In a terminal it reads what you type, and the shell can connect it to a file instead, with <, or to another program's output, with a pipe, |. This site's editor has no file or pipe to connect, so the programs in this part were run from the command line, on Python 3.10 to 3.14, with the same results on each. count.py passes sys.stdin to the streaming count_lines():
# count.py
import sys
def count_lines(stream):
return sum(1 for line in stream)
if __name__ == "__main__":
print(count_lines(sys.stdin))printf 'a\nb\nc\n' | python3 count.py
seq 1 1000000 | python3 count.pyThey printed 3 and 1000000. seq prints the numbers from 1 to 1,000,000, one per line, and count.py counted them one line at a time. errors.py passes on only the lines that contain ERROR. sys.stdout.write(line) writes a line exactly as it came in, where print() would add a second \n:
# errors.py
import sys
for line in sys.stdin:
if "ERROR" in line:
sys.stdout.write(line)printf 'INFO a\nERROR b\nINFO c\nERROR d\n' | python3 errors.py | python3 count.pyThat printed 2. The shell started both programs at once and connected the standard output of errors.py to the standard input of count.py, so count.py read the lines as the buffer of errors.py sent them on.
A pipe has no going back to its start. rewind.py reads all of its standard input, then tries to return to the beginning:
# rewind.py
import sys
sys.stdin.read()
sys.stdin.seek(0)
print("rewound")printf 'a\nb\n' | python3 rewind.py
python3 rewind.py < count.pyFed through a pipe, it stopped at seek(0) with io.UnsupportedOperation: underlying stream is not seekable. Given the file count.py with <, it printed rewound. Compressed input works too: gzip.open(sys.stdin.buffer, "rt", encoding="utf-8"), from the next part, read a gzip-compressed pipe of 100,000 numbers and counted 100,000 lines. sys.stdin.buffer is the binary layer under sys.stdin.
Compressed files and zip members, read as streams
gzip.open() opens a .gz file and decompresses it as you read, a piece at a time, so a compressed file is read like any other, without unpacking it first. It takes open()'s modes for reading, writing, appending and creating, though not the ones with +. One difference catches people out: its default mode is "rb", binary, where open()'s is text. "rt" and "wt" give text:
import gzip
with gzip.open("visits.log.gz", "wt", encoding="utf-8", newline="\n") as f:
f.write("2024-06-01 /home\n")
f.write("2024-06-02 /pricing\n")
with gzip.open("visits.log.gz", "rt", encoding="utf-8") as f:
for line in f:
print(repr(line))
with gzip.open("visits.log.gz") as f:
for line in f:
print(line)
# ✅ Expected output:
# '2024-06-01 /home\n'
# '2024-06-02 /pricing\n'
# b'2024-06-01 /home\n'
# b'2024-06-02 /pricing\n'Without "rt", each line is a bytes value. Writing a string to a gzip file opened in a binary mode stops with a TypeError worded differently by version: on Python 3.10, 3.11 and in this site's editor it's TypeError: memoryview: a bytes-like object is required, not 'str', and from 3.12 on TypeError: a bytes-like object is required, not 'str'. A big compressed file streams like a plain one:
import gzip
with gzip.open("big.log.gz", "wt", encoding="utf-8") as f:
for n in range(100_000):
level = "ERROR" if n % 1000 == 0 else "INFO"
f.write(f"{level} request {n}\n")
lines = 0
errors = 0
with gzip.open("big.log.gz", "rt", encoding="utf-8") as f:
for line in f:
lines += 1
if line.startswith("ERROR"):
errors += 1
print(lines, errors)
# ✅ Expected output:
# 100000 100gzip checks the first bytes it reads, so a file that isn't gzip-compressed fails at once:
import gzip
with open("notes.txt", "w", encoding="utf-8") as f:
f.write("Hello\n")
with gzip.open("notes.txt", "rt", encoding="utf-8") as f:
print(f.read())
# ✅ Expected error:
# gzip.BadGzipFile: Not a gzipped file (b'He')bz2.open() works the same way for .bz2 files, with the same modes and the same binary default, and so does lzma.open() for .xz files.
A zip file holds several files, called members. Packaging & Publishing Python Libraries to PyPI wrote zip files with zipfile. ZipFile.open(name) reads one member as a binary stream, decompressing it as it goes. Wrapped in a TextIOWrapper, with newline="" for the csv module, it feeds csv.DictReader one row at a time, and nothing is unpacked to disk:
import csv
import io
import zipfile
with zipfile.ZipFile("export.zip", "w", compression=zipfile.ZIP_DEFLATED) as zf:
zf.writestr("sales.csv", "item,units\r\ntea,3\r\ncake,1\r\n")
zf.writestr("notes.txt", "two files\n")
with zipfile.ZipFile("export.zip") as zf:
print(zf.namelist())
with zf.open("sales.csv") as raw:
for row in csv.DictReader(io.TextIOWrapper(raw, encoding="utf-8", newline="")):
print(row)
# ✅ Expected output:
# ['sales.csv', 'notes.txt']
# {'item': 'tea', 'units': '3'}
# {'item': 'cake', 'units': '1'}ZIP_DEFLATED compresses the members; without it, zipfile stores them as they are. ZipFile.open() has no text mode: zf.open("sales.csv", "rt") stops with ValueError: open() requires mode "r" or "w". Handing its binary stream straight to the csv module is the usual slip:
import csv
import zipfile
with zipfile.ZipFile("export.zip", "w") as zf:
zf.writestr("sales.csv", "item,units\r\ntea,3\r\n")
with zipfile.ZipFile("export.zip") as zf:
with zf.open("sales.csv") as f:
for row in csv.reader(f):
print(row)
# ✅ Expected error:
# _csv.Error: iterator should return strings, not bytes (the file should be opened in text mode)CSV rows and JSON Lines, one record at a time
Reading CSV & Excel Files read CSV files row by row with csv.reader and csv.DictReader, and said to loop over a reader rather than call list() on it. A row isn't always a line, though. A quoted value can hold a line break, so a file can have more lines than rows:
import csv
with open("notes.csv", "w", encoding="utf-8", newline="") as f:
writer = csv.writer(f)
writer.writerow(["id", "note"])
writer.writerow([1, "first line\nsecond line"])
writer.writerow([2, "short"])
with open("notes.csv", encoding="utf-8", newline="") as f:
print(sum(1 for line in f), "lines")
with open("notes.csv", encoding="utf-8", newline="") as f:
reader = csv.reader(f)
rows = sum(1 for row in reader)
print(rows, "rows, the last ending on line", reader.line_num)
# ✅ Expected output:
# 4 lines
# 3 rows, the last ending on line 4csv.writer put the note in quotes, line break and all, so the file has four lines but three rows. reader.line_num, printed at the end, is how many lines the reader has read so far, so inside the loop it's the line the current row ended on. That makes it the number to report when a row holds a bad value:
import csv
import io
text = io.StringIO("item,units\ntea,3\ncake,x\njam,2\n")
reader = csv.DictReader(text)
total = 0
for row in reader:
try:
total += int(row["units"])
except ValueError:
print("line", reader.line_num, "has units", repr(row["units"]))
print(total)
# ✅ Expected output:
# line 3 has units 'x'
# 5Reading one file and writing another a row at a time filters a CSV file of any size. Context Managers & the with Statement in Depth put several context managers in one with, separated by commas, and two open files work the same way. reader.fieldnames holds the column names DictReader read from the header, and gives DictWriter the same columns:
import csv
with open("orders.csv", "w", encoding="utf-8", newline="") as f:
writer = csv.writer(f)
writer.writerow(["id", "status", "pence"])
for n in range(1, 10_001):
writer.writerow([n, "refunded" if n % 500 == 0 else "paid", n % 7 * 100])
kept = 0
with open("orders.csv", encoding="utf-8", newline="") as src, open("refunds.csv", "w", encoding="utf-8", newline="") as dst:
reader = csv.DictReader(src)
writer = csv.DictWriter(dst, fieldnames=reader.fieldnames)
writer.writeheader()
for row in reader:
if row["status"] == "refunded":
writer.writerow(row)
kept += 1
print(kept)
with open("refunds.csv", encoding="utf-8", newline="") as f:
print(f.readline().strip())
print(f.readline().strip())
# ✅ Expected output:
# 20
# id,status,pence
# 500,refunded,300pandas can read a CSV file in pieces too: pd.read_csv(path, chunksize=10_000) gives DataFrames of up to 10,000 rows, one after another. The pandas course's Reading & Writing Data (CSV, Excel, JSON) covers read_csv().
JSON Lines, in files usually named .jsonl, stores one JSON value per line, most often an object. Working with JSON used json.load(), which reads a whole file and parses it as one value. With one record per line, json.dumps() writes each record as a line and json.loads() reads each line back, so the file streams. json.dumps() writes a character outside ASCII as an escape, \u00e9 for é, unless you pass ensure_ascii=False; json.loads() turns it back into é:
import json
orders = [{"id": 1, "item": "tea", "pence": 120}, {"id": 2, "item": "café", "pence": 275}]
with open("orders.jsonl", "w", encoding="utf-8") as f:
for order in orders:
f.write(json.dumps(order) + "\n")
with open("orders.jsonl", encoding="utf-8") as f:
print(f.read(), end="")
total = 0
with open("orders.jsonl", encoding="utf-8") as f:
for line in f:
total += json.loads(line)["pence"]
print(total)
# ✅ Expected output:
# {"id": 1, "item": "tea", "pence": 120}
# {"id": 2, "item": "caf\u00e9", "pence": 275}
# 395json.load() can't read such a file, because it expects one value and finds a second on line 2:
import json
with open("orders.jsonl", "w", encoding="utf-8") as f:
f.write('{"id": 1, "pence": 120}\n')
f.write('{"id": 2, "pence": 275}\n')
with open("orders.jsonl", encoding="utf-8") as f:
orders = json.load(f)
# ✅ Expected error:
# json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 24)A blank line, easily left behind by editing a file by hand or joining two files, isn't a JSON value either:
import json
with open("orders.jsonl", "w", encoding="utf-8") as f:
f.write('{"id": 1, "pence": 120}\n\n{"id": 2, "pence": 275}\n')
total = 0
with open("orders.jsonl", encoding="utf-8") as f:
for line in f:
total += json.loads(line)["pence"]
print(total)
# ✅ Expected error:
# json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)So skip blank lines, and decide what a bad record should do. Here the loop reports it by line number and carries on. e.msg is the error's message without the position, which here would count from the start of the line rather than the file:
import json
with open("orders.jsonl", "w", encoding="utf-8") as f:
f.write('{"id": 1, "pence": 120}\n')
f.write('\n')
f.write('{"id": 2, "pence": 275\n')
f.write('{"id": 3, "pence": 99}\n')
total = 0
with open("orders.jsonl", encoding="utf-8") as f:
for number, line in enumerate(f, start=1):
if not line.strip():
continue
try:
total += json.loads(line)["pence"]
except json.JSONDecodeError as e:
print("line", number, "skipped:", e.msg)
print(total)
# ✅ Expected output:
# line 3 skipped: Expecting ',' delimiter
# 219Practice: count lines in 4,096-byte pieces
Finish chunks(), a generator that reads f in pieces of size bytes until read() returns b"", then count the lines of a 10,000-line log by counting the newline bytes in each piece. Each blank has its own hint line at the bottom of the code, in order.
def chunks(f, size):
while True:
data = f.___(size)
if not data:
___
yield data
with open("app.log", "w", encoding="utf-8") as f:
for n in range(1, 10_001):
f.write(f"{n} INFO request\n")
lines = 0
biggest = 0
with open("app.log", "rb") as f:
for data in chunks(f, 4096):
lines += data.count(___)
biggest = max(biggest, len(data))
print(lines, biggest)
# 👉 replace ___ with read
# 👉 replace ___ with break
# 👉 replace ___ with b"\n"
# ✅ Expected output:
# 10000 4096On your own: refunds as compressed JSON Lines
orders.csv holds 10,000 orders, with the columns id, status and pence; the starter below writes it. Without loading the whole file, write every order whose status is refunded to refunds.jsonl.gz, as gzip-compressed JSON Lines, with id and pence as numbers. Then read refunds.jsonl.gz back and print how many refunds there are, their total in pence, and the first one.
import csv
with open("orders.csv", "w", encoding="utf-8", newline="") as f:
writer = csv.writer(f)
writer.writerow(["id", "status", "pence"])
for n in range(1, 10_001):
writer.writerow([n, "refunded" if n % 500 == 0 else "paid", n % 7 * 100])
# your code here: refunds.jsonl.gz, then read it backimport csv
import gzip
import json
with open("orders.csv", "w", encoding="utf-8", newline="") as f:
writer = csv.writer(f)
writer.writerow(["id", "status", "pence"])
for n in range(1, 10_001):
writer.writerow([n, "refunded" if n % 500 == 0 else "paid", n % 7 * 100])
with open("orders.csv", encoding="utf-8", newline="") as src, gzip.open("refunds.jsonl.gz", "wt", encoding="utf-8") as dst:
for row in csv.DictReader(src):
if row["status"] == "refunded":
record = {"id": int(row["id"]), "pence": int(row["pence"])}
dst.write(json.dumps(record) + "\n")
count = 0
total = 0
first = None
with gzip.open("refunds.jsonl.gz", "rt", encoding="utf-8") as f:
for line in f:
record = json.loads(line)
count += 1
total += record["pence"]
if first is None:
first = record
print(count, "refunds,", total, "pence")
print("first:", first)
# ✅ Expected output:
# 20 refunds, 6300 pence
# first: {'id': 500, 'pence': 300}
Practice quiz
app.log has 100,000 lines and is open as f. Which of these keeps every line in memory before it gives a result?
- sum(1 for line in f)
- max(f, key=len)
- sorted(f)
- any("ERROR" in line for line in f)
Answer: sorted(f). sorted() has to see every line before it can return the first, and it returns a list of all 100,000. The other three look at one line at a time and keep only a count, the longest line so far, or nothing.
menu.txt holds café crème, written as UTF-8. t is the file opened in text mode with encoding="utf-8", and b the same file opened with "rb". What does print(repr(t.read(4)), b.read(4)) print?
- 'café' b'café'
- 'caf\xc3' b'caf\xc3'
- 'caf' b'caf'
- 'café' b'caf\xc3'
Answer: 'café' b'caf\xc3'. In text mode, read(4) counts characters, so it returns café. In binary mode it counts bytes, and é takes two bytes in UTF-8, so the fourth byte is the first half of é.
photo.bin holds 1,000,000 bytes and is open with "rb" as f. What does for chunk in iter(f.read, b""): print(len(chunk)) print?
- 4096, many times
- 1000000, once
- 65536 fifteen times, then 16960
- Nothing: the loop stops at once
Answer: 1000000, once. iter(f.read, b"") calls f.read() with no size, and read() with no size returns everything that's left. The second call returns b"", which ends the loop. A lambda passes the size: iter(lambda: f.read(65_536), b"").
log = open("log.txt", "w", encoding="utf-8") is followed by log.write("09:00 start "). Before log is flushed or closed, a second open("log.txt", encoding="utf-8") reads the file. What does repr() of its read() show?
- ''
- '09:00 start '
- '09:00 start'
- It stops: the file is already open
Answer: ''. write() put the text in log's buffer, in memory. It reaches the file when the buffer fills, on flush() or on close(), so the second reader finds the file empty.
todo.txt holds Buy milk and Walk the dog, one per line, and is open in text mode as f. What does f.seek(-3, 2) do?
- It moves 3 characters back from the end, so f.read() gives 'og '
- It stops with io.UnsupportedOperation: can't do nonzero end-relative seeks
- It stops with ValueError: negative seek position -3
- It moves to the start of the file
Answer: It stops with io.UnsupportedOperation: can't do nonzero end-relative seeks. In text mode, seek() counted from the start takes a number tell() gave you, or 0, and counted from the end or the current position it takes only 0, as in seek(0, 2) for the end. Opened with "rb", the same seek works, and read() then gives b'og '.
What does this print? buffer = io.StringIO(), then buffer.write("first "), then print(repr(buffer.read()))
- 'first '
- 'first'
- None
- ''
Answer: ''. A StringIO keeps a position, as a file does. Writing leaves it at the end, so read() finds nothing after it. seek(0) before reading, or getvalue(), gives the text.
visits.log.gz was written with gzip.open("visits.log.gz", "wt", encoding="utf-8", newline=" ") and holds the line 2024-06-01 /home. It is opened again with f = gzip.open("visits.log.gz"), no mode given. What does for line in f: print(line) print?
- 2024-06-01 /home
- '2024-06-01 /home '
- b'2024-06-01 /home '
- It stops with UnicodeDecodeError
Answer: b'2024-06-01 /home '. gzip.open() opens in binary mode unless the mode says "rt", so each line is a bytes value, and print() shows it in its b'...' form.
csv.writer wrote notes.csv: a header row, a row whose note holds a line break, and one more row. Opened with newline="", the file gives lines = sum(1 for line in f), and, opened again, rows = sum(1 for row in csv.reader(f)). What does print(lines, rows) print?
- 4 3
- 3 3
- 4 4
- 3 4
Answer: 4 3. csv.writer put the note in quotes with its line break inside, so the file has 4 lines. csv.reader keeps a quoted value whole, so it gives 3 rows.
Continue this course
- Previous: Packaging & Publishing Python Libraries to PyPI
- Next: Advanced Collections, itertools & functools — Use Counter, defaultdict, chain, groupby, partial and more
- Quick reference: Python cheat sheet
Frequently asked questions
How do I follow a log file that's still being written, like tail -f?
Open it, move to the end with seek(0, 2), and call readline() in a loop. readline() returns an empty string while nothing new has been written, so wait a moment with time.sleep() and try again. A line that comes back without its is one the writer hasn't finished, so keep it until the rest arrives. The program writing the log decides when its lines reach the file, as the part on flush() showed. This site's editor can't show it working: nothing else writes to the file while your program runs.
Can I read one huge JSON array without loading all of it?
Not with json.load(). It reads the whole file and parses it as one value: wrapped in a reader that counts calls, as in this lesson, loading a file of 1,000 objects made a single read() call. If you produce the data yourself, write it as JSON Lines instead. Otherwise, the easy way is a parser from outside the standard library that reads JSON a piece at a time. The standard library can do it with a loop of your own: json.JSONDecoder().raw_decode(text) parses one value from the start of a string and returns it with the position where it ended, so a loop that reads the file in pieces can take the array apart one element at a time. Written that way, a loop reading 64 characters at a time got all 1,000 objects out of the same file.
How do I hash a big file without reading all of it?
Feed it to the hash a piece at a time: read(n) in a loop, passing each piece to the hash object's update() method, so the result is the same as hashing the whole file at once. Hashing with hashlib shows the loop.
What is mmap, and do I need it?
The mmap module maps a file into memory, so you can slice and search it like a bytes value without reading it into one first. With a file holding 0123456789 opened with "rb", m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) gave b'345' for m[3:6], 10 for len(m) and 7 for m.find(b"7"), on Python 3.10 to 3.14 and in this site's editor. For reading a file from start to end, the loops in this lesson are all you need.
Why does my script print BrokenPipeError when I pipe it into head?
head stops reading after the lines it shows and closes its end of the pipe, so Python's next write fails. Measured with python3 -c 'for i in range(100000): print(i)' | head -3: the numbers 0, 1 and 2 appeared, then a traceback ending in BrokenPipeError: [Errno 32] Broken pipe, and Python's exit status was 1. Python 3.14 then also printed Exception ignored while flushing sys.stdout: and the error a second time, and exited with 120. The signal module's documentation shows the fix: catch BrokenPipeError, point standard output at os.devnull, and exit with status 1.
How can I find the open() calls that forgot encoding=?
Run the program with python3 -X warn_default_encoding. Each text-mode open() without encoding= then prints an EncodingWarning. Measured on Python 3.10 to 3.14, open('x.txt', 'w') run with -c printed <string>:1: EncodingWarning: 'encoding' argument not specified, and with encoding='utf-8' added it printed nothing. Without encoding=, Python uses the computer's default encoding, which File Handling showed isn't the same everywhere.
Related lessons
- File Handling
- Generators & Iterators Mastery
- itertools
- Memory Management & Garbage Collection Internals
- Object-Oriented Programming
- Lambda Functions
- Walrus
- Strings, Bytes & Unicode Encodings
- Modern File Paths with pathlib
- Context Managers & the with Statement in Depth
- Logging, Debugging & Error Handling at Scale
- Building Command-Line Tools with argparse & Typer