High-Level File Operations (shutil)
Published by Brayan K
The shutil module provides high-level file operations — copying, moving, archiving, and deleting whole files and directory trees — in a single, convenient function call each.
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
Choose the copy contract before choosing the function
A copy can mean only file contents, contents plus permission mode, or an attempt to preserve additional metadata. shutil provides different contracts for these jobs. None is a universal backup system: copy2 still cannot preserve every platform-specific attribute, owner or access-control entry. Keep the original until you have checked the result your application actually needs.
from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import copyfile
with TemporaryDirectory() as directory:
root = Path(directory)
source = root / "source.txt"
destination = root / "destination.txt"
source.write_text("approved", encoding="utf-8")
destination.write_text("old draft", encoding="utf-8")
copyfile(source, destination)
print(destination.read_text(encoding="utf-8"))
print(source.read_bytes() == destination.read_bytes())
# ✅ Expected output:
# approved
# Truecopyfile takes a complete destination filename and overwrites an existing destination file. It copies contents rather than metadata. copy also accepts a destination directory and copies permission mode; copy2 additionally attempts to preserve metadata through copystat. Metadata support differs across filesystems, so these examples check contents instead of claiming a portable timestamp or permission result.
from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import copy
with TemporaryDirectory() as directory:
root = Path(directory)
source = root / "record.txt"
target = root / "copies"
target.mkdir()
source.write_text("record", encoding="utf-8")
copied = Path(copy(source, target))
print(copied.parent == target)
print(copied.read_text(encoding="utf-8"))
print(source.exists())
# ✅ Expected output:
# True
# record
# TrueThe copied file retains its basename under the target directory. Do not interchange copy and copyfile while assuming their destination rules are identical. Building and inspecting Path objects is covered in Modern File Paths with pathlib; this lesson owns the operation applied to those paths.
A stream copy starts at the current position
from io import BytesIO
from shutil import copyfileobj
source = BytesIO(b"HEADER:payload")
source.seek(7)
destination = BytesIO()
copyfileobj(source, destination, length=3)
print(destination.getvalue().decode("ascii"))
print(source.closed, destination.closed)
source.close()
destination.close()
# ✅ Expected output:
# payload
# False Falsecopyfileobj consumes from the current source position. The positive length controls its buffer size, not the total number of bytes to copy. It neither closes the supplied streams nor guarantees that a buffered destination has been flushed before another handle reads it. The caller owns those lifetime and visibility decisions. Working with Files, Streams & Large Datasets owns buffering and stream interfaces in detail.
from io import BytesIO, StringIO
from shutil import copyfileobj
with StringIO("text") as source, BytesIO() as destination:
copyfileobj(source, destination)
# ✅ Expected error:
# TypeError: a bytes-like object is required, not 'str'The read method returns strings here, while the destination expects bytes. Choose matching stream types or put an explicit encoding layer between them. A copy operation does not infer an encoding from a filename extension.
Copying a tree is not synchronizing a tree
from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import copytree, ignore_patterns
with TemporaryDirectory() as directory:
root = Path(directory)
source = root / "source"
source.mkdir()
(source / "nested").mkdir()
(source / "keep.txt").write_text("A", encoding="utf-8")
(source / "cache.tmp").write_text("skip", encoding="utf-8")
(source / "nested" / "keep.txt").write_text("B", encoding="utf-8")
destination = root / "copied"
copytree(source, destination, ignore=ignore_patterns("*.tmp"))
print((destination / "keep.txt").read_text(encoding="utf-8"))
print((destination / "nested" / "keep.txt").read_text(encoding="utf-8"))
print((destination / "cache.tmp").exists())
# ✅ Expected output:
# A
# B
# FalseBy default copytree expects the destination tree not to exist. ignore_patterns supplies a callback that excludes matching names as each source directory is visited. It is a copy-selection rule, not a deletion rule for files already present at the destination. Wildcard behavior is owned by Finding Files with glob & fnmatch.
from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import copytree
with TemporaryDirectory() as directory:
root = Path(directory)
source = root / "source"
destination = root / "destination"
source.mkdir()
destination.mkdir()
(source / "shared.txt").write_text("new", encoding="utf-8")
(destination / "shared.txt").write_text("old", encoding="utf-8")
(destination / "extra.txt").write_text("retained", encoding="utf-8")
copytree(source, destination, dirs_exist_ok=True)
print((destination / "shared.txt").read_text(encoding="utf-8"))
print((destination / "extra.txt").read_text(encoding="utf-8"))
# ✅ Expected output:
# new
# retaineddirs_exist_ok allows merging into existing directories; matching files can be overwritten, while extra destination files remain. That is not a mirror or an atomic transaction. A mid-copy failure can leave partial results. For valuable data, build a new destination, validate it, and publish it under an application-specific replacement policy.
Moving may rename or copy and then remove
from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import move
with TemporaryDirectory() as directory:
root = Path(directory)
source = (root / "draft.txt").resolve()
destination = (root / "final.txt").resolve()
if source.parent != root.resolve() or destination.parent != root.resolve():
raise ValueError("move must stay inside the owned directory")
source.write_text("final text", encoding="utf-8")
move(source, destination)
print(source.exists())
print(destination.read_text(encoding="utf-8"))
# ✅ Expected output:
# False
# final textWhen possible move uses a rename. Across filesystems it may instead copy and remove the source. Do not promise atomicity or identical metadata behavior for every move. Existing-target rules also depend on whether the destination is a directory and on the underlying rename semantics; choose a fresh destination when the operation must not overwrite another result.
This example creates both paths itself and checks their resolved parent before moving anything. Real applications need an explicit destination and overwrite policy; accepting an arbitrary path and calling move is not a complete file-management feature.
Removal needs a known boundary, not a guessed path
from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import rmtree
with TemporaryDirectory() as directory:
root = Path(directory).resolve()
target = (root / "generated").resolve()
target.mkdir()
(target / "result.txt").write_text("scratch", encoding="utf-8")
if target == root or target.parent != root:
raise ValueError("refuse removal outside the owned child")
rmtree(target)
print(target.exists())
print(root.exists())
# ✅ Expected output:
# False
# Truermtree removes the tree, including its contents. The guard refuses the root itself and any location outside the explicitly owned child boundary. It catches a wrong-path programming error in this controlled workspace; it is not a complete defense against hostile concurrent filesystem changes. Avoid using this small example as a security sandbox.
ignore_errors=True hides removal failures. Do not use it by default when successful cleanup matters: an open handle, permissions issue or unexpected target should remain visible. The lifetime management of the outer scratch directory is owned by Temporary Files & Directories.
Archive creation and extraction have different trust questions
from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import make_archive
from zipfile import ZipFile
with TemporaryDirectory() as directory:
root = Path(directory)
source = root / "source"
source.mkdir()
(source / "report.txt").write_text("ready", encoding="utf-8")
archive = make_archive(str(root / "bundle"), "zip", root_dir=source)
with ZipFile(archive) as bundle:
print(len(bundle.namelist()))
print(bundle.read("report.txt").decode("utf-8"))
# ✅ Expected output:
# 1
# readyroot_dir determines the tree to include; base_name names the archive without its format suffix. The generated archive can contain timestamps and other metadata, so equal file contents do not imply identical archive bytes. Check the logical members and data rather than a fixed compressed size.
from shutil import unpack_archive
unpack_archive("unused.archive", format="imaginary")
# ✅ Expected error:
# ValueError: Unknown unpack format 'imaginary'Extract only archives whose contents and trust level you understand. Archive members can attempt to escape an intended destination or create problematic links, and extraction defaults differ by format and Python version. This lesson creates and inspects its own ZIP fixture without extracting untrusted data. Streaming archive members belongs to Working with Files, Streams & Large Datasets.
Complete a filtered tree copy
Create a fresh copy that excludes .tmp files. Fill in the copy operation and the ignore pattern, then check the retained content and excluded fixture.
from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import copytree, ignore_patterns
with TemporaryDirectory() as directory:
root = Path(directory)
source = root / "source"
source.mkdir()
(source / "keep.txt").write_text("keep", encoding="utf-8")
(source / "scratch.tmp").write_text("skip", encoding="utf-8")
target = root / "target"
___(source, target, ignore=ignore_patterns(___))
print((target / "keep.txt").read_text(encoding="utf-8"))
print((target / "scratch.tmp").exists())
# 👉 replace ___ with copytree
# 👉 replace ___ with "*.tmp"
# ✅ Expected output:
# keep
# FalsePrepare and verify a fresh release directory
Build an owned source directory containing two text reports and one temporary file. Copy it into a fresh release directory while excluding .tmp files. Compare the retained bytes against the source, then archive the release as ZIP and print the sorted member contents. Leave the source intact and keep every operation inside one temporary workspace.
from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import copytree, ignore_patterns, make_archive
from zipfile import ZipFile
with TemporaryDirectory() as directory:
root = Path(directory)
source = root / "source"
source.mkdir()
fixtures = {"a.txt": "alpha", "b.txt": "beta", "cache.tmp": "skip"}
for name, text in fixtures.items():
(source / name).write_text(text, encoding="utf-8")
release = root / "release"
copytree(source, release, ignore=ignore_patterns("*.tmp"))
kept = ["a.txt", "b.txt"]
verified = all((source / name).read_bytes() == (release / name).read_bytes()
for name in kept)
if not verified:
raise ValueError("copied content mismatch")
archive = make_archive(str(root / "release-package"), "zip", root_dir=release)
with ZipFile(archive) as bundle:
print([bundle.read(name).decode("utf-8")
for name in sorted(bundle.namelist())])
print((source / "cache.tmp").exists())
print((release / "cache.tmp").exists())
# ✅ Expected output:
# ['alpha', 'beta']
# True
# False
Practice quiz
What does this program print? from io import BytesIO from shutil import copyfileobj source = BytesIO(b"abcdef") source.seek(2) target = BytesIO() copyfileobj(source, target, length=1) print(target.getvalue())
- b'cdef'
- b'a'
- b'abcdef'
- b'c'
Answer: b'cdef'. length is the copy buffer size; copying continues from the current position to EOF.
Which function requires a complete destination filename rather than accepting a destination directory?
- copy
- copyfile
- copy2
- copytree
Answer: copyfile. copyfile copies contents to the exact named destination file.
What does dirs_exist_ok=True do with extra files already in the destination?
- Deletes them
- Moves them to the source
- Leaves them unless another operation changes them
- Guarantees a byte-identical mirror
Answer: Leaves them unless another operation changes them. copytree merges and may overwrite matching files; it does not synchronize deletions.
What does this program print? from io import BytesIO from shutil import copyfileobj a = BytesIO(b"x") b = BytesIO() copyfileobj(a, b) print(a.closed, b.closed)
- True True
- True False
- False True
- False False
Answer: False False. The caller retains ownership of both supplied streams.
Which claim about copy2 is accurate?
- It attempts additional metadata preservation but cannot preserve everything
- It always preserves owners and ACLs everywhere
- It creates a compressed backup
- It never overwrites an existing file
Answer: It attempts additional metadata preservation but cannot preserve everything. Filesystem and platform metadata capabilities limit what can be preserved.
Why can move across filesystems behave differently from rename?
- It never removes the source
- It may copy data and then remove the source
- It changes every file into text
- It always creates a ZIP archive
Answer: It may copy data and then remove the source. The fallback is not the same single filesystem operation as an in-filesystem rename.
Which error ends this program? from shutil import unpack_archive unpack_archive("unused.archive", format="imaginary")
- FileNotFoundError
- None
- ValueError: Unknown unpack format 'imaginary'
- True
Answer: ValueError: Unknown unpack format 'imaginary'. An explicitly unknown format is rejected before extraction.
What should be checked before recursively removing a generated directory?
- Only that its string is nonempty
- Only that it ends with a slash
- Only that it has a short name
- That its resolved target stays within the explicitly owned boundary
Answer: That its resolved target stays within the explicitly owned boundary. A concrete ownership boundary prevents routine wrong-target errors; arbitrary strings are insufficient.
Continue this course
- Previous: Finding Files with glob & fnmatch
- Next: Network Programming with sockets — Build TCP clients and servers with Python's socket module
- Quick reference: Python cheat sheet