High-Level File Operations (shutil)

Published by Brayan K

The shutil module provides high-level file operations — copying, moving, archiving, and deleting whole files and directory trees — in a single, convenient function call each.

Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Choose the copy contract before choosing the function

A copy can mean only file contents, contents plus permission mode, or an attempt to preserve additional metadata. shutil provides different contracts for these jobs. None is a universal backup system: copy2 still cannot preserve every platform-specific attribute, owner or access-control entry. Keep the original until you have checked the result your application actually needs.

from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import copyfile

with TemporaryDirectory() as directory:
    root = Path(directory)
    source = root / "source.txt"
    destination = root / "destination.txt"
    source.write_text("approved", encoding="utf-8")
    destination.write_text("old draft", encoding="utf-8")
    copyfile(source, destination)
    print(destination.read_text(encoding="utf-8"))
    print(source.read_bytes() == destination.read_bytes())

# ✅ Expected output:
# approved
# True

copyfile takes a complete destination filename and overwrites an existing destination file. It copies contents rather than metadata. copy also accepts a destination directory and copies permission mode; copy2 additionally attempts to preserve metadata through copystat. Metadata support differs across filesystems, so these examples check contents instead of claiming a portable timestamp or permission result.

from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import copy

with TemporaryDirectory() as directory:
    root = Path(directory)
    source = root / "record.txt"
    target = root / "copies"
    target.mkdir()
    source.write_text("record", encoding="utf-8")
    copied = Path(copy(source, target))
    print(copied.parent == target)
    print(copied.read_text(encoding="utf-8"))
    print(source.exists())

# ✅ Expected output:
# True
# record
# True

The copied file retains its basename under the target directory. Do not interchange copy and copyfile while assuming their destination rules are identical. Building and inspecting Path objects is covered in Modern File Paths with pathlib; this lesson owns the operation applied to those paths.

A stream copy starts at the current position

from io import BytesIO
from shutil import copyfileobj

source = BytesIO(b"HEADER:payload")
source.seek(7)
destination = BytesIO()
copyfileobj(source, destination, length=3)
print(destination.getvalue().decode("ascii"))
print(source.closed, destination.closed)
source.close()
destination.close()

# ✅ Expected output:
# payload
# False False

copyfileobj consumes from the current source position. The positive length controls its buffer size, not the total number of bytes to copy. It neither closes the supplied streams nor guarantees that a buffered destination has been flushed before another handle reads it. The caller owns those lifetime and visibility decisions. Working with Files, Streams & Large Datasets owns buffering and stream interfaces in detail.

from io import BytesIO, StringIO
from shutil import copyfileobj

with StringIO("text") as source, BytesIO() as destination:
    copyfileobj(source, destination)

# ✅ Expected error:
# TypeError: a bytes-like object is required, not 'str'

The read method returns strings here, while the destination expects bytes. Choose matching stream types or put an explicit encoding layer between them. A copy operation does not infer an encoding from a filename extension.

Copying a tree is not synchronizing a tree

from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import copytree, ignore_patterns

with TemporaryDirectory() as directory:
    root = Path(directory)
    source = root / "source"
    source.mkdir()
    (source / "nested").mkdir()
    (source / "keep.txt").write_text("A", encoding="utf-8")
    (source / "cache.tmp").write_text("skip", encoding="utf-8")
    (source / "nested" / "keep.txt").write_text("B", encoding="utf-8")
    destination = root / "copied"
    copytree(source, destination, ignore=ignore_patterns("*.tmp"))
    print((destination / "keep.txt").read_text(encoding="utf-8"))
    print((destination / "nested" / "keep.txt").read_text(encoding="utf-8"))
    print((destination / "cache.tmp").exists())

# ✅ Expected output:
# A
# B
# False

By default copytree expects the destination tree not to exist. ignore_patterns supplies a callback that excludes matching names as each source directory is visited. It is a copy-selection rule, not a deletion rule for files already present at the destination. Wildcard behavior is owned by Finding Files with glob & fnmatch.

from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import copytree

with TemporaryDirectory() as directory:
    root = Path(directory)
    source = root / "source"
    destination = root / "destination"
    source.mkdir()
    destination.mkdir()
    (source / "shared.txt").write_text("new", encoding="utf-8")
    (destination / "shared.txt").write_text("old", encoding="utf-8")
    (destination / "extra.txt").write_text("retained", encoding="utf-8")
    copytree(source, destination, dirs_exist_ok=True)
    print((destination / "shared.txt").read_text(encoding="utf-8"))
    print((destination / "extra.txt").read_text(encoding="utf-8"))

# ✅ Expected output:
# new
# retained

dirs_exist_ok allows merging into existing directories; matching files can be overwritten, while extra destination files remain. That is not a mirror or an atomic transaction. A mid-copy failure can leave partial results. For valuable data, build a new destination, validate it, and publish it under an application-specific replacement policy.

Moving may rename or copy and then remove

from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import move

with TemporaryDirectory() as directory:
    root = Path(directory)
    source = (root / "draft.txt").resolve()
    destination = (root / "final.txt").resolve()
    if source.parent != root.resolve() or destination.parent != root.resolve():
        raise ValueError("move must stay inside the owned directory")
    source.write_text("final text", encoding="utf-8")
    move(source, destination)
    print(source.exists())
    print(destination.read_text(encoding="utf-8"))

# ✅ Expected output:
# False
# final text

When possible move uses a rename. Across filesystems it may instead copy and remove the source. Do not promise atomicity or identical metadata behavior for every move. Existing-target rules also depend on whether the destination is a directory and on the underlying rename semantics; choose a fresh destination when the operation must not overwrite another result.

This example creates both paths itself and checks their resolved parent before moving anything. Real applications need an explicit destination and overwrite policy; accepting an arbitrary path and calling move is not a complete file-management feature.

Removal needs a known boundary, not a guessed path

from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import rmtree

with TemporaryDirectory() as directory:
    root = Path(directory).resolve()
    target = (root / "generated").resolve()
    target.mkdir()
    (target / "result.txt").write_text("scratch", encoding="utf-8")
    if target == root or target.parent != root:
        raise ValueError("refuse removal outside the owned child")
    rmtree(target)
    print(target.exists())
    print(root.exists())

# ✅ Expected output:
# False
# True

rmtree removes the tree, including its contents. The guard refuses the root itself and any location outside the explicitly owned child boundary. It catches a wrong-path programming error in this controlled workspace; it is not a complete defense against hostile concurrent filesystem changes. Avoid using this small example as a security sandbox.

ignore_errors=True hides removal failures. Do not use it by default when successful cleanup matters: an open handle, permissions issue or unexpected target should remain visible. The lifetime management of the outer scratch directory is owned by Temporary Files & Directories.

Archive creation and extraction have different trust questions

from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import make_archive
from zipfile import ZipFile

with TemporaryDirectory() as directory:
    root = Path(directory)
    source = root / "source"
    source.mkdir()
    (source / "report.txt").write_text("ready", encoding="utf-8")
    archive = make_archive(str(root / "bundle"), "zip", root_dir=source)
    with ZipFile(archive) as bundle:
        print(len(bundle.namelist()))
        print(bundle.read("report.txt").decode("utf-8"))

# ✅ Expected output:
# 1
# ready

root_dir determines the tree to include; base_name names the archive without its format suffix. The generated archive can contain timestamps and other metadata, so equal file contents do not imply identical archive bytes. Check the logical members and data rather than a fixed compressed size.

from shutil import unpack_archive

unpack_archive("unused.archive", format="imaginary")

# ✅ Expected error:
# ValueError: Unknown unpack format 'imaginary'

Extract only archives whose contents and trust level you understand. Archive members can attempt to escape an intended destination or create problematic links, and extraction defaults differ by format and Python version. This lesson creates and inspects its own ZIP fixture without extracting untrusted data. Streaming archive members belongs to Working with Files, Streams & Large Datasets.

Complete a filtered tree copy

Create a fresh copy that excludes .tmp files. Fill in the copy operation and the ignore pattern, then check the retained content and excluded fixture.

from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import copytree, ignore_patterns

with TemporaryDirectory() as directory:
    root = Path(directory)
    source = root / "source"
    source.mkdir()
    (source / "keep.txt").write_text("keep", encoding="utf-8")
    (source / "scratch.tmp").write_text("skip", encoding="utf-8")
    target = root / "target"
    ___(source, target, ignore=ignore_patterns(___))
    print((target / "keep.txt").read_text(encoding="utf-8"))
    print((target / "scratch.tmp").exists())

# 👉 replace ___ with copytree
# 👉 replace ___ with "*.tmp"

# ✅ Expected output:
# keep
# False

Prepare and verify a fresh release directory

Build an owned source directory containing two text reports and one temporary file. Copy it into a fresh release directory while excluding .tmp files. Compare the retained bytes against the source, then archive the release as ZIP and print the sorted member contents. Leave the source intact and keep every operation inside one temporary workspace.

from pathlib import Path
from tempfile import TemporaryDirectory
from shutil import copytree, ignore_patterns, make_archive
from zipfile import ZipFile

with TemporaryDirectory() as directory:
    root = Path(directory)
    source = root / "source"
    source.mkdir()
    fixtures = {"a.txt": "alpha", "b.txt": "beta", "cache.tmp": "skip"}
    for name, text in fixtures.items():
        (source / name).write_text(text, encoding="utf-8")
    release = root / "release"
    copytree(source, release, ignore=ignore_patterns("*.tmp"))
    kept = ["a.txt", "b.txt"]
    verified = all((source / name).read_bytes() == (release / name).read_bytes()
                   for name in kept)
    if not verified:
        raise ValueError("copied content mismatch")
    archive = make_archive(str(root / "release-package"), "zip", root_dir=release)
    with ZipFile(archive) as bundle:
        print([bundle.read(name).decode("utf-8")
               for name in sorted(bundle.namelist())])
    print((source / "cache.tmp").exists())
    print((release / "cache.tmp").exists())

# ✅ Expected output:
# ['alpha', 'beta']
# True
# False

Practice quiz

What does this program print? from io import BytesIO from shutil import copyfileobj source = BytesIO(b"abcdef") source.seek(2) target = BytesIO() copyfileobj(source, target, length=1) print(target.getvalue())

  • b'cdef'
  • b'a'
  • b'abcdef'
  • b'c'

Answer: b'cdef'. length is the copy buffer size; copying continues from the current position to EOF.

Which function requires a complete destination filename rather than accepting a destination directory?

  • copy
  • copyfile
  • copy2
  • copytree

Answer: copyfile. copyfile copies contents to the exact named destination file.

What does dirs_exist_ok=True do with extra files already in the destination?

  • Deletes them
  • Moves them to the source
  • Leaves them unless another operation changes them
  • Guarantees a byte-identical mirror

Answer: Leaves them unless another operation changes them. copytree merges and may overwrite matching files; it does not synchronize deletions.

What does this program print? from io import BytesIO from shutil import copyfileobj a = BytesIO(b"x") b = BytesIO() copyfileobj(a, b) print(a.closed, b.closed)

  • True True
  • True False
  • False True
  • False False

Answer: False False. The caller retains ownership of both supplied streams.

Which claim about copy2 is accurate?

  • It attempts additional metadata preservation but cannot preserve everything
  • It always preserves owners and ACLs everywhere
  • It creates a compressed backup
  • It never overwrites an existing file

Answer: It attempts additional metadata preservation but cannot preserve everything. Filesystem and platform metadata capabilities limit what can be preserved.

Why can move across filesystems behave differently from rename?

  • It never removes the source
  • It may copy data and then remove the source
  • It changes every file into text
  • It always creates a ZIP archive

Answer: It may copy data and then remove the source. The fallback is not the same single filesystem operation as an in-filesystem rename.

Which error ends this program? from shutil import unpack_archive unpack_archive("unused.archive", format="imaginary")

  • FileNotFoundError
  • None
  • ValueError: Unknown unpack format 'imaginary'
  • True

Answer: ValueError: Unknown unpack format 'imaginary'. An explicitly unknown format is rejected before extraction.

What should be checked before recursively removing a generated directory?

  • Only that its string is nonempty
  • Only that it ends with a slash
  • Only that it has a short name
  • That its resolved target stays within the explicitly owned boundary

Answer: That its resolved target stays within the explicitly owned boundary. A concrete ownership boundary prevents routine wrong-target errors; arbitrary strings are insufficient.

Continue this course

Related lessons