Logging, Debugging & Error Handling
Published by Brayan K
When your codebase is tiny, print() and random try/except blocks "work". Once you have APIs, background workers, cron jobs, queues, and multiple servers, that approach falls apart. Learn professional-grade debugging + logging + error-handling systems used by real engineering teams.
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
Standard output and standard error
This lesson starts where Exception Handling stops. That lesson catches an error and decides what happens next; this one records it, with Python's logging module, so you can read it later, and then shows how to stop a running program and look inside it with Python's debugger, pdb. Exception classes of your own and raise … from belong to Custom Exceptions & Exception Chaining, and warnings to The warnings Module.
A program has two streams for text. print() writes to standard output. Standard error is the second stream, meant for problems and reports, and it's where logging writes unless you tell it otherwise. A terminal shows both, so they look like one stream until you separate them. On Linux and macOS, these two commands do:
python app.py 2>errors.log
python app.py >/dev/nullWith an app.py that prints report ready and then calls logging.warning("disk almost full"), the first command shows report ready and puts WARNING:root:disk almost full in errors.log; the second throws standard output away and shows only the warning. A logger you make yourself writes to standard error too:
import logging
log = logging.getLogger("shop")
print("report ready")
log.info("3 orders read")
log.warning("disk almost full")
print("2 files written")
# ✅ Expected output:
# report ready
# 2 files writtenIn a terminal a third line appears between those two: disk almost full, on standard error. 3 orders read appears nowhere. Until logging is set up, a logger passes on only WARNING and above, and they come out through Python's last-resort handler, which writes the message alone to standard error. This site's editor shows such a line after everything printed, in yellow and marked ⚠️. The panels show standard output, so the examples from here on send their log lines there.
Every logger reports to one at the top, the root logger. Before anything is set up, its level is WARNING and it has no handlers, the objects that write lines somewhere:
import logging
root = logging.getLogger()
print(root.level, logging.getLevelName(root.level))
print(len(root.handlers))
# ✅ Expected output:
# 30 WARNING
# 0The module's own functions, such as logging.warning("disk almost full"), set the root logger up the first time one is called: a handler on standard error, with the format %(levelname)s:%(name)s:%(message)s. That's where WARNING:root:disk almost full above came from. The examples below set logging up themselves instead.
Set logging up once, with basicConfig()
logging.basicConfig() sets up the root logger: the lowest level to show, the format of each line, and where the lines go. A program does it once, where it starts:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="%(levelname)s %(name)s: %(message)s")
log = logging.getLogger("shop")
log.debug("basket has 3 items")
log.info("order 7 received")
log.warning("card declined for order 7")
# ✅ Expected output:
# INFO shop: order 7 received
# WARNING shop: card declined for order 7The DEBUG line is below the INFO level, so it's dropped. Two of the arguments are there for this site. stream=sys.stdout sends the lines to standard output, so the panel can show them; a real program would usually leave them on standard error. force=True replaces any set-up that's already there: Python keeps its loggers for as long as the program runs, and this site's editor keeps one Python for every Run on a page, so without it a second Run would print nothing. A program run in a terminal starts fresh each time and doesn't need it.
Without force=True, a second call does nothing at all:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.WARNING, force=True,
format="%(levelname)s %(name)s: %(message)s")
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG,
format="%(message)s")
log = logging.getLogger("shop")
log.debug("basket has 3 items")
log.warning("card declined")
# ✅ Expected output:
# WARNING shop: card declinedIts documentation says so: it does nothing "if the root logger already has handlers configured, unless the keyword argument force is set to True". The DEBUG line stays hidden, and the format is still the first one. With force=True the second call wins:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.WARNING, force=True,
format="%(levelname)s %(name)s: %(message)s")
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG, force=True,
format="%(message)s")
log = logging.getLogger("shop")
log.debug("basket has 3 items")
log.warning("card declined")
# ✅ Expected output:
# basket has 3 items
# card declinedThe usual way to fall into this: a module logs something as it's imported, before the program reaches its basicConfig().
import logging
import sys
logging.info("loading settings")
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG,
format="%(levelname)s %(name)s: %(message)s")
logging.info("settings loaded")
root = logging.getLogger()
print(len(root.handlers), logging.getLevelName(root.level))
# ✅ Expected output:
# 1 WARNINGThe first logging.info() set the root logger up, with one handler on standard error and the level WARNING. So the later basicConfig() was ignored, and neither INFO line was shown.
Loggers have names, and the dots make a tree
logging.getLogger() returns the same logger every time it's given the same name, as Design Patterns in Python (Singleton, Factory, Strategy) pointed out. So each part of a program asks for its logger by name instead of having one passed to it:
import logging
a = logging.getLogger("shop")
b = logging.getLogger("shop")
c = logging.getLogger("admin")
print(a is b, a is c)
print(a.name, c.name)
# ✅ Expected output:
# True False
# shop adminThe usual name is the module's own, __name__. In the program you run that's "__main__", as Modules and Packages showed, and in a module of a package it's the module's dotted name:
from pathlib import Path
Path("shop").mkdir(exist_ok=True)
Path("shop/__init__.py").write_text("", encoding="utf-8")
Path("shop/checkout.py").write_text(
"import logging\n"
"\n"
"log = logging.getLogger(__name__)\n"
"\n"
"def pay(order):\n"
" log.info('order %s paid', order)\n",
encoding="utf-8",
)
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="%(levelname)s %(name)s: %(message)s")
import shop.checkout
shop.checkout.pay(7)
# ✅ Expected output:
# INFO shop.checkout: order 7 paidThe dots make a tree: shop.checkout is a child of shop, and every logger is a descendant of the root. A line goes up that tree, and each handler on the way writes it. Neither of these loggers has a handler, so the root's writes both lines:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="%(levelname)s %(name)s: %(message)s")
logging.getLogger("shop.checkout").info("order 7 paid")
logging.getLogger("shop.stock").warning("only 2 left")
# ✅ Expected output:
# INFO shop.checkout: order 7 paid
# WARNING shop.stock: only 2 leftSo a handler on shop and another on the root write each of its lines twice:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="root's handler: %(message)s")
shop = logging.getLogger("shop")
shop.handlers.clear()
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logging.Formatter("shop's handler: %(message)s"))
shop.addHandler(handler)
logging.getLogger("shop.checkout").info("order 7 paid")
# ✅ Expected output:
# shop's handler: order 7 paid
# root's handler: order 7 paidpropagate = False stops a line at that logger, so the root's handler never sees it. (shop.handlers.clear() is there for this site's editor, which keeps loggers between Runs: without it, every Run would add one more handler.)
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="root's handler: %(message)s")
shop = logging.getLogger("shop")
shop.handlers.clear()
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logging.Formatter("shop's handler: %(message)s"))
shop.addHandler(handler)
shop.propagate = False
logging.getLogger("shop.checkout").info("order 7 paid")
# ✅ Expected output:
# shop's handler: order 7 paidA set-up function that runs twice does the same thing, one handler per call:
import logging
import sys
def setup():
log = logging.getLogger("shop")
log.setLevel(logging.INFO)
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logging.Formatter("%(levelname)s %(name)s: %(message)s"))
log.addHandler(handler)
return log
logging.getLogger("shop").handlers.clear()
setup()
log = setup()
log.info("order 7 paid")
print(len(log.handlers), "handlers")
# ✅ Expected output:
# INFO shop: order 7 paid
# INFO shop: order 7 paid
# 2 handlersSo set logging up once, where the program starts, and let each module only ask for its logger. A library goes further and sets nothing up, leaving that to the program that uses it; the most it adds is logging.NullHandler(), a handler that writes nothing and keeps the last-resort handler quiet.
Levels: two checks decide whether a line appears
Each level is a number: DEBUG 10, INFO 20, WARNING 30, ERROR 40 and CRITICAL 50. A line is shown when its level passes two checks, the logger's level first and then the level of each handler that would write it. A logger with no level of its own, 0, uses its parent's, and so on up to the root:
import logging
logging.getLogger("shop").setLevel(logging.DEBUG)
db = logging.getLogger("shop.db")
print(db.level, db.getEffectiveLevel())
print(logging.getLogger("admin").getEffectiveLevel())
# ✅ Expected output:
# 0 10
# 30getEffectiveLevel() says which level applies. That's how to quiet one noisy part of a program, by giving its logger a higher level:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG, force=True,
format="%(levelname)s %(name)s: %(message)s")
logging.getLogger("shop.db").setLevel(logging.WARNING)
logging.getLogger("shop.db").debug("SELECT * FROM orders")
logging.getLogger("shop.db").warning("slow query")
logging.getLogger("shop.checkout").debug("basket has 3 items")
# ✅ Expected output:
# WARNING shop.db: slow query
# DEBUG shop.checkout: basket has 3 itemsA handler's level can only drop more lines. This handler takes DEBUG, but shop has no level and the root's is WARNING, so the DEBUG line never gets as far as the handler:
import logging
import sys
log = logging.getLogger("shop")
log.handlers.clear()
log.setLevel(logging.NOTSET)
log.propagate = False
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)
log.addHandler(handler)
log.debug("basket has 3 items")
log.warning("card declined")
# ✅ Expected output:
# card declinedA handler at ERROR next to one that takes everything sends the serious lines to a second place as well; the practice exercise below builds one. Levels can also be given by name, in capitals: log.setLevel("INFO") works. In lower case it stops the program:
import logging
log = logging.getLogger("shop")
log.setLevel("info")
# ✅ Expected error:
# ValueError: Unknown level: 'info'Give values as arguments, not in an f-string
f-strings & String Formatting mentioned that % formatting still turns up in logging calls. This is why: a logging call takes a message with %s placeholders and the values after it, and fills them in only for a line it's going to show.
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="%(levelname)s %(name)s: %(message)s")
log = logging.getLogger("shop")
class Basket:
def __str__(self):
print(" (making the basket's text)")
return "3 items"
basket = Basket()
log.debug("basket: %s", basket)
log.info("basket: %s", basket)
# ✅ Expected output:
# (making the basket's text)
# INFO shop: basket: 3 itemsThe basket's text was made once, for the INFO line; the DEBUG line was dropped before anything was filled in. An f-string is built before the call, so it's made for the hidden line too:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="%(levelname)s %(name)s: %(message)s")
log = logging.getLogger("shop")
class Basket:
def __str__(self):
print(" (making the basket's text)")
return "3 items"
basket = Basket()
log.debug(f"basket: {basket}")
log.info(f"basket: {basket}")
# ✅ Expected output:
# (making the basket's text)
# (making the basket's text)
# INFO shop: basket: 3 itemsThe values themselves are still worked out, though: log.debug("items: %s", count_items()) calls count_items() whether the line is shown or not. For something slow, check first, with if log.isEnabledFor(logging.DEBUG):.
The comma that works in print() doesn't work here:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="%(levelname)s %(name)s: %(message)s")
log = logging.getLogger("shop")
total = 12.5
log.info("Total:", total)
print("still running")
# ✅ Expected output:
# still running12.5 is a value with no %s to go into, so formatting fails inside logging, and logging doesn't stop the program over it. The line is lost, and a report goes to standard error instead. It starts --- Logging error ---, names TypeError: not all arguments converted during string formatting, lists the calls that led there, and ends with Message: 'Total:' and Arguments: (12.5,).
With no values after it, a message is written as it is, so a % sign is fine. With values, the message is a % format, and a percent sign is written %%:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="%(levelname)s %(name)s: %(message)s")
log = logging.getLogger("shop")
log.info("50% done")
log.info("50%% done for order %s", 7)
# ✅ Expected output:
# INFO shop: 50% done
# INFO shop: 50% done for order 7The format decides what each line says
A format names the fields each line shows, each written %(name)s, or %(name)d for a number. -8 pads a field to eight characters, and funcName and lineno say where the call was made:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="%(levelname)-8s %(name)s %(funcName)s:%(lineno)d %(message)s")
def charge(order):
logging.getLogger("shop").warning("card declined for %s", order)
charge("A-7")
# ✅ Expected output:
# WARNING shop charge:8 card declined for A-7A format written with braces, as in str.format(), needs the argument style="{" as well. Without it, the set-up stops:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="{levelname} {name}: {message}")
logging.getLogger("shop").info("order 7 received")
# ✅ Expected error:
# ValueError: Invalid format '{levelname} {name}: {message}' for '%' styleextra= adds fields of your own to a line's record, and the format shows them by name:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="%(levelname)s user=%(user)s %(message)s")
log = logging.getLogger("shop")
log.warning("card declined", extra={"user": "ada"})
# ✅ Expected output:
# WARNING user=ada card declinedA field the format doesn't name is on the record but not in the line, so the default format leaves user_id out here. (A key the record already has, such as message, stops the call with KeyError: "Attempt to overwrite 'message' in LogRecord".)
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True)
logging.getLogger("web").info("user logged in", extra={"user_id": 7})
# ✅ Expected output:
# INFO:web:user logged inPrograms that collect log lines can search them by field when each line is JSON. A formatter of your own decides the whole line; this one writes one JSON object per line with json.dumps(), as Working with JSON showed:
import json
import logging
import sys
class JsonLines(logging.Formatter):
def format(self, record):
entry = {"level": record.levelname, "logger": record.name,
"message": record.getMessage()}
entry.update(getattr(record, "fields", {}))
return json.dumps(entry)
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonLines())
logging.basicConfig(level=logging.INFO, handlers=[handler], force=True)
log = logging.getLogger("web")
log.info("user logged in", extra={"fields": {"user_id": 7, "ip": "10.0.0.5"}})
log.warning("card declined")
# ✅ Expected output:
# {"level": "INFO", "logger": "web", "message": "user logged in", "user_id": 7, "ip": "10.0.0.5"}
# {"level": "WARNING", "logger": "web", "message": "card declined"}A filter on a handler sees every line that handler writes, and can add a field to each one, such as the id of the request being handled:
import logging
import sys
class AddRequestId(logging.Filter):
def __init__(self, request_id):
super().__init__()
self.request_id = request_id
def filter(self, record):
record.request_id = self.request_id
return True
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logging.Formatter("[%(request_id)s] %(name)s: %(message)s"))
handler.addFilter(AddRequestId("r-1"))
logging.basicConfig(level=logging.INFO, handlers=[handler], force=True)
logging.getLogger("web").info("GET /checkout")
logging.getLogger("web.payments").warning("card declined")
# ✅ Expected output:
# [r-1] web: GET /checkout
# [r-1] web.payments: card declinedThe filter is on the handler for a reason. A filter on a logger sees only that logger's own calls, not its children's: on web, it would miss web.payments' line, which would then fail with a --- Logging error --- report, since its format names a field that line doesn't have.
Some fields change from run to run, so no panel here shows them: %(asctime)s, the date and time of the line, and %(process)d, the number of the running process. A real format usually starts with %(asctime)s.
Recording an error you've caught
Inside an except block, log.exception("charge failed") logs an ERROR line followed by the whole traceback, the report Python prints for an error nothing catches. A traceback's middle lines name files and line numbers, and they differ between computers, Python versions and this site's editor, so the examples here use a formatter that keeps only its last line:
import logging
import sys
import traceback
class ShortErrors(logging.Formatter):
def formatException(self, exc_info):
return " " + traceback.format_exception_only(exc_info[1])[-1].strip()
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(ShortErrors("%(levelname)s %(name)s: %(message)s"))
logging.basicConfig(level=logging.INFO, handlers=[handler], force=True)
log = logging.getLogger("shop")
def charge(amount):
if amount > 1000:
raise ValueError("amount over limit")
return amount
try:
charge(5000)
except ValueError:
log.exception("charge failed")
print("still running")
# ✅ Expected output:
# ERROR shop: charge failed
# ValueError: amount over limit
# still runningWith the usual formatter, the same call writes ERROR shop: charge failed, then the traceback from Traceback (most recent call last): down to ValueError: amount over limit, and the program carries on. log.error() with the error in the message records what went wrong, but not where:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="%(levelname)s %(name)s: %(message)s")
log = logging.getLogger("shop")
def charge(amount):
if amount > 1000:
raise ValueError("amount over limit")
return amount
try:
charge(5000)
except ValueError as e:
log.error("charge failed: %s", e)
# ✅ Expected output:
# ERROR shop: charge failed: amount over limitlog.exception() is log.error() with exc_info=True: it logs at ERROR and adds the error being handled. Outside an except block there's none to add:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="%(levelname)s %(name)s: %(message)s")
log = logging.getLogger("shop")
log.error("charge failed", exc_info=True)
# ✅ Expected output:
# ERROR shop: charge failed
# NoneType: NoneLogging an error where it's caught and then re-raising it writes the same failure twice:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="%(levelname)s %(name)s: %(message)s")
log = logging.getLogger("shop")
def charge(amount):
if amount > 1000:
raise ValueError("amount over limit")
return amount
def checkout(amount):
try:
return charge(amount)
except ValueError:
log.error("checkout failed")
raise
try:
checkout(5000)
except ValueError:
log.error("request failed")
# ✅ Expected output:
# ERROR shop: checkout failed
# ERROR shop: request failedLog an error once, in the place that handles it. A program that runs for a long time, such as a batch job or a server, logs the error and carries on with the next piece of work, so one bad row costs one line in the log rather than the whole run:
import logging
import sys
import traceback
class ShortErrors(logging.Formatter):
def formatException(self, exc_info):
return " " + traceback.format_exception_only(exc_info[1])[-1].strip()
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(ShortErrors("%(levelname)s %(name)s: %(message)s"))
logging.basicConfig(level=logging.INFO, handlers=[handler], force=True)
log = logging.getLogger("import")
rows = [("A-1", "2"), ("A-2", "two"), ("A-3", "5")]
imported = 0
for order, qty in rows:
try:
count = int(qty)
except ValueError:
log.exception("order %s skipped", order)
continue
imported += 1
log.info("order %s: %d items", order, count)
log.info("%d of %d orders imported", imported, len(rows))
# ✅ Expected output:
# INFO import: order A-1: 2 items
# ERROR import: order A-2 skipped
# ValueError: invalid literal for int() with base 10: 'two'
# INFO import: order A-3: 5 items
# INFO import: 2 of 3 orders importedAn error nothing catches can be logged too. Just before such an error stops the program, Python calls sys.excepthook with it:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="%(levelname)s %(name)s: %(message)s")
log = logging.getLogger("shop")
def log_uncaught(kind, error, tb):
log.critical("uncaught %s: %s", kind.__name__, error)
sys.excepthook = log_uncaught
print("starting")
int("ten")Run with python3, this prints starting and then CRITICAL shop: uncaught ValueError: invalid literal for int() with base 10: 'ten' instead of the usual report, and the program ends with exit status 1, as any failed program does. This site's editor shows its own report of the error and doesn't call the hook.
Reading a traceback in code: the traceback module
Introduction to Python read a traceback from the bottom up, and Exception Handling's traceback.print_exc() prints the report and lets the program carry on; it writes to standard error, as logging does. The traceback module also hands you the report's parts. extract_tb() lists the calls that led to the error, from the outside in:
import traceback
def charge(amount):
if amount > 1000:
raise ValueError("amount over limit")
return amount
def checkout(amount):
return charge(amount)
try:
checkout(5000)
except ValueError as e:
for frame in traceback.extract_tb(e.__traceback__):
print(frame.name, "line", frame.lineno)
print(traceback.format_exception_only(e))
# ✅ Expected output:
# <module> line 12
# checkout line 9
# charge line 5
# ['ValueError: amount over limit\n']Line 12 is the call at the bottom of the program, line 9 the call inside checkout(), and line 5 the raise. format_exc() returns the whole report as text, the same text print_exc() prints:
import traceback
def charge(amount):
if amount > 1000:
raise ValueError("amount over limit")
return amount
try:
charge(5000)
except ValueError:
report = traceback.format_exc()
lines = report.splitlines()
print(lines[0])
print(lines[-1])
# ✅ Expected output:
# Traceback (most recent call last):
# ValueError: amount over limitAn error raised while another is being handled gives a report in two parts, joined by a sentence that says how they're related:
import traceback
prices = {"tea": 2}
try:
try:
prices["coffee"]
except KeyError:
int("ten")
except ValueError:
for line in traceback.format_exc().splitlines():
if not line.startswith(" "):
print(line)
# ✅ Expected output:
# Traceback (most recent call last):
# KeyError: 'coffee'
#
# During handling of the above exception, another exception occurred:
#
# Traceback (most recent call last):
# ValueError: invalid literal for int() with base 10: 'ten'raise … from joins them with a different sentence, "The above exception was the direct cause of the following exception:". Custom Exceptions & Exception Chaining covers both.
Log files
A FileHandler writes lines to a file, and the file can be read back like any other. (This site's editor empties its folder before each Run, so the file starts empty every time.)
import logging
from pathlib import Path
handler = logging.FileHandler("app.log", encoding="utf-8")
handler.setFormatter(logging.Formatter("%(levelname)s %(name)s: %(message)s"))
logging.basicConfig(level=logging.INFO, handlers=[handler], force=True)
log = logging.getLogger("shop")
log.info("order 7 received")
log.warning("card declined for order 7")
handler.close()
print(Path("app.log").read_text(encoding="utf-8"), end="")
# ✅ Expected output:
# INFO shop: order 7 received
# WARNING shop: card declined for order 7A log file grows for as long as the program runs. RotatingFileHandler starts a new file when the current one would pass maxBytes, keeps backupCount old ones, and deletes the oldest. The limit here is tiny, so every line gets a file of its own:
import logging
from logging.handlers import RotatingFileHandler
from pathlib import Path
handler = RotatingFileHandler("app.log", maxBytes=40, backupCount=2, encoding="utf-8")
handler.setFormatter(logging.Formatter("%(message)s"))
log = logging.getLogger("shop")
log.handlers.clear()
log.addHandler(handler)
log.propagate = False
for n in range(1, 8):
log.warning("order %d is waiting for payment", n)
handler.close()
for path in sorted(Path(".").glob("app.log*")):
print(path.name, "|", path.read_text(encoding="utf-8").strip())
# ✅ Expected output:
# app.log | order 7 is waiting for payment
# app.log.1 | order 6 is waiting for payment
# app.log.2 | order 5 is waiting for paymentFor a bigger set-up, logging.config.dictConfig() takes all of it as one dictionary: formats, handlers, and a level for each logger.
import logging
import logging.config
logging.config.dictConfig({
"version": 1,
"disable_existing_loggers": False,
"formatters": {"short": {"format": "%(levelname)s %(name)s: %(message)s"}},
"handlers": {"console": {"class": "logging.StreamHandler", "formatter": "short",
"stream": "ext://sys.stdout"}},
"loggers": {"shop.db": {"level": "WARNING"}},
"root": {"level": "DEBUG", "handlers": ["console"]},
})
logging.getLogger("shop.checkout").debug("basket has 3 items")
logging.getLogger("shop.db").debug("SELECT * FROM orders")
logging.getLogger("shop.db").warning("slow query")
# ✅ Expected output:
# DEBUG shop.checkout: basket has 3 items
# WARNING shop.db: slow query"disable_existing_loggers": False keeps existing loggers enabled. With the default True, existing non-root loggers are disabled unless they or their ancestors are named in the configuration.
Debugging with pdb
A log says what a program did. A debugger stops it at a line you choose so you can look at its variables there. Python's is pdb, and it needs a terminal to type its commands into, so this part is for a terminal rather than this site's editor. Take this checkout.py:
def total(prices):
result = 0
for price in prices:
result += price
return result
print(total([2, 3]))python -m pdb checkout.py starts it under pdb, stopped before its first line, with a (Pdb) prompt waiting for a command. These commands, typed one at a time, give the same answers on Python 3.10–3.14:
- break 4 puts a breakpoint on line 4, result += price, and continue runs until it's reached.
- p price, result prints (2, 0): the line hasn't run yet.
- continue stops at the same line on the next turn of the loop, and p result prints 2.
- clear 1 deletes the breakpoint, and continue runs to the end: the program prints 5, then pdb says The program finished and will be restarted.
- quit leaves pdb.
next runs one line, step goes into the function a line calls, and where shows the calls that led to the current line. breakpoint() written in your code stops there when the program is run normally. Where exactly differs: Python 3.10–3.12 stop on the line after it, and 3.13 and later on the breakpoint() line itself. python -m pdb -c continue crash.py runs a program until it fails, then stops at the failing line with its variables still there to print.
breakpoint() calls sys.breakpointhook(), which starts pdb, and a program can replace it:
import sys
def report(*args, **kwargs):
print("breakpoint() called sys.breakpointhook")
sys.breakpointhook = report
print("before")
breakpoint()
print("after")
# ✅ Expected output:
# before
# breakpoint() called sys.breakpointhook
# afterWith Python's default sys.breakpointhook, setting PYTHONBREAKPOINT to 0 disables breakpoint(). A custom hook, such as the preceding example's, decides its own behavior and need not read that environment variable. Start a fresh editor or press Reset before trying the default-hook example. In a terminal it's PYTHONBREAKPOINT=0 python app.py; from inside the program:
import os
os.environ["PYTHONBREAKPOINT"] = "0"
print("before")
breakpoint()
print("after")
# ✅ Expected output:
# before
# afterassert: a check that can be switched off
assert checks something that should always be true at that point, and stops the program with an AssertionError, and your message, when it isn't:
def add_line(total, price):
assert price >= 0, f"price {price} is negative"
return total + price
print(add_line(10, 5))
print(add_line(10, -5))
# ✅ Expected output:
# 15
# ✅ Expected error:
# AssertionError: price -5 is negativeRunning a program with python -O removes every assert from it: the same function then returns 5 for the bad price, without a word. So an assert catches your own mistakes while you work, and a check on data from outside, such as what a user typed, needs an if and an error that's always raised. Tests use assert too, in Testing with pytest: Fixtures, Parametrisation & Mocks.
Practice: send errors to a second handler
The logger shop should write every line, from DEBUG up, through the handler everything, and the errors once more through alerts. Fill in the two levels: the logger's own, so nothing is dropped before the handlers, and the alerts handler's.
import logging
import sys
log = logging.getLogger("shop")
log.handlers.clear()
log.setLevel(logging.___)
log.propagate = False
everything = logging.StreamHandler(sys.stdout)
everything.setFormatter(logging.Formatter("log: %(levelname)s %(message)s"))
alerts = logging.StreamHandler(sys.stdout)
alerts.setLevel(logging.___)
alerts.setFormatter(logging.Formatter("ALERT: %(levelname)s %(message)s"))
log.addHandler(everything)
log.addHandler(alerts)
log.debug("basket has 3 items")
log.info("order 7 paid")
log.error("payment service down")
# 👉 replace ___ with DEBUG
# 👉 replace ___ with ERROR
# ✅ Expected output:
# log: DEBUG basket has 3 items
# log: INFO order 7 paid
# log: ERROR payment service down
# ALERT: ERROR payment service downOn your own: an order import that logs
Write the import for this list of orders:
- Set logging up to print LEVEL name: message lines on standard output, at INFO, and log through a logger named orders.import.
- A quantity that isn't a number: a WARNING, order A-2 skipped: quantity 'two' is not a number, and carry on.
- A quantity of 0 or less: an ERROR, order A-4 skipped: quantity -1.
- Any other order: an INFO line, order A-1: 2 items.
- At the end, an INFO line, 2 of 4 orders imported. Give the values as arguments, not in f-strings.
import logging
import sys
orders = [("A-1", "2"), ("A-2", "two"), ("A-3", "5"), ("A-4", "-1")]
# your code hereimport logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True,
format="%(levelname)s %(name)s: %(message)s")
log = logging.getLogger("orders.import")
orders = [("A-1", "2"), ("A-2", "two"), ("A-3", "5"), ("A-4", "-1")]
imported = 0
for order, qty in orders:
try:
count = int(qty)
except ValueError:
log.warning("order %s skipped: quantity %r is not a number", order, qty)
continue
if count <= 0:
log.error("order %s skipped: quantity %d", order, count)
continue
imported += 1
log.info("order %s: %d items", order, count)
log.info("%d of %d orders imported", imported, len(orders))
# ✅ Expected output:
# INFO orders.import: order A-1: 2 items
# WARNING orders.import: order A-2 skipped: quantity 'two' is not a number
# INFO orders.import: order A-3: 5 items
# ERROR orders.import: order A-4 skipped: quantity -1
# INFO orders.import: 2 of 4 orders imported
Practice quiz
Nothing is set up. A program runs logging.info("starting") and then logging.warning("low disk"). What does the terminal show?
- INFO:root:starting, then WARNING:root:low disk
- WARNING:root:low disk
- low disk
- Nothing
Answer: WARNING:root:low disk. The first call sets the root logger up with a handler on standard error and the format levelname:name:message. The root's level is WARNING, so only the second line is shown.
logging.basicConfig(stream=sys.stdout, level=logging.WARNING, force=True), then logging.basicConfig(stream=sys.stdout, level=logging.DEBUG), then logging.debug("checking stock") and logging.warning("low stock"). What prints?
- DEBUG:root:checking stock, then WARNING:root:low stock
- low stock
- WARNING:root:low stock
- It stops with ValueError
Answer: WARNING:root:low stock. The second basicConfig() does nothing, because the root logger already has a handler; only force=True would replace it.
basicConfig(stream=sys.stdout, format="%(message)s", level=logging.INFO, force=True). The logger "shop" also gets its own StreamHandler(sys.stdout). Then logging.getLogger("shop.checkout").info("paid"). What prints?
- paid
- Nothing: shop.checkout has no handler
- paid, then INFO:shop.checkout:paid
- paid, twice
Answer: paid, twice. The record goes up the tree to every handler on the way: shop's handler writes it, then the root's handler writes it again.
Logging is set up to print INFO lines. A program runs logging.getLogger("shop").info("Total:", 12.5) and then print("done"). What happens?
- done is printed, and a report starting --- Logging error --- goes to standard error
- Total: 12.5, then done
- It stops with TypeError: not all arguments converted during string formatting
- Total:, then done
Answer: done is printed, and a report starting --- Logging error --- goes to standard error. Extra values fill %s placeholders. With none in the message, formatting fails inside logging, which reports it on standard error and lets the program carry on.
The level is INFO. Basket's __str__ prints "str called". The program runs log.debug(f"basket: {basket}") and then log.debug("basket: %s", basket). How many times is "str called" printed?
- Never
- Twice
- Once
- Three times
Answer: Once. The f-string is built before debug() is called. With %s, logging fills the message only for a line it is going to show, and DEBUG lines aren't shown.
Outside any except block, with the format "%(levelname)s %(message)s": log.error("charge failed", exc_info=True). What prints?
- ERROR charge failed
- It stops with ValueError
- ERROR charge failed, then a traceback of the error() call
- ERROR charge failed, then NoneType: None
Answer: ERROR charge failed, then NoneType: None. exc_info=True records the error being handled. There is none, so the traceback part says NoneType: None.
A program runs logging.debug("loading settings"), then logging.basicConfig(stream=sys.stdout, level=logging.DEBUG, format="%(message)s"), then logging.debug("settings loaded") and print("end"). What does standard output show?
- end
- settings loaded, then end
- loading settings, settings loaded, end
- It stops with ValueError
Answer: end. The first logging.debug() call set the root logger up (standard error, level WARNING), so the later basicConfig() did nothing and no DEBUG line is shown.
load() runs log.error("load failed") in its except block and re-raises; the caller catches the same error and runs log.error("request failed"). The format is "%(levelname)s %(message)s". What prints?
- ERROR request failed
- ERROR load failed, then ERROR request failed
- ERROR load failed
- It stops with ValueError
Answer: ERROR load failed, then ERROR request failed. Each except block logs it, so one failure gives two error lines. Log an error once, where it's handled.
Continue this course
- Previous: Module & Package Architecture for Large Codebases
- Next: Testing with pytest: Fixtures, Parametrisation & Mocks — Write reliable tests with pytest fixtures, parametrize, and unittest.mock
- Quick reference: Python cheat sheet › Logging & Debugging
- From the blog: Error Handling Best Practices in Python
Frequently asked questions
Is logging safe to use from several threads?
Yes: each handler takes a lock while it writes a line, so lines from different threads don't mix. Four threads writing 2,000 lines each through one handler gave 8,000 whole lines on Python 3.10–3.14. This site's editor can't start threads; Concurrency in Python: Threads vs Processes covers them.
How do I get warnings from the warnings module into my logs?
Call logging.captureWarnings(True). From then on, a warning goes to the logger named py.warnings at level WARNING instead of straight to standard error, so your handlers and formats apply to it. The warnings Module covers warnings themselves.
Can every line of one request carry its id without passing the id around?
Yes, with a contextvars.ContextVar that a handler filter reads, like the request-id filter above. Set the variable when a request starts; each asyncio task gets its own copy, so two requests handled at once keep their own ids. AsyncIO: Event Loop, Tasks & Futures covers tasks.
Can I keep my logging set-up in a file?
dictConfig() takes a dictionary, and a dictionary can come from a JSON file, as Working with JSON shows. Keep "disable_existing_loggers": False to leave existing loggers enabled; otherwise existing non-root loggers neither named by the configuration nor descended from a named logger are disabled. logging.config.fileConfig() reads the older INI-style files that Config Files with configparser covers.
Why does logger.warn() work?
It's an old spelling of warning(). It still logs on Python 3.10–3.14, and adds a DeprecationWarning on standard error: The 'warn' method is deprecated, use 'warning' instead.
What is %(taskName)s in a format?
The name of the asyncio task that made the line, from Python 3.12: inside a task named fetch-1 a line can read fetch-1: fetching, and outside any task the field is None. Python 3.10 and 3.11 have no such field, nor does this site's editor, which runs Python 3.11: there every line with it fails with a --- Logging error --- report.
Why do log lines show up in yellow, after everything printed, in this site's editor?
They're on standard error. The editor shows what the program printed first, then each line written to standard error, marked ⚠️. A terminal shows the two streams in the order they were written.