Strings, Bytes & Unicode Encodings

Published by Brayan K

In Python a str is readable Unicode text while bytes is raw binary data, and you move between them with encode and decode using an encoding such as UTF-8 — the foundation for working correctly with files, networks, and every language's characters.

Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Text and its encoded bytes count different things

A Python string stores Unicode code points. An encoded byte sequence stores integer byte values. UTF-8 can use several bytes for one code point, so a text position and a byte position are not interchangeable. Encode when converting text for a binary boundary; decode when interpreting bytes as text.

text = "café"
data = text.encode("utf-8")
print("Text length:", len(text))
print("Byte length:", len(data))
print("Last text item:", text[-1])
print("Last byte:", data[-1])
print("Bytes:", list(data))
print("Round trip:", data.decode("utf-8") == text)

# ✅ Expected output:
# Text length: 4
# Byte length: 5
# Last text item: é
# Last byte: 169
# Bytes: [99, 97, 102, 195, 169]
# Round trip: True

Indexing bytes returns an integer, whereas slicing bytes returns another byte sequence. The last two bytes here jointly represent the final text character. Ordinary string transformations such as stripping and splitting are covered by String Methods.

data = "café".encode("utf-8")
print(str(data))
print(data.decode("utf-8"))
print(data[:3])

# ✅ Expected output:
# b'caf\xc3\xa9'
# café
# b'caf'

Calling str on bytes produces a representation with a b prefix and escapes. It is useful for inspection, not for turning the payload back into its intended text. Use decode with the encoding specified by the producer or format.

Do not mix the two types at a text operation

heading = "Name: "
payload = b"Asha"
print(heading + payload)

# ✅ Expected error:
# TypeError: can only concatenate str (not "bytes") to str

Decode the payload before joining it to text, or encode the heading if the result must be bytes. The choice depends on the boundary, not on whichever conversion removes the exception fastest. Keep the middle of a text-processing program in str, with explicit conversions where data enters or leaves.

record = b"Asha,tea"
parts = record.split(b",")
print(parts)
print([part.decode("utf-8") for part in parts])

# ✅ Expected output:
# [b'Asha', b'tea']
# ['Asha', 'tea']

A decoding error points to bytes and an attempted encoding

data = b"caf\xe9"
print(data.decode("utf-8"))

# ✅ Expected error:
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data

The error reports the byte position and why the chosen decoder cannot interpret it. This payload was deliberately created with Latin-1 data. In a real problem, inspect the format or source metadata before choosing another decoder; success alone does not establish that the resulting text is correct.

legacy_data = b"caf\xe9"
text = legacy_data.decode("latin-1")
utf8_data = text.encode("utf-8")
print(text)
print(utf8_data)
print(utf8_data.decode("latin-1") == text)

# ✅ Expected output:
# café
# b'caf\xc3\xa9'
# False

The last decode succeeds but produces different text. Latin-1 assigns every byte a code point, so it can hide a mistaken encoding choice instead of detecting it. Preserve the original bytes while investigating corrupt text. Repeated guessing can make the original problem harder to reconstruct.

damaged = b"A\xffB"
print(damaged.decode("utf-8", errors="ignore"))
print(ascii(damaged.decode("utf-8", errors="replace")))

# ✅ Expected output:
# AB
# 'A\ufffdB'

ignore drops undecodable data. replace marks it with a replacement character, but does not preserve the original byte in the resulting text. These are explicit loss policies, not repairs. Use strict decoding when a malformed record should be rejected and investigated.

A byte chunk can end halfway through a character

data = "café".encode("utf-8")
print(data[:4].decode("utf-8"))

# ✅ Expected error:
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 3: unexpected end of data

The whole payload is valid; this selected chunk is incomplete. When processing a sequence of byte chunks, an incremental decoder carries unfinished sequences into the next call. The final call must say that the stream ended so a genuinely truncated final character is reported.

import codecs

data = "café".encode("utf-8")
decoder = codecs.getincrementaldecoder("utf-8")()
first = decoder.decode(data[:4])
last = decoder.decode(data[4:], final=True)
print(repr(first))
print(repr(last))
print(first + last)

# ✅ Expected output:
# 'caf'
# 'é'
# café

This example keeps every byte in memory so its behavior is reproducible. Real file buffering, text-mode reads and stream positions are covered in Working with Files, Streams & Large Datasets. A text-mode reader already handles its encoding boundary; do not manually decode strings it has returned.

Visually identical text can have different code points

import unicodedata

composed = "\u00e9"
decomposed = "e\u0301"
print(len(composed), len(decomposed))
print(composed == decomposed)
print(unicodedata.normalize("NFC", composed) == unicodedata.normalize("NFC", decomposed))

# ✅ Expected output:
# 1 2
# False
# True

Length counts code points, not necessarily visible characters. NFC normalization makes these canonically equivalent spellings comparable. It does not remove all visual lookalikes, implement case-insensitive matching, or define every application’s identity rules. Decide whether normalization is appropriate before storing or comparing text, and preserve original spelling when it matters.

Complete an explicit text-to-byte round trip

Encode a name as UTF-8, measure its byte length, then decode it back. The two missing method names should make the direction of each conversion visible. Avoid str(data), which would describe the bytes rather than decode them.

name = "Zoë"
data = name.___("utf-8")
restored = data.___("utf-8")
print(len(name), len(data))
print(restored)
print(restored == name)

# 👉 replace ___ with encode
# 👉 replace ___ with decode

# ✅ Expected output:
# 3 4
# Zoë
# True

Accept valid text records and report invalid ones

Process three byte records: UTF-8 for café, bytes for tea, and the single byte 255. Decode each strictly as UTF-8. Keep valid text in arrival order and print the one-based record number for each invalid record. Do not silently drop the bad byte or guess a different encoding. This brief distinguishes rejecting a record from discarding bytes inside it.

records = ["café".encode("utf-8"), b"tea", bytes([255])]
accepted = []
for number, data in enumerate(records, start=1):
    try:
        text = data.decode("utf-8")
    except UnicodeDecodeError:
        print("Rejected record:", number)
    else:
        accepted.append(text)
print("Accepted:", accepted)

# ✅ Expected output:
# Rejected record: 3
# Accepted: ['café', 'tea']

Only a decoding failure is handled here. Other programming errors should still be visible. A production format would also define validation after decoding, such as allowed fields; valid UTF-8 alone does not prove a record is meaningful.

Practice quiz

What does this program print? text = "café" print(len(text), len(text.encode("utf-8")))

  • 4 4
  • 5 5
  • 4 5
  • 5 4

Answer: 4 5. The final code point takes two UTF-8 bytes.

What does this program print? data = b"AB" print(data[0], data[:1])

  • 65 b'A'
  • b'A' b'A'
  • 65 65
  • 'A' 'A'

Answer: 65 b'A'. Indexing bytes gives an integer; slicing retains the bytes type.

What does this program print? print(str(b"tea")) print(b"tea".decode("utf-8"))

  • tea tea
  • b'tea' b'tea'
  • tea b'tea'
  • b'tea' tea

Answer: b'tea' tea. str displays a representation while decode interprets the encoded text.

What does this program print? data = bytes([65, 255, 66]) print(data.decode("utf-8", errors="ignore"))

  • A?B
  • AB
  • AÿB
  • UnicodeDecodeError

Answer: AB. Ignoring malformed bytes removes them from the text.

What does this program print? import unicodedata a = "é" b = "e" + chr(0x301) print(a == b) print(unicodedata.normalize("NFC", a) == unicodedata.normalize("NFC", b))

  • True True
  • False False
  • False True
  • True False

Answer: False True. The original code point sequences differ but these NFC forms agree.

Why can decoding each arbitrary byte chunk independently fail on valid UTF-8 data?

  • A chunk can end inside a multibyte sequence
  • UTF-8 does not support accents
  • bytes must always be converted using str
  • A text character always takes four bytes

Answer: A chunk can end inside a multibyte sequence. Use an incremental decoder or a text stream that manages this boundary.

What does this program print? import codecs data = "é".encode("utf-8") d = codecs.getincrementaldecoder("utf-8")() print(repr(d.decode(data[:1]))) print(repr(d.decode(data[1:], final=True)))

  • 'é' ''
  • '' 'é'
  • '' ''
  • 'Ã' '©'

Answer: '' 'é'. The first byte is buffered until the second completes the code point.

A decoder accepts a payload without error. What has that established?

  • The text is authentic
  • The chosen encoding was certainly correct
  • The record has valid fields
  • Those bytes were accepted by that decoder

Answer: Those bytes were accepted by that decoder. Successful decoding does not prove the producer used that encoding or that the record is meaningful.

Continue this course

Related lessons