Language Integration: C & Rust
Published by Brayan K
Master Python + C/Rust integration using ctypes, cffi, Cython, and PyO3 for maximum performance while maintaining Python's productivity
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
Choose the boundary before choosing a binding tool
Python can exchange messages with another process, call a C-compatible shared library, or import a compiled extension module. These are different interfaces. Loading a library does not compile its source, and giving Python a pointer does not tell it who owns the memory.
| Boundary | What crosses it | Main tradeoff |
|---|---|---|
| Separate executable | Bytes or a documented text protocol | Process isolation, but startup and serialization costs |
| C ABI through ctypes/cffi | C-compatible values, pointers and lengths | Direct calls, but the exact binary contract is your responsibility |
| Cython or Rust/PyO3 extension | Values exposed through a Python module | A convenient Python API, with a compiled build and distribution step |
A foreign-function interface (FFI) calls code compiled outside Python. An application binary interface (ABI) specifies how arguments, return values and memory are represented at that boundary. Matching function names alone is insufficient: calling convention, integer width and ownership rules also have to match.
Measure a real bottleneck before moving work out of Python. Profiling & Optimising Python Performance owns that measurement; no fixed “95/5” split or promised speed multiplier applies to every application.
A text character is not a byte
Agree on an encoding before passing text to another language. The count of Python characters can differ from the length of the encoded buffer. This matters when a native API asks for a byte length or a worker reads a framed message.
text = "café"
payload = text.encode("utf-8")
print("characters:", len(text))
print("bytes:", len(payload))
print(payload.decode("utf-8"))
# ✅ Expected output:
# characters: 4
# bytes: 5
# caféChoosing an incompatible decoder is an error, not a reason to silently discard bytes:
payload = "café".encode("utf-8")
print(payload.decode("ascii"))
# ✅ Expected error:
# UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)Use the encoding specified by the producer, here UTF-8. A replacement-error handler can be appropriate for displaying damaged text, but it would change data in a machine protocol. File/text stream details are covered in Working with Files, Streams & Large Datasets.
Call a real C function with its declared signature
The next example calls strlen in the operating system's C runtime. This is a real native call, not a Python imitation of a C library. Run it with local CPython on Windows, Linux or macOS; browser Python cannot load the computer's native libraries. The remaining ctypes examples also need local CPython.
The C declaration is size_t strlen(const char *text). Set argtypes to a C string pointer and restype to c_size_t before calling it. Leaving a native result at ctypes' default C-int type can misread a wider result or pointer. Consult the actual library's header instead of guessing from a Python value.
import ctypes
from ctypes.util import find_library
import os
library_name = "ucrtbase" if os.name == "nt" else find_library("c")
if library_name is None:
raise RuntimeError("a system C runtime is required")
library = ctypes.CDLL(library_name)
strlen = library.strlen
strlen.argtypes = [ctypes.c_char_p]
strlen.restype = ctypes.c_size_t
payload = "café".encode("utf-8")
print("native byte count:", strlen(payload))
with_zero = b"tea" + bytes([0]) + b"milk"
print("Python byte count:", len(with_zero))
print("C string count:", strlen(with_zero))
# ✅ Expected output:
# native byte count: 5
# Python byte count: 8
# C string count: 3A C string ends at its first zero byte. It cannot represent an arbitrary binary payload merely by accepting a pointer. A binary API normally takes a pointer and an explicit length. Never ask strlen to inspect an arbitrary memory address; the bytes object here supplies a valid terminated buffer for the duration of the call.
The system runtime selection above is specific to the supported desktop operating systems. For your own library, load a known trusted build using its documented calling convention. A missing library, missing exported symbol or wrong CPU architecture is an installation/ABI problem; changing the function's Python spelling will not repair it.
Validate ranges before narrowing an integer
Python integers can grow beyond a native integer's fixed width. Converting to a signed 32-bit ctypes value can wrap instead of raising an error. The following example makes the narrowing visible without depending on how wide a platform's plain C int happens to be.
from ctypes import c_int32
value = 2 ** 31
print("Python:", value)
print("narrowed:", c_int32(value).value)
# ✅ Expected output:
# Python: 2147483648
# narrowed: -2147483648If the contract requires a signed 32-bit integer, reject out-of-range values before converting them. Do not mistake the wrapped value for a result computed by your C or Rust algorithm.
def signed_32(value):
if type(value) is not int or not -(2 ** 31) <= value < 2 ** 31:
raise ValueError("value must fit a signed 32-bit integer")
return value
signed_32(2 ** 31)
# ✅ Expected error:
# ValueError: value must fit a signed 32-bit integerThe validation itself is plain Python and runs in the browser. Exact type checking also rejects a boolean when the contract requires a numeric argument. Python type annotations alone would not enforce this rule at runtime.
Keep the buffer owner alive while a pointer is used
A pointer describes where data lives; it does not generally describe the allocation's length or who frees it. A wrapper needs all three facts. The ctypes array below owns three doubles, while the pointer gives indexed access to them. Keep the owner reachable until the native operation has finished, and never index past the documented count.
from ctypes import c_double, POINTER, cast
owner = (c_double * 3)(1.5, 2.0, 3.5)
pointer = cast(owner, POINTER(c_double))
count = len(owner)
print("elements:", count)
print("sum:", sum(pointer[index] for index in range(count)))
pointer[1] = 4.0
print("owner values:", list(owner))
# ✅ Expected output:
# elements: 3
# sum: 7.0
# owner values: [1.5, 4.0, 3.5]This snippet demonstrates shared buffer access; its summation is still a Python loop and makes no native-speed claim. For a real foreign allocation, use that library's documented release function. Do not mix allocators or free borrowed storage. If a native function retains the pointer after returning, keeping the owner alive only until that return is insufficient.
“Zero copy” is a condition to establish, not a default speed setting. Dtype, byte order, contiguity, alignment, mutability and lifetime must match the native interface. Sometimes a deliberate copy into a known layout is the correct boundary. Test empty arrays and rejected inputs as well as the happy path.
A separate worker needs a protocol, not shared pointers
For an existing command-line tool written in C, Rust or another language, a process boundary can avoid in-process ABI handling. Define its input, output, encoding and failure status. This example starts a Python worker solely to test that protocol without requiring a compiler; it does not claim to execute Rust or C.
import json
import subprocess
import sys
worker = '''import json, sys
request = json.load(sys.stdin)
values = request["values"]
if not isinstance(values, list) or any(type(value) is not int for value in values):
raise ValueError("values must be integers")
json.dump({"total": sum(values)}, sys.stdout)
'''
completed = subprocess.run(
[sys.executable, "-c", worker],
input=json.dumps({"values": [4, 6, 8]}),
capture_output=True, text=True, encoding="utf-8", check=True, timeout=5,
)
reply = json.loads(completed.stdout)
if not isinstance(reply, dict) or type(reply.get("total")) is not int:
raise ValueError("worker total must be an integer")
print("total:", reply["total"])
print("successful exit:", completed.returncode == 0)
# ✅ Expected output:
# total: 18
# successful exit: TrueStandard output contains only the machine-readable response. Debug logs belong on standard error; mixing them into stdout corrupts the JSON protocol. Nonzero exit and malformed output are distinct failures. Running Commands with subprocess owns process invocation and timeouts; Working with JSON owns serialization.
A separate process can contain a worker crash, but it still needs request-size limits, a deadline and a defined restart policy. Reusing a long-lived worker also requires message framing: decide where one response ends before sending another request.
When to build an extension instead
If you control the native code and need a normal Python import, choose an extension tool and a supported build matrix. Cython translates its source to C/C++ before compilation. PyO3 exposes Rust functions/classes through Python bindings; maturin helps build and package those bindings. cffi offers C declarations and either ABI-level calls or compiled wrappers. These tools do not all use the same build steps.
A Rust/PyO3 project needs a Rust toolchain, the intended Python environment and a compatible build configuration. Start with one typed function, define how its errors become Python exceptions, build/install it into that environment, then test imports and rejected inputs. Publish wheels matching the platforms and interpreter ABI you actually tested. A Python-callable extension module is different from an arbitrary DLL with a C function.
Rust reduces many memory hazards in safe Rust code, but an FFI boundary and unsafe dependencies can still be wrong. Likewise, importing native code does not automatically provide parallel execution. On a GIL-enabled interpreter, whether a long native operation permits other Python threads to run depends on its binding; free-threaded builds have their own compatibility requirements. Do not infer any of this from the language name.
This lesson's checked examples validate text, ranges, buffer access, an actual C-runtime call and a process protocol. They do not compile a Cython or PyO3 package. Follow the tool's complete build guide when taking that additional step: PyO3 installation and builds, PyO3 parallelism and cffi's interface modes. Python package distribution is covered in Packaging & Publishing Python Libraries to PyPI.
Complete the byte-count boundary
A foreign worker expects UTF-8 bytes and their byte count. Fill the encoding and count operation. Then decode its echoed payload using the same encoding.
text = "naïve"
payload = text.encode(___)
byte_count = len(___)
print(byte_count)
print(payload.decode("utf-8"))
# 👉 replace ___ with "utf-8"
# 👉 replace ___ with payload
# ✅ Expected output:
# 6
# naïvePrepare a label for a C-string interface
Write encode_label(text) for an API that accepts UTF-8 C strings. Reject text containing a zero character with ValueError("label contains a zero character"). Otherwise return its encoded bytes and byte count. Print the count and decoded text for café, then demonstrate rejection of "tea" + chr(0) + "milk". This solution prepares a valid argument without loading a native library.
def encode_label(text):
if chr(0) in text:
raise ValueError("label contains a zero character")
payload = text.encode("utf-8")
return payload, len(payload)
payload, count = encode_label("café")
print(count, payload.decode("utf-8"))
try:
encode_label("tea" + chr(0) + "milk")
except ValueError as error:
print(type(error).__name__ + ":", error)
# ✅ Expected output:
# 5 café
# ValueError: label contains a zero characterIn the native-call example, ctypes supplies the C-string argument representation from these bytes. A different API that accepts arbitrary binary data should use its documented pointer-and-length contract instead of this terminator restriction. The ctypes reference documents the conversion and ownership APIs.
Practice quiz
What does print(len("naïve"), len("naïve".encode("utf-8"))) print?
- 5 5
- 5 6
- 6 6
- 6 5
Answer: 5 6. The string contains five Unicode code points. In this word, UTF-8 uses two bytes for ï and one for each other character, so character and byte counts differ.
A native function returns size_t, but a ctypes wrapper leaves the default int return type. What repairs the declaration before the first call?
- Convert the returned Python value to int after calling
- Infer the return type from the Python argument values
- Store the result in a variable named for the native type
- Set argtypes and restype from the native function header
Answer: Set argtypes and restype from the native function header. ctypes needs the native signature. A wrong return type can truncate the result before Python sees it; converting or renaming that result afterward cannot recover lost bits.
In local CPython, after import ctypes, what does print(ctypes.c_int32(2**31).value) print?
- -2147483648
- 2147483648
- 0
- OverflowError
Answer: -2147483648. This conversion narrows to a signed 32-bit value instead of validating the application's range. Check type and bounds before crossing the boundary when wrapping is not intended.
A native API borrows a buffer pointer and keeps using it after the Python wrapper returns. When may the wrapper release its last reference to the buffer owner?
- As soon as the wrapper has returned to its caller
- After the pointer has been cast to the required type
- After the native API confirms it has finished using it
- As soon as the native function first returns a status
Answer: After the native API confirms it has finished using it. The allocation must remain alive for the full native-use lifetime. A cast or the end of the Python call does not end the native library's ownership contract.
A worker protocol requires exactly one JSON document on stdout. Adding a debug print makes decoding fail. Where should that diagnostic go?
- Before the JSON document, with a recognizable prefix
- To stderr, leaving stdout for the protocol document
- After the JSON document, for the caller to skip later
- Into stdout after merging stderr into the same stream
Answer: To stderr, leaving stdout for the protocol document. Keep the machine-readable protocol on stdout and diagnostics on stderr. Parsing only the last line would change the stated protocol instead of honoring it.
Two systems share a buffer address, but disagree about the element layout. What is still needed for a valid zero-copy interface?
- Matching layout and an agreed ownership and lifetime contract
- Matching variable names on both sides of the language boundary
- Matching printed values from one small example of the buffer
- Matching pointer text copied into both programs' configuration
Answer: Matching layout and an agreed ownership and lifetime contract. Sharing an address does not establish compatible types, byte order, alignment or lifetime. Those contracts must agree before both sides can safely interpret the same storage.
A native kernel is faster in isolation, but the Python-to-native workflow is slower. Which measurement could explain the difference?
- The kernel alone with input conversion excluded
- The number of source lines in each implementation
- The native language's usual reputation for speed
- The complete call including conversions and data transfer
Answer: The complete call including conversions and data transfer. Conversion, copying and call overhead can dominate the kernel work. Measure the complete boundary operation on representative inputs; this lesson makes no speed claim.
A developer passes C source text to ctypes and expects compilation. What does the ctypes loading step actually need?
- The source text renamed with a shared-library suffix
- The same C function rewritten with a Python-style name
- A compiled shared library matching the expected interface
- The compiler's console output saved as a Python string
Answer: A compiled shared library matching the expected interface. ctypes loads existing shared libraries and calls their exported functions. Compiling a C extension or building another language's package is a separate toolchain step.
Continue this course
- Previous: Automation & Scripting for DevOps and System Tasks
- Next: Architecture Patterns for Python Apps (MVC, Clean Architecture) — Structure large Python applications with proven architectural patterns
- Quick reference: Python cheat sheet