Transformer
A transformer is a neural network architecture that processes sequences (like sentences) using a mechanism called attention, which lets the model weigh how relevant each word is to every other word.
Data & AI
Introduced in 2017, the transformer replaced older sequential models and now underpins nearly all modern language AI, including large language models. Its key innovation, self-attention, lets it look at an entire sequence at once and capture long-range relationships between words.
Because transformers process all positions in parallel rather than one word at a time, they train far faster on modern hardware. The same architecture has since been adapted for images, audio, and more.
Examples
Python
# Self-attention idea: each word attends to others
sentence = ["The", "cat", "sat"]
# 'sat' looks back at 'cat' to know who sat
attention = {"sat": {"cat": 0.7, "The": 0.2, "sat": 0.1}}
print(attention["sat"])Output:
{'cat': 0.7, 'The': 0.2, 'sat': 0.1}