HTTP Requests & APIs (requests)
Published by Brayan K
An API lets your program talk to other programs over the internet — weather data, currency rates, your own backend. The requests library makes calling them feel as easy as calling a function.
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
What an HTTP request is
Every time you load a web page, your browser sends an HTTP request to a server and gets a response back: a status code saying how it went, some headers, and a body. An API answers the same way, usually with JSON in the body instead of a web page. Each request names a method:
- GET fetches data: "give me the weather for London".
- POST sends data to create something: "add this book".
- PUT and PATCH change something that exists.
- DELETE removes it.
requests isn't part of Python. Install it with the line below; Virtual Environments & Dependency Management Best Practices explains where installed packages go.
pip install requestsIt also needs a network connection, which this site's editor can't give it, so the requests code in this lesson is shown without a Run button and without output. The mistakes people make around those calls, with the URL, the status and the JSON, run below with Python's own tools, and their output is checked.
A GET request
import requests
resp = requests.get("https://api.github.com/users/octocat", timeout=10)
print(resp.status_code)
print(resp.headers["Content-Type"])
user = resp.json()
print(user["login"], user["public_repos"])resp.status_code is the number that says how it went, resp.headers a dictionary of the headers, and resp.text the body as text, which is what a web page gives you; Web Scraping with BeautifulSoup reads it. resp.json() runs json.loads() on that text, as Working with JSON showed, and hands back the lists and dictionaries inside. timeout=10 sets connection and read-inactivity limits, not a ten-second limit for the whole download; the timeout section below explains more.
Pass parameters instead of pasting them into the URL
Searches and filters go after a ? in the URL, as name=value pairs joined by &. Text with a space or an & in it has to be escaped first, or the server reads it wrong. urlencode(), from Python's urllib.parse, does the escaping:
from urllib.parse import urlencode, quote
params = {"q": "fish & chips", "page": 2}
print(urlencode(params))
print(quote("fish & chips"))
print("https://api.example.com/search?" + urlencode(params))
# ✅ Expected output:
# q=fish+%26+chips&page=2
# fish%20%26%20chips
# https://api.example.com/search?q=fish+%26+chips&page=2+ stands for a space and %26 for &. Typed straight into the URL, the & splits the search in two. parse_qs() reads a query the way a server does:
from urllib.parse import urlsplit, parse_qs
typed = "https://api.example.com/search?q=fish & chips&page=2"
print(parse_qs(urlsplit(typed).query))
built = "https://api.example.com/search?q=fish+%26+chips&page=2"
print(parse_qs(urlsplit(built).query))
# ✅ Expected output:
# {'q': ['fish '], 'page': ['2']}
# {'q': ['fish & chips'], 'page': ['2']}The typed URL searched for "fish ", and the rest became a nameless parameter that the server drops. requests does the escaping for you when you pass a dictionary: params={"q": "fish & chips", "page": 2} sends ?q=fish+%26+chips&page=2, the same text as urlencode().
import requests
resp = requests.get(
"https://api.example.com/search",
params={"q": "fish & chips", "page": 2},
headers={"Accept": "application/json"},
timeout=10,
)
print(resp.url)headers= adds headers the same way. requests sends a User-Agent header naming itself on every call; put your own in headers when an API asks you to say which program is calling.
Check the status before using the body
A request that fails still comes back with a response, often with a JSON body describing the error. To try this here, fake_get() stands in for requests.get(): it looks the URL up in a dictionary and answers with a status and a body. SimpleNamespace, from Python's types module, makes an object with whatever names you give it, here status_code, text and json, which is all these examples need from a response:
import json
from types import SimpleNamespace
SERVER = {
"https://api.example.com/users/ada": (200, '{"login": "ada", "repos": 8}'),
}
def fake_get(url, timeout=None):
status, text = SERVER.get(url, (404, '{"message": "Not Found"}'))
return SimpleNamespace(status_code=status, text=text, json=lambda: json.loads(text))
for name in ["ada", "adaa"]:
resp = fake_get("https://api.example.com/users/" + name, timeout=10)
if resp.status_code == 200:
print(name, "->", resp.json()["repos"], "repos")
else:
print(name, "->", resp.status_code, resp.json()["message"])
# ✅ Expected output:
# ada -> 8 repos
# adaa -> 404 Not FoundRead the body without checking, and the error turns up far from its cause:
import json
from types import SimpleNamespace
SERVER = {
"https://api.example.com/users/ada": (200, '{"login": "ada", "repos": 8}'),
}
def fake_get(url, timeout=None):
status, text = SERVER.get(url, (404, '{"message": "Not Found"}'))
return SimpleNamespace(status_code=status, text=text, json=lambda: json.loads(text))
resp = fake_get("https://api.example.com/users/adaa", timeout=10)
user = resp.json()
print(user["login"])
# ✅ Expected error:
# KeyError: 'login'The 404's body was valid JSON, just not a user, so the program stops with a KeyError about login rather than anything that mentions the request. Status codes come in families, named by their first digit:
| Family | Meaning | Examples |
|---|---|---|
| 2xx | It worked | 200 OK, 201 Created |
| 3xx | Redirect or reuse a cached response | 301 Moved Permanently, 304 Not Modified |
| 4xx | The request was refused | 401 Unauthorized, 404 Not Found, 429 Too Many Requests |
| 5xx | The server failed | 500 Internal Server Error, 503 Service Unavailable |
from http import HTTPStatus
for code in [200, 201, 301, 404, 429, 503]:
print(code, code // 100, HTTPStatus(code).phrase)
# ✅ Expected output:
# 200 2 OK
# 201 2 Created
# 301 3 Moved Permanently
# 404 4 Not Found
# 429 4 Too Many Requests
# 503 5 Service Unavailablecode // 100 gives the family, and HTTPStatus, from Python's http module, knows each code's name. requests has two shortcuts. resp.ok is True for any status below 400, redirects included, so it isn't the same as 200. resp.raise_for_status() does nothing for a status below 400, and stops the program with an HTTPError for any other, with a message such as 404 Client Error: Not Found for url: https://api.example.com/users/adaa:
import requests
resp = requests.get("https://api.github.com/users/octocat", timeout=10)
resp.raise_for_status()
user = resp.json()To carry on after a failure instead of stopping, catch requests.RequestException, which covers HTTPError, timeouts and connection errors; Exception Handling covers try and except.
.json() is a method, and only JSON parses
Leave off the brackets and you have the method itself, not the data:
import json
from types import SimpleNamespace
text = '{"login": "ada", "repos": 8}'
resp = SimpleNamespace(status_code=200, text=text, json=lambda: json.loads(text))
user = resp.json
print(user["login"])
# ✅ Expected error:
# TypeError: 'function' object is not subscriptableThe stand-in's json is a plain function, so the message says 'function'; with requests it says 'method' object is not subscriptable. And .json() only works on JSON. A server in trouble often answers with an HTML error page, or with nothing at all:
import json
body = "<!DOCTYPE html><html><body><h1>502 Bad Gateway</h1></body></html>"
data = json.loads(body)
# ✅ Expected error:
# json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)An empty body gives the same message. requests raises its own JSONDecodeError with the same text, so check the status first, and the Content-Type header if you're unsure, before calling .json().
Sending data: json= or data=
To send data with POST, requests can encode it in two ways. These are the two bodies:
import json
from urllib.parse import urlencode
payload = {"title": "Dune", "pages": 412}
print(json.dumps(payload))
print(urlencode(payload))
# ✅ Expected output:
# {"title": "Dune", "pages": 412}
# title=Dune&pages=412json=payload sends the first, JSON text, with a Content-Type header of application/json, which is what most APIs expect. data=payload sends the second, the encoding an HTML form uses. Pass only one: given both, requests sends the data= body and ignores json=. And data=json.dumps(payload) sends the JSON text without the header that says it's JSON. A POST that creates something usually answers 201 Created.
import requests
payload = {"title": "Dune", "pages": 412}
resp = requests.post("https://api.example.com/books", json=payload, timeout=10)
print(resp.status_code)Always set a timeout
Without timeout=, requests waits for an answer for as long as it takes: its default is no limit at all. A server that accepts the connection and never answers stops the program for good. timeout=10 makes requests give up after 10 seconds of silence with a Timeout error, a kind of RequestException. Every call in this lesson has one.
Keep API keys out of the code
Many APIs want a key that identifies you. Typed into the code, the key goes wherever the code goes, into a shared file or a public repository. Keep it in an environment variable instead, a named value the operating system hands to the program, and read it with os.environ:
import os
from unittest.mock import patch
# Use an empty environment for this demonstration; restore it on exit.
with patch.dict(os.environ, {}, clear=True):
key = os.environ.get("WEATHER_API_KEY")
print("key present:", key is not None)
if key is None:
print("Set WEATHER_API_KEY first")
# ✅ Expected output:
# key present: False
# Set WEATHER_API_KEY firstos.environ.get() gives None when the variable isn't set. These demonstrations temporarily use an empty environment and restore the original afterwards, so their results don't depend on your own keys. They report whether a key exists without printing its value. Square brackets stop the program instead:
import os
from unittest.mock import patch
with patch.dict(os.environ, {}, clear=True):
key = os.environ["WEATHER_API_KEY"]
# ✅ Expected error:
# KeyError: 'WEATHER_API_KEY'import os
import requests
key = os.environ.get("WEATHER_API_KEY")
resp = requests.get(
"https://api.example.com/weather",
params={"city": "London", "key": key},
timeout=10,
)More than one page, and trying again
APIs hand back long lists a page at a time, with a way to ask for the next page. Here the pages are a dictionary of JSON texts, and each one says which page comes next, or null on the last:
import json
PAGES = {
1: '{"results": ["Dune", "Emma"], "next": 2}',
2: '{"results": ["Ulysses", "Beloved"], "next": 3}',
3: '{"results": ["Ivanhoe"], "next": null}',
}
def get_page(number):
return json.loads(PAGES[number])
titles = []
page = 1
while page is not None:
data = get_page(page)
titles.extend(data["results"])
print("page", page, "gave", len(data["results"]))
page = data["next"]
print(titles)
# ✅ Expected output:
# page 1 gave 2
# page 2 gave 2
# page 3 gave 1
# ['Dune', 'Emma', 'Ulysses', 'Beloved', 'Ivanhoe']A busy server may answer 503, or 429 when you call too often. The polite fix is to wait and try again, waiting longer each time. Real code calls time.sleep(delay); this example prints the wait instead, so it runs at once:
answers = [503, 503, 200]
delay = 1
for attempt in range(1, 5):
status = answers[attempt - 1]
if status == 200:
print("try", attempt, "got 200")
break
print("try", attempt, "got", status, "- wait", delay, "s")
delay = delay * 2
# ✅ Expected output:
# try 1 got 503 - wait 1 s
# try 2 got 503 - wait 2 s
# try 3 got 200REST API Clients & External Service Integrations builds both into a reusable client.
Practice: a search that finds something
Fill the three blanks with the values in the hint lines, in order. The stand-in only finds results for a correctly escaped search, so a hand-typed query gets a 400.
from types import SimpleNamespace
from urllib.parse import urlencode
def fake_get(url, timeout=None):
if url.endswith("?q=fish+%26+chips"):
return SimpleNamespace(status_code=200, json=lambda: {"count": 2})
return SimpleNamespace(status_code=400, json=lambda: {"error": "bad query"})
query = ___({"q": "fish & chips"})
url = "https://api.example.com/search?" + query
print(url)
resp = fake_get(url, timeout=10)
if resp.status_code == ___:
print("found", resp.___()["count"])
else:
print("failed:", resp.status_code)
# 👉 replace ___ with urlencode
# 👉 replace ___ with 200
# 👉 replace ___ with json
# ✅ Expected output:
# https://api.example.com/search?q=fish+%26+chips
# found 2On your own: collect every page
Write a fake_get(url, params=None, timeout=None) that answers like a paged book API: page 1 gives Dune and Emma, page 2 gives Ulysses, and each page's JSON names the next page. Page 3 is down and answers 503. Collect every title, stop with a message such as page 3 failed with 503 at the first page that isn't 200, then print the titles. Then compare with this solution:
import json
from types import SimpleNamespace
PAGES = {
1: (200, '{"results": ["Dune", "Emma"], "next": 2}'),
2: (200, '{"results": ["Ulysses"], "next": 3}'),
3: (503, '{"error": "busy"}'),
}
def fake_get(url, params=None, timeout=None):
status, text = PAGES[params["page"]]
return SimpleNamespace(status_code=status, json=lambda: json.loads(text))
titles = []
page = 1
while page is not None:
resp = fake_get("https://api.example.com/books", params={"page": page}, timeout=10)
if resp.status_code != 200:
print("page", page, "failed with", resp.status_code)
break
data = resp.json()
titles.extend(data["results"])
page = data["next"]
print(titles)
# ✅ Expected output:
# page 3 failed with 503
# ['Dune', 'Emma', 'Ulysses']The status check comes before resp.json(), so the 503's body is never read as a page, and the titles collected before the failure are kept.
Practice quiz
What does urlencode({"q": "fish & chips", "page": 2}) give?
- q=fish & chips&page=2
- q=fish%20&%20chips&page=2
- q=fish+%26+chips&page=2
- q=fish+chips&page=2
Answer: q=fish+%26+chips&page=2. A space becomes + and & becomes %26, so the & that separates the pairs can't be confused with the one in the search.
What does parse_qs(urlsplit("https://api.example.com/search?q=fish & chips&page=2").query) give?
- {'q': ['fish & chips'], 'page': ['2']}
- {'q': ['fish '], 'page': ['2']}
- {'q': ['fish', 'chips'], 'page': ['2']}
- An error
Answer: {'q': ['fish '], 'page': ['2']}. The typed & splits the search in two, so the server sees q=fish and a nameless " chips".
A 404 response's body is {"message": "Not Found"}. What does resp.json()["login"] do?
- It stops with KeyError: 'login'
- It gives None
- It gives 'Not Found'
- It stops with an HTTPError
Answer: It stops with KeyError: 'login'. The error body is valid JSON, just not a user. Check resp.status_code first.
user = resp.json, then user["login"]. What happens?
- It gives the login
- It stops with a TypeError, because the method itself isn't subscriptable
- It gives None
- It stops with KeyError: 'login'
Answer: It stops with a TypeError, because the method itself isn't subscriptable. Without brackets, resp.json is the method, not the data. Call it: resp.json().
What does json.loads("<!DOCTYPE html><html><body>Error</body></html>") do?
- It gives the HTML as a string
- It gives {}
- It gives None
- It stops with JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Answer: It stops with JSONDecodeError: Expecting value: line 1 column 1 (char 0). An HTML page isn't JSON, and the very first character, <, can't start a JSON value.
Waiting 1 second after the first failure and doubling the wait each time, how long in all does a request wait if it fails three times and then works?
- 3 seconds
- 8 seconds
- 7 seconds
- 4 seconds
Answer: 7 seconds. 1 + 2 + 4 seconds, one wait after each of the three failures.
With WEATHER_API_KEY not set, what does os.environ.get("WEATHER_API_KEY", "missing") give?
- missing
- None
- It stops with KeyError
- An empty string
Answer: missing. get() gives the default you pass when the variable isn't there. os.environ["WEATHER_API_KEY"] would stop with a KeyError.
What does print(429 // 100, 503 // 100) show?
- 429 503
- 4.29 5.03
- 2 3
- 4 5
Answer: 4 5. Floor division by 100 gives a status code's family: 4xx means the request was refused, 5xx that the server failed.
Continue this course
- Previous: Working with JSON
- Next: Reading CSV & Excel Files — Read and write spreadsheet data with csv and openpyxl/pandas
- Quick reference: Python cheat sheet
Frequently asked questions
Why can't this site's editor run requests?
The editor runs Python inside your web page, where a program can't open the network connections requests uses, and requests isn't installed there. Run requests code on your own computer after pip install requests, or in an online notebook such as Google Colab.
Should I use requests or urllib.request?
urllib.request comes with Python, so it needs no install. requests is a separate package, built on another one called urllib3, and does more of the work for you: params=, json=, timeouts and sessions that reuse one connection for many calls.
Can a program wait for several requests at once?
Yes, with asynchronous code, which starts many requests and handles each answer as it arrives instead of waiting for one before sending the next. AsyncIO: Event Loop, Tasks & Futures shows how.