Checkpoint: A Real-World Script
Published by Brayan K
You've learned regex, datetime, JSON, HTTP requests, CSV/Excel, web scraping, itertools and pathlib — let's build a real script. No new concepts here: this checkpoint is about wiring your skills together into one program that fetches data, cleans it, analyses it, and saves a report.
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
The job: a report on one user's repositories
The script asks GitHub's API for a user's repositories, keeps the ones the user made, works out how long ago each was last changed, totals the stars per language, and saves the list as a CSV file. Each section below is one place where two of the last eight lessons meet and catch people out; the brief at the end puts them together.
On a computer with the requests package, the fetch is the call HTTP Requests & APIs teaches:
import requests
repos = []
page = 1
while True:
resp = requests.get("https://api.github.com/users/octocat/repos",
params={"per_page": 100, "page": page}, timeout=10)
resp.raise_for_status()
batch = resp.json()
if not batch:
break
repos.extend(batch)
page += 1GitHub sends at most 100 repositories per request, so the loop asks for one page after another, as that lesson's paging example does, until a page comes back empty.
This site's editor and the check this site's build makes can't reach the network, so the examples here use that lesson's stand-in: fake_get() hands back an answer with a status_code and a json() method, built from JSON text in the program. To carry on after a failed request instead of stopping, catch requests.RequestException, which Exception Handling covers.
import json
from types import SimpleNamespace
BODY = """[
{"name": "web-scraper", "language": "Python", "stargazers_count": 130,
"pushed_at": "2024-06-01T08:00:00Z", "fork": false},
{"name": "notes", "language": null, "stargazers_count": 5,
"pushed_at": "2024-02-15T08:00:00Z", "fork": false}
]"""
def fake_get(url, params=None, timeout=None):
return SimpleNamespace(status_code=200, text=BODY, json=lambda: json.loads(BODY))
resp = fake_get("https://api.github.com/users/octocat/repos", params={"per_page": 100}, timeout=10)
print(resp.status_code)
repos = resp.json()
print(len(repos))
print(repos[1])
# ✅ Expected output:
# 200
# 2
# {'name': 'notes', 'language': None, 'stargazers_count': 5, 'pushed_at': '2024-02-15T08:00:00Z', 'fork': False}The JSON null arrived as None and false as False, as Working with JSON showed. That None is where the first trap is.
A JSON null meets an f-string width
A repository with no code in it has no language, so the API sends null. A width such as :<8 from f-strings & String Formatting works on text and numbers, but not on None. or gives a default, as in Operators and Expressions:
import json
repo = json.loads('{"name": "notes", "language": null, "fork": false}')
print(f"{repo['name']:<8}|{repo['language'] or 'none':<8}|")
print(f"{repo['name']:<8}|{repo['language']:<8}|")
# ✅ Expected output:
# notes |none |
# ✅ Expected error:
# TypeError: unsupported format string passed to NoneType.__format__or gives its right side whenever the left side counts as false, and that includes a real 0 or an empty string. For a language that's what you want; for a count of stars it isn't:
stars = 0
language = None
print(language or "none", stars or "no stars", stars)
# ✅ Expected output:
# none no stars 0The T, the Z, and what .days counts
GitHub writes times such as 2024-05-10T08:00:00Z. As Dates & Times (datetime) showed, letters that aren't codes match themselves, so the format spells out the T and the Z. Taking one datetime from another gives a timedelta, and its .days counts whole days only:
from datetime import datetime
TODAY = datetime(2024, 6, 17)
pushed = datetime.strptime("2024-05-10T08:00:00Z", "%Y-%m-%dT%H:%M:%SZ")
print(pushed)
print(TODAY - pushed)
print((TODAY - pushed).days)
print((TODAY.date() - pushed.date()).days)
# ✅ Expected output:
# 2024-05-10 08:00:00
# 37 days, 16:00:00
# 37
# 38The push was at 08:00, so on 17 June only 37 full days have passed, though the calendar dates are 38 apart. Both answers are right; decide which one the report means and use it everywhere. Leave the Z out of the format and strptime() refuses, because every character has to be accounted for:
from datetime import datetime
datetime.strptime("2024-05-10T08:00:00Z", "%Y-%m-%dT%H:%M:%S")
# ✅ Expected error:
# ValueError: unconverted data remains: ZAnd a date can't be taken from a datetime, or the other way round:
from datetime import date, datetime
pushed = datetime.strptime("2024-05-10T08:00:00Z", "%Y-%m-%dT%H:%M:%SZ")
print((date(2024, 6, 17) - pushed).days)
# ✅ Expected error:
# TypeError: unsupported operand type(s) for -: 'datetime.date' and 'datetime.datetime'datetime.fromisoformat() reads the same text without a format, but only from Python 3.11, and it gives a datetime that knows its time zone. Taking that from a TODAY that doesn't know one is a TypeError, so the report keeps strptime(). Time Zones with zoneinfo covers datetimes that carry a zone.
match() checks the start, fullmatch() the whole name
The report keeps names made of lower-case letters, digits and hyphens that start with a letter. Regular Expressions (re) showed that match() only needs the start of the text to fit:
import re
NAME = re.compile(r"[a-z][a-z0-9-]*")
for name in ["data-pipeline", "2fast2furious", "old_site", "web-scraper"]:
print(f"{name:<14}", bool(NAME.match(name)), bool(NAME.fullmatch(name)))
# ✅ Expected output:
# data-pipeline True True
# 2fast2furious False False
# old_site True False
# web-scraper True Truematch() accepts old_site because old fits; fullmatch() turns it down, because the underscore isn't allowed. A name starting with a digit fails both.
groupby() needs the same sort first
To total the stars per language, itertools has groupby(). It only joins repositories that sit next to each other, so the list is sorted by the same key first. The key is used twice, so it's a def, as Lambda Functions advises:
from itertools import groupby
def language_of(repo):
return repo["language"]
kept = [
{"name": "data-pipeline", "language": "Python", "stars": 42},
{"name": "dotfiles", "language": "Shell", "stars": 3},
{"name": "web-scraper", "language": "Python", "stars": 130},
]
for language, group in groupby(kept, key=language_of):
print(language, [repo["name"] for repo in group])
print("---")
for language, group in groupby(sorted(kept, key=language_of), key=language_of):
print(language, sum(repo["stars"] for repo in group))
# ✅ Expected output:
# Python ['data-pipeline']
# Shell ['dotfiles']
# Python ['web-scraper']
# ---
# Python 172
# Shell 3Without the sort, Python turns up twice. Once a missing language has become "none", it sorts after "Python" and "Shell", because capital letters come before small ones.
DictWriter refuses keys it wasn't given
The report is built as CSV text in memory, as Reading CSV & Excel Files showed with io.StringIO(). The API's dictionaries have more keys than the report's columns, and DictWriter won't quietly leave one out:
import csv
import io
repo = {"name": "notes", "stars": 5, "fork": False}
buffer = io.StringIO()
writer = csv.DictWriter(buffer, fieldnames=["name", "stars"], lineterminator="\n")
writer.writeheader()
writer.writerow(repo)
# ✅ Expected error:
# ValueError: dict contains fields not in fieldnames: 'fork'Build each row from the keys the report wants:
import csv
import io
repo = {"name": "notes", "stars": 5, "fork": False}
buffer = io.StringIO()
writer = csv.DictWriter(buffer, fieldnames=["name", "stars"], lineterminator="\n")
writer.writeheader()
writer.writerow({"name": repo["name"], "stars": repo["stars"]})
print(buffer.getvalue(), end="")
# ✅ Expected output:
# name,stars
# notes,5Saving a report the next run can write again
A script like this runs again tomorrow, into folders that already exist. With parents=True and exist_ok=True, mkdir() makes what's missing and leaves the rest, and write_text() replaces the file, as Modern File Paths with pathlib showed. Running the same lines twice proves it:
from pathlib import Path
folder = Path("reports") / "2024-06-17"
folder.mkdir(parents=True, exist_ok=True)
folder.mkdir(parents=True, exist_ok=True)
report = folder / "repos.csv"
print(report.write_text("name,stars\nweb-scraper,130\n", encoding="utf-8"))
print(report.write_text("name,stars\nweb-scraper,130\n", encoding="utf-8"))
print(sorted(p.as_posix() for p in Path("reports").rglob("*.csv")))
print(report.read_text(encoding="utf-8"), end="")
# ✅ Expected output:
# 27
# 27
# ['reports/2024-06-17/repos.csv']
# name,stars
# web-scraper,130write_text() returns how many characters it wrote, 27 both times, and there is still one file. Without parents=True, the first mkdir() stops with FileNotFoundError, because reports doesn't exist yet.
Practice: stars per language
Total the stars for each language, with a missing language counted as "none", and print the totals as CSV text. The blanks are the default for a missing language, the sort that groupby() needs, and the call that takes the text out of the buffer; the hints at the end name them.
import csv
import io
from itertools import groupby
repos = [
{"name": "data-pipeline", "language": "Python", "stars": 42},
{"name": "notes", "language": None, "stars": 5},
{"name": "web-scraper", "language": "Python", "stars": 130},
{"name": "dotfiles", "language": "Shell", "stars": 3},
]
def language_of(repo):
return repo["language"] ___ "none"
buffer = io.StringIO()
writer = csv.DictWriter(buffer, fieldnames=["language", "stars"], lineterminator="\n")
writer.writeheader()
for language, group in groupby(___(repos, key=language_of), key=language_of):
writer.writerow({"language": language, "stars": sum(repo["stars"] for repo in group)})
print(buffer.___(), end="")
# 👉 replace ___ with or
# 👉 replace ___ with sorted
# 👉 replace ___ with getvalue
# ✅ Expected output:
# language,stars
# Python,172
# Shell,3
# none,5On your own: the repository report
The starter holds the API's answer for seven repositories and the stand-in. Write the rest:
- Fetch the repositories with fake_get() and print the status (HTTP Requests & APIs).
- Keep the repositories that aren't forks, whose pushed_at isn't None, and whose whole name is lower-case letters, digits and hyphens, starting with a letter. GitHub's description of this answer allows null for pushed_at, and strptime() can't read None (Working with JSON, Regular Expressions).
- For each, make a row with the name, the language or "none", the stars and the days since the last push (Working with JSON, Operators and Expressions, Dates & Times).
- Print the rows, most stars first, and then the stars per language (Lambda Functions, f-strings & String Formatting, itertools).
- Build the rows as CSV text with DictWriter (Reading CSV & Excel Files).
- Save it as reports/2024-06-17/repos.csv, then print the CSV files under reports and the report itself (Modern File Paths with pathlib).
import csv
import io
import json
import re
from datetime import datetime
from itertools import groupby
from pathlib import Path
from types import SimpleNamespace
BODY = """[
{"name": "data-pipeline", "language": "Python", "stargazers_count": 42,
"pushed_at": "2024-05-10T08:00:00Z", "fork": false},
{"name": "test-scratch", "language": "Python", "stargazers_count": 0,
"pushed_at": "2023-01-02T08:00:00Z", "fork": true},
{"name": "web-scraper", "language": "Python", "stargazers_count": 130,
"pushed_at": "2024-06-01T08:00:00Z", "fork": false},
{"name": "notes", "language": null, "stargazers_count": 5,
"pushed_at": "2024-02-15T08:00:00Z", "fork": false},
{"name": "2fast2furious", "language": "Go", "stargazers_count": 8,
"pushed_at": "2024-04-20T08:00:00Z", "fork": false},
{"name": "dotfiles", "language": "Shell", "stargazers_count": 3,
"pushed_at": "2023-11-30T21:15:00Z", "fork": false},
{"name": "new-idea", "language": null, "stargazers_count": 0,
"pushed_at": null, "fork": false}
]"""
def fake_get(url, params=None, timeout=None):
return SimpleNamespace(status_code=200, text=BODY, json=lambda: json.loads(BODY))
TODAY = datetime(2024, 6, 17)
# your code here: the six steps aboveimport csv
import io
import json
import re
from datetime import datetime
from itertools import groupby
from pathlib import Path
from types import SimpleNamespace
BODY = """[
{"name": "data-pipeline", "language": "Python", "stargazers_count": 42,
"pushed_at": "2024-05-10T08:00:00Z", "fork": false},
{"name": "test-scratch", "language": "Python", "stargazers_count": 0,
"pushed_at": "2023-01-02T08:00:00Z", "fork": true},
{"name": "web-scraper", "language": "Python", "stargazers_count": 130,
"pushed_at": "2024-06-01T08:00:00Z", "fork": false},
{"name": "notes", "language": null, "stargazers_count": 5,
"pushed_at": "2024-02-15T08:00:00Z", "fork": false},
{"name": "2fast2furious", "language": "Go", "stargazers_count": 8,
"pushed_at": "2024-04-20T08:00:00Z", "fork": false},
{"name": "dotfiles", "language": "Shell", "stargazers_count": 3,
"pushed_at": "2023-11-30T21:15:00Z", "fork": false},
{"name": "new-idea", "language": null, "stargazers_count": 0,
"pushed_at": null, "fork": false}
]"""
def fake_get(url, params=None, timeout=None):
return SimpleNamespace(status_code=200, text=BODY, json=lambda: json.loads(BODY))
TODAY = datetime(2024, 6, 17)
NAME = re.compile(r"[a-z][a-z0-9-]*")
def language_of(row):
return row["language"]
def stars_of(row):
return row["stars"]
def make_row(repo):
pushed = datetime.strptime(repo["pushed_at"], "%Y-%m-%dT%H:%M:%SZ")
return {
"name": repo["name"],
"language": repo["language"] or "none",
"stars": repo["stargazers_count"],
"days": (TODAY - pushed).days,
}
resp = fake_get("https://api.github.com/users/octocat/repos", params={"per_page": 100}, timeout=10)
print("Status:", resp.status_code)
repos = resp.json()
rows = [
make_row(repo)
for repo in repos
if not repo["fork"] and repo["pushed_at"] is not None and NAME.fullmatch(repo["name"])
]
rows.sort(key=stars_of, reverse=True)
print(f"Kept {len(rows)} of {len(repos)} repos")
for row in rows:
print(f" {row['name']:<14}{row['language']:<8}{row['stars']:>4}{row['days']:>5} days")
for language, group in groupby(sorted(rows, key=language_of), key=language_of):
print(f"{language}: {sum(stars_of(row) for row in group)} stars")
buffer = io.StringIO()
writer = csv.DictWriter(buffer, fieldnames=["name", "language", "stars", "days"], lineterminator="\n")
writer.writeheader()
writer.writerows(rows)
folder = Path("reports") / TODAY.strftime("%Y-%m-%d")
folder.mkdir(parents=True, exist_ok=True)
report = folder / "repos.csv"
report.write_text(buffer.getvalue(), encoding="utf-8")
print(sorted(p.as_posix() for p in Path("reports").rglob("*.csv")))
print(report.read_text(encoding="utf-8"), end="")
# ✅ Expected output:
# Status: 200
# Kept 4 of 7 repos
# web-scraper Python 130 15 days
# data-pipeline Python 42 37 days
# notes none 5 122 days
# dotfiles Shell 3 199 days
# Python: 172 stars
# Shell: 3 stars
# none: 5 stars
# ['reports/2024-06-17/repos.csv']
# name,language,stars,days
# web-scraper,Python,130,15
# data-pipeline,Python,42,37
# notes,none,5,122
# dotfiles,Shell,3,199
Practice quiz
TODAY = datetime(2024, 6, 17) and pushed = datetime.strptime("2024-05-10T08:00:00Z", "%Y-%m-%dT%H:%M:%SZ"). What is (TODAY - pushed).days?
- 38
- 37.6
- 37
- 36
Answer: 37. The gap is 37 days and 16 hours, and .days counts only the whole days. (Dates & Times)
What does datetime.strptime("2024-05-10T08:00:00Z", "%Y-%m-%dT%H:%M:%S") do?
- It stops with ValueError: unconverted data remains: Z
- It gives 2024-05-10 08:00:00
- It gives 2024-05-10 08:00:00+00:00
- It gives None
Answer: It stops with ValueError: unconverted data remains: Z. The format has to account for every character. The Z is left over, so strptime() refuses. (Dates & Times)
What does json.loads('{"language": null, "fork": false}') give?
- {'language': null, 'fork': false}
- {'language': 'null', 'fork': 'false'}
- It stops with an error
- {'language': None, 'fork': False}
Answer: {'language': None, 'fork': False}. JSON's null becomes None and false becomes False. (Working with JSON)
language, stars = None, 0, then print(language or "none", stars or "no stars"). What prints?
- None 0
- none no stars
- none 0
- None no stars
Answer: none no stars. or gives its right side whenever the left side counts as false, and a real 0 does too. (Operators and Expressions)
NAME = r"[a-z][a-z0-9-]*". What do bool(re.match(NAME, "old_site")) and bool(re.fullmatch(NAME, "old_site")) give?
- True True
- True False
- False False
- False True
Answer: True False. match() is happy with the start, old; fullmatch() needs the whole name to fit, and the underscore doesn't. (Regular Expressions)
What does [key for key, group in groupby(["Python", "Shell", "Python"])] give?
- ['Python', 'Shell']
- ['Shell', 'Python']
- ['Python', 'Python', 'Shell']
- ['Python', 'Shell', 'Python']
Answer: ['Python', 'Shell', 'Python']. groupby() only joins neighbours, so the second Python starts a group of its own. Sort first to get one group per language. (itertools)
A DictWriter with fieldnames=["name", "stars"] is given {"name": "notes", "stars": 5, "fork": False}. What happens?
- It stops with ValueError: dict contains fields not in fieldnames: 'fork'
- It writes notes,5
- It writes notes,5,False
- It stops with KeyError: 'fork'
Answer: It stops with ValueError: dict contains fields not in fieldnames: 'fork'. A key the writer wasn't told about is an error, not something it leaves out. (Reading CSV & Excel Files)
report.write_text("name ", encoding="utf-8"), then report.write_text("name notes ", encoding="utf-8"). What does report.read_text(encoding="utf-8").splitlines() give?
- ['name', 'name', 'notes']
- ['name']
- ['name', 'notes']
- It stops with an error
Answer: ['name', 'notes']. write_text() replaces the file, so only the second text is left. (Modern File Paths with pathlib)
Continue this course
- Previous: Modern File Paths with pathlib
- Next: File Handling — Read from and write to files — txt, csv, and more
- Quick reference: Python cheat sheet
Frequently asked questions
Which lessons does this checkpoint use?
The eight before it: Regular Expressions, Dates & Times, Working with JSON, HTTP Requests & APIs, Reading CSV & Excel Files, Web Scraping with BeautifulSoup, itertools and Modern File Paths with pathlib, plus f-strings and Operators. The report never reads a web page, because GitHub's API already answers in JSON; Web Scraping with BeautifulSoup is for data that only a page shows. Each step of the brief names the lesson it comes from.
How do I run the report against the real GitHub API?
On a computer with requests installed (pip install requests), replace the entire stand-in fetching section, including fake_get and the calls that build repos, with the real pagination loop from the first section. It calls requests.get and resp.raise_for_status() for each page and builds repos until a page comes back empty. Keep the filtering and report-writing steps after that loop. This site's editor can't send requests, so keep the stand-in version there.
Why is TODAY a fixed date instead of datetime.now()?
So the report prints the same day counts every time it runs, here and in the check this site's build makes. A script you run for real would read the clock, and its numbers would change from day to day. Read it in UTC, as GitHub's times are (that's what their Z means): import timezone from datetime too, and set TODAY = datetime.now(timezone.utc).replace(tzinfo=None). The replace() drops the time zone, so TODAY can be taken from the datetimes strptime() gives. Plain datetime.now() is your computer's local time, which can be hours away from UTC, so near midnight a count could be a day off.