PYTHON / CAPSTONE PROJECTS
Project: a command-line data cleaning tool
Build a CLI data cleaner in Python where pure per-field cleaning functions return values plus problem reports, wired up with argparse at the edges.
What you will learn
- Write cleaning rules as pure functions returning (value, problem) instead of printing
- Stream rows with csv.DictReader/DictWriter so file size never matters
- Send cleaned rows to stdout and the rejection report to stderr
- Check that cleaning a already-clean value returns it unchanged (idempotence)
Understanding Project: a command-line data cleaning tool
A data cleaning tool has two jobs that people usually tangle together: deciding what a value should become, and moving bytes in and out of files. Keep them apart. Each rule is a small function that takes one raw string and returns a pair: the cleaned value and either None or a short description of what was wrong. Because such a function touches no files, no argv and no global state, you can test it with a single assert, and you can reuse the same rule from a CLI, a test or a notebook.
The second half of the design is deciding what a rejected row means. A crash on line 40000 of a 50000-row file is the worst outcome: you lose the run and you learn about exactly one bad value. Instead, collect problems per row and let the caller choose the policy, which is what an --on-error flag with drop, keep and fail options expresses. Then the count of rejections becomes part of the tool's output, and a run that quietly dropped 900 rows is visible rather than invisible.
Because the cleaned rows are the tool's real product, they belong on stdout so the tool can sit in a shell pipeline; the human-facing summary belongs on stderr so it never lands inside the CSV. This also settles the in-place question: never open the input file for writing while you are still reading it, since truncating the file you are streaming destroys the data. Write to a new file or to stdout, and let the user redirect. One more property worth enforcing: cleaning an already-clean value must return it unchanged, so re-running the tool on its own output is safe.
import csv import io RAW = """name,email,signups Ada Lovelace ,ADA@Example.COM ,12 Grace Hopper,grace@example.com, Charles Babbage,charles[at]example.com,3 """ def clean_name(value): return " ".join(value.split()), None def clean_email(value): value = value.strip().lower() return value, None if "@" in value else "email has no @" def clean_signups(value): value = value.strip() if value == "": return 0, None if not value.isdigit(): return None, "signups is not a whole number: " + repr(value) return int(value), None CLEANERS = {"name": clean_name, "email": clean_email, "signups": clean_signups} def clean_row(row): cleaned, problems = {}, [] for field, value in row.items(): cleaned[field], problem = CLEANERS[field](value) if problem: problems.append(problem) return cleaned, problems for lineno, row in enumerate(csv.DictReader(io.StringIO(RAW)), start=2): cleaned, problems = clean_row(row) if problems: print("line", lineno, "rejected:", "; ".join(problems)) else: print("line", lineno, "ok:", cleaned)
import csv
import io
RAW = """name,email,signups
Ada Lovelace ,ADA@Example.COM ,12
Grace Hopper,grace@example.com,
Charles Babbage,charles[at]example.com,3
"""
def clean_name(value):
return " ".join(value.split()), None
def clean_email(value):
value = value.strip().lower()
return value, None if "@" in value else "email has no @"
def clean_signups(value):
value = value.strip()
if value == "":
return 0, None
if not value.isdigit():
return None, "signups is not a whole number: " + repr(value)
return int(value), None
CLEANERS = {"name": clean_name, "email": clean_email, "signups": clean_signups}
def clean_row(row):
cleaned, problems = {}, []
for field, value in row.items():
cleaned[field], problem = CLEANERS[field](value)
if problem:
problems.append(problem)
return cleaned, problems
for lineno, row in enumerate(csv.DictReader(io.StringIO(RAW)), start=2):
cleaned, problems = clean_row(row)
if problems:
print("line", lineno, "rejected:", "; ".join(problems))
else:
print("line", lineno, "ok:", cleaned)Cleaning rules are pure functions that return a cleaned value plus a problem report, and argparse plus file handles stay at the outer edge of the program.
Worked examples
The command-line surface
Defines the tool's flags with argparse and inspects the parsed options without running a shell.
import argparse
import contextlib
import io
parser = argparse.ArgumentParser(prog="cleancsv")
parser.add_argument("infile", nargs="?", default="-")
parser.add_argument("--on-error", choices=["drop", "keep", "fail"], default="drop")
parser.add_argument("--quiet", action="store_true")
args = parser.parse_args(["contacts.csv", "--on-error", "fail"])
print(args.infile, args.on_error, args.quiet)
args = parser.parse_args([])
print(args.infile, args.on_error, args.quiet)
with contextlib.redirect_stderr(io.StringIO()):
try:
parser.parse_args(["--on-error", "guess"])
except SystemExit as exc:
code = exc.code
print("bad choice rejected with exit code", code)Example explained
Line 1nargs="?" with default="-" makes the input file optional, so a missing argument means read stdin.
Line 2choices= turns the error policy into a validated set, so a typo cannot silently select the wrong behaviour.
Line 3argparse reports the bad choice on stderr and raises SystemExit with code 2, which is why the shell sees a failure.
Line 4--on-error becomes the attribute args.on_error: argparse converts the dash to an underscore.
Idempotent phone normalisation
Shows a cleaner that produces a canonical form, then asserts that re-cleaning that form changes nothing.
from collections import Counter
def clean_phone(value):
digits = "".join(ch for ch in value if ch.isdigit())
if len(digits) != 10:
return None, "phone needs exactly 10 digits"
return "(%s) %s-%s" % (digits[:3], digits[3:6], digits[6:]), None
problems = Counter()
for raw in ["555 867 5309", "+1 (555) 010-9999", "12345", "5550109999"]:
value, problem = clean_phone(raw)
if problem:
problems[problem] += 1
print(repr(raw), "-> rejected")
else:
print(repr(raw), "->", value)
assert clean_phone(value) == (value, None)
print(dict(problems))Example explained
Line 1Stripping to digits first means the rule does not care which separators the source used.
Line 2The assert states the idempotence property: feeding the cleaned value back in returns it untouched.
Line 3Counter accumulates one line per distinct problem, which becomes the end-of-run summary.
Line 4The 11-digit number is rejected rather than truncated, because guessing which digit to drop would invent data.
Important notes
str.isdigit() is not a number check: it rejects "-3", "4.5" and "1,000", and accepts non-ASCII digit characters, so use int(value) inside try/except when signs or separators are legal.
When you open real CSV files, pass newline="" to open(); without it the csv module can emit blank lines between rows on some platforms and mishandle embedded newlines.
Common mistakes
Calling open() or print() inside the cleaning rules: the rules can then only be exercised by running the whole program, and the report cannot be redirected.
Opening the input file with mode "w" or "r+" to clean it in place while iterating it, which truncates the file and destroys rows you have not read yet.
Turning bad values into 0 or "" instead of reporting them, so a file where half the numeric column failed to parse looks like a successful run.
Try it yourself
Change, predict, then run
Add a clean_joined_on rule to the main example that turns "2024-3-7" into "2024-03-07" and rejects anything that is not three integers separated by dashes, then print a Counter of problem messages after the loop.
Open the Python workspaceCheck your understanding
Your tool writes cleaned rows plus a "rejected 3 rows" summary. Why should the summary go to stderr rather than stdout?
- Because stdout carries the cleaned CSV, and the summary would otherwise appear as a row in whatever consumes the pipe
- Because stderr is unbuffered and therefore faster for short messages
- Because print() cannot write to stdout once csv.writer has written to it
- Because the shell automatically saves anything on stderr to a log file
Show answer
stdout is the data channel: a summary line printed there becomes a malformed record for the next command in the pipeline, while stderr stays visible on the terminal even when stdout is redirected to a file. Buffering differences are real but irrelevant here, and the shell does not log stderr anywhere unless you redirect it yourself.