PYTHON / FILE HANDLING
Reading whole files and reading line by line
Read a text file in one call with read(), or stream it one line at a time by iterating the file object, and know when each is correct.
What you will learn
- Use f.read() for a single string, f.readlines() for a list of lines
- Iterate `for line in f` to process a file of any size with constant memory
- Know that each line keeps its trailing newline unless you strip it
- Recognise that a file object holds a cursor, so a second read() returns ''
Understanding Reading whole files and reading line by line
An open text file is not a string; it is a stream with a position. When you call f.read(), Python decodes every remaining byte from the current position to the end and hands you one string, then leaves the cursor at end of file. That is why a second f.read() on the same object returns '' rather than the contents again, and why f.seek(0) is needed to start over.
Iterating the file object instead (`for line in f`) asks the stream for the next chunk up to and including the next newline. Only that one line exists as a Python string at a time, so a 2 GB log file costs a few kilobytes of memory instead of 2 GB plus decoding overhead. The split points are newline characters, so the last piece has no trailing '\n' if the file does not end with one.
Because the newline is part of the data, a line read from disk is 'gamma\n', not 'gamma'. Comparisons and int() conversions on the raw line usually still need line.strip() or line.rstrip('\n'), and printing a raw line produces a blank line because print() adds its own newline. Choose read() when you need the whole text as one value, such as for a regex over the file or json.loads; choose iteration when the file is processed record by record or is large enough that its size is unknown.
with open("notes.txt", "w") as f:
f.write("alpha\nbeta\ngamma\n")
with open("notes.txt") as f:
whole = f.read()
print(repr(whole))
print(len(whole))
with open("notes.txt") as f:
for line in f:
print(repr(line))A file object is a cursor over a stream: read() consumes everything at once, while iteration yields one newline-terminated line at a time.
Worked examples
The cursor moves as you read
Shows that reading consumes the stream and that seek(0) rewinds it.
with open("data.txt", "w") as f:
f.write("one\ntwo\n")
with open("data.txt") as f:
print("first: ", repr(f.read()))
print("second:", repr(f.read()))
f.seek(0)
print("rewound:", repr(f.readline()))Example explained
Line 1The first f.read() returns everything and leaves the position at end of file.
Line 2The second f.read() finds nothing left to decode, so it returns the empty string, not None and not an error.
Line 3f.seek(0) moves the position back to byte 0 of the file.
Line 4f.readline() then returns just the first line, including its newline.
Newlines and the last line
Compares readlines(), iteration with rstrip, and splitlines() on a file with no final newline.
with open("cities.txt", "w") as f:
f.write("Oslo\nLima\nCairo")
with open("cities.txt") as f:
print(f.readlines())
with open("cities.txt") as f:
print([line.rstrip("\n") for line in f])
with open("cities.txt") as f:
print(f.read().splitlines())Example explained
Line 1readlines() keeps every newline, and 'Cairo' has none because the file does not end with one.
Line 2rstrip('\n') removes only the trailing newline and leaves other whitespace intact.
Line 3read().splitlines() gives the same clean list but first loads the entire file into memory.
Line 4All three reopen the file because the previous cursor was already at end of file.
Streaming an aggregate
Processes numbers one line at a time while tracking line numbers and skipping blanks.
with open("temps.txt", "w") as f:
f.write("21.5\n19.0\n\n23.5\n")
total = 0.0
count = 0
with open("temps.txt") as f:
for lineno, line in enumerate(f, start=1):
text = line.strip()
if not text:
print(f"line {lineno}: blank, skipped")
continue
total += float(text)
count += 1
print(f"{count} values, mean {total / count:.2f}")Example explained
Line 1enumerate(f, start=1) numbers lines as they arrive without building a list of them.
Line 2line 3 is '\n', so after strip() it is '' and float() would raise ValueError on it.
Line 3float(text) needs the stripped text; float('21.5\n') happens to work, but int('7\n') style input from other formats is safer stripped.
Line 4Memory use does not grow with the file because only one line is held at a time.
Important notes
Text mode translates '\r\n' and '\r' to '\n' while reading, so len(f.read()) can be smaller than the file size in bytes.
The file's line iterator dies with the file: looping over lines after the with block ends raises ValueError: I/O operation on closed file, though a list from readlines() survives.
Common mistakes
Calling f.read() a second time and concluding the file is empty; the cursor is already at end of file, so it returns ''.
Comparing a raw line to text, as in `if line == "gamma":`, which is False because the line is actually 'gamma\n'.
Using f.read().split('\n') on a file that ends with a newline, which produces a trailing '' element; splitlines() or iteration does not.
Try it yourself
Change, predict, then run
Write a file whose last line has no trailing newline, then loop over it printing repr(line) and len(line) for each line, and confirm which single line differs in length from the others.
Open the Python workspaceCheck your understanding
A script counts lines containing 'ERROR' in a 4 GB log. Why is `for line in f` preferred over `f.readlines()`?
- readlines() builds a list holding the whole decoded file plus one string per line at once, while iteration holds only the current line
- readlines() strips the newline characters, so lines ending in 'ERROR\n' would not match
- iterating the file is faster because it bypasses the operating system's buffering
- readlines() only works on files opened in binary mode
Show answer
The difference is memory: readlines() materialises every line simultaneously, so peak usage scales with file size, whereas iteration keeps one line alive at a time. Option 2 is tempting but wrong in the opposite direction, since readlines() preserves each trailing newline rather than stripping it; that affects string comparisons, not this substring search.