PYTHON / FILE HANDLING
Temporary files and safe atomic writes
Write files without risking corruption: build the new content in a temp file beside the target, then publish it with one os.replace call.
What you will learn
- Write a temp file beside the target, then publish it with a single os.replace()
- Use os.replace, not os.rename, so overwriting an existing file works on Windows
- Call flush() then os.fsync() before the replace when data must survive a crash
- Reach for mkstemp or NamedTemporaryFile(delete=False) when you move the file yourself
Understanding Temporary files and safe atomic writes
open(path, "w") truncates the file the instant it is opened, before a single byte of new content exists. If your program raises halfway through serializing, or the process is killed, the file on disk is now shorter than the old version and not yet the new one. Nothing in the write path can undo that, so the fix is structural: never write into the file readers are using.
The only cheap operation a filesystem gives you that is atomic is rename. So the pattern is: create a unique scratch file, write the complete new content into it, close it, then call os.replace(tmp, target). Any reader either opens the old inode or the new one, never a half-filled file. The temp file must live in the same directory as the target, because rename is atomic only within one filesystem; if tmp is on another mount, os.replace raises OSError with Errno 18, Invalid cross-device link.
tempfile gives you the scratch file safely. mkstemp and NamedTemporaryFile open with O_CREAT|O_EXCL and mode 0o600, so they cannot collide with an existing name or be hijacked through a symlink in a shared directory, and each call picks a fresh random name so two processes writing the same target do not stomp on each other. Atomic and durable are different properties: os.replace makes the switch indivisible, but only flush() plus os.fsync(fd) before the replace guarantees the bytes are actually on the platter if power drops.
import json
import os
import tempfile
from pathlib import Path
target = Path("settings.json")
target.write_text('{"theme": "light", "font": 12}', encoding="utf-8")
def atomic_write(path, write_body, encoding="utf-8"):
path = Path(path)
fd, tmp_name = tempfile.mkstemp(dir=path.parent,
prefix=path.name + ".",
suffix=".tmp")
try:
with open(fd, "w", encoding=encoding) as f:
write_body(f)
f.flush()
os.fsync(f.fileno())
os.replace(tmp_name, path)
except BaseException:
os.unlink(tmp_name)
raise
def broken_body(f):
f.write('{"theme": "dark"')
raise ValueError("record 3 is malformed")
try:
atomic_write(target, broken_body)
except ValueError as exc:
print("write failed:", exc)
print("target intact:", target.read_text(encoding="utf-8"))
atomic_write(target, lambda f: json.dump({"theme": "dark", "font": 14}, f))
print("target updated:", target.read_text(encoding="utf-8"))
print("temp files left:", sorted(p.name for p in Path(".").glob("settings.json.*")))A safe write is a complete write to a throwaway file in the same directory followed by one atomic rename over the target.
Worked examples
What mode "w" actually costs you
Shows that opening for writing destroys the old content before the new content is produced.
from pathlib import Path
p = Path("data.txt")
p.write_text("original contents\n", encoding="utf-8")
try:
with open(p, "w", encoding="utf-8") as f:
f.write("new ")
raise RuntimeError("serializer failed on row 3")
except RuntimeError as exc:
print("error:", exc)
text = p.read_text(encoding="utf-8")
print("bytes on disk now:", repr(text))
print("original still there:", "original" in text)Example explained
Line 1open(p, "w") truncates data.txt to zero length at open time, not at close time.
Line 2The exception propagates, but the with statement still closes the file, which flushes the 4 characters already buffered.
Line 3The result is a file that is neither the old nor the new version, which is exactly the state os.replace prevents.
Line 4Note that the with statement did its job perfectly here; context managers protect the handle, not the content.
Scratch files that clean up, and ones that do not
Contrasts TemporaryFile (anonymous, auto-deleted) with NamedTemporaryFile(delete=False), the form you need when you plan to rename the file yourself.
import os
import tempfile
with tempfile.TemporaryFile(mode="w+", encoding="utf-8") as f:
f.write("id,score\n7,91\n")
f.seek(0)
print("scratch file:", f.read().splitlines())
with tempfile.NamedTemporaryFile(mode="w", suffix=".csv",
delete=False, encoding="utf-8") as nf:
nf.write("id,score\n8,73\n")
kept = nf.name
print("survives the block:", os.path.exists(kept))
with open(kept, encoding="utf-8") as f:
print("contents:", f.read().splitlines())
os.unlink(kept)
print("after unlink:", os.path.exists(kept))Example explained
Line 1mode="w+" is required to read a temp file back; f.seek(0) rewinds, otherwise read() returns an empty string.
Line 2TemporaryFile is deleted when the block ends, so it is only useful for data that never needs a path.
Line 3delete=False keeps the file after close, which is the whole point when the next step is os.replace.
Line 4With delete=False you own the cleanup, so every failure path must unlink the name yourself.
A whole directory as the scratch space
Uses TemporaryDirectory when a job produces several intermediate files that should all disappear together.
import tempfile
from pathlib import Path
with tempfile.TemporaryDirectory(prefix="report-") as name:
work = Path(name)
(work / "a.txt").write_text("alpha\n", encoding="utf-8")
(work / "b.txt").write_text("beta\n", encoding="utf-8")
print("contents:", sorted(p.name for p in work.iterdir()))
print("prefix used:", work.name.startswith("report-"))
print("directory removed:", not work.exists())Example explained
Line 1TemporaryDirectory yields a string path, so wrap it in Path() to use the pathlib API on it.
Line 2prefix makes the random name recognisable if you ever have to debug leftovers by hand.
Line 3On exit the whole tree is removed recursively, even files you created after entering the block.
Line 4This is not a substitute for the atomic-write recipe: a temp dir usually sits on a different filesystem than your target.
Important notes
os.replace hands the target the temp file's metadata, and mkstemp creates files with mode 0o600, so a file that other users or services need to read must have its permissions restored with shutil.copymode(target, tmp_name) before the replace.
Atomic is not the same as durable: after os.replace the directory entry change may still sit in the OS cache, so crash-critical code also opens the parent directory and fsyncs it. On Windows, os.replace raises PermissionError if another process currently has the target open.
Common mistakes
Using tempfile.NamedTemporaryFile() with the default delete=True and then os.replace-ing it: when the block exits, the cleanup tries to unlink a name that no longer exists and you get FileNotFoundError instead of a successful write.
Leaving dir at its default so the temp file lands in /tmp while the target is on another disk or a mounted volume: os.replace fails with OSError Errno 18, Invalid cross-device link, and switching to shutil.move silently turns the publish step back into a non-atomic copy.
Using a fixed scratch name like "settings.json.tmp" instead of a random one: two processes writing the same target interleave their bytes in that single file, and one of them then renames the mixture over the real data.
Try it yourself
Change, predict, then run
Write atomic_json_dump(path, obj) that dumps obj as JSON using the mkstemp plus os.replace pattern, then prove it works: create the file with a valid dict first, call your function with {"tags": {1, 2}} so json raises TypeError, and check afterwards that the file still parses and that no .tmp files remain in the directory.
Open the Python workspaceCheck your understanding
Why must the temporary file be created in the same directory as the file you are replacing?
- Because rename is atomic only within a single filesystem, and a cross-device move degrades into a non-atomic copy
- Because tempfile.mkstemp refuses to create files outside the current working directory
- Because the system temp directory is cleared on reboot, so the new content could be lost
- Because os.fsync can only flush a file that shares a parent directory with its target
Show answer
os.replace maps to the rename syscall, which the kernel can only perform atomically inside one filesystem; if the temp file is on a different mount, os.replace raises Invalid cross-device link, and the usual workaround (shutil.move) copies bytes into the destination in place, reintroducing exactly the half-written window you were trying to eliminate. The reboot answer describes a real property of /tmp but is irrelevant here, since the temp file exists for only a few milliseconds before the rename.