PYTHON / PANDAS
Type conversion and categorical data
Convert pandas columns between string, numeric, nullable-integer, and category dtypes, and control how categorical order drives sorting and comparison.
What you will learn
- Parse messy numeric text with pd.to_numeric(errors='coerce') instead of astype
- Keep missing values in an integer column by using the Int64 nullable dtype
- Convert low-cardinality label columns to category to store codes, not repeated strings
- Define category order with CategoricalDtype(ordered=True) so <, >, sort and max work
Understanding Type conversion and categorical data
Every pandas column has one dtype, and that dtype decides both how the values sit in memory and which operations are legal. An object column of price strings stores a pointer to a separate Python str for each row, so df['price'].mean() either fails or concatenates text; the same column as float64 is a contiguous block of 8-byte doubles that NumPy can sum in one pass. Converting types is therefore not cosmetic tidying, it is choosing the machine representation that makes the arithmetic you want possible.
There are two conversion styles and they fail differently. astype is strict: it demands that every value be convertible and raises on the first one that is not, which is what you want when you believe the data is already clean. pd.to_numeric with errors='coerce' is lenient: unparseable entries such as 'n/a' become NaN, and because NaN is a float the whole result is float64 even when every real value is a whole number. If you need whole numbers plus missing values, convert onward to the nullable 'Int64' dtype, which carries a separate boolean mask and uses pd.NA instead of NaN.
The category dtype is a different idea from int64 or float64: it is a two-part structure. Pandas keeps one array of the distinct values (the categories) and one array of small integers (the codes) that point into it, so 100000 rows of 'S'/'M'/'L' become 100000 int8 codes plus three strings. That is where the memory saving comes from, and it also explains the surprising behaviours: comparisons and sorting operate on the codes, astype('category') assigns codes by sorting the unique values, and assigning a label that has no code yet is rejected rather than silently added.
import pandas as pd
df = pd.DataFrame({
"id": ["1", "2", "3", "4"],
"price": ["10.50", "12.50", "n/a", "7.00"],
"size": ["S", "M", "S", "L"],
})
print("before:", [str(t) for t in df.dtypes])
df["id"] = df["id"].astype("int64")
df["price"] = pd.to_numeric(df["price"], errors="coerce")
df["size"] = df["size"].astype("category")
print("after: ", [str(t) for t in df.dtypes])
print("categories:", df["size"].cat.categories.tolist())
print("codes:", df["size"].cat.codes.tolist())
print("mean price:", df["price"].mean())A column's dtype is its storage format and its contract for what operations are allowed, and category stores integer codes into a fixed list of labels rather than the labels themselves.
Worked examples
Ordered categories
Declaring the category order makes comparison operators and max follow size order instead of the alphabet.
import pandas as pd
from pandas.api.types import CategoricalDtype
sizes = CategoricalDtype(categories=["S", "M", "L"], ordered=True)
s = pd.Series(["L", "S", "M", "S"], dtype=sizes)
print(s.sort_values().tolist())
print((s > "S").tolist())
print(s.max())Example explained
Line 1CategoricalDtype fixes the code assignment: S=0, M=1, L=2, regardless of alphabetical order.
Line 2sort_values sorts the codes, so the labels come back in S, M, L order rather than L, M, S.
Line 3s > "S" is only allowed because ordered=True; on an unordered categorical the same comparison raises TypeError.
Line 4s.max() returns the label with the highest code, which is a meaningful answer only for ordered categoricals.
Categories are a closed set
Writing a label that is not already a category is rejected until you add it explicitly.
import pandas as pd
s = pd.Series(["cat", "dog", "cat"], dtype="category")
try:
s.iloc[1] = "bird"
except (TypeError, ValueError):
print("assignment rejected")
s = s.cat.add_categories(["bird"])
s.iloc[1] = "bird"
for name, count in s.value_counts().items():
print(name, int(count))Example explained
Line 1The assignment fails because 'bird' has no integer code, and pandas will not invent one behind your back.
Line 2cat.add_categories returns a new Series whose category list has room for 'bird', so the same assignment now succeeds.
Line 3value_counts on a categorical reports every category, so 'dog' still appears with a count of 0 after its only row was overwritten.
Line 4int(count) is used because the counts come back as NumPy integers.
Important notes
category only saves memory when the number of distinct values is small relative to the number of rows; on a near-unique ID column it stores the codes and all the original values, so it costs more than object.
Concatenating or comparing two categorical columns whose category lists differ falls back to object dtype, quietly discarding both the memory saving and any declared order.
Common mistakes
Calling .astype("int64") on a column that contains NaN: pandas raises instead of rounding or dropping, because float64 NaN has no int64 representation. Coerce to numeric first, then use "Int64".
Assuming astype("category") preserves the order the labels appeared in the data. It sorts the unique values, so 'Large', 'Medium', 'Small' get codes 0, 1, 2 and every later sort_values or comparison is wrong.
Filtering rows out of a categorical column and expecting the unused labels to disappear. The category list is unchanged, so value_counts and groupby keep reporting empty groups until you call cat.remove_unused_categories().
Try it yourself
Change, predict, then run
Build a Series from ["90", "75", "missing", "60"] and convert it to a nullable Int64 column without raising, then create a second Series of ["C", "A", "B", "A"] as an ordered category with A above B above C and print its sorted values.
Open the Python workspaceCheck your understanding
A column of shirt sizes is converted with astype("category") and then sorted with sort_values(); the result comes out L, M, S. What explains that order?
- sort_values orders by the integer codes, and astype("category") assigned codes by sorting the unique labels, giving L=0, M=1, S=2
- Unordered categoricals cannot be compared, so sort_values silently falls back to comparing the label strings
- The codes are assigned in order of first appearance in the data, and L happened to appear first
- Unordered categoricals are never reordered, so sort_values returned the original row order unchanged
Show answer
astype builds the category list by sorting the distinct labels, and every sort or comparison on a categorical works on the codes that index that list, so alphabetical categories produce an alphabetical sort. The string-comparison option looks identical here but is the wrong mechanism: pass CategoricalDtype(categories=["S","M","L"], ordered=True) and sort_values returns S, M, L, which no string comparison would produce.