The most common reason is that your script is loading the entire CSV into memory at once. A CSV file may be several gigabytes on disk, but once parsed into Python objects (strings, lists, dictionaries, or DataFrames), it can consume many times more memory.
Here are the most common causes and fixes.
1. You're reading the whole file at once
Memory-heavy:
import pandas as pd
df = pd.read_csv("huge.csv")
or
with open("huge.csv") as f:
rows = list(csv.reader(f))
Both load everything into RAM.
Better: process one chunk at a time
import pandas as pd
for chunk in pd.read_csv("huge.csv", chunksize=100_000):
# Process chunk
print(len(chunk))
Or with the standard library:
import csv
with open("huge.csv", newline="") as f:
reader = csv.reader(f)
for row in reader:
# Process one row
pass
2. Pandas is using larger data types than necessary
For example:
int64 instead of int32float64 instead of float32- strings stored as generic Python objects
Specify dtypes when possible:
df = pd.read_csv(
"data.csv",
dtype={
"age": "int16",
"price": "float32"
}
)
If a column contains repeated strings, converting it to a categorical type can save substantial memory:
df["state"] = df["state"].astype("category")
3. You're creating unnecessary copies
Operations like these can temporarily double memory usage:
df2 = df.copy()
filtered = df[df["value"] > 0]
When practical:
- overwrite variables you no longer need
- delete unused objects
del df
4. You only need a few columns
Instead of loading every column:
df = pd.read_csv(
"huge.csv",
usecols=["id", "price", "date"]
)
5. You're storing processed data in a list
This defeats streaming:
results = []
for row in reader:
results.append(process(row))
Instead, write results as you go:
with open("output.csv", "w", newline="") as out:
writer = csv.writer(out)
for row in reader:
writer.writerow(process(row))
6. Strings are expensive
A CSV with millions of unique text values can use much more memory than expected because every string becomes a Python object.
If possible:
- process rows individually
- use categorical columns for repeated values
- avoid keeping all strings in memory
7. Estimate the memory needed
A CSV that's:
- 2 GB on disk might occupy 5–15 GB in memory.
- 10 GB on disk may require 20–50+ GB depending on data types.
The in-memory representation is much larger than the text file because Python objects and pandas structures have overhead.
Recommended pattern for huge files
import pandas as pd
total = 0
for chunk in pd.read_csv(
"huge.csv",
chunksize=100_000,
usecols=["price"],
dtype={"price": "float32"}
):
total += chunk["price"].sum()
print(total)
This keeps memory usage roughly constant regardless of the file size.
If the file is extremely large
If you're working with tens or hundreds of gigabytes, consider tools designed for out-of-core processing, such as:
- DuckDB for SQL-style analysis directly on CSV and Parquet files.
- Polars with streaming or lazy execution.
- Dask for distributed or out-of-core pandas-like workflows.
These tools can process data that doesn't fit entirely in RAM.
If you share your code (especially the part that reads and processes the CSV) and roughly how large the file is, I can point out the specific memory bottlenecks and suggest targeted improvements.
