The right choice depends less on dataset size than on how your team works.
Here's a practical way to think about it:
| Situation | Recommendation |
|---|
| Mostly ML experiments with files on S3/GCS/Azure | DVC |
| Data lake with many users, ETL pipelines, and branching needs | LakeFS |
| Small team, simple workflows | Object storage + manifests + Git |
| Mostly tabular data in a warehouse | Warehouse-native versioning (e.g. snapshots/time travel) |
Option 1: DVC
Best for: ML teams tracking datasets alongside model code.
Pros
- Git-like workflow (
add, push, pull)
- Stores only metadata in Git
- Large files live in cloud object storage
- Good experiment tracking integration
- Easy reproducibility
Cons
- Can become cumbersome with many TB-scale datasets
- File-oriented rather than data-lake-oriented
- Developers need to learn DVC commands
Storage costs:
- DVC doesn't duplicate data by default if your remote storage is configured well.
- Large immutable datasets are uploaded once.
- New versions only upload changed files (not changed bytes within a file).
Works especially well when datasets consist of many partitioned files rather than one huge monolithic file.
Option 2: LakeFS
Best for: Organizations treating data lakes like Git repositories.
Instead of versioning files directly, it versions an object store.
Features include:
- branches
- commits
- merges
- rollback
- isolated experimentation
Advantages:
- Works well with Spark
- Works well with Airflow
- Compatible with many existing tools
- Handles petabyte-scale lakes
Example workflow:
main
├── experiment-a
├── experiment-b
└── production
Each branch references existing objects instead of copying everything.
Storage stays efficient because unchanged objects are shared.
This is a major advantage over naïvely copying datasets.
Option 3: Simple manifest approach
Many teams don't actually need DVC or LakeFS.
Store data like:
s3://datasets/
customers/
v1/
v2/
v3/
transactions/
2026/
2027/
Then in Git:
manifest.yaml
training_data:
version: v3
or
dataset:
uri: s3://datasets/customers/v3
Advantages:
- extremely simple
- almost zero tooling
- works everywhere
- very low maintenance
For many startups this is enough.
Keeping storage costs low
This matters more than the versioning tool.
1. Never rewrite whole datasets
Instead of
dataset.parquet
prefer
2026/
part-001.parquet
part-002.parquet
If one partition changes, only that object changes.
2. Use immutable objects
Avoid:
sales.parquet
Prefer:
sales/
snapshot-2026-07-01/
snapshot-2026-07-15/
Or partition by date.
Immutable objects enable reuse and make cleanup policies safer.
3. Partition intelligently
Instead of
4 TB file
use
year=2026/
month=07/
day=17/
Only changed partitions require new storage.
4. Use columnar formats
Parquet and ORC are generally much smaller than CSV because they support efficient compression and columnar storage.
Savings of 3–10× over CSV are common, depending on the data.
5. Separate raw and derived data
Keep:
raw/
processed/
features/
If feature engineering changes, you don't need to duplicate raw data.
6. Use lifecycle policies
Cloud storage can automatically:
- archive old versions
- move cold data to cheaper storage classes
- delete temporary experiment data after a retention period
This often has a bigger impact on costs than the choice of versioning tool.
7. Content-addressable storage
Tools like DVC and LakeFS use content hashes so identical objects are stored once.
If ten experiments reference the same 100 GB dataset, storage stays close to 100 GB rather than 1 TB.
What many mature ML teams do
A common architecture looks like:
Object Storage (S3/GCS/Azure Blob)
│
├── raw
├── processed
├── features
│
LakeFS (branch/version)
│
Spark / Airflow / dbt
│
ML pipeline
│
DVC or experiment tracker
│
Model registry
In this setup, LakeFS versions the shared data lake, while DVC (or another experiment-tracking tool) can be used selectively for model artifacts or smaller datasets tied closely to code.
A practical recommendation
- Small team (1–5 people): Use cloud object storage with immutable, partitioned datasets and version manifests in Git. Add DVC only if you need stronger reproducibility.
- Growing ML team (5–20 people): DVC is a good fit if workflows are centered on model development and dataset snapshots.
- Large platform or data engineering organization: LakeFS becomes compelling because Git-style branching and commits for the entire data lake help multiple teams collaborate without duplicating large datasets.
Regardless of the tool, the biggest drivers of storage cost are how you organize the data (immutable, partitioned, columnar) and your retention/lifecycle policies—not the versioning system itself.
