At scale (billions to trillions of ticks), there usually isn't a single storage engine. Most production market data platforms use a tiered architecture, where each component is optimized for a different access pattern.
Here's how the common options compare.
| System | Best for | Typical query latency | Weaknesses |
|---|
| KDB+ | Real-time analytics, quant research, HFT | microseconds–milliseconds | Expensive licensing, specialized language (q) |
| ClickHouse | Historical analytics, dashboards, backtests | milliseconds–seconds | Not ideal for sub-millisecond interactive strategies |
| ArcticDB | Python-native research datasets | milliseconds | Less suited as a production tick database |
| Custom storage | Specialized HFT workloads | microseconds | High engineering cost |
KDB+
KDB+ has been the de facto standard across many banks, exchanges, and proprietary trading firms because it was designed specifically around time-series market data.
Typical production architecture:
Feed Handler
↓
Ticker Plant
↓
Real-time Database (RDB)
↓
Intraday Database (IDB)
↓
Historical Database (HDB)
Storage characteristics:
- columnar
- memory-mapped
- partitioned by date
- sorted by timestamp + symbol
Example query:
select last price by sym from trade where time within (09:30;16:00)
Production performance commonly looks like:
- last price lookup: <100 µs
- VWAP over millions of rows: 1–10 ms
- one trading day's ticks: a few milliseconds
- years of historical data: tens of milliseconds
The biggest advantage isn't raw speed alone—it's that the query language is built around time-series operations.
ClickHouse
ClickHouse has become very popular for firms that don't want KDB licensing costs.
Strengths:
- extremely fast scans
- excellent compression
- distributed clusters
- SQL interface
- inexpensive hardware scaling
Typical schema:
ORDER BY
(
symbol,
timestamp
)
Compression often reaches:
- 5–15×
- sometimes 20× on quote data
Production queries:
SELECT
symbol,
avg(price)
FROM ticks
WHERE timestamp BETWEEN ...
GROUP BY symbol
Typical latency:
- 100 million rows: 20–200 ms
- 1 billion rows: hundreds of milliseconds
- trillions of rows distributed: seconds
For historical research, ClickHouse is exceptionally competitive.
ArcticDB
ArcticDB targets quantitative research more than production execution.
Typical workflow:
Python
↓
Pandas
↓
ArcticDB
Advantages:
- versioned datasets
- immutable snapshots
- excellent integration with pandas
- easy research reproducibility
Example:
library.read("ES_ticks")
Performance:
- loading millions of rows: milliseconds to tens of milliseconds
- optimized for data science rather than live trading
It's an excellent "research database."
Custom solutions
Some HFT firms build entirely custom storage engines.
Common design choices:
FPGA feed
↓
Shared Memory
↓
Lock-free ring buffers
↓
Compressed append-only storage
Features often include:
- proprietary compression
- custom binary formats
- SIMD decompression
- NUMA-aware memory layout
- direct memory mapping
Query latency may be measured in microseconds because the engine only supports the firm's required query patterns.
The trade-off is significantly higher engineering complexity.
Storage formats
Tick data is almost always stored in a columnar layout.
Instead of:
timestamp symbol price size
timestamp symbol price size
it's stored as:
timestamps[]
symbols[]
prices[]
sizes[]
This enables:
- vectorized execution
- cache-efficient scans
- higher compression
- SIMD acceleration
Partitioning
Large deployments commonly partition by:
Date
↓
Exchange
↓
Symbol
or
Year/
Month/
Day/
Within each partition, records are sorted by timestamp.
Indexes are often sparse because sequential scans over compressed columnar data are very fast.
Compression
Market data compresses particularly well due to repeated values and predictable patterns.
Common techniques include:
- delta encoding for timestamps
- dictionary encoding for symbols
- run-length encoding
- Gorilla compression for floating-point values
- Zstandard (ZSTD)
- LZ4 for faster decompression
Compression ratios of 8–15× are common for tick data.
Typical production architecture
Many modern trading firms use a combination rather than a single database:
Exchange
↓
Feed handlers
↓
Kafka / Aeron
↓
Real-time cache
↓
KDB+ or custom engine
↓
ClickHouse
↓
Parquet on object storage
In this pattern:
- KDB+ (or a custom in-memory engine) serves latency-sensitive intraday queries.
- ClickHouse powers historical analytics, monitoring, and ad hoc SQL.
- Parquet on object storage provides a low-cost archival layer.
- ArcticDB or similar tooling may sit alongside this stack for reproducible quantitative research.
What query performance looks like in production
Actual performance depends heavily on hardware, data layout, query selectivity, and concurrency, but representative ranges are:
- Single symbol, one trading day: sub-millisecond to a few milliseconds.
- Hundreds of symbols over several days: a few to tens of milliseconds.
- Full-market scans over billions of ticks: tens to hundreds of milliseconds on well-tuned columnar systems.
- Multi-year historical analytics across trillions of rows: typically hundreds of milliseconds to a few seconds on distributed clusters.
For ultra-low-latency trading, firms generally avoid querying a historical database during strategy execution. Live strategies consume data from in-memory order books or lock-free shared-memory structures, while historical databases are used for research, backtesting, surveillance, and post-trade analytics. This separation keeps execution latency predictable while allowing rich historical analysis.
