The main challenge is that realism and speed tend to pull in opposite directions. A backtester that models every microsecond event like a live exchange often becomes unusably slow, while a vectorized backtester misses many of the behaviors that actually determine P&L.
A good compromise is to separate market simulation from strategy evaluation and make realism configurable.
1. Use an event-driven core, but avoid per-tick bookkeeping when unnecessary
Instead of processing everything synchronously:
Market Data
│
▼
Event Queue
│
┌───┴────┐
│ │
Strategy Exchange Simulator
│ │
└───┬────┘
▼
Portfolio
Each event has:
- timestamp
- source
- sequence number
- event type
- payload
Example:
09:30:00.000001 quote
09:30:00.000003 trade
09:30:00.000004 order_ack
09:30:00.000011 partial_fill
09:30:00.000020 cancel_ack
The sequence number resolves events with identical timestamps.
Rather than replaying every nanosecond, jump directly between events.
2. Model latency explicitly
Don't assume an order appears instantly.
Instead:
strategy decides
│
▼
network latency
│
▼
exchange receives
│
▼
matching engine
│
▼
execution
│
▼
fill notification latency
This naturally creates asynchronous behavior.
Example:
Market @ 10.000
Strategy sends buy
Latency = 350 μs
By arrival:
Market = 10.03
Order joins book at 10.03
That single change often has a larger impact on results than sophisticated fill models.
3. Partial fills should come from available liquidity
Instead of
if price touched:
fill 100%
maintain available size.
Example:
Book:
Ask
100.01 120
100.02 400
Buy order:
300 shares
Result:
120 @100.01
180 @100.02
If your order rests:
Queue position = 950 shares
Incoming trades:
300
150
200
400
Only after 950 shares trade do you begin filling.
Queue modeling matters much more than adding random slippage.
4. Queue position approximation
Full order book replay is expensive.
A common approximation tracks:
queue_ahead
When new trades hit:
queue_ahead -= executed_volume
if queue_ahead <= 0:
begin fills
You don't need every individual order.
Just maintain aggregate size ahead.
This is surprisingly accurate for many strategies.
5. Venue-specific matching rules
Different venues behave differently.
Rather than:
fill(order)
use:
venue.execute(order, book)
Each venue implements:
Price-time priority
Pro-rata
Maker/taker fees
Hidden liquidity
Icebergs
Auction logic
Minimum quantity
Odd-lot rules
Self-trade prevention
Pegged orders
Then swapping venues becomes
NYSEVenue()
CMEVenue()
CryptoVenue()
without changing the engine.
6. Asynchronous acknowledgements
Orders don't transition directly:
NEW -> FILLED
Instead:
NEW
↓
PENDING_SUBMIT
↓
ACKNOWLEDGED
↓
PARTIALLY_FILLED
↓
FILLED
or
↓
CANCEL_PENDING
↓
CANCELLED
This lets you model races like:
Cancel sent
↓
Fill happens before cancel processed
↓
Remaining cancelled
which happens frequently in live trading.
7. Keep state mutations localized
A fast engine usually has only a few mutable objects:
Order book
Open orders
Portfolio
Cash
Positions
Everything else is immutable events.
This minimizes copying and cache misses.
8. Avoid recalculating indicators
Strategies often dominate runtime.
Instead of:
every event:
EMA(history)
maintain rolling state:
EMA_new =
alpha * price +
(1-alpha) * EMA_old
The same applies to:
- VWAP
- rolling variance
- ATR
- RSI
- moving averages
This reduces many updates from O(n) to O(1).
9. Process events in batches when possible
Many timestamps contain numerous updates.
Instead of:
quote
trade
quote
quote
trade
quote
group by timestamp:
Update book
Update trades
Run strategy once
unless your strategy depends on every intermediate state.
This often yields substantial speedups.
10. Make realism configurable
Not every simulation needs the same fidelity. For example:
| Mode | Fill model | Queue | Latency | Speed |
|---|
| Fast | Touch = fill | None | None | Very high |
| Medium | Volume-aware | Approximate | Fixed | High |
| Detailed | Queue model | Yes | Stochastic | Moderate |
| Exchange replay | Full matching | Exact | Historical | Lower |
This lets researchers iterate quickly and reserve the most detailed simulation for validating promising strategies.
11. Separate exchange logic from portfolio logic
A clean architecture might look like:
Market Feed
│
▼
Exchange Simulator
│
fills / cancels
▼
Execution Engine
│
▼
Portfolio
│
▼
Strategy
The portfolio should not know how fills were generated, and the strategy should not know how the exchange matched orders. This separation makes it easier to swap in different market models or venue implementations.
Performance considerations
Well-designed event-driven backtesters remain surprisingly fast because most work happens only when something changes. Some practical techniques include:
- Using contiguous arrays (or columnar storage) for market data instead of many small objects.
- Keeping hot-path objects (orders, positions, order book levels) mutable to avoid excessive allocations.
- Maintaining incremental state (book levels, indicators, P&L) rather than recomputing from history.
- Using a priority queue for future events (latency, acknowledgements, scheduled actions).
- Parallelizing across independent simulations (different parameter sets or days) rather than within a single event stream, since a single market timeline is inherently sequential.
A useful mental model is to think of the backtester as a discrete-event simulator rather than a loop over bars or ticks. Each market update, order acknowledgment, fill, cancel, or timer becomes an event in a time-ordered queue. That naturally captures asynchronous behavior while allowing you to dial the fidelity—from simple fill heuristics to venue-specific matching—without changing the overall architecture.
