For high-frequency or tick-level research, I treat data validation as a first-class part of the research pipeline. It's often worth spending as much effort on data quality as on modeling because many "alpha signals" disappear once the data is cleaned correctly.
A typical pipeline looks like this:
| Stage | Checks | Typical Action |
|---|
| Raw ingestion | Missing files, corrupt records, duplicate files | Reject batch, alert |
| Schema validation | Timestamp format, field types, required columns | Fail ingestion |
| Tick validation | Invalid prices, sizes, crossed markets | Flag or remove |
| Sequence validation | Out-of-order timestamps, duplicate ticks | Sort/deduplicate |
| Market microstructure | Bid > ask, stale quotes, impossible spreads | Repair or discard |
| Corporate events | Splits, symbol changes | Adjust historical data |
| Session validation | Trades outside session, auction handling | Filter or label |
| Statistical anomaly detection | Extreme outliers, feed glitches | Flag for review |
| Research dataset generation | Aggregate into bars/features with QA metrics | Store clean version |
1. Timestamp integrity
I verify:
- monotonic timestamps per symbol
- no backwards time jumps
- exchange timestamp vs receive timestamp
- timezone consistency
- daylight savings handling
- nanosecond precision preserved
Example checks:
timestamp[i] >= timestamp[i-1]
Measure
- duplicate timestamps
- clock drift
- feed latency
2. Duplicate detection
Duplicates are surprisingly common.
Examples:
Same
- timestamp
- price
- size
- exchange
- trade ID
appearing multiple times.
Pipeline:
- exact duplicate removal
- trade-ID deduplication
- quote deduplication
Keep statistics such as
duplicate_rate = duplicates / total_ticks
A spike often indicates feed problems.
3. Price sanity
Reject
price <= 0
Reject
NaN
Inf
Flag
price changes > X%
within Y milliseconds
These are often busted trades or feed glitches.
4. Bid/ask consistency
For quote data:
Reject
bid > ask
Flag
spread > threshold
Example
spread > 10 × rolling median
This usually indicates stale quotes.
5. Trade-through checks
Verify execution prices satisfy market conditions.
For example
trade << bid
trade >> ask
may indicate
- delayed quote feed
- incorrect timestamps
- bad prints
Rather than deleting them immediately, I often label these for downstream filtering because quote/trade synchronization issues can produce false positives.
6. Sequence numbers
If the exchange provides sequence numbers:
Check
missing sequences
duplicate sequences
out-of-order sequences
This catches packet loss.
7. Volume validation
Reject
size <= 0
Flag
size >> historical percentile
Example
>99.99 percentile
Large trades aren't necessarily wrong but deserve inspection.
8. Auction handling
Opening
Closing
Intraday auctions
should usually be tagged separately.
Many intraday signals are distorted by auction prints.
9. Session filtering
Remove or separately label
- pre-market
- after-hours
- holidays
- half-days
Many microstructure signals only make sense during regular trading hours.
10. Corporate action adjustment
Adjust for
- stock splits
- reverse splits
- symbol changes
- ticker mergers
Otherwise you'll observe apparent 50% overnight moves that are purely mechanical.
11. Outlier detection
Use rolling statistics.
Example
z = (price - rolling_median) / rolling_MAD
If
|z| > 10
Flag it.
Median and median absolute deviation (MAD) are generally more robust than mean and standard deviation for tick data.
12. Quote staleness
Detect
same quote
for several seconds
This often means
- exchange outage
- frozen feed
- disconnected venue
Signals based on spread or imbalance become unreliable during these periods.
13. Locked/crossed markets
Measure frequency of
bid == ask
bid > ask
Some exchanges legitimately produce locked markets briefly.
Persistent crossed markets usually indicate data issues.
14. Trade direction consistency
If using aggressor-side classification (e.g., Lee–Ready), compare classifications against available exchange-provided aggressor flags where possible. Large disagreements may indicate synchronization or timestamp issues.
15. Feature-level validation
After feature construction, verify:
- no NaNs
- no infinities
- expected ranges
- stable distributions
For example:
imbalance ∈ [-1,1]
spread >= 0
volatility >= 0
Unexpected values often indicate upstream problems.
16. Distribution monitoring
For every symbol and day, compute summary metrics such as:
- tick count
- average spread
- median spread
- trade count
- quote count
- average trade size
- return volatility
- duplicate rate
- invalid quote rate
Compare these to rolling historical baselines. Sudden shifts can reveal feed changes or parsing errors before they affect research.
17. Cross-source validation
When multiple feeds are available:
- compare prices across vendors
- compare daily volume
- compare OHLC values
- compare VWAP
- compare corporate action adjustments
Differences beyond defined tolerances are flagged for investigation.
18. Research-specific leakage checks
Before fitting models, I verify that:
- features are computed only from information available up to the prediction timestamp
- labels are generated strictly from future data
- joins between trades, quotes, and reference data cannot introduce look-ahead bias
- train/validation/test splits respect time order
Many apparent "data quality" issues are actually subtle forms of data leakage.
Example automated pipeline
Raw feed
↓
Schema validation
↓
Timestamp validation
↓
Deduplication
↓
Price/size validation
↓
Quote consistency checks
↓
Session & auction labeling
↓
Corporate action adjustment
↓
Outlier & staleness detection
↓
Feature generation
↓
Feature QA
↓
Dataset versioning
One operational practice that pays off is to make the pipeline fail loudly. Instead of silently dropping bad records, produce a daily QA report with metrics such as duplicate rate, invalid quote rate, crossed-market rate, stale-quote percentage, missing sequence counts, and outlier counts. Those metrics can be tracked over time and often provide early warning of feed degradation or exchange-side changes before they contaminate research results.
