For most analytics-heavy products in 2026, I'd avoid treating this as an "ETL vs. streaming" decision. The best architecture is usually a combination, with complexity introduced only where it creates measurable value.
A practical default looks like this:
Applications
│
Events (clicks, API calls, business events)
│
Event bus (Kafka / Redpanda / cloud equivalent)
│
┌──────────────┬────────────────┐
│ │ │
Real-time Object storage Operational consumers
analytics (raw event log) (notifications, fraud, etc.)
│ │
│ ETL/ELT transforms
│ │
Warehouse/Lakehouse
│
dbt or SQL models
│
Dashboards / ML / Product analytics
Here's how I'd think about each piece:
| Component | Recommendation |
|---|
| Event collection | Instrument everything with well-defined events and schemas. |
| Storage | Keep immutable raw events in object storage. Storage is cheap; historical replay is invaluable. |
| Warehouse | Use a modern warehouse/lakehouse as the source of truth for analytics. |
| Transformations | Favor SQL + dbt-style transformations over complex visual ETL workflows whenever possible. |
| Streaming | Use only where low latency actually matters (seconds instead of minutes). |
| BI | Query the warehouse rather than building separate reporting databases. |
When I'd choose streaming
Streaming is worth the operational cost if you need things like:
- live dashboards (<5 s latency)
- personalization while users are active
- fraud detection
- anomaly detection
- recommendation engines
- operational alerts
If your PMs are perfectly happy with dashboards that refresh every 15 minutes, streaming often isn't buying much.
When simpler batch pipelines win
Many SaaS products overestimate their need for real-time analytics.
If your workload is:
- customer reporting
- growth metrics
- retention analysis
- cohort analysis
- executive dashboards
- experimentation analysis
then scheduled ELT every 5–30 minutes is usually dramatically simpler and almost indistinguishable from real time for decision-making.
My preferred stack
A stack I'd be comfortable building today would look something like:
- Event collection: OpenTelemetry or a product analytics SDK with strong schema governance
- Event bus: Kafka-compatible platform (managed if possible)
- Object storage: S3-compatible bucket
- Warehouse/lakehouse: Snowflake, BigQuery, ClickHouse, or DuckDB-based lakehouse depending on scale and budget
- Transformations: dbt
- Orchestration: Dagster or Airflow (Dagster if starting fresh)
- BI: Metabase, Superset, Looker, or similar
Principles I'd optimize for
- Events are immutable. Never rewrite history.
- Version schemas. Analytics breaks when event definitions drift.
- One source of truth. Avoid duplicate metric definitions across systems.
- Replayability. You should be able to rebuild your warehouse from raw events.
- Start with batch. Add streaming only for workloads that have a concrete latency requirement.
If I were starting a new product in 2026
I'd build a hybrid pipeline:
- Events are published once.
- They're written immediately to durable storage.
- The same events feed both a warehouse (batch/ELT) and a streaming layer for the few use cases that require real-time processing.
- Business metrics are computed in the warehouse, not inside application code.
That approach scales from thousands to billions of events while keeping operational complexity manageable. It also lets you add real-time capabilities incrementally instead of committing your entire analytics stack to streaming from day one.
