Each replay uses a recreated ChatGPT interface so the placement can be read in its original conversational context. It is not a live or embedded chat.

1. ObservedJul 17, 2026Conversation1 turnsPlacementAfter turn 01 turns · After turn 0Replay chat
   Observed conversationJul 17, 2026 · interface recreation

   ChatGPTArchived view

   Designing a new data pipeline for training. Dagster, Prefect, or Airflow? We're dealing with both batch and streaming data. What's working well for you in 2026?

   If you're starting fresh in 2026, my default recommendation is:

   • Dagster for ML/data platform teams building modern data pipelines.
   • Prefect 3 if you want lightweight orchestration with minimal operational overhead.
   • Airflow if you're operating at enterprise scale and already have a significant Airflow ecosystem or need compatibility with existing tooling.

   The biggest shift over the past few years is that orchestration has become less about "running Python scripts on a schedule" and more about coordinating data assets, event-driven workflows, and external compute systems.

   Here's how I'd compare them.

   Category	Dagster	Prefect	Airflow
   ML workflows	Excellent	Very good	Good
   Batch ETL	Excellent	Excellent	Excellent
   Streaming orchestration	Good (coordinates streaming systems rather than replacing them)	Good	Fair
   Data asset model	Excellent	Moderate	Limited
   Local development	Excellent	Excellent	Fair
   UI	Excellent	Good	Mature but more operational
   Learning curve	Medium	Low	High
   Kubernetes	Excellent	Excellent	Excellent
   Ecosystem	Growing rapidly	Growing	Huge

   Dagster

   This is probably the strongest choice if you're building a modern ML platform.

   Strengths:

   • asset-centric instead of DAG-centric
   • first-class data lineage
   • partitions are excellent
   • great developer experience
   • testing is straightforward
   • integrates nicely with dbt, Spark, Snowflake, BigQuery, DuckDB, Ray, Kubernetes, etc.

   Example workflow:

   Kafka
      ↓
   Spark Streaming
      ↓
   Iceberg
      ↓
   Dagster asset
      ↓
   Feature engineering
      ↓
   Training
      ↓
   Evaluation
      ↓
   Model registry
   

   That asset model ends up matching how ML teams actually think.

   Instead of:

   run task A then B then C

   you define

   build customer_features

   which depends on

   transactions

   which depends on

   raw_events

   That becomes much easier to maintain over time.

   ---

   Prefect

   Prefect has become very pleasant if your goal is:

   "Just orchestrate my Python."

   Advantages:

   • almost no boilerplate
   • async support
   • dynamic workflows
   • nice retries
   • easy deployments
   • less operational complexity than Airflow

   If your pipelines are mostly Python functions and cloud jobs, it's hard to beat for simplicity.

   It's particularly attractive for smaller ML teams (roughly 5–20 engineers).

   ---

   Airflow

   Airflow is still everywhere.

   It still shines for:

   • thousands of scheduled jobs
   • enterprise governance
   • mature operational practices
   • massive plugin ecosystem
   • lots of engineers already know it

   Its weaknesses haven't changed much:

   • verbose
   • scheduler complexity
   • dynamic DAGs are less natural
   • weaker abstractions for ML assets

   I wouldn't choose Airflow today unless:

   • your company already runs Airflow
   • another team supports it
   • you need compatibility with an existing ecosystem

   ---

   Batch + Streaming

   One thing I'd avoid is trying to make your orchestrator be your streaming engine.

   Instead:

   Kafka
        │
        ▼
   Flink / Spark Streaming
        │
        ▼
   Iceberg / Delta
        │
        ▼
   Dagster or Prefect
        │
        ▼
   training
   evaluation
   feature generation
   backfills
   

   Streaming jobs should generally run continuously.

   The orchestrator should:

   • deploy them
   • monitor them
   • restart them
   • trigger downstream work
   • manage backfills

   rather than process every individual event.

   ---

   ML Training Pipelines

   For training specifically, modern stacks often look something like:

   Kafka
   ↓
   
   Flink
   
   ↓
   
   Iceberg
   
   ↓
   
   dbt
   
   ↓
   
   Dagster
   
   ↓
   
   Ray
   or
   Kubernetes Jobs
   
   ↓
   
   MLflow
   
   ↓
   
   Model Registry
   
   ↓
   
   Deployment
   

   Notice that the orchestrator coordinates work—it doesn't perform heavy computation itself.

   ---

   Scalability

   Approximate comfort zones:

   • Prefect: hundreds to a few thousand flows/day with relatively little operational effort.
   • Dagster: thousands of asset materializations and complex dependency graphs; scales well with proper infrastructure.
   • Airflow: proven at very large enterprise scale, including tens of thousands of scheduled tasks per day, though operating it can require more platform engineering effort.

   In practice, the execution engine (Spark, Ray, Kubernetes, Flink, etc.) is usually the limiting factor before the orchestrator is.

   What I'd choose today

   For a new ML training platform handling both batch and streaming:

   • Dagster as the orchestrator.
   • Kafka (or another streaming platform) plus Flink or Spark Structured Streaming for continuous ingestion.
   • Apache Iceberg or Delta Lake as the storage layer.
   • Ray or Kubernetes Jobs for distributed model training.
   • MLflow for experiment tracking and model registry.

   That combination keeps each component focused on what it does best: streaming systems process events continuously, the orchestrator manages dependencies and scheduling, and distributed compute handles large-scale training.

   A couple of questions would help narrow the recommendation further:

   • Roughly how many pipelines and training runs do you expect per day?
   • Are you deploying primarily on Kubernetes, or are you targeting a managed cloud platform like AWS, GCP, or Azure?

   

   Temporal TechnologiesAd

   Outgrowing Celery Jobs?Move Python jobs to durable workflows with retries, state, schedules, and visibility built in.