Prefect

Prefect makes Python workflows resilient with two decorators: @flow and @task. No DAG files, no operators, no boilerplate — write normal Python functions and Prefect handles retries, scheduling, logging, and observability. It's what Airflow would be if built today.

Dynamic task creation at runtime is the killer feature over Airflow's static DAG parsing. Your ML pipeline can branch based on data, not just config. The hybrid execution model (local code, cloud orchestration) keeps your data on your infrastructure. Compared to Airflow (static DAGs, more ecosystem), Prefect is simpler. Compared to Dagster (asset-centric), Prefect is more flexible. Compared to Temporal (durable execution), Prefect is Python-only but easier.

Use this when you need resilient Python workflows without the Airflow learning curve — data pipelines, ML training, ETL. Skip this if you need multi-language support or your workflows are simple enough for cron.

The catch: Prefect Cloud is the path of least resistance, but it's a paid service. Self-hosting Prefect Server is possible but less documented. And the v1 to v2 migration broke a lot of workflows — check which version tutorials target. Apache 2.0 license.