Workflow orchestration handles retries, timeouts, validation, notifications, and dependencies that would otherwise make pipeline maintenance dominated by failure handling.
2
Prefect Orion makes workflows more dynamic by using ordinary Python control flow, passing task results directly, and creating the execution graph during runtime.
3
Pandera, Tune, and Evidently can be combined with Prefect to validate data, run parallel hyperparameter searches, and stop or notify on detected drift.
Summary
Kevin Kho introduces workflow orchestration through a machine learning pipeline that pulls data from a database and API, transforms it, trains a model, records artifacts, and deploys the result. He explains how retries, timeouts, malformed data, failed APIs, and non-converging models create a large amount of failure-handling code, which he calls negative engineering. He then demonstrates Prefect Orion, the upcoming 2.0 version, with a small Dogecoin monitoring workflow. Orion wraps functions as tasks, adds retry settings, tracks states, passes Python objects between tasks, and allows ordinary if statements and loops. The larger example combines Pandera for data validation, Tune for hyperparameter search, Dask for parallel execution, and Evidently for drift detection. The presentation also shows Orion's local API, SQLite-backed state, and dashboard. Kevin is open about the transition from Prefect 1.0 to Orion and about some integrations still being worked out.
Pipeline failures create more maintenance work than the model code
Kevin describes a typical machine learning pipeline that combines database data, API data, Pandas transformations, model training, artifact storage, and deployment. Each step introduces decisions about what happens after failure. An API call may need retries, a database may be temporarily unavailable, and a model may run too long or fail to converge. Bad data can also pass technical checks while causing downstream problems. Kevin calls the explicit retry, timeout, notification, and alternate-path code used to handle these cases "negative engineering." He says teams can end up spending more time on this work than on the original model or business problem.
Prefect Orion removes the requirement that workflows behave like fixed DAGs
Kevin explains that Prefect 1.0 and many orchestration systems are built around a directed acyclic graph whose structure is registered before execution. He says this can constrain workflows and force workarounds. Prefect Orion, presented as Prefect 2.0, makes the workflow experience more like ordinary Python. It supports dynamic workflows, tasks added during execution, and first-class subflows. A section of a workflow can be retried without rebuilding the entire fixed graph. Kevin also notes that Orion was in technical preview at the time, while Prefect 1.0 was the production option.
A small polling script becomes a managed workflow with little code
Kevin uses a Dogecoin price monitor to show the difference between a plain Python loop and Prefect. The script calls the CoinDesk API, examines minute-level prices from the previous ten minutes, calculates whether the drop exceeds a threshold, and sends a Slack alert. Running it in an infinite loop on an EC2 instance works until an API or notification failure interrupts the process. With Prefect, he marks the API call as a task and configures three tries with a ten-second retry delay. He wraps the overall function as a flow, allowing Prefect to track the run and its task states.
Orion passes ordinary Python values while building dependencies from task calls
In the Dogecoin example, the API task returns a Pandas data frame and the detection task receives it directly. Kevin contrasts this with systems that require intermediate persistence before downstream tasks can consume data. Prefect infers that the data-fetch task is upstream because its result is passed into the detector. Orion also lets the code resolve a task future into a normal Python object, then use a regular if statement. This allows later tasks to be submitted conditionally during execution. The resulting graph shows the task dependencies that Prefect can observe, while native Python sections that are not tasks do not appear as separate dashboard nodes.
Kevin changes the example from one coin to Dogecoin, Bitcoin, and Ethereum, then adds a Dask task runner. The loop remains ordinary Python, while Prefect submits the repeated work through Dask. He describes the same pattern for a model-training workflow where multiple hyperparameter configurations can run in parallel, including on the local machine. He also discusses larger data systems. Prefect 1.0 could persist intermediate task results so a failed downstream task could restart from an earlier successful point, but Kevin warns that persisting large data frames repeatedly can be expensive and may need to be disabled explicitly.
Pandera turns data assumptions into checks with useful failure cases
Kevin introduces Pandera as a validation library mainly for Pandas data frames. His example defines rules for columns, including ranges, nullability, type coercion, and relationships between columns. One feature must fall between two other features, so the rule operates at the data-frame level rather than on a single column. Validation returns the data frame when checks pass and reports failing values and indexes when they do not. Kevin points out that lazy validation is needed when the user wants all failures collected before an error is raised. He also shows decorators for checking function inputs and outputs.
Tune expresses grid, random, and optimization search spaces
Kevin presents Tune as a library for hyperparameter tuning. Its search-space objects can describe grids, random values, and choices. A grid over two values combined with another grid produces every combination, while random expressions can be sampled with a seed. He says unsampled expressions can also be passed to an optimizer that chooses values during the search. Search spaces can be added or multiplied to combine them, and Tune can create models from a search space. The example uses scikit-learn models and can partition the search by a column such as gender. Tune can submit combinations to Dask or Spark.
Evidently can turn drift reports into a pipeline decision
Kevin uses Evidently AI to compare baseline data with new data. A report can show distribution changes for a target or individual features, but he says generating dashboards is less practical inside a recurring pipeline. Instead, the workflow can calculate the underlying JSON profile and read the specific metric it needs. In his example, the pipeline counts drifted features and compares that count with a threshold. If the count is too high, it sends a Slack alert and avoids making predictions. Otherwise, it loads the saved model, produces predictions, and writes the result to a Parquet file. Prefect also validates typed parameters such as the drift threshold.
