Andrew Dye moved from low-level systems and custom silicon work at Microsoft into distributed training infrastructure at Meta, where systems engineering and machine learning met.
2
Large-scale training creates scheduling, fault-tolerance, observability, and coordination problems because jobs use many interconnected machines and workloads differ widely.
3
Flyte and Union aim to let teams move from experiments to production while preserving repeatability, caching, checkpoint recovery, and collaboration across engineering roles.
Summary
Andrew Dye describes moving from low-level systems engineering on Microsoft's HoloLens to distributed training infrastructure at Meta and then to Union AI. His career brought him close to machine learning without starting as an ML engineer. At Meta, he worked on distributed data parallel systems for Caffe2, which later informed PyTorch DDP, and on reliability for large training jobs. He explains why distributed training needs careful scheduling, checkpointing, observability, and analysis across workloads. The conversation then turns to Flyte and Union. Andrew sees orchestration as a shared working layer for systems engineers, ML engineers, data scientists, and MLOps practitioners. He argues that abstractions should be designed for a specific audience and should avoid exposing every lower-level control. Flyte provides strong typing, extensibility, caching, version management, and intra-task checkpointing, while Union Cloud reduces the infrastructure burden for teams that want to experiment without managing a cluster.
Low-level systems work can provide a practical route into machine learning
Andrew Dye started at Microsoft in 2014 as a low-level systems engineer working on HoloLens. His work involved custom silicon, a custom instruction set, real-time positional tracking, and communication between hardware and algorithms. He initially understood neither the layer above nor the layer below his component. Seeing a game respond to images, data, and computation helped him understand how hardware complexity enabled user-facing machine learning experiences. He says machine learning made mathematics less intimidating because he could represent concepts with code, although research proofs still challenge him.
Andrew learned ML infrastructure by combining systems experience with help from specialists
When Andrew joined Meta in 2017, engineers went through boot camp and a team-finding process. He wanted to learn more about machine learning and found a distributed training team that also contained systems problems he already understood. This gave him a middle ground from which he could grow. He learned by working with experts, trying things, and applying intuition. Andrew is honest that an introductory course such as Andrew Ng's Coursera course gave him only basic familiarity, while the work quickly involved model parallelism across tens of nodes and service-oriented infrastructure.
Distributed training turns resource scheduling into a coordination problem
Andrew describes Meta's training environment as a setting with limited GPU capacity, large jobs, small jobs, data placement concerns, and different network behaviors. A scheduler cannot simply wait until enough GPUs are free for a large job, because that wastes available resources. It may allow smaller jobs to run while waiting, then bring in the larger job when resources are available. That approach requires cooperation from the training stack so progress is not lost. He says the range and heterogeneity of workloads create cascading problems across scheduling, placement, networking, and execution.
Observability must cover patterns across jobs, not only one job's logs
Andrew says anticipating failures, instrumenting likely behaviors, and logging them are necessary for diagnosing distributed systems. ML makes this difficult because teams rapidly try and discard ideas, so it is unclear when to invest in tests or visibility. He recommends infrastructure that puts observability and failure information near the center of the system. Tabular logs help explain detailed states, while time-series data such as counters, latency averages, and standard deviations reveal patterns. Comparing jobs and examining outliers can expose network or data-center causes that a single workload log would miss.
Flyte gives different engineering roles a shared way to build pipelines
Andrew says the wider ML ecosystem brought him into contact with systems engineers, MLOps practitioners, ML engineers, and other roles working together. He sees Flyte's community as an example of this collaboration. Different teams can build different parts of an orchestration pipeline, such as data inputs or performance-sensitive training steps, and connect them through the tool. In his view, the tool becomes a medium for communication because teams can discuss their constraints and contributions in a shared system rather than treating each discipline as an isolated group.
Abstractions should hide complexity while keeping lower-level escape routes intentional
Andrew warns that a clean abstraction can gradually accumulate exceptions. Each new requirement exposes another lower-level knob until users face a confusing collection of controls and the abstraction becomes frustrating. He says teams should decide who an abstraction targets, which capabilities it exposes, and when users may work at another layer. There is no single abstraction that fits every use case. Flyte takes a strong position on typing while remaining flexible about how tasks are performed through plugins, including integrations with existing or future tools.
Union focuses on moving workloads from local experiments to managed production
Andrew describes Union ML as a higher-level layer on Flyte and Flytekit for instantiating models and inference servers quickly. Union also works to remove infrastructure management from teams that want to experiment without deploying and configuring a production-grade Flyte cluster themselves. Union Cloud provides a running environment where users can execute workloads without managing the underlying system. Andrew says Flyte was designed to support a path from local scripts and notebooks through increasingly complex pipelines to production without forcing a rewrite in a different tool.
Large training jobs need checkpoint-aware fault tolerance
At Meta, jobs running across tens or hundreds of machines created problems during failures and fleet updates. Updating firmware or drivers could interrupt the same distributed job many times unless the system understood which machines were connected to that job. Andrew describes coordinating interruptions around saved checkpoints so GPU work was not discarded. Flyte addresses a related need through intra-task checkpointing. A Python API writes a file to a location, and Flyte provides that file again when the task restarts. PyTorch integrations can build on this mechanism.
