Adobe built an internal compute platform to schedule generative AI workloads across multiple Kubernetes clusters and manage shared GPU capacity.
2
Snapshotting lets the platform preempt idle interactive development sessions without losing their configured environments.
3
An automated recovery system monitors jobs and hardware, recycles unhealthy nodes, and resumes long-running training after failures.
Summary
Sahil Khanna describes how Adobe built a compute platform for Firefly's generative AI workloads. The platform supports hundreds of distributed jobs and thousands of GPU nodes, where demand exceeds available capacity and hardware failures are common. Adobe built an internal scheduler to place jobs across clusters, manage quotas, reserve capacity for critical workloads, and preempt lower-priority work. Interactive development sessions created a separate problem because they contain user-configured state and are often idle. The team added environment snapshotting so sessions can be stopped and resumed without forcing engineers to rebuild them. A reclaim policy recovers idle GPU capacity. For reliability, an automated recovery system monitors job progress, node health, and failure signals, then recycles hardware and resumes training. Khanna also explains the platform architecture, including its APIs, global scheduler, per-cluster managers, node agents, port agents, and storage in S3.
Firefly requires a shared platform for large generative AI workloads
Adobe's Firefly organization supports applications that generate images, videos, and audio. Khanna's team built a compute platform to simplify access to GPU instances and improve developer productivity. The platform runs hundreds of distributed jobs and manages thousands of GPU nodes. Training and experimentation require large machines such as H800 and H200 systems, so research engineers compete for limited capacity. Adobe reserves capacity ahead of time, then has to allocate it fairly across projects while keeping the reserved resources in use.
The scheduler manages capacity across clusters and workload priorities
Adobe needed to schedule jobs across multiple clusters because one cluster could not support the required scale. Critical projects need dedicated capacity, while noncritical projects should use capacity when it is free. Khanna says the team built an internal scheduler because available open source options did not provide the combination they needed: a state of the world shared across clusters, quota management, and preemption. The scheduler can reclaim capacity from lower-priority work when a critical job needs it.
Interactive sessions need preemption without destroying user state
Development users access GPU nodes through interactive sessions and Jupyter notebooks. These sessions are often idle, but they contain custom environments that users may have spent hours configuring. Stopping a session outright would make preemption disruptive. Adobe added snapshotting, which saves the environment state before a job stops. When the user starts another job, the session can resume from the saved state within seconds instead of requiring the environment to be configured again.
Reclaim policies recover idle GPUs from development workloads
Snapshotting made it practical to preempt interactive sessions, and Adobe added policies that reclaim resources when jobs are idle. The reclaimed GPUs can then be assigned to applications that need them. This allows the same shared infrastructure to support production training and development workflows. Khanna presents reclaiming idle capacity as part of the platform's effort to improve overall GPU use without giving development users a poor restart experience.
Hardware failures reduce utilization at distributed training scale
Cloud GPU infrastructure frequently encounters hardware failures, configuration problems, overheating, and connectivity issues. These failures lengthen training, interrupt jobs, and leave GPUs idle while systems recover. Adobe therefore built an internal automatic recovery system. A central component tracks job progress, collects node health information, and processes failure signals from other parts of the platform. It decides how to recover each failure and resume training.
Automatic recovery matters because training runs last days or weeks
Generative model training jobs can run for days or weeks, so a failure cannot require engineers to restart from the beginning. Adobe's recovery system recycles bad hardware and resumes training automatically. Khanna says this helps jobs complete successfully and recover faster from hardware and connectivity problems. During the question period, he adds that training runs commonly encounter several incidents per day, although he does not provide a percentage or a complete failure statistic.
The architecture separates global placement from cluster execution
Users interact with the platform through a UI, Python SDK, and CLI. An API and event layer records requirements and triggers actions. The global scheduler understands running jobs, available nodes, and project quotas, then decides where a job should run. A cluster manager for each Kubernetes cluster turns that decision into actual pods and infrastructure. Node and pod agents collect health and workload data and perform actions such as taking snapshots, capturing metrics, and running tracing. Snapshot data is stored in S3.
Jobs usually stay within one cluster for network performance
Adobe uses multiple clusters to provide enough total capacity, but Khanna says the team tries to fit each individual training job inside one cluster and, especially, one availability zone. This avoids spreading a job across clusters and preserves the fastest available connectivity. He describes bin packing as the way the platform places jobs under this constraint. The approach lets Adobe use many clusters for aggregate scale without requiring each training run to span them.
"We try to fit a single job within a cluster so that we can especially within a zone use the fastest connectivity possible and have the maximum throughput."21:30
Who should watch
You run GPU training jobs across shared clusters and need scheduling, quotas, or preemption.
Interactive development sessions consume expensive GPUs but cannot be stopped safely without preserving user state.
Your distributed training jobs lose time to unhealthy hardware, overheating, or connectivity failures, and you are considering automatic recovery.
