An SRE's main concern is the reliability and resilience of the infrastructure on which business workloads run.
2
For machine learning, resilience means helping long-running jobs finish, resume after interruptions, and tolerate network failures through checkpoints, retries, timeouts, and circuit breakers.
3
MLOps teams should choose infrastructure and data stores around workload needs, while treating security, monitoring, capacity planning, and cost as shared engineering responsibilities.
Summary
Neeran Gul describes SRE work from his experience managing the SRE team at Benevolent AI. The team supports research workloads by keeping infrastructure available, reducing developer blockers, and making pipeline runs resilient from start to finish. Gul explains that SRE is usually a secondary discipline for experienced software engineers who can code and understand algorithms, data structures, system design, and architecture. In MLOps, he treats machine learning workloads largely as long-running batch jobs. They need checkpoints, retries, timeouts, and circuit breakers so they can resume after failures. Infrastructure choices depend on the workload, including data access patterns, storage needs, compute requirements, and whether jobs are linear or parallel. Gul also discusses security responsibilities for developers, monitoring at several levels, budget limits, incident response, and capacity planning. He encourages MLOps practitioners to contribute to open source projects, including experimental ones.
SRE work keeps the company's pipelines running without blocking developers
Neeran Gul says he manages and leads the SRE team at Benevolent AI. Its aim is to provide uptime and resilience to failures, so a pipeline can run from start to finish without interruptions. The team also optimizes infrastructure to remove blockers for developers. At the company's stage of growth, the work includes release management and change management, along with day-to-day technical operations. Gul describes the role as broad because responsibilities are later spread across more specialized teams as the business grows.
Stored procedures taught Gul how business logic can live inside the database
Gul explains that stored procedures put the application's business logic inside SQL functions. The application performs basic create, update, and delete operations, then calls the database to execute the logic. This differs from using an object-relational mapper, where application objects and their dependencies hold more of the logic. He worked with large PostgreSQL functions containing complicated joins and other operations. The experience was difficult, but it gave him a strong understanding of databases and prepared him for later operations work.
Operations has different meanings depending on the company and location
Gul says operations can refer to IT, DevOps, or a mixture of the two, depending on the organization and its location. In some startups, DevOps is the operations function and IT is separate. In London, he sees DevOps and IT as more clearly separated, while other parts of Europe use a mixed model. IT often handles office equipment, networking, firewalls, and access. There is still overlap, and Gul notes that historical responsibilities can place physical data-center work on an SRE even though it is not normally part of the role.
SRE is an experienced software engineering discipline focused on reliability
Gul agrees with the definition of SRE as operations work done by engineers with software expertise. He summarizes the concern as infrastructure reliability and resilience. He says people generally move into SRE after several years as software engineers and some exposure to operations or infrastructure. SRE interviews can include coding problems, algorithms, data structures, system design, and architecture. Experience matters because system design and architecture are learned through working on real projects, databases, and production systems.
MLOps infrastructure covers a growing set of machine learning systems
Gul defines MLOps as the infrastructure built to support machine learning. New libraries and tools keep appearing, and each can affect operations across the stack. He uses Kubeflow as an example because deploying it can involve an entire Kubernetes cluster and its surrounding ecosystem. Polyaxon is another example of a machine learning platform for Kubernetes. MLOps also includes the systems that hold and provide data, since machine learning depends on data. Gul says many of these practices existed inside analytics teams before they acquired the name MLOps.
Resilient ML jobs should survive interruptions and resume work
From an SRE perspective, Gul categorizes much machine learning work as batch jobs or long-running jobs. Resilience means allowing a job to run from start to finish without stopping. Since network problems can occur even with well-built infrastructure, jobs should be able to resume from where they stopped. Gul recommends operational additions such as checkpoints, circuit breakers, timeouts, and retries. These mechanisms help infrastructure support the workload without requiring every interruption to become a manual recovery exercise.
Infrastructure choices should follow workload and data requirements
Gul says SRE teams should support the technologies developers choose while advising on how to make them resilient. The team may change infrastructure when developers need a capability such as gRPC. For compute, the team must handle CPU and GPU demand, memory requirements, autoscaling, and limits. Resilience has a cost, both through extra technical debt and additional infrastructure spending. Data storage decisions depend on read and write patterns, concurrency, latency, relational needs, temporary storage, object storage, and data warehousing. Gul recommends different technologies for a linear CSV job and a large parallel workload, such as one using Spark and the Hadoop ecosystem.
Security and monitoring belong to every engineer, not only SRE
Gul says security work includes risk assessment, access removal after employees leave, and policies for people with different levels of access. Developers also have a responsibility to respect restricted data and choose maintainable dependencies. They need to know how to respond when secrets or passwords are accidentally committed. Monitoring gives SREs visibility into infrastructure, from pod and service status to memory failures, network traffic, storage, and team spending. It also supports incident response, budget decisions, and capacity planning, such as predicting when a storage volume will run out of space.
Open source contributions should include experimental work
Gul encourages MLOps practitioners to contribute to open source projects and bring their expertise to the systems they use. He appreciates libraries that cover a broad part of the stack and says MLOps needs people who understand its different areas. He also argues that experimental projects have value, even when they are not yet polished or widely exposed. His advice is aimed at people who build useful systems privately and could share them so others can use, improve, or combine them with related work.
