# MLOps from the Perspective of an SRE

Neeran Gul, Benevolent AI | MLOps Coffee Sessions | Episode 8 | 57:30
Hosted by David Aponte

Source: https://www.youtube.com/watch?v=CQhgP-E1jhY
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/mlops-from-the-perspective-of-an-sre
Published: 2020-09-08
Tags: data-engineering, monitoring, reliability

## TL;DR
- An SRE's main concern is the reliability and resilience of the infrastructure on which business workloads run.
- For machine learning, resilience means helping long-running jobs finish, resume after interruptions, and tolerate network failures through checkpoints, retries, timeouts, and circuit breakers.
- MLOps teams should choose infrastructure and data stores around workload needs, while treating security, monitoring, capacity planning, and cost as shared engineering responsibilities.

## Summary
Neeran Gul describes SRE work from his experience managing the SRE team at Benevolent AI. The team supports research workloads by keeping infrastructure available, reducing developer blockers, and making pipeline runs resilient from start to finish. Gul explains that SRE is usually a secondary discipline for experienced software engineers who can code and understand algorithms, data structures, system design, and architecture. In MLOps, he treats machine learning workloads largely as long-running batch jobs. They need checkpoints, retries, timeouts, and circuit breakers so they can resume after failures. Infrastructure choices depend on the workload, including data access patterns, storage needs, compute requirements, and whether jobs are linear or parallel. Gul also discusses security responsibilities for developers, monitoring at several levels, budget limits, incident response, and capacity planning. He encourages MLOps practitioners to contribute to open source projects, including experimental ones.

## Key ideas
### SRE work keeps the company's pipelines running without blocking developers
[00:36](https://www.youtube.com/watch?v=CQhgP-E1jhY&t=36s)
Neeran Gul says he manages and leads the SRE team at Benevolent AI. Its aim is to provide uptime and resilience to failures, so a pipeline can run from start to finish without interruptions. The team also optimizes infrastructure to remove blockers for developers. At the company's stage of growth, the work includes release management and change management, along with day-to-day technical operations. Gul describes the role as broad because responsibilities are later spread across more specialized teams as the business grows.

### Stored procedures taught Gul how business logic can live inside the database
[09:08](https://www.youtube.com/watch?v=CQhgP-E1jhY&t=548s)
Gul explains that stored procedures put the application's business logic inside SQL functions. The application performs basic create, update, and delete operations, then calls the database to execute the logic. This differs from using an object-relational mapper, where application objects and their dependencies hold more of the logic. He worked with large PostgreSQL functions containing complicated joins and other operations. The experience was difficult, but it gave him a strong understanding of databases and prepared him for later operations work.

### Operations has different meanings depending on the company and location
[15:16](https://www.youtube.com/watch?v=CQhgP-E1jhY&t=916s)
Gul says operations can refer to IT, DevOps, or a mixture of the two, depending on the organization and its location. In some startups, DevOps is the operations function and IT is separate. In London, he sees DevOps and IT as more clearly separated, while other parts of Europe use a mixed model. IT often handles office equipment, networking, firewalls, and access. There is still overlap, and Gul notes that historical responsibilities can place physical data-center work on an SRE even though it is not normally part of the role.

### SRE is an experienced software engineering discipline focused on reliability
[20:39](https://www.youtube.com/watch?v=CQhgP-E1jhY&t=1239s)
Gul agrees with the definition of SRE as operations work done by engineers with software expertise. He summarizes the concern as infrastructure reliability and resilience. He says people generally move into SRE after several years as software engineers and some exposure to operations or infrastructure. SRE interviews can include coding problems, algorithms, data structures, system design, and architecture. Experience matters because system design and architecture are learned through working on real projects, databases, and production systems.

### MLOps infrastructure covers a growing set of machine learning systems
[28:31](https://www.youtube.com/watch?v=CQhgP-E1jhY&t=1711s)
Gul defines MLOps as the infrastructure built to support machine learning. New libraries and tools keep appearing, and each can affect operations across the stack. He uses Kubeflow as an example because deploying it can involve an entire Kubernetes cluster and its surrounding ecosystem. Polyaxon is another example of a machine learning platform for Kubernetes. MLOps also includes the systems that hold and provide data, since machine learning depends on data. Gul says many of these practices existed inside analytics teams before they acquired the name MLOps.

### Resilient ML jobs should survive interruptions and resume work
[31:53](https://www.youtube.com/watch?v=CQhgP-E1jhY&t=1913s)
From an SRE perspective, Gul categorizes much machine learning work as batch jobs or long-running jobs. Resilience means allowing a job to run from start to finish without stopping. Since network problems can occur even with well-built infrastructure, jobs should be able to resume from where they stopped. Gul recommends operational additions such as checkpoints, circuit breakers, timeouts, and retries. These mechanisms help infrastructure support the workload without requiring every interruption to become a manual recovery exercise.

### Infrastructure choices should follow workload and data requirements
[34:30](https://www.youtube.com/watch?v=CQhgP-E1jhY&t=2070s)
Gul says SRE teams should support the technologies developers choose while advising on how to make them resilient. The team may change infrastructure when developers need a capability such as gRPC. For compute, the team must handle CPU and GPU demand, memory requirements, autoscaling, and limits. Resilience has a cost, both through extra technical debt and additional infrastructure spending. Data storage decisions depend on read and write patterns, concurrency, latency, relational needs, temporary storage, object storage, and data warehousing. Gul recommends different technologies for a linear CSV job and a large parallel workload, such as one using Spark and the Hadoop ecosystem.

### Security and monitoring belong to every engineer, not only SRE
[43:51](https://www.youtube.com/watch?v=CQhgP-E1jhY&t=2631s)
Gul says security work includes risk assessment, access removal after employees leave, and policies for people with different levels of access. Developers also have a responsibility to respect restricted data and choose maintainable dependencies. They need to know how to respond when secrets or passwords are accidentally committed. Monitoring gives SREs visibility into infrastructure, from pod and service status to memory failures, network traffic, storage, and team spending. It also supports incident response, budget decisions, and capacity planning, such as predicting when a storage volume will run out of space.

### Open source contributions should include experimental work
[55:04](https://www.youtube.com/watch?v=CQhgP-E1jhY&t=3304s)
Gul encourages MLOps practitioners to contribute to open source projects and bring their expertise to the systems they use. He appreciates libraries that cover a broad part of the stack and says MLOps needs people who understand its different areas. He also argues that experimental projects have value, even when they are not yet polished or widely exposed. His advice is aimed at people who build useful systems privately and could share them so others can use, improve, or combine them with related work.

## Notable quotes
- Neeran Gul: "SRE cares about resiliency and reliability." (21:04)
- Neeran Gul: "You want to put in checkpoints, you want to put in circuit breakers, you want to put in timeouts, you want to put in retrying." (33:21)
- Neeran Gul: "Without monitoring we cannot know what's going on." (50:08)
- Neeran Gul: "There's a cost to resiliency." (37:56)
- Neeran Gul: "Every engineer has that duty." (46:11)

## Tools & references mentioned
- Benevolent AI
- AWS
- Azure
- GCP
- Terraform
- Ansible
- PHP
- Python
- Django
- PostgreSQL
- Microsoft
- Yammer
- Docker
- Kubernetes
- Kubeflow
- Polyaxon
- gRPC
- Spark
- Hadoop
- S3
- BigQuery
- Redshift
- Prometheus
- Thanos

## Who should watch
- You work on ML systems and want to understand how an SRE evaluates reliability, compute, storage, monitoring, and operational cost.
- You are moving from software engineering or operations into SRE and want a practical description of the experience and skills involved.
- Your team runs long machine learning jobs, handles sensitive data, or relies on shared infrastructure that needs better failure recovery.

## Editor's note

Neeran Gul says SRE teams should support the technologies developers choose while advising on how to make them resilient. ZenML lets teams keep pipeline code unchanged while choosing an orchestrator, artifact store, and other infrastructure by configuration. That makes it easier to run the same workflow across a laptop, Kubernetes, Airflow, Kubeflow, or cloud services.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Machine Learning SRE](https://mlopstalks.com/talks/machine-learning-sre) (Niall Murphy, Microsoft Azure, 48:29)
- [MLOps in Practice: Common Challenges and Lessons Learned](https://mlopstalks.com/talks/mlops-in-practice-common-challenges-and-lessons-learned) (Marouen Hizaoui & Mo Basirati, Machine Learning Reply, 1:03:01)
- [MLOps: Isn't That Just DevOps?](https://mlopstalks.com/talks/mlops-isnt-that-just-devops) (Ryan Dawson, Seldon, 1:06:32)
- [The Godfather Of MLOps](https://mlopstalks.com/talks/the-godfather-of-mlops) (D. Sculley, Google, 51:25)
- [Doing MLOps](https://mlopstalks.com/talks/doing-mlops) (Noah Gift, Pragmatic AI Labs, 1:01:22)