# Open Source and Fast Decision Making

Rob Hirschfeld, RackN | MLOps Podcast | Episode 164 | 1:00:02
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=ysv0gFUU7AE
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/open-source-and-fast-decision-making
Published: 2023-07-04
Tags: deployment, open-source, security, workflows

## TL;DR
- Rob Hirschfeld argues that operations has gained layers of services without solving the underlying problems of repeatability, access control, and shared data.
- He says teams should exercise complete automation across environments instead of removing unfamiliar steps, because repeated use exposes bugs and creates reusable fixes.
- He recommends immutable, version-controlled artifacts and reliable reset processes so teams can rebuild infrastructure instead of guessing which patches have been applied.

## Summary
Rob Hirschfeld and Demetrios Brinkmann discuss what has changed in infrastructure since Hirschfeld and Dave McCrory worked on early cloud systems. Hirschfeld says the industry has added services and microservices while making limited progress on operations, security, standards, and shared sources of truth. SaaS removes operational work from customers, though it also reduces control over data and automation. Hirschfeld's answer is to keep automation versioned, repeatable, and exercised across environments. He explains this through RackN's Digital Rebar, which builds infrastructure pipelines from reusable operations across cloud, virtual machines, and bare metal. He also argues for immutable artifacts and regular system resets, since rebuilding a known state can be faster and safer than applying fragile, partial patches. The discussion connects these ideas to machine learning infrastructure, where data is harder to change but access and replication can still be automated and inspected.

## Key ideas
### Cloud technology and the cloud business model became confused
[09:00](https://www.youtube.com/watch?v=ysv0gFUU7AE&t=540s)
Rob Hirschfeld says the early work he and Dave McCrory did involved renting infrastructure and delivering applications over the internet. He distinguishes that business model from the technology he associates with cloud, which is API-driven, elastic, and dynamic infrastructure. In 1999, their company needed several physical servers for every customer, so they used an early ESX beta to put the stack on one server. They then wrote patents around managing virtual machines across multiple machines and providing an API for infrastructure managed by another company. Hirschfeld says Amazon later found the right business model, while his team treated the system as a development and test platform and did not monetize it effectively.

### Operations has accumulated complexity without enough improvement
[09:00](https://www.youtube.com/watch?v=ysv0gFUU7AE&t=540s)
Hirschfeld says many assumptions about the cloud have changed less than people think. The operational work behind infrastructure has not improved as much as the services built on top of it. Microservices and narrowly focused SaaS products have increased the number of independent systems that teams must use and maintain. He describes information spread across many applications, with each new service potentially receiving broad access for a narrow purpose. He is especially concerned about tools that can scan repositories, Slack, or drives without tightly limited permissions. The industry also keeps building products with their own data stores and sources of truth, which makes sharing information and maintaining control harder.

### Security and access control remain difficult because standards are weak
[14:05](https://www.youtube.com/watch?v=ysv0gFUU7AE&t=845s)
When Demetrios asks how teams should limit access, Hirschfeld admits that he does not have a complete answer. He says the industry has avoided the problem because good security and access-control experiences are hard to build. Kubernetes ingress controllers provide some movement toward standards, but they do not solve the wider issue. Single sign-on is also inconsistent, and some providers use it to collect information about users' habits and behavior. Hirschfeld argues that products should make it easier for operators to keep data local and retain control. That may require accepting slower systems, shared protocols, and products that do not assume they are the only source of truth.

### A standard can multiply variations when vendors implement it differently
[18:08](https://www.youtube.com/watch?v=ysv0gFUU7AE&t=1088s)
Hirschfeld uses Redfish as an example of the limits of standardization. HP helped establish Redfish as an open protocol and formed a vendor consortium for server out-of-band management, replacing older and fragmented approaches such as IPMI. The result created common ground, but each vendor still added its own variation. Teams therefore cannot assume that management tooling will work identically across Dell and HP servers. Time also changes the problem. Operators need to know which protocol version they use, whether a security bug was fixed, and which systems in their fleet have received the relevant patch. Hirschfeld says this operational work is valuable, but it remains work that teams often hand to SaaS providers.

### Automation becomes reusable when teams keep the full system under exercise
[26:21](https://www.youtube.com/watch?v=ysv0gFUU7AE&t=1581s)
Hirschfeld describes installing OpenStack or Hadoop on successive customer sites with an automated process. Each installation improved the team's knowledge, but the first ten sites became unique because there was no way to apply later fixes to earlier installations. He calls that a failure of software, because a software provider should be able to move customers forward together. His proposed remedy is to exercise the same automation repeatedly across environments. If ten customers run nearly the same automation, a bug found at one site is likely to exist elsewhere, and the fix can benefit the group. This only works when users retain versioned, reusable automation instead of copying it and removing unfamiliar parts.

### Immutable artifacts can make resets faster than patching
[35:54](https://www.youtube.com/watch?v=ysv0gFUU7AE&t=2154s)
Hirschfeld initially resisted rebuilding whole servers from immutable artifacts because he expected that process to be slow and risky. His colleague argued for destroying a server and recreating it from a known artifact. After adopting the practice, Hirschfeld found that a full reset could be faster than applying surgical patches. Package updates often pull from multiple repositories, introduce unexpected changes, and fail in fragile ways. A prepared artifact can be built once, cloned across a fleet, and used to restore a known state. Hirschfeld says the prerequisite is reliable automation. If operators do not trust the reset process, they should fix that reliability problem before depending on immutability.

### Regular resets expose whether infrastructure can recover
[39:38](https://www.youtube.com/watch?v=ysv0gFUU7AE&t=2378s)
Hirschfeld says teams should become comfortable turning systems over instead of celebrating servers that have run unchanged for a year. He proposes that infrastructure should be capable of cycling every 30 days or faster, whether it runs in the cloud or on premises. That practice keeps patching and refreshing processes active and gives teams a way to respond to an emergency update. He connects this to resilience: an operator should be able to queue a fix and roll it out rather than spend months discovering, patching, and repairing systems. Legacy code is not automatically the problem. The greater danger is code that nobody can rebuild, test, patch, or upgrade.

### Digital Rebar exposes complex infrastructure through reusable workflows
[46:11](https://www.youtube.com/watch?v=ysv0gFUU7AE&t=2771s)
Hirschfeld describes RackN's Digital Rebar as an infrastructure workflow engine. It sequences operations such as Terraform, Ansible, shell scripts, and API calls into version-managed pipelines that move infrastructure toward a desired state. The same pipeline can abstract over cloud, virtual machines, and bare metal while presenting a simpler API to a developer or data scientist who only wants a usable machine-learning cluster. Digital Rebar also supports smaller work orders, such as running a security scan without changing the system, and it keeps shared state available to other tools. Customers download immutable automation artifacts and control when they adopt new versions, so they retain ownership of their infrastructure and data.

## Notable quotes
- Rob Hirschfeld: "The antidote for complexity is exercise." (26:21)
- Rob Hirschfeld: "If you have systems that haven't been reset in 30 days, you should actually be like, 'Oh wait, all my servers are too old.'" (40:21)
- Rob Hirschfeld: "Immutability doesn't just mean that I have a whole bunch of stuff sitting in a locked vault." (43:19)
- Rob Hirschfeld: "The legacy is the lack of repeatability in the system." (44:00)
- Rob Hirschfeld: "We might be entering an era where there is the possibility of a 10x operator." (58:04)

## Tools & references mentioned
- RackN
- Digital Rebar
- Dave McCrory
- Dell
- Amazon
- VMware
- ESX
- Diane Greene
- OpenStack Foundation
- Redfish
- HP
- IPMI
- OpenStack
- Hadoop
- Terraform
- Ansible
- Chef
- ChatGPT

## Who should watch
- Infrastructure engineers deciding whether to run systems themselves or rely on SaaS providers will hear a detailed account of the control, access, and maintenance trade-offs.
- Teams with brittle deployment scripts or incomplete patch histories should pay attention to Hirschfeld's case for immutable artifacts, repeatable builds, and regular resets.
- Machine-learning infrastructure teams that need clusters across cloud and bare-metal environments will find a concrete description of how reusable infrastructure workflows can hide operational details without taking away control.

## Editor's note

Rob Hirschfeld says teams should exercise the same automation repeatedly across environments so fixes become reusable instead of each installation becoming unique. ZenML lets teams write workflows as Python pipelines and run the same code on different infrastructure through a configurable stack. That gives the repeatability Hirschfeld wants without copying automation for each environment.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [MLOps Critiques](https://mlopstalks.com/talks/mlops-critiques) (Matthijs Brouns, Xccelerated.io, 49:44)
- [MLOps Investments](https://mlopstalks.com/talks/mlops-investments) (Sarah Catanzaro, Amplify Partners, 46:18)
- [Who's MLOps for Anyway?](https://mlopstalks.com/talks/whos-mlops-for-anyway) (Jonathan Rioux, EPAM Systems, 1:10:15)
- [The Godfather Of MLOps](https://mlopstalks.com/talks/the-godfather-of-mlops) (D. Sculley, Google, 51:25)
- [Most Underrated MLOps Topics](https://mlopstalks.com/talks/most-underrated-mlops-topics) (Marian Ignev, CloudStrap.io & SashiDo.io, 53:56)