# Considerations and Optimizations for Deploying Open Source LLMs at Your Company

Oscar Rovira, Mystic AI | MLOps Community | 11:31

Source: https://www.youtube.com/watch?v=TkJLx46yVJQ
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/considerations-and-optimizations-for-deploying-open-source-llms-at-your-company
Published: 2023-08-16
Tags: build-vs-buy, deployment, gpus, model-serving

## TL;DR
- Deploying an open source LLM is similar to deploying other machine learning models, but it requires more GPU memory and more attention to model-specific libraries and environments.
- Teams need infrastructure that can handle GPU scarcity, dynamic scaling, cost control, low latency, streaming responses, monitoring, privacy, and deployment across clouds or private environments.
- Oscar presents Mystic AI's Pipeline Core as a way for data scientists to define model pipelines with Python decorators, upload them, and receive a monitored, scalable API endpoint.

## Summary
Oscar Rovira describes the engineering work required to turn an open source LLM into a fast, secure, scalable API. Each model may need different libraries and packages, while the large memory footprint makes GPU selection and availability difficult. Production systems also need dynamic scaling, cost controls, low latency, streaming, monitoring, and deployment inside a company's own premises or cloud VPC. He compares three approaches: building on Kubernetes and Docker, using a cloud provider's managed service, or adopting a platform that provides the deployment infrastructure. The first gives control but needs specialist expertise and significant development time. The second creates cloud dependence and still requires maintenance. Oscar then presents Mystic AI's Pipeline Core, which lets data scientists define pipelines with decorators and ordinary Python code, deploy them across infrastructure, and obtain an API endpoint with scaling and monitoring. He shows a Falcon 87B example and a customer dashboard reporting a P95 of 25 milliseconds.

## Key ideas
### An LLM API creates a familiar machine learning deployment problem with heavier infrastructure demands
[01:03](https://www.youtube.com/watch?v=TkJLx46yVJQ&t=63s)
Oscar frames the starting point as turning a large LLM into a fast, secure, scalable API endpoint, similar to the experience provided by OpenAI for private models. The software engineering problem resembles deployment for other machine learning models, but LLMs may need much more memory. Each model can also require different libraries and packages, so the deployment system must manage the environment for the specific model. The infrastructure has to account for both the model's software dependencies and its GPU memory requirements. Oscar's point is that users should be able to think about those requirements without rebuilding deployment machinery for every model.

### GPU availability and model memory make infrastructure selection a central concern
[02:01](https://www.youtube.com/watch?v=TkJLx46yVJQ&t=121s)
Oscar says LLMs are among the larger models teams put into production, which makes GPU capacity and memory important from the beginning. Teams need to choose a GPU that can run the model, yet available GPUs are limited and expensive because many companies want them. A deployment system may therefore need access to a large pool across cloud providers. Moving between providers matters when the provider selected first does not have the required hardware. The underlying challenge is to hide those infrastructure decisions while still selecting a machine that can support the model's library and memory requirements.

### Production systems must scale with demand while controlling GPU cost
[02:55](https://www.youtube.com/watch?v=TkJLx46yVJQ&t=175s)
Running a model on one GPU is only the first step. Oscar asks teams to consider how many users will access it and how many requests it must handle per second. Hundreds of concurrent requests could require hundreds of GPUs, so the software needs to scale dynamically with demand. Cost also needs active management because running these machines can become very expensive. Oscar mentions spot instances and using multiple cloud providers as possible cost techniques. The deployment layer should decide how much capacity is needed and which available GPU is the most economical fit for the workload, rather than leaving every scaling decision to the application team.

### Latency, caching, and streaming shape the user's experience
[03:57](https://www.youtube.com/watch?v=TkJLx46yVJQ&t=237s)
Oscar says the industry should target 50 milliseconds of API latency to support an easier streaming experience. The system also needs to have the model ready on the GPU before a request arrives. He calls this preemptive caching, where the model is cached ahead of request processing. Streaming matters because users generally prefer words to appear as they are generated instead of receiving one large block of text. A useful deployment system therefore needs to combine low request latency, model readiness, and streaming output. These requirements add work beyond simply placing a model on a GPU and exposing an endpoint.

### Data scientists should be able to deploy without passing work through many software teams
[04:52](https://www.youtube.com/watch?v=TkJLx46yVJQ&t=292s)
Oscar argues that deployment should be closer to the data scientist's existing workflow. Teams may have models and Jupyter notebooks ready, but asking several software teams to package and operate them creates a barrier. The platform should let data scientists deploy their own models while still providing reliable infrastructure. Monitoring is part of that requirement because systems fail and teams need to identify and fix problems quickly. Privacy and security also matter for open source models, since companies may want to run them on their own premises or inside their own cloud VPC. The deployment software therefore has to fit the company's infrastructure and security constraints.

### Building on Kubernetes gives control but demands specialist infrastructure work
[05:43](https://www.youtube.com/watch?v=TkJLx46yVJQ&t=343s)
The first approach Oscar describes is to use the skills a software engineering team already has, including Kubernetes and Docker. Each Docker file can contain a different model, while Kubernetes handles scaling. This approach is familiar from CPU-based microservices and gives a company control over where the model runs. Oscar warns that GPU workloads create additional challenges, so the assumptions from CPU services do not carry over directly. A team needs the expertise to solve those infrastructure problems and connect them to its existing systems. Building an internal machine learning platform for production scale also takes substantial time.

### Managed cloud services reduce some work while creating provider dependence
[07:08](https://www.youtube.com/watch?v=TkJLx46yVJQ&t=428s)
The second approach is to use a cloud provider's managed offering, such as Vertex AI or the equivalent service from another provider. This can avoid building all of the infrastructure internally, but the company becomes limited by the cloud provider it chooses. Oscar describes this as cloud lock-in. Teams still need expertise with that vendor's services, and they still spend time and resources maintaining the infrastructure. The approach can be practical for an organization already committed to one provider, but it does not remove the operational work or the need to understand the selected cloud's deployment model.

### Pipeline Core packages deployment work behind a Python-based data scientist workflow
[07:43](https://www.youtube.com/watch?v=TkJLx46yVJQ&t=463s)
Oscar presents Pipeline Core, the product Mystic AI built for companies that want an API endpoint without implementing the deployment infrastructure themselves. He says it is intended to shorten the path to production and work across cloud providers. The workflow starts with a small number of decorators around functions that define the machine learning pipeline. It is not limited to one framework or file type. A pipeline can combine frameworks and include preprocessing, postprocessing, and other Python code. In his example, one function loads Falcon 87B and another defines the inference path step by step. After uploading the pipeline, the user receives an API endpoint, load testing, scaling, and a dashboard for monitoring.

## Notable quotes
- "How do I go from this LLM big model into getting a fast secure scalable API endpoint?" (01:03)
- "You need to have a system that is able to manage whatever is the environment required to run this model." (01:41)
- "Ideally in the industry should be targeting 50 milliseconds of API latency." (03:57)
- "How can we empower the data scientists to deploy these models in reliable infrastructure immediately?" (08:32)
- "With a simple SDK we really give the companies the power of an experience that we've seen teams for ML workloads." (10:19)

## Tools & references mentioned
- Mystic AI
- Y Combinator
- OpenAI
- Kubernetes
- Docker
- Vertex AI
- Pipeline Core
- Falcon 87B
- Stripe

## Who should watch
- You are deciding whether to build an internal platform for serving open source LLMs and need a clear comparison with managed cloud services.
- Your team has models in notebooks but lacks a straightforward path to GPU-aware, scalable API deployment.
- You need to think through GPU availability, latency, streaming, privacy, and cost before putting an open source LLM into production.

## Editor's note

Oscar Rovira says data scientists should be able to deploy models without passing their work through several software teams. ZenML lets them define ML workflows as Python steps and run the same pipeline code on different infrastructure by changing the configured stack. That gives teams a way to keep deployment code separate from choices such as Kubernetes or a cloud provider.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Scalable Evaluation and Serving of Open Source LLMs](https://mlopstalks.com/talks/scalable-evaluation-and-serving-of-open-source-llms) (Waleed Kadous, Anyscale, 34:57)
- [Efficiently Scaling and Deploying LLMs](https://mlopstalks.com/talks/efficiently-scaling-and-deploying-llms) (Hanlin Tang, MosaicML, 25:14)
- [MLOps + Machine Learning](https://mlopstalks.com/talks/mlops-machine-learning) (James Sutton, Algorithmia, 1:01:50)
- [LLMs For the Rest of Us](https://mlopstalks.com/talks/llms-for-the-rest-of-us) (Vikram Sreekanti, Aqueduct & Joseph Gonzalez, UC Berkeley and Aqueduct, 24:33)
- [Enterprises Using MLOps, the Changing LLM Landscape, MLOps Pipelines](https://mlopstalks.com/talks/enterprises-using-mlops-the-changing-llm-landscape-mlops-pipelines) (Chris Van Pelt, Weights & Biases, 47:51)