# Gen AI Buy vs Build, Commercial vs Open Source

Ilona Logvinova, McKinsey & Mohamed Abusaid, QuantumBlack, AI by McKinsey & Nayur Khan, Goldman Sachs | AI in Production 2024 | 56:21

Source: https://www.youtube.com/watch?v=IpXZGXeuHt4
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/gen-ai-buy-vs-build-commercial-vs-open-source
Published: 2024-02-02
Tags: build-vs-buy, governance, open-models, testing

## TL;DR
- Build or buy decisions should start with the business problem, then work backward to the required model, data, infrastructure, and controls.
- Commercial models and products can provide speed and managed safeguards, while open models offer more control, customization, and local hosting at the cost of additional skills and infrastructure.
- Gen AI adds to traditional ML rather than replacing it, so production systems still need analytics, data platforms, testing, governance, and model evaluation.

## Summary
The panel separates several decisions that are often mixed together. An organization might buy a product with an embedded model, build a custom application around a commercial API, or host and adapt an open model. Buying can provide speed and managed controls, while building can improve customization, data control, and fit with the organization's policies. The panel argues that the choice should begin with the problem and the sensitivity of the data. Enterprise offerings may keep data inside an organization, but users often cannot see how a model was trained or which model is operating behind a product. The speakers also discuss hallucinations, regulatory debt, regional language and cultural needs, and the difficulty of moving between models. Gen AI does not remove traditional ML. Recommendation systems, customer data platforms, analytics, and other existing components still have work to do. A chatbot proof of concept can start with a managed API, then move to self-hosting if production risk and control requirements grow.

## Key ideas
### The model market divides into closed APIs and open models
[08:08](https://www.youtube.com/watch?v=IpXZGXeuHt4&t=488s)
Nayur Khan describes closed models as commercial systems accessed through an API, such as OpenAI or Google Gemini. Their inner workings are opaque and the infrastructure belongs to someone else. Open models, including Mistral, Mixtral, Falcon, and Meta's Llama 2, can be taken off the shelf and hosted or adapted by an organization, although their licenses and restrictions differ. Hugging Face contains a very large collection of models. This distinction frames the later choices between buying a ready-made capability, building around a model, and choosing commercial or open technology.

### Buying can mean purchasing a whole product or only accessing a model
[10:19](https://www.youtube.com/watch?v=IpXZGXeuHt4&t=619s)
The panel separates products that already include generative AI from custom applications that an organization builds itself. Microsoft, Salesforce, SAP, and GitHub are examples of existing products adding LLM capabilities. With an off-the-shelf product, the user may not choose the underlying model and may have limited visibility into how it works. A custom application gives the organization more choices about the model, integration, data, and controls, but requires more engineering work. This distinction matters because build versus buy is not one decision. It can involve the user-facing product, the model engine, and the surrounding infrastructure.

### Data boundaries and model behavior should guide the risk decision
[12:00](https://www.youtube.com/watch?v=IpXZGXeuHt4&t=720s)
Ilona Logvinova describes an enterprise model as a closed circle. Data can remain on premises or in the organization's cloud environment, with insights generated from that internal data. A second arrangement keeps organizational data inside the boundary while allowing the tool to draw information from external sources. The trade-off includes confidentiality, data use, and the quality of generated answers. Logvinova explains that generative systems can hallucinate because they are designed for creativity, while multi-step reasoning, including some mathematical tasks, is harder for them. Organizations must decide how much creativity they need and how much correctness and containment their use case requires.

### The right build decision starts with the problem and the organization's constraints
[18:19](https://www.youtube.com/watch?v=IpXZGXeuHt4&t=1099s)
Mohamed Abusaid says buying can provide a capability quickly, but usually limits customization and control. Building can produce a better fit, yet requires more money, time, skills, data, infrastructure, and often GPUs. He rejects starting with a technology and searching for a problem afterward. The panel recommends defining the problem, understanding why it matters, and then deciding whether an LLM is needed at all. Ilona adds that the market is immature and changes quickly, so organizations should ask whether a capability is core intellectual property, central to the product, or likely to become a standard capability supplied by the market.

### Model transparency is still limited, even when organizations want it
[27:04](https://www.youtube.com/watch?v=IpXZGXeuHt4&t=1624s)
The panel does not identify a ready-made product that reliably tells users how a model was trained, which datasets were used, or which model is operating behind an application. Even open models may disclose code without making the complete training data reproducible. Managed services can provide data-use agreements, but that is different from giving an end user a clear view of model provenance. Ilona notes that providers such as Anthropic, Cohere, and OpenAI have different principles and behavior, but users can assess those differences only when they know which provider and model a product uses.

### A chatbot can move from a managed proof of concept to a self-hosted system
[30:24](https://www.youtube.com/watch?v=IpXZGXeuHt4&t=1824s)
For an internal chatbot, an organization might keep its existing ServiceNow interface, use a managed model such as Azure OpenAI, and rely on existing cloud contracts and data protections. Production also requires an API, integration work, a vector store, infrastructure, and connections to organizational data. If the chatbot becomes external-facing and handles sensitive customer data, the organization may decide to host an open model itself. The panel presents this as a practical progression: use a commercial API to test value quickly, then consider a self-hosted model when production scale, control, and risk requirements justify the additional work.

### Gen AI adds a generative layer to traditional ML systems
[39:20](https://www.youtube.com/watch?v=IpXZGXeuHt4&t=2360s)
Nayur Khan gives a retail example in which traditional analytics uses purchase history and behavior to decide which customer should receive which message, through which channel, and at what time. Gen AI can then turn that decision into a message with an appropriate tone. The analytical model, customer data platform, and MLOps processes still matter. Gen AI is useful for summarizing, extracting, paraphrasing, and generating language, but it is weak at precise mathematics and some financial calculations. Production teams therefore need to combine generative systems with existing analytics, data, testing, and governance practices.

### Changing models requires prompt versioning, testing, and an abstraction layer
[51:29](https://www.youtube.com/watch?v=IpXZGXeuHt4&t=3089s)
Abusaid compares changing an LLM to changing a car engine. Prompts can behave differently across models and versions, including between OpenAI model versions. Teams should abstract prompts, store metadata about which model and version each prompt supports, and test when they change providers. A model gateway or similar indirection can make it easier to switch models during evaluation. The panel also recommends recording prompts and responses and continuously checking that outputs remain within acceptable bounds. Managed model behavior can change, so a proof of concept should be made reproducible and testable before it becomes a production dependency.

## Notable quotes
- Mohamed Abusaid: "Build versus buy, it depends. That's the punch line." (20:18)
- Ilona Logvinova: "Don't have the tech identify the problem for you. Identify the problem first and address it." (35:59)
- Mohamed Abusaid: "The GenAI technology is great, but it's not great at numbers. Not great at maths." (45:43)
- Mohamed Abusaid: "If you change the engine of your car, you might find that something doesn't work." (51:29)
- Nayur Khan: "Make sure you have written enough prompts and recorded those responses and communicated some sort of mechanism to constantly evaluate what you're running against." (53:35)

## Tools & references mentioned
- OpenAI
- Google Gemini
- Mistral AI
- Mistral
- Mixtral
- Technology Innovation Institute
- Falcon
- Meta
- Llama 2
- Hugging Face
- Microsoft
- Windows 11
- Salesforce
- SAP
- GitHub Copilot
- Azure OpenAI
- ServiceNow
- Anthropic
- Cohere
- Llama 3
- ChatGPT
- OpenAI API
- MLOps Coffee Sessions
- QuantumBlack
- McKinsey
- Goldman Sachs

## Who should watch
- You are deciding whether to buy a Gen AI product, build a custom application, or host an open model, and need a way to separate those choices.
- Your organization handles confidential, regulated, or region-specific data and needs to understand model boundaries, transparency, and vendor questions.
- You have a proof of concept built on a commercial API and want to prepare for model changes, production testing, or a move to self-hosting.

## Related talks

- [Build vs Buy an ML Platform](https://mlopstalks.com/talks/build-vs-buy-an-ml-platform) (Diego Oppenheimer, Algorithmia, 57:20)
- [GenAI in Production - Challenges and Trends](https://mlopstalks.com/talks/genai-in-production-challenges-and-trends) (Verena Weber, Verena Weber, 48:43)
- [Extending AI: From Industry to Innovation](https://mlopstalks.com/talks/extending-ai-from-industry-to-innovation) (Sophia Rowland & David Weik, SAS, 1:01:37)
- [We Can All Be AI Engineers and We Can Do It with Open Source Models](https://mlopstalks.com/talks/we-can-all-be-ai-engineers-and-we-can-do-it-with-open-source-models) (Luke Marsden, HelixML, 54:31)
- [Building AI Products across Multiple Domains: Commonalities & Non-Commonalities](https://mlopstalks.com/talks/building-ai-products-across-multiple-domains-commonalities-non-commonalities) (Dhruv Ghulati, 12:32)