# Making Your Company LLM-native

Francisco Ingham, Pampa Labs | MLOps Podcast | Episode 266 | 56:14
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=ey6OG3C8g1Y
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/making-your-company-llm-native
Published: 2024-10-04
Tags: agents, evals, rag

## TL;DR
- An LLM-native company uses language models wherever they fit in both its products and its internal work.
- Teams should start agents with vibe checks, then add more detailed evaluations as demand, scale, cost, and latency make the architecture more stable.
- Domain expertise remains necessary because people must know what good work and a good user experience look like before they can direct an LLM.

## Summary
Francisco Ingham defines an LLM-native company as one that knows how to use LLMs wherever they fit. That includes customer-facing products and the daily workflows of employees. He argues for adding models to operational work while keeping humans responsible for creative work, decisions, and quality. At Pampa Labs, the team tests this idea with small internal agents for food orders and expense splitting. Francisco describes a development process that starts with quick vibe checks, then adds end-to-end and component-level evaluations as a product gains users and its architecture settles. He prefers rule-based tests where possible because LLM judges introduce unstable measurements. The conversation also covers interfaces, coding tools, retrieval, recommendation systems, and the role of domain experts. Francisco's view is practical: LLM features should fit the user's existing experience, and teams should choose chat, clicks, or automation according to the problem.

## Key ideas
### An LLM-native company applies models across products and internal work
[02:52](https://www.youtube.com/watch?v=ey6OG3C8g1Y&t=172s)
Francisco Ingham defines an LLM-native company as one that understands how to use LLMs anywhere they fit. He includes two sides of the business: products with LLM features and people who use LLMs in their daily work. Demetrios Brinkmann gives marketing production as an example, where a model might help with research, podcast uploads, transcription, or title suggestions without producing the entire podcast. Francisco says teams can overuse or underuse models, so they need experimentation to find the point where an LLM saves time instead of adding work.

### Internal agents are useful when natural language removes friction
[07:40](https://www.youtube.com/watch?v=ey6OG3C8g1Y&t=460s)
At Pampa Labs, the team deliberately forces LLMs into some workflows to see whether they help. One example is a WhatsApp agent for daily food orders. Team members write what they want, and the agent creates the list for the person placing the order. Another agent collects shared expenses and helps split them at the end of the month. Francisco says ordinary automation works when inputs are structured, but an LLM can interpret a voice note or an unstructured message. The benefit is often convenience, especially when users can stay in the platform they already use.

### LLMs can speed up development, while production work needs a data flywheel
[11:08](https://www.youtube.com/watch?v=ey6OG3C8g1Y&t=668s)
Francisco describes an agent architecture with tools that produce structured output for back-end processing. He also uses LLM-assisted development, including Cursor, so developers spend less time writing code and more time deciding what to build and how it should work. Once a system has real users and needs production quality, the team adds a progressively more systematic evaluation process. Pampa Labs starts with vibe checks, moves to a small evaluation set, and later evaluates separate modules as the product grows. The amount of evaluation work should match the evidence that the product and its architecture will remain in use.

### Evaluation becomes stricter when teams optimize cost or latency
[14:20](https://www.youtube.com/watch?v=ey6OG3C8g1Y&t=860s)
Francisco says evaluation becomes necessary when a system has enough traffic to justify optimization. Changing the model, reducing a prompt, or splitting a prompt into smaller stages can reduce cost or latency, but the team needs evidence that accuracy stayed mostly the same. He describes an end-to-end evaluation set as an integration test for the whole system. Pampa Labs tracks experiments with LangSmith, using datasets, notebooks, metrics, and the product interface. Successful experiments show that an architectural change or prompt change lowered cost or latency without materially changing the answers.

### Rule-based tests give teams more stable measurements than LLM judges
[20:55](https://www.youtube.com/watch?v=ey6OG3C8g1Y&t=1255s)
Francisco prefers evaluations that do not include LLMs where that is possible. Tool inputs can often be checked with strict rules, especially when the expected parameters are known. Retrieval queries may need semantic similarity or an LLM, and natural-language answers are difficult to assess without one. LLM judges add stochasticity, so results can change between runs. A team may need repeated runs, careful judge prompts, and tests of the evaluator itself. When Francisco does use an LLM judge, he prefers binary judgments or comparisons between two answers over a broad numerical score, with separate checks for aspects such as content and style.

### Existing interfaces should gain LLM help instead of forcing everything into chat
[26:09](https://www.youtube.com/watch?v=ey6OG3C8g1Y&t=1569s)
Francisco has tried agents that create presentation slides from a known structure, but he has not found a solution that produces designs he likes. He prefers tools that keep the model close to the work. Raycast can apply custom AI shortcuts inside apps such as Slack and email, while Cursor keeps coding assistance inside the IDE. He calls these low-command-tab solutions because users do not need to switch screens to get help. Demetrios questions whether chat should become a universal control centre. Francisco says people still want to see and interact with things, so integrated interfaces with chat, buttons, and other controls feel better than a blank chat screen.

### Domain knowledge becomes more valuable when LLMs do more of the execution
[30:19](https://www.youtube.com/watch?v=ey6OG3C8g1Y&t=1819s)
Francisco argues that LLMs enhance domain expertise rather than remove its need. A marketing specialist still needs to know what a good SEO result looks like. An AI engineer cannot decide what a good construction-industry user experience is without help from domain experts. In his view, an AI engineer is someone who can add LLM features to a product and use LLMs in their own development process. Traditional roles still contain expertise that existed before LLMs, including evaluation and experimental practice. An LLM-native data scientist combines that foundation with the ability to use LLMs.

### RAG systems need decisions about the data and use case before the technique
[40:38](https://www.youtube.com/watch?v=ey6OG3C8g1Y&t=2438s)
When clients ask Pampa Labs to add AI, Francisco says the work may begin with a proof of concept and a small group of users, or with optimization of an existing production system. Retrieval-augmented generation has many possible designs. Teams must decide what to embed, how to split documents, whether to include titles or summaries, which embedding model to use, and how to retrieve relevant vectors. A document question-answering system has different requirements from extraction, where the desired data points are known in advance. Extraction can use rules, keywords, or hybrid search. Francisco connects this work to his earlier experience with ranking, retrieval, and embeddings in recommendation systems.

### Good LLM product experiences combine chat, clicks, and existing user behaviour
[52:12](https://www.youtube.com/watch?v=ey6OG3C8g1Y&t=3132s)
Francisco sees recommendation systems as a promising place for LLMs. A shopper might ask for a darker shade or another colour after seeing products, creating an interaction closer to a sales conversation. The system can combine language with product data and user behaviour such as hovering over an image. He also likes interfaces that offer a small set of clickable suggestions, as in Notion, instead of requiring users to type every request. The right design depends on the experience a company wants to provide. Francisco argues for experimentation with users and a practical mix of interface elements rather than a rule that every feature must be chat-based.

## Notable quotes
- Francisco Ingham: "My definition is a company that understands how to leverage LLMs anywhere that they fit basically." (02:52)
- Francisco Ingham: "Invest in proportion to the evidence that you have that your architecture is going to stay in that way." (19:41)
- Francisco Ingham: "If you can do it without an LLM, do it without an LLM." (21:35)
- Francisco Ingham: "Domain specific knowledge for me is at an all-time high in value." (30:38)
- Francisco Ingham: "These systems need direction and the direction comes from knowing your users very well." (32:41)

## Tools & references mentioned
- Pampa Labs
- MLOps Community
- Cursor
- Raycast
- LangSmith
- Google Slides
- Shopify
- FashionCLIP
- Notion
- ChatGPT
- embeddings
- retrieval-augmented generation

## Who should watch
- You are deciding where LLMs belong in an existing product or operational workflow and need a way to test small ideas before committing to a larger build.
- Your agent works in a prototype, but you need to know when evaluation, cost tracking, and production safeguards are worth the effort.
- You work with retrieval, recommendations, or a domain-specific product and want to think through data choices and user interfaces before choosing a standard LLM pattern.

## Related talks

- [Real LLM Success Stories: How They Actually Work](https://mlopstalks.com/talks/real-llm-success-stories-how-they-actually-work) (Alex Strick van Linschoten, ZenML, 48:20)
- [Fireside Chat - The Future of LLMs](https://mlopstalks.com/talks/fireside-chat-the-future-of-llms) (David Hershey, Unusual Ventures & Daniel Jeffries, AI Infrastructure Alliance, 36:07)
- [Product Engineering for LLMs](https://mlopstalks.com/talks/product-engineering-for-llms) (Charles Frye, Full Stack Deep Learning & Sahar Mor, Stripe & Sarah Guo, Conviction & Shyamala Prayaga, NVIDIA & Willem Pienaar, 31:46)
- [The Next Revolution in AI: LLMs and Beyond](https://mlopstalks.com/talks/the-next-revolution-in-ai-llms-and-beyond) (, 13:47)
- [LLMs in Focus: From One-Size Fits All to Verticalized Solutions](https://mlopstalks.com/talks/llms-in-focus-from-one-size-fits-all-to-verticalized-solutions) (Venky Ganti & Laurel Orr, Numbers Station, 55:15)