# LLMs in Focus: From One-Size Fits All to Verticalized Solutions

Venky Ganti & Laurel Orr, Numbers Station | MLOps Podcast | Episode 196 | 55:15
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=gjXc6y39Hjg
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/llms-in-focus-from-one-size-fits-all-to-verticalized-solutions
Published: 2023-12-19
Tags: enterprise, evals, fine-tuning, inference

## TL;DR
- Verticalized language models can handle enterprise data tasks more accurately because they are given the data schema, query history, and business context that general models lack.
- Smaller, task-specific models can reduce hosting costs, run inside an enterprise VPC, and support high-volume jobs such as entity resolution.
- Model evaluation should measure the result a customer cares about, such as the output table produced by SQL, rather than comparing generated SQL strings alone.

## Summary
Venky Ganti and Laurel Orr explain why general-purpose LLMs often struggle with enterprise data. A public model can write SQL for familiar schemas, but real companies have obscure column names, many tables, private query logs, and business rules that the model has never seen. Numbers Station focuses on private data models for conversational analytics, entity resolution, data wrangling, and related work. The guests argue for verticalization, with model size chosen for the workload. A larger model may suit occasional questions from business users, while a small tuned model is better for repeated production jobs. They discuss the practical work of collecting and labeling customer data, building evaluations, and keeping models inside a customer VPC. Laurel expects more local models and longer context windows, while Venky expects enterprise demand for private models to grow. Both describe model selection as an ongoing tradeoff between accuracy, cost, latency, and system complexity.

## Key ideas
### Number Station builds private models for enterprise data work
[05:35](https://www.youtube.com/watch?v=gjXc6y39Hjg&t=335s)
Venky Ganti describes Number Station's focus as private data LLMs for enterprises. One application lets business users ask questions of company data in a conversational way, with the model using enterprise context to produce more accurate answers. He also names entity resolution and data wrangling as customer needs. Laurel Orr joined after working with Chris Re at Stanford and becoming convinced that large models would change how engineers build applications and interact with data.

### Data wrangling still needs engineers and domain experts
[10:06](https://www.youtube.com/watch?v=gjXc6y39Hjg&t=606s)
Venky explains that wrangling includes schema and data transformations, cleaning, standardizing column names, filling null values, and matching records that refer to the same entity. Customers in insurance and real estate are interested in replacing rule-based entity-resolution systems that take time to build and maintain. He is clear that LLMs do not remove data engineers. Engineers still tune the models, and domain experts guide them toward the right behavior.

### General models know SQL syntax but miss private business context
[15:08](https://www.youtube.com/watch?v=gjXc6y39Hjg&t=908s)
Laurel says models trained on public sources can learn general SQL, including common schemas and query patterns. They struggle when a company has hundreds of tables, columns with unexplained short names, private pipelines, and business logic known only by a few engineers. The model has not seen the company's data or query logs, so its answer may be a plausible guess rather than a correct interpretation. This is why enterprise text-to-SQL requires access to the customer's own context.

### Verticalization improves accuracy and gives models useful guard rails
[17:45](https://www.youtube.com/watch?v=gjXc6y39Hjg&t=1065s)
Venky identifies two steps for complex enterprise tasks. The model must first be specialized for a data task such as text-to-SQL or entity resolution. It must then learn the customer's schema, values, query patterns, reports, and the purpose behind those queries. A private data model can also reject questions outside its scope, such as a weather question, or say when it lacks enough information to answer accurately. Those guard rails are harder to define in a general-purpose chatbot.

### Smaller models make private hosting and customization more practical
[20:17](https://www.youtube.com/watch?v=gjXc6y39Hjg&t=1217s)
Laurel argues that a model focused on text-to-SQL does not need most of the information stored in a broad model that also handles recipes, document summaries, and many other tasks. A smaller model can run locally with more reasonably priced hardware, which lets an enterprise host and customize it inside its own VPC. Venky adds that reducing the model's footprint can lower cost while making it easier to provide the specific context needed for accurate answers and to limit hallucinations.

### The right model structure depends on the workload
[24:11](https://www.youtube.com/watch?v=gjXc6y39Hjg&t=1451s)
Laurel prefers tighter task isolation when engineers need to customize models, since changing one behavior in a large multitask model can hurt another task. Venky separates human-driven questions from system-driven jobs. A business user asking for a sales aggregation can tolerate a larger model and more latency. Entity resolution may run millions of times, so it needs a small, efficient model tuned to a specific customer's data. Related tasks may share a model when transfer learning can carry useful information between them.

### Customer data is the hard part of fine-tuning
[32:03](https://www.youtube.com/watch?v=gjXc6y39Hjg&t=1923s)
Laurel says that companies rarely have a ready-made collection of text-to-SQL examples. Teams must curate logs and data dumps, create evaluation sets, label examples, and deal with privacy restrictions. Fine-tuning an existing model generally needs less data than training from scratch, but the quality of the available customer data still determines the result. Venky says Number Station starts with a verticalized model and adds the customer's schema, queries, and reports in a process he describes as taking weeks, followed by continued fine-tuning as the context changes.

### Evaluation should follow the customer's actual outcome
[51:44](https://www.youtube.com/watch?v=gjXc6y39Hjg&t=3104s)
Laurel says evaluation is custom because LLM tasks can combine free-form text with classifications, actions, or API calls. Standard metrics are easier for classification and less useful for free-form outputs. For text-to-SQL, comparing two SQL strings misses the point because many different queries can produce the same answer. The more useful test is whether the resulting table matches what the user needs. For some constrained classes of SQL, that comparison can be automated.

## Notable quotes
- Venky Ganti: "We're building private data LLMs for enterprises." (06:47)
- Laurel Orr: "They're very much there to help people faster but they're not there to replace people." (13:21)
- Laurel Orr: "The single most important factor to getting a high quality model is the data that comes in." (25:26)
- Laurel Orr: "The models guess, and it's not a smart guess." (16:48)
- Laurel Orr: "At the end of the day what they really care about is does the output table match." (52:50)

## Tools & references mentioned
- Numbers Station
- Chris Re
- Stanford
- Hazy Research Labs
- Google
- Mesh Dynamics
- Alation
- NSQL
- GitHub Copilot
- OpenAI
- Meta
- Llama
- Llama 2
- Claude
- DSPy
- Snowflake
- MongoDB

## Who should watch
- You are deciding whether a general LLM is suitable for text-to-SQL or enterprise data cleaning, and need to understand where private schema and business context matter.
- Your team is choosing between one large model and several smaller models for human questions, batch jobs, or high-volume entity resolution.
- You need a practical approach to collecting customer data and evaluating generated SQL by its result rather than by string similarity.

## Related talks

- [Challenges in Providing LLMs as a Service](https://mlopstalks.com/talks/challenges-in-providing-llms-as-a-service) (Hemant Jain, Cohere AI, 11:43)
- [Challenges and Opportunities in Building Data Science Solutions with LLMs](https://mlopstalks.com/talks/challenges-and-opportunities-in-building-data-science-solutions-with-llms) (Pascal Brokmeier, McKinsey and Company & Daniel Herde & Viktoriia Oliinyk, QuantumBlack, AI by McKinsey, 37:19)
- [Want High Performing LLMs? Hint: It Is All About Your Data](https://mlopstalks.com/talks/want-high-performing-llms-hint-it-is-all-about-your-data) (Vikram Chatterji, Galileo, 33:56)
- [Fireside Chat - The Future of LLMs](https://mlopstalks.com/talks/fireside-chat-the-future-of-llms) (David Hershey, Unusual Ventures & Daniel Jeffries, AI Infrastructure Alliance, 36:07)
- [LLMs For the Rest of Us](https://mlopstalks.com/talks/llms-for-the-rest-of-us) (Vikram Sreekanti, Aqueduct & Joseph Gonzalez, UC Berkeley and Aqueduct, 24:33)