# The Truth About LLM Training

Paul van der Boor & Zulkuf Genc, Prosus Group | Agents in Production Series | 55:47
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=QfwJYpDlslw
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/the-truth-about-llm-training
Published: 2025-08-09
Tags: evals, fine-tuning, gpus, model-serving

## TL;DR
- Prosus uses private, task-specific eval sets because public benchmarks do not predict how models perform on its real products and languages.
- The team chooses between model APIs, hosted open-source models, and its own GPUs based on privacy, utilization, cost, workload shape, and software support.
- Prosus trains specialized models on top of changing base models, so it can replace a depreciated base model without rebuilding the whole training effort.

## Summary
Paul van der Boor and Zulkuf Genc explain how Prosus evaluates, trains, and runs models for products used across its group companies. Their evaluation system covers tasks such as tool calling, summarization, open-book question answering, entity extraction, and programming questions in several languages. The datasets are private, manually curated, refreshed as models learn from older data, and run through automated pipelines. The team also tests models as judges and uses task-specific rubrics. On infrastructure, they describe a workload with sharp peaks for training and changing demand for inference. That makes fixed GPU commitments difficult. Provider support, framework compatibility, data access, privacy, networking, availability, and scaling matter as much as GPU price. They use commercial APIs where possible, hosted open-source models for some workloads, and their own infrastructure for proprietary models. Zulkuf and Paul expect specialized models fine-tuned from existing bases to replace more commercial traffic for specific tasks.

## Key ideas
### Public benchmarks do not predict performance on Prosus products
[00:49](https://www.youtube.com/watch?v=QfwJYpDlslw&t=49s)
Paul says Prosus evaluates models against the tasks its products actually need, rather than trusting public leaderboards. Examples include call summaries for Polish car dealers and recipe ingredient prediction for Brazilian food, along with voice calls in African, Polish, Brazilian, Hindi, and other languages. The team also cares about whether a model can scale to hundreds of millions of calls at an affordable cost. That leads to work on smaller models, distillation, open-source models, fine-tuning, GPU selection, utilization, and inference SLAs. Demetrios describes this as covering the problem from top to bottom, while Paul says Prosus has had to solve the whole range for production systems.

### Private evaluation sets give the team a fast way to test new models
[04:07](https://www.youtube.com/watch?v=QfwJYpDlslw&t=247s)
Zulkuf describes an internal evaluation capability that runs new models through pipelines and returns results in hours or days. The sets began with Stack Overflow data, where the team wanted to compare models trained with and without that data. They expanded the evaluations to cover different programming languages and question types, then added tool calling, summarization, open-book Q&A, entity extraction, and other tasks. Stack unseen uses newer Stack Overflow questions because older questions can enter model training data. The evaluations are private, but the resulting model comparisons are shared through ProLLM. This lets Prosus test claims about small or newly released models on its own workloads.

### Evaluation data becomes outdated and needs regular maintenance
[15:29](https://www.youtube.com/watch?v=QfwJYpDlslw&t=929s)
Zulkuf says evaluation sets start to get outdated after about six months because models and libraries change. Prosus tries to refresh Stack unseen every three months, while other datasets move more slowly because maintenance takes resources. The scope also changes with team needs. When a team starts using Go, for example, Go coverage becomes an evaluation requirement. Paul says the pipeline can place a new model on the leaderboard within an hour or a few hours, showing its task performance and cost. That quick signal helps the team decide whether a model is worth the larger effort of putting it into an agent or product.

### LLM judges work when the evaluation has a credible reference
[23:22](https://www.youtube.com/watch?v=QfwJYpDlslw&t=1402s)
For technical questions, Zulkuf says there may be several valid answers, so exact string comparison is not enough. The team gives a judging model the question, a trusted answer or highly rated examples, and the candidate answer, then asks whether the candidate is correct. He says the approach reached around 90% accuracy for judging correctness when a credible reference was available. Prosus first evaluates potential judges with human-labeled examples, then selects a judge, currently GPT-4.1 in the conversation. The scoring rubric depends on the task. Technical answers use a 0-to-3 scale tied to whether the answer lets the user continue, while summarization checks whether the model follows instructions such as returning exactly two takeaways.

### Model selection has become a moving production responsibility
[20:11](https://www.youtube.com/watch?v=QfwJYpDlslw&t=1211s)
Paul says the team has used continuous champion and challenger testing for its internal productivity platform, Token. More than a hundred models have entered production for tasks such as summarization and image generation, and he expects the models in use today to be different within six months. Zulkuf describes a shift from OpenAI models holding a clear lead to Google, Anthropic, Chinese, and open-source models becoming competitive on particular tasks. Prosus therefore routes different tasks to different models. A smaller open-source model may handle a token-heavy summarization job while a more expensive model handles work that needs it. The team is agnostic about the provider when choosing a model, then decides how to host it.

### GPU procurement depends on workload shape and provider support
[25:40](https://www.youtube.com/watch?v=QfwJYpDlslw&t=1540s)
Zulkuf says Prosus explored providers including NVIDIA, CoreWeave, Together AI, and MosaicAI, which later became part of Databricks. Providers often asked for reserved capacity over a fixed period, but Prosus training demand is spiky. The team may need many GPUs during a training week, then far fewer while it prepares data and studies results. Its preferred arrangement combines on-demand capacity for exploration with short reservations for larger runs. Inference has a different pattern because daily or weekly demand changes, so fixed GPU allocations can have poor utilization. The team also checks framework and model compatibility, Hugging Face integration, secure access to S3 data, deletion and replica policies, network speed, autoscaling, GPU availability, and the provider's support.

### Hosted open-source models and self-hosting solve different problems
[30:30](https://www.youtube.com/watch?v=QfwJYpDlslw&t=1830s)
Paul compares several ways to use an open-source model such as DeepSeek. Prosus could call the model provider directly, download it from Hugging Face to bare-metal machines, use a token-based hosted service such as Together AI, or run a proprietary fine-tuned model itself. Privacy can rule out calling the original provider. Bare-metal infrastructure can become expensive when usage fluctuates. A hosted token service handles the shared model, while a proprietary model still requires Prosus to provision and operate its own compute. Paul says switching GPU providers is fairly manageable when workloads are not tied to a provider's framework, although data routes and practical framework support can create work.

### Privacy-preserving tagging lets Prosus understand Token usage
[39:57](https://www.youtube.com/watch?v=QfwJYpDlslw&t=2397s)
Zulkuf says nobody on the team can read Token conversations without the user's consent, including database administrators. To understand usage, Prosus sends whole conversations to an LLM for tagging rather than inspecting them manually. Tags cover the domain, task type, use case, and estimated time saved. The system can identify coding, debugging, finance, HR, and data science patterns while avoiding the private content itself. Open-ended tagging created inconsistent labels such as Python, script writing, and programming for similar work. Direct embedding or prompting a model with more than about a hundred texts also produced poor clusters. The team instead builds an initial hierarchy, uses the model to refine it, and creates several levels of detail. This powers Flow, a data analyst agent that continuously analyzes large volumes of Token questions.

### Training specialized models does not require rebuilding from scratch
[49:12](https://www.youtube.com/watch?v=QfwJYpDlslw&t=2952s)
Zulkuf says Prosus trains models because its companies have valuable domain data and need systems tuned to their businesses, languages, and regions. The aim is to build specialized models that support customer-facing systems and agents. He distinguishes the depreciating base model from the work Prosus adds through fine-tuning, further pre-training, and alignment. When a newer base model replaces an older one, the team can put the new base into its existing data and training pipelines. Paul expects proprietary models to replace a larger share of commercial model traffic for specific tasks over the next 12 to 18 months, mainly because they can offer better task performance and lower scaling costs. Their development process starts with an evaluation baseline, then measures data, prompts, and training changes against it.

## Notable quotes
- Paul van der Boor: "The evaluations sets themselves are the biggest problem that we have for eval sets or for evaluations." (13:25)
- Paul van der Boor: "The models we have today in Token, right, that are answering the questions in six months all those models will be different." (21:53)
- Paul van der Boor: "These models are the quickest depreciating assets, right? They've got a half life." (23:51)
- Zulkuf Genc: "We cannot read any message and not even in our team without the consent of the person." (40:12)
- Zulkuf Genc: "You must train the models if you are in that game and you are serious about it and then you have a really good domain data." (49:32)

## Tools & references mentioned
- Prosus Group
- MLOps Community
- ProLLM
- Stack Overflow
- OpenAI
- Google
- Anthropic
- DeepSeek
- Llama
- Mistral
- Qwen
- Together AI
- MosaicAI
- Databricks
- NVIDIA
- CoreWeave
- Hugging Face
- GPT-4.1
- Token
- Flow
- S3
- Llama 2
- XGBoost

## Who should watch
- You run agents across several products and need evaluations tied to your own languages, domains, and tasks.
- Your team is deciding between commercial model APIs, hosted open-source models, and self-managed GPU infrastructure.
- You need to measure private user activity without giving engineers direct access to conversation content.

## Editor's note

Zulkuf Genc says Prosus can replace a depreciated base model by putting the new model into its existing data and training pipelines. ZenML records each pipeline run's steps, inputs, outputs, and code version, so the team can trace how a new base model changed the resulting model without losing the work behind it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Challenges in Providing LLMs as a Service](https://mlopstalks.com/talks/challenges-in-providing-llms-as-a-service) (Hemant Jain, Cohere AI, 11:43)
- [The Art and Science of Training LLMs](https://mlopstalks.com/talks/the-art-and-science-of-training-llms) (Bandish Shah & Davis Blalock, MosaicML/Databricks, 1:15:12)
- [Scalable Evaluation and Serving of Open Source LLMs](https://mlopstalks.com/talks/scalable-evaluation-and-serving-of-open-source-llms) (Waleed Kadous, Anyscale, 34:57)
- [Considerations and Optimizations for Deploying Open Source LLMs at Your Company](https://mlopstalks.com/talks/considerations-and-optimizations-for-deploying-open-source-llms-at-your-company) (Oscar Rovira, Mystic AI, 11:31)
- [Building LLM Applications for Production](https://mlopstalks.com/talks/building-llm-applications-for-production) (Chip Huyen, Claypot AI, 35:23)