Prosus uses private, task-specific eval sets because public benchmarks do not predict how models perform on its real products and languages.
2
The team chooses between model APIs, hosted open-source models, and its own GPUs based on privacy, utilization, cost, workload shape, and software support.
3
Prosus trains specialized models on top of changing base models, so it can replace a depreciated base model without rebuilding the whole training effort.
Summary
Paul van der Boor and Zulkuf Genc explain how Prosus evaluates, trains, and runs models for products used across its group companies. Their evaluation system covers tasks such as tool calling, summarization, open-book question answering, entity extraction, and programming questions in several languages. The datasets are private, manually curated, refreshed as models learn from older data, and run through automated pipelines. The team also tests models as judges and uses task-specific rubrics. On infrastructure, they describe a workload with sharp peaks for training and changing demand for inference. That makes fixed GPU commitments difficult. Provider support, framework compatibility, data access, privacy, networking, availability, and scaling matter as much as GPU price. They use commercial APIs where possible, hosted open-source models for some workloads, and their own infrastructure for proprietary models. Zulkuf and Paul expect specialized models fine-tuned from existing bases to replace more commercial traffic for specific tasks.
Public benchmarks do not predict performance on Prosus products
Paul says Prosus evaluates models against the tasks its products actually need, rather than trusting public leaderboards. Examples include call summaries for Polish car dealers and recipe ingredient prediction for Brazilian food, along with voice calls in African, Polish, Brazilian, Hindi, and other languages. The team also cares about whether a model can scale to hundreds of millions of calls at an affordable cost. That leads to work on smaller models, distillation, open-source models, fine-tuning, GPU selection, utilization, and inference SLAs. Demetrios describes this as covering the problem from top to bottom, while Paul says Prosus has had to solve the whole range for production systems.
Private evaluation sets give the team a fast way to test new models
Zulkuf describes an internal evaluation capability that runs new models through pipelines and returns results in hours or days. The sets began with Stack Overflow data, where the team wanted to compare models trained with and without that data. They expanded the evaluations to cover different programming languages and question types, then added tool calling, summarization, open-book Q&A, entity extraction, and other tasks. Stack unseen uses newer Stack Overflow questions because older questions can enter model training data. The evaluations are private, but the resulting model comparisons are shared through ProLLM. This lets Prosus test claims about small or newly released models on its own workloads.
Evaluation data becomes outdated and needs regular maintenance
Zulkuf says evaluation sets start to get outdated after about six months because models and libraries change. Prosus tries to refresh Stack unseen every three months, while other datasets move more slowly because maintenance takes resources. The scope also changes with team needs. When a team starts using Go, for example, Go coverage becomes an evaluation requirement. Paul says the pipeline can place a new model on the leaderboard within an hour or a few hours, showing its task performance and cost. That quick signal helps the team decide whether a model is worth the larger effort of putting it into an agent or product.
LLM judges work when the evaluation has a credible reference
For technical questions, Zulkuf says there may be several valid answers, so exact string comparison is not enough. The team gives a judging model the question, a trusted answer or highly rated examples, and the candidate answer, then asks whether the candidate is correct. He says the approach reached around 90% accuracy for judging correctness when a credible reference was available. Prosus first evaluates potential judges with human-labeled examples, then selects a judge, currently GPT-4.1 in the conversation. The scoring rubric depends on the task. Technical answers use a 0-to-3 scale tied to whether the answer lets the user continue, while summarization checks whether the model follows instructions such as returning exactly two takeaways.
Model selection has become a moving production responsibility
Paul says the team has used continuous champion and challenger testing for its internal productivity platform, Token. More than a hundred models have entered production for tasks such as summarization and image generation, and he expects the models in use today to be different within six months. Zulkuf describes a shift from OpenAI models holding a clear lead to Google, Anthropic, Chinese, and open-source models becoming competitive on particular tasks. Prosus therefore routes different tasks to different models. A smaller open-source model may handle a token-heavy summarization job while a more expensive model handles work that needs it. The team is agnostic about the provider when choosing a model, then decides how to host it.
GPU procurement depends on workload shape and provider support
Zulkuf says Prosus explored providers including NVIDIA, CoreWeave, Together AI, and MosaicAI, which later became part of Databricks. Providers often asked for reserved capacity over a fixed period, but Prosus training demand is spiky. The team may need many GPUs during a training week, then far fewer while it prepares data and studies results. Its preferred arrangement combines on-demand capacity for exploration with short reservations for larger runs. Inference has a different pattern because daily or weekly demand changes, so fixed GPU allocations can have poor utilization. The team also checks framework and model compatibility, Hugging Face integration, secure access to S3 data, deletion and replica policies, network speed, autoscaling, GPU availability, and the provider's support.
Hosted open-source models and self-hosting solve different problems
Paul compares several ways to use an open-source model such as DeepSeek. Prosus could call the model provider directly, download it from Hugging Face to bare-metal machines, use a token-based hosted service such as Together AI, or run a proprietary fine-tuned model itself. Privacy can rule out calling the original provider. Bare-metal infrastructure can become expensive when usage fluctuates. A hosted token service handles the shared model, while a proprietary model still requires Prosus to provision and operate its own compute. Paul says switching GPU providers is fairly manageable when workloads are not tied to a provider's framework, although data routes and practical framework support can create work.
Zulkuf says nobody on the team can read Token conversations without the user's consent, including database administrators. To understand usage, Prosus sends whole conversations to an LLM for tagging rather than inspecting them manually. Tags cover the domain, task type, use case, and estimated time saved. The system can identify coding, debugging, finance, HR, and data science patterns while avoiding the private content itself. Open-ended tagging created inconsistent labels such as Python, script writing, and programming for similar work. Direct embedding or prompting a model with more than about a hundred texts also produced poor clusters. The team instead builds an initial hierarchy, uses the model to refine it, and creates several levels of detail. This powers Flow, a data analyst agent that continuously analyzes large volumes of Token questions.
Training specialized models does not require rebuilding from scratch
Zulkuf says Prosus trains models because its companies have valuable domain data and need systems tuned to their businesses, languages, and regions. The aim is to build specialized models that support customer-facing systems and agents. He distinguishes the depreciating base model from the work Prosus adds through fine-tuning, further pre-training, and alignment. When a newer base model replaces an older one, the team can put the new base into its existing data and training pipelines. Paul expects proprietary models to replace a larger share of commercial model traffic for specific tasks over the next 12 to 18 months, mainly because they can offer better task performance and lower scaling costs. Their development process starts with an evaluation baseline, then measures data, prompts, and training changes against it.
