# Cut One AI Cost and Two More Grow Back

Ioannis Zempekakis, Prosus | MLOps Community | 41:44
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=_ctB0shWyL4
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/cut-one-ai-cost-and-two-more-grow-back
Published: 2026-11-05
Tags: context-engineering, cost, evals, fine-tuning

## TL;DR
- Cloud API spending stops scaling when agents can use sandboxes, storage, integrations, and large amounts of context.
- Controlling context and routing tasks to simpler models reduced Prosus's cost by 10x, although new features continue to add spending.
- In-house engineering pays off when a workflow runs at enough scale, because it builds reusable evals, model-routing skills, and control over the stack.

## Summary
Ioannis Zempekakis explains why AI cost is larger than the price of a model API. Agents can use sandboxes, store information, connect to company systems, and repeat work without direct human effort. Employees can also recreate applications, schedule unnecessary runs, and discard codebases when the first attempt fails. Prosus reduced costs by controlling context and using a basic model router, while continuing to add new features that create fresh demand. Zempekakis argues that model progress could stop and companies would still have about two years of process, user-experience, and engineering improvements to make. Smaller fine-tuned models can pay off for repetitive work at large scale. Building internally also produces eval suites and preserves the option to change vendors or projects. He closes with hiring advice: test system design, resistance to over-engineering, AI intuition, and judgment about when an agent should make the implementation decision.

## Key ideas
### Agent harnesses make hidden usage costs grow quickly
[00:48](https://www.youtube.com/watch?v=_ctB0shWyL4&t=48s)
Zempekakis says calling a cloud endpoint from GCP or OpenAI feels simple, but the economics change when an agent harness gets a computer, a sandbox, storage, and integrations. An agent may pull information from Slack, email, Google Docs, Granola, Notion, and Linear, then check whether a conversation became a pull request or another work item. These actions can happen repeatedly without a person directly consuming the output. The sandbox itself is inexpensive, he says. The agent's ability to store information, run skills, and connect to more systems creates the larger cost.

### Giving everyone an AI tool creates repeated and unnecessary work
[03:03](https://www.youtube.com/watch?v=_ctB0shWyL4&t=183s)
Once a company releases a powerful tool, employees try many ideas. They build apps, documents, and presentations, and they may schedule work every five minutes or every Saturday even when that frequency is unnecessary. Zempekakis compares this with data systems, where teams learned to reduce refresh frequency when fresh data was not needed. AI makes the pattern easier to reproduce across the company. Users can also build the same application several times because the first attempt did not work, then throw away the codebase and start again. Each repetition consumes more context and tokens.

### Small amounts of context are often better for an agent
[05:16](https://www.youtube.com/watch?v=_ctB0shWyL4&t=316s)
The cost of rebuilding applications quickly leads to context management. Zempekakis mentions finding information about skills through semantic search, then says the simplest setup works best. The agent should receive as little instruction as it needs, because too much context can overcrowd the task and make the result worse. He says Prosus first followed the cloud API path with Cloud and OpenAI, then saw costs grow rapidly. If a task costs $10 and people can live without it, it may not be viable. If it costs 0.0001, it can become part of a large pipeline that saves substantial time.

### Evals let a company match models to individual tasks
[10:31](https://www.youtube.com/watch?v=_ctB0shWyL4&t=631s)
Zempekakis describes evals as a way to make engineering work more systematic. A company can cluster its evals by task and benchmark models against the relevant work. That makes it possible to choose one model for one task and another model elsewhere, rather than using the most expensive model everywhere. He admits this creates more work than simply calling an API, whose service levels are already dependable. The payoff is control over model choice and a clearer view of the cost and performance of each part of an agent system.

### Model progress could stop while process improvements continue
[14:18](https://www.youtube.com/watch?v=_ctB0shWyL4&t=858s)
Zempekakis says that even if model development stopped immediately, companies could still spend about two years improving how they use the existing level of intelligence. Those improvements include removing or adding processes, changing the user experience, and making engineering changes. He calls context control the easiest optimisation, followed by choosing the right model for each task. A basic router is enough to go far. Prosus reduced its cost by 10x with this approach, but Zempekakis says the company is also building ten new features each month, so new spending keeps appearing.

### Fine-tuning smaller models pays when repetitive work reaches scale
[17:31](https://www.youtube.com/watch?v=_ctB0shWyL4&t=1051s)
Smaller models can make sense when a workflow repeats often and the desired outcome is clear. Zempekakis gives logistics as an example, where training on logistics traces could produce similar or better performance at 20 or 30 times lower cost and with five or ten times lower latency. Data analytics is another candidate. The exact question may change, but the surrounding process, data, metadata, databases, and prior questions can remain similar. He says the economics depend on scale. A workflow for five people may not justify the work, while one used by 5,000 data analysts or engineers might.

### Building internally creates reusable engineering capabilities
[21:15](https://www.youtube.com/watch?v=_ctB0shWyL4&t=1275s)
The value of in-house work includes more than the immediate model or workflow. Zempekakis says a model router requires a strong eval suite, and the process of building one creates capabilities that can be reused elsewhere. Once a company learns how to build good evals, it can apply them to other agents and package the practice for the wider organisation. Prosus has about 30,000 employees in its ecosystem, so Zempekakis sees this as a way to spread engineering practices across the company. He rejects the idea that generative AI removes hard engineering. Fast building makes solid infrastructure and observability more important.

### Owning part of the stack preserves optionality
[24:48](https://www.youtube.com/watch?v=_ctB0shWyL4&t=1488s)
Fast-moving technology makes vendor and project commitments risky. Zempekakis says building some technology in-house prevents a company from being fully locked into OpenAI, Claude, a startup, or an open-source library that may change direction. Prosus began with a GPT-3 assistant in Slack, moved to a web application with selectable roles, then built an agent that selected tools and eventually a harness. He says this progression was possible because Prosus controlled a large part of the stack. A company that depends entirely on another project may have to accept a change, maintain a fork, or rebuild for compatibility.

### AI engineering hiring requires system design and judgment
[32:12](https://www.youtube.com/watch?v=_ctB0shWyL4&t=1932s)
Zempekakis does not hire for the title 'AI engineer'. He looks for specialisation relevant to the product, plus good software engineering and design thinking. Candidates should be able to design lean, scalable systems, because coding itself has become easier to delegate. He still uses whiteboard system-design interviews and checks whether candidates can spot over-engineering. Candidates also need practical AI intuition, including how agents work, what their limits are, how to build evals, and when to let an agent make an implementation decision. The hard hiring problem is finding people who combine software engineering experience with curiosity about new AI tools.

## Notable quotes
- Ioannis Zempekakis: "The easiest optimization is actually controlling the context and then the models." (14:59)
- Ioannis Zempekakis: "Even if it stops tomorrow, the development of the models, there's still for next two years with the same level of intelligence we can do optimizations." (14:18)
- Ioannis Zempekakis: "I think contrary because now you can build so fast, you need the basis to be very solid." (21:52)
- Ioannis Zempekakis: "Coding now more or less has been abstracted, but how do you design your solution has not been." (33:33)
- Ioannis Zempekakis: "You hire for specialization in a specific area of your product." (32:56)

## Tools & references mentioned
- GCP
- OpenAI
- Slack
- Google Docs
- Granola
- Notion
- Linear
- Cloud
- GPT-3
- Claude
- ChatGPT
- Open source
- Evals
- AI engineer
- SRE
- GPU
- LangChain
- Claude Code

## Who should watch
- You are setting an AI budget for a large company and need to account for agent context, storage, integrations, and repeated runs.
- Your team is deciding whether to use a hosted API, route across models, fine-tune a smaller model, or build part of the stack in-house.
- You are hiring AI engineers and want an interview process that tests system design, evals, agent judgment, and resistance to over-engineering.

## Related talks

- [Cutting Costs with Artificial Intelligence](https://mlopstalks.com/talks/cutting-costs-with-artificial-intelligence) (, 12:36)
- [The Cost of AI: FinOps Strategies for Intelligent Agents](https://mlopstalks.com/talks/the-cost-of-ai-finops-strategies-for-intelligent-agents) (Advait Patel, Broadcom, 31:05)
- [The Hidden Bottlenecks Slowing Down AI Agents](https://mlopstalks.com/talks/the-hidden-bottlenecks-slowing-down-ai-agents) (Paul van der Boor & Bruce Martens, Prosus Group, 48:00)
- [What Drives GenAI Development in the Next 3 Years](https://mlopstalks.com/talks/what-drives-genai-development-in-the-next-3-years) (Euro Beinat, Prosus, 18:54)
- [How to Optimize AI Agents in Production](https://mlopstalks.com/talks/how-to-optimize-ai-agents-in-production) (, 27:06)
