Agent evaluation is limited by the quality of curated eval sets and product feedback, rather than by the evaluation tool itself.
2
Prosus encourages engineers to buy tools when they solve a real problem, but production use is restricted by maturity, legal review, privacy, and missing capabilities.
3
Building core agent infrastructure in-house gives Bruce Martens's team more control over debugging, reliability, and product-specific features.
Summary
Paul van der Boor and Bruce Martens explain why buying tools has not removed the hardest work in agent development. Evaluation depends on curated datasets, real user conversations, and a feedback loop. Their team holds labeling parties to create evaluation sets because a vendor cannot supply that product knowledge. Prosus encourages engineers to test external tools, yet many fail to meet production needs around privacy, legal approval, reliability, or technical control. Bruce describes building an internal agent platform in Go, including orchestration and evaluation. Direct API access let the team measure delays between streamed response chunks from Vertex AI and add a targeted retry. The discussion also covers coding agents. Cursor, Devin, and GitHub Copilot are useful for existing, documented codebases, onboarding, and simple changes, while fully generating maintainable production systems remains unreliable. The speakers are candid that vendor tools work well for prototypes, but compliant production systems often require in-house work or self-hosting.
Evaluation needs data and feedback before it needs another tool
Paul van der Boor says the difficult part of evaluation is creating a curated eval set and collecting new conversations from real users. The team needs to measure whether models improve, regress, or behave differently across cases. Bruce Martens describes labeling parties where people from across the company score answers for food images, code, and customer conversations. The participants may need different language skills, such as Polish or Brazilian Portuguese. Paul says tools such as Spellbook, Humanloop, and Orca AI can help, but they do not remove the work of generating evaluation data. The feedback loop is what lets the team compare new models and improve over time.
Prosus buys tools to save engineering time, even when production adoption is slow
Paul says there are fewer external tools in the team's production agents than he would like, partly because time and people are scarce. He tells engineers there are no rewards for rebuilding something that could have been bought. The team is encouraged to test new open-source and commercial tools, learn which problems they solve, and share useful tools with engineers across Prosus's portfolio companies. This testing also helps the investment arm identify companies worth partnering with. Paul admits that his team sees problems earlier than many customers because it trains models and builds AI products at scale, so tools designed for smaller or less technical teams may use drag-and-drop interfaces that his engineers do not need.
Coding agents work better on existing codebases than on new production systems
Paul says the team uses coding agents such as Cursor and Devin rather than building one. Cursor changed the team's workflow, while newer agents can create environments and attempt tasks end to end. Devin works well for small personal projects and incremental changes in documented repositories with CI/CD. It is less suitable for creating an entire production codebase because the architecture may not match the team's preferences and can be hard to maintain. The team runs a rotating "Devin duty" process, where an engineer first gives a Jira task to the coding agent. They track whether the agent produces useful pull requests and may later set a target share of agent-ready PRs.
AI-generated code must reduce work without creating a larger review burden
The team previously built an AI pull request reviewer that produced so many comments that people disabled it. Paul frames the problem as finding the point where the work completed by an agent outweighs the extra effort needed to understand its output. The answer depends on the task, the engineer's seniority, and familiarity with the codebase. Coding agents are also useful for onboarding. A new hire can ask Cursor to describe a repository's services, endpoints, conventions, and utility code. Paul uses the same approach when he needs to understand a codebase without working in it every day.
Evaluation and prompting stay in-house when they are central to the product
Bruce works on Token, Prosus's internal agent platform, which is distributed to portfolio companies. His team keeps prompts in its own systems and built its own evaluation flow. Prompt quality is central to the product, so the team wants to control it directly. The evaluation system includes an internal evaluation team and leaderboard. Sending complete conversations to an outside evaluation provider would also require legal approval and would raise concerns about exposing personally identifiable information. A self-hosted or bring-your-own-cloud option could reduce that concern, but it does not remove the need to trust the vendor and its handling of the data.
Direct infrastructure can make narrow reliability fixes possible
Bruce gives an example from the team's in-house orchestrator. Requests to Vertex AI sometimes exceeded a five-minute timeout. Because the team implemented the API directly instead of hiding it behind an SDK or orchestration library, it could measure the time to the first response chunk and the time between later chunks. The team found cases where the gap between chunks lasted minutes. It added a timeout for that condition and retried the request, while also recording the provider's request ID in its logs. Bruce says an external orchestrator might not expose these details. The tradeoff is clear: frameworks make prototypes faster, but direct code gives the team more control over unusual production failures.
Go adds restrictions that reduce uncertainty, while limiting access to AI tooling
Bruce's platform uses Go even though much of the AI ecosystem targets Python. Go's compiled code and stricter structure reduce the amount of uncertainty around the software surrounding a model. In Python, the team would need extra work to make code fail-safe, and some small changes would rely on monkey patching. The cost is that many model providers and orchestration tools offer better SDK support in Python. Bruce says the team sometimes does more work up front, then saves time when it needs to debug an edge case or expand a core feature such as memory. Building the memory system internally lets the team add tool-call history later without rewriting around a vendor's limits.
Privacy and compliance can turn a promising vendor into extra engineering work
Bruce tested vendor tools for OAuth-based integrations and agent memory, including Composio and another memory service. A tool that looked promising had many integrations, but testing revealed missing scopes, undocumented setup steps, and tools that failed in staging. More seriously, dashboards exposed user conversations and memories that contained personally identifiable information. Bruce contrasts this with DataDog and Sentry, where the team avoids logging PII and uses controls around user data. A self-hosted option can help, but the team still has to inspect how the product handles privacy. Vendor tools remain useful for prototypes, yet production agents need stronger compliance and scaling controls.
"If something is so core to your product and you want to expand the feature that it's easy to do instead of having to hack around or change the library."Bruce Martens30:50
Who should watch
You are deciding whether to build or buy evaluation, orchestration, observability, or agent integrations for a production system.
Your team is testing coding agents and needs practical guidance on where they help, where they create review work, and how to measure adoption.
You need to assess vendor tools while dealing with legal approval, PII, self-hosting, and the limits of your current language or platform.
