# Experiment Tracking in the Age of LLMs

Piotr Niedźwiedź, Neptune | MLOps Podcast | Episode 168 | 45:11
Hosted by Vishnu Rachakonda

Source: https://www.youtube.com/watch?v=zbkXoONGDGg
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/experiment-tracking-in-the-age-of-llms
Published: 2023-08-01
Tags: evals, experiment-tracking, prompt-engineering

## TL;DR
- Neptune's mission is to give AI teams control and confidence while they build, test, deploy, and monitor models.
- Experiment tracking covers model debugging, comparison, versioning, reproducibility, and handing models over to operations, so the work extends beyond research experiments.
- Prompt engineering will need testing, validation, lineage, and connections to classical models before foundation-model systems can be used with the same confidence as conventional ML systems.

## Summary
Piotr Niedźwiedź explains how his path from software engineering and competitive programming led to Codilime, DeepSense.ai, and Neptune. Neptune began as an internal experiment tracker for a consulting business, created because teams were managing models with spreadsheets and homegrown tools. Piotr says the hardest product question is not which problem can be solved, but whether it will remain valuable after cloud providers or larger platforms address it. He connects this discipline to Neptune's mission: giving AI teams control and confidence over model development and testing. Large language models change the building process because prompts, agents, and foundation models become part of the system. They do not remove the need for experiment tracking. Teams still need to understand behavior, compare versions, validate changes, reproduce results, and trace which component uses which version. Piotr expects numerical validation methods, combinations of foundation and classical models, and a more integrated ecosystem of focused tools rather than one end-to-end platform.

## Key ideas
### Competitive programming gave Piotr confidence to start companies
[02:11](https://www.youtube.com/watch?v=zbkXoONGDGg&t=131s)
Piotr describes competitive programming as an important part of his education and early career. He won a silver medal at the national level in the Polish Olympiad in Informatics and later competed in Google Code Jam and Facebook Hacker Cup. These experiences gave him access to strong universities and a broad international network. They also gave him confidence that he could attempt difficult work and build a company. After receiving offers from large technology companies in the Valley, he chose entrepreneurship instead. His first company, Codilime, began as a way to generate money for a future product company and later grew to more than 400 people, with his brother as CEO.

### Neptune started after Piotr saw teams managing models in spreadsheets
[10:25](https://www.youtube.com/watch?v=zbkXoONGDGg&t=625s)
Piotr co-founded DeepSense.ai after seeing the potential of deep learning and wanting to build expertise with other people. The company began by entering Kaggle competitions and later provided machine learning services. Neptune came from an internal tool created by one of the data scientists, who combined experiment tracking with infrastructure management. At the time, training infrastructure required a lot of engineering work. Piotr then asked data scientists and leaders at other companies how they managed model development. He repeatedly heard that they used spreadsheets or homegrown systems. That repetition convinced him there was a real problem, even though it did not yet prove that a lasting business could be built around it.

### The hard product decision is separating temporary problems from lasting needs
[13:49](https://www.youtube.com/watch?v=zbkXoONGDGg&t=829s)
Piotr says that spotting problems in MLOps or large language model operations is relatively easy. A capable engineer can often build a quick solution for a specific issue faster than a cloud provider. The harder questions are whether the provider will solve the issue within six months or a year, and whether a company can defend its position over time. He frames product work around the jobs that must be done, why they matter, and why another company should not solve them. Neptune once passed on enterprise deals that would have doubled its revenue because the requested work did not fit its core jobs to be done. Piotr sees this market as a marathon rather than a winner-takes-all contest.

### Neptune's mission stays stable while the definition of a model changes
[22:19](https://www.youtube.com/watch?v=zbkXoONGDGg&t=1339s)
Piotr distinguishes between a company mission, which should change very rarely, and a product vision, which can adjust more often. Neptune's mission is to help machine learning or AI teams gain the control and confidence they need while building and testing models. Large language models complicate the meaning of model building because a system may involve a prompt, an agent, several models, or a combination of these parts. Piotr still sees the same underlying need as in software development: teams should be able to develop, test, deploy, and monitor systems in a controlled way. He expects the mission to remain stable even as Neptune's product changes to support new methods.

### Large language model systems still lack dependable validation
[25:05](https://www.youtube.com/watch?v=zbkXoONGDGg&t=1505s)
Piotr says many production uses of large language models still include a human in the loop, especially where the model can be wrong. Conventional deep learning systems have validation metrics and established ways to build confidence, while prompt-based systems often depend on a person deciding which prompt looks better. He expects new methods for testing and validating prompts and other foundation-model systems. These may include a foundation model judging prompt sets, an agent calling a classical model for a prediction, or a combination of fine-tuning and additional models. The specific solution is not yet clear to him. The need is clear: production systems require evidence for why one version is better than another.

### Experiment tracking covers production lineage, not only research
[30:00](https://www.youtube.com/watch?v=zbkXoONGDGg&t=1800s)
Piotr says Neptune is seeing more demand for experiment tracking than expected, even with the attention around large language models. He thinks the name is too narrow because the work is not limited to experimentation. Teams need to understand what happens while building a model, debug the process, compare it with other runs, version it, reproduce results, share it for feedback, and hand it to an operations team. Prompt engineering adds another form of building, with prompt chains and agents replacing some conventional model-development steps. Neptune plans to support prompt and chain visualizations, including integration with LangChain. Piotr still treats this as part of a broader system for tracking how components are built and used.

### A mature ecosystem will connect focused tools instead of replacing them with one platform
[39:53](https://www.youtube.com/watch?v=zbkXoONGDGg&t=2393s)
Piotr predicts that systems built around foundation models will gain numerical methods for testing and validation. Classical models will remain useful for narrower tasks such as predictive maintenance and recommender systems, and teams will need ways to combine them with foundation models. He also expects the MLOps ecosystem to become more mature through clearer categories and stronger integrations. Market pressure may reduce the number of vendors, making it easier for a focused product such as Neptune to decide which partners to integrate with. Piotr is skeptical that one end-to-end platform will dominate, since software has many focused tools that work together. He is much less certain about the ultimate capability ceiling of current foundation-model methods.

## Notable quotes
- Piotr Niedźwiedź: "The challenging thing is to decide what problems are temporary problems." (15:08)
- Piotr Niedźwiedź: "We want to provide them the same level of control and confidence when they are building and testing their models." (23:29)
- Piotr Niedźwiedź: "If you are building a solution based on a foundational model that is doing something specific with your data, then maybe fine-tuning, maybe it would be done by an agent that would ask your smaller classical model for some prediction." (35:46)
- Piotr Niedźwiedź: "I think that set of jobs to be done, at least for our understanding of experiment tracker, is a little bit broader." (36:59)
- Piotr Niedźwiedź: "I think we will have more clear categories of point solutions that are well integrated." (42:45)

## Tools & references mentioned
- Neptune
- OpenAI
- Wojtek Zaremba
- Google Code Jam
- Facebook Hacker Cup
- Codilime
- DeepSense.ai
- Kaggle
- Steve Jobs
- Jeff Bezos
- Walter Isaacson
- ChatGPT
- GPT-3
- GPT-3.5
- GPT-4
- Google Search
- Wolfram Alpha
- LangChain

## Who should watch
- You are deciding whether an experiment-tracking product should expand into prompt, agent, or foundation-model workflows.
- Your team uses spreadsheets or homegrown tools and needs reproducibility, comparison, versioning, and evidence for model decisions.
- You are building an MLOps product and need a way to separate short-lived customer requests from problems that can support a lasting product.

## Editor's note

Piotr Niedźwiedź says experiment tracking has to cover more than research: teams need to debug model development, compare runs, version models, reproduce results, and hand them to operations. ZenML records each run's steps, inputs, outputs, and code version, so a model or artifact can be traced back to the data and code that produced it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Continuous Evaluation & Model Experimentation](https://mlopstalks.com/talks/continuous-evaluation-model-experimentation) (Danny Ma, Sydney Data Science, 1:00:46)
- [Pitfalls and Best Practices: 5 Lessons from LLMs in Production](https://mlopstalks.com/talks/pitfalls-and-best-practices-5-lessons-from-llms-in-production) (Raza Habib, Humanloop, 30:27)
- [Taming AI Product Development Through Test-driven Prompt Engineering](https://mlopstalks.com/talks/taming-ai-product-development-through-test-driven-prompt-engineering) (Maxime Beauchemin, Preset, 21:26)
- [MLOps vs LLMOps](https://mlopstalks.com/talks/mlops-vs-llmops) (Richa Sachdev, JPMorgan Chase & Willem Pienaar, Feast & Chris Van Pelt, Weights & Biases & Aparna Dhinakaran, Arize AI & Alex Ratner, Snorkel, 34:57)
- [Enterprises Using MLOps, the Changing LLM Landscape, MLOps Pipelines](https://mlopstalks.com/talks/enterprises-using-mlops-the-changing-llm-landscape-mlops-pipelines) (Chris Van Pelt, Weights & Biases, 47:51)