PodcastReal World AI Agent StoriesClaimMost of the project time is spent evaluating quality, debugging false positives, and coordinating with other departments rather than building the initial proof of concept.25:19
Threads / Watching models in production / 2025
Watching models in production in 2025
69 sessions
PodcastReal World AI Agent Stories
PodcastAI Careers Insights from Ex Meta Staff EngPushed backIlya Reznik disputes the assumption that a high benchmark score demonstrates real-world usefulness.18:18
PodcastReal LLM Success Stories: How They Actually WorkClaimLLM evaluation is often added only after a prototype is being prepared for a wider audience, rather than being part of the initial prototype.34:43
PodcastAutonomous AI SRE: The Future of Site Reliability EngineeringClaimWillem Pienaar says production issues are often associated with configuration or code changes, so monitoring deployments, Slack discussions and releases is highly useful.20:37
Reading groupAI Agents: The Future of ML Engineering?Pushed backThe speakers questioned whether the benchmark's contamination analysis adequately established that model performance was not inflated by memorization.27:50
PodcastLook At Your ****ing Data 👀ClaimPrompting or retrieval-augmented generation should be tried before fine-tuning, but fine-tuning remains useful when those methods cannot solve the problem.15:45
PodcastThe Agent Landscape - Lessons Learned Putting Agents Into ProductionClaimPaul van der Boor says agents in production at Prosus typically use well-defined APIs because they are easier to test and evaluate than open-ended web browsing.10:21
PodcastThe Challenge with AI Voice AgentsClaimBuilding a voice agent requires multiple capabilities, including transcription, speaker detection, language detection, translation, voice generation, and evaluation across languages.10:46
PodcastWeb Agents: The Cutting Edge of AI is Here?Pushed backThe team found that performance on the WebArena benchmark did not translate reliably to the web-agent tasks they cared about.17:49
PodcastAI SQL Data AnalystClaimThe team added a separate clarity pre-check that evaluates whether a question contains enough information before starting the agent.21:46
PodcastFuture of Software, Agents in the Enterprise, and Inception Stage Company BuildingClaimEliot Durbin says enterprise agent systems are developing application, monitoring, infrastructure, and compute layers, with pricing and billing still unresolved.12:41
PodcastI Let An AI Play Pokémon! - Claude plays Pokémon CreatorClaimClaude Plays Pokémon became useful as a long-horizon evaluation because reaching milestones such as defeating a gym leader shows whether a model can stay coherent, use new information, learn, and make progress over many hours.16:41
AI in Production 2025 | KeynotePushed backThe speaker argued that graph RAG does not have a single established best practice and that retrieval strategies require experimentation.44:10
PodcastBuilding Trust Through Technology: Responsible AI in PracticeClaimOrganizations should plan to iterate and should treat failures as opportunities to identify gaps in evaluation, experimentation, and alignment with organizational values.25:25
PodcastWe're All Finetuning IncorrectlyClaimAn AI system should close the loop from generation to evaluation to improvement, because evaluation without improvement leaves the system static.50:46
PodcastReal-Time Forecasting Faceoff: Time Series vs. DNNsClaimAutoregressive models mainly use recent history because short-term marketplace behavior tends to repeat even when the overall market is unstable.12:40
Reading groupA-MEM: Agentic Memory for LLM AgentsPushed backA participant questions whether enriching text with links and then using vector similarity provides meaningful benefit over semantic retrieval of the original content.51:12
PodcastMaking AI Reliable is the Greatest Challenge of the 2020sClaimAlon Bochman recommends defining an evaluation with examples of desired and undesired outputs so different models and configurations can be benchmarked on the team's own task.4:45
Evaluation of Agentic SystemPushed backAditya Gautam said that existing standard LLM benchmarks do not clearly apply to every agentic-system scenario.19:39
MCP is not going to change everything (yet)Pushed backA2A was presented as potentially important for business delegation, but Sam Partee questioned whether it was needed because ordinary tool calls can already implement observable agent handoffs.16:05
How Product Metrics Become LLM EvaluationsPushed backDemetrios Brinkmann suggests hallucinations may be an inherent nature of LLMs, while Raza Habib argues the central issue is broader system reliability and the need to evaluate desired behavior even with highly capable models.12:02
Everything Hard About Building AI Agents TodayClaimShreya Shankar says data-processing pipelines have many of the same evaluation and tool-use challenges as AI agents.2:14
Testing AI Intelligence: The Benchmarking BattleClaimPhysics, capitalism, and human preference are three possible evaluation engines, with physics being Greg Kamradt’s preferred option.27:20
ML Engineers Who Ignore LLMs Are Voluntarily Retiring EarlyClaimYoni Michael says reliable AI systems need multi-stage tracing and evaluation so teams can identify which model call or pipeline stage caused a bad final result.57:48
PodcastInside Uber's AI Revolution: Everything About How They Use AI/MLPushed backGenerative-AI evaluation is fundamentally different from predictive-machine-learning evaluation and remains an industry-wide problem.39:07
AI Agent Development Tradeoffs You NEED to KnowPushed backSherwood disagreed with treating agents as requiring fundamentally different observability from ordinary software, saying they remain software that should use metrics, logs, and traces.43:10
MLflow 3.0: The Future of AI AgentsClaimIn agent development, training is largely replaced by tuning or prompting, along with tools, function calling, tracing, and evaluation.2:42
The Hidden Bottlenecks Slowing Down AI AgentsPushed backThe speakers reject the idea that the evaluation tool itself is normally the main evaluation bottleneck.1:04
PodcastThe Truth About LLM TrainingPushed backPublic benchmarks do not reliably indicate performance on Prosus’s real use cases because models can be optimized for the benchmark test set.1:29
Advanced Context EngineeringClaimDex says that prompt content, memory, retrieval, agent history, and structured output are all parts of context engineering.15:09
The Science of Improving AI AgentsPushed backEval-Driven Development is presented as a better alternative to iterating based on personal feelings or “vibe” checks.1:05
Why Language Models Need a Lesson in EducationPushed backThe evaluation should be calibrated to the task and what matters for the project, not merely to what can be measured in the underlying data.14:18
Evaluation-Driven Development with MLflow 3.0ClaimEvaluation-Driven Development uses evaluation as the cornerstone of agent development, similar to how test-driven development uses tests.2:14
APICA: The Digital Colleague at the Port of Antwerp-BrugesClaimAPICA's safety approach includes monitoring and alerting, access control, guardrails, and a security evaluation framework.7:17
Iterating on Your AI EvalsPushed backHuman-in-the-loop evaluation may be introduced too late in the advanced-system example.12:22
AI traces are worth a thousand logsPushed backTraces should not be treated as part of the agent's context; they are a separate view of the operations the agent performs.10:48
Fast, Trustworthy, Reliable Voice Agents: MLOps That Blend LLM Annotation with Human QAClaimHappyRobot is building a holistic monitoring and evaluation platform that connects north-star metrics with targeted machine-learning metrics.7:59
From Spikes to Stories: AI-Augmented Troubleshooting in the Network WildClaimThousandEyes provides path traces across a digital experience, including the application server, internet service provider, cloud platform, and internet backbone.0:51
The Hidden Infrastructure Behind Every AI AgentClaimAI gateway challenges include dynamic routing, access control, cost-aware load balancing, model-aware failover, and observability.7:37
Building Real-Time, Reliable Voice AI: From Simulation to ProductionClaimCoval is building simulation and evaluation tools to help voice agents move from demos to highly reliable production conversations.0:26
Smart Agents Start with Smart LLM ChoicesClaimNova Pro V1 was less similar to the other models in the experiment, so engineers could investigate its different behavior with human evaluation.17:01
A Deep Discussion with the Author of "Context Rot"Pushed backKelly Hong rejected shuffling as a practical retrieval strategy and favored filtering or reranking to remove irrelevant context.51:19
Advancing the Cost-Quality Frontier in Agentic AIClaimAgent evaluation is difficult because many tasks are ambiguous and often lack labels, but quality still needs to be measured before production.4:41
Cutting Costs with Artificial IntelligenceClaimRobust evaluation is needed before changing models, prompts, or tool calls in a production agentic or multi-agent system.11:26
Catastrophic agent failure and how to avoid itClaimIn the healthcare example, an agent reached what appeared to be a successful state without booking the requested appointment, and the failure was not detected because the output was evaluated incorrectly.6:46
Zero Trust for Multi-Agent SystemsClaimMulti-agent systems are already in production, but scaling autonomy also scales risk, so systems need agent-level identity, isolation, context-aware access and strong observability.19:37
Before Building AI Agents Watch These Hard Earned LessonsClaimNishikant Dhanuka says evaluation should begin before launch with synthetic or manually collected queries and should use simple business metrics before more complex tool-level analysis.38:58
Building Multi-Player AI Systems (and why it's SO hard)ClaimMeshAgent Studio provides logging and observability for agent tool calls, messages, execution times, and other activities.7:58
Designing AI Agents for the Complex Realities of HealthcareClaimA key difference between good and bad residents is knowing when to ask for help, and AI agents should be evaluated on the same ability.4:58
The Cost of AI: FinOps Strategies for Intelligent AgentsClaimAdvait Patel recommends using infrastructure cost estimation, Kubernetes cost monitoring, observability tools, and built-in cloud provider recommendations to understand and optimize agent usage.13:25
An AI Company By AccidentClaimLiveKit has a transport network and an agents framework, but it is still filling gaps in testing, evaluation, deployment, scaling, and observability.20:26
Evaluating AI Agents: Why It Matters and How We Do ItClaimAI agents are software systems with nondeterministic parts, so they need evaluation at least as extensive as traditional software testing.2:07
How to Optimize AI Agents in ProductionClaimAI agent configurations involve choices about models, temperature, retrieval, prompting strategy, examples, styles, and roles.2:08
Beyond Chatbots: How to build Agentic AI systems with Google GeminiPushed backAgent evaluation should focus on reliability across repeated runs, not only capability or correctness on a single run.13:14
Why You Should Care About Observability in LLM WorkflowsClaimOpenTelemetry lets the team instrument agentic flows, record compliance-related business processes, and connect governance and auditing to deployed microservices.6:21
Evals Aren't Useful? Really?ClaimAgents need evaluations to understand how they are performing before being released to millions of users.0:43
How to build agents that take ACTIONClaimAlex Salazar recommends starting agent development with simple evaluations that define expected inputs and outputs for business scenarios.8:21
Big updates to MLflow 3.0Pushed backCorey Zumar argues that simple prompt-based LLM judges are limited because they cannot inspect the full trace, and presents agentic judges as a broader alternative.16:16
Meta-Prompting: The Hack That's Changing Production AIClaimArize is an engineering platform for people building AI agents that helps them evaluate and observe those agents in production.0:05
The Next Evolution of AI AgentsClaimRetrieval augmented generation brings real-time data into agentic applications and provides citations.7:10
Sub-Agent Architectures: What You Can LeverageClaimVibes-based testing is useful when starting out and for judging an agent's user experience, while rigorous test suites help measure small improvements and prevent regressions.8:35
MCP-Enabled AgentsPushed backSophia Skowronski questioned whether adding many distractor MCP servers truly replicates production conditions, rather than merely making the benchmark harder.27:08
PodcastVoice AI's Biggest Weakness ExposedClaimBrooke Hopkins says voice agents can be built by adding evaluations as new capabilities are added and improved over time.0:30
PodcastHow Sierra AI Does Context EngineeringClaimSierra runs simulations multiple times and uses an evaluator agent to check whether conversations meet a checklist.1:07
PodcastDoes AgenticRAG Really Work?ClaimSatish Bhambri says RAG-agent evaluation should separately examine whether the correct documents were retrieved, whether the dynamic prompt is relevant, and whether the language model produced the correct output.58:51
PodcastThe Future of AI Agents are SandboxesClaimRunloop.ai benchmarks run agents in known sandbox states and use scoring functions to measure how well they complete tasks.24:05