PodcastLanguage, Graphs, and AI in IndustryPushed backPaco Nathan disputes the model-centric focus on benchmark scores without equal attention to data quality, cost, security and domain-specific evaluations.33:10
Threads / Watching models in production / 2024
Watching models in production in 2024
80 sessions
PodcastLanguage, Graphs, and AI in Industry
PodcastPioneering AI Models for Regional LanguagesPushed backDemetrios Brinkmann suggested that large language models may already beat humans on many exams, while Aleksa Gordić disputed the interpretation because evaluation data may have appeared in training data.22:54
PodcastThe Myth of AI BreakthroughsClaimJonathan Frankle says major breakthroughs are rare, and that AI researchers should focus more on incremental improvements, evaluation, synthetic data, and practical problems than on searching for the next Transformer.1:02:03
PodcastRAG Has Been OversimplifiedClaimProduction RAG systems should use observability and evaluation tools to check whether their answers are relevant and sufficiently correct.18:49
PodcastLLM Evaluation with Arize AI's Aparna DhinakaranPushed backAparna Dhinakaran argues that binary or multiclass evaluations are more useful than numeric score evaluations because LLM scores often have no reliable meaning on a spectrum.36:50
PodcastAds Ranking Evolution at PinterestClaimPinterest monitors offline data pipelines, online serving pipelines, model predictions, features, and end-to-end pipeline health, and can stop models from being promoted when checks fail.22:46
PodcastEvaluating and Integrating ML ModelsClaimMorgan McGuire says Clean RL provides clean implementations and evaluations of standard reinforcement-learning models and benchmarks, with its work instrumented in Weights & Biases.15:23
LLM Use Cases in ProductionClaimReside now ships an initial workflow, evaluates it continuously in production, and uses collected examples to develop scoring categories.13:03
PodcastBecoming an AI EvangelistClaimAlex Volkov learned ML operations by scaling his translation startup, putting models into production, and monitoring them.13:56
PodcastThe Real E2E RAG StackPushed backSam Bean disputes the idea that neural networks should be added to search or evaluation because they automatically make systems simpler.44:18
Model Merging and Mixtures of ExpertsClaimEvaluating merged models is more difficult and costly than creating them because creation can be done with a CPU, while evaluation requires running the models.7:47
Security and PrivacyClaimKatharine Jarmul says teams should define evaluation criteria early and update them with beta-tester feedback because model behavior can change with small version changes.28:03
PodcastThe Art and Science of Training LLMsPushed backDavis Blalock recommends increasing system complexity only when a simpler API, model, prompt, retrieval system, or fine-tuning approach is insufficient.54:29
PodcastDesigning ML Infra for ML & LLM Use CasesClaimModel-monitoring and retraining needs depend on the use case because some applications do not receive ground-truth outcomes quickly.18:31
Charting LLMOps OdysseyClaimYinxi Zhang describes four generative AI application patterns: prompt engineering, retrieval-augmented generation, fine-tuning, and pre-training.3:12
Navigating via Retrieval Evaluation to Demystify LLM WonderlandClaimInformation retrieval is a key enabler of applications such as online shopping, report building, assistants, films, and music.2:58
The State of Production Machine Learning in 2024ClaimProduction machine learning systems include experimentation, training data, model artifacts, deployment, inference, monitoring, and metadata flowing across the system.8:45
Graduating from Proprietary to Open Source Models in ProductionPushed backPhilip Kiely disputes treating benchmark scores as the complete basis for evaluating language models.9:37
From MVP to ProductionClaimEric Peter works on model training capabilities and retrieval-augmented generation tools at Databricks.0:59
Lessons from Building LLM-based Social Media ProductsClaimFaizaan Charania recommends treating a prompt like a machine-learning model and evaluating prompt versions against the same data and rules.11:20
Making Sense of LLMOpsClaimA retrieval-augmented generation system embeds a user prompt, retrieves similar context from a vector database, adds that context to the prompt, and sends it to an LLM.3:39
Shipping LLMs: Buckle Up & Enjoy the RideClaimShipping LLMs introduces challenges including prompt injection, hallucinations, inference costs, privacy, latency, and evaluation.8:25
Reliable Hallucination Detection in Large Language ModelsClaimHallucination research includes studying causes, detection and benchmarks, and mitigation.6:33
LLMOps and GenAI at Enterprise Scale - Challenges and OpportunitiesPushed backAndy McMahon noted that some articles were already arguing that retrieval-augmented generation was going away even though it had just emerged.6:44
PodcastWhat is AI Quality?ClaimTest coverage should be built from scenarios that reflect functional requirements rather than being treated only as an undifferentiated benchmark dataset.32:48
PodcastFedML Nexus AI: Your Generative AI Platform at ScaleClaimSalman Avestimehr says ownership, scalability, observability, privacy, and safety are major challenges when building generative AI applications.5:25
PodcastRecSys at SpotifyPushed backRecommender systems are not identical to retrieval-augmented generation because recommender systems lack the generative step that produces an explanatory paragraph.5:51
Building AI Products across Multiple Domains: Commonalities & Non-CommonalitiesClaimAI product managers need to think about training-data strategy, post-launch monitoring, orchestration layers, system components, problem decomposition, ensembles or hybrid systems, and annotation strategy.4:44
Productionizing AI: How to Think From the EndClaimAnnie describes her recent experience with LLMs as adapting from traditional regression and tree-based models while often feeling behind the pace of change.1:43
PodcastRetrieval Augmented GenerationClaimSyed Asad says a 133 MB CSV file was too difficult to embed directly for a retrieval-augmented generation system.4:00
Beyond Guess-and-Check: Towards AI-assisted Prompt EngineeringPushed backThe speakers disagreed about whether prompt iteration should have no default metrics and rely on human judgment or should include evaluation metrics for some analyses.14:27
Evaluating Language ModelsPushed backMultiple-choice evaluations are poor benchmarks for language models because they do not represent production use.6:31
Evaluating Quality and Improving LLM Products at ScalePushed backSystem evaluation should be prioritized over evaluating models alone.12:16
PodcastBuild Reliable Systems with Chaos EngineeringClaimObservability and chaos engineering need to work closely together so teams can distinguish injected effects from cascading failures and stop unsafe experiments.25:36
PodcastUber's Michelangelo: Strategic AI Overhaul and ImpactClaimMichelangelo includes an evaluation framework that lets teams compare new models against existing models using use-case-specific evaluation frameworks and datasets.28:46
How to Build Production-Ready AI Models for ManufacturingClaimIn semiconductor manufacturing, AI is used both to develop manufacturing processes and to monitor those processes during volume production.8:26
PodcastNavigating the AI Frontier: The Power of Synthetic Data and Agent Evaluations in LLM DevelopmentClaimEvaluations establish baselines and allow teams to iterate through synthetic data generation, model improvement, and fine-tuning.9:41
PodcastAI For Good - Detecting Harmful Content at ScaleClaimThe content evaluated by ActiveFence can include audio, video, images, text, comments, chats, usernames, descriptions, titles, podcasts and short-form content in many languages.6:41
PodcastExtending AI: From Industry to InnovationClaimComputer-vision models can monitor industrial workplaces for unsafe behavior and alert managers when a potentially dangerous situation occurs.19:41
PodcastEvaluating the Effectiveness of Large Language ModelsClaimAniket Singh evaluates what language models can do as human-like decision makers rather than focusing mainly on model benchmarks or the systems around the models.3:58
PodcastAI in HealthcarePushed backDemetrios Brinkmann asked whether using multiple guardrail systems would be redundant overkill, while Eric Landry described using Ragas for evaluation and NeMo Guardrails for runtime protections and intent-based responses.41:36
A Blueprint for Scalable & Reliable Enterprise AI/ML SystemsPushed backSteven Eliuk said that having the model-building team define its own evaluation criteria is a bad idea and that an outside group should audit the model.23:46
PodcastReliable LLM Products, Fueled by FeedbackClaimAI products should be analyzed in production with usage data and user feedback, rather than relying only on model evaluation metrics.19:21
Balancing Speed and SafetyClaimEvaluation should use human feedback and account for the difficulty of comparing language model outputs with ground truth.21:36
PodcastHarnessing AI APIs for Safer, Accurate, & Reliable ApplicationsClaimRon Heichman says prompting a language model is fundamentally about constructing the text that enters its context window, regardless of how many agent or retrieval components surround it.14:34
PodcastDesign and Development Principles for LLMOpsPushed backAndy McMahon disputes using generative AI for tasks that can be handled more cheaply, quickly and controllably with traditional methods such as logistic regression or regular expressions.40:40
PodcastBigQuery Feature StoreClaimPoint-in-time retrieval requires reconstructing the freelancer and project state at the time an interaction occurred, which is complex, error-prone, and expensive when done with historical joins.8:53
PodcastMLOps for GenAI ApplicationsClaimHarcharan Kabbay recommends designing RAG applications as microservices with separate embedding or vector-store, LLM, retrieval, and orchestration components.3:35
PodcastVisualize - Bringing Structure to Unstructured DataPushed backReal user feedback should be used early to judge Retrieval-Augmented Generation quality rather than relying mainly on automatic Retrieval-Augmented Generation metrics.12:18
PodcastAx a New Way to Build Complex Workflows with LLMsPushed backVikram Rangnekar says that frameworks often overcomplicate LLM development and lack production features such as telemetry.33:15
PodcastRAG Quality Starts with Data QualityClaimTonic Textual is designed to build data pipelines for people creating retrieval-augmented generation systems.4:41
Reading groupIntegrating Knowledge Graphs & Vector RAG for Efficient Information ExtractionClaimHybrid RAG combines vector retrieval and knowledge graphs to improve information extraction from unstructured financial documents.2:26
PodcastMaking Your Company LLM-nativePushed backLLM judges should be avoided when rule-based evaluations can test the system instead.20:55
PodcastCentralized or Decentralized ML Platform?ClaimPicnic's machine learning engineers own deployment and monitoring, while business analysts help define the business outcomes and the tests that should be automated.27:54
Data Contracts: The Missing Piece of the Data PuzzlePushed backThe answer to whether data observability and data contracts are still needed was corrected from no to yes.0:26
GenAI in production with MLflowClaimBen Wilson says MLflow evaluation can assess retrieval relevance by showing which document chunks were returned and what they contain.10:03
How Data Capture Transforms ML ObservabilityClaimPushkar says data capture on model endpoints enables data observability for models deployed in production.0:39
Scaling Data Reliably: A Journey in Growing Through Data Pain PointsClaimService-level agreements, service-level objectives, and service-level indicators make data-system expectations and failures observable.9:05
Supercharging Your RAG System: Techniques and ChallengesClaimTengyu Ma says retrieval remains necessary even if large language models eventually approach AGI, because it supplies proprietary company knowledge.4:33
The Future of Data: Composability & the Modular Data StackClaimData systems should follow a consistent pattern for setup and monitoring so that teams can simplify their architecture as they deploy and retire systems.6:07
Turn Data Chaos into AI Strategy with Programmatic AI Data DevelopmentClaimGenerative AI systems still require labeling to evaluate outputs, provide feedback, check document use, and align the system to enterprise requirements.4:29
PodcastSystematically Test and Evaluate Your LLMs AppsClaimGideon Mendels says LLM application evaluations generally use deterministic assertions, heuristic or distance metrics, and an LLM-as-a-judge metric.4:00
PodcastThe EU AI Act: Navigating New LegislationPushed backPetar Tsankov rejects the general narrative that regulation is inherently bad and argues that powerful technology needs baseline safety requirements made actionable through benchmarks.15:45
PodcastWe Can All Be AI Engineers and We Can Do It with Open Source ModelsPushed backAI-spec tests are not intended to replace general-purpose evaluation tools; they focus on the knowledge and API-calling features defined by the spec.37:00
Cleric AI SRE: Towards Self-healing Autonomous SoftwarePushed backWillem Pienaar disputed the idea that simply adding more logs, metrics, and observability dashboards solves the production operations problem.4:24
AI Agents: The Future of Productivity, or Just a Fad?Pushed backSam Partee says retrieval systems are not agents and that an agent must be able to call tools.4:50
Few Shot Code Generation to Autonomous Software Engineering AgentsClaimSWE-bench evaluates whether language-model systems can perform software engineering on real GitHub issues.3:02
How AI Agents Will Change Customer SupportClaimProduction AI agents must be evaluated not only for whether they work well and make good decisions, but also for the many ways they can fail.3:10
The Open Source AI Coding RevolutionPushed backBenchmark results can show overfitting, but better SWE-bench models still tend to work better in practical use.22:18
How to Make AI Agents that ACTUALLY WORKPushed backPatrick Marlo disputes evaluating only the outer result of a multi-stage RAG pipeline as sufficient for diagnosing problems.20:47
Building Replit Agent - Hard Lessons LearnedPushed backJames rejected the idea that public evaluations were sufficient for judging an agent built for a specific product and user group.15:25
Maximize Your Productivity with LLMs: Task Utility ExplainedPushed backJulia Kiseleva says calling the process evaluation is overstated and prefers calling it assessment.16:32
Is More Really Better: Delve Into Document StrategyClaimThe inference parameter vector for DRAG includes the number of retrieved documents and the number of in-context examples, while IterDRAG is the number of retrieval and generation iterations.1:32
Simulation Techniques for AI Agents from Self-DrivingClaimSelf-driving development moved from rigid, manually maintained test cases toward probabilistic evaluation over large collections of generated or logged scenarios.6:26
PodcastHolistic Evaluation of Generative AI SystemsPushed backJineet Doshi says there is no universally correct evaluation method because the appropriate technique depends on the use case.5:30
Reading groupHow AgentOps Enables ObservabilityClaimThe field lacks a generally accepted standard or single source of truth for agent observability.6:36