MeetupThe Motivation for MLOpsThreads / Watching models in production / 2023
Watching models in production in 2023
65 sessions
PodcastML in Production: A DS from Ubisoft PerspectiveClaimJean-Michel Daignan describes an end-to-end recommendation pipeline that processes game telemetry, builds targets, trains models with Spark, schedules production jobs with Airflow, monitors predictions, and runs A/B tests.16:49
MeetupDeclarative MLOps: Streamlining Model Serving on KubernetesClaimRahul Parundekar says unit tests should check serving behavior and returned scores, rather than replace the training-time evaluation performed on held-out data.26:47
Challenges and Opportunities in Building Data Science Solutions with LLMsClaimEvaluating language-model output is difficult because prompt engineering and quality assessment are less standardized than traditional machine-learning metrics such as R-squared.21:35
Want High Performing LLMs? Hint: It Is All About Your DataPushed backVikram Chatterji says there is not yet a broadly adequate metric for evaluating prompts across models, while practitioners often default to BLEU scores and similar measures.31:56
PodcastThe Birth and Growth of Spark: An Open Source Success StoryClaimMatei Zaharia says Databricks is integrating machine learning with data analytics by recording model inputs and outputs in tables that can be monitored and analyzed with data-warehouse features.28:55
DevTools for Language Models: Unlocking the Future of AI-Driven ApplicationsClaimKnowledge retrieval improves model accuracy and behavior by providing relevant context, and can also reduce operating cost.10:11
Cost Optimization and PerformanceClaimMonitoring can use end-user feedback, behavioral signals, human labeling, or synthetic evaluation by another language model.24:51
Agentic Relationship ManagementClaimAs agent action layers become more integrated with external platforms and social networks, they will need monitoring, rollback or interruption capabilities, and human involvement.7:47
Building Defensible Products with LLMsClaimHumanloop builds developer tools for prototyping LLM applications, understanding their production performance, and using evaluation data to improve them over time.0:36
Using LLMs to Punch Above Your Weight!ClaimCameron Feenstra says evaluation metrics are needed to determine whether a model performs as well as it did during testing and whether its performance changes over time.17:49
PodcastWhy is MLOps Hard in an Enterprise?ClaimMaria Vechtomova says their MLOps maturity assessment was conducted at the product level, including questions about version control and whether the code, model artifact, and data used for a deployment could be traced.17:42
LLMs in Production Conference - Part IIClaimThe LLM application stack is coalescing around foundational models, vector databases, model tuning and building, developer SDKs and tools, and model operations such as monitoring and experimentation.0:00
The Emerging Toolkit for Reliable, High-quality LLM ApplicationsClaimReliable LLM applications must address operational issues such as cost, performance, availability, model drift, timeliness, and privacy.5:14
Pitfalls and Best Practices: 5 Lessons from LLMs in ProductionClaimTeams should use objective evaluation during prompt engineering, in production monitoring, and when changing models or prompts to avoid regressions.7:27
Scalable Evaluation and Serving of Open Source LLMsClaimWaleed Kadous says open-source models can be good enough for summarization, retrieval, and assistant generation, even though they are not universal replacements for GPT-4.7:07
Beyond the Hype: Monitoring LLMs in ProductionClaimClaire Longo leads ML Solutions Engineering for Customer Success at Arize and helps customers integrate Arize as an ML observability solution.0:21
Building Production CopilotsClaimTristan Zajonc says conversational copilots can support knowledge discovery, ad hoc queries, and information retrieval.6:00
Embeddings and Retrieval for LLMs: Techniques and ChallengesPushed backAnton Troynikov argued that heuristics in robotics and retrieval systems are brittle and that models should eventually handle more of the required tasks.32:54
Building and Curating Datasets for RLHF and LLM Fine-tuningClaimProduction interactions with a prompt-based model can be monitored and reviewed by users or domain experts to create evaluation data and possible fine-tuning data.27:01
Evaluating LLM-based ApplicationsPushed backJosh Tobin disputes the common view that human evaluation is always the best way to evaluate language models, arguing that automated model evaluation can be useful and that the best approach combines automated and human evaluation.13:34
EvaluationPushed backJosh Tobin disputes relying on public benchmarks for application development, saying they are nearly useless when they do not measure a company's user data and outcomes.18:17
PodcastTreating Prompt Engineering More Like CodePushed backMaxime Beauchemin favors static, code-defined prompt cases rather than having an AI dynamically generate and evaluate the entire test set.37:50
Building Reliable AI AgentsClaimAgent workflow nodes can be tested and evaluated using practices from traditional software engineering, including unit tests, evaluations, and distilling a generic model into a smaller specialized model.15:43
Building ProductsClaimNatalia Burina proposes evaluating use cases along accuracy and fluency axes, because LLMs are useful when fluency matters more than accuracy and risky when high accuracy is required.11:51
LLMs For the Rest of UsPushed backJoseph Gonzalez says benchmark rankings can be misleading because some models abstain from unusual questions while people often prefer an answer.10:25
Lessons Learned Productionising LLMs for Stripe SupportClaimOffline evaluations and agent testing looked positive, but agents used the generated prompts much less than expected after launch.4:36
Combining LLMs with Knowledge Bases to Prevent HallucinationsClaimScott Mackie says information retrieval systems will still be required because language models do not have access to real-time data through their model weights.10:23
Unleashing Code Completion with LLMsClaimMonmayuri Ray evaluates code-completion outputs using honesty, harmlessness, and helpfulness.3:31
PodcastMLOps at the Age of Generative AIClaimBarak Turovsky's framework evaluates generative AI use cases by the required accuracy, fluency, and consequences of mistakes.18:38
Incorporating LLMs in High-stake Use CasesClaimBreaking large tasks into smaller tasks can make them easier; for example, information retrieval can evaluate each paragraph rather than an entire document.4:47
The Confidence Checklist for LLMs in ProductionClaimRohit Agarwal recommends captchas, rate limits, and monitoring to limit the cost of abusive traffic and DDoS attacks.7:47
Everything We've Been Taught About ML is WrongClaimEmmanuel Ameisen says that models can sometimes evaluate generated outputs better than human graders because they agree more closely with gold ratings.13:36
UX of an LLM UserClaimJasper monitors latency and error spikes with Datadog and uses fallback models or providers when performance problems occur.28:29
LIMA: Less is More for AlignmentClaimEvaluating truthfulness, especially for domain-specific questions without expert annotators, is difficult.10:06
MLOps vs LLMOpsClaimAparna Dhinakaran says LLM observability needs to trace whether the retrieved context was correct and relevant, especially when vector stores are used.23:01
Using LLMs to Power Consumer Search at ScaleClaimPerplexity is a retrieval-augmented generation system that combines a language model with a live search index so answers can use current information while retaining conversational reasoning capabilities.14:19
Fireside Chat - The Future of LLMsPushed backDavid Hershey says LLM application observability is not the same as traditional monitoring because teams first need to determine whether the application works at all.28:47
Taming AI Product Development Through Test-driven Prompt EngineeringPushed backMaxime Beauchemin says prompt-evaluation results should not be treated as purely Boolean because language-model outputs can be partly correct.11:11
PodcastAll About Evaluating LLM ApplicationsPushed backShahul Es disputes the idea that benchmarks can guarantee how an LLM will perform in a particular application.12:29
PodcastMLOps at GetYourGuidePushed backJean Machado argued that people should not be blocked from using LLMs because they do not use the platform's Python-centered templates, although the platform will provide managed tooling for use cases that need observability and controls.54:11
PodcastMLOps vs ML OrchestrationClaimKetan Umare says Union provides capabilities such as dynamic multi-cluster execution, role-based access control, single sign-on, task-level monitoring, and managed Kubernetes infrastructure.33:45
Observability for LLMsClaimObservability is part of the toolkit needed to make large language model applications reliable over time.1:23
The Truth About AI AgentsClaimSilen Naihin says AutoGPT needed an objective reward function or benchmark because qualitative inspection of prompt changes did not show whether an agent had improved.17:10
LLMs in Production at GetYourGuidePushed backThe GPT-4 evaluator was not consistently better than human evaluators, although it sometimes detected missed information and false positives.20:58
Current State of LLMs in ProductionPushed backLLM-based evaluation is subject to evaluator bias, including a model favoring answers from itself or a related model.8:32
Data Quality's Impact on Large Language ModelsClaimData observability adds historical analysis to data quality by learning data behavior and identifying when current values or business metrics are anomalous.18:55
From Building Self-driving Cars to Building LLM ApplicationsClaimEffy Zhang is the CEO of Baserun, a testing and observability platform for applications.0:04
PodcastBuilding Effective Products with GenAIPushed backUsing thumbs-up and thumbs-down feedback alone is not sufficient for evaluating generative AI output.50:26
Building RAG-based LLM Applications for ProductionPushed backThe speakers said that human evaluation cannot be replaced completely by language-model evaluation, even though language models can make evaluation more scalable.9:56
GenAI: An Unreliable Information StoreClaimNoble Ackerson says retrieval augmented generation can provide fresher data when a model's training has stopped and enterprise data continues to change.16:59
Product Engineering for LLMsClaimRetrieval-augmented generation makes it easier to turn large knowledge sources such as vehicle owner manuals into conversational assistants.12:05
Authoring Interactive, Shareable AI Evaluation Reports with ZenoPushed backAlex Cabrera says labeled evaluation generally gives a stronger signal than label-free evaluation.19:06
Evaluating LLMs for AI RiskClaimThorough LLM evaluation should address operational, ethical, and security risks using a combination of manual and automated red teaming.9:39
PodcastDesigning for Forward Compatibility in Gen AIClaimRohit Agarwal identifies summarization, question answering with retrieval-augmented generation, and generative writing tasks as three major production application types.16:35
Model Blind Spot Discovery for Better ModelsClaimA systematic ingestion evaluation should hold all variables constant except one and compare results on a representative subset of the data.34:32
PodcastEnterprises Using MLOps, the Changing LLM Landscape, MLOps PipelinesPushed backChris Van Pelt argues that a thumbs-up or thumbs-down is not sufficient as the complete evaluation of a model or application.40:34
MeetupScaling MLOps for Computer VisionClaimFabio Grätz says Recogni triggers benchmark workflows from feature-branch commits so proposed conversion changes can be evaluated across many networks.25:23
PodcastBuilding the Future of AI in Software DevelopmentClaimCodeium evaluates code models by deleting code from open-source repositories and checking whether the generated code makes the repositories' unit tests pass.42:08
PodcastModel Management in a Regulated EnvironmentClaimModel Management covers the tools and processes across a model's whole life cycle, from collecting data and choosing an algorithm through training, evaluation, deployment, monitoring, and replacement.4:22
PodcastLLMs in Focus: From One-Size Fits All to Verticalized SolutionsClaimFor customer-specific text-to-SQL, the required training data usually does not already exist in a ready-made form, so it must be curated, labeled, evaluated, and handled with privacy constraints.33:24