31 sessions
MeetupJust Build It! Tips for Making ML Engineering and MLOps RealClaimAndy McMahon recommends bootstrapping capability by adding practices incrementally, such as version control, packaging, unit testing, CI/CD, infrastructure as code, experiment tracking and performance monitoring.14:04
PodcastTowards Observability for ML PipelinesPushed backShreya Shankar rejects the common practice of monitoring thousands of feature-level KL divergences as the primary way to operate ML systems.18:14
PodcastBuild a Culture of ML Testing and Model QualityClaimMohamed Elgendy says that model quality depends on both the model and the test data, while test data is often treated as an afterthought or a benchmark dataset that is incrementally expanded.10:26
PodcastCalibration for ML at EtsyClaimSeoyoon Park said that growing the team and separating machine-learning work from other stack responsibilities reduced context switching and enabled more focus on data quality and evaluation metrics.9:08
MeetupBuilding an Open Source MLOps Stack with ZenMLClaimHamza Tahir says the NBA data changed after Stephen Curry's long three-point shot, which he presents as an example of data drift.14:02
PodcastPlatform Thinking: A Lemonade Case StudyPushed backAutomatic model monitoring was not considered suitable for Lemonade's process, so data scientists had to configure monitors manually.14:09
PodcastPractitioners Guide to MLOpsClaimThe Practitioners' Guide to MLOps describes six integrated and iterative processes: development and experimentation, training operationalization, continuous training, model deployment, prediction serving, and continuous monitoring.12:37
PodcastBetter Use Cases for Text EmbeddingsClaimVincent Warmerdam says benchmark datasets can contain enough label errors that reported state-of-the-art performance may partly reflect overfitting to incorrect labels.22:28
PodcastLessons from Studying FAANG ML SystemsPushed backErnest Chan pushed back on the idea that shadow mode would necessarily solve the problem of a model facing major COVID-related data drift, saying it might not help the existing model unless that model were turned off.36:39
MeetupOrchestrating Machine Learning Workflows with PrefectClaimEvidently AI can compare baseline and new data to detect data drift and can expose JSON metrics for use in pipeline decisions.50:29
PodcastMLOps at StripeClaimEmmanuel Ameisen says machine-learning teams should automate data gathering, feature generation, model training and evaluation wherever possible.15:35
MeetupApplications of Data ScienceClaimPallet monitors classification confidence, reviews predictions below a threshold, and uses human checks to investigate possible misclassifications.27:14
PodcastModel Monitoring in Practice: Top TrendsClaimModel monitoring is needed after deployment to check whether models behave as intended, whether input distributions drift, and whether relationships between targets and features change.5:27
MeetupMLOps EngineeringLabsClaimA Flyte workflow is a set of tasks or processes that orchestrates work while providing guarantees for completion, observability, control, and infrastructure abstraction.5:26
MeetupDataOps is a Software Engineering ChallengePushed backMicha Kunze says commercial data-observability tooling was not valuable enough for his team's use case because they needed integrated checks that could stop pipelines, rather than only post hoc metrics.46:09
Scaling Real-time Machine Learning at ChimeClaimChime built monitoring and alerting into its machine-learning platform so newly deployed real-time models receive them by default.16:56
PodcastMLOps CritiquesPushed backMatthijs Brouns says monitoring tools should account for the whole software stack rather than only ML monitoring.29:21
PodcastFixing Your ML Data Blind SpotsClaimYash Sheth says unstructured data can experience semantic drift, which makes it important to refresh production datasets at scale.17:35
MeetupDevOps, Security, and Observability in MLClaimLuke Marsden says the MLOps process turns data and code into a model, deploys the model to deliver business value, monitors it, and feeds monitoring results back into iterative improvements.3:17
PodcastMLflow Pipelines: Opinionated ML Pipelines in MLflowClaimXiangrui Meng says real-time machine learning requires automating the model lifecycle, monitoring data and model performance, and retraining or updating models when data changes.15:46
MeetupFrom Expectations to Synthetic Data GenerationClaimThe right way to evaluate synthetic data depends on its downstream use, such as machine learning, fidelity, privacy, or utility.42:01
PodcastMLOps at DoorDashClaimDoorDash's model development process uses version-controlled and reviewed code so models can be reproduced and traced back to their source.10:02
PodcastRecommender System: Why They Update Models 100 Times a DayClaimFunCorp reuses features across two or three models and evaluates models through A/B tests.12:26
PodcastTrustworthy Machine LearningPushed backKrishnaram Kenthapadi suggested that academic research focuses more on data preparation and validation than monitoring, while Kush Varshney agreed that monitoring is an important open research area.33:34
Monitoring Unstructured DataClaimArize monitors every embedding, identifies patterns of change, and provides tools to export problematic samples for labeling or data-quality work.6:31
MeetupObtain New Insights on Model Behavior with FiddlerClaimFiddler helps organizations register models, centralize model performance monitoring, and understand input drift, output drift, and accuracy over time.2:53
PodcastManaging Machine Learning ProjectsPushed backSimon Thompson disputes the common focus on model accuracy alone and argues that business value and the consequences of errors should determine evaluation.33:06
PodcastLet's Continue Bundling into the DatabasePushed backEthan Rosenthal argued that streaming databases may support feature stores and model monitoring, while Mike Del Balso said they were not yet a complete replacement for feature stores and remained early for large-scale requirements.30:52
PodcastSystems Engineer Navigating the World of MLClaimObservability for machine learning systems should include instrumentation, failure information, tabular logs, counters, and time-series data across workloads.20:25
PodcastExplainability in the MLOps CycleClaimDattaraj Rao says fairness checks, explainability, bias checks and drift monitoring remain important for machine learning systems just as they were important for rule-based systems.27:50
