MeetupOur 1st MLOps MeetupClaimModel monitoring differs from regular software monitoring because production data is unlabeled, so a model can have normal latency and error rates while producing incorrect results.23:00
Threads / Watching models in production / 2020
Watching models in production in 2020
27 sessions
MeetupOur 1st MLOps Meetup
MeetupHierarchy of Machine Learning NeedsClaimMonitoring for machine learning applications often relies on software engineering tools, basic statistics, and low-resolution summary data, leaving a gap for monitoring distributions and comparing them.10:20
MeetupHigh Stakes ML: Active Failures, Latent FactorsPushed backFlavio Clesio rejects using large technology companies as the main reliability benchmark for high-stakes systems.14:06
MeetupTrueLayer's MLOps PipelineClaimTrueLayer uses Prometheus and Grafana to monitor service latency, error rates, input distributions, and prediction-category frequencies.25:41
MeetupMLOps - The Blind Men and the ElephantPushed backSaurav Chakravorty argued that continuous retraining is not necessary in every setup, while continuous monitoring of scoring data is necessary.29:49
MeetupMachine Learning at Scale in Mercado LibreClaimFury is a cloud-independent platform for developing and deploying microservices that abstracts infrastructure, provides continuous integration, and automatically supplies monitoring, logs, and metrics.20:26
MeetupVenture Capital in Machine Learning StartupsPushed backJohn Spindler disputes the common preference for deep learning by saying that linear regression is often the better choice when it fits the problem.18:07
MeetupRunning a Fintech on Machine LearningClaimNubank can put models into production and change them relatively quickly, but model monitoring still requires substantial human review.16:06
MeetupBuild vs Buy an ML PlatformClaimDiego Oppenheimer says a production machine learning system must handle model versioning, model libraries, pipelines, source-code management, APIs, data connections, scaling, monitoring, security, and governance.26:00
MeetupMonitoring the Machine Learning StackPushed backLina Weichbrodt said that real-time response monitoring is needed in addition to offline data-quality checks such as Great Expectations or TensorFlow Data Validation.21:00
MeetupFeature Stores: An Essential Part of the ML Stack to Build Great DataClaimKevin Stumpf says Tecton automates feature backfills, streaming jobs, storage and monitoring after a feature definition is deployed.40:27
MeetupHow to Leverage ML Tooling EcosystemClaimThe production workflow uses Airflow for pipelines, Seldon for deployment, and Grafana for monitoring.23:09
PodcastMLOps: Isn't That Just DevOps?ClaimMLOps is what is necessary to make the machine-learning build, deploy, and monitor lifecycle as smooth and safe as possible.3:28
MeetupML ObservabilityClaimDemetrios Brinkmann says machine learning observability is one of his favorite subjects and that Aparna Dhinakaran is an expert in it.0:00
PodcastMLOps and DevOps, Parallels and DeviationsPushed backManual evaluation can be preferable to automating every check in some machine-learning workflows.22:39
PodcastMLOps from the Perspective of an SREClaimNeeran Gul says monitoring is essential for SRE because without it the team is effectively blind to what is happening in the infrastructure.50:08
MeetupStreaming Machine Learning with Apache Kafka and Tiered StoragePushed backKai Waehner said Kafka is not a silver bullet for every monitoring problem.37:56
PodcastContinuous Delivery and Automation Pipelines in ML, Part 2ClaimRetraining can be triggered by outlier detection or drift detection.7:34
PodcastAnalyzing "Continuous Delivery and Automation Pipelines in ML", Part 3Pushed backThe speakers dispute the idea that accuracy alone is the most important evaluation measure, emphasizing business metrics as well.14:13
MeetupMLOps #37 When You Say Data Scientist Do You Mean Data Engineer? Lessons Learned From Startup LifeClaimA data product in production needs software rules, testing, monitoring, and maintenance because it can break quickly.1:35
MeetupOperationalize Open Source Models with SAS Open Model ManagerClaimIvan Nardini says operationalizing analytics requires an end-to-end cycle covering governance, deployment, monitoring, retraining, and decisioning.7:38
PodcastCI/CD & Continuous Training in MLClaimAutomated triggers can retrain a model or rerun the pipeline when a score crosses a threshold, new data arrives, or outliers and concept drift are detected.28:00
PodcastLuigi in ProductionClaimLuigi Patruno started ML in Production because online information focused heavily on building classifiers and regressors on laptops rather than building machine-learning products that people use.2:33
PodcastIntroducing Data Downtime: From Firefighting to WinningClaimOrganizations can build support for data observability by tying it to business outcomes and explaining how data problems affect each team.21:05
MeetupThe Current MLOps LandscapeClaimNathan Benaich says the most active MLOps areas are data labeling and annotation, data quality, monitoring, and explainability.8:20