Vishnu Prathish built a bespoke MLOps platform because data cleaning and feature engineering had to run the same way in notebooks, retraining, batch inference, and real-time inference.
2
A cloud-native approach with Amazon SageMaker reduced the integration and DevOps work needed to connect training, deployment, CI/CD, monitoring, and retraining.
3
Automated retraining needs a mature model and human review. Poor data can make a prediction problem impossible regardless of the model architecture.
Summary
Vishnu Prathish describes the MLOps platform Innovyze built for digital twins of water infrastructure. Each customer has different assets, sensor quality, and operating conditions, so models are partly hand-curated while moving through one standardized deployment pipeline. The platform records data cleaning and feature engineering as configuration so the same transformations can be reproduced during training, retraining, batch inference, and real-time inference. Innovyze chose a cloud-native approach built around Amazon SageMaker because integrating separate open source tools would require substantial DevOps work. The pipeline moves artifacts from development through QA to production, with monitoring and retraining support. Vishnu is direct about the limits: sparse or delayed data can make a model impossible to build. The team filters customers through sales checks and domain experts before involving data scientists. Retraining raises alerts when metrics cross thresholds, but a person still decides whether to retrain.
The platform had to make messy data science repeatable across production paths
Innovyze works with unreliable sensors and sparse measurements from water infrastructure. Vishnu says a data scientist may clean data, drop sensor anomalies, interpolate values, and create features in a Jupyter notebook. Those same operations must run in real-time inference, batch inference, and retraining. The team therefore made data cleaning and feature engineering configuration-driven. The configuration captures how raw data becomes model input, so the process can be reproduced across environments and over time. This became necessary after handcrafted models worked in theory but failed to materialize in production with their first customers.
Innovyze combined a bespoke pipeline with Amazon SageMaker building blocks
The team first built a model repository that stored models alongside the feature engineering and data cleaning information needed to use them. Existing end-to-end products did not fit their workflow, so Innovyze built its own system and used Amazon SageMaker and AWS for many of the underlying pieces. The resulting pipeline covers feature engineering, cleaning, analysis, continuous retraining, CI/CD promotion from development to QA to production, production monitoring, and eventually automated retraining. Vishnu describes it as a work in progress rather than a finished package.
Vishnu prefers cloud-native tools when they reduce integration work
Vishnu separates MLOps approaches into cloud-agnostic and cloud-native patterns. A cloud-agnostic setup can combine separate tools for data testing, experiment tracking, notebooks, infrastructure, and CI/CD. That gives teams flexibility, but connecting and maintaining those tools becomes a full-time DevOps task. SageMaker provides a connected environment for feature selection, distributed training, reproducible training, and hyperparameter tuning. Vishnu says this helps the team reach business value faster and avoid maintaining infrastructure before it knows whether the approach fits. It also makes it cheaper to try and fail while the MLOps process is still evolving.
Digital twins are connected graphs of models for physical assets
Innovyze models the individual components of a water network, such as lakes, pumps, reservoirs, and treatment processes. Different asset types use different model families. Vishnu mentions random forest models, neural networks, and autoregressive models for demand forecasting. The models are connected into a directed acyclic graph, which becomes the digital twin. The system runs these models through Amazon SageMaker's real-time and batch inference capabilities, with a custom framework on top. A digital twin is built for each customer because networks and assets differ, while the deployment pipeline remains shared.
Data quality must be assessed before expensive modeling work begins
Some water measurements arrive once a day after manual collection and laboratory processing, while the system is expected to optimize operations in real time. Innovyze uses virtual or soft sensors in some cases, combining sparse measurements with other sensors that report more often. Vishnu says the broader MLOps task is to preserve the transformations that turn bad raw data into usable features. The team also checks data before a data scientist starts. Sales assesses customer maturity and willingness to install sensors, domain experts inspect sensor placement and anomalies, and automated model tests check whether useful correlation appears. Some failures still become clear only after modeling begins.
Reproducibility connects historical data with live sensor streams
Model development often starts with historical data supplied through email or Dropbox, while production receives a separate stream from live sensors. Units, sensor coverage, and data behavior can change between those sources. Vishnu says the team needs a centralized repository and connected data stream so historical and real-time data can support one reproducible training process. Rebuilding the model in a separate environment helps test whether the result was repeatable. It also supports compliance, model explanation, and later retraining. He points to 2020 as an example of data drift, because changed rainfall patterns and human behavior made earlier time-series models obsolete.
Collaboration needs access controls without forcing data scientists into a rigid process
Vishnu calls data science a messy, creative process and does not want to impose a fixed workflow that prevents iteration. The team began with local Jupyter notebooks, which made sharing and reproducibility difficult. SageMaker Studio improved collaboration while allowing access to be limited by region, contracts, and permissions. A centralized notebook server could expose too much data, especially when contractors join a team. Vishnu is open about the remaining gap: sharing a large notebook does not mean another person can simply run it successfully, because data structures, saved state, and generated configuration can be missing or inconsistent.
Automated retraining should follow model maturity and still keep a person involved
Innovyze has too many models to monitor and retrain manually, but Vishnu does not recommend full automation for every use case. The team emits metrics such as RMSE, MAPE, and MASE, with thresholds set during training. When drift pushes a metric beyond its threshold, the system sends an alert. A person reviews the result and clicks retrain. This is automated execution with human approval, rather than unattended retraining. Vishnu says the approach fits water assets because a pump usually keeps the same basic dynamics, while models such as fraud or churn prediction can face larger behavior changes and may need a redesigned model.
"The key is to enable data scientists to do whatever they want and make sure that's reproducible in a pipeline."Vishnu Prathish22:02
Who should watch
You are building a deployment system for many customer-specific models and need one pipeline to handle different model artifacts and configurations.
Your team is deciding between a cloud-native platform and a collection of open source tools, and you want to understand the integration and DevOps tradeoff.
You are considering automatic retraining and need a practical approach that keeps metric thresholds and human approval in the loop.
