Joel Grus still thinks Jupyter notebooks create serious hidden-state and reproducibility problems, although newer tools address some of those issues.
2
At Capital Group, security rules constrain the team's MLOps choices to AWS services, so the team built its own abstractions around SageMaker's bring-your-own-algorithm mode.
3
Grus treats machine learning code as production code and uses mandatory code review to improve correctness, readability, and software engineering habits.
Summary
Joel Grus explains how he moved from mathematics and quantitative finance into data science, software engineering, and machine learning engineering. He revisits his 2018 JupyterCon criticism of notebooks, saying their interactivity and low barrier to entry are useful, while hidden state makes them harder to reason about as work grows. He points to Streamlit and reactive notebook systems as alternatives for some notebook use cases. At Capital Group, his team works under strict security controls and uses AWS services such as Lambda and SageMaker. Rather than coupling their workflow to SageMaker's standard training API, they use the bring-your-own-algorithm path and run the same Docker images locally on small data sets before using larger cloud jobs. Grus also discusses the limits of MLOps automation. Retraining and monitoring depend on the business problem and the errors a model makes. His strongest engineering practice is mandatory code review, with emphasis on readable code and production standards.
Grus entered data science through a long route from mathematics and finance
Joel Grus studied pure mathematics, entered a PhD program, and then moved into quantitative finance after leaving graduate school. When a hedge fund failed around 2006, he joined Faircast, an online travel company that predicted whether airfares would rise or fall. His first work there involved SQL queries, spreadsheets, and pivot tables rather than model development. The company called its machine learning staff data miners, and they built their own C++ toolkits for Sun Grid Engine. Around 2011, Grus moved toward data science when the field was still new. At one startup, a CEO hired him after showing him a SQL query and asking whether he understood it.
His objection to notebooks began with a capable engineer trapped by hidden state
While working at the Allen Institute for Artificial Intelligence, Grus helped a senior software engineer who was struggling with Python. Her problems came from a Jupyter notebook rather than from Python itself. She had run cells in an order that left her unable to reason about the values of her variables or what the code was doing. Grus said the experience showed how notebook features can confuse even a very talented engineer. He then wrote a frustrated tweet offering to give a talk about why notebooks were bad. JupyterCon invited him to submit it, and he spent more time researching and writing that talk than he had spent on any previous presentation.
Notebook criticism has led to better tools and alternatives for specific use cases
Grus says he would give an updated version of his notebook talk today. He accepts that notebooks are useful because they are interactive and have a low barrier to entry, especially when someone wants to explore a model. He also says the problems become worse as notebook use grows, because hidden state and line-by-line execution make the work harder to understand. Newer tooling helps with source control and code review, but he is unsure whether it removes enough friction. Streamlit can turn Python code into a web application, which covers cases where a notebook was being used mainly to let someone interact with a model. He also discusses reactive kernels that track dependencies between cells and rerun affected cells automatically.
Security rules determine much of Capital Group's MLOps stack
Capital Group is heavily committed to AWS and has strict security requirements because it is an investment company. Grus cannot freely choose the best tool for each problem. His team uses approved platform services such as Lambda functions, SageMaker endpoints, and message queues. When raw data in S3 needs preprocessing, they use Lambda and notifications to trigger the work. For model training, the security team effectively gave them one approved choice, SageMaker. This makes tool selection simple, although it also limits the range of systems they can consider. Grus describes the resulting workflows as bespoke systems built from standard platform components.
The team built around SageMaker's bring-your-own-algorithm mode
Grus did not want to depend on SageMaker's standard inference API because he could not assume every future model would fit its interface. His team instead used the bring-your-own-algorithm mode, which requires a Docker image to follow an incompletely documented contract for training, serving, ports, and data formats. They spent substantial time reverse engineering that contract and adding abstractions around it. The approach lets them run training images locally, in SageMaker, or in theory on an EC2 instance. Grus values this because a new model can first be tested locally on a very small data set. If it trains correctly there, the team can send the same image to the cloud for a larger run.
MLOps cannot decide model retraining without business judgment
Grus considers his team immature in its MLOps thinking because it is still focused on getting models deployed and used. He says monitoring and retraining decisions depend on the business problem, the kinds of errors a model makes, and how a newer model would perform. Tools can help track production predictions, but they cannot fully determine when a model should be retrained, replaced, or decommissioned. The team is also approaching the point where deployed models create service obligations. With a small team, Grus is concerned about supporting models under an SLA without having to carry pagers around the clock.
Data scientists should treat analytics code as production code
Grus rejects the idea that data science code is exempt from software engineering standards because it is only used for analysis. If analysis informs a business decision, he considers that decision to be something the organization is effectively shipping. He believes data scientists should become more like software engineers and should treat their code as production code. He is skeptical of people who use the phrase 'this isn't production' to avoid those standards. He also says his earlier prediction that standalone data science would disappear into machine learning engineering was wrong. He still expects data scientists to have a role, although he believes their engineering habits need to improve.
Mandatory code review is Grus's main way to teach engineering practice
Every change on Grus's team must receive a code review, including his own code. He uses reviews to explain why a different implementation would be clearer, safer, or easier to maintain. His comments often concern function names, variable names, convoluted loops, and readability rather than obvious correctness errors. He prefers tools such as Black and Flake8 to handle formatting and basic style checks so that human review can focus on understanding the code. Early reviews are often done by Grus because the team members work in separate areas such as AWS infrastructure, NLP, and search. He wants more people to understand the whole stack so reviews do not depend on one person.
"Every one of her problems were not caused by Python, were not caused by her misunderstanding of Python, but they were caused by the hidden state in the notebook."Joel Grus10:05
Who should watch
You are deciding whether notebooks should remain part of a team's workflow as experiments become shared code and scheduled jobs.
Your organization has strict cloud and security requirements, and you need to design an MLOps system around approved AWS services.
You manage data scientists who need practical guidance on code review, readability, local testing, and production standards.
