Baking model files into Docker images makes production rollbacks and reproducibility easier because the model and its serving software have one recorded version.
2
Downloading models at startup reduces the number of container images, but it adds a dependency on the model registry and can make production fail when that registry is unavailable.
3
A central service built with Kaniko and Kubernetes can let data scientists publish MLflow models while DevOps controls secure, multi-tenant image builds.
Summary
Luke Marsden explains how to package a trained machine learning model and its serving code into a Docker image. He starts with a Flask server and a scikit-learn model, then compares baking model files into an image with downloading them from MLflow or object storage when the server starts. His preference is baking because it gives one versioned artifact, makes rollbacks clearer, and keeps the model registry outside the production serving path. He also describes the security problem with building Docker images inside Kubernetes, especially when mounting the Docker socket gives users effective root access to the host. His Chassis project uses Kaniko to build images in a centralized Kubernetes service. An MLflow model can be published through a Python SDK, have its dependencies and files baked into an image, and be pushed to Docker Hub or deployed to KFServing. The talk ends with advice for ML and DevOps teams to share responsibility through an internal model-building service.
A model is a set of learned files that needs its runtime dependencies
Marsden describes a machine learning model as files containing learned weights, together with files that describe the network architecture or other model structure. Docker packages those files with the server and dependencies needed to run them. A Docker image can start from a Debian or Ubuntu base layer, add a model server such as TensorFlow Serving or a Flask Python server, and then add the model weights. This creates a portable unit that can run in development or production without depending on whatever library versions happen to be installed on a server.
The first demonstration uses a Flask REST server that opens a model file at startup and calls model.predict when it receives a POST request. The model is a pre-trained scikit-learn model saved as model.pickle. Marsden adds that file to the Dockerfile with a COPY instruction, builds the image with Docker build -t, runs it in the background, and sends prediction features over HTTP. The example shows the basic workflow: write serving code, include the model file, build the image, and run the same artifact wherever the container is deployed.
Downloading model files at startup adds an operational dependency
Marsden compares baking the model into the image with having the model server download weights from a location such as MLflow when it starts. The downloading approach can reduce the number of images because one generic image can run multiple model versions. It also means the MLflow server or another model store must be online whenever a container starts. If Kubernetes restarts a pod while that service is unavailable, the model can fail to come back even though the rest of the deployment appears healthy.
Baking makes model updates and rollbacks easier to track
For updates and rollbacks, Marsden strongly prefers baking. With a separately downloaded model, production depends on a tuple containing the Docker image and the external model files. Operators must track both the serving software version and the model version, and a rushed rollback could restore one without the other. A baked image records the libraries, server, and model together, so a previous known-good image rolls back both parts. He also warns against replacing model files in S3 and restarting containers without recording which version was active.
Low-latency serving requires model reuse and concurrency planning
In response to a question about prediction responses below 10 milliseconds, Marsden says the server should load and initialize the model once at startup, then reuse the cached model object for requests. The model itself must also be benchmarked, since a model that takes longer than the target cannot meet the target through server changes alone. Production compute, including GPUs where appropriate, may help. The server must handle concurrent requests as well, because a naive Python process that handles one request at a time can develop severe tail latency under load.
Building Docker inside a container creates a serious security problem
Data scientists often work inside Kubernetes pods, such as Kubeflow Notebooks, where Docker does not run naturally. Building images on individual laptops creates ownership and continuity problems if someone leaves or loses their machine. Mounting the host Docker socket into a container is also dangerous because it can give users root access to the machine running the cluster. Marsden presents Kaniko as an alternative. It executes Dockerfile commands in its own container, then packages the resulting filesystem and sends it to a container registry without requiring the Docker daemon or socket.
Chassis turns secure model builds into a shared service
Chassis, developed with the Modzy team, provides a centralized model-build service for multi-tenant Kubernetes clusters. A data scientist publishes an MLflow model through chassis.ml.publish. The Chassis API starts a Kubernetes job that uses Kaniko to install the model's specified dependencies, bake in the model files, and push the resulting image to a registry or the Modzy platform. The service also provides a Python SDK, so data scientists do not need to learn Docker or manage the image-build environment themselves.
An internal build service gives ML teams autonomy without removing DevOps control
Marsden recommends that DevOps teams offer a model-building service that ML teams can use through a Python API. The ML team can build its own production image without sending model files to DevOps or asking data scientists to manage infrastructure. DevOps can control the build process, its security settings, and its production requirements. He argues that this reduces the number of people involved in each model release, which otherwise increases the time required to ship a model.
"And even if I did want to run kind of Docker from inside Kubernetes then you have to do this thing that's called bind mounting the Docker socket and this is something this is a technique that gives security and compliance people like a big headache."Luke Marsden49:59
Who should watch
You need to decide whether production model artifacts should be baked into images or downloaded when services start.
Your ML team works in Kubernetes and needs a way to build images without giving data scientists access to the Docker socket.
DevOps and ML teams are sharing model releases and want clearer ownership, versioning, and rollback procedures.
