# Vertex AI Workshop

Sascha Heyer, DoiT | MLOps Meetup | Episode 117 | 1:24:30
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=VY03WeGTd8U
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/vertex-ai-workshop
Published: 2022-12-30
Tags: model-registry, model-serving, orchestration, training-pipelines

## TL;DR
- Vertex AI can train models with managed infrastructure, custom containers, selected machine types, GPUs or TPUs, distributed training, logging, and debugging options.
- Custom containers give teams control over training and serving code, versioning, dependencies, preprocessing, logging, and model artifacts.
- Vertex AI Pipelines can connect training, container building, model upload, endpoint deployment, experiments, and model lineage without teams managing their own Kubernetes cluster.

## Summary
Sascha Heyer gives a hands-on introduction to using Vertex AI across model training, serving, and pipelines. He starts with a training application packaged in a custom container, then shows how to build the image with Cloud Build, push it to the Google Container Registry, and submit a training job through the Vertex AI SDK. The training job can specify machine types and accelerators, while Vertex AI manages the infrastructure. He then explains how to serve a trained model with a pre-built or custom container, including the HTTP health and prediction endpoints required by Vertex AI. The final section turns these steps into a Kubeflow pipeline. Components can train a model, build a serving container, upload the model to the Model Registry, and deploy it to an endpoint. Vertex AI records artifacts, metrics, experiments, and model lineage. Heyer is also direct about limits, including request sizes, model sizes, and the fact that endpoints do not scale down to zero.

## Key ideas
### Managed training moves teams beyond notebooks without requiring cluster administration
[03:15](https://www.youtube.com/watch?v=VY03WeGTd8U&t=195s)
Sascha describes the common path from a notebook proof of concept to larger datasets and production deployment. Vertex AI provisions the infrastructure needed for training, so teams do not have to manage a virtual machine or a Kubernetes cluster themselves. Training jobs can specify machine types and accelerators such as GPUs or TPUs, and distributed training is available for large datasets. The service also provides logging and debugging options. Sascha says the experience is similar to local training, while allowing the workload to run at a larger scale. He recommends moving the implementation into a code repository rather than continuing to train manually from a notebook.

### Custom training containers keep code, dependencies, and versions together
[06:09](https://www.youtube.com/watch?v=VY03WeGTd8U&t=369s)
Vertex AI needs the training code and its dependencies packaged before it can run a job. Sascha presents two options, a custom Docker container or a Python source distribution stored in Google Cloud Storage. He recommends the custom container because the image is versioned in the container registry and can be used in a CI/CD process. The same image can run in environments that support Docker, which reduces dependency differences between local development and training. Older training versions remain available, so a team can roll back after a bad change. The workflow is to create a Dockerfile, build the image with Cloud Build, push it to the Google Container Registry, and submit the image as a training job.

### Training jobs can be sized and inspected through the SDK or other interfaces
[19:18](https://www.youtube.com/watch?v=VY03WeGTd8U&t=1158s)
For a training job, Sascha defines a region, a display name, a worker pool, a machine type, and an accelerator. His example uses an N1 standard 8 machine with an Nvidia Tesla T4 GPU. Vertex AI provisions the requested hardware and shows the selected machine, container, elapsed time, and CPU and GPU utilization. Those measurements help a team decide whether the machine is oversized or whether more capacity is needed. Jobs can be started through the gcloud command, the Vertex AI SDK, another client library, the Google API, or the user interface. Multiple jobs can run in parallel when the project needs several models or experiments.

### Serving requires a model registry entry and an endpoint deployment
[32:04](https://www.youtube.com/watch?v=VY03WeGTd8U&t=1924s)
After training, Sascha uploads the model to Google Cloud Storage and explains three serving choices: a Google-provided container, a custom container, or a custom prediction routine. The custom container gives control over prediction code, logging, preprocessing such as tokenization, and hosting multiple models at one endpoint. The deployment flow has three steps. First, create an endpoint. Second, upload the model to the Vertex AI Model Registry. Third, deploy the model to the endpoint. Deployment can set the machine type, accelerator, and traffic split. A team can send some traffic to a newer model version while retaining an older version, then change the split after checking the result.

### A custom prediction container has a small HTTP contract
[38:19](https://www.youtube.com/watch?v=VY03WeGTd8U&t=2299s)
A custom serving container must run a Dockerized application that listens for requests, normally on 0.0.0.0 and port 8080. It must expose a health-check path so Vertex AI can determine whether the model is ready, along with a prediction path. Requests and responses use JSON. The rest of the implementation is flexible. Sascha's example uses FastAPI, loads a tokenizer and model, performs preprocessing, runs prediction, and formats the result. He recommends putting model artifacts and tokenizers inside the container. Downloading them when each instance starts can add cold-start time, especially when autoscaling creates several instances.

### Vertex AI has practical serving limits that affect architecture choices
[56:24](https://www.youtube.com/watch?v=VY03WeGTd8U&t=3384s)
Sascha names several limitations as of the workshop. Request and response payloads are limited to 1.5 megabytes, which can matter for large images. He suggests reducing or encoding inputs when necessary. He also describes a model size limit of 15 gigabytes at that time, while newer Transformer models can be larger. Vertex AI endpoints do not scale down to zero, so deploying many models can create a cost problem. Cloud Run or Kubernetes can be alternatives when a project does not need Vertex AI features or needs to work around a specific limit. The choice depends on the team's requirements and its ability to maintain virtual machines or Kubernetes.

### Pipelines turn manual training and deployment steps into reusable components
[1:02:29](https://www.youtube.com/watch?v=VY03WeGTd8U&t=3749s)
Sascha introduces Vertex AI Pipelines as a managed way to run Kubeflow or TensorFlow Extended pipelines. A pipeline is made from components, and each component performs one step such as preprocessing, training, evaluation, container building, model upload, or deployment. In his example, a Python function becomes a component through a decorator. The pipeline connects outputs from one component to inputs of the next, then a compiler creates a pipeline specification that Vertex AI can run. His larger example trains a model, builds a serving container, and deploys the resulting model. The pipeline can also request a GPU and disable caching for demonstration purposes.

### Pipeline metadata supports experiments, lineage, reuse, and rollback
[1:12:11](https://www.youtube.com/watch?v=VY03WeGTd8U&t=4331s)
Vertex AI Pipelines records artifacts and metrics that move into and out of components. Sascha uses this metadata to answer which data produced a model and which model came from a particular pipeline run. Pipeline runs can record metrics such as accuracy, and Vertex AI Experiments allows runs and parameter values to be compared. The Model Registry stores multiple model versions, so a team can identify an older version and roll back when needed. Sascha also mentions pre-built components for Google services, reusable component specifications, parameterized pipelines, and scheduled execution through Cloud Scheduler calling a Cloud Function.

## Notable quotes
- Sascha Heyer: "We should keep things simple until we make them complicated." (02:50)
- Sascha Heyer: "Vertex AI helps you to automatically scale and automatically provision the infrastructure needed to train your models." (05:02)
- Sascha Heyer: "I always recommend to put the model into your data container itself to reduce the cold start times." (41:50)
- Sascha Heyer: "The pipeline itself contains the components we created, again defined by a pipeline decorator." (59:55)
- Sascha Heyer: "Vertex AI Pipelines provides a serverless product to run a Kubeflow or TFX pipeline." (1:03:12)

## Tools & references mentioned
- Vertex AI
- Google Cloud Storage
- Google Container Registry
- Cloud Build
- Kubernetes
- Kubeflow
- TensorFlow Extended
- TensorFlow
- XGBoost
- scikit-learn
- Hugging Face
- FastAPI
- Cloud Run
- Cloud Scheduler
- Cloud Functions
- Google Cloud
- AWS
- SageMaker
- Vertex AI Model Registry
- Vertex AI Experiments
- Cloud TPU
- Nvidia Tesla T4

## Who should watch
- You are moving a model from a notebook into repeatable cloud training and need a concrete Vertex AI workflow.
- Your team wants managed model endpoints but needs custom preprocessing, logging, container images, or traffic splits.
- You are deciding whether a managed Vertex AI pipeline is preferable to maintaining Kubeflow, Kubernetes, or virtual machines yourself.

## Editor's note

Sascha Heyer shows how manual training and deployment steps can become reusable Vertex AI Pipeline components, with metadata recording which data produced a model. ZenML records each pipeline run's steps, inputs, outputs, and code version, so artifacts and models can be traced to the data and code that produced them. Its pipelines can run on different infrastructure through configuration.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Ship Agents: A Virtual Conference Track 2](https://mlopstalks.com/talks/ship-agents-a-virtual-conference-track-2) (Adam Boaz Becker & Sarmad Absil, Trial Cyber & Divia Mahajan, Amazon Alexa, 1:38:26)
- [AWS Trainium and Inferentia](https://mlopstalks.com/talks/aws-trainium-and-inferentia) (Kamran Khan, Annapurna ML & Matthew McClean, AWS, Annapurna Labs, 45:23)
- [AI Is Fast. AI Projects Are Slow. Let's Fix That.](https://mlopstalks.com/talks/ai-is-fast-ai-projects-are-slow-lets-fix-that) (JRocketRide's Joe Maionchi, 56:48)
- [Building Out GPU Clouds](https://mlopstalks.com/talks/building-out-gpu-clouds) (Mohan Atreya, Rafay Systems, 47:58)
- [The Next Evolution of AI Agents](https://mlopstalks.com/talks/the-next-evolution-of-ai-agents) (Alon Horev, Vast Data, 13:38)