# How to Actually Use Cost Effective AI in Your Business

Eddie Mattia, Outerbounds & Scott Perry, AWS | MLOps Community Mini Summit #9 | 49:05

Source: https://www.youtube.com/watch?v=nWrgiwsLgO8
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/how-to-actually-use-cost-effective-ai-in-your-business
Published: 2024-11-06
Tags: cost, deployment, gpus, orchestration

## TL;DR
- Eddie Mattia shows how Metaflow connects AWS Trainium jobs to data, orchestration, monitoring, versioning, and deployment workflows.
- Scott Perry explains how AWS Trainium and Inferentia use purpose-built hardware and the Neuron SDK to run deep learning workloads with lower cost and energy use.
- Both speakers argue that teams should access large AI compute through ordinary workflow systems such as AWS Batch or Kubernetes instead of managing clusters by hand.

## Summary
The session covers cost-efficient AI infrastructure from workflow orchestration down to custom accelerator hardware. Eddie Mattia walks through an integration between Metaflow and AWS Trainium. Metaflow can deploy workflows, connect them to AWS Batch, configure training resources, track runs, monitor Neuron cores, and manage model artifacts. He places this work inside a broader machine learning stack that still needs data storage, versioning, deployment, and model-serving systems. Scott Perry then explains AWS Inferentia and Trainium, including their hardware architecture, instance types, and the Neuron SDK. The SDK integrates with PyTorch, Hugging Face Transformers, Optimum Neuron, and several AWS services. Perry also discusses Neuron tools for monitoring and the Neuron Kernel Interface for writing lower-level accelerator code. The closing discussion focuses on making large distributed workloads accessible through batch jobs and Kubernetes, so smaller teams can use them without operating clusters manually.

## Key ideas
### Metaflow connects Trainium to an ordinary cloud workflow
[00:00](https://www.youtube.com/watch?v=nWrgiwsLgO8&t=0s)
Eddie Mattia introduces an integration that lets Metaflow run workflows on AWS Trainium through AWS Batch. The setup uses CloudFormation to deploy the Metaflow stack and configure the job queues and training resources. Metaflow can also run on other clouds, on-premise systems, Kubernetes, and different compute providers. The goal is to give data scientists a familiar Python workflow while dispatching jobs to specialized AWS hardware when needed.

### The surrounding machine learning stack still matters for large models
[06:33](https://www.youtube.com/watch?v=nWrgiwsLgO8&t=393s)
Mattia argues that large language model workloads do not remove the ordinary layers of a machine learning system. Teams still need storage for datasets, metadata, checkpoints, and evaluation results. They need ways to connect workflows to compute, package dependencies, version runs, deploy models, and expose models through usable applications. Metaflow provides a way to connect these layers while allowing the modeling layer to change as new architectures and techniques appear.

### Metaflow makes accelerator jobs configurable and observable
[08:03](https://www.youtube.com/watch?v=nWrgiwsLgO8&t=483s)
Metaflow lets users declare runtime needs in Python, including CPU counts, Trainium or Inferentia devices, Docker images, and software dependencies. Mattia describes a monitoring integration that runs the Neuron Monitor at a chosen frequency and displays accelerator utilization in the Metaflow UI. This helps users see whether their functions are using the available Neuron cores effectively while they optimize a workflow for production.

### Smaller accelerator instances are useful for testing before large training runs
[11:31](https://www.youtube.com/watch?v=nWrgiwsLgO8&t=691s)
Mattia recommends thinking about scale as part of the workflow. A team can first test whether its Trainium environment works on smaller training instances before moving to the larger systems needed for pre-training a model. He says the smaller devices can cost substantially less, which makes them useful for checking configuration and execution before committing to a large distributed job.

### AWS designed Trainium and Inferentia for different parts of the model lifecycle
[19:39](https://www.youtube.com/watch?v=nWrgiwsLgO8&t=1179s)
Scott Perry explains that Inferentia targets inference and Trainium targets training. AWS launched Inferentia in 2019 for workloads such as ResNet and YOLO, followed by Inferentia2 for larger Transformer and generative AI models. Trainium launched in 2022 as a training-focused chip for large distributed workloads. Perry gives AWS-reported comparisons of up to 40 percent better price performance for Inferentia2 and up to 50 percent training-cost savings for Trainium against comparable EC2 instances.

### The accelerator architecture is specialized rather than a general-purpose GPU
[22:08](https://www.youtube.com/watch?v=nWrgiwsLgO8&t=1328s)
Perry describes Inferentia2 and Trainium as chips built around the operations common in deep learning. Their components include tensor engines for matrix operations, engines for vector and scalar operations, general-purpose SIMD processors, high-bandwidth memory, collective communication hardware, and links between chips. The general-purpose processors let users write C code for operations that were not known when the chips were designed. This gives developers an additional path when a model needs an operation outside the standard compiler flow.

### The Neuron SDK lets existing framework code run on AWS accelerators
[30:33](https://www.youtube.com/watch?v=nWrgiwsLgO8&t=1833s)
The Neuron SDK includes a runtime, framework integrations, a compiler, monitoring tools, profiling, and the Neuron Kernel Interface. Perry says users can move PyTorch models and tensors onto the accelerator through framework integration rather than rewriting the whole model. The compiler extracts computation graphs, optimizes them for Trainium or Inferentia, and produces executable artifacts. Optimum Neuron extends the Hugging Face Transformers and Trainer experience to these chips.

### Batch and Kubernetes reduce the amount of cluster management required
[39:42](https://www.youtube.com/watch?v=nWrgiwsLgO8&t=2382s)
In the closing discussion, the speakers describe a shift away from logging into a cluster and manually launching distributed jobs. AWS Batch can provide ephemeral Trainium and Inferentia capacity when a job needs it. Kubernetes and EKS can also use accelerator node groups that scale for machine learning workloads. Their point is that teams can submit large jobs through the same kinds of systems used for other software, while Metaflow handles workflow tracking, checkpointing, and orchestration.

## Notable quotes
- Eddie Mattia: "The idea is that we need to be able to connect these different computer environments and all of the things that we talked about at the data layer and all these other layers." (08:48)
- Scott Perry: "With Trainium we're able to drive up to 50% savings on training costs compared to comparable EC2 instances." (21:32)
- Scott Perry: "You could basically take your PyTorch model that you have already working today, move the model and move your tensors that are going into the model onto XLA devices and run your model as normal." (31:09)
- Eddie Mattia: "You can start with a team of two people and you can just submit some batch jobs and not actually have to think about kernels." (45:14)

## Tools & references mentioned
- Outerbounds
- Metaflow
- AWS Trainium
- AWS Inferentia
- Inferentia2
- AWS Batch
- AWS CloudFormation
- AWS Neuron SDK
- Neuron Monitor
- Neuron Top
- Neuron LS
- Neuron Kernel Interface
- Optimum Neuron
- Hugging Face
- PyTorch
- TensorFlow
- XLA
- AWS EC2
- Amazon EKS
- Kubernetes
- AWS SageMaker
- AWS Step Functions
- Argo Workflows
- Apache Airflow
- Ray
- Weights & Biases
- Lightning
- TGI
- vLLM
- TorchServe
- Ray Serve
- Snowflake
- AWS S3
- Llama 2
- ResNet
- YOLO
- Ninjatech
- Leonardo AI
- AWS Lambda

## Who should watch
- You are deciding whether specialized AWS accelerators can fit into an existing PyTorch or Hugging Face workflow.
- Your team needs to run large training or inference jobs but does not want to operate a cluster manually.
- You use Metaflow or AWS Batch and want practical details on tracking, monitoring, and scaling Trainium workloads.

## Editor's note

Eddie Mattia says large language model workloads still need storage, metadata, checkpoints, versioning, deployment, and ways to connect workflows to compute. ZenML records each pipeline run's steps, inputs, outputs, and code version, so a model or artifact can be traced to the data and code that produced it. The same pipeline code can then run on different infrastructure chosen by configuration.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [AWS Trainium and Inferentia](https://mlopstalks.com/talks/aws-trainium-and-inferentia) (Kamran Khan, Annapurna ML & Matthew McClean, AWS, Annapurna Labs, 45:23)
- [The Cost of AI: FinOps Strategies for Intelligent Agents](https://mlopstalks.com/talks/the-cost-of-ai-finops-strategies-for-intelligent-agents) (Advait Patel, Broadcom, 31:05)
- [AI Is Fast. AI Projects Are Slow. Let's Fix That.](https://mlopstalks.com/talks/ai-is-fast-ai-projects-are-slow-lets-fix-that) (JRocketRide's Joe Maionchi, 56:48)
- [Cutting Costs with Artificial Intelligence](https://mlopstalks.com/talks/cutting-costs-with-artificial-intelligence) (, 12:36)
- [Why You Need More Than Airflow](https://mlopstalks.com/talks/why-you-need-more-than-airflow) (Ketan Umare, Union.ai, 1:11:12)