# Bringing Audio ML Models into Production

Valerio Velardo, Utopia Music | MLOps Coffee Sessions | Episode 90 | 50:42
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=fGKZljg_SQI
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/bringing-audio-ml-models-into-production
Published: 2022-04-03
Tags: deployment, experiment-tracking, platform-teams

## TL;DR
- Audio, music, and speech data need different processing choices, even when they share some machine learning techniques.
- Small companies should move quickly from a rough prototype into production work, then build pipelines for repeatable experiments, evaluation, and deployment.
- Large companies benefit from a separate engineering or MLOps team that treats research teams as customers and builds modular tools around their needs.

## Summary
Valerio Velardo explains why audio ML has different production needs from image and language work. Music, environmental sound, and speech are related areas, but each has its own properties. Raw audio is too large for many models, so teams often convert wave files into representations such as spectrograms, mel spectrograms, or MFCCs. Velardo argues that smaller companies should let one team handle both research and productization, spending limited time on a working prototype before moving to clean code, infrastructure, experiment tracking, model registries, evaluation, and deployment. Larger companies can split research and engineering into two teams, with the MLOps team treating internal research groups as customers. He measures platform progress partly by the number of use cases it covers and relies on regular feedback from those teams. He also recommends modular interfaces and open-source components to reduce vendor lock-in and make changes local to one module.

## Key ideas
### Audio ML covers different problems that need different domain knowledge
[08:32](https://www.youtube.com/watch?v=fGKZljg_SQI&t=512s)
Valerio places music, environmental sound, and speech within the wider audio AI field, while stressing that they differ in important ways. Environmental sound includes events such as breaking windows, city noise, or a passing siren, with tasks such as audio scene classification. Music systems may recognize chords, separate vocals and instruments, classify genres, judge whether music is happy or sad, or generate melodies and sounds. Shared techniques exist, but the data and tasks are not interchangeable. He says engineers need a deep understanding of how music differs from ordinary environmental noise and from speech before choosing how to process it.

### Audio data must be transformed into compact representations before many models can use it
[18:18](https://www.youtube.com/watch?v=fGKZljg_SQI&t=1098s)
Raw wave files have very high dimensionality, especially when the audio is long, so passing them directly into a deep learning model is difficult. Valerio describes turning wave files into representations such as spectrograms and mel spectrograms. Fourier transforms and filter banks help produce these representations, while mel filter banks make the result closer to human perception. He also discusses MFCCs, or mel-frequency cepstral coefficients, which use transformations related to how the vocal tract produces speech. Earlier systems often relied on features such as zero-crossing rate and loudness. The research continues because teams want representations that are both compact and rich in perceptual information.

### Small companies should keep research and productization close together
[23:00](https://www.youtube.com/watch?v=fGKZljg_SQI&t=1380s)
For a small startup, Valerio recommends one team that can handle AI audio or music research and then move the work into production. The first stage can be a short prototype, with imperfect code accepted because the goal is to show that the idea works. Once that point is reached, the team should shift to clean code, sound infrastructure, and repeatable practices. He says companies often spend their attention on the research while underestimating the work required to turn a promising prototype into a usable product. His advice is to spend at least as much time productizing the work as was spent prototyping it.

### A production pipeline needs more than training code
[25:27](https://www.youtube.com/watch?v=fGKZljg_SQI&t=1527s)
Valerio describes an initial MLOps pipeline as something that can reproduce experiments, track training runs, keep models in a central registry, automate evaluation, and support deployment. In audio work, he also wants a shared framework with a high-level API for reading and transforming audio, such as converting files into mel spectrograms. Without common tooling, people on the same team can independently recreate the same transformations. That wastes time and can produce inconsistent or incorrect experiments. He is direct that moving fast with improvised code can result in more wasted work when the underlying data processing is wrong.

### Large companies should treat MLOps as an internal product team
[28:46](https://www.youtube.com/watch?v=fGKZljg_SQI&t=1726s)
At larger companies, Valerio recommends separating research and engineering into two teams, while keeping them in close contact. The MLOps team should treat research groups as customers. That means discovering their needs, building a small solution, getting feedback, and improving it in short cycles. At Utopia Music, this may mean serving machine learning work across more than 10 or 15 business units, each with different use cases and sometimes different libraries. Valerio compares the platform team to a startup whose customers are internal teams. The relationship needs regular communication rather than a handoff between researchers and engineers.

### Platform coverage is a better early measure than building impressive features
[31:14](https://www.youtube.com/watch?v=fGKZljg_SQI&t=1874s)
Valerio uses the number of use cases covered by a centralized platform as an early measure of progress. Different teams may work on highly specialized problems or use different tools, so the platform should first support the most useful common needs. This can be broken down into how many models can be built from configuration, how many can be evaluated with suitable metrics, and how much of deployment and monitoring is supported. He combines these measures with frequent conversations. Teams provide feedback at the start and end of sprints, which helps the platform group decide what to build next and test whether the result addresses real problems.

### Education and shared work help research teams value production code
[36:08](https://www.youtube.com/watch?v=fGKZljg_SQI&t=2168s)
The main cultural problem Valerio encounters is that AI researchers often come from academic backgrounds and focus on making a model work. They may not see why clean, scalable, repeatable code matters. He addresses this through customer and feedback conversations, workshops that include researchers and engineers, and sessions about how MLOps can help with experiments and models. He does not want the two groups to avoid each other. Keeping researchers involved gives the engineering team a better understanding of their needs, while showing researchers how production practices can reduce work and make their experiments more useful.

### Modular interfaces reduce lock-in and allow one platform to support different data types
[42:51](https://www.youtube.com/watch?v=fGKZljg_SQI&t=2571s)
Valerio designs platforms as collections of modules with high-level interfaces. Configuration, training, experimentation, tracking, evaluation, deployment, monitoring, and inference can be separate components, with specialized implementations for audio, language, or other data. A team can then add or remove modules for a particular use case. He says the full design cannot be decided at the start, so the platform needs broad interfaces that leave room for more specific decisions later. He prefers open-source tools because they reduce dependence on a paid service, but he also isolates third-party SDKs and APIs behind internal modules. That limits the code changes required when an external library changes.

## Notable quotes
- Valerio Velardo: "The problem with audio data is that if you're dealing with raw audio, so for example a wave file, this is highly highly dimensional." (18:18)
- Valerio Velardo: "We need something that proves that the thing that we're trying to build actually works to a certain extent." (24:15)
- Valerio Velardo: "The real problems and challenges most of the time come from the other side, which is getting this amazing initial prototype that you have and putting it into production." (24:40)
- Valerio Velardo: "Every time I do an import of a third party library, well, I'm kind of getting married in a sense with that library." (49:19)

## Tools & references mentioned
- MLOps Community
- Utopia Music
- Spotify
- The Sound of AI
- University of Huddersfield
- digital signal processing
- Fourier transform
- MFCCs

## Who should watch
- You are building audio or music ML systems and need to decide how much work belongs in data processing, research, and production engineering.
- Your research team has working prototypes, but experiments are hard to reproduce and models are difficult to evaluate or deploy.
- You are designing an internal ML platform for teams with different use cases and want to limit dependence on external tools.

## Editor's note

Valerio Velardo says improvised audio code can create more wasted work when the underlying data processing is wrong, so teams need repeatable production practices after prototyping. ZenML records each pipeline run's steps, inputs, outputs, and code version, letting teams trace an artifact or model back to the data and code that produced it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Scaling AI in Production](https://mlopstalks.com/talks/scaling-ai-in-production) (Srivatsan Srinivasan, AIEngineering, 51:56)
- [Tecton Round-table // Get your ML Application Into Production](https://mlopstalks.com/talks/tecton-round-table-get-your-ml-application-into-production) (Kevin Stumpf, Derek Salama, Eddie Esquivel & Isaac Cameron, Tecton, 55:42)
- [Just Build It! Tips for Making ML Engineering and MLOps Real](https://mlopstalks.com/talks/just-build-it-tips-for-making-ml-engineering-and-mlops-real) (Andy McMahon, NatWest Group, 48:17)
- [Luigi in Production](https://mlopstalks.com/talks/luigi-in-production) (Luigi Patruno, ML in Production, 47:23)
- [Extending AI: From Industry to Innovation](https://mlopstalks.com/talks/extending-ai-from-industry-to-innovation) (Sophia Rowland & David Weik, SAS, 1:01:37)