Audio, music, and speech data need different processing choices, even when they share some machine learning techniques.
2
Small companies should move quickly from a rough prototype into production work, then build pipelines for repeatable experiments, evaluation, and deployment.
3
Large companies benefit from a separate engineering or MLOps team that treats research teams as customers and builds modular tools around their needs.
Summary
Valerio Velardo explains why audio ML has different production needs from image and language work. Music, environmental sound, and speech are related areas, but each has its own properties. Raw audio is too large for many models, so teams often convert wave files into representations such as spectrograms, mel spectrograms, or MFCCs. Velardo argues that smaller companies should let one team handle both research and productization, spending limited time on a working prototype before moving to clean code, infrastructure, experiment tracking, model registries, evaluation, and deployment. Larger companies can split research and engineering into two teams, with the MLOps team treating internal research groups as customers. He measures platform progress partly by the number of use cases it covers and relies on regular feedback from those teams. He also recommends modular interfaces and open-source components to reduce vendor lock-in and make changes local to one module.
Audio ML covers different problems that need different domain knowledge
Valerio places music, environmental sound, and speech within the wider audio AI field, while stressing that they differ in important ways. Environmental sound includes events such as breaking windows, city noise, or a passing siren, with tasks such as audio scene classification. Music systems may recognize chords, separate vocals and instruments, classify genres, judge whether music is happy or sad, or generate melodies and sounds. Shared techniques exist, but the data and tasks are not interchangeable. He says engineers need a deep understanding of how music differs from ordinary environmental noise and from speech before choosing how to process it.
Audio data must be transformed into compact representations before many models can use it
Raw wave files have very high dimensionality, especially when the audio is long, so passing them directly into a deep learning model is difficult. Valerio describes turning wave files into representations such as spectrograms and mel spectrograms. Fourier transforms and filter banks help produce these representations, while mel filter banks make the result closer to human perception. He also discusses MFCCs, or mel-frequency cepstral coefficients, which use transformations related to how the vocal tract produces speech. Earlier systems often relied on features such as zero-crossing rate and loudness. The research continues because teams want representations that are both compact and rich in perceptual information.
Small companies should keep research and productization close together
For a small startup, Valerio recommends one team that can handle AI audio or music research and then move the work into production. The first stage can be a short prototype, with imperfect code accepted because the goal is to show that the idea works. Once that point is reached, the team should shift to clean code, sound infrastructure, and repeatable practices. He says companies often spend their attention on the research while underestimating the work required to turn a promising prototype into a usable product. His advice is to spend at least as much time productizing the work as was spent prototyping it.
A production pipeline needs more than training code
Valerio describes an initial MLOps pipeline as something that can reproduce experiments, track training runs, keep models in a central registry, automate evaluation, and support deployment. In audio work, he also wants a shared framework with a high-level API for reading and transforming audio, such as converting files into mel spectrograms. Without common tooling, people on the same team can independently recreate the same transformations. That wastes time and can produce inconsistent or incorrect experiments. He is direct that moving fast with improvised code can result in more wasted work when the underlying data processing is wrong.
Large companies should treat MLOps as an internal product team
At larger companies, Valerio recommends separating research and engineering into two teams, while keeping them in close contact. The MLOps team should treat research groups as customers. That means discovering their needs, building a small solution, getting feedback, and improving it in short cycles. At Utopia Music, this may mean serving machine learning work across more than 10 or 15 business units, each with different use cases and sometimes different libraries. Valerio compares the platform team to a startup whose customers are internal teams. The relationship needs regular communication rather than a handoff between researchers and engineers.
Platform coverage is a better early measure than building impressive features
Valerio uses the number of use cases covered by a centralized platform as an early measure of progress. Different teams may work on highly specialized problems or use different tools, so the platform should first support the most useful common needs. This can be broken down into how many models can be built from configuration, how many can be evaluated with suitable metrics, and how much of deployment and monitoring is supported. He combines these measures with frequent conversations. Teams provide feedback at the start and end of sprints, which helps the platform group decide what to build next and test whether the result addresses real problems.
Education and shared work help research teams value production code
The main cultural problem Valerio encounters is that AI researchers often come from academic backgrounds and focus on making a model work. They may not see why clean, scalable, repeatable code matters. He addresses this through customer and feedback conversations, workshops that include researchers and engineers, and sessions about how MLOps can help with experiments and models. He does not want the two groups to avoid each other. Keeping researchers involved gives the engineering team a better understanding of their needs, while showing researchers how production practices can reduce work and make their experiments more useful.
Modular interfaces reduce lock-in and allow one platform to support different data types
Valerio designs platforms as collections of modules with high-level interfaces. Configuration, training, experimentation, tracking, evaluation, deployment, monitoring, and inference can be separate components, with specialized implementations for audio, language, or other data. A team can then add or remove modules for a particular use case. He says the full design cannot be decided at the start, so the platform needs broad interfaces that leave room for more specific decisions later. He prefers open-source tools because they reduce dependence on a paid service, but he also isolates third-party SDKs and APIs behind internal modules. That limits the code changes required when an external library changes.
"The real problems and challenges most of the time come from the other side, which is getting this amazing initial prototype that you have and putting it into production."Valerio Velardo24:40
Who should watch
You are building audio or music ML systems and need to decide how much work belongs in data processing, research, and production engineering.
Your research team has working prototypes, but experiments are hard to reproduce and models are difficult to evaluate or deploy.
You are designing an internal ML platform for teams with different use cases and want to limit dependence on external tools.
