Daniel Jeffries built an ambient music generator with Google Brain's Magenta Music Transformer, using MIDI to model musical notes and structure.
2
The project uses Pachyderm to version data, package experimental code in containers, and connect audio conversion, MIDI transcription, preprocessing, and model training into a pipeline.
3
The generated music still needs a human to choose instruments and shape the result, while future music tools could suggest riffs, chords, and arrangements in near real time.
Summary
Daniel Jeffries explains why he chose ambient music as a practical machine learning project instead of a small tutorial dataset. He trained Google Brain's Magenta Music Transformer on a curated ambient playlist. The system converts MP3 files to WAV, transcribes them into MIDI, preprocesses the MIDI for TensorFlow, and trains the transformer on GPU hardware. MIDI captures notes, timing, and duration, but loses much of the original performance's expression. Jeffries uses Logic Pro to assign software instruments to generated MIDI files, so the final sound remains a human-directed part of the process. Pachyderm handles data versioning, lineage, immutable repositories, and containerised pipeline stages. He is frank about failed experiments, outdated research code, broken dependencies, bad transcriptions, frozen pipelines, and generated songs that repeat only a few notes. He expects AI music tools to become useful when they can respond quickly enough to suggest alternatives while an artist is working.
A useful AI project should show a real future use case
Jeffries rejects the equivalent of "hello world" projects in AI, such as the MNIST or iris datasets, because they show software in isolation. He wanted a project that worked in a recognisable creative setting. Ambient music gave him that context and connected to a future where artists work with AI. He imagines a musician playing a few bars or chords, then asking the system for many variations. The artist could listen, select ideas, and keep developing the piece. He describes this as a centaur model, where a human and an AI work together and the machine augments abilities rather than simply replacing the person.
Jeffries connects the music project to his view that critical thinking matters across technology, business, and art. Machine learning systems can study very large collections of existing patterns and optimise within them. He argues that humans can still conceive an idea that has never existed and reason about how to build it. That distinction affects creative work. AI can generate material from learned examples, while a person decides what the material means, whether it is useful, and where to take it. The project therefore keeps a human involved in judging and shaping the output.
Transformers address long-term musical structure better than earlier approaches
Jeffries chose Google Brain's Magenta Music Transformer after considering convolutional neural networks and WaveNet. He says those earlier approaches could produce short sections but often lost the structure of the song, creating a different musical personality after several seconds. Transformers use attention mechanisms to relate distant parts of a sequence. Magenta adapted that idea for music, allowing one note to connect with other parts of a piece over a longer span. The model does not directly generate a waveform. It works with MIDI, which records notes, timing, and duration but leaves out much of the nuance of a complete performance.
The music pipeline has to solve transcription before training
Jeffries could not obtain the original MIDI files from the artists in his playlist, so he had to convert the music himself. The pipeline starts with MP3 files, converts them to WAV, uses another Magenta neural network to transcribe the audio into MIDI, converts that output into a TensorFlow-compatible format, and then trains the Music Transformer. The transcription stage is difficult because research projects often have incomplete code and old dependencies. If the original recordings had been available as MIDI, he says the model would have had a much better input dataset.
Pachyderm ties immutable data and containerised stages together
Pachyderm gives each stage an input repository and an output repository, with data versioning and lineage across the pipeline. Jeffries argues that metadata alone is unsafe when the files it refers to can be overwritten. A later change to an image or video mount can make an earlier training record point to data that no longer exists. In his setup, JSON or YAML definitions specify the container image, command, input, output, and GPU requirements. The same approach packages awkward research dependencies into containers that can be called again without rebuilding the environment from scratch.
Ambient music leaves more room for imperfect notes
Jeffries says ambient music was a practical choice as well as a personal preference. Its soft, flowing sound can absorb an occasional wrong note, while a drum rhythm makes a mistake immediately obvious. He tried Berlin techno, but the result was less convincing. He attributes that partly to a corpus that was too small and partly to software instruments that lacked the hard, bass-heavy character he wanted. The generated MIDI still needs a human to select suitable instruments in Logic Pro. Some outputs sound good through several instruments, while others remain poor regardless of the instrument choice.
AI music tools need faster inference before artists can use them interactively
Jeffries expects early AI music software to suggest instruments, chords, and alternate versions of a piece. His current system takes about ten minutes to generate a song of roughly 2048 bars, so it is not ready for a live musical exchange. He expects better algorithms, more computing power, and inference chips to reduce that delay. The more useful form would work almost in real time, allowing an artist to play part of a song while the AI fills in sections, smooths the arrangement, or proposes alternatives. He compares this future direction with an advanced writing assistant that makes broader suggestions than grammar correction.
Research code creates as much work as the model itself
Jeffries describes outdated libraries, hard-coded variables, failed MIDI-generation methods, dependency problems, corrupted data, frozen pipelines, and long training runs. Some seed files produce "brain damage songs" that repeat two or three notes for minutes. A Berlin techno model trained for several days did not produce results he liked. He says the team spent months getting experimental components to work together, then packaged working pieces into containers. His account is honest about the amount of persistence required. The model is only one part of the effort, and getting the surrounding software to run is a large part of the project.
