# The Variational Book

Yuri Plotkin, ML Scientist | MLOps Podcast | Episode 253 | 55:23
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=RBzYKL_dO84
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/the-variational-book
Published: 2024-08-13
Tags: classical-ml

## TL;DR
- Yuri Plotkin wrote The Variational Book to connect Bayesian models, latent-variable methods, variational inference, score matching, normalizing flows, and diffusion models.
- Diffusion models produce high-quality samples through repeated noising and denoising steps, while newer methods reduce the number of inference steps to make generation faster.
- The book leaves out LLMs and focuses on foundational generative AI methods whose ideas have influenced newer models.

## Summary
Yuri Plotkin discusses how he moved from biomedical engineering and wet-lab research into computer science and generative AI. His book, The Variational Book, grew from his attempt to understand research papers by deriving the mathematics and explaining how separate methods fit together. He focuses on Bayesian modeling, latent-variable models, variational inference, GANs, score matching, normalizing flows, and diffusion models, while leaving out LLMs because that area already has extensive theoretical and practical coverage. Plotkin explains diffusion models through their repeated corruption and denoising process, and describes how later work reduced inference from thousands of steps toward far fewer steps. He also explains the difference between Markovian and non-Markovian diffusion processes, including how DDIM changes the inference structure. The conversation returns to a practical concern: engineers need to understand what model architectures assume, where they are strong, and where their costs make another approach more suitable.

## Key ideas
### Curiosity led Yuri Plotkin from biomedical engineering to computer science
[03:11](https://www.youtube.com/watch?v=RBzYKL_dO84&t=191s)
Yuri Plotkin says biology interested him from childhood, including asking his grandparents whether ketchup killed germs. He studied biomedical engineering at undergraduate and master's level because it combined biology with engineering and practical problem-solving. He later moved into wet-lab research, then returned to mathematics and began studying computer science. Learning is the thread he sees across his career. He likes the scientific method because it allows people to test ideas against observations and values the reproducibility that comes with systematic work. His explanation is personal rather than planned: curiosity kept pulling him toward new subjects until computer science became his main focus.

### The book connects separate generative AI methods into one lineage
[06:19](https://www.youtube.com/watch?v=RBzYKL_dO84&t=379s)
The Variational Book covers roughly the last decade of generative AI algorithms, excluding LLMs. Plotkin derives the algorithms and explains why their steps work, adding intuitive and anecdotal examples. He began with papers and planned to write a short blog post, but continued writing until the project became a book. He describes the literature as a puzzle, where individual publications contribute pieces that can be difficult to connect. The book starts with Bayesian modeling and latent models, then moves toward variational methods, score matching, normalizing flows, and diffusion models. Plotkin says the goal is to reduce the effort needed to learn the material without hiding its technical demands.

### LLMs are outside the book because Plotkin chose a different foundation
[10:48](https://www.youtube.com/watch?v=RBzYKL_dO84&t=648s)
Plotkin deliberately leaves LLMs out of the book. His focus is Bayesian latent models, variational inference, and methods concerned with fitting distributions. He also points out that LLMs already have many books covering theory and techniques such as LoRA, RAG, and RLHF. He wanted to give more attention to traditional approaches that remain foundational and whose theory carries into newer methods. Demetrios Brinkmann describes the book as a family tree of models, and Plotkin agrees that the historical connections are part of what he tried to make clear. He also admits that the book is selective because he chose methods that worked and became widely cited.

### Diffusion models trade repeated computation for sample quality
[13:25](https://www.youtube.com/watch?v=RBzYKL_dO84&t=805s)
Plotkin explains diffusion models as a repeated version of a latent-space generation process. During training, a data sample is corrupted over many steps by adding noise until it becomes Gaussian noise. During inference, the model reverses that process and learns to generate a sample from the original data distribution. He contrasts this with GANs, whose generator can produce a sample in one step. Diffusion models can produce highly faithful samples, but their original inference process was expensive because it required many sequential steps. Plotkin cites DDPM as a seminal 2020 paper and says that generating samples with the early approach could take hours, while GAN sampling could take thousands of samples in minutes.

### Diffusion research has focused on reducing inference steps
[15:47](https://www.youtube.com/watch?v=RBzYKL_dO84&t=947s)
Plotkin says later diffusion work concentrated on the inference bottleneck. Methods moved from around a thousand steps toward a hundred, then ten, with current work attempting one or two steps. The training process can still use many steps, since the practical concern is how quickly the model can generate output. He connects faster inference with video generation, where lower inference cost could allow longer videos. Demetrios Brinkmann compares the denoising process to watching a Polaroid image form and notes that too many steps could over-bake an image in some interfaces. Plotkin also distinguishes algorithmic changes from architecture-specific changes, since his book concentrates on the core algorithms rather than every model variation.

### DDIM changes the diffusion inference assumption
[28:08](https://www.youtube.com/watch?v=RBzYKL_dO84&t=1688s)
Plotkin defines a first-order Markov model as one where the future depends only on the previous state. In the original denoising diffusion setup, each forward step is conditioned on the immediately preceding step, which makes the process sequential. DDIM, or diffusion denoising implicit models, uses a non-Markovian inference structure. Plotkin explains that the distribution can depend on the previous time step and the original data sample. This allows the model to preserve the relevant marginals while reducing the number of inference steps. He stresses that the important practical result is faster generation, while acknowledging that the mathematical explanation is difficult to convey without working through the equations.

### Model understanding helps engineers choose architectures for real constraints
[18:18](https://www.youtube.com/watch?v=RBzYKL_dO84&t=1098s)
Demetrios Brinkmann uses the contrast between GANs and diffusion models to make a practical point. A GAN may be faster, while a diffusion model may offer different generative properties at a higher inference cost. The right choice depends on the use case and the allowed response time. Plotkin agrees that engineers need to understand why an objective or architecture is being used and what its consequences are. The conversation broadens this to ML engineering, where decisions include the model, infrastructure, databases, data, and the business outcome. Plotkin says theory and engineering both matter because proposing a useful change requires understanding the method rather than simply coding an available objective.

## Notable quotes
- Yuri Plotkin: "The more you learn, the more you realize you know less." (05:54)
- Yuri Plotkin: "I create that common thread that lets you kind of in a stepwise fashion understand things." (08:42)
- Yuri Plotkin: "The bottlenecks are the inference where you have to do a thousand steps to generate one example." (15:11)
- Yuri Plotkin: "You care about the inference, the reverse, because you care about the generation, and you want that to be quick." (37:54)
- Yuri Plotkin: "In order to be able to make changes or propose changes, you have to understand why you are using it and what the implications are." (52:38)

## Tools & references mentioned
- The Variational Book
- LLMs
- GANs
- continuous normalizing flows
- normalizing flows
- score matching
- variational autoencoders
- diffusion models
- Flux
- DDPM
- Stable Diffusion
- DDIM
- diffusion denoising implicit models
- Joe Reis
- The Fundamentals of Data Engineering
- LoRA
- RAG
- RLHF
- Bayesian modeling
- Bayes theorem
- KL divergence
- Wasserstein GANs
- style-based generators

## Who should watch
- You are an ML engineer who wants to understand how variational methods, GANs, normalizing flows, and diffusion models relate before choosing an architecture.
- You work with generative models and need a clear explanation of why diffusion inference is slow and how methods such as DDIM reduce the number of steps.
- You are moving from data engineering or another technical field into ML and want more theory for judging model assumptions, costs, and use cases.

## Related talks

- [Creative AI: Using ML to Create Art, Music, and Jokes](https://mlopstalks.com/talks/creative-ai-using-ml-to-create-art-music-and-jokes) (Suyash Joshi, MLOps Community, 1:12:57)
- [Innovative Gen AI Applications: Beyond Text](https://mlopstalks.com/talks/innovative-gen-ai-applications-beyond-text) (Diana C. Montañes Mondragon & Nick Schenone, QuantumBlack, 54:45)
- [MLOps at the Age of Generative AI](https://mlopstalks.com/talks/mlops-at-the-age-of-generative-ai) (Barak Turovsky, Scale Venture Partners, 56:56)
- [DevTools for Language Models: Unlocking the Future of AI-Driven Applications](https://mlopstalks.com/talks/devtools-for-language-models-unlocking-the-future-of-ai-driven-applications) (Diego Oppenheimer, Factory, 29:55)
- [The Next Revolution in AI: LLMs and Beyond](https://mlopstalks.com/talks/the-next-revolution-in-ai-llms-and-beyond) (, 13:47)