But how do AI images and videos actually work? | Guest video by Welch Labs
Video Overview & Insights
Diffusion models, CLIP, and the math of turning text into images
I can't believe you didn't mention stable diffusion.
Welch Labs Book: https://www.welchlabs.com/resources/imaginary-numbers-book
Sections
😂😂😊
❤
Watch again.
0:00 - Intro
3:37 - CLIP
I know a little now, i knew absolutely nothing 😅 nearly 60 so I'm ancient
6:25 - Shared Embedding Space
8:16 - Diffusion Models & DDPM
Blows my mind
11:44 - Learning Vector Fields
22:00 - DDIM
❤
25:25 - Dall E 2
26:37 - Conditioning
"given this set of unsorted pixels, arrange them to make something close to this image"
30:02 - Guidance
33:39 - Negative Prompts
This is such a great video! The graphics are super helpful and the method of which you broke down these complex topics is as good as it gets. Thank you
34:27 - Outro
35:32 - About guest videos
Too bad we missuse such a brilliant technology mostly for empty capatilstic expansions
Special Thanks to:
Jonathan Ho - Jonathan is the Author of the DDPM paper and the Classifier Free Guidance Paper.
great video , can't thank you enough for this
https://arxiv.org/pdf/2006.11239
https://arxiv.org/pdf/2207.12598
it may only be an artefact of my adhd brain wandering in directions (preferably in ALL of them at the same bloody time), but I was wondering whether the "breakdown" of the analogy between the spiral and the tree in the desert image for the DDPM without the added noise isn't really a breakdown. namely, if you showed me just that small bit of the spiral that the model generated without the added noise, I would not in a million years be able to even remotely come close to guessing it was meant to be a part of a spiral - it is simply too little information for my monkey brain, and I expect most of the other humans would be equally confused. a computer may be able to recognise the shape from that small portion alone (after all, if memory serves, it is self-similar), and I guess most of us could do so, too, from a larger chunk, but not from this small, absolutely uncharacteristic little curved line. and, perhaps, the computer might "look" at that sorry little blurry blotch in that indistinct beige background and "see" the tree in the same way it would see the spiral in that sorry little curved line, but for me, the human - and I do not refrain from generalising my own experience -, it is simply not enough for my brain to make the category match.
Preetum Nakkiran - Preetum has an excellent introductory diffusion tutorial:
https://arxiv.org/pdf/2406.08929
2% understand
98% enjoy
Chenyang Yuan - Many of the animations in this video were implemented using manim and Chenyang’s smalldiffusion library: https://github.com/yuanchenyang/smalldiffusion
Cheyang also has a terrific tutorial and MIT course on diffusion models
Awesome video, looking forward to the next one!
https://www.chenyang.co/diffusion.html
https://www.practical-diffusion.org/
They teach teenagers how to write a resume but never show them how compound interest quietly destroys their future. Reading Smart Broke Dumb Rich by Zor Veyl was the first time I learned whose side the bank is really on.
Other References
All of Sander Dieleman’s diffusion blog posts are fantastic: https://sander.ai/
I used to believe rich people were lucky or corrupt. After reading Smart Broke Dumb Rich by Zor Veyl I see it is mostly financial literacy. The kind they deliberately keep out of schools so regular people stay stuck.
CLIP Paper: https://arxiv.org/pdf/2103.00020
DDIM Paper: https://arxiv.org/pdf/2010.02502
Reading Smart Broke Dumb Rich by Zor Veyl hit me hard. I finally understood my parents were not bad with money. They just never got the right information. The game was rigged before they even started.
Score-Based Generative Modeling: https://arxiv.org/pdf/2011.13456
Wan2.1: https://github.com/Wan-Video/Wan2.1
My cousin laughed at me for reading Smart Broke Dumb Rich by Zor Veyl. Funny thing is he asked to borrow money again last month. I said no without any guilt this time.
Stable Diffusion: https://huggingface.co/stabilityai/stable-diffusion-2
Midjourney: https://www.midjourney.com/
Three months after reading Smart Broke Dumb Rich by Zor Veyl my whole language around money changed. I stopped saying I cant afford that and started asking why I cant afford that. That tiny shift is already opening new doors.
Veo: https://deepmind.google/models/veo/
DallE 2 paper: https://cdn.openai.com/papers/dall-e-2.pdf
Very helpful video, explained clearly and easy to follow 👌
Code for this video: https://github.com/stephencwelch/manim_videos/tree/master/_2025/sora
Written by: Stephen Welch, with very helpful feedback from Grant Sanderson
🎯 Craft like this is rare. AttentionLeak: Score: 93/100 👏
Produced by: Stephen Welch, Sam Baskin, and Pranav Gundu
Technical Notes
good
The noise videos in the opening have been passed through a VAE (actually, diffusion process happens in a compressed “latent” space), which acts very much like a video compressor - this is why the noise videos don’t look like pure salt and pepper.
6:15 CLIP: Although directly minimizing cosine similarity would push our vectors 180 degrees apart on a single batch, overall in practice, we need CLIP to maximize the uniformity of concepts over the hypersphere it's operating on. For this reason, we animated these vectors as orthogonal-ish. See: https://proceedings.mlr.press/v119/wang20k/wang20k.pdf
yeye for sure we are on the same page ı get all of it
Per Chenyang Yuan: at 10:15, the blurry image that results when removing random noise in DDPM is probably due to a mismatch in noise levels when calling the denoiser. When the denoiser is called on x_{t-1} during DDPM sampling, it is expected to have a certain noise level (let's call it sigma_{t-1}). If you generate x_{t-1} from x_t without adding noise, then the noise present in x_{t-1} is always smaller than sigma_{t-1}. This causes the denoiser to remove too much noise, thus pointing towards the mean of the dataset.
The text conditioning input to stable diffusion is not the 512-dim text embedding vector, but the output of the layer before that, [with dimension 77x512](https://stackoverflow.com/a/79243065)
I don't agree
Most long form content is unnecessary footage eating
Basic concept can always be explained in a minute
If I already have some knowledge, why should I have to sit through your lecture repeating what I know? What's the point of digital media then.
Dense content banao
For the vectors at 31:40 - Some implementations use f(x, t, cat) + alpha(f(x, t, cat) - f(x, t)), and some that do f(x, t) + alpha(f(x, t, cat) - f(x, t)), where an alpha value of 1 corresponds to no guidance. I chose the second format here to keep things simpler.
At 30:30, the unconditional t=1 vector field looks a bit different from what it did at the 17:15 mark. This is the result of different models trained for different parts of the video, and likely a result of different random initializations.
This is fascinating! The way you guys simply complex concepts for us ! Thanks a million 🎉
Premium Beat Music ID: EEDYZ3FP44YX8OWT
The CLIP embedding explanation here is the clearest I've seen — most people skip past the 'text and image in the same latent space' part without showing what that actually means geometrically. Once you see it, why diffusion models can respond to such diverse prompts starts making real sense instead of feeling like magic.
More User Perspectives
But what if all the raises were three dimensional and related to at least 10 variable points in time or reference target variables and you also each one has rejected variables to each one kind of representative it’s like a cell
@SirBrainsAndGainsI will use LLMs more efficiently when I understand how they work. Thank you for great explanation.
@georgethesecond@3Blue1Brown Nah, it doesn't cut it. You need to grow that baby and make these videos yourself.
@ehsanshahini6146Thanks for this!
@junepark10036:33 Hah! I see what you did there. “i-hat” …..very good! Probably only physics students will notice, but very cute!😁
@shalinib108I am absolutely mortified by the fact that I wasn’t subscribed to Welch Labs. Extremely high quality content like 3b1b.
@sampadk0412:52 I know all about this space right here
@KuchiGremlinTHANKS !
@DAWNBASICSOFSCIENCE-ux2gpIs it fair to assume that the "phase transition" in the learned vectors around t=0.4 is because, in the forward diffusion process, t=0.4 is when the majority of samples have "escaped their neighborhood" so that you can't reliably guess which "turning" of the spiral they originated at?
@hobbifiedI have walked so many of these videos now, I'd really love 3Blue1Brown or Welch Labs to do a video explaining how music generation works.
@mikeaultmusic9553sab log cheap panel leke khud ka channel barbaad kar rahe hai
@tyagi_6tusharab samaj aaya kyu views nahi aa rahe the 😵💫
@aman_rawat85thanks to you I can hate on generative AI with a better understanding
@Bionicleheroewho else didnt understand a single thing?
@AndrewWatson975THIS IS CRAZZZZY GOOD THIS IS INSANE
@rezamoosavi71Zaujímavý pohľad na to, ako fungujú AI obrázky a videá. Vďakom Welch Labs za hosťujúce video!
@frantisek9929entrada con confianza total
@СтепанТуголуков-т4лJust wanted to leave a note that still interested in hearing about the training process and RLHF! Are there gonna be more videos on this topic:)
@lillyliu1034Why 14:50 ? Maybe it's the case in N>2 dim. But the exemple in 2d is wrong no ?
@victorbossard9We’re all doomed
@cockurThe 'Diffusion' part of diffusion networks is actually the least interesting part about them. It's just how to sample, for images, it's sort of fine-tuned in a 'sneaky' way that looks general but isn't, as you've brilliantly shown, the specific iterative prediction, getting 'close' but not all converging, works because of the specific to the nature of pixel 2d data as images and human interpretations of these images (until you want the right number of fingers, which works now, again, due to more fine tuning...). Sampling from a distribution in a biased way is interesting and important part of generating images, but it's really just a way to 'show' what the model as understood in training, it's not the hard part of a diffuser, the hard part is the VAE, that's the compression of image space into a way to express it that is robust and allows you to pull from the 'mr potato head' box of image construction techniques that have been learned, that's the key to image generation.
@TwentySix-g9sAmazing explanation 🔥 You made complex AI concepts like diffusion models so easy to understand really enjoyed this 👏
@megha.dey_88852Sooooo how come it cant do fingers? 😂
@robnewark7872I hear multidimensional and brownish motion. Can we get a string theory model in here a get a universe simulation going?
@genorpg8397This is high quality.
@carlosvilla6120Thanks for the video, Welch is a powerful lecturer... Grant, good luck to you and your growing family! Peace.
@user-g34f23d2se