Diffusion, Multimodal & Generative Models

The forward process is a fixed Markov chain that gradually adds Gaussian noise to data over $T$ steps: $q(x_t|x_{t-1})=\mathcal{N}(\sqrt{1-\beta_t}\,x_{t-1},\beta_t I)$, eventually destroying all structure into near-pure noise. The reverse process is learned: a network parameterizes $p_\theta(x_{t-1}|x_t)$ to denoise step by step back to data. In DDPM the network typically predicts the noise $\epsilon$ added at step $t$ (an $\epsilon$-prediction objective), equivalent up to scaling to predicting the score $\nabla_{x}\log p(x_t)$. Sampling starts from Gaussian noise and iteratively denoises.

VAEs maximize an evidence lower bound (reconstruction + KL to a prior), giving a tractable likelihood proxy and stable training, but tend to produce blurrier samples because the Gaussian likelihood averages modes. GANs train a generator against a discriminator in a minimax game, yielding sharper samples but no likelihood, unstable training, and risk of mode collapse. VAEs offer good mode coverage and a smooth latent useful for interpolation/representation; GANs win on perceptual fidelity. Pick VAEs when you need a principled latent or density estimate; GANs (historically) for crisp images, though diffusion now dominates fidelity.

CFG steers a conditional diffusion model toward conditioning $c$ (e.g., a text prompt) without a separate classifier. During training you randomly drop the condition (e.g., 10–20% of the time) so one network learns both conditional $\epsilon_\theta(x_t,c)$ and unconditional $\epsilon_\theta(x_t,\varnothing)$ predictions. At sampling you extrapolate: $\tilde\epsilon=\epsilon_\theta(x_t,\varnothing)+w(\epsilon_\theta(x_t,c)-\epsilon_\theta(x_t,\varnothing))$, where $w$ is the guidance scale. Higher $w$ increases prompt adherence and contrast but reduces diversity and can oversaturate or add artifacts; $w=1$ recovers plain conditional sampling. It trades mode coverage for alignment.

Latent diffusion (Rombach et al.) runs the diffusion process in the compressed latent space of a pretrained VAE/autoencoder rather than on raw pixels. Pixel-space diffusion must denoise high-dimensional images at every step, which is computationally enormous. The autoencoder downsamples (e.g., 512×512×3 → 64×64×4), removing imperceptible high-frequency detail so diffusion focuses on semantic/perceptual content. The U-Net learns the distribution in this efficient latent; the decoder maps the denoised latent back to pixels. This cuts training and inference cost by roughly an order of magnitude while preserving quality, and enables conditioning via cross-attention on text embeddings.

CLIP trains an image encoder and a text encoder jointly on hundreds of millions of (image, caption) pairs with a symmetric InfoNCE contrastive loss. For a batch of $N$ pairs it computes the $N\times N$ cosine-similarity matrix of image and text embeddings, then applies cross-entropy in both directions to pull matched pairs together and push the $N-1$ mismatched pairs apart, with a learned temperature. This aligns the two modalities into one space where semantically related images and text are close. Zero-shot classification embeds class names as prompts ('a photo of a {class}') and picks the nearest text embedding to the image — no task-specific training needed.

The dominant pattern (LLaVA-style, Flamingo-style) keeps a pretrained LLM and a pretrained vision encoder (often CLIP/ViT) and bridges them with a learned connector. The vision encoder produces patch embeddings; a projector — an MLP, a linear layer, or a resampler like Flamingo's Perceiver or a Q-Former — maps these into the LLM's token embedding space, turning the image into 'visual tokens' prepended or interleaved with text tokens. The LLM then attends over both jointly. Training is staged: first align the connector on image-caption data (encoder/LLM often frozen), then instruction-tune. The connector does the modality alignment cheaply.

Choose a text-to-image latent diffusion foundation model (the cloud's hosted image-generation model, customized via DreamBooth/LoRA-style adaptation on brand assets). Prioritize the classifier-free guidance scale to balance prompt adherence (higher) against diversity and artifact risk (lower), and exploit the latent space for efficient large-scale inference. A GAN is the wrong default: GANs train unstably, are prone to mode collapse (limited diversity across a catalog), lack a text-conditioning interface as flexible as cross-attention on prompt embeddings, and are largely superseded by diffusion for controllable, high-fidelity text-to-image. Diffusion also fine-tunes well from a foundation checkpoint.

The ELBO decomposes into per-step KL terms between $q(x_{t-1}|x_t,x_0)$ and $p_\theta(x_{t-1}|x_t)$, both Gaussian, so each KL reduces to a weighted squared error between their means. Using the closed form $x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\,\epsilon$ and reparameterizing the mean in terms of $\epsilon$, the term becomes proportional to $\|\epsilon-\epsilon_\theta(x_t,t)\|^2$. Ho et al. drop the time-dependent weighting, giving $L_{simple}=\mathbb{E}_{t,x_0,\epsilon}\|\epsilon-\epsilon_\theta(\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon,t)\|^2$. This is valid because the variational bound's Gaussian KLs are exactly MSEs on the predicted noise; unweighting empirically improves quality by emphasizing harder, higher-noise steps.

Mode collapse is when the generator maps many latents to a few outputs, covering only some modes of the data while ignoring others — sharp samples but low diversity. It arises because the generator can fool the current discriminator by exploiting one mode; the minimax game has no explicit coverage incentive, and unstable gradients let the generator collapse before the discriminator adapts. Mitigations: minibatch discrimination / minibatch standard deviation (lets D see batch diversity), unrolled GANs, feature matching, Wasserstein loss with gradient penalty (WGAN-GP) for smoother gradients, spectral normalization for Lipschitz stability, packing (PacGAN), and historical-averaging tricks.

Inception Score (IS) uses an Inception classifier: it rewards low-entropy per-image label distributions (each image clearly an object) and high-entropy marginal (diverse classes), via $\exp(\mathbb{E}_x\mathrm{KL}(p(y|x)\|p(y)))$. FID fits Gaussians to Inception-pool features of real and generated sets and computes the Fréchet/2-Wasserstein distance $\|\mu_r-\mu_g\|^2+\mathrm{Tr}(\Sigma_r+\Sigma_g-2(\Sigma_r\Sigma_g)^{1/2})$; lower is better. FID failure modes: (1) it is biased by sample size — fewer samples inflate FID, so counts must match; (2) the Gaussian assumption and ImageNet-trained features make it domain-mismatched on non-ImageNet data and blind to memorization/overfitting.

DDIM reformulates the reverse process as a non-Markovian update (deterministic or partially stochastic) that reuses DDPM's training objective but lets you sample on a sub-sequence of timesteps (e.g., 20–50) instead of all $T$. With variance parameter $\eta=0$ it is fully deterministic, giving a consistent, invertible noise-to-image mapping (useful for interpolation/editing), at some cost to the stochastic diversity DDPM provides. Unifying view: diffusion learns the score $\nabla_x\log p_t(x)$, and sampling integrates the reverse-time SDE; DDIM corresponds to its deterministic 'probability-flow ODE.' Faster ODE solvers (DPM-Solver) exploit this to reach high quality in ~10–20 steps.

Training step: t = randint(1,T); eps = randn_like(x0); xt = sqrt(abar[t])*x0 + sqrt(1-abar[t])*eps; loss = mse(model(xt, t, cond), eps); loss.backward(). Sampling loop: x = randn(shape); for t in reversed(range(1,T+1)): eps_hat = model(x, t, cond); x0_hat = (x - sqrt(1-abar[t])*eps_hat)/sqrt(abar[t]); mean = sqrt(abar[t-1])*x0_hat + sqrt(1-abar[t-1])*eps_hat # DDIM-style; or use the DDPM posterior mean; z = randn() if t>1 else 0; x = mean + sigma[t]*z. The network always receives noisy $x_t$, the timestep embedding $t$, and any conditioning, and outputs predicted noise. abar is the cumulative product of $\alpha_t=1-\beta_t$.

Autoregressive models factorize the joint exactly via the chain rule $p(x)=\prod_i p(x_i|x_{<i})$, giving exact tractable likelihoods and stable maximum-likelihood training; diffusion only optimizes a variational lower bound. The cost is sampling: AR generation is inherently sequential — one element at a time — so an $N$-element output needs $N$ forward passes, brutal for high-resolution images or high-sample-rate audio. Discrete tokenizers (VQ-VAE/VQGAN for images, neural codecs like EnCodec for audio) shrink the sequence: a Transformer models a short grid of discrete latent tokens instead of raw pixels/samples, slashing AR steps while keeping exact likelihood over the token sequence.

Start from $\log p_\theta(x)=\log\int p_\theta(x|z)p(z)dz$. Insert the approximate posterior $q_\phi(z|x)$ and apply Jensen: $\log p_\theta(x)=\log\mathbb{E}_q[\frac{p_\theta(x,z)}{q_\phi(z|x)}]\ge\mathbb{E}_q[\log p_\theta(x|z)]-D_{KL}(q_\phi(z|x)\,\|\,p(z))$, the ELBO. The gap equals $D_{KL}(q_\phi\|p_\theta(z|x))\ge0$, so it is always a lower bound, tight when $q$ matches the true posterior. The reparameterization trick writes $z=\mu_\phi+\sigma_\phi\odot\epsilon$, $\epsilon\sim\mathcal{N}(0,I)$, moving stochasticity off $\phi$ so $\nabla_\phi$ passes through a deterministic path — lower-variance gradients than the score-function/REINFORCE estimator.

CFG extrapolates $\tilde\epsilon=\epsilon_\varnothing+w(\epsilon_c-\epsilon_\varnothing)$, which is no longer the score of any normalized distribution for $w>1$ — it implicitly samples from $p(x)\,p(c|x)^w$, sharpening the conditional toward a near-mode-seeking, lower-temperature distribution. This over-concentrates mass, collapsing diversity, and the inflated update pushes predicted $x_0$ out of the training range, producing oversaturated/clipped pixels and high-contrast artifacts. Fixes: dynamic thresholding (Imagen) rescales/clamps predicted $x_0$ to the valid range each step to counter saturation; guidance scheduling lowers $w$ over timesteps; applying guidance only on a middle interval of noise levels recovers diversity, since high guidance hurts most at very low and very high noise.

For a Gaussian-perturbed variable $x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon$, Tweedie's formula gives the posterior mean $\mathbb{E}[x_0|x_t]=\frac{1}{\sqrt{\bar\alpha_t}}\big(x_t+(1-\bar\alpha_t)\nabla_{x_t}\log p_t(x_t)\big)$ — the optimal denoiser equals data plus a score correction. Since $x_t-\sqrt{\bar\alpha_t}x_0=\sqrt{1-\bar\alpha_t}\epsilon$, the score satisfies $\nabla_{x_t}\log p_t(x_t)=-\epsilon/\sqrt{1-\bar\alpha_t}$. Hence training $\epsilon_\theta$ with MSE to the true noise is exactly denoising score matching (Vincent's equivalence): minimizing $\|\epsilon-\epsilon_\theta\|^2$ estimates $-\sqrt{1-\bar\alpha_t}\,\nabla\log p_t$. This is why diffusion = learning the score across noise scales, and why sampling is reverse-SDE / probability-flow-ODE integration of that learned score.

The forward process gives $x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon$. The ELBO decomposes into KL terms between the tractable forward posterior $q(x_{t-1}|x_t,x_0)$ (Gaussian, known mean/variance) and $p_\theta(x_{t-1}|x_t)$. Matching their means, and substituting $x_0$ in terms of $x_t,\epsilon$, the per-step term becomes a weighted $\|\epsilon-\epsilon_\theta(x_t,t)\|^2$. Ho et al. drop the time-dependent weights to get $L_{simple}=\mathbb{E}_{t,x_0,\epsilon}\|\epsilon-\epsilon_\theta(x_t,t)\|^2$. $\epsilon$-prediction targets are unit-variance across all $t$, giving a well-conditioned, scale-stable regression; $x_0$-prediction has wildly varying signal at high noise, and the raw posterior-mean weighting over-emphasizes low-noise steps.

DDIM defines a non-Markovian forward family that shares the same marginals $q(x_t|x_0)$ as DDPM, so a network trained with the DDPM objective is reusable without retraining. Its update first estimates $\hat{x}_0=(x_t-\sqrt{1-\bar\alpha_t}\,\epsilon_\theta)/\sqrt{\bar\alpha_t}$, then steps to $x_{t-1}=\sqrt{\bar\alpha_{t-1}}\hat{x}_0+\sqrt{1-\bar\alpha_{t-1}-\sigma_t^2}\,\epsilon_\theta+\sigma_t\epsilon$. Setting $\sigma_t=\eta\sqrt{\dots}$ with $\eta=0$ removes all injected noise: the trajectory becomes a deterministic ODE-like map, so you can skip to a sparse subsequence of timesteps (e.g. 20–50) with little quality loss. $\eta=1$ recovers stochastic DDPM. Determinism also gives a usable latent code (invertibility, interpolation).

For the Gaussian perturbation $x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon$, the marginal score satisfies $\nabla_{x_t}\log p(x_t)=-\,\epsilon_\theta(x_t,t)/\sqrt{1-\bar\alpha_t}$ — so noise-prediction is denoising score matching up to a scale. Tweedie's formula formalizes this: for $x_t\sim\mathcal{N}(\sqrt{\bar\alpha_t}x_0,(1-\bar\alpha_t)I)$, the posterior mean $\mathbb{E}[x_0|x_t]=(x_t+(1-\bar\alpha_t)\nabla\log p(x_t))/\sqrt{\bar\alpha_t}$, i.e. the optimal denoiser is the score. Song's framework casts the forward process as an SDE; sampling solves the reverse-time SDE $dx=[f-g^2\nabla\log p]dt+g\,d\bar w$ (or its probability-flow ODE), with the learned score plugged in — unifying DDPM, DDIM (the ODE), and score-based models.