Compiling Denoisers into Generators

Denoising is an elegant self-supervised learning primitive. Corrupt a real example, ask a model to recover what was lost, and supervision comes for free. Gaussian corruption makes this especially convenient: it provides a continuous difficulty scale and a mathematical connection between reconstruction and the score of a probability distribution.

But a principled learning primitive is not necessarily the best pretraining task for the system we ultimately want to deploy.

If the goal is an image or video generator that produces a convincing sample in one or a few forward passes, the success of representation alignment, structured corruption, and step-distillation should make us question the default recipe. We train a network to estimate what a corrupted observation probably came from, then spend considerable effort teaching it useful representations and converting its predictions into decisive generative outputs.

My thesis is that these interventions expose a mismatch between the reconstruction task used for training and the sampling task required at inference. They are evidence against treating conventional Gaussian-denoising pretraining as an optimal design, although they do not establish that Gaussian noise itself must be abandoned.

The distinction starts with what the network actually learns.

Suppose we corrupt an image with Gaussian noise and train a model to recover the original, using squared reconstruction error. We may also provide a text prompt or previous video frames as context.

The best possible prediction under this objective is the conditional mean: the average of all clean examples compatible with the observation and context, weighted by how plausible each example is. This is a property of the loss itself. Moving away from that average increases the expected squared error, even when the alternative is a perfectly realistic image.

When several incompatible explanations remain plausible, their average can lie between them.

Imagine a video context in which a ball could next appear on either side of an obstacle. A pixelwise average of those futures may contain two faint balls. It minimizes squared error while describing neither possible outcome. The problem is not inadequate optimization: the ambiguity is real, and averaging is exactly what the objective requests.

At complete corruption, the input is pure noise and contains no information about the particular training example. If noise and training examples are paired independently, the best reconstruction depends only on the remaining context. Every seed receives the same optimal prediction: the average image or future compatible with that context.

A perfect one-shot reconstructor at that endpoint therefore ignores the seed. A useful generator must use the seed to select among plausible outcomes. The seed supplies a choice, not evidence about which original training example was destroyed.

Adding an extra independent random input does not repair the squared-error objective: conditioned on that input, the same conditional mean remains optimal. What must change is the target, the coupling between inputs and targets, or the criterion used to judge outputs.

There is a crucial qualification. Diffusion models are not mathematically condemned to generate averages. Under Gaussian corruption, the conditional mean tells us how the probability density of noisy data changes around the current input. That local information is called the score.

A diffusion sampler uses this information repeatedly to move an initial noise sample toward a generated image. With the appropriate dynamics, these trajectories transform a distribution of random seeds into the data distribution. The complete iterative system is already a generator. The network supplies local information; the sampling procedure turns it into an outcome. Song et al. establish this score-based formulation, while Tweedie’s formula explains the connection between denoising and score estimation.

The issue is therefore computational: knowing which direction to move is different from knowing where an entire journey will end. A network that supplies the former does not automatically compute the latter in one evaluation. A theory explaining why iterative sampling works does not establish that its training objective is the most efficient preparation for a tiny inference budget.

Changing from predicting the clean image to predicting noise or velocity does not, by itself, remove this distinction. Squared-error regression still learns a conditional expectation of the chosen target. Parameterization and weighting matter greatly in practice, but they do not automatically turn a local prediction into a complete transport map.

This is where representation learning becomes revealing.

REPA, or Representation Alignment for Generation, aligns projected hidden states of a denoising model with clean-image features from a pretrained visual encoder. DINOv2 is a particularly effective teacher. The reported SiT experiment matches the unguided performance of a baseline trained for seven million steps in fewer than 400,000 steps: over 17.5 times fewer training steps.

DINOv2 supplies visual representations learned through a separate self-supervised training process. REPA uses those representations to guide what the generative network learns internally, instead of expecting reconstruction supervision alone to discover equally useful features efficiently.

My reading is that this exposes a weakness in the allocation of learning effort. A denoising objective can support semantic learning without making it the easiest or fastest route to reducing loss. Explicit representation supervision changes that route. The improvement does not invalidate score matching; it questions whether score matching alone provides the best curriculum for a finite model and training budget. The headline speedup also excludes the separate cost of pretraining the external encoder.

Self-Flow brings that lesson inside the generative model. It assigns two different noise levels across tokens and trains a student to predict features from an exponential-moving-average teacher that observes a cleaner view, alongside the flow-matching objective.

The connection to masked autoencoders, or MAE, is the deliberate creation of missing-information problems. MAE removes patches and reconstructs their pixels from visible context. Self-Flow uses heterogeneous corruption and teacher-feature prediction to encourage contextual inference. It retains noisy tokens and is not simply MAE attached to a diffusion network.

The shared idea is that the structure of corruption shapes the representations learned. How information is withheld matters, beyond how much noise is added.

Self-Flow also reports that naive full masking and independent per-token noising hurt generation in its evaluated setup, attributing this to a training–inference mismatch. Its results support carefully designed asymmetry, rather than arbitrary corruption complexity. Gaussian noise remains part of the method; the intervention changes the learning problem built around it. Uneven noise has to be designed carefully, and more randomness isn’t automatically better.

The same design freedom appears in Diffusion Forcing. Instead of assigning one shared noise level to an entire sequence, it trains with independent per-token noise levels in a causal model. This allows relatively clean history to coexist with uncertain future tokens and supports flexible sequence generation and guidance.

For video, that distinction matters. A generated past and an unresolved future have different informational roles. A single global noise clock is a modeling choice, not a requirement imposed by the task.

Diffusion Forcing retains a probabilistic foundation, including a variational bound over subsequences. It therefore illustrates how diffusion theory can accommodate a different training formulation, rather than refuting that theory. Together with Self-Flow, it suggests a practical question: which corruption patterns prepare the model for the conditional problems it will actually face at inference?

A related mismatch appears before the denoiser even sees its input.

Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models shows that increasing a tokenizer’s per-token latent dimension can improve reconstruction while making downstream diffusion training harder. Its VA-VAE aligns the latent space with pretrained vision features to improve the reconstruction–generation tradeoff.

This is a different reconstruction problem from denoising: the paper studies the tokenizer and the learnability of its latent space. But it reinforces the same warning about proxy objectives. Preserving more information does not guarantee that a finite generative model can organize and synthesize that information efficiently. Reconstruction quality alone cannot rank representations for generation.

Step-distillation addresses the mismatch most directly.

The central conceptual obstacle is underdetermination. A corrupted observation or a video context admits multiple valid completions. A generator must resolve that ambiguity into individual samples while reproducing the appropriate distribution across many seeds. It must avoid both averaging incompatible outcomes and always selecting the same plausible outcome.

This is not the only source of diffusion’s computational cost: representation quality, trajectory geometry, numerical integration, and approximation error also matter. But it explains why better reconstruction and better one-step generation are not interchangeable goals.

Consistency models learn to send different points along the same sampling trajectory to a common endpoint. In consistency distillation, the teacher tells the student which points belong to that trajectory. This teaches the student to anticipate where the sampling process will finish. Its target is the outcome selected by that process, which can differ from the average of all original examples compatible with a noisy observation.

The conceptual shift is from estimating a local quantity to approximating the result of a computation. Consistency training can also learn such models without a pretrained diffusion teacher, so distillation is one route to this objective rather than a universal requirement.

Distribution Matching Distillation, or DMD, changes the criterion more explicitly. It trains a fast generator using the difference between scores of the target distribution and the generator’s own distribution, evaluated after re-noising. Its original formulation also includes a teacher-output regression loss.

The student need not reconstruct the particular training image that happened to be paired with an independent noise sample. Instead, its outputs are judged collectively through a distributional objective. The target score alone would encourage seeking high-density outcomes; comparing it with the generated-distribution score accounts for how the student already distributes its samples. Approximate scores and optimization still limit the result, so this is not an automatic guarantee of complete mode coverage.

Consistency distillation and DMD thus offer different ways to change what the network is asked to compute: an endpoint associated with a trajectory, or samples whose distribution matches a target.

That is why step-distillation deserves to be understood as more than removing redundant solver calls. It trains a different computational object.

Taken together, these papers motivate a more deliberate separation between learning useful representations, specifying conditional generation tasks, and learning a fast sampling map. Canonical denoising provides a powerful foundation for all three, but its mathematical elegance does not settle their engineering design.

For a system whose final requirement is a handful of forward passes, we should evaluate pretraining by the generator it enables after adaptation, including the total training cost. There is no reason to assume that the best many-step denoiser is automatically the best starting point for that objective.

In this sense, step-distillation is the way we compile what is inherently a denoiser into what is effectively a generator: the iterative sampler’s work becomes a learned mapping from a seed to a committed, coherent sample.

Written by Francois Belletti, wording polished by AI, 09/20/2026