Can Generative Models Finally Be Trained End-to-End? The Core Is a For Loop
This article introduces a novel training paradigm for generative models called Explorative Modeling (XM), which enables true end-to-end training. Traditionally, powerful generative models like autoregressive and diffusion models are not trained end-to-end. They are trained to predict a single small step but require iterative multi-step sampling for inference. This "exposure bias" leads to error accumulation and limits performance.
The core challenge XM addresses is "mode blurring." In generative tasks, a single input (e.g., "generate a dog") corresponds to many valid outputs (multiple modes). Standard training objectives like reconstruction loss force the model to average these modes, producing unrealistic, blurry outputs. To avoid this, existing models break generation into many small, almost deterministic steps, sacrificing end-to-end training.
XM tackles this by restructuring the training loop itself. Its key insight is to amplify "generative expressivity." For each training input, instead of generating one sample, the model generates K candidate outputs. Only the candidate closest to the real data is used for computing the loss and updating the model via backpropagation. This simple "best-of-K" mechanism is implemented as a short for-loop. By exploring multiple possibilities, the model learns to distribute its guesses across different modes rather than collapsing to their uninformative average.
The paper demonstrates that "exploration" acts as a new, powerful scaling axis. Gains from XM increase with model size, data scale, and compute. Experiments show improvements in FID scores for image generation and significant efficiency gains, sometimes outperforming larger models without exploration. When pushed to the limit, XM enables fully single-step, end-to-end generative models. In robotics tasks, an "Explorative Policy" matched the performance of a 100-step Diffusion Policy with a single forward pass, drastically improving inference speed.
While the best-of-K concept is not entirely new, the authors' contribution lies in formally understanding it as a direct method to boost generative expressivity without fragmenting the generation process. This work suggests that as models scale, enhancing exploration during training may become crucial for overcoming fundamental performance bottlenecks.
marsbit1 h fa
