Text-to-image models have consistently scaled along dimensions of model size, data, and compute. However, the amount of information contained in the Captions paired with training images has rarely been systematically studied as an independent variable. ByteDance's Seed team discovered that longer natural language captions do not necessarily provide more usable visual supervision to the model; compared to length, the amount of information bound to the image within a caption is a better predictor of the final training loss a diffusion model can achieve.
Based on this finding, the team proposed Structured Prompt, aiming to enhance text conditioning from both sides of Diffusability and Promptability. This approach led to significant improvements on tasks involving complex composition, reasoning, and world knowledge generation.
Figure 1 from the paper | Natural language length saturates quickly; structured conditioning continues to increase image information and consistently lowers the diffusion training loss along a unified relationship.
In recent years, the advancement of text-to-image models has largely followed a familiar path: larger models, more data, and increased training compute.
However, there is an easily overlooked difference between text-to-image models and language models. Language models can learn directly from text sequences via self-supervision; text-to-image models rely on image-caption pairs to learn "what kind of text corresponds to what kind of visual content." An image may contain numerous objects, attributes, positions, actions, and relations, but only the parts that are accurately described and clearly bound in the caption can be passed to the model as text-conditioned supervision.
Thus, a fundamental question arises: Beyond scaling up models, data, and compute, can we improve the learning of generation models by increasing the image information carried by captions?
In this new work, ByteDance's Seed team investigated this question. The core conclusion can be summarized in one sentence:
What truly scales with text conditioning is not the number of tokens in a caption, but the image information within it that can be utilized by the model.
- Paper Title: Scaling Properties of Text Conditioning in Visual Generation
- Authors: Zilong Chen, Chaorui Deng, Kunchang Li, Hongyi Yuan, Haoqi Fan Affiliation: ByteDance Seed
- Paper: https://arxiv.org/abs/2607.29679
- Project Page: https://heheyas.github.io/context-scaling
- Code: https://github.com/heheyas/context-scaling
- Models: https://huggingface.co/collections/heheyas/context-scaling
- Online Demo: https://heheyas-context-scaling.hf.space/
- Hugging Face Paper: https://huggingface.co/papers/2607.29679
Why Don't Models Get Stronger When Prompts Get Longer?
An intuitive approach is to write training captions or user prompts longer and in more detail. More tokens seem like they should mean more supervision and help the model generate more complex images.
However, experiments gave a different answer. On various existing open-source text-to-image systems, natural language prompts quickly saturated with increasing length, with final performance even falling below that of their respective shortest prompts. Even training diffusion models specifically on the same set of long-text captions yielded limited benefits.
To observe this phenomenon more clearly, the team designed an image reconstruction experiment with a fixed backbone network. For the same reference image, the team generated four natural language captions of progressively increasing detail from the same set of complete annotations, then used the same Qwen-Image model and random seed to attempt to reconstruct the image. These captions described the same entities and relations, with later versions mainly adding length by supplementing and expanding the natural language expressions.
Surprisingly, although the captions became significantly longer, the quality of image reconstruction hardly improved. The added prose mostly explained, rephrased, or connected already present content, rather than continuously adding new, stably usable visual variables.
Figure 3 from the paper | Fixed-backbone reconstruction experiment: Reconstruction plateaus as NL Caption continues to lengthen, but gradually restoring SP fields yields continuous improvement.
This indicates that caption length is only a weak proxy variable. A text can be very long yet still fail to clearly specify which attribute belongs to which object, what relation exists between two objects, their respective locations, and their front-back order in the scene.
How to Measure the True Image Information in a Caption?
If token count is insufficient to gauge supervision strength, we need to directly measure the information bound to the image within a caption. For this purpose, the team adapted two complementary metrics from existing work: Grounded Perplexity Gain (GPG) and Effective Detailness (ED).
GPG: How much does the image make the caption "more predictable"?
GPG is a white-box metric that requires reading model token probabilities. For the same caption, the team separately had a frozen vision-language model see and not see the paired image, and calculated the increase in log-likelihood for the caption's content tokens after the image was presented. If the caption contains a large amount of information tightly bound to that image, seeing the image should significantly enhance the model's predictive ability for those tokens.
ED: How many reliable image attributes does the caption cover?
ED is a black-box semantic metric that does not rely on token probabilities. It extracts attributes with entity contexts from the image and the caption separately, then calculates the accuracy of caption attributes and the recall of image attributes. It finally uses F0.5, which places more weight on accuracy, applying stronger penalties to descriptions in the caption without visual grounding.
The two metrics approach the same problem from different angles: GPG focuses on the statistical dependency between image and text, while ED focuses on whether the caption accurately covers verifiable visual content.
Figure 6 from the paper | Definitions and measurement trends of GPG and ED.
Caption Information Content Can Predict Diffusion Model Training Loss
Next, the team fixed the images, model architecture, initialization method, optimization configuration, and training budget, varying only the training captions. The entire experiment included 15 caption configurations: three natural language versions of different lengths, six Structured Prompt versions gradually restoring fields, and six variants with spatial expression or field masking. Each configuration started from the same BAGEL continued-training checkpoint and independently trained a diffusion model.
The results showed no consistent relationship between the token count of natural language captions and training outcomes. However, when the x-axis was changed to caption information content, configurations of different formats and detail levels fell onto highly regular curves:
- The converged diffusion loss had an approximately linear relationship with GPG, Pearson r = -0.984.
- The converged diffusion loss followed a power-law trend with ED, with Pearson r = -0.971 in log-log space.
- GPG and ED also showed high consistency in ranking different caption configurations, Spearman ρ = 0.96.
The team refers to this as the scaling properties of text conditioning. It is not a theoretical law holding for all models, but an empirical calibration obtained under fixed architecture and training recipe. However, it provides two immediate benefits.
First, it transforms caption information content from a vague "data quality" concept into a trainable variable that can be controlled and measured: when model, images, and compute are fixed, information content can predict the final training loss the model achieves.
Second, after performing one calibration, candidate caption schemes can be compared using GPG or ED under the same training recipe before deciding whether to invest in expensive diffusion model training. For six caption variants not involved in the fitting, the two metrics still accurately predicted their convergence loss.
Figure 7 from the paper | Under a fixed training recipe, the convergence loss shows a stable relationship with caption information content.
Structured Prompt: Making Information Not Just More, But Easier for the Model to Use
The earlier experiments reveal a crucial point: merely adding natural language prose is not enough; the new information also needs to be organized in a stable and unambiguous manner.
Therefore, the team proposes Structured Prompt (SP), using structured JSON to represent the visual variables in an image. It consists of three layers:
- Global Layer: Scene intent, setting, atmosphere, style, lighting, and photographic information.
- Element Layer: Each subject's identity, attributes, actions, position, optional depth, and local photographic information.
- Relation Layer: Positional, occlusion, interaction, and semantic relations between different elements.
Compared to free text, the key of SP is not just the "JSON" appearance, but placing different visual variables into stable named fields, reducing ambiguity in attribute assignment, spatial relations, and object binding. In the fixed-backbone reconstruction experiment, reconstruction quality continuously improved as SP fields were gradually restored; in the full training sweep, increased field coverage also consistently raised GPG, ED, and lowered the converged diffusion loss.
Figure 5 from the paper | Structured Prompt organizes global, element-level, and cross-element visual variables into named fields.
To generate complete SP for large-scale training data, the team constructed an image-to-SP annotation pipeline. A general VLM handles global semantics and local content, Sapiens supplements human pose evidence, DepthAnything V2 provides relative depth, SAM 2.1 provides masks and occlusion cues, and finally a VLM unifies this information into a consistent, complete SP.
Figure 8 from the paper | VLM and experts for pose, depth, and segmentation collaboratively construct a complete Structured Prompt.
The team terms the ability of a caption representation to expose and organize image supervision for a diffusion model as Diffusability. SP enhances precisely this aspect: without changing the diffusion model architecture, it allows the model to learn more and clearer visual variables from the text condition.
Promptability: With a Good Structure, You Still Need an LLM to Fill It Well
During training, complete SP can be extracted from paired images, but during actual generation, only the user's sentence is available—there is no reference image or oracle annotation. The system also requires an LLM prompter to expand the user request into a detailed, coherent SP that does not violate the original constraints.
The team calls this ability to instantiate structured conditions from user requests Promptability. End-to-end generation quality depends on the joint effect of both sides:
Generation Quality = Diffusability × Promptability
The multiplication sign here is an organizational perspective, not a mathematically derived formula: Diffusability describes what the diffusion model can learn from the caption representation, and Promptability describes whether the LLM can truly produce a high-quality caption instance during inference.
Figure 4 from the paper | Structured Prompt connects annotation, measurement, diffuser training, prompter training, and final generation.
First, the team fixed the SP schema and Qwen-Image diffuser, only replacing the zero-shot LLM prompter. As Qwen3.5 scaled from 0.8B to 397B, GenEval++ score in thinking mode improved from 46.4% to 86.8%. Apart from the smallest model which tended to repeat during thinking and failed to output valid JSON, chain-of-thought provided further improvements at other scales. This indicates that the model capability and reasoning ability of general LLMs can be directly translated into better image generation results through the caption interface.
However, zero-shot LLMs still tend to generate SP lacking in information and with relatively simple composition. To further improve Promptability, the team adopted three-stage training:
SFT learns the distribution of SP content expected by the diffusion model, not just the JSON format.
Cold-start distills "how to deduce SP solely from user requests" from privileged reasoning traces paired with images.
RFT continues optimization on rollouts generated and rendered by the prompter itself, where a verifier selects high-confidence trajectories, and then provides dense token supervision through on-policy self-distillation from an image-conditioned teacher.
Ablation experiments show the three stages serve different purposes: SFT brings the largest single-stage structural improvement, cold-start strengthens the deduction from user requests to SP, and verifier-gated OPSD achieves the strongest results within the prompter's own distribution.
Figure 9 from the paper | With fixed schema and diffuser, generation quality improves with LLM prompter scale and reasoning mode.
Figure 10 from the paper | Three-stage prompter training: SFT, cold-start, and verifier-gated RFT.
Structured Representation Also Makes the Generation Process Easier to Iteratively Correct
Another natural advantage of SP's field-based representation is that when errors appear in the generated image, the system can locate and modify the corresponding object, attribute, relation, or layout fields, rather than rewriting the entire natural language prompt.
Based on this, the team built a refine-render-judge loop. In each round, the prompter generates or revises SP based on user request and historical feedback, the fixed diffuser renders an image, and an online judge provides PASS/FAIL decisions along with specific issues regarding prompt adherence, structure, and visual quality. If it fails, the next round only needs adjustments around the relevant fields.
Experiments show that increasing iteration budget can further improve structural alignment, adherence, and GSB performance; however, effective reasoning length is not long. For the trained prompter, even when allowed up to 8 rounds, an average of only 2.31 rounds were used; increasing Tmax from 4 to 8 yielded minimal additional benefit. This indicates that text-to-image generation does benefit from iterative error correction, but under the current setup, does not require very long prompt-side reasoning trajectories: after fixing major specification errors, additional rounds saturate quickly.
Figure 14 from the paper | Agentic reasoning loop of refine-render-judge.
Figure 15 from the paper | The loop can correct issues with object splitting, relations, and overall layout.
How Much Improvement Does the Structured Interface Bring to the Same Qwen-Image Base?
The final system consists of an SP-trained diffuser and a trained LLM prompter. It outperforms all compared open-weight models on almost every reported metric and reaches or surpasses most compared closed-source systems on the majority of evaluations, with advantages particularly pronounced on composition, reasoning, and world knowledge tasks.
More crucially, the matched control. To rule out explanations like "it just trained more," the team trained an additional system using the exact same Qwen-Image architecture, training images, training stages, and budget, but consistently using free natural language captions. The results are as follows:
Table 2 from the paper | Complete comparison with representative text-to-image systems. Screenshot retains evaluation definitions, bold text, and footnotes from the paper.
The additional training for the matched NL system indeed brought some improvements, but far from enough to replicate the SP system's results. This indicates that the gains cannot be simply attributed to a larger backbone or more training but are closely related to the structured caption interface used between the prompter and diffuser.
Figure 11 from the paper | Qualitative comparison on complex spatial relations, quantities, and attribute binding.
The Next Step is Not Just to Scale the Model, But Also to Scale the Condition Itself
The starting point of this work is simple: For text-to-image models, captions are not irrelevant metadata, but the primary interface through which image content enters text-conditioned learning.
When captions merely become longer, the new tokens may just rephrase and elaborate; when image information is accurately extracted, clearly bound, and stably organized, the same generation model can learn more from it. GPG and ED make this information a measurable variable, Structured Prompt improves Diffusability, and LLM scaling, post-training, and short-range agentic refinement improve Promptability.
Therefore, the next step in scaling text-to-image should not only focus on "how large the rendering model is" but also ask:
How much image information—information it can truly learn and use—is the text condition passed to the model actually carrying?
This article is from the WeChat official account "Machine Heart"
