Fix qwen image prompt padding #12075

[sambhavnoobcoder]

Fix QwenImage Prompt Embedding Padding for Deterministic Outputs

What was the issue?

Issue #12075 reported that QwenImage pipelines were producing non-deterministic outputs when using the same prompt across different batch sizes. The same text prompt would generate different images depending on whether it was batched alone or with other prompts of varying lengths.

This inconsistency violated a fundamental expectation: identical prompts with the same seed should always produce identical outputs, regardless of batch composition.

How I identified the problem

After reviewing the issue report and examining the QwenImage pipeline implementation, I discovered the root cause in the prompt embedding padding logic.

The pipelines were dynamically padding prompt embeddings to the maximum sequence length within each batch, rather than using a fixed padding length. This meant:

• A short prompt batched alone would be padded to its own length
• The same short prompt batched with a longer prompt would be padded to the longer prompt's length
• Different padding created different Rotary Position Embedding (RoPE) position assignments
• RoPE uses a shared position space for text and image tokens, so inconsistent text positions led to inconsistent image generation

The problem existed across all 8 QwenImage pipeline variants (main, img2img, inpaint, edit, edit_inpaint, edit_plus, controlnet, controlnet_inpaint) and the modular encoder functions.

How I solved it

The solution involved ensuring all prompt embeddings are padded to a consistent, fixed length determined by the max_sequence_length parameter (default 512, configurable up to the model's 1024 token limit).

I modified the padding logic in all affected locations to:

1. Use fixed-length padding: Changed from batch-maximum padding to max_sequence_length padding
2. Propagate the parameter: Added max_sequence_length parameter to all internal prompt encoding methods
3. Update RoPE sequence lengths: Changed txt_seq_lens to reflect the padded length instead of actual token counts
4. Handle vision tokens: Added truncation logic in Edit pipelines where image tokens are processed through the text encoder
5. Maintain backward compatibility: Kept the default max_sequence_length=512 to preserve existing behavior for users

The fix ensures that any prompt will always receive the same padding and RoPE positions, regardless of batch composition, making outputs deterministic and reproducible.

How the fix was tested

I created a comprehensive test test_prompt_embeds_padding() that verifies three critical behaviors:

1. Fixed padding to max_sequence_length: Confirms short prompts are padded to the full length (not just to their token count)
2. Batch consistency: Validates that prompts in a mixed-length batch all receive the same padding length
3. Custom max_sequence_length support: Tests that specifying a custom value (e.g., 512) correctly truncates and pads to that length

Additionally, I ran the entire QwenImage test suite to ensure no regressions were introduced. All structural tests pass successfully, with only expected value assertion changes (since fixing the padding changes the numerical outputs).

Test Results

Fixes : #12075
cc : @sayakpaul @yiyixuxu

[sambhavnoobcoder]

can i get some initial reviews on this @sayakpaul ?

[sambhavnoobcoder]

@yiyixuxu could you please give me an initial review on this PR ? I'll start making changes according to the same .

[sambhavnoobcoder]

Hi @yiyixuxu @sayakpaul ,
I was wondering if i could get some initial comments and reviews on this please ? I'll make the necessary changes to this accordingly .

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sambhavnoobcoder]

not stale

[sayakpaul]

Take note of #12702

[sambhavnoobcoder]

Cool @sayakpaul . took a pull from main , updated this to the latest version , as well as accounted for all the changes from #12702 . Would really appreciate a review now here for the same .

[sayakpaul]

Could you walk us through the main changes after you pulled in the changes introduced in #12702?

[sambhavnoobcoder]

yes sure , so after integrating PR #12702, which removed the redundant txt_seq_lens parameter and shifted to attention mask-based inference for cleaner code, I first preserved my fixed padding approach from PR #12643 by retaining the tokenizer_max_length (default 1024) in encoding functions like get_qwen_prompt_embeds_edit in encoders.py; this ensures deterministic outputs by always padding prompts to a consistent length, preventing RoPE position variations across batch sizes for the same prompt and seed. Next, I eliminated the txt_seq_lens variables and parameters entirely from all pipeline files, such as pipeline_qwenimage.py and pipeline_qwenimage_controlnet.py, allowing the transformer to infer sequence lengths directly from the generated attention masks, which simplifies maintenance and reduces code redundancy without conflicting with my padding fix. Then, I adopted PR's prompt template constants by importing them from prompt_templates.py into the encoding functions, improving code organization and modularity while keeping my fixed padding intact. Additionally, I added config specs for tokenizer_max_length to classes like QwenImageEditTextEncoderStep in encoders.py, enabling proper propagation of the fixed length value through modular pipelines. Finally, I performed pipeline-specific cleanups by removing txt_seq_lens from transformer and controlnet calls in specialized files like pipeline_qwenimage_edit.py and pipeline_qwenimage_inpaint.py, where special logic like summing attention masks is now handled internally by the transformer, ensuring compatibility and maintaining overall functionality as verified by passing tests like test_prompt_embeds_padding.

Do let me know if i missed anything here , i'll incorporate that as well . Else you can review the code , i would resolve anything you would point out in the same .