DiffusionGemma generates text by starting with 256 random tokens and refining them over several passes. I began with a simple question:

Are all of those random tokens equivalent, or do the thousands that never appear in real text act like hidden mask symbols?

The question mattered because the model draws noise from its entire vocabulary. That vocabulary contains 6,227 literal <unusedN> entries. I found none in ten million tokens of normal text, yet a fully random starting sequence contains about six on average. If the model treated them as special, the design of its vocabulary would quietly influence generation.

I tested the idea across 2,304 conditions, with the rules fixed before I saw any results. The answer was no: unused tokens behaved no differently from the ordinary tokens used for comparison. But one of my safeguards found something more important. The same input produced different predictions when processed alone and when duplicated into a batch. I reran the original study one sample at a time, then investigated why batching changed the answer.

One effect. Different machinery.

On an H100, batch sizes one and two matched each other, while sizes three and four matched each other, but the two groups disagreed. I traced the first difference to the model's router, which decides which internal experts handle each token. Only 0.051% of its scores changed. Within two layers, every token position was affected; by the final output, 98.82% of token scores differed.

I repeated the test on an A100. The output still depended on batch size, but the first difference appeared in the model's attention layer instead. The problem appeared on both GPUs, but it began in a different part of the model on each one.

How the difference spreads

A tiny rounding difference reaches the entire output.

Router · first difference

0.051%

of the recorded router scores changed.

Two layers later

256 / 256

token positions carried the difference.

Final output

98.82%

of the final token scores differed.

H100 A100
Batch pattern B1 = B2 | B3 = B4 Every size differed
First difference Model router First attention layer
Schematic view of the measured spread. The percentages and counts come from the H100 study; the grids show their relative scale.

Results at a glance

Measurement Result
Generation runs that diverged 24 / 24
Final texts that changed 22 / 24
Stopping points that changed 19 / 24
Denoising passes matching after both fixes 293 / 293

Problems uncovered

To test the cause, I made the router process each sequence separately. The first denoising pass then matched exactly. The second still diverged when the model fed its previous predictions back into itself, a process called self-conditioning. I traced that difference to the matrix multiplication that turns token probabilities into embeddings.

Two points in the denoising loop

The first fix exposed the second problem.

DiffusionGemma denoising loop and the two repaired operations Tokens pass through embeddings, the transformer, token scoring, and sampling. Predictions loop back into the next pass through self-conditioning. The first repaired operation is in the model router and the second converts probabilities back into embeddings. next denoising pass self-conditioning Token canvas256 positions Embeddingtokens + prior prediction Transformerrouter projection Token scoresper position Samplercommit Probabilitiesprevious pass Probability × embeddingmatrix multiplication 1 2
1

Router projection. Processing each sequence separately made the complete first pass match.

2

Self-conditioning calculation. Processing it row by row made every later pass match.

The solution

I ran both operations one sequence at a time throughout generation. The batch-of-four run then matched the single-input run in every recorded output: sampling decisions, random choices, stopping point, final tokens, and final text. All 24 test cases matched.

After both fixes, every target-token score matched byte for byte across all 293 denoising passes.

157.3 billion output bytes checked across both runs

What this establishes

Batching is supposed to change throughput, not the answer. Here, the number of identical inputs packed onto the GPU changed the model's generation path and final text. On the tested H100 setup, I traced the behavior to two matrix operations and showed that fixing them reproduced the single-input output exactly.

This study covers one model and one software setup. It does not show an accuracy change, a speedup, or a production-ready fix. It does show that low-level GPU calculations can steer a model's generation and should be treated as part of the system's behavior.

Use the controls below to see how each batch size changed the denoising path, and how the two fixes brought every run back into exact agreement.

pass 13 / 13

Batch sizes 1–4 · identical inputs · only the number of rows changes

Schematic token states; all reported counts come from the study.
noise committed token differs from B1