SYSTEM NOTICE

Auto translation by AI. Be sure, accuracy, nuances and authorial intent may not be fully reflected.
見出し画像

Introducing DiffusionGemma: An Attempt to Speed Up Text Generation by Blocks

Webtech Watcher

What you will learn in this article

In this article, we will organize the following points regarding DiffusionGemma, which was released by Google DeepMind.

  • What kind of model is DiffusionGemma?

  • How is it different from conventional text generation?

  • What does it mean to "generate blocks simultaneously"?

  • In what situations might it be useful?

  • Points to keep in mind before using it

It is easier to understand DiffusionGemma as an experimental model for exploring text generation speed from a different angle, rather than a finished model that will immediately replace all use cases.

For developers interested in speed, this is a step worth watching.

What is DiffusionGemma?

DiffusionGemma is an experimental open-weights model released by Google DeepMind.

It is based on the Gemma 4 26B A4B Mixture-of-Experts architecture, which consists of multiple expert components. While it is introduced as a 26B scale model in the announcement, the model card states that the total parameter count is 25.2B, with 3.8B parameters active during inference.

The license is Apache 2.0. A major feature is that the weights are publicly available, making it easy to access for research and development.

Note that DiffusionGemma is also a multimodal model that can include images and video frames as input, in addition to text. However, the output is text. In this article, we will focus specifically on the mechanisms related to the speed of text generation.

What is important here is not to view DiffusionGemma simply as a "new high-quality version of Gemma 4," but to see it as an experimental model that attempts to change the very mechanism of text generation.

Many large language models generate text from left to right, one token at a time. DiffusionGemma deviates slightly from this by handling chunks of text simultaneously.

This difference affects speed and areas of expertise.

Conventional models operate by proceeding "one token at a time"

Many of the language models currently in common use operate on a method called autoregressive.

While it may sound complicated, the concept is quite straightforward. It decides the next item one by one while looking at the words or characters that came before it.

The tokens referred to here are close to the smallest unit the model uses to handle text. In Japanese, this can be a single character, or it can be part of a word or a short phrase.

For example, the process flows as follows.

Today
Today the weather
Today the weather is
Today the weather is nice so for a walk

In this way, the continuation is added little by little. As a sentence, it is natural and tends to be of high quality. On the other hand, the mechanism involves waiting in line.

To output the next token, the previous token must be determined. This often becomes a bottleneck for generation speed.

When processing many users at once in the cloud, there are many situations where it can run efficiently. However, in scenarios where one person uses it on a local GPU, the GPU's computing power may not be fully utilized.

Even with a high-performance GPU, the model continues the flow of "determining the next one by one," leaving wait times. This is the room for improvement that DiffusionGemma is targeting.

DiffusionGemma thinks like "revising a draft all at once"

The concept of DiffusionGemma is close to the diffusion models used in image generation AI.

In image generation AI, you start from a noise-like state and gradually make the image clearer. DiffusionGemma brings a similar flow to text.

Admittedly, research into using the idea of diffusion models for text generation has existed for some time. The feature of DiffusionGemma is that it makes this direction an open-weights model based on Gemma 4, in a form that developers can easily try out.

Instead of creating a perfect sentence from left to right from the start, it first prepares a "canvas" for the text. It places temporary tokens there and refines them into a sentence-like form while reviewing them multiple times.

According to official information, this canvas is handled in units of 256 tokens. In other words, instead of determining one token at a time in sequence, it looks at a block of up to 256 tokens in parallel and refines them through several reviews.

However, 256 tokens are not completed as is in a single process. By making the entire block visible at once, it gradually approaches a more certain form through multiple denoisings, or processes that remove noise.

To use a familiar example, conventional models are close to the method of "writing a sentence in order from the beginning." DiffusionGemma is close to the method of "placing a rough draft of the entire paragraph and revising it multiple times while looking at the whole thing."

This point of "revising while looking at everything at once" is the major difference from the conventional method of proceeding one token at a time.

The reason for the speed is being able to hand over a substantial amount of work to the GPU

DiffusionGemma is introduced as a model that aims for up to 4x faster text generation on a GPU.

The speedup referred to here does not mean it will always be 4 times faster in every environment. It is said to be particularly effective in local inference using a dedicated GPU or in situations with few concurrent requests.

Why does it become faster?

In conventional token-by-token generation, the GPU repeatedly reads model weights while producing results bit by bit. This process is prone to bottlenecks in memory read/write operations.

On the other hand, DiffusionGemma processes a 256-token canvas all at once. For the GPU, this means receiving large calculations at once, making it easier to leverage its computational performance.

The official announcement indicates figures of over 1,000 tokens per second on an NVIDIA H100 and over 700 tokens per second on a GeForce RTX 5090. While these numbers look fast, they should be viewed as results under specific hardware and conditions.

Since DiffusionGemma is designed to speed up processing by passing large calculations to dedicated GPUs, the effects may be less noticeable in unified memory environments like Apple Silicon.

The Meaning of "Seeing the Whole"

A key feature of DiffusionGemma is its ability to view context bidirectionally.

Conventional autoregressive models basically decide the next token while looking at what was previously generated. While they perform sophisticated processing internally, the flow of generation proceeds from left to right.

DiffusionGemma adjusts the 256-token canvas currently being generated by looking at it bidirectionally. It refines the text by looking not only at the preceding parts but also at the candidates placed ahead.

In long texts, it proceeds to the next block while using the confirmed blocks as context. In other words, rather than seeing the entire long text simultaneously, it is designed to view tokens in parallel within a block and proceed sequentially between blocks.

This property is considered effective for tasks such as the following:

  • Naturally filling in the middle of a sentence

  • Completing parts of code

  • Maintaining the consistency of Markdown brackets and headings

  • Handling problems that require looking at overall constraints, such as Sudoku

The ability to see the preceding and following context simultaneously is particularly helpful for "filling in the middle" tasks.

For example, in a situation where you are adding an explanation to the middle of an existing text, you want to match the flow not only with what comes before but also with what comes after. Writing sequentially from left to right can sometimes lead to unnatural connections with the following text.

A method like DiffusionGemma seems well-suited for this type of work.

What kind of usage can be envisioned?

Since DiffusionGemma is an experimental model, it is best suited for testing in "small workflows where speed matters."

For example, text completion within an editor. Rather than having it write an entire long article, it is better suited for purposes such as refining parts of existing text or adding short explanations under headings.

When responses are fast, it is harder to break the rhythm of writing. Just having a short wait time significantly changes how easy it is to experiment.

Another use case is code completion or making small corrections.

Rather than generating an entire function from scratch, it fills in missing processes within existing code. The task of filling in gaps while looking at the surrounding context aligns well with the concept of adjusting the whole in block units.

It also has potential for interactive tools running in local environments. In scenarios such as summarizing internal documents locally, integrating into local development support tools, or repeatedly regenerating candidates on a UI, the ability to return results quickly and try again multiple times is more valuable than generating the highest-quality long text in one go.

The direction of DiffusionGemma is close to these types of use cases.

However, standard Gemma 4 has the advantage in quality.

This is a point worth looking at carefully.

Even in the official announcement, DiffusionGemma is described as an experimental model that prioritizes speed and parallel generation, and standard Gemma 4 is recommended for applications requiring maximum quality.

In fact, in the model card's key benchmarks, the standard Gemma 4 26B A4B outperforms DiffusionGemma in many categories.

In other words, DiffusionGemma is not a model that is "always better because it is fast."

For products where quality is the top priority or for long-form generation where accuracy is critical, standard Gemma 4 is more suitable. Especially for business documents where errors are not tolerated, professional answers, or complex reasoning, it is safer not to judge based on speed alone.

DiffusionGemma is an option for testing speed and new generation methods. If you view it that way, you can maintain a balanced perspective without expecting too much or underestimating it.

Things to check before using it

It is noted that DiffusionGemma can be used via Hugging Face Transformers, vLLM, SGLang, MLX, and others. If you are trying it locally, you should check the GPU memory requirements first.

The official announcement states that when quantized, it fits within about 18GB of VRAM on high-performance consumer GPUs. However, actual usability will vary depending on the quantization method, inference framework, input length, and the number of concurrent processes.

Rather than assuming large-scale use cases from the start, it is better to observe its behavior with short questions or small completions.

If you are going to try it, the following flow is a good starting point.

  1. Check normal response speed with short questions

  2. Try completing text in the middle of a sentence

  3. Try small code completions

  4. Checking how quality degrades in longer texts

  5. Comparing with existing autoregressive models

Comparing them this way makes it easier to see not only the speed but also which types of output remain stable.

In particular, with diffusion-based generation, the text can be rewritten mid-process before completion. Since this differs from the natural left-to-right growth seen in autoregressive models, some ingenuity will likely be needed when displaying intermediate progress in a UI.

Key points for developers to consider

The interest in DiffusionGemma goes beyond just the news of a 'faster model' being released.

Text generation has primarily been a long, one-token-at-a-time process from left to right. Of course, that method is robust and will likely remain the mainstream for the foreseeable future.

However, not all use cases are optimized for that approach.

Rewriting parts of a text, filling in blanks in code, generating well-structured short outputs at high speed, or building local assistants that react instantly to user input.

For these purposes, a method that considers blocks simultaneously becomes a new option.

To handle long texts, DiffusionGemma also adopts a mechanism that combines block-by-block generation with a conventional flow. It generates a 256-token canvas, confirms it, and then proceeds to the next block.

It does not completely abandon sequence; rather, it proceeds in parallel within blocks and stably between blocks. It appears to be a design that is both experimental and mindful of practical use.

It seems particularly well-suited for cases where you want to build low-latency AI tools on a local GPU, or for scenarios where you need to provide short responses repeatedly, such as in-editor completion. The mechanism of reviewing by blocks may also work well for tasks like naturally filling in parts of text or code.

On the other hand, for tasks requiring the highest quality long-form writing, complex reasoning, or strict accuracy, it is best to continue comparing it with standard high-quality models.

DiffusionGemma is less of a finished answer and more of a model that presents a new question.

Is it really necessary for text generation to proceed only one token at a time? Is there a way to generate text that utilizes the power of GPUs more naturally? This is a concrete attempt to address such questions.

It may not immediately change our daily AI usage, but it certainly seems to be connected to the trend of creating AI that is fast, local, and capable of being tested interactively many times.

It is a model I would like to start with by finding its strengths, without rushing.

Reference links

いいなと思ったら応援しよう！