[Image Generation AI] What are CLIP and T5xxl? Text encoders make illustrations look this beautiful!
Introduction
Hello, I'm Easygoing.
Today, I'm thinking about text encoders for image generation AI.
Text encoders are dictionaries
AI understands the text we input by converting it into a format that machines can comprehend.
The component that performs this translation is the text encoder, and its function is equivalent to a dictionary that translates human language into machine language.
How does changing the text encoder in an image generation AI affect image quality?
Comparing actual images!
Now, let's take a look at how images actually change in the new image generation AI, Flux.1.
Flux.1 is equipped with two types of text encoders.
T5xxl: Primarily understands the context of the prompt
CLIP-L: Primarily understands short sentences and words
This time, I will try replacing these T5xxl and CLIP-L with higher-precision versions.
T5xxl-FP16 + CLIP-L-FP16 (Original)
Flan-T5xxl-FP16 + CLIP-L-FP16
Flan-T5xxl-FP32 + CLIP-L-FP32
Flan-T5xxl-FP32 + CLIP-GmP-ViT-L-14-FP32
Flan-T5xxl-FP32 + Long-CLIP-GmP-ViT-L-14-FP32
The text encoders used this time perform better the further down the list they are.
By changing the text encoder, you can see that the details of the building on the right become finer and the image quality improves.
Note that while the Long-Clip-L model at the bottom can be used in ComfyUI, it cannot be used in Stable Diffusion webUI Forge.
Also, to use FP32 format text encoders, the --fp32-text-enc setting described later is required.
Let's take a closer look at text encoders
Now, let's take a closer look at text encoders.
First, the text encoders installed in major image generation AIs are as follows.
T5xxl and CLIP are the text encoders, while UNET and Transformer are the parts that generate images based on the analyzed information.
CLIP is the foundation of everything
CLIP is an AI developed by OpenAI that can understand both text and images.
Open-CLIP was developed by reverse-engineering CLIP as open source, and nowadays, Open-CLIP is often shortened to just CLIP.
There are the following types of CLIP, depending on their performance:
CLIP-L is an improved version of CLIP-B, and many AI image generators are equipped with CLIP-L.
Long-CLIP-L is a model improved from CLIP-L to be able to understand long sentences as well.
CLIP-G is a model that increases the number of parameters compared to CLIP-L to improve overall performance; while the number of tokens remains the same, it can now understand long prompts of over 200 words by emphasizing and reproducing important elements.
T5xxl understands context
T5xxl is a text-to-text generation model developed by Google, and it has become the fundamental technology for today's AI services, such as AI chat and translation AI.
Theoretically, T5xxl can handle very long sentences, but even so, accuracy decreases as the text gets longer.
T5xxl v1.1 and Flan-T5xxl have the same number of parameters, but overall accuracy has improved due to efficient additional training.
The number of text encoders is increasing
New AI image generators are now equipped with multiple text encoders to improve the accuracy of prompt understanding.
Stable Diffusion 1: Understanding with words and short sentences
Stable Diffusion 1, released in July 2022, used CLIP-L as its text encoder.
Since CLIP-L could only understand a small number of tokens, it was necessary to use techniques such as writing in short, word-separated segments and placing important keywords at the beginning.
Stable Diffusion XL: Understanding long sentences
Stable Diffusion XL, released in July 2023, was equipped with a new CLIP-G in addition to the conventional CLIP-L as a text encoder.
CLIP-G has higher performance than CLIP-L, can understand longer sentences, and now allows prompts to be entered as long natural language sentences.
The SDXL model is 7GB in size, and text encoders like CLIP-G account for 1.8 GB of that, showing that SDXL places a high priority on prompt understanding.
Stable Diffusion 3: Understanding Context
Stable Diffusion 3, released in June 2024, improved its text comprehension by incorporating T5xxl in addition to CLIP-L and CLIP-G.
T5xxl is high-performance, but it is also large, with the encoder portion alone being 9GB in size.
Because the text encoder has become so massive, it has become common since Stable Diffusion 3 to operate the text encoder separately from the rest of the model.
Flux.1: CLIP-G is not included
Flux.1, released in August 2024, uses two text encoders, CLIP-L and T5xxl, and does not include CLIP-G.
This is likely based on the idea that the functionality of CLIP-G can be sufficiently covered by T5xxl.
Based on the presence or absence of CLIP-G, Stability AI claims that Stable Diffusion 3.5 has superior language understanding compared to Flux.1, but in reality, there are almost no situations where one feels a lack of prompt understanding with Flux.1.
Furthermore, even with the previous generation CLIP-G, practical understanding is sufficient even when entering long prompts.
Improved Text Encoders!
Now, I will introduce the links to the improved text encoders used this time.
Improved CLIP-L
CLIP-GmP-ViT-L-14
CLIP-GmP-ViT-L-14 is an improved model of CLIP-L developed and released for free by an individual named Zer0int.
The reason for development is apparently because I like CLIP, and he says he trained it on his home PC using an RTX 4090.
CLIP-GmP-ViT-L-14 improves accuracy over the conventional CLIP-L using a method called Global mean Pooling (GmP: Geometric Parameterization), and on the ImageNet/ObjectNet benchmarks, performance is significantly improved with a 90% accuracy rate compared to the 85% accuracy rate of the original CLIP-L.
According to Zer0int, CLIP-GmP-ViT-L-14 improves upon the excessive fixation in image understanding of CLIP-L.
There are multiple files on the CLIP-GmP-ViT-L-14 download page, and in addition to the original FP32 version, an FP16 version of the further improved ViT-L-14-BEST-smooth-GmP-TE-only-HF-format.safetensors is also available.
If you are unsure which one to use, you should download this FP16 version.
Long-CLIP-GmP-ViT-L-14 (ComfyUI only)
Long-CLIP-L is a model improved to support inputs of up to 248 tokens by extending the 77-token limit of the standard CLIP-L model, which allows even CLIP-L to handle longer prompts.
Currently, only ComfyUI can use Long-CLIP-L, and it cannot be used in Stable Diffusion webUI Forge.
On the download page, in addition to the original FP32 version, an FP16 performance-improved version, Long-ViT-L-14-BEST-GmP-smooth-ft.safetensors, is also distributed.
Updated 2024.12.31
I compared the effects of the improved CLIP-L using actual images.
Flan-T5xxl (Improved T5xxl)
Next is the improved version of T5xxl. Flan-T5xxl is a model that has been further trained on the regular T5xxl to improve its accuracy.
Flan-T5xxl Original (Split version)
The original Flan-T5xxl released by Google is distributed in split parts because of its large size (44GB for the FP32 format).
Flan-T5xxl Merged Version
This is a file based on the original that has been merged for use in AI image generation.
In addition to the simple merged version, I am also distributing a TE-only version that extracts only the text encoder portion used in Flux.1 / SD 3.5.
How to use the Flan-T5xxl model
It is distributed in FP32, FP16, and GGUF formats.
Using the FP32 format text encoder
Text encoders are usually processed in FP16 format.
When using an FP32 format text encoder, you need to configure settings at startup.
Here, I will introduce how to set it up in Stability Matrix.
ComfyUI
Set --fp32-text-enc at startup.
Stable Diffusion webUI Forge
Set --clip-in-fp32 at startup.
It is difficult to upgrade the text encoder for SDXL
This time, I tried upgrading the text encoder for Flux.1.
Flux.1 and SD 3.5 operate with separate text encoders, so they can be easily upgraded, but SDXL and SD 1.5 are integrated into the main body so upgrading them is somewhat more difficult.
I would like to write another article about this next time!
Summary: Let's try changing the text encoder!
CLIP converts text into vectors
T5xxl understands context
Improved text encoders have been released
When thinking about image generation quality, I think we often focused on the transformer part that generates the image and put the text encoder on the back burner.
Through this verification, I found that the text encoder also significantly affects image quality.
The improved CLIP-L I introduced this time clearly improves image quality despite its small size, so I recommend that everyone try it out.
Thank you for reading to the end!
Creator Introduction
Now, the creator I would like to introduce this time is Shiki. Shiki expresses a unique world with cute cat-eared characters and beautiful use of color.
It seems that the illustrations are generated using Leonardo AI, and then finished by adjusting the colors in the Google Photos app.
The impression of an illustration changes significantly depending on the color expression. The beautiful skin tone expression is a must-see!
Introduction to the comparison article between Flan-T5xxl and T5xxl_v1.1
matataByy has published an article that compares T5xxl in detail, including Flan-T5xxl and T5xxl_v1.1.
According to it,
Flan-T5xxl-FP32 tends to have better prompt comprehension
Flux.1[dev] seems to have better prompt adherence than Flux.1[shnell]
Everyone, please check it out!
Update History
2025.8.12
Partially revised the article content
2025.4.20
Added instructions on how to launch the FP32 format text encoder in Stable Diffusion webUI Forge
2025.3.9
Partially revised the article following the release of Flan-T5xxl_TE-only
2025.1.12
matataByy added an introduction to the article by
2024.12.15
Added instructions on how to use the FP32 format text encoder in ComfyUI