SYSTEM NOTICE

Auto translation by AI. Be sure, accuracy, nuances and authorial intent may not be fully reflected.
見出し画像

Trying out Kolors in ComfyUI (Supports Chinese/English (+Japanese), VRAM 8GB)

まゆひらa

※ Last update 12-30-2024
※ Japanese prompts are only "usable to an extent," so English or Chinese is recommended. It seems to be more proficient in Chinese.



■ 0. Overview

▼ 0-0. Introduction

　In this article, we will use the image generation AI "Kolors" in ComfyUI. Kolors was developed by the Chinese short video streaming service Kuaishou Technology and released around the summer of 2024.

　Kolors uses ChatGLM3 as its Text Encoder, and ChatGLM3 supports English and Chinese. However, in practice, it seems that Japanese is also understood. The VAE is shared with SDXL.

　The prompt for the image posted below is: "On a hill in the countryside, a girl in a pink dress is walking with a dog. Flowers are blooming around, and the blue sea can be seen in the distance." While the bare feet are a bit concerning, you can see its understanding of the prompt, such as flowers blooming only in the surroundings.

　Please refer to the links below for other information.

▼ 0-1. Related Links


■ 1. Preparation for Use

▼ 1-1. Overview

　We will use ComfyUI-Kolors-MZ, which was linked earlier. The installation steps are written there, and there is also information on IPAdapter and ControlNet. Please check the link for details. This article only covers up to simple generation.

　Also, to avoid confusion regarding model types, we have adopted a policy of creating a directory in the default location and placing them under it.

▼ 1-2. Installing Custom Nodes

Open the command prompt, navigate to "ComfyUI\custom_nodes", and run the following command (Git is required). Of course, you can also install it from within ComfyUI using ComfyUI-Manager.

git clone https://github.com/MinusZoneAI/ComfyUI-Kolors-MZ

▼ 1-3. Setting up models, etc.

For models, I will use the official original and an anime model found on Civitai. Please download either or both as you prefer.

For the Text Encoder, ChatGLM3, choose 8bit; if you are concerned about insufficient resources, it is a good idea to compare it with 4bit.

The VAE is the same as SDXL. If you haven't installed it yet, please download it.


■ 2. Generation using the workflow

▼ 2-1. Overview

Considering output quality, the workflow in this article performs t2i followed by upscaling and i2i. I have prepared two types, with and without Tiled VAE. Both are simple originals, so feel free to modify and redistribute them.

▼ 2-2. Supplementary notes

To reiterate, Kolors prompts support English and Chinese, but Japanese seems to work as well. The effectiveness of the prompt seems to vary depending on the language (my impression is "Chinese > English >> Japanese").

▼ 2-3. Workflow (No Tiled)

This simply processes in the order of t2i, upscaling, and i2i. If you have sufficient resources, choose this one; if you are short on VRAM, try the one with Tiled.

Download the workflow and drag and drop it onto the ComfyUI screen. If you also want to save the first-stage image, select the node in the top right and press Ctrl+M to unmute it.

When the workflow was executed
The prompt is "Anime style watercolor painting. A girl with long hair is smiling, wearing a white dress and sandals. There is a black cat next to the girl. It is a natural cliff with flowers blooming around, and the sea and islands can be seen beyond it."

The steps value is set on the lower side, at 15 for the first pass and 12 for the second. Usually, it seems to be around 25 to 30. I have also set the cfg lower. Please adjust it according to the style you want to output and the quality you require.

▼ 2-4. Workflow (with Tiled)

Basically, it is the same as 2-3. By using Tiled (a standard ComfyUI node that processes in segments, currently in beta) for VAE Encode and Decode after upscaling, you can support high-resolution generation. However, the time required for generation will also increase. I have increased the upscale factor from 1.5 to 2.5, so please feel free to change it to your preference.

The usage is the same. Since the resolution is high, it takes 45 seconds (60 seconds including model loading) in my environment.

When the workflow was executed
The prompt is: "A soft-colored, single frame from a Japanese shoujo manga. A little girl with short hair and big eyes, wearing light purple pajamas, reading a book by the window. There is a white rabbit beside her, and the night sky, full moon, and city lights can be seen outside the window."


■ 3. Bonus 1

▼ 3-1. Overview

Using the workflow from 2-3, I will try the prompt in English, Japanese, and Chinese respectively. Please note that the output content may change depending on how it is translated, so use this only as a reference. I used ChatGPT to create the prompts.

▼ 3-2. Anime Style (Amusement Park and Magical Girl)

First is English.

The prompt is: "Japanese flat shade anime style. A cute, infant magical girl with long dark brown hair, wearing a pink and white frilly costume with ribbons and bows. Her cheeks are rosy, her eyes shyly downcast, and she leans in with an innocent, playful smile. Holding a heart-shaped gem wand, she stands in a lively amusement park at sunset, with colorful rides and blurred crowds bathed in a warm golden glow. Soft, watercolor-style with a dreamy, nostalgic vibe."

Next is Japanese. It seems the prompt is not being interpreted well. There is a possibility that it could be improved with some adjustments (not verified).

The prompt is: "Japanese flat-shaded anime style. A cute young magical girl with long dark brown hair. Wearing a pink and white frilly costume, decorated with ribbons and bows. Her cheeks are flushed, her eyes are shyly cast down, and she has an innocent and playful smile. Holding a heart-shaped gem wand, she stands in a bustling amusement park wrapped in the sunset. Colorful rides and blurry crowds are bathed in warm golden light. Soft watercolor style, with a dreamy, nostalgic atmosphere."

Finally, Chinese (Simplified). Due to the design of Kolors, Chinese might be the most suitable language for prompts.

The prompt is: "Japanese flat-shaded anime style. A cute young magical girl with long dark brown hair, wearing a pink and white frilly costume, embellished with ribbons and bows. Her cheeks are slightly flushed, her eyes shyly cast down, revealing an innocent and playful smile, with her body leaning slightly forward. She holds a heart-shaped gem wand, standing in a lively amusement park under the sunset, surrounded by colorful rides and a blurred crowd bathed in warm golden light. The overall presentation is a soft watercolor style, full of dreamy and nostalgic atmosphere."

▼ 3-3. Photorealistic Style (Shopping Mall and Girl)

First is English.

The prompt is: "A close-up of a young girl in a shopping mall, with a glass dome ceiling in the background. She has medium brown hair in twin braids, gray eyes, and a gentle smile. She holds out one hand while lifting one leg and grasping the hem of her sky-blue gothic dress with frilled hems and a large chest ribbon. Her outfit includes a headband, white frilled socks, and black Mary Jane shoes."

Next is Japanese. I will leave the evaluation to you. Note that for some reason, it may output in an anime style.

The prompt is: "A close-up of a girl in a shopping mall with a glass dome ceiling in the background. She has medium brown hair in twin braids, gray eyes, and a gentle smile. She is reaching out one hand, lifting one leg slightly, and holding the hem of her sky-blue gothic dress. The dress has a ruffled hem and a large bow on the chest, and she is wearing a hairband, white ruffled socks, and black Mary Jane shoes."

Finally, here is Chinese (Simplified). In my experience, the prompt fidelity seems higher with Chinese than with English.

The prompt is: "In a shopping mall, with a glass dome in the background, a close-up shot of a little girl. She has medium brown twin tails, gray eyes, and a gentle smile. She reaches out one hand, lifts one leg, and grabs the hem of her sky-blue gothic-style dress, which has ruffles and a large bow on the chest. She is wearing a headband, white ruffled socks, and black Mary Jane shoes."

It looked a bit cramped in the pose, so I swapped the vertical and horizontal dimensions.

The prompt is the same as the previous one.


■ 4. Bonus 2

▼ 4-1. Overview

I used the model that I only introduced in 1-3 to generate images using the prompts from before. As expected, the Chinese prompt seems to be the most suitable. If you are interested, try generating with the first prompt translated into Chinese.

▼ 4-2. Bonus Images

The prompt is: "Anime-style watercolor painting. A girl with long hair is smiling, wearing a white dress and sandals. There is a black cat next to the girl. It is a natural cliff with flowers blooming around, and the sea and islands can be seen beyond it."
The prompt is: "A soft-colored frame from a Japanese girls' manga. A little girl with short hair and big eyes is wearing light purple pajamas and reading a book by the window. There is a white rabbit beside her, and the night sky, full moon, and city lights can be seen outside the window."
The prompt is: "Japanese flat color block animation style. A cute young magical girl with long dark brown hair, wearing a pink and white ruffled outfit adorned with ribbons and bows. Her cheeks are slightly flushed, her eyes are shyly cast down, and she has an innocent yet playful smile, with her body leaning slightly forward. She is holding a heart-shaped gem wand, standing in a bustling amusement park under the sunset, surrounded by colorful rides and a blurred crowd bathed in warm golden light. The overall look is a soft watercolor style, full of dreamy and nostalgic atmosphere."

The composition was having trouble generating in landscape, so I made it square.

The prompt is: "In a shopping mall, with a glass dome in the background, a close-up shot of a little girl. She has medium brown twin tails, gray eyes, and a gentle smile. She reaches out one hand, lifts one leg, and grabs the hem of her sky-blue gothic-style dress, which has ruffles and a large bow on the chest. She is wearing a headband, white ruffled socks, and black Mary Jane shoes."


■ 5. Other

Please follow the menu to find other articles I have written.

If you have any questions regarding this article, please contact @riddi0908 on X.


いいなと思ったら応援しよう！