[SDXL] The True Value of Refiner! What is Born from Destructive Creation
Introduction
Hello, I'm Easygoing.
This time, I would like to take a deep dive into the Refiner for Stable Diffusion XL!
It Started with the Refiner
When I first started using note because I found generative AI interesting, the first thing I was interested in was the Refiner.
In SDXL, you can improve the quality of illustrations drawn with the Base model by using the Refiner model for finishing.
The Refiner has a unique effect and adds personality when used, but it was a difficult feature to use because it breaks the structure.
Three months have passed since I started using the Refiner, and I've noticed a few things during that time, so I'd like to share them today.
Let's Compare First!
First, let's compare illustrations with and without the Refiner.
First, I will generate an illustration at the base resolution of SDXL.
I was able to generate an illustration of a beautiful female operator.
The Refiner Destroys Structure!
Next, I will try using the Refiner on this illustration.
As mentioned earlier, the Refiner significantly disrupts the structure.
In this illustration as well, adding the Refiner causes the outlines to distort, and parts like the hands have collapsed.
It is recommended to use the Refiner for finishing touches after the Base model has finished drawing, but because it disrupts the structure so much, it cannot be applied strongly.
What changed with the Refiner?
So, what exactly changed with the Refiner?
First, let's compare the brightness of the illustration from earlier.
Compared to the original image, the illustration with the Refiner added shows a decrease in the mid-tones and an increase at both extremes.
This indicates that the image's contrast has increased, giving the entire illustration a sharper, more defined impression.
Looking at the actual illustration, it is clear that not only has the contrast simply increased, but the shadows on the face and hair are emphasized, creating a sense of depth, and the saturation is also enhanced, making it more vivid.
Showing its true potential with high-resolution upscaling!
Next, let's try upscaling the illustration to a higher resolution.
Both illustrations look beautiful after being upscaled.
In particular, the Refiner illustration has its structure restored through redrawing after upscaling.
Refiner is not just for the end, but can be used effectively by inserting it in between, which is where it truly shows its worth.
Highlights are emphasized!
The effect of Refiner also appears in unexpected places.
When using Refiner, the contrast increases, and the highlights in the eyes are emphasized.
Furthermore, by breaking the structure once, the position of the highlights has shifted slightly.
When generating illustrations, AI is good at accurately depicting subjects, but struggles with stylized or broken expressions.
Using Refiner can create subtle shifts, which leads to an expression of warmth similar to something drawn by human hands.
Let's compare a few
Now, let's look at a few illustrations. The top is the original illustration upscaled, and the bottom is with Refiner added.
Woman in a blue dress
Couple in autumn leaves
When you use the Refiner, the contrast is emphasized in a natural way, and the saturation also increases.
On the other hand, the expressions and softness of the characters remain intact.
When you use the Refiner, you can increase the texture while maintaining softness.
The Refiner is difficult
The Refiner is interesting, but how to use it is quite difficult.
The effectiveness of the Refiner varies completely depending on the base illustration, and it takes some trial and error to make it work effectively without completely destroying it.
It is good when taking time to complete a single illustration, but it is a difficult feature to use when generating illustrations with a good tempo.
Why was it implemented as a standard?
Why did SDXL implement such a difficult-to-handle feature as a standard?
This is my personal opinion, but I think that at the time SDXL was released, Midjourney was considered the standard for art, while Stable Diffusion was considered the standard for realistic expression.
Stability AI (the developer of Stable Diffusion) at the time was approaching development with an aggressive stance, and perhaps the Refiner was one of the answers to expressing art with the newly released Stable Diffusion XL.
The stance of daring to implement a feature that is not for general users as a standard makes me feel the aggressive posture of Stability AI at the time.
What I learned from the Refiner
Personally, I have learned a lot from the Refiner.
Using multiple models in combination
Once breaking down an illustration and regenerating it
This is simple in hindsight, but precisely because it is simple, it is an idea that does not come to mind at first.
The lessons I learned from the Refiner have become the origin of my current SDXL to Flux.1 workflow.
Although the Refiner did not become widespread, I believe it has had a significant influence on the aesthetics of subsequent generative AI, including Flux.1.
Summary
The Refiner enhances texture
The Refiner destroys structure
Reconstruct through high-resolution upscaling
Precisely because generative AI is in its infancy, new ideas are being born one after another.
This excitement is a privilege of the present moment, accessible only to those who engage with unfinished things.
Why not take this opportunity to try out AI for yourself?
Thank you for reading until the end!
Workflow
I have published a workflow on the AI platform service SeaArt AI where you can compare the effects of the Refiner.
Clicking the link will display the workflow, and you can download it as is.
If you register for a free account, you can generate illustrations on the spot, even if you don't usually use ComfyUI.
Click here for an article explaining how to register for SeaArt AI.
Model
anima_pencil-XL_v500
SDXL-v1.0_Refiner
2024.10.24
There was a report of an error, so I changed the Refiner link from Civitai to Hugging Face.
Creator Introduction
Although it is not directly related to this theme, I had the opportunity to see some wonderful works that utilize AI.
Miwa-san uses AI to visualize and introduce doujin works from the 1980s and 1990s.
In this video, it seems they pose the characters using figures with movable joints.
The smooth movement of the characters is excellent. Everyone, please take a look at it once.