[AI Image Generation] Flux.1 runs comfortably on 6GB VRAM! --novram recommendation

Introduction

Hello, this is Easygoing.

I have been writing articles about AI image generation, and recently my posts have mainly been about Flux.1.

Flux.1 is an AI model capable of high-quality image generation, but its high VRAM usage is a drawback.

Today, I will consider ways to run Flux.1 comfortably even on low-spec PCs.

Conclusion: Use the --novram option

First, the conclusion.Flux.1 runs comfortably on 6GB of VRAM when using the --novram option!

Verification Environment

This verification was conducted in the following environment.

• ComfyUI

• System RAM: 32GB

• GPU: RTX 4060 Ti (VRAM 16GB)

I used a workflow that generates high-quality illustrations through image generation followed by upscaling.

I will measure the VRAM usage for each process and create a graph.

Actual Results

First, I will check the VRAM usage in the simple original FP16 format.

• T5xxl-fp16 9.6 GB

• blue_pencil-flux1_v001-fp16 33GB

The first generation is the blue graph, and the second is the green graph.

Since VRAM usage reaches 15 GB in all cases, generation time increases dramatically in environments with 12GB of VRAM or less.

Does lowering the resolution not save VRAM?

I will try lowering the image resolution to find a way to generate images with less VRAM.

When I set the resolution to 128 x 128 (1/64th of the original size), the image generation time became shorter, but the VRAM usage remained almost the same as a result.

It turns out that lowering the resolution is not very effective for saving VRAM.

I will look for other ways to reduce VRAM usage.

--lowvram and --novram options

ComfyUI has two options to make VRAM usage more efficient when VRAM is low.

--lowvram

--lowvram is an option that moves part of the model to system RAM to reduce VRAM usage.

You can save VRAM, but the overall processing speed becomes slower.

--novram

--novram saves VRAM even more aggressively.

While VRAM usage becomes extremely low, the processing speed becomes even slower.

--lowvram is almost ineffective!

First, let's look at the VRAM usage when using the --lowvram option.

• T5xxl-fp16 9.6 GB

• blue_pencil-flux1_v001-fp16 33GB

With the --lowvram setting, VRAM usage hardly changed.

--lowvram was somewhat effective at saving VRAM when I tried it with SDXL, but perhaps because Flux.1 was released recently, it does not seem to be sufficiently optimized.

--novram is quite effective!

Looking at --novram on the same graph,it significantly reduces VRAM usage, keeping total consumption almost within 6GB.

VRAM usage only exceeds 6GB during the upscale processing stage, but this process is short, and if a resolution of 1440 x 1440 is sufficient, the upscale process itself is unnecessary.

On the other hand, since --novram moves VRAM data to system RAM for processing,it uses the full 32GB of system RAM.

If system RAM is used more efficiently, it seems possible to generate images a bit faster.

Change checkpoint to gguf format

To save system RAM, I will try changing the checkpoint to be loaded to the smaller gguf format.

• T5xxl-fp16 9.6 GB

• blue_pencil-flux1_v001-Q_8.gguf 12.5 GB (-20GB)

By reducing the model size, system RAM usage stays within 32GB, and image generation became 2 minutes faster.

The generation time is not much different from when VRAM is 16GB, and there is almost no degradation in image quality.

The gguf format is extremely effective for saving RAM.

Try reducing system RAM to 16GB

Next, I will try under conditions with less system RAM.

Keeping the --novram setting, I will remove one stick of main memory to reduce system RAM to 16GB.

RAM usage is pegged at the maximum of 16GB, and the generation time has increased by 2 minutes due to memory shortage.

As before, this seems like it could be faster if the model size is reduced.

Further lightweighting the model

To reduce system RAM usage, I will try making the model even lighter.

T5xxl-Q_5_K_M.gguf 3.4GB (-6.2 GB)
blue_pencil-flux1_v001-Q_4_K_M.gguf 6.8GB (-5.7 GB)

When the model is lightened, the image generation time becomes the same as the original.

However, when lightened to Q_4, the generated images have less detail.

In an environment with 16GB of system RAM, I think it is best to use Q_4 for daily use and Q_8 when you want to improve image quality.

Summary of verification results

I will summarize the results of this verification in a table.

Flux.1 can be used sufficiently with 6GB of VRAM if you use --novram and the gguf format.

Images can be generated with 16GB of system RAM, but 32GB is desirable to balance image quality and speed.

What I found in this verification is that while --novram has significant benefits, there are almost no downsides.

If you use --novram to appropriately adjust the model capacity, you can maintain almost the original speed.

What are the recommended settings?

Based on the verification results, I will introduce recommended settings.

VRAM 16GB or more

• T5xxl-fp16

• Checkpoint fp16

VRAM 6GB to 12GB

• --novram setting

• T5xxl-fp16

• Checkpoint Q_8.gguf

System RAM 16GB or less

• T5xxl-Q_5_K_M.gguf

• Checkpoint Q_4.gguf

How to set up --novram

This is how to set up --novram in Stability Matrix, which I use.

Launch Stability Matrix, click the settings button for ComfyUI, and scroll down to find the --novram checkbox.

Just check this box and launch ComfyUI, and you're good to go.

Points to note

This verification was strictly for measuring VRAM usage; I did not physically swap out GPUs to perform the measurements.

When I tried it on my own GTX 1660Ti 6GB, the VRAM usage was higher even with --novram enabled, so I believe there is significant optimization depending on the GPU generation.

However, there is no downside to trying --novram, so if your VRAM usage is exceeding limits, it is definitely worth trying first.

Is there no --novram setting in Forge?

Stable Diffusion webUI Forge automates VRAM management, so you cannot manually set --novram.

Instead, Forge has the --always-offload-from-vram option to reduce VRAM usage, but in my environment, using this did not change the VRAM usage.

For now, ComfyUI is more advantageous for low-spec PCs because it allows for more granular VRAM settings, but Forge performs quite advanced VRAM management, so it is possible that VRAM operation will improve in future updates.

Isn't it difficult to install ComfyUI?

That is not the case.

On 2024.9.14, Zuntan released a tool to install ComfyUI with one click!

(To be precise, it is a tool that installs both Forge and ComfyUI with one click.)

From now on, if you are going to use ComfyUI, I recommend installing this EasyForge + ComfyUI.

I will introduce this in detail in the next article.

Summary

• Flux.1 runs comfortably on 16GB VRAM

• It runs on 6GB VRAM with the --novram setting

• System RAM should preferably be 32GB

Thank you for reading until the end!

Reference Articles

Comparison of image quality in FP16 / Q_8 / Q_4 formats

I learned that --novram is effective in ComfyUI this time thanks to this article by GJL.

Thank you for the very helpful information!

English Page