Before replacing your GPU... The final resort for Flux1 acceleration: "novram + --fast"
Well, in an AI environment where dramatic, one might even say "violent," evolution never stops, I find myself feeling the limitations of my current nVidia RTX 4070 12GB more and more every day.
Especially with Flux1, the speed gap between 12GB and 16GB of VRAM is significant, and I am seriously considering upgrading to an RTX 4070 Ti Super 16GB. However, it is my principle to do everything I can software-wise before resorting to that.
I have pursued every possible measure I could think of, assuming there isn't some dramatic new support in CUDA or CuDNN.
Even when it comes to Pytorch settings, I doubt many people have gone as far as I have in the following steps. (Though, setting aside Pytorch 2.3.1 or earlier, it is debatable how effective these steps are for Pytorch 2.4.0+cu12.4.)
Anyway, let's get straight to the point.
With this measure, provided you use the Sampler made by ControlAltAI (a group I am a support member of), I have succeeded in reducing VRAM consumption during sampler processing to the following level.
And this is while using the heaviest model, flux1-dev.sft (now safetensors), and applying Xlabs' ControlNet.
The generation time takes 150 seconds, but since the VRAM consumption would exceed 12GB during sampler processing otherwise, making it impossible to finish, this is a dramatic speedup.
I haven't done anything particularly difficult.
I simply changed the startup options as shown below.
.\python_embeded\python.exe -s ComfyUI\main.py --novram --windows-standalone-build --fastI never thought I would end up having to use "no vram" with an RTX 4070 12GB... but for Flux1, one has to consider the RTX 4070 12GB as barely above the lower limit of low-end hardware.
Note that this trick works, as far as I have confirmed at this moment, with the Flux Sampler by ControlAltAI and the standard KSampler. With Xlabs' Sampler, even using this trick, flux1-dev.sft still results in an OutOfMemory error and fails to complete.
According to ControlAltAI, the Xlabs nodes seem to have a different processing logic from the legacy ones (which is likely the cause of the exclusivity of Xlabs nodes), and that is probably one of the reasons why they crash even with novram.
Also, even with the ControlAltAI Flux Sampler or KSampler, they may crash depending on how the nodes are connected.
Regarding --fast, some information appeared the other day, and its effectiveness on the RTX 4090 seems to have been confirmed. It appears to have an effect on each card in the RTX 4000 series and above.
By the way, for quantized models like GGUF or NF4, you don't necessarily need to use novram; the lowvram setting keeps sampler processing under 12GB even with ControlNet applied (excluding IP-Adapter), so there is no necessity to use novram.
In fact, there are cases where nodes that completed using shared VRAM with lowvram will crash with an OutOfMemory error when using novram.
There were even situations where it was better to launch with lowvram, use shared VRAM, and take advantage of schnell's 4 steps.
I recommend creating multiple batch files to switch between startup options. I have created three types.
As a final aside, I have modified the workflow that used the Xlabs Sampler, which has issues with exclusivity, to apply the legacy LoRA Loader.
A further aside.
Since there was an error in how the IP-Adapter node I released the other day was set up, I have corrected it and also added a LoRA Loader.