How to convert Flux fp16 to GGUF by flux tools

I follow generative AI-related information every day, but naturally, there is a limit to what one person can do. There is a lot of information that I miss.

The following article provided me with extremely useful information at this moment. I am grateful for the description of such very important information.

First of all, I didn't even know about a tool called EasyForge,

The important thing here is that this author, separately from the main Forge body,provides a tool to convert Flux1 fp16 format files to GGUF formatis the point.

Actually, I had been trying to perform the same conversion using Llama.cpp, but it didn't go well.

Or rather, for safetensors files for image generation, I would seriously like someone who has succeeded with this to teach me how they did it.

In the first place, this was originally developed for language generation AI uploaded to HuggingFace, and even if you download the repository for image generation AI via git clone, it cannot be loaded.

With the methods for language generation LLMs currently circulating, a file containing model information called config.json is required, but image generation AI models do not have this.
(By the way, I succeeded in quantizing language generation LLMs normally.)

For those who have quantized image generation AI models using Llama.cpp, I seriously want to know how you are doing it.

…I was facing this challenge, but the "flux tools" included in the EasyForge package solved this problem easily.

As shown in the figure below, I succeeded in creating the Q8_0 format on my own.

Since I have already built the environment for the main Forge body, I do not need all of EasyForge. After executing the installation batch file, I delete everything except the following two folders.

Open the flux_tool folder and execute the "ConvertGguf" batch file. Then, a screen like the one below will appear.

All you have to do is follow the messages.

Note that the initial settings are configured to create Q6_K and Q8_0 formats as shown below. By changing the descriptions of all related files, it is possible to perform quantization with other different settings.

@echo off
chcp 65001 > NUL

set PS_CMD=PowerShell -Version 5.1 -NoProfile -ExecutionPolicy Bypass
set "LANG_TAG="
for /f "tokens=*" %%i in ('%PS_CMD% -c "(Get-WinUserLanguageList).LanguageTag"') do set "LANG_TAG=%%i"

set "CONVERT_GGUF_MSG=Convert Flux model (fp16, 22GB~ recommended) to GGUF."
if "%LANG_TAG%" == "ja" (
	set "CONVERT_GGUF_MSG=Flux モデル（fp16, 22GB~ 推奨）を GGUF に変換します。"
)
echo %CONVERT_GGUF_MSG%

set "SAFETENSOR_MODEL=%~1"
if "%SAFETENSOR_MODEL%" == "" ( call "%~dp0env\GetFluxModel.bat" )
if "%SAFETENSOR_MODEL%" == "" ( set "SAFETENSOR_MODEL=%FLUX_MODEL%" )
if not exist "%SAFETENSOR_MODEL%" (
	echo [Error] Model not found: %SAFETENSOR_MODEL%
	pause & exit /b 1
)
echo SAFETENSOR_MODEL: %SAFETENSOR_MODEL%

if not exist "%~dp0env\ConvertGgufFormat.txt" ( echo Q8_0 Q6_K >"%~dp0env\ConvertGgufFormat.txt")
set /p CONVERT_GGUF_FORMAT=<"%~dp0env\ConvertGgufFormat.txt"

call "%~dp0env\ConvertGguf.bat" "%SAFETENSOR_MODEL%" %CONVERT_GGUF_FORMAT%
if %ERRORLEVEL% neq 0 ( exit /b 1 )

The image was also generated properly.

This has opened a path to easily quantize original merged Flux1 models that are uploaded to Civitai and HuggingFace every day. I am grateful for this.

Finally, a side note.

This "EasyForge" has an extremely unique way of building virtual environments. It's interesting. I was impressed that such a method existed.

It is not the venv format of A1111, nor is it the Portable format of Forge or ComfyUI.

At first glance, it looks like an A1111-style venv format, but it is actually different. This is because Forge launches using Python 3.10.6 in my environment where Python 3.11.9 is installed.

If that were all, the Portable formats for Forge and ComfyUI would be doing the same thing, but EasyForge achieves "this" using the venv format.

I carefully examined the folder structure and, for the most part, understood the general logic.

I was impressed, thinking, "I see, there is such a method..." Although this is not the time to dive deep into it, so I will move on, but I would like to investigate this EasyForge virtual environment more thoroughly at a later date.