Steps to use LCM LoRA in AUTOMATIC1111 Stable Diffusion web UI

Last update 11-26-2023

▼ 0. An easy way to use LCM LoRA

　Before diving into the main article, I will introduce the easiest way to experience LCM LoRA. Please see section 1 and beyond for standard usage.

0-1. Models with LCM LoRA merged

　Stable Diffusion models with LCM LoRA merged are now available, and you can use them to generate images immediately. Please note that you need to lower the Sampling steps and CFG Scale significantly (steps around 8, CFG 1-2). Also, do not include prompts that specify LCM LoRA.

　Note that the LCM Sampler is not mandatory; it seems better to use Euler a, DPM++ 2S a Karras, etc. If you want to use the LCM Sampler, please see section 3.

　You can find LCM models by filtering in the Civitai model list. Change the Model types selection to "Checkpoint" and the Base model selection to "SD 1.5 LCM" or "SDXL 1.0 LCM". There are few right now, but they are expected to increase in the future.

0-2. LCM LoRA merged models and generated images

　These are examples using models that exist at the time of writing.

EveryjourneyLCM (SD 1.5 LCM)
https://civitai.com/models/210006

blue_pencil-XL-LCM (SDXL 1.0 LCM)
https://civitai.com/models/202108

Reproduction_LCM (SDXL 1.0 LCM)
https://civitai.com/models/202308

▼ 1. About this article

1-1. Overview

One famous tool for generating images with Stable Diffusion format models is AUTOMATIC1111's Stable Diffusion web UI (hereinafter referred to as AUTOMATIC1111 web UI). I have written an article about the TensorRT extension, which can be used with this to generate images at high speed.

In this article, I will explain the procedure for using a different method called "LCM LoRA." Since it significantly reduces the number of steps during generation, it also features faster Hires. fix (i2i upscaling).

For information on LCM and LCM LoRA, please see the article below. Note that it also appears to be compatible with TensorRT.

▼ 2. Points to note about LCM LoRA

2-1. Notes on usage

While generation becomes faster with LCM LoRA, you may have to pay some costs.

• The generated images will be different. Since Sampling steps and CFG Scale are set quite low, changes in behavior are inevitable. Therefore, it is necessary to adjust prompts and settings based on the premise of using LCM LoRA.

• There is compatibility with models, and you may not get the desired results. This is likely because LCM LoRA is intended for base models. Try finding a compatible model or use a model that has LCM LoRA incorporated.

• Compatibility with Sampling methods is also intense. Using the LCM Sampler is safe, but it reduces the level of detail. Euler a and DPM++ 2S a Karras are relatively suitable.

▼ 3. Steps to use LCM LoRA

3-1. Overview

There are two steps: adding the LCM Sampler (though not strictly required) and specifying the LCM LoRA during generation. This article assumes the following conditions.

• You are already using the AUTOMATIC1111 web UI, or a new installation is complete and ready to use. If there is a possibility that it is not working correctly, consider updating.

• "torch: 2.0.1+cu***" is displayed at the bottom of the operation screen. Operation in cases other than this has not been verified, and it may require an update to the AUTOMATIC1111 web UI itself or PyTorch.
  (*Other than the LCM Sampler, it is just applying a LoRA, so it might be possible to use it with older versions, but this has not been verified.)

3-2. Adding LCM to Sampling method

It seems this is not strictly required. If you are interested, please try adding it. Upon checking, there were two methods. This article selects the former method. Note that both can be installed using almost the same procedure.

• Install AnimateDiff for Stable Diffusion WebUI (LCM Sampler will also be installed).
  https://github.com/continue-revolution/sd-webui-animatediff

• Install the standalone LCM Sampler (at the time of writing, it appears the name is "LCM Test").
  https://github.com/light-and-ray/sd-webui-lcm-sampler

　Once you have launched the AUTOMATIC1111 web UI, navigate to the "Extensions" tab, then "Install from URL", enter the official repository URL for AnimateDiff, and click "Install". If you have already installed it but cannot select the LCM Sampler, please update AnimateDiff.

　Once the installation is complete, a message will appear below the "Install" button. After that, go to the "Installed" tab, click "Apply and restart UI", and wait for it to restart. If the LCM Sampler is still missing, try restarting the AUTOMATIC1111 web UI itself.

3-3. Adding LCM LoRA

　As the name suggests, LCM LoRA is a LoRA, so it can be used just like any other LoRA. Please download it from the URL below and move it to your LoRA directory.

• For Stable Diffusion 1.5
  https://huggingface.co/latent-consistency/lcm-lora-sdv1-5

• For Segmind Stable Diffusion 1B (SSD-1B)
  https://huggingface.co/latent-consistency/lcm-lora-ssd-1b

• For Stable Diffusion XL
  https://huggingface.co/latent-consistency/lcm-lora-sdxl

　All of these have the same filename (pytorch_lora_weights.safetensors), so you need to be careful if you want to keep them together. The easiest solution is to rename the files.

　By the way, when using models and LoRAs together, the major version of Stable Diffusion (1, 2, XL) must match. I got confused as I started mixing these up, so I decided to separate the installation directories by version. I plan to introduce how to do this in another article. I introduced it in the article below.

Personal configuration notes for Stable Diffusion web UI (Forge)
https://note.com/mayu_hiraizumi/n/nd05a329ff982

3-4. Generating images

　Once you are ready, perform the generation. There are a few points to note in the settings. If there are sample images for the model, please refer to those as well.

• Specify the LCM LoRA in the prompt (not necessary if using a model with LCM LoRA merged). Write the LoRA filename at the end in the format ",<lora:pytorch_lora_weights:1>". The value 1 can be adjusted, and it might be worth trying lower values.

• Select one of the following for Sampling method: Euler a, DPM++ 2S a Karras, or LCM. LCM tends to produce simple images, so choose according to your preference. Upon checking, DPM++ 2M SDE Exponential, DPM++ 2M SDE Heun Exponential, DPM++ 3M SDE Exponential, and Restart also appear to be supported.

• Sampling steps should be around 4 to 10. At 4, the image may appear blurry. If there are no issues, you can fix it at 8.

• CFG Scale should be around 1 to 2. If there are no issues, you can fix it at 2. Note that while the slider is in 0.5 increments, you can actually specify finer values (a little trick).

• Use Hires. fix according to your preference. Since some models have output resolutions they struggle with, I basically use Hires. fix. There are setting examples in 0-2. and 4.

• Other extensions have not been verified. Since something might cause issues, it is desirable to start from a simple state or check the impact of each one.

▼ 4. LCM LoRA Sample Images

4-1. Overview

　I will post the generated sample images along with their settings. The launch options are as follows. If you specify opt-sdp-no-mem-attention and remove xformers, the reproducibility under the same settings will be higher (the same image will always be generated).

COMMANDLINE_ARGS=--opt-sdp-no-mem-attention --no-half-vae --medvram

4-2. Settings during generation

I used LittleStepMix_A and VAE distributed at the following URL.
https://huggingface.co/sazyou-roukaku/LittleStepMix

The settings below are for when Hires. fix is enabled.

pale pastel color anime,1girl, softly smile :d, upper body, yellow green pigtail hair, skyblue seaside beach, some palm-tree, gray eyes, pale pink ribbon dress, distant mountain, <lora:pytorch_lora_weights:1>
Negative prompt: (worst quality, low quality:1.2), backlit,
CFG scale: 2, Seed: 4184699122, Size: 768x384, Model hash: 1493009b1f, Model: LittleStepMIx_A, VAE hash: f6dbafc61e, VAE: sr_SDv2vae_kl-f8anime2.safetensors, Denoising strength: 0.55, Clip skip: 2, ENSD: 31337, Hires upscale: 2, Hires upscaler: R-ESRGAN 4x+ Anime6B, Lora hashes: "pytorch_lora_weights: aaebf6360f7d", Version: v1.6.0-2-g4afaaf8a

4-3. Without Hires. fix

　The output resolution is 768x384. If the Steps are too low, the output seems to be insufficient. Characteristics for each Sampler can also be seen.

4-4. Sampler: Fixed to Euler a

　From here on, the output resolution is 1536x768 (Upscale by 2) and Hires steps is 0. Differences due to Sampling steps and Hires sampler can be seen.

4-5. Sampler: Fixed to DPM++ 2S a Karras

　It has an atmosphere that seems to provide a good level of detail. You can choose between Euler a or this one based on your preference.

4-6. Sampler: Fixed to LCM

　It is cleaner than Euler a. You can choose this one based on your preference as well.

4-7. Conclusion

　Just to be sure, I tried changing the Sampler and Hires sampler, but I don't think it's necessary to go out of your way to change them (it might depend on the model?). Since the overall settings are largely a matter of preference, please try various things.

　If you like, please also try DPM++ 2M SDE Exponential, DPM++ 2M SDE Heun Exponential, DPM++ 3M SDE Exponential, and Restart. It is also a good idea to use an X/Y plot to check the output for each Sampler.

▼ 5. Bonus

The prompt and other details for the cover image are provided below. To generate the same image, the settings "torch: 2.0.1+cu118" and "--opt-sdp-no-mem-attention" are required.

upper body, child 2girl, rise arms, cute summer dress, happy smile, brown pigtail, meadow, distant town,
ultra detailed animation, hug together,
Negative prompt: (bad quality, low quality:1.2), teeth, large breast, braid hair,
Steps: 7, Sampler: Euler a, CFG scale: 1.5, Seed: 1830664579, Size: 1024x512, Model hash: 9d592e9646, Model: blue_pencil-XL-v1.0.0-lcm, Denoising strength: 0.55, Clip skip: 2, ENSD: 31337, Hires upscale: 1.5, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.6.0-2-g4afaaf8a

▼ 6. Other

Please navigate through the menu to find other articles I have written.

My note account is linked to my main account @Mayu_Hiraizumi, but please direct any inquiries regarding these articles to my sub-account @riddi0908.