Steps to use StreamDiffusion on a Windows PC

Last update 1-6-2024
*(1-6-2024) Added information on the RealTimeScreen binary version. Added the missing pnpm installation command.
*(1-4-2024) Due to updates, I have added "Real-Time Img2Img Demo" to section 4 and "Update Procedure" to section 7-5. Also, the execution procedure for the Demo has been slightly changed.
*The title image shows two "optimal-performance" instances running simultaneously. For the one on the right, I added " --prompt "close-up, anime, cat, downtown street"".
*If you want to quickly check the results, please run the sample programs "screen", "optimal-performance", or the "Demo program".

▼ 1. About this article

1-1. Overview

StreamDiffusion is a pipeline aimed at high-speed, real-time image generation. It uses a dedicated program with accelerated processing that combines Stable Diffusion, SD-Turbo, TensorRT, and LCM-LoRA.

This article explains the steps to install StreamDiffusion and run the sample and demo programs. Next, I will briefly explain how to change settings. Furthermore, after introducing third-party programs, I will provide information such as uninstallation.

1-2. Official repository, paper, explanatory videos, etc.

StreamDiffusion
https://github.com/cumulo-autumn/StreamDiffusion
README
https://github.com/cumulo-autumn/StreamDiffusion/blob/main/README.md

StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation
https://arxiv.org/abs/2312.12491

[Real-time image generation] StreamDiffusion, the author explains how to use it [Over 100 frames per second]
https://www.youtube.com/watch?v=CnVbZv-6lGE

▼ 2. Installation

This article assumes that Python 3.10, CUDA Toolkit 11.8, and Git are already installed. Operation was verified on an RTX 3060 with 12GB of VRAM. If you have less VRAM than this, some or all programs may not work. 8GB or more is likely recommended.

The following steps are current as of the time of writing and are subject to change. Please also refer to the repository's README.

2-1. Installation 1

I am using "C:\aiwork" as the working directory, so please adjust accordingly. Open the command prompt and execute the following commands in order, ensuring the directory "C:\aiwork\StreamDiffusion" does not exist.

cd \aiwork

git clone https://github.com/cumulo-autumn/StreamDiffusion.git
cd StreamDiffusion

python -m venv venv
venv\Scripts\activate
（ここで行頭に (venv) が付いていることを確認する）

python -m pip install --upgrade pip

2-2. Installation 2

Please execute the following command. Large packages will be downloaded.

pip install torch==2.1.0 torchvision==0.16.0 xformers --extra-index-url https://download.pytorch.org/whl/cu118

2-3. Installation 3

Please execute one of the following commands. Note that the author of StreamDiffusion recommends the latest version.

（最新版をダウンロードする場合）
pip install git+https://github.com/cumulo-autumn/StreamDiffusion.git@main#egg=streamdiffusion[tensorrt]

（リリース版をダウンロードする場合）
pip install streamdiffusion[tensorrt]

2-4. Installation 4

Please execute the command below. A relatively large package will be downloaded.

python -m streamdiffusion.tools.install-tensorrt

2-5. Installation 5

Only for the release version, please execute the command below. Note that if it is already installed, it will display "Requirement already satisfied: pywin32...". There is no harm in executing it by mistake.

pip install pywin32

2-6. Installation 6 (Optional)

The following is for developers, so you do not need to execute it.

python setup.py develop easy_install streamdiffusion[tensorrt]
python -m streamdiffusion.tools.install-tensorrt

2-7. Closing the Command Prompt

Please close the command prompt you have been using until now. For the sake of explaining the procedure, I would like you to continue in a new window.

2-8. Renaming the directory

Since a cuDNN issue occurred in some programs, I will explain how to handle it. Please change the name of the "cudnn" directory located in "StreamDiffusion\venv\Lib\site-packages\nvidia\" to something like "cudnn.bak". This will change the location of the referenced cuDNN, and I believe the error dialog will no longer appear.

▼ 3. Sample Programs

Referring to the document below, we will try running various sample programs. I recommend "screen" and "optimal-performance". You do not need to run them in order, so please choose as you like. However, please do not forget to run 3-1. first.

StreamDiffusion Examples
https://github.com/cumulo-autumn/StreamDiffusion/tree/main/examples

Note that when using TensorRT, it will pause for a while with a display like the one below to create a dedicated model. If it has already been created, this will be skipped.

[I] Building engine with configuration:
    Flags                  | [FP16]
    Engine Capability      | EngineCapability.DEFAULT
    Memory Pools           | [WORKSPACE: 4239.00 MiB, TACTIC_DRAM: 12287.38 MiB]
    Tactic Sources         | []
    Profiling Verbosity    | ProfilingVerbosity.DETAILED
    Preview Features       | [FASTER_DYNAMIC_SHAPES_0805, DISABLE_EXTERNAL_TACTIC_SOURCES_FOR_CORE_0805]

Please remember that you can display usage instructions by adding " --help".

python screen/main.py --help

3-1. Preparation before execution

After opening a new command prompt, please execute the following commands in order first. This will switch to the environment built in the previous steps and move to the sample program directory.

cd \aiwork\StreamDiffusion
venv\Scripts\activate
（ここで行頭に (venv) が付いていることを確認する）

cd examples

3-2. Running screen

This program captures a specified area of the desktop to use as input and performs real-time generation using img2img. CPU load is high during execution, and high performance may be required.

For the first time only, execute the following commands in order. The necessary packages will be installed. Note that in the author's environment, installing Triton did not work. It seems to work without it, so if you choose not to install it, please omit the second line.

pip install -r screen/requirements.txt
pip install https://huggingface.co/r4ziel/xformers_pre_built/resolve/main/triton-2.0.0-cp310-cp310-win_amd64.whl

To run screen, please execute the following command.

python screen/main.py

A semi-transparent window will appear; move it to the area you wish to capture and press Enter. However, the author has observed an issue where the set coordinates may be misaligned.

The semi-transparent window will disappear, and an Image Viewer window will appear. The results of real-time generation using the specific area as input will be displayed in the Image Viewer. Try moving the window or playing a video.

When you want to exit, press Ctrl+C in the command prompt window. If you accidentally close the Image Viewer window, please close the command prompt window (this may take a little time).

If you want to use TensorRT, please execute it with the options as shown below. The first time, it will take a while to start because it creates a model specifically for TensorRT.

python screen/main.py --acceleration tensorrt

It seems to be about twice as fast compared to not using TensorRT.

3-3. Running the benchmark

This is a benchmark program for measuring performance. Please execute the following command. In addition to "single.py", there is "multi.py"; the former is for single-batch processing, and the latter appears to be multi-batch processing optimized for the RTX 4090. Usually, you should select the former.

python benchmark/single.py

After waiting a short while, a progress bar will be displayed, and when it reaches 100%, the results will be shown.

100%|████████████████████████████████████████████████████████████████████████████████| 100/100 [00:22<00:00,  4.50it/s]
Average time: 215.93773559570312ms
Average FPS: 4.630964556710387
Max FPS: 4.733658059656689
Min FPS: 4.200737513516701
Std: 0.09281142820303803

If you want to use TensorRT, please execute it with the options as shown below. The first time, it will take a while to start because it creates a model specifically for TensorRT.

python benchmark/single.py --acceleration tensorrt

When the progress bar reaches 100%, the results will be displayed, so please compare them with the case where TensorRT was not used.

100%|████████████████████████████████████████████████████████████████████████████████| 100/100 [00:13<00:00,  7.60it/s]
Average time: 124.47206359863281ms
Average FPS: 8.033931237972855
Max FPS: 8.220270537913702
Min FPS: 7.763662255577496
Std: 0.10069756838199544

3-4. Running optimal-performance

This is a program that uses SD-Turbo model and TensorRT to perform real-time txt2img generation and display it continuously. Please execute the following command. In addition to "single.py", there is "multi.py"; the former performs single-batch processing, and the latter performs multi-batch processing optimized for the RTX 4090. Usually, you should select the former.

python optimal-performance/single.py

The first time, it will take a while to start because it creates a model specifically for TensorRT.

Once preparation is complete, an Image Viewer window will appear, and a cat wearing a hat and sunglasses will continue to be generated.

You can change the prompt to alter the generated content. Below is one example.

python optimal-performance/single.py --p "close-up, anime, cat, downtown street"

To exit, press Ctrl+C in the command prompt window. If you accidentally close the Image Viewer window, please close the command prompt window (this may take a moment).

3-5. Executing img2img

Specify the image file or directory to perform generation via img2img and save it to a file. If you would like to try this, please refer to the following. Note that if you do not have any suitable images on hand, you can also try executing Txt2Img in the next section first.

In this article, the goal is to perform numerous generations and measure the execution time. First, I created a directory "C:\aiwork\i2i_in" containing only the input images, and an output directory "C:\aiwork\i2i_out".

Then, I generated 100 images for input. For reference, the time taken for generation was 9 minutes and 7 seconds. The parameters are listed in the image description.

First, let's try generating just one image using "single.py". The command is as follows. Please change the file path as appropriate.

python img2img/single.py --i C:/aiwork/i2i_in/test.png --o C:/aiwork/i2i_out/test1.png

Next, let's specify a prompt. I added " --p \"1girl, smile :d, brown pigtail hair, white frill dress, red eyes, farm rural village, spring flower, meadow, distant mountain\"" and changed the output filename.

python img2img/single.py --i C:/aiwork/i2i_in/test.png --o C:/aiwork/i2i_out/test2.png --p "1girl, smile :d, brown pigtail hair, white frill dress, red eyes, farm rural village, spring flower, meadow, distant mountain"

Next, let's try using TensorRT. I added " --acceleration tensorrt" and changed the output filename. The first time, it will take a while to start execution because it creates a model specifically for TensorRT.

python img2img/single.py --i C:/aiwork/i2i_in/test.png --o C:/aiwork/i2i_out/test3.png --p "1girl, smile :d, brown pigtail hair, white frill dress, red eyes, farm rural village, spring flower, meadow, distant mountain" --acceleration tensorrt

Up to this point, three images have been generated. It is hard to tell at a glance, but the output differs slightly depending on whether TensorRT is used or not.

Since it seems to be working correctly, next I will perform 100 consecutive generations. I changed "single.py" in the command to "multi.py" and updated the input/output locations to directories. Note that "--output-dir" for the output destination appears to be a mistake for "--output". In this article, I have standardized them to "--i" and "--o".

First is the case without using TensorRT. From the start to the end of the command execution, it took approximately 48.9 seconds. Also, the difference in the update time between the first and last file was approximately 32.82 seconds, and dividing by 99 resulted in approximately 0.33 seconds per image, and taking the reciprocal resulted in approximately 3.02 images per second (values excluding the preparation time after the program starts).

python img2img/multi.py --i C:/aiwork/i2i_in --o C:/aiwork/i2i_out --p "1girl, smile :d, brown pigtail hair, white frill dress, red eyes, farm rural village, spring flower, meadow, distant mountain"

Finally, here is the case using TensorRT. When the TensorRT-specific model has already been created, from the start to the end of the command execution, it took approximately 40.2 seconds. Also, the difference in the update time between the first and last file was approximately 22.66 seconds, and dividing by 99 resulted in approximately 0.23 seconds per image, and taking the reciprocal resulted in approximately 4.37 images per second (values excluding the preparation time after the program starts).

python img2img/multi.py --i C:/aiwork/i2i_in --o C:/aiwork/i2i_out --p "1girl, smile :d, brown pigtail hair, white frill dress, red eyes, farm rural village, spring flower, meadow, distant mountain" --acceleration tensorrt

Compared to the txt2img in the next section, it is possible that file input is becoming a bottleneck.

3-6. Executing txt2img

Specify a prompt to perform generation via txt2img and save it to a file. If you would like to try this, please refer to the following.

In this section, the goal is to perform a large number of generations and measure the execution time. I reused the output directory "C:\aiwork\i2i_out" (unplanned).

First, I will try generating just one image using "single.py". Please change the output file path as appropriate.

python txt2img/single.py --o C:/aiwork/i2i_out/test4.png --p "1girl, smile :d, brown pigtail hair, white frill dress, red eyes, downtown street"

Next, I will try using TensorRT. I added "--acceleration tensorrt" and changed the output filename. The first time it runs, it takes a while to start because it creates a model specifically for TensorRT.

python txt2img/single.py --o C:/aiwork/i2i_out/test5.png --p "1girl, smile :d, brown pigtail hair, white frill dress, red eyes, downtown street" --acceleration tensorrt

Since it seems to be working correctly, I will now use "multi.py" to generate 100 images. Note that this consumed about 25GB of VRAM during generation. Like the other sample programs, it is possible that it is optimized for the RTX 4090.

First is the case without using TensorRT. It took about 64.4 seconds from the start to the end of the command execution. Also, the difference in the modification time between the first and last files was about 10.01 seconds, and dividing by 99 resulted in about 0.10 seconds per image, and taking the reciprocal resulted in about 9.89 images per second (values excluding the preparation time after the program starts).

python txt2img/multi.py --f 100 --o C:/aiwork/i2i_out --p "1girl, smile :d, brown pigtail hair, white frill dress, red eyes, downtown street"

Finally, here is the case using TensorRT. In my environment, only in this case was the model created every time it was executed, and red error messages were also displayed. However, if you just wait, it will execute.

python txt2img/multi.py --f 100 --o C:/aiwork/i2i_out --p "1girl, smile :d, brown pigtail hair, white frill dress, red eyes, downtown street" --acceleration tensorrt

It took about 203 seconds from the start to the end of the command execution. Also, the difference in the modification time between the first and last files was about 10.11 seconds, and dividing by 99 resulted in about 0.10 seconds per image, and taking the reciprocal resulted in about 9.79 images per second (values excluding the preparation time after the program starts).

The reason why the speed did not increase even when using TensorRT is likely due to a lack of VRAM. Whether using TensorRT or not, the state during generation became like the image below.

Note that simply lowering the "--f" value does not seem to be the solution. In my environment, I have confirmed a phenomenon where it stops progressing after displaying "[W] UNSUPPORTED_STATE". In this state, you have no choice but to close the command prompt.

[I] Building engine with configuration:
    Flags                  | [FP16]
    Engine Capability      | EngineCapability.DEFAULT
    Memory Pools           | [WORKSPACE: 3305.00 MiB, TACTIC_DRAM: 12287.38 MiB]
    Tactic Sources         | []
    Profiling Verbosity    | ProfilingVerbosity.DETAILED
    Preview Features       | [FASTER_DYNAMIC_SHAPES_0805, DISABLE_EXTERNAL_TACTIC_SOURCES_FOR_CORE_0805]
[W] UNSUPPORTED_STATESkipping tactic 0 due to insufficient memory on requested size of 11056491520 detected for tactic 0x0000000000000000.Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().

3-7. vid2vid execution

Specify a video to perform generation using vid2vid and save it to a file. If you would like to try it, please refer to the following to execute it.

Only for the first time, please execute the following command. The necessary packages will be installed.

pip install -r vid2vid/requirements.txt

Please prepare a video file for conversion. You may use the file below. Please rest assured that it was filmed and processed by me. You are free to repost it at your own risk, but it would be helpful if you could introduce this article.

To execute vid2vid, please run the following command. Please change the file path and prompt as appropriate. Note that if you want to include a comma in the prompt for vid2vid, it seems you need to add a "\" (backslash) immediately before it (it will result in an error if you don't).

python vid2vid/main.py --i C:/aiwork/riddi0908-20231229a.mp4 --o C:/aiwork/output.mp4 --p "sheep standing in a meadow"

The entire PC may become sluggish during execution. Once execution is complete, please check the generated video.

▼ 4. Demo program

　We will try running the Demo program by referring to the documentation below.

Real-Time Txt2Img Demo
https://github.com/cumulo-autumn/StreamDiffusion/tree/main/demo/realtime-txt2img

Real-Time Img2Img Demo
https://github.com/cumulo-autumn/StreamDiffusion/tree/main/demo/realtime-img2img

4-1. Installing Node.js (First time only)

　The Demo program requires Node.js. Please access https://nodejs.org/ and download it using the "LTS" button. You can install it simply by running the downloaded file and following the prompts. There is no need to check any boxes during the process.

　There are two programs, so please proceed as follows.

• Real-Time Txt2Img Demo → Go to 4-2.

• Real-Time Img2Img Demo → Go to 4-5.

4-2. (Txt2Img) Preparation before execution

　After opening a new command prompt, first execute the following commands in order. This will switch to the environment built in the previous steps and move to the Demo program directory.

cd \aiwork\StreamDiffusion
venv\Scripts\activate
（ここで行頭に (venv) が付いていることを確認する）

cd demo\realtime-txt2img

4-3. (Txt2Img) Setup (First time/When updating)

　If this is your first time or if you have updated StreamDiffusion, please execute the following commands in order.

npm install -g pnpm
pip install -r requirements.txt
cd frontend
pnpm i
pnpm run build
cd ..\

4-4. (Txt2Img) Execution

　Please execute the following command.

python main.py

　When a display like the one below appears, access http://127.0.0.1:9090/ in your web browser. 9090 is the port number, so if it is a different value, please change it accordingly.

INFO:     Uvicorn running on http://127.0.0.1:9090 (Press CTRL+C to quit)

　When you enter a prompt in the input field at the bottom, images will be generated in real-time. It will update when you rewrite the prompt. It is hard to express in words, so please try it out for yourself.

　When you want to finish, press Ctrl+C in the command prompt window.

4-5. (Img2Img) Preparation before execution

　After opening a new command prompt, first execute the following commands in order. This will switch to the environment built in the previous steps and move to the Demo program directory.

cd \aiwork\StreamDiffusion
venv\Scripts\activate
（ここで行頭に (venv) が付いていることを確認する）

cd demo\realtime-img2img

4-6. (Img2Img) Setup (Initial/Update)

If this is your first time or you have updated StreamDiffusion, please execute the following commands in order.

pip install -r requirements.txt
cd frontend
npm i
npm run build
cd ..\

4-4. (Img2Img) Execution

Please execute the following command. The first time you run it, it will take a while to start as it creates a TensorRT-specific model.

python main.py  --acceleration tensorrt

When a display like the one below appears, access http://127.0.0.1:7860/ in your web browser. 7860 is the port number, so please change it accordingly if the number is different.

INFO:     Uvicorn running on http://0.0.0.0:7860 (Press CTRL+C to quit)

Click "Start" to begin. Changing the prompt in the input field at the bottom seems to reflect in real-time. Since I do not have a webcam in my environment, I was unable to verify its operation.

When you want to exit, press Ctrl+C in the command prompt window.

▼ 5. Changing Sample Program Settings

The sample program comes with default settings, so it is designed to work right away without any configuration. This section introduces an example of changing some of the settings.

Please note that the configurable items vary depending on the program. You can check the details by running it with only " --help" appended.

python screen/main.py --help

5-1. Changing the Prompt

The prompt is a string that specifies the content of the image to be generated. Providing a suitable prompt can change the content or improve the output.

For example, if you look at the source for screen, you will find the following content. The reason it tends to output images with glasses is that "thick glasses" is included in the default prompt.

prompt: str = "1girl with brown dog hair, thick glasses, smiling",
negative_prompt: str = "low quality, bad quality, blurry, low resolution",

As a test, let's run screen with the following content. If you are using TensorRT, please add " --acceleration tensorrt".

python screen/main.py --p "1girl with red eyes, open mouth"

The eyes turned red and the mouth opened. Please note that since this is img2img, it does not always result in the expected changes.

Let's also change the prompt for optimal-performance again. However, since this one specifically uses a special model (SD-Turbo), it is better not to expect much in terms of style changes.

python optimal-performance/single.py --p "flat anime style, face close-up, little 1girl, smile, dress, castle"

5-2. Changing the Model

Programs other than optimal-performance are set to use the model "KBlueLeaf/kohaku-v2.1" by default. Let's try changing this to an arbitrary Stable Diffusion 1.5 model. Note that LoRA can also be applied.

To change the model, specify either the Diffusers ID or the file path for a safetensors format file. Also, if you want to use TensorRT, the initial conversion takes time, so please run it without adding " --acceleration tensorrt" first to verify that it works.

Here, we will use "ToraFurryMix v2.0" as the model and specify prompts for cat ears, a muzzle, etc. Please change the model file path as appropriate.

python screen/main.py --p "furry, kitten girl, cat ears, muzzle" --m "C:/aiwork/sd-data/sdv1/models/ToraFurryMix_v20.safetensors"

Please note that since this is img2img, it may not always result in the changes you expect.

Please refer to the article below for Stable Diffusion 1.5 models recommended by the author. These do not contain, or are unlikely to contain, components from NAI leaked models.

Actually, it is also possible to change the model for optimal-performance. However, since no other SD-Turbo models exist, I do not think you will get good output regardless of which model you use.

▼ 6. Utilizing StreamDiffusion

Since the StreamDiffusion package only provides sample-like programs, you have to modify them or create your own to do anything more. Initiatives using StreamDiffusion are already underway, and some of the code has been made public. It is for intermediate users and above, but it might be a good challenge to try.

6-1. Explanation of basic code and customization

The article below is helpful for further mastering StreamDiffusion.

6-2. Converting camera footage

The article below is an attempt to convert webcam footage in real time.

6-3. RealTimeScreen

A program has been released that performs real-time generation while capturing a specified area, which can be used for drawing assistance, etc. Since there is a binary version, it is relatively easy to install.

▼ 7. Notes

These are the procedures for deleting data, uninstalling, and updating.

7-1. Deleting pip cache

Packages installed via the pip command are saved on your PC. While this is convenient as it saves the trouble of downloading them again when reused, some users may be concerned about the size, as PyTorch, for example, is over 2GB.

To delete the package cache, execute the "pip cache purge" command or manually delete the contents of "%LOCALAPPDATA%\pip\Cache". Note that they will be downloaded again if needed.

7-2. Deleting the Diffusers cache

Data including Diffusers-format models downloaded from HuggingFace during program execution is saved on your PC. These will remain unless you delete them, so it is a good idea to organize them as needed. Note that they will be downloaded again if needed.

The storage location is "%USERPROFILE%\.cache\huggingface\hub". The "models..." directory shown in the image below is likely related to StreamDiffusion, so it is recommended to leave it as is while in use.

7-3. Deleting TensorRT format models

TensorRT-specific models used in StreamDiffusion are also saved on your PC. They are located in "StreamDiffusion\examples\engines", so please decide whether to keep them by checking their names, etc.

7-4. Deleting StreamDiffusion

If you wish to delete StreamDiffusion, please move any necessary files before deleting the "StreamDiffusion" directory. Also, delete the caches mentioned in the previous sections if they are no longer needed.

7-5. Update procedure

Although unconfirmed, you can update StreamDiffusion using the following steps. Open the Command Prompt and execute the following commands in order. Please adjust the first line as appropriate.

cd \aiwork\StreamDiffusion

git pull

venv\Scripts\activate
（ここで行頭に (venv) が付いていることを確認する）

Please execute one of the following commands.

（最新版をダウンロードする場合）
pip install git+https://github.com/cumulo-autumn/StreamDiffusion.git@main#egg=streamdiffusion[tensorrt]

（リリース版をダウンロードする場合）
pip install streamdiffusion[tensorrt]

Please execute the following command. It may not be necessary.

python -m streamdiffusion.tools.install-tensorrt

For developers, please also execute 2-6.

If you are running the Demo program, please also re-execute the setup steps.

▼ 8. Other

Please follow the menu to find other articles I have written.

　My note account is linked to my main @Mayu_Hiraizumi account, but please direct any inquiries regarding my articles to my sub-account @riddi0908.