[New Preprocessor] The "reference_adain" and "reference_adain+attn" are added

[lllyasviel]

V1.1.171 adds two new reference preprocessors:

reference_adain
AdaIn (Adaptive Instance Normalization) from
Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization
Xun Huang and Serge Belongie, Cornell University
https://arxiv.org/abs/1703.06868

reference_adain+attn
AdaIn (Adaptive Instance Normalization) + Attention link same as "reference_only"

Comparison

Input image (midjourney v5, https://twitter.com/kajikent/status/1654409097041817601)

meta

woman in street, masterpiece, best quality,
Negative prompt: lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry
Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 12345, Size: 768x512, Model hash: c0d1994c73, Model: realisticVisionV20_v20, Version: v1.2.0, ControlNet 0: "preprocessor: reference_????, model: None, weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (64, 0.5, 64)"

reference_only (style fidelity=0.5):

reference_adain (style fidelity=0.5):

reference_adain+attn (style fidelity=0.5):

without CN (all controlnets disabled):

You need at least 1.1.171 to use them.

[lllyasviel]

In our test:

1. "reference_adain+attn, Style Fidelity=1.0" is the current state-of-the-art best method but it is sometimes too strong and we do not recommend to use it as default.
2. we still recommend to use "reference_only + Style Fidelity=0.5" as default since it is very robust.

[kft334]

Can style fidelity be added to the API as threshold_a. Looking forward to fully using all of these new preprocessors.

[lllyasviel]

i think it is already - but not tested

[BahzBeih]

sorry for repeating request, but i don't know if you see my reply in previous thread, i hope you add option to reference feature, to use multiple photos as reference instead of one photo, i guess this will solve output images is not look alike the original person 🤞🤞🤞🤞🤞

[lllyasviel]

multi-controlnet is in setting, implemented several months ago. see also the readme

[BahzBeih]

yah i know about it, but i didn't get good results with it in this case, my request is like make it like lora training, by add ability to add multiple photos to the same controlnet reference with same person or style "Architecture style for example" in different angels and resolutions to make the final photo, and if possible produce a file like lora form this photos to be used with controlnet reference later, instead of adding the same multiple photos again every time, this will be revolutionary 🤞🤞🤞🤞🤞🤞🤞🤞🤞🤞🤞

[kdhckd]

it is not working at batch tool in img2img. Could you check it?

[lllyasviel]

hi all we fixed the multi-reference weight bug in 1.1.172

[extra-large-onions]

[lllyasviel]

prompt conflicts style - try "prompt is more important" or remove conflicting prompts
style fidelity conflicts base model - try fidelity = 0.5

[extra-large-onions]

I figured it out - the mugshot lora was causing the color defects - probably for anime only

[2blackbar]

so remove your tests if it was mistake on your part , they are confusing

[extra-large-onions]

I did. My bad I'm not familiar with github discussion culture.

[IndolentKheper]

Did something change with lineart_standard? I'm getting a weird hidden halo effect when using lineart_standard that doesn't appear with the same input using the other lineart preprocessors. The background warps to form an outline around the input image's lines.

Also, tons of artifacts especially on faces. Again, just with lineart_standard, doesn't appear when using other lineart preprocessors.

[lllyasviel]

check input image resolution - mistakenly used low-resolution or thumbnail image

[IndolentKheper]

Unfortunately I'm working with some old legacy files that are low resolution in this case, and upscaling is causing too many changes. :/
Controlnet is totally nailing the linework though, even with the low res inputs it's catching the lines perfectly, it's just the color that's being botched.

[lllyasviel]

then just use [invert] instead of lineart_standard

[IndolentKheper]

Invert's also giving a weird artifact pattern with low res inputs, actually. But I figured out a workaround. Since Controlnet was already getting the lines right, I used the low-res image in txt2img then prompted for black and white lineart and used LoRAs to encourage it to remake the image as just lineart. Eventually I was able to recreate the input at 512 a little better than upscaling, which was causing gaps.

[orfeaus]

The new feature is powerful and will have a lot of use cases. Thanks for this!

One thing I've noticed is that the images it produces are often slightly blurry compared to images generated without using the new feature. Is this a known issue?

[aleqniko]

Still, there is almost no face recognition,

[xyzzart]

Still, there is almost no face recognition,

the photorealistic results also depends of the checkpoint u use as they trained with limited dataset. i used to play around with face generations in artbreeder back in a days) that thing is still great

[ghost]

Hi so previously I made a bug report about an issue that consists of incorrect color tones that I assumed was only limited to "Controlnet is more important"

But my assumption seems to be wrong. It has been happening on balanced mode with style fidelity 0.5 and 1.0 too and in all 3 "reference" pre-processors

Here is the reference image:

Here is all reference pre-processors with Style fidelity 1.0 in Balanced mode

And here is all reference pre-processors with Style fidelity 0.5 in Balanced mode

As you can see, it seems to be collapsing even at 0.5 style fidelity and the color tone seems to be more dull too. What I also did notice is this does not happen much on realistic reference images but it happens a lot more on Anime art style images that u choose for reference images. So I am guessing that is where some issue lies in
Can someone give this a try by downloading the reference image I used above and see if the problem exists?

[lllyasviel]

it can happen when your base model refuse to draw colors as saturated as the example - you can disable cn and generate a few images with your base model to see what color range your base model supports

[PaperOrb]

Hi, I noticed this issue as well and my base model is very colorful/vibrant without cnet enabled. I'm on 1.1.189. Here's my generation info minus prompt since it's nsfw:

Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 4017221646, Size: 512x768, Model hash: f725cdf24b, Model: euclaseBlend_v12, Clip skip: 2, ENSD: 31337, ControlNet 0: "preprocessor: reference_only, model: None, weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (512, 1, 64)"

[Rkkss]

I did some testing with ref_adain+attn. Style Fidelity=1.0

Input image

first gen with only quality tags:

best quality, masterpiece, highres, (photorealistic:1.4)
Negative prompt: (worst quality:2), (low quality:2), (normal quality:2), lowres, ((monochrome)), ((grayscale)), (nsfw:1.2)
Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 123456789, Size: 512x680, Model hash: 5d22b29ecd, Model: majicmixRealistic_v2, Clip skip: 2, ENSD: 31337, Version: v1.2.1, ControlNet 0: "preprocessor: reference_adain+attn, model: None, weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (64, 1, 64)"

No surprise, he looks like a female because of model's bias

Now I add 1boy in prompt of the prompt: 1boy, best quality, masterpiece, highres, (photorealistic:1.4)

Close enough. Facial hair, black hair is there, I don't need to use more prompt.

This is the interesting part.
Cfg 3-5 seems to be closer to input image, cfg 7+ starts introducing weird patterns & artifacts (hiresfix can fix this but these images are still further away from input image)

Example that hires fix fixes some artifact at cfg9

---

So the conclusion is use fewer tags + low cfg? Need more testing to confirm this.

[orfeaus]

I also noticed lowering the CFG value resolves issues where certain models output very dark generations when using the new reference_only feature.

[ghost]

I also noticed lowering the CFG value resolves issues where certain models output very dark generations when using the new reference_only feature.

Could this be related to the problem I am having, scroll up a bit and you will see a post about a pink haired anime girl problem. The generations it produces make it very dark even after testing with multiple checkpoints. Checkpoints that can even do vibrant colors do this too. I noticed this because there are times where it would spit out a color correct generation once in a while so I am not 100% sure if its a model checkpoint problem or not. Although I was able to get it working with just one checkpoint model that has vibrant colors. But still its hard to believe something like Abyssmix wont be able to produce vibrant colors like the pink haired girl

[upbit]

Thank you for providing this awesome preprocessor! Through reference_adain+attn, it is possible to learn illustration composition very well, and can fine-tuning character through prompts.

Using Eunice Ye's Naraka character image as a reference, the generated images are compared (the left one is the original picture):

Prompt: (masterpiece, best quality:1.2), (extremely detailed), ultra high res, 8k, 1boy, male focus, solo, fingerless gloves, shoulder armor, looking at viewer, muscular, fighting stance, black hair, armor, pauldrons, > manly, pectorals, open clothes, facial hair, jacket, wind around
Negative prompt: EasyNegative, (worst quality, low quality:1.4), watermark, NG_DeepNegative_V1_75T, BadHandV4
ControlNet: reference_adain+attn, weight=1.0, mode=My prompts is more important

[zhuofengli]

Is there a way to use them with diffusers yet?

[XavierXiao]

Can someone explain how the method work? What is the attention mechanism of the reference-only?

[NormXU]

I draw a simple graph for your convenience.

The hidden features of a reference image within a self-attention block are initially stored in a list. These attributes are subsequently employed as image conditions to perform "cross-attention" within the same self-attention block.

From my understanding, we actually perform two rounds of diffusion in this pipeline.

Regarding the reference_latent, it is obtained through a linear combination of two elements: the noisy reference image encoded by the VAE encoder, and a randomly initialized latent feature for the standard diffusion process.

check the code here

[betterze]

Thx for the explanation. What is reference_adain and style fidelity?

[NormXU]

I wrote a blog post about reference-only mode. please check here

[PaperOrb]

I didn't see it mentioned so figured I would ask: does it improve anything if we use a txt2img reference image that's the same size as the txt2img output? example: txt2img outputs a 512x512 image and uses a 512x512 image for the reference

[2blackbar]

you want image cropped or stretched? i personally match aspect ratio or crop, also i dont use below 512res as ref

[PaperOrb]

Hmm I noticed that lllyasviel posted some reference photos that weren't the same size as the output. So you're saying you prefer matching the reference aspect ratio to the output aspect ratio, or you crop the reference so that it matches the output size exactly?

If I summarized that correctly, does that have visible differences? I'm not noticing anything on my end regardless of absolute size or aspect ratio, but I'm still unsure.

[orfeaus]

I just updated my PyTorch version to 2.0 on MacOS and images are collapsing again on some models when using reference_only / reference_adain / reference_adain+attn. This wasn't the case on the previous version of PyTorch.

I'm running Controlnet 1.1.189.

[PaperOrb]

Are you on the latest version of the webui?

[orfeaus]

Yea, updated to the latest (89f9faa) to see if it fixed the issues, but it didn't.

[Joeyd348039]

Any luck? Same problem here.

[H4dr1en]

Can the reference preprocessors be used in combination with controlnet models? I want to do img2img and preserve the semantic of the input image (depth, segmentation masks), and apply the style of a second image to the first image, would that be possible with the current state?

[dan4ik94]

yes, reference_only works best for me

[rbychn]

is there a way to get a bunch of MidJourney images and pass it through this without prompting but still get a great output?

want to do an article comparing the progress but won't be efficient to manually prompt for 50 images :(

[ghost]

Would it be possible to have a negative reference? I give ControlNet an example of how it is not supposed to look like - that would be a cool feature!

[usamaehsan]

when these will be available outside web ui?

[2blackbar]

When you use reference model and lineart model (two controlnets) then you can recreate reference image kinda almost perfectly

Steps: 32, Sampler: DPM++ 2M Karras, CFG scale: 9.5, Seed: 2228333885, Size: 768x768, Model hash: fc52756a74, Model: dreamlikePhotoreal20_, Denoising strength: 0.66, ControlNet 0: "preprocessor: reference_only, model: None, weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (64, 1, 64)", ControlNet 1: "preprocessor: lineart_realistic, model: control_v11p_sd15_lineart_fp16 [5c23b17d], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (512, 64, 64)"

[xyzzart]

When you use reference model and lineart model (two controlnets) then you can recreate reference image kinda almost perfectly

Steps: 32, Sampler: DPM++ 2M Karras, CFG scale: 9.5, Seed: 2228333885, Size: 768x768, Model hash: fc52756a74, Model: dreamlikePhotoreal20_, Denoising strength: 0.66, ControlNet 0: "preprocessor: reference_only, model: None, weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (64, 1, 64)", ControlNet 1: "preprocessor: lineart_realistic, model: control_v11p_sd15_lineart_fp16 [5c23b17d], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (512, 64, 64)"

looks great! would be even more cool if it possible to change composition keeping the referance face/ how is that possible to do?

[aaronkingdom]

Is there any resolution requirement for uploaded images?

I upload a picture with an indeterminate scale resolution, for example

712x1321 image as source, error will occur

RuntimeError: shape '[2, -1, 8, 40]' is invalid for input of size 24640
Time taken: 8.95sTorch active/reserved: 12648/12822 MiB, Sys VRAM: 8118/22528 MiB (36.04%)

[aaronkingdom]

One more mistake:

RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 1 but got size 2 for tensor number 1 in the list.

Strangely enough, when I reduced the number of prompt words, the errors disappeared,

It appears when I start controlnet reference_only and other renference_xx

[aaronkingdom]

Well, I tested it again, and it didn't work.

My cn version:
ControlNet v1.1.219

sd Startup parameters :--opt-channelslast --opt-sdp-no-mem-attention --api --device-id=1 --autolaunch --update-check --update-all-extensions

txt2img resolution Settings:
720x1280

cn upload resolution: 540x960

What's the problem? Or am I using it the wrong way? Can I get some help? Thank you

好吧,我又测试了一次,还是不行.
我的cn版本:
ControlNet v1.1.219

sd启动参数:--opt-channelslast --opt-sdp-no-mem-attention --api --device-id=1 --autolaunch --update-check --update-all-extensions
txt2img分辨率设置:
720x1280

cn upload resolution: 540x960

问题出现在哪里呢? 还是我用错方法了?? 有人帮忙一下吗,谢谢

[lllyasviel]

remove --opt-channelslast

[aaronkingdom]

hi, i remove, also the same error

RuntimeError: shape '[2, -1, 8, 40]' is invalid for input of size 24640

and this is my running environment: version:
v1.3.2  •  python: 3.10.9  •  torch: 2.0.1+cu118  •  xformers: N/A  •  gradio: 3.32.0  •  checkpoint: fc2511737a

when i change the args to : --device-id=1 --autolaunch
it show me another error:
RuntimeError: einsum(): subscript b has size 8 for operand 1 which does not broadcast with previously seen size 16
Time taken: 5.55sTorch active/reserved: 20079/20408 MiB, Sys VRAM: 10270/22528 MiB (45.59%)

but the other controlnet preprocessor working welll

just preprocessor reference_ not working :(

[aaronkingdom]

solve:
#1571 (comment)

[andupotorac]

Guys, I see you have preprocessor params. Where does one set these?

[gasvn]

Hi, I achieve a cross-image region drag based on the reference scheme. 1. Use inpaint controlnet to extract inpainted region feature from another image. 2. Use the segment-anything controlnet to keep reasonable pose.

Code in https://github.com/sail-sg/EditAnything

[chunyu-li]

Hey, can you give more mathematical details about how reference-only is implemented? I am very curious about it.

[betterze]

And what is reference_adain and style fidelity?

[tripathiarpan20]

I wonder if it's possible to use this method to restrict the Color scheme in the output, much like T2I-Adapter Color (but it's not available for SDXL)

[priteshakhaja02]

Greetings 👋,

I'm employing the Fooocus V2, Fooocus Photograph, Fooocus Negative model in replication to create realistic photos. However, despite providing four high-quality input photos, I'm encountering challenges in achieving accurate face matching in the AI-generated images, and the quality of the eyes is consistently subpar and very bad, not real like venom.

Any suggestions on how to address this issue?

[oldrena]

about reference_only，I can't reproduce it photos after May 16.Even with the same parameters.
For example, ControlNet 0: "Module: reference_only, Model: None, Weight: 1, Resize Mode: Resize and Fill, Low Vram: True, Threshold A: 0.8, Guidance Start: 0, Guidance End: 0.8, Pixel Perfect: True, Control Mode: ControlNet is more important, Hr Option: Both, Save Detected Map: True", Dynamic thresholding enabled: True, Mimic scale: 7.
It produces different pictures.
Since it is mechanically translated, it may not be the exact translation.

[oldrena]

Even if I go back to the version, I can't reproduce it

[BingLiHanShuang]

Hello, if we want to use reference_only with python, how could we do?