[Investigation] Smile Test: Investigating the Elysium_Anime_V3 Issue #3
I have been investigating the "first token ignored" issue over the last two posts, and I believe I have identified the corrupted section and the method for fixing it.
Furthermore, I have summarized the verification of the fix's effectiveness and its potential side effects.
Quick Summary
・Identified the corrupted section within the model regarding the "first token ignored" issue
・Confirmed multiple methods for fixing it (see within the article)
・Summarized the impact of applying the fix
・Added a check extension for the model (06/02)
Corrupted section discovered?
Arena (the author of sd-tagging-helper) has recently been investigating internal model structures, and I asked them about it after being intrigued by the examples they provided regarding corruption and damage to internal model data.
cond_stage_model.transformer.text_model.embeddings.position_idsThe content of this key has changed to torch.float32, but it should be torch.int64. When an Add Difference is performed, it is not a total multiplier of 1, so this value, which should be an integer, deviates from being an integer. Furthermore,
It is not an exact quote.
Example: Values in Elysium_Anime_V3
Elysium_Anime_V3.safetensors
tensor([[ 0.0000, 1.0000, 2.0000, 2.9995, 3.9999, 4.9985, 5.9990, 6.9994,
7.9999, 9.0004, 9.9971, 11.0013, 11.9979, 13.0021, 13.9988, 15.0031,
15.9998, 16.9965, 18.0007, 19.0049, 19.9942, 20.9984, 22.0025, 23.0068,
23.9958, 25.0000, 26.0042, 26.9935, 27.9976, 29.0019, 30.0061, 30.9954,
31.9996, 33.0037, 33.9930, 35.0121, 36.0015, 36.9907, 38.0098, 38.9991,
39.9884, 41.0074, 41.9967, 42.9861, 44.0051, 44.9944, 46.0137, 47.0028,
47.9917, 49.0107, 50.0000, 50.9897, 52.0084, 52.9983, 53.9869, 55.0066,
55.9953, 56.9852, 58.0039, 58.9935, 60.0122, 61.0019, 61.9908, 63.0102,
63.9991, 64.9880, 66.0075, 67.0262, 67.9860, 69.0047, 70.0242, 70.9832,
72.0030, 73.0214, 73.9815, 75.0000, 76.0197]])This deviated float value is truncated at the decimal point when the model is loaded because it is cast to int64, resulting in a different value. It is thought that when converting to fp16, rounding occurs, which (in some cases?) fixes it.
The way to fix this phenomenon is very similar to what I have been investigating in the previous two posts.
This also explains the mechanism behind why "converting to fp16 fixes it."
How to check (Added 06/02)
I had released this as an extension but hadn't provided instructions, so I am adding them here. There is no fix function included, so please refer to the article below.
Fixing Method
Based on this discovery, I have fixed 8528d-fix and Elysium_Anime_V3.
Specifically, I forcibly re-initialized the tensor within the key mentioned above.
As a result, it appears to be correctly fixed.
The prompt given was "smile sleepy girl"
In the top version, the first token "smile" was ignored, but in the keychange version (bottom), a smile can be seen
Fixing Method 1: Arena's Extension
※ A CLIP-fix option has been added to the extension (※2) released by Arena. If you want to process it from the Web UI, you can use that as well. (I haven't tried it myself, so please use it after verifying it.)
(※2) arenatemp/stable-diffusion-webui-model-toolkit
Fixing Method 2: Fixing Script
As an alternative method, I have provided a fix script.
This is a CLI tool, not an extension.
・First edition (4.19 kB): Deleted because it lacked support for NovelAI-specific keys・Fixed version @ 01/15. (4.16 kB)
How to use the fix script
This is for CLI. Please use it from the command line.
It is built with argparse, so please first check the commands using `python fix_postion_ids.py -h`.
--model <Full path to the model to load (required)>
If only the model is specified, it will only perform the check.
--verbose
This will increase the amount of output.
--out <Full path for the fixed model output, including model name>
If you enter an output filename, it will fix the model and save it. If the model name ends in .safetensors, it will be saved as a safetensors file.Command example
python fix_position_ids.py --model E:\tool\sd\model_sd\need_fix\Elysium_Anime_V3.safetensors --verboseFix Method 3: Use "Skip/Reset CLIP key" in Merge Block Weighted
"Merge Block Weighted" Extension
https://github.com/bbc-mc/sdweb-merge-block-weighted-gui
None: Do nothing (same calculation method as before)
Skip: Skip calculation for the target key
Force Reset: Rewrite to integers, same as the fix script
Investigation 1: Status of target keys within the model
Although the explanation is out of order, I will first check the status of each model.
For each model, I investigated and compared the status of the target key (torch.float32) and what happens when it is cast using .to(torch.int64).
I also added output for anomaly detection to check where the discrepancies or issues are.
How to read the output results
The output results display four tensors.
From the top:
・1. Current tensor: Data as loaded from the model
・2. Tensor converted to torch.int64: What happens when type-converted to torch.int64
・3. Proposed fix content(constant)
・4. Judgment result: Whether there is a difference between the current data and the proposed fix (True means no difference, False means there is a difference)
Furthermore, if anomalies are found, the following two pieces of information are displayed.
・corrupt: A list of which token number is "shifted"
・missing: A list of numbers that are missing from the data as a result of the shift
stable diffusion 1.5
loading ... sd-v1-5-pruned.ckpt
# current data is:
tensor([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
72, 73, 74, 75, 76]])
<class 'torch.Tensor'>
torch.int64
== if changed to torch.int64 ==
<class 'torch.Tensor'>
tensor([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
72, 73, 74, 75, 76]])
torch.int64
# change to:
tensor([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
72, 73, 74, 75, 76]])
<class 'torch.Tensor'>
torch.int64
#
tensor([[True, True, True, True, True, True, True, True, True, True, True, True,
True, True, True, True, True, True, True, True, True, True, True, True,
True, True, True, True, True, True, True, True, True, True, True, True,
True, True, True, True, True, True, True, True, True, True, True, True,
True, True, True, True, True, True, True, True, True, True, True, True,
True, True, True, True, True, True, True, True, True, True, True, True,
True, True, True, True, True]])The data type within the tensor is torch.int64.
The values are neatly ordered and there are no gaps or skips.
8528d-fix
loading ... 8528d-fix.ckpt
# current data is:
tensor([[ 0.0000, 1.0000, 1.9999, 2.9996, 3.9998, 5.0000, 5.9992, 6.5492,
7.9996, 9.0008, 10.0000, 10.9992, 11.9984, 12.9977, 13.5477, 15.0000,
15.9992, 16.9984, 18.0016, 18.5516, 20.0000, 20.9953, 21.9984, 23.0016,
23.9969, 25.0000, 25.9953, 26.9984, 27.5484, 28.9969, 30.0000, 30.9953,
31.9984, 32.5484, 33.9969, 35.0000, 36.0031, 36.9906, 37.5406, 38.9969,
40.0000, 41.0031, 41.9906, 42.5406, 43.9969, 45.0000, 46.0031, 46.9906,
47.9938, 48.9969, 50.0000, 51.0031, 51.9906, 52.9938, 53.9969, 55.0000,
55.5500, 56.9906, 57.9938, 58.9969, 60.0000, 60.5500, 61.9906, 62.9938,
63.9969, 64.5469, 65.5500, 66.9906, 67.9938, 68.9969, 70.0000, 70.5500,
72.0062, 72.9938, 73.9812, 75.0000, 75.5500]])
<class 'torch.Tensor'>
torch.float32
# == if changed to torch.int64 ==
<class 'torch.Tensor'>
tensor([[ 0, 0, 1, 2, 3, 5, 5, 6, 7, 9, 10, 10, 11, 12, 13, 15, 15, 16,
18, 18, 20, 20, 21, 23, 23, 25, 25, 26, 27, 28, 30, 30, 31, 32, 33, 35,
36, 36, 37, 38, 40, 41, 41, 42, 43, 45, 46, 46, 47, 48, 50, 51, 51, 52,
53, 55, 55, 56, 57, 58, 60, 60, 61, 62, 63, 64, 65, 66, 67, 68, 70, 70,
72, 72, 73, 75, 75]])
torch.int64
# change to:
tensor([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
72, 73, 74, 75, 76]])
<class 'torch.Tensor'>
torch.int64
#
tensor([[ True, False, False, False, False, True, False, False, False, True,
True, False, False, False, False, True, False, False, True, False,
True, False, False, True, False, True, False, False, False, False,
True, False, False, False, False, True, True, False, False, False,
True, True, False, False, False, True, True, False, False, False,
True, True, False, False, False, True, False, False, False, False,
True, False, False, False, False, False, False, False, False, False,
True, False, True, False, False, True, False]])
corrupt token indexes : [1, 2, 3, 4, 6, 7, 8, 11, 12, 13, 14, 16, 17, 19, 21, 22, 24, 26, 27, 28, 29, 31, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 65, 66, 67, 68, 69, 71,
73, 74, 76]
missing token numbers : [4, 8, 14, 17, 19, 22, 24, 29, 34, 39, 44, 49, 54, 59, 69, 71, 74, 76]First, let's start with 1. Current Tensor.
The tensor type has changed to torch.float32.
Also, the values are slightly shifted to the negative in some places, with some being affected down to around -0.5 (index=76, value=75.55).
Next, looking from 2. torch.int64 tensor
onwards, an anomaly is immediately found: it reads tensor([[0, 0, 1, 2, 3 ... . As you can see by comparing it with the SD15 data, it should be 0, 1, 2, 3 ... here, so this is incorrect.
Furthermore, from 4. Deviation Judgment, the list of deviated numbers listed below it, and the list of missing values, it is clear that this deviation is occurring over a wide range.
Elysium_Anime_V3
# current data is:
tensor([[ 0.0000, 1.0000, 2.0000, 2.9995, 3.9999, 4.9985, 5.9990, 6.9994,
7.9999, 9.0004, 9.9971, 11.0013, 11.9979, 13.0021, 13.9988, 15.0031,
15.9998, 16.9965, 18.0007, 19.0049, 19.9942, 20.9984, 22.0025, 23.0068,
23.9958, 25.0000, 26.0042, 26.9935, 27.9976, 29.0019, 30.0061, 30.9954,
31.9996, 33.0037, 33.9930, 35.0121, 36.0015, 36.9907, 38.0098, 38.9991,
39.9884, 41.0074, 41.9967, 42.9861, 44.0051, 44.9944, 46.0137, 47.0028,
47.9917, 49.0107, 50.0000, 50.9897, 52.0084, 52.9983, 53.9869, 55.0066,
55.9953, 56.9852, 58.0039, 58.9935, 60.0122, 61.0019, 61.9908, 63.0102,
63.9991, 64.9880, 66.0075, 67.0262, 67.9860, 69.0047, 70.0242, 70.9832,
72.0030, 73.0214, 73.9815, 75.0000, 76.0197]])
<class 'torch.Tensor'>
torch.float32
# == if changed to torch.int64 ==
<class 'torch.Tensor'>
tensor([[ 0, 0, 1, 2, 3, 4, 5, 6, 7, 9, 9, 11, 11, 13, 13, 15, 15, 16,
18, 19, 19, 20, 22, 23, 23, 25, 26, 26, 27, 29, 30, 30, 31, 33, 33, 35,
36, 36, 38, 38, 39, 41, 41, 42, 44, 44, 46, 47, 47, 49, 50, 50, 52, 52,
53, 55, 55, 56, 58, 58, 60, 61, 61, 63, 63, 64, 66, 67, 67, 69, 70, 70,
72, 73, 73, 74, 76]])
torch.int64
# change to:
tensor([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
72, 73, 74, 75, 76]])
<class 'torch.Tensor'>
torch.int64
#
tensor([[ True, False, False, False, False, False, False, False, False, True,
False, True, False, True, False, True, False, False, True, True,
False, False, True, True, False, True, True, False, False, True,
True, False, False, True, False, True, True, False, True, False,
False, True, False, False, True, False, True, True, False, True,
True, False, True, False, False, True, False, False, True, False,
True, True, False, True, False, False, True, True, False, True,
True, False, True, True, False, False, True]])
corrupt token indexes : [1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 17, 20, 21, 24, 27, 28, 31, 32, 34, 37, 39, 40, 42, 43, 45, 48, 51, 53, 54, 56, 57, 59, 62, 64, 65, 68, 71, 74, 75]
missing token numbers : [8, 10, 12, 14, 17, 21, 24, 28, 32, 34, 37, 40, 43, 45, 48, 51, 54, 57, 59, 62, 65, 68, 71, 75]1. Current Tensor The tensor type is torch.float32.
There are places where the values have shifted to negative and places where they have shifted to positive.2. int64 0, 0, 1, 2, 3, 4, 5, 6, 7, 9 ... and the values are shifted.
Missing list 8, 10, 12, but the numbers missing are different from 8528d-fix, and no particular pattern is observed.
8528d-final (fp16)
Investigating as well.
From the top: current state, result when casting current state to int64, and whether each value is shifted.
loading ... 8528d-final.ckpt
no state_dict. direct model.
# current data is:
tensor([[ 0.0000, 1.0000, 2.0000, 3.0000, 4.0000, 5.0000, 6.0000, 6.5508,
8.0000, 9.0000, 10.0000, 11.0000, 12.0000, 13.0000, 13.5469, 15.0000,
16.0000, 17.0000, 18.0000, 18.5469, 20.0000, 21.0000, 22.0000, 23.0000,
24.0000, 25.0000, 26.0000, 27.0000, 27.5469, 29.0000, 30.0000, 31.0000,
32.0000, 32.5625, 34.0000, 35.0000, 36.0000, 37.0000, 37.5312, 39.0000,
40.0000, 41.0000, 42.0000, 42.5312, 44.0000, 45.0000, 46.0000, 47.0000,
48.0000, 49.0000, 50.0000, 51.0000, 52.0000, 53.0000, 54.0000, 55.0000,
55.5625, 57.0000, 58.0000, 59.0000, 60.0000, 60.5625, 62.0000, 63.0000,
64.0000, 64.5625, 65.5625, 67.0000, 68.0000, 69.0000, 70.0000, 70.5625,
72.0000, 73.0000, 74.0000, 75.0000, 75.5625]], dtype=torch.float16)
<class 'torch.Tensor'>
torch.float16
# == if changed to torch.int64 ==
<class 'torch.Tensor'>
tensor([[ 0, 1, 2, 3, 4, 5, 6, 6, 8, 9, 10, 11, 12, 13, 13, 15, 16, 17,
18, 18, 20, 21, 22, 23, 24, 25, 26, 27, 27, 29, 30, 31, 32, 32, 34, 35,
36, 37, 37, 39, 40, 41, 42, 42, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
54, 55, 55, 57, 58, 59, 60, 60, 62, 63, 64, 64, 65, 67, 68, 69, 70, 70,
72, 73, 74, 75, 75]])
torch.int64
# change to:
tensor([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
72, 73, 74, 75, 76]])
<class 'torch.Tensor'>
torch.int64
#
tensor([[ True, True, True, True, True, True, True, False, True, True,
True, True, True, True, False, True, True, True, True, False,
True, True, True, True, True, True, True, True, False, True,
True, True, True, False, True, True, True, True, False, True,
True, True, True, False, True, True, True, True, True, True,
True, True, True, True, True, True, False, True, True, True,
True, False, True, True, True, False, False, True, True, True,
True, False, True, True, True, True, False]])
corrupt token indexes : [7, 14, 19, 28, 33, 38, 43, 56, 61, 65, 66, 71, 76]
missing token numbers : [7, 14, 19, 28, 33, 38, 43, 56, 61, 66, 71, 76]1. Current Tensor The internal values have changed to torch.float16, but this is a reasonable change since fp16 processing is being performed.
2. int64 The first half is 0, 1, 2, 3, 4, 5, 6, 6, 8, so it is going well, but there is a slight deviation.
Corrupt/Missing list Compared to 8528d-fix, the list has become much shorter, but there is still a deviation. Also, there are missing values.
Results of Investigation 1
From the above investigation results, it was found that the values of the target keys are shifted.
Also, it was found that the ways in which the values shift—both positive and negative—are mixed even within the same model.
Investigation 2: Impact of shifted values within target keys
Regarding the shift in values within the tensor found in this investigation, I briefly mentioned at the beginning that fixing this can resolve the 'ignoring the 1st token' issue.
From this, it is understood that this value is a setting related to the relationship between the prompt and the model. However, it has not been determined how a shift occurs and what kind of impact it has.
Therefore, I investigated which position's values have what kind of impact by intentionally corrupting the correction tensor in the above correction script.
Image output conditions
The prompt is as follows. Seed is fixed, CLIP skip=1, eta=0
smile sleepy girl standing bear
Steps: 40, Sampler: Euler a, CFG scale: 7.5, Seed: 651296271, Face restoration: CodeFormer, Size: 512x512
Investigation 2-1. Trying to set each location to 0
The meaning of the numbers at the end of each filename is as follows.
1-0: tensor([[ 0,0,2,3,4,5 ...
2-0: tensor([[ 0,1,0,3,4,5 ...
3-0: tensor([[ 0,1,2,0,4,5 ...
3-1: tensor([[ 0,1,2,1,4,5 ...
1. The top one is the standard 8528d-fix. Smile is ignored.
2. keychange_8528dfix is the one with the fix script applied. Smile has returned.
3. break_1-0 Smile is lost again.
4. break_2-0 Sleepy is lost, and the eyes are clearly wide open.
5. break_3-0 Girl is lost, and it turned into a bear.
6. break_3-1 The value for the girl part was set to 1. Girl is not lost, and the smile of 1 is not lost either.
It is difficult to interpret the results of break_3-1.
break_3-1 Hypothesis #1: "Does it move as 'receive as the 3rd token, replacing 3.girl with 1.smile'?"
元 : smile sleepy girl standing bear
新？: smile sleepy smile standing bear=> This hypothesis can likely be rejected.
Since we know that losing girl in 3-0 results in a bear, it would be strange if it didn't also become a bear if girl is lost in 3-1.
break_3_1 Hypothesis #2: "The 3rd token should go to the 1st lane"
元 : smile sleepy girl standing bear
新？: smile sleepy (null) standing bear
girlThis might be possible.
In this case, girl might behave "as if it were the 1st token".
If so, the influence of girl should be increased.
Also, although I noted (null), it is unclear how the 3rd token, which no longer receives anything, behaves.
Investigation 2-2. Try moving values that seem to have influence to index=1
Therefore, I conducted an experiment with the following settings.
Original: 0, 1, 2, 3, 4, 5
4-1: 0, 1, 2, 3, 1, 4
5-1: 0 ,1, 2, 3, 4, 1
Since the prompt is 'smile sleepy girl standing bear',
in 4-1, standing should be affected.
In 5-1, bear should be affected.
Result of 4-1:
If standing joins the 1st token, does it become more... standing?
I can't tell the difference. I feel like the experimental setup is a bit flawed.
Result of 5-1:
If bear joins the 1st token, should the bear-ness increase?
In the 1st column, the bear ears became ribbons.
The 2nd column zoomed in a bit, so perhaps smile was emphasized?
The 3rd column has larger bear ears.
The 4th column has a slightly suppressed smile and a slightly closer face shot.
In 3-1 to 5-1, the meaning of the corresponding prompt number was strengthened.
(Can I really say that?!!!!!!)
I don't really understand, so I'll do a follow-up test.
Create 1-5. If the hypothesis is correct, 1.smile becomes (null) and overlaps with 5.bear, so the smile should weaken.
(Though it does look like the smile has become slightly more subdued...)
From this, it is thought that processing is being performed using the data in this key such that "the n-th token is treated as the x-th."
With this hypothesis in mind, looking at the results of 1-5 shows that while 5:bear has moved to the position of 1:smile, the effect of 1:smile has not disappeared, and it can be seen that the smile has weakened.
From the above, it can be said that the impact of the issue where the numerical values of the key content are shifted this time is that
"in a normal prompt, the influence should be in descending order from the front, but because the influence for positions where the values are shifted is calculated at the destination location, the influence gradient for each location in the prompt is no longer smooth."
Roughly speaking, this means
"if you arrange prompts based on your previous intuition, you will sometimes get waves or roughness in the effects."
Detailed reproduction of the corruption flow
Based on the knowledge gained so far, I will reproduce the corruption flow more concretely.
The reproduction steps and recipe are as follows.
Add Difference Merge
Add Difference Merge
Important point
Looking at the reproduction steps above, one might want to say "You knew this from the start!" but the point of doubt was that "it is not the case that the model always breaks when you do an Add Difference." This procedure will reproduce that part as well.
Step 1. Perform Add Difference
O1 = model_A + model_B + model_C, 1.0
Any combination is fine here.
The multiplier can also be anything.
Assume that models A, B, and C all have normal keys.
This combination does not cause problems with the target key.
O1 should often be able to obtain normal output (depending on the multiplier).
Step 2. Perform Add Difference
O2 = model_D + model_E + O1, 1.0
*Assume that models D and E all have normal keys.
This is the important part.
I brought the Add Difference-processed model O1 to (C) in the Add Difference operation. At this time, the following calculation is performed in the internal keys.
For simplicity, let's consider the part where the value in the key is 1.
(In other words, the part corresponding to the "1st token." This became 0 and caused the problem.)
$$ O2 = D + 1.0 * (E - O1) $$
Looking at what kind of calculation was performed in O1 here,
$$
O1 = A + 1.0 * (B - C) \\
= 1 + 1.0 * (1 - 1)
$$
"So O1 = 0, right????" is what it should be, but this is where it differs.
1.0 * (1-1) is a floating-point calculation, so the representation of 0 is not strictly 0.
To confirm this, I examined the same keys for a model that had undergone "difference extraction" using the method from the previous article, and the results are as follows.
(Investigation Results)
As a result, the calculation regarding O1 effectively becomes the following. If we let α be something close to 0 obtained by $ 1.0 * (1-1) $,
$$
O1 = A + 1.0 * ( B - C ) \\
= 1 + 1.0 * ( 1 - 1 ) \\
= 1 + α
$$
Now, we apply the O1 obtained from this calculation to the formula for O2. The calculation of 1-1 is performed again in the O2 calculation, and if we represent the residual value at that time as β,
$$
O2 = D + 1.0 * ( E - O1) \\
= 1 + 1.0 * ( 1 - 1 - α ) \\
= 1 + 1.0 * ( β - α ) \\
= 1 + ( β - α )
$$
This becomes. Since the definitions of α and β are very small values, their signs are undetermined, and their relative magnitudes are uncertain, 1 + (β - α) is
‘sometimes slightly less than 1’.※I do not have the expertise to know whether α or β is consistently larger, as both are very small values.
This being less than 1 leads to the phenomenon shown at the beginning of this article. "When loading the model, it is converted to torch.int64, but the calculation is performed by truncating the decimal part." This shows how the values within the keys become smaller than the original numerical values.
Summary
Damaged Areas
Regarding this issue where the first token is ignored, I believe I have almost identified the damaged area within the model structure.
Problems with Correction Methods
Also, while there were already provisional correction methods, each had its own problems.
Replacing the CLIP is a somewhat destructive method, and if you have also trained the CLIP, such as with DreamBooth, there was a possibility that the learning results would be lost.
Conversion to fp16 may not be a sufficient fix. Furthermore, since the numerical values in the keys mentioned above remain shifted, there was the problem that models treated this way would pass on this damage to the next generation.
Benefits of the Correction Method Created This Time
With the method of correcting the keys presented this time,
1. Weights and other learning results within the CLIP can be preserved
2. Correspondence with prompts is normalized
Remaining Issues
If the correction is applied, there is a possibility that you will feel the art style has changed because the way prompts affect the output changes. Also, especially if you are training/fine-tuning,
if learning was proceeding under the premise of this broken CLIP, it may cause changes in the model's drawing performance.
In any case, since "the art style changing" is a major event in image creation, you need to think carefully before applying the fix.
(I have shown that it is possible to handle each of these properly.)
That is all.