SYSTEM NOTICE

Auto translation by AI. Be sure, accuracy, nuances and authorial intent may not be fully reflected.

Complete Guide: How to fix A1111 for Flash Attention-3 & 2

2025年7月13日 19:48

On August 19, 2025, we added logic to prioritize FA-2 as the second option. Since FA-2 is not built-in, it requires a separate installation.

Introduction

xformers has evolved since version 0.0.31 to include Flash-Attention-3 by default.

However, A1111 has not been updated for over a year and does not support the latest logic, so even if you install 0.0.31post1, it lacks the logic to utilize FA-3.

In this update, I have modified it to prioritize the use of FA-3. As a result, it has been improved to 'use FA-3 or the kernel deemed most optimal' as shown below.

xformers.ops.memory_efficient_attention

This time, I have successfully analyzed the xformers kernel selection and am outputting logs to determine whether 'FA-3 is being used' or 'Cutlass is being used'.

In the image above, you can see that FA-3 is used for basic t2i, while Cutlass is used for FaceDetailer and HandDetailer.

When you run the following command in an environment where xformers is installed,

python -m xformers.info

If it is displayed as follows, FA-3 is ready to be used. Note that the following uses a self-built whl rather than the official whl.

xFormers 0.0.32+8354497d.d20250716
memory_efficient_attention.ckF:                    unavailable
memory_efficient_attention.ckB:                    unavailable
memory_efficient_attention.ck_decoderF:            unavailable
memory_efficient_attention.ck_splitKF:             unavailable
memory_efficient_attention.cutlassF-pt:            available
memory_efficient_attention.cutlassB-pt:            available
memory_efficient_attention.fa2F@0.0.0:             unavailable
memory_efficient_attention.fa2B@0.0.0:             unavailable
memory_efficient_attention.fa3F@2.8.0.post2-3-g3ba6f82: available
memory_efficient_attention.fa3B@2.8.0.post2-3-g3ba6f82: available
memory_efficient_attention.fa3F_splitKV@2.8.0.post2-3-g3ba6f82: available
memory_efficient_attention.triton_splitKF:         available
indexing.scaled_index_addF:                        available
indexing.scaled_index_addB:                        available
indexing.index_select:                             available
sp24.sparse24_sparsify_both_ways:                  available
sp24.sparse24_apply:                               available
sp24.sparse24_apply_dense_output:                  available
sp24._sparse24_gemm:                               available
sp24._cslt_sparse_mm_search@0.0.0:                 available
sp24._cslt_sparse_mm@0.0.0:                        available
swiglu.dual_gemm_silu:                             available
swiglu.gemm_fused_operand_sum:                     available
swiglu.fused.p.cpp:                                available
is_triton_available:                               True
pytorch.version:                                   2.7.1+cu128
pytorch.cuda:                                      available
gpu.compute_capability:                            8.9
gpu.name:                                          NVIDIA GeForce RTX 4070
dcgm_profiler:                                     unavailable
build.info:                                        available
build.cuda_version:                                1208
build.hip_version:                                 None
build.python_version:                              3.11.13
build.torch_version:                               2.7.1+cu128
build.env.TORCH_CUDA_ARCH_LIST:                    8.9
build.env.PYTORCH_ROCM_ARCH:                       None
build.env.XFORMERS_BUILD_TYPE:                     None
build.env.XFORMERS_ENABLE_DEBUG_ASSERTIONS:        None
build.env.NVCC_FLAGS:                              None
build.env.XFORMERS_PACKAGE_FROM:                   None
build.nvcc_version:                                12.8.93
source.privacy:                                    open source

The reason I built the whl myself is that the official whl is created with cu126. Since I am using a cu128 environment, I built it to match Pytorch 2.7.1+cu128.

The method for building xformers is explained below.

State of A1111 before the fix

Originally, FA-3 should have been usable just by using --xformers

However, does this mean that A1111, in its initial state, was unable to load the optimal attention?

### 1. The Original Logic

- **xformers is designed to automatically select the fastest available kernel (including FA-3) when A1111 is launched with the '--xformers' option.**

- **If the conditions of RTX 4070 + xformers 0.0.31.post1 or later + PyTorch/CUDA are met, FA-3 should have been used automatically.**

---

### 2. The Reality of A1111's Issues

- **The attention optimization logic in the A1111 core is outdated and has not kept up with the new xformers API and kernel specifications.**

- Therefore, it was incorrectly judged as "unusable" by old judgment logic such as "supports()" or "not_supported_reasons()", and FA-3 was not actually being used.

---

### 3. What was done in this fix

- Removed A1111's "pre-judgment logic" and modified it to call xformers.ops.memory_efficient_attention directly at all times.

- As a result, the "automatic kernel selection" logic of the xformers core is now 100% utilized, and FA-3 is now truly being used.

- Logs have also been organized to appear only once per generation.

---

### 4. Conclusion

- Originally, the design was such that FA-3 could be used with just "--xformers", but A1111's own old attention optimization logic was acting as an obstacle.

- With this fix, FA-3 can now be used as originally intended by the xformers design.

A1111 is originally designed to automatically use FA-3 with just "--xformers", but if the core's attention optimization logic remains old, there are cases where the latest version of xformers' FA-3 cannot be used correctly.

This time, I removed A1111's judgment logic and left it to xformers' automatic selection, fixing it so that FA-3 is truly used.-

The reality of SDP enforcement

Does this mean that in the initial state, only sdp or conventional cutlass could be used?

Yes, in the initial state (A1111 standard, with the latest xformers + FA-3 available environment), it is correct to assume that effectively only conventional kernels such as SDP (Scaled Dot Product Attention) or cutlass were being used. without mistake.

Reason

A1111's attention optimization logic relied on old judgments such as "supports()" and "not_supported_reasons()"
- It could not keep up with the specification changes of the latest xformers and FA-3
Even if FA-3 was available, it was incorrectly judged as "unusable", and memory_efficient_attention itself was not called
Therefore, A1111 automatically fell back to conventional kernels such as SDP or cutlass that have existed for a long time
FA-3 was never actually used

Summary

In the initial state, even in an environment where 'FA-3 available' was true, A1111 could only use SDP, cutlass, etc.
With this fix, 'FA-3 is now truly being used'.

Please explain the 'evidence that only conventional kernels were being used' and the 'basis for inferring which kernel was being used'.

That is a very good question.

Regarding the **'evidence that only conventional kernels were being used' and the 'basis for inferring which kernel was being used'**, I will technically explain the behavior in an A1111 + latest xformers + FA-3 available environment.

---

## 1. **Evidence that only conventional kernels were being used**

### A. **Flow of A1111's attention optimization logic**

- A1111's standard `modules/sd_hijack_optimizations.py` performed a pre-check on whether 'FA-3 can be used' via supports() or not_supported_reasons().

- Because it could not keep up with the latest xformers or FA-3 specification changes, **supports() was incorrectly judged as 'cannot be used' due to reasons like False or 'operator wasn't built'**.

### B. **Behavior when judged as 'cannot be used'**

- It did not call memory_efficient_attention itself and automatically fell back to conventional kernels such as SDP (Scaled Dot Product Attention) or cutlass.

- In other words, even in an FA-3 available environment, FA-3 was never used because A1111's attention optimization logic 'did not call it'.

### C. **Actual logs and errors**

- Reasons such as 'operator wasn't built' or 'not supported' appeared in the judgment of supports() or not_supported_reasons().

- Therefore, logs like '[A1111] Flash Attention 3 (FA-3) is NOT supported for this input. Fallback to other attention op.' were appearing.

---

## 2. **Basis for inferring which kernel was being used**

### A. **xformers' automatic kernel selection logic**

- xformers.ops.memory_efficient_attention automatically selects the "fastest available kernel (FA-3, FA-2, cutlass, triton, SDP, etc.)" when called.

- However, if A1111 does not call memory_efficient_attention itself, **PyTorch's standard SDP or A1111's own legacy implementation will be used** instead.

### B. **A1111 Fallback Implementation**

- If supports() returns NG, A1111 is designed to **automatically switch to legacy SDP (Scaled Dot Product Attention) or kernels like cutlass**.

- Therefore, it can be inferred that **even if FA-3 was available, SDP or cutlass was actually being used**.

## 3. **Summary**

- **With A1111's default settings, even in environments where FA-3 is available, memory_efficient_attention was not called due to "incorrect supports() evaluation," meaning only SDP or similar was being used.**

- **By modifying it to "always call memory_efficient_attention" in this fix, xformers' internal automatic kernel selection now functions, allowing FA-3 to be truly utilized.**

---

### [Supplementary Note]

- **To know 100% strictly which kernel was actually used, xformers' internal debug output or source code modification is required**, but based on A1111's logic, it can be asserted that "not being called" means "FA-3 was definitely not being used."

---

Modified file

modules/sd_hijack_optimizations.py

Published below

modules/processing.py

Published below

\venv\Lib\site-packages\xformers\xformers/ops/fmha/init.py

Published below

\venv\Lib\site-packages\xformers\ops\fmha\dispatch.py

Published below

Fix details

A comparison of the 'initial state' and 'post-fix state' for `modules/sd_hijack_optimizations.py` and `modules/processing.py` is summarized here with key points and explanations.

1. modules/sd_hijack_optimizations.py

[Initial state (A1111 standard)]

Pre-check whether FA-3 is truly usable using supports() or not_supported_reasons()
If the check fails, memory_efficient_attention itself is not called
Alignment of q, k, v dtypes is limited
Log output is not finely controlled

Example (excerpt/summary):

def xformers_attention_forward(self, x, context=None, mask=None, **kwargs):
    # ...（q, k, vの生成）...
    # supports()やnot_supported_reasons()で判定
    log_fa_operation_runtime(q, k, v)
    # 判定OKなら
    out = xformers.ops.memory_efficient_attention(q, k, v, ...)
    # NGなら従来のSDP等にフォールバック
    # ...（後略）...

[Post-fix]

Completely remove pre-checks like supports()completely removed
Always call xformers.ops.memory_efficient_attention directly
Always align q, k, v dtypes, prioritizing float16
Output logs only once per generation (FA3_LOGGED_THIS_GEN flag + reset_fa3_log)
Safely fallback to the original forward in case of exceptions

Example (excerpt/summary):

FA3_LOGGED_THIS_GEN = False

def reset_fa3_log():
    global FA3_LOGGED_THIS_GEN
    FA3_LOGGED_THIS_GEN = False

def xformers_attention_forward(self, x, context=None, mask=None):
    global FA3_LOGGED_THIS_GEN
    try:
        # ...（q, k, vの生成）...
        # dtypeをfloat16優先で揃える
        # 必ずmemory_efficient_attentionを呼ぶ
        out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=mask)
        if not FA3_LOGGED_THIS_GEN:
            print("[A1111] xformers.memory_efficient_attention called (FA-3 or best available kernel will be used)")
            FA3_LOGGED_THIS_GEN = True
        # ...（後略）...
    except Exception as e:
        # フォールバック
        if hasattr(self, "_old_attention_forward"):
            return self._old_attention_forward(x, context, mask)
        raise

Furthermore, to support extensions, the following modifications were implemented.

2. Current state (after modifications)

def xformers_attention_forward(self, x, context=None, mask=None, **kwargs):
    global FA3_LOGGED_THIS_GEN
    # Remove unsupported kwargs for xformers/SDP
    kwargs.pop('additional_tokens', None)
    try:
        import xformers.ops
        h = self.heads
        q_in = self.to_q(x)
        context = context if context is not None else x
        k_in = self.to_k(context)
        v_in = self.to_v(context)
        q, k, v = (t.reshape(t.shape[0], t.shape[1], h, -1) for t in (q_in, k_in, v_in))
        # dtype揃え
        dtypes = [q.dtype, k.dtype, v.dtype]
        if torch.float16 in dtypes:
            target_dtype = torch.float16
        else:
            target_dtype = q.dtype
        q = q.to(target_dtype)
        k = k.to(target_dtype)
        v = v.to(target_dtype)
        out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=mask)
        if not FA3_LOGGED_THIS_GEN:
            print("[A1111] xformers.memory_efficient_attention called (FA-3 or best available kernel will be used)")
            FA3_LOGGED_THIS_GEN = True
        out = out.to(x.dtype)
        b, n, h, d = out.shape
        out = out.reshape(b, n, h * d)
        return self.to_out(out)
    except Exception as e:
        print(f"[A1111] xformers.memory_efficient_attention failed, falling back to SDP. Exception: {e}")
        # Try PyTorch's scaled_dot_product_attention (SDP)
        try:
            h = self.heads
            q_in = self.to_q(x)
            context = context if context is not None else x
            k_in = self.to_k(context)
            v_in = self.to_v(context)
            q, k, v = (t.reshape(t.shape[0], t.shape[1], h, -1) for t in (q_in, k_in, v_in))
            dtype = q.dtype
            q = q.contiguous()
            k = k.contiguous()
            v = v.contiguous()
            out = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=mask, dropout_p=0.0, is_causal=False)
            out = out.to(dtype)
            b, n, h, d = out.shape
            out = out.reshape(b, n, h * d)
            print("[A1111] Fallback: torch.nn.functional.scaled_dot_product_attention (SDP) used.")
            return self.to_out(out)
        except Exception as e2:
            print(f"[A1111] SDP also failed, falling back to legacy. Exception: {e2}")
            if hasattr(self, "_old_attention_forward"):
                return self._old_attention_forward(x, context, mask)
            raise

Explanation

Add `**kwargs` so that it does not result in a TypeError when receiving additional keyword arguments (e.g., `additional_tokens`) passed by extensions
Pop and ignore keyword arguments not supported by xformers/SDP
First attempt FA-3 (memory_efficient_attention), and if it fails, automatically switch to SDP (scaled_dot_product_attention)
If that also fails, fall back to the traditional legacy implementation
Identify which kernel was used via logs
Full compatibility with extensions such as ControlNet and MultiDiffusion

Key Points

The current state is an ideal implementation that supports all of "extensions + FA-3/SDP/legacy automatic switching."
No matter what keyword arguments an extension passes, the A1111 core handles them flexibly without causing errors.
Since you can see which Attention kernel was used in the logs, debugging and verification are easy.

2. modules/processing.py

[Initial State (A1111 Standard)]

No reset process exists for FA-3 logs
Nothing specific is done at the beginning of process_images_inner

Example (excerpt/summary):

def process_images_inner(p: StableDiffusionProcessing) -> Processed:
    """this is the main loop ..."""
    # ...（画像生成処理）...

[After Modification]

Reset the FA3_LOGGED_THIS_GEN flag for each image generation
Call reset_fa3_log() at the beginning of process_images_inner

Example (excerpt/summary):

from modules.sd_hijack_optimizations import reset_fa3_log

def process_images_inner(p: StableDiffusionProcessing) -> Processed:
    reset_fa3_log()
    """this is the main loop ..."""
    # ...（画像生成処理）...

Addition

def process_images_inner(p: StableDiffusionProcessing) -> Processed:
    # --- Add for xformers kernel log reset and A1111 log ---
    try:
        from xformers.ops.fmha import reset_kernel_log
        reset_kernel_log()
    except ImportError:
        pass
    print("[A1111] xformers.memory_efficient_attention called (FA-3 or best available kernel will be used)")
    # --- End add ---
    reset_fa3_log()
    # ...既存の処理...

・Call reset_kernel_log() for every generation to reset the xformers kernel log
・Output the A1111 explanation log for every generation as well
・This ensures that the "A1111 log" and "xformers kernel name log" always appear
　together exactly once

3. xformers/ops/fmha/init.py

[Before modification (default)]

def _memory_efficient_attention_forward(
    inp: Inputs, op: Optional[Type[AttentionFwOpBase]]
) -> torch.Tensor:
    inp.validate_inputs()
    output_shape = inp.normalize_bmhk()
    if op is None:
        op = _dispatch_fw(inp, False)
    else:
        _ensure_op_supports_or_raise(ValueError, "memory_efficient_attention", op, inp)

    out, *_ = op.apply(inp, needs_gradient=False)
    return out.reshape(output_shape)

・Cannot tell which kernel (FA-3, cutlass, triton, etc.) is being used
・No log output

[After modification]

_kernel_log_shown = False

def reset_kernel_log():
    global _kernel_log_shown
    _kernel_log_shown = False

def _memory_efficient_attention_forward(
    inp: Inputs, op: Optional[Type[AttentionFwOpBase]]
) -> torch.Tensor:
    global _kernel_log_shown
    inp.validate_inputs()
    output_shape = inp.normalize_bmhk()
    if op is None:
        op = _dispatch_fw(inp, False)
    else:
        _ensure_op_supports_or_raise(ValueError, "memory_efficient_attention", op, inp)

    # 1生成ごとに1回だけカーネル名をログ出力
    if not _kernel_log_shown:
        print(f"[xformers] memory_efficient_attention: selected kernel = {getattr(op, 'NAME', str(op))}")
        _kernel_log_shown = True

    out, *_ = op.apply(inp, needs_gradient=False)
    return out.reshape(output_shape)

Addition

def process_images_inner(p: StableDiffusionProcessing) -> Processed:
    # --- Add for xformers kernel log reset and A1111 log ---
    try:
        from xformers.ops.fmha import reset_kernel_log
        reset_kernel_log()
    except ImportError:
        pass
    print("[A1111] xformers.memory_efficient_attention called (FA-3 or best available kernel will be used)")
    # --- End add ---
    reset_fa3_log()
    # ...既存の処理...

Explanation

Added a print + reset function to the xformers core that outputs "which kernel was selected" exactly once

Added "reset per generation + A1111 explanation log" to the A1111 core

This makes it possible to fully visualize which kernel was used at what timing

...

In August 2025, after upgrading my GPU to an RTX 5050 Ti 16GB, I became unable to use FA-3. To use FA-2 as a substitute, the following modifications were necessary.

The following modifications involve completely overwriting the files above, but since FA-3 works perfectly in an RTX 4070 12GB and Pytorch 2.7.1+cu128 environment, I am keeping the information above as an archive.

How to Fix Flash-Attention-2

# A1111 FA-2 Optimization Modification - Explanation of Modified Code

## 1. `modules/sd_hijack_optimizations.py`

### **xformers import section**

# Always try to import xformers if available
try:
    import xformers.ops
    shared.xformers_available = True
    print("[A1111] xformers successfully imported")
except Exception as e:
    print(f"[A1111] Cannot import xformers: {e}")
    shared.xformers_available = False

**Explanation**: Changed from conditional import to constant import attempt. Always enable xformers if available, and display a detailed error message if it fails.

### **xformers availability check**

**Explanation**: Removed the Compute Capability upper limit (`<= (9, 0)`). This allows xformers to be used on newer GPUs such as the RTX 5060 Ti.

def is_available(self):
    # Enable xformers if it's available and CUDA is available (no upper cap on compute capability)
    return shared.xformers_available and torch.cuda.is_available() and (6, 0) <= torch.cuda.get_device_capability(shared.device)

### **xformers_attention_forward function**

def xformers_attention_forward(self, x, context=None, mask=None, **kwargs):
    global FA3_LOGGED_THIS_GEN
    kwargs.pop('additional_tokens', None)
    try:
        import xformers.ops
        h = self.heads
        q_in = self.to_q(x)
        context = context if context is not None else x
        k_in = self.to_k(context)
        v_in = self.to_v(context)
        
        # シンプルなreshape（head dimension制限は削除済み）
        q, k, v = (t.reshape(t.shape[0], t.shape[1], h, -1) for t in (q_in, k_in, v_in))
        
        del q_in, k_in, v_in

        # 出力dtypeは入力xに合わせる
        dtype = x.dtype
        if shared.opts.upcast_attn:
            q, k, v = q.float(), k.float(), v.float()
        else:
            # Forgeに寄せて、FAカーネルを使うためfloat32なら半精度で実行→出力は元dtypeへ戻す
            if q.dtype == torch.float32:
                q = q.half(); k = k.half(); v = v.half()

        # Forgeのシンプルな実装を参考に、カスタムopを指定
        out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=mask, op=get_xformers_flash_attention_op(q, k, v))
        
        if not FA3_LOGGED_THIS_GEN:
            # 使用されたカーネルを確認
            kernel_info = get_xformers_kernel_info()
            if kernel_info:
                print(f"[A1111] xformers.memory_efficient_attention called - {kernel_info} kernel detected")
            else:
                print("[A1111] xformers.memory_efficient_attention called (FA-3→FA-2→Cutlass priority order)")
            FA3_LOGGED_THIS_GEN = True
        
        out = out.to(dtype)
        b, n, h, d = out.shape
        out = out.reshape(b, n, h * d)
        return self.to_out(out)
    except Exception as e:
        print(f"[A1111] xformers.memory_efficient_attention failed, falling back to SDP. Exception: {e}")
        # ... フォールバック処理

**Explanation**:

- Simplified by removing unnecessary head dimension restrictions

- Actively prioritize FA-2 by specifying custom ops

- Log detailed kernel information

- Removed intermediate variables for memory efficiency

### **Newly added functions**

#### **get_xformers_flash_attention_op**

def get_xformers_flash_attention_op(q, k, v):
    # Forgeの実装を参考に、より積極的にFA-2を優先
    try:
        # FA-2を優先的に試す
        flash_attention_op = xformers.ops.MemoryEfficientAttentionFlashAttentionOp
        fw, bw = flash_attention_op
        if fw.supports(xformers.ops.fmha.Inputs(query=q, key=k, value=v, attn_bias=None)):
            return flash_attention_op
    except Exception as e:
        # FA-2が使用できない場合はNoneを返してxformersに自動選択させる
        pass
    return None

**Explanation**: Checks if the FA-2 kernel is available and explicitly specifies it if it is. If not available, returns None to delegate to xformers' automatic selection.

#### **get_xformers_kernel_info**

def get_xformers_kernel_info():
    """Get information about the last used xformers kernel"""
    try:
        from xformers.ops.fmha import get_last_used_kernel
        kernel_name = get_last_used_kernel()
        if kernel_name:
            if 'fa2' in kernel_name.lower() or ('flash' in kernel_name.lower() and '2' in kernel_name):
                return "FA-2 (2nd priority)"
            elif 'fa3' in kernel_name.lower() or ('flash' in kernel_name.lower() and '3' in kernel_name):
                return "FA-3 (1st priority)"
            elif 'cutlass' in kernel_name.lower():
                return "Cutlass (3rd priority)"
            else:
                return f"{kernel_name} (unknown priority)"
    except ImportError:
        pass
    return None

**Explanation**: Retrieves detailed information about the last used kernel and displays it with priority ranking.

#### **check_and_configure_fa3**

def check_and_configure_fa3():
    """Check FA-3 availability and configure xformers accordingly"""
    try:
        import xformers.ops.fmha.dispatch
        import xformers.ops.fmha.flash3
        
        # Check if FA-3 is actually available
        fa3_available = xformers.ops.fmha.dispatch.fa3_available()
        
        if fa3_available:
            print("[A1111] FA-3 is available - enabling FA-3 priority")
            xformers.ops.fmha.dispatch._set_use_fa3(True)
        else:
            print("[A1111] FA-3 is not available - disabling FA-3 priority (FA-2 will be used)")
            xformers.ops.fmha.dispatch._set_use_fa3(False)
            
    except Exception as e:
        print(f"[A1111] Error checking FA-3 availability: {e}")
        # Default to False if we can't check
        try:
            import xformers.ops.fmha.dispatch
            xformers.ops.fmha.dispatch._set_use_fa3(False)
        except:
            pass

**Explanation**: Dynamically checks FA-3 availability at startup; enables it if available, or disables it to prioritize FA-2 if not.

def check_and_configure_fa3():
    """Check FA-3 availability and configure xformers accordingly"""
    try:
        import xformers.ops.fmha.dispatch
        import xformers.ops.fmha.flash3
        
        # Check if FA-3 is actually available
        fa3_available = xformers.ops.fmha.dispatch.fa3_available()
        
        if fa3_available:
            print("[A1111] FA-3 is available - enabling FA-3 priority")
            xformers.ops.fmha.dispatch._set_use_fa3(True)
        else:
            print("[A1111] FA-3 is not available - disabling FA-3 priority (FA-2 will be used)")
            xformers.ops.fmha.dispatch._set_use_fa3(False)
            
    except Exception as e:
        print(f"[A1111] Error checking FA-3 availability: {e}")
        # Default to False if we can't check
        try:
            import xformers.ops.fmha.dispatch
            xformers.ops.fmha.dispatch._set_use_fa3(False)
        except:
            pass

---

## 2. `modules/sd_hijack.py`

### **Automatic selection logic**

if selection == "Automatic" and len(optimizers) > 0:
    # Prefer xformers when available or explicitly requested
    try:
        xformers_opt = next((x for x in optimizers if isinstance(x, sd_hijack_optimizations.SdOptimizationXformers)), None)
    except Exception:
        xformers_opt = None

    if xformers_opt is not None and xformers_opt.is_available() and (getattr(shared.cmd_opts, "xformers", False) or getattr(shared.cmd_opts, "xformers_flash_attention", False) or getattr(shared, "xformers_available", False)):
        matching_optimizer = xformers_opt
    else:
        matching_optimizer = next(iter([x for x in optimizers if x.cmd_opt and getattr(shared.cmd_opts, x.cmd_opt, False)]), optimizers[0])

**Explanation**: If xformers is available, it is selected with priority over other optimizations. Even without explicit specification, it is automatically selected if xformers_available is True.

---

## 3. `venv\Lib\site-packages\xformers\ops\fmha\init.py`

### **Kernel Selection Log**

# 1生成ごとに1回だけカーネル名をログ出力
if not _kernel_log_shown:
    global _last_used_kernel
    kernel_name = getattr(op, 'NAME', str(op))
    _last_used_kernel = kernel_name
    
    # カーネル名をより詳細に解析
    if 'flash' in kernel_name.lower():
        if '3' in kernel_name or 'fa3' in kernel_name.lower():
            kernel_type = "Flash Attention 3 (FA-3)"
            priority_info = "✓ Highest priority (FA3→FA2→Cutlass)"
        elif '2' in kernel_name or 'fa2' in kernel_name.lower():
            kernel_type = "Flash Attention 2 (FA-2)"
            priority_info = "✓ Second priority (FA3→FA2→Cutlass)"
        else:
            kernel_type = "Flash Attention"
            priority_info = "? Unknown Flash Attention version"
    elif 'cutlass' in kernel_name.lower():
        kernel_type = "Cutlass"
        priority_info = "⚠ Third priority (FA3→FA2→Cutlass) - FA3/FA2 unavailable"
    elif 'ck' in kernel_name.lower():
        kernel_type = "CK"
        priority_info = "? CK kernel selected"
    elif 'triton' in kernel_name.lower():
        kernel_type = "Triton"
        priority_info = "? Triton kernel selected"
    else:
        kernel_type = kernel_name
        priority_info = "? Unknown kernel type"
    
    print(f"[xformers] memory_efficient_attention: selected kernel = {kernel_type} ({kernel_name}) - {priority_info}")
    _kernel_log_shown = True

**Explanation**: Analyzes kernel names in detail and displays them along with priority information. Clarifies the identification and priority of FA-3/FA-2/Cutlass.

### **Newly Added Functions**

#### **get_last_used_kernel**

def get_last_used_kernel():
    """Get the name of the last used kernel"""
    global _last_used_kernel
    return _last_used_kernel

**Explanation**: A function to retrieve the name of the last used kernel.

#### **reset_kernel_log**

def reset_kernel_log():
    """Reset the kernel log for the next generation"""
    global _kernel_log_shown, _last_used_kernel
    _kernel_log_shown = False
    _last_used_kernel = None

**Explanation**: A function to reset the log state for the next generation.

---

## 4. `venv\Lib\site-packages\xformers\ops\fmha\dispatch.py`

### **FA-3 Configuration**

_USE_FLASH_ATTENTION_3 = True

**Explanation**: Enables FA-3 by default. It is disabled only if it is unavailable during the dynamic check at startup.

### **Priority List Construction**

def _dispatch_fw_priority_list(
    inp: Inputs, needs_gradient: bool
) -> Sequence[Type[AttentionFwOpBase]]:
    if torch.version.cuda:
        flash3_op = [flash3.FwOp] if _get_use_fa3() else []
        
        # Check if FA-2 is available
        fa2_available = False
        fa2_reasons = []
        try:
            from xformers.ops.fmha import flash
            fa2_available = flash.FwOp.supports(inp)
            if not fa2_available:
                fa2_reasons = flash.FwOp.not_supported_reasons(inp)
        except Exception as e:
            fa2_reasons = [f"Exception: {e}"]
        
        # Debug output for first few calls
        import os
        if not hasattr(_dispatch_fw_priority_list, '_debug_count'):
            _dispatch_fw_priority_list._debug_count = 0
        _dispatch_fw_priority_list._debug_count += 1
        
        if True:  # デバッグ情報を有効化
            print(f"[xformers] Dispatch debug #{_dispatch_fw_priority_list._debug_count}:")
            print(f"  Input shapes: q={inp.query.shape}, k={inp.key.shape}, v={inp.value.shape}")
            print(f"  Input dtypes: q={inp.query.dtype}, k={inp.key.dtype}, v={inp.value.dtype}")
            print(f"  FA-3 enabled: {_get_use_fa3()}")
            print(f"  FA-2 available: {fa2_available}")
            if not fa2_available:
                print(f"  FA-2 not supported reasons: {fa2_reasons}")
        
        # Build priority list based on availability
        if fa2_available:
            priority_list_ops = deque(
                flash3_op
                + [
                    flash.FwOp,  # FA-2
                    cutlass.FwOp,
                ]
            )
        else:
            priority_list_ops = deque(
                flash3_op
                + [
                    cutlass.FwOp,
                ]
            )

**Explanation**:

- Dynamically check FA-2 availability

- Output detailed debug information

- Include FA-2 in the priority list if available

- If FA-3 is enabled, it is the highest priority, followed by FA-2, then Cutlass

---

## 5. `modules/initialize.py`

### **Initialization Process**

from modules import script_callbacks, sd_hijack_optimizations, sd_hijack
script_callbacks.on_list_optimizers(sd_hijack_optimizations.list_optimizers)
sd_hijack.list_optimizers()

# Check and configure FA-3 availability
sd_hijack_optimizations.check_and_configure_fa3()

startup_timer.record("scripts list_optimizers")

**Explanation**: Checks for FA-3 availability at startup and configures it appropriately. This ensures that FA-3 is prioritized in environments where it is available, and FA-2 is used in environments where it is not.

---

## 6. `modules/processing.py`

### **Log Messages**

print("[A1111] xformers.memory_efficient_attention called (FA-3→FA-2→Cutlass priority order)")

**Explanation**: Changed to log messages that clearly indicate priority.

---

## Summary of Key Improvements

1. **Forced xformers activation**: Changed from conditional import to constant import attempt.

2. **Removal of Compute Capability limit**: xformers can now be used on newer GPUs.

3. **Automatic selection logic fix**: Prioritizes xformers above all else.

4. **FA-2 priority implementation**: Active use of FA-2 through custom op specification.

5. **Detailed log output**: Displays detailed information on kernel selection and priority.

6. **FA-3 dynamic configuration**: Proper configuration via availability check at startup.

7. **Simplification**: Improved memory efficiency by removing unnecessary restrictions.

8. **Debug functionality**: Facilitates problem identification with detailed debug information.

With these changes, FA-2 is now used correctly even on the RTX 5060 Ti, and FA-3 is prioritized in environments where it is available.

SYSTEM NOTICE

Complete Guide: How to fix A1111 for Flash Attention-3 & 2

Introduction

xformers.ops.memory_efficient_attention

State of A1111 before the fix

### 1. **The Original Logic**

### 2. **The Reality of A1111's Issues**

### 3. What was done in this fix

### 4. Conclusion

The reality of SDP enforcement

Modified file

modules/sd_hijack_optimizations.py

modules/processing.py

\venv\Lib\site-packages\xformers\xformers/ops/fmha/init.py

\venv\Lib\site-packages\xformers\ops\fmha\dispatch.py

Fix details

1. modules/sd_hijack_optimizations.py

2. modules/processing.py

3. xformers/ops/fmha/init.py

How to Fix Flash-Attention-2

## **1. `modules/sd_hijack_optimizations.py`**

## **2. `modules/sd_hijack.py`**

## **3. `venv\Lib\site-packages\xformers\ops\fmha\__init__.py`**

## **4. `venv\Lib\site-packages\xformers\ops\fmha\dispatch.py`**

## **5. `modules/initialize.py`**

## **6. `modules/processing.py`**

## **Summary of Key Improvements**

いいなと思ったら応援しよう！

### 1. The Original Logic

### 2. The Reality of A1111's Issues

## 1. `modules/sd_hijack_optimizations.py`

## 2. `modules/sd_hijack.py`

## 3. `venv\Lib\site-packages\xformers\ops\fmha\init.py`

## 4. `venv\Lib\site-packages\xformers\ops\fmha\dispatch.py`

## 5. `modules/initialize.py`

## 6. `modules/processing.py`

## Summary of Key Improvements