深度学习日常学习记录——归一化

最新推荐文章于 2026-06-19 13:00:33 发布

原创最新推荐文章于 2026-06-19 13:00:33 发布 · 1.5k 阅读

19 ·

本内容遵循CC 4.0 BY-SA版权协议

GEO检测

标签

#深度学习 #学习 #人工智能

                    
                    深度学习日常学习记录——归一化

总结
深度学习归一化技术总结

归一化方法数学公式归一化维度适用场景Batch依赖
Batch Norm (BN)μB=1m∑i=1mxiσB2=1m∑i=1m(xi−μB)2x^i=xi−μBσB2+ϵyi=γx^i+β\begin{aligned} \mu_B &= \frac{1}{m}\sum_{i=1}^m x_i \\ \sigma_B^2 &= \frac{1}{m}\sum_{i=1}^m (x_i - \mu_B)^2 \\ \hat{x}_i &= \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} \\ y_i &= \gamma \hat{x}_i + \beta \end{aligned}μB​σB2​x^i​yi​​=m1​i=1∑m​xi​=m1​i=1∑m​(xi​−μB​)2=σB2​+ϵ​xi​−μB​​=γx^i​+β​N,H,WCNN强依赖
Layer Norm (LN)μL=1H∑i=1HxiσL2=1H∑i=1H(xi−μL)2x^i=xi−μLσL2+ϵyi=γx^i+β\begin{aligned} \mu_L &= \frac{1}{H}\sum_{i=1}^H x_i \\ \sigma_L^2 &= \frac{1}{H}\sum_{i=1}^H (x_i - \mu_L)^2 \\ \hat{x}_i &= \frac{x_i - \mu_L}{\sqrt{\sigma_L^2 + \epsilon}} \\ y_i &= \gamma \hat{x}_i + \beta \end{aligned}μL​σL2​x^i​yi​​=H1​i=1∑H​xi​=H1​i=1∑H​(xi​−μL​)2=σL2​+ϵ​xi​−μL​​=γx^i​+β​C,H,WRNN/Transformer不依赖
Instance Norm (IN)μnc=1HW∑h,wxnchwσnc2=1HW∑h,w(xnchw−μnc)2x^nchw=xnchw−μncσnc2+ϵynchw=γx^nchw+β\begin{aligned} \mu_{nc} &= \frac{1}{HW}\sum_{h,w} x_{nchw} \\ \sigma_{nc}^2 &= \frac{1}{HW}\sum_{h,w} (x_{nchw} - \mu_{nc})^2 \\ \hat{x}_{nchw} &= \frac{x_{nchw} - \mu_{nc}}{\sqrt{\sigma_{nc}^2 + \epsilon}} \\ y_{nchw} &= \gamma \hat{x}_{nchw} + \beta \end{aligned}μnc​σnc2​x^nchw​ynchw​​=HW1​h,w∑​xnchw​=HW1​h,w∑​(xnchw​−μnc​)2=σnc2​+ϵ​xnchw​−μnc​​=γx^nchw​+β​H,W风格迁移不依赖
Group Norm (GN)μng=1(C/G)HW∑c,g,h,wxnchwσng2=1(C/G)HW∑c,g,h,w(xnchw−μng)2x^nchw=xnchw−μngσng2+ϵynchw=γx^nchw+β\begin{aligned} \mu_{ng} &= \frac{1}{(C/G)HW}\sum_{c,g,h,w} x_{nchw} \\ \sigma_{ng}^2 &= \frac{1}{(C/G)HW}\sum_{c,g,h,w} (x_{nchw} - \mu_{ng})^2 \\ \hat{x}_{nchw} &= \frac{x_{nchw} - \mu_{ng}}{\sqrt{\sigma_{ng}^2 + \epsilon}} \\ y_{nchw} &= \gamma \hat{x}_{nchw} + \beta \end{aligned}μng​σng2​x^nchw​ynchw​​=(C/G)HW1​c,g,h,w∑​xnchw​=(C/G)HW1​c,g,h,w∑​(xnchw​−μng​)2=σng2​+ϵ​xnchw​−μng​​=γx^nchw​+β​(C/G),H,W小batch CNN不依赖
Weight Norm (WN)w=g∣v∣vw = \frac{g}{|v|}vw=∣v∣g​v-全连接/RNN不依赖
Spectral Norm (SN)WSN=Wσ(W)W_{SN} = \frac{W}{\sigma(W)}WSN​=σ(W)W​-GAN不依赖
AdaINAdaIN(x,y)=σ(y)(x−μ(x)σ(x))+μ(y)\text{AdaIN}(x,y) = \sigma(y)\left(\frac{x - \mu(x)}{\sigma(x)}\right) + \mu(y)AdaIN(x,y)=σ(y)(σ(x)x−μ(x)​)+μ(y)H,W风格迁移不依赖

归一化方法	数学公式	归一化维度	适用场景	Batch依赖
Batch Norm (BN)	$μB=1m∑i=1mxiσB2=1m∑i=1m(xi−μB)2x^i=xi−μBσB2+ϵyi=γx^i+β\begin{aligned} \mu_B &= \frac{1}{m}\sum_{i=1}^m x_i \\ \sigma_B^2 &= \frac{1}{m}\sum_{i=1}^m (x_i - \mu_B)^2 \\ \hat{x}_i &= \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} \\ y_i &= \gamma \hat{x}_i + \beta \end{aligned}$	N,H,W	CNN	强依赖
Layer Norm (LN)	$μL=1H∑i=1HxiσL2=1H∑i=1H(xi−μL)2x^i=xi−μLσL2+ϵyi=γx^i+β\begin{aligned} \mu_L &= \frac{1}{H}\sum_{i=1}^H x_i \\ \sigma_L^2 &= \frac{1}{H}\sum_{i=1}^H (x_i - \mu_L)^2 \\ \hat{x}_i &= \frac{x_i - \mu_L}{\sqrt{\sigma_L^2 + \epsilon}} \\ y_i &= \gamma \hat{x}_i + \beta \end{aligned}$	C,H,W	RNN/Transformer	不依赖
Instance Norm (IN)	$μnc=1HW∑h,wxnchwσnc2=1HW∑h,w(xnchw−μnc)2x^nchw=xnchw−μncσnc2+ϵynchw=γx^nchw+β\begin{aligned} \mu_{nc} &= \frac{1}{HW}\sum_{h,w} x_{nchw} \\ \sigma_{nc}^2 &= \frac{1}{HW}\sum_{h,w} (x_{nchw} - \mu_{nc})^2 \\ \hat{x}_{nchw} &= \frac{x_{nchw} - \mu_{nc}}{\sqrt{\sigma_{nc}^2 + \epsilon}} \\ y_{nchw} &= \gamma \hat{x}_{nchw} + \beta \end{aligned}$	H,W	风格迁移	不依赖
Group Norm (GN)	$μng=1(C/G)HW∑c,g,h,wxnchwσng2=1(C/G)HW∑c,g,h,w(xnchw−μng)2x^nchw=xnchw−μngσng2+ϵynchw=γx^nchw+β\begin{aligned} \mu_{ng} &= \frac{1}{(C/G)HW}\sum_{c,g,h,w} x_{nchw} \\ \sigma_{ng}^2 &= \frac{1}{(C/G)HW}\sum_{c,g,h,w} (x_{nchw} - \mu_{ng})^2 \\ \hat{x}_{nchw} &= \frac{x_{nchw} - \mu_{ng}}{\sqrt{\sigma_{ng}^2 + \epsilon}} \\ y_{nchw} &= \gamma \hat{x}_{nchw} + \beta \end{aligned}$	(C/G),H,W	小batch CNN	不依赖
Weight Norm (WN)	$\frac{g}{\|v\|}v$	-	全连接/RNN	不依赖
Spectral Norm (SN)	$WSN=Wσ(W)W_{SN} = \frac{W}{\sigma(W)}$	-	GAN	不依赖
AdaIN	$AdaIN(x,y)=σ(y)(x−μ(x)σ(x))+μ(y)\text{AdaIN}(x,y) = \sigma(y)\left(\frac{x - \mu(x)}{\sigma(x)}\right) + \mu(y)$	H,W	风格迁移	不依赖