在batchsize128的情況下，變化ramp-up，看最佳的ramp-upepoch是多少。
ramp-up
20
50
80
FRN
91.74%
92.30%
92.28%
感覺最優的rampup應在50~80個epoch之間，暫用50.
cifar100上FRN效果太差，復現的結果有問題。需要更正。
測試3 的方法究竟有什麼好處？更快的收斂究竟意味着什麼？目前FRN面臨的問題就是訓練困難。需要採用warmup和cosine形式的lr下降方式才能work。這種新的訓練方式能否降低對這些技巧的依賴，使得該方法更加普適？
σ \sigma σ 在ramp-up 50的情況下，變化batchsize，看各種算法的accuracy。
Image per GPU
128
32
BN
93.01%
92.01%
FRN
92.30%
92.31%
Max_Min_FRN
91.25%
91.25%
No_sigma_FRN_V1
91.51%
91.50%
No_sigma_FRN_V2
91.71%
91.71%
三種layer的公式如下：
B N : x ^ = γ x − μ b n σ b n + β , σ b n = ∑ ( x i − μ ) 2 H ∗ W ∗ B BN:
\hat{x}=\gamma\frac{x-\mu_{bn}}{\sigma_{bn}}+\beta,\sigma_{bn}=\sqrt{\frac{\sum
(x_i-\mu)^2}{H*W*B}} B N : x ^ = γ σ b n x − μ b n + β , σ b n = H ∗ W ∗ B ∑ ( x i − μ ) 2
F R N : x ^ = m a x ( γ x σ f r n + β , τ ) , σ f r n = ∑ x i 2 H ∗ W FRN:\hat{x}=max(\gamma\frac{x}{\sigma_{frn}}+\beta,\tau),\sigma_{frn}=\sqrt{\frac{\sum
x_i^2}{H*W}} F R N : x ^ = m a x ( γ σ f r n x + β , τ ) , σ f r n = H ∗ W ∑ x i 2
M a x _ M i n _ F R N : x ^ = m a x ( γ x σ r a n g e + β , τ ) , σ r a n g e = x m a x − x m i n 2 ∗ l n ( H ∗ W ) , x m a x , x m i n 是 在 H W 兩 個 維 度 統 計 的 Max\_Min\_FRN:\hat{x}=max(\gamma\frac{x}{\sigma_{range}}+\beta,\tau),\sigma_{range}=\frac{x_{max}-x_{min}}{\sqrt{2*ln(H*W)}},x_{max},x_{min}是在HW兩個維度統計的 M a x _ M i n _ F R N : x ^ = m a x ( γ σ r a n g e x + β , τ ) , σ r a n g e = 2 ∗ l n ( H ∗ W ) x m a x − x m i n , x m a x , x m i n 是 在 H W 兩 個 維 度 統 計 的
N o _ σ _ F R N _ V 1 : x ^ = m a x ( γ x + β , τ ) No\_\sigma\_FRN\_V1:\hat{x}=max(\gamma x +\beta,\tau) N o _ σ _ F R N _ V 1 : x ^ = m a x ( γ x + β , τ )
N o _ σ _ F R N _ V 2 : x ^ = m a x ( x , τ ) No\_\sigma\_FRN\_V2:\hat{x}=max( x,\tau) N o _ σ _ F R N _ V 2 : x ^ = m a x ( x , τ )
FRN在兩種batchsize中表現一致。這不僅依賴於FRNlayer的設計(σ f r n \sigma_{frn} σ f r n 0.1 ∗ b a t c h _ s i z e / 256 0.1*batch\_size/256 0 . 1 ∗ b a t c h _ s i z e / 2 5 6
FRN在Images per GPU 128的情況下，是差於BN的。但是paper裏面最大的Images per GPU是32，而在Images per GPU 32的情況下，FRN性能優於BN。所以這個結果是與paper裏面結果一致的。
Max_Min_FRN效果較FRN差距比較大。但是在小batchsize下，與BN基本持平。與FRN差距較大的原因，初步分析，是由於σ r a n g e = x m a x − x m i n 2 ∗ l n ( H ∗ W ) ≈ σ b n = ∑ ( x i − μ ) 2 H ∗ W ∗ B < < σ f r n = ∑ x i 2 H ∗ W \sigma_{range}=\frac{x_{max}-x_{min}}{\sqrt{2*ln(H*W)}}\approx\sigma_{bn}=\sqrt{\frac{\sum
(x_i-\mu)^2}{H*W*B}}<<\sigma_{frn}=\sqrt{\frac{\sum
x_i^2}{H*W}} σ r a n g e = 2 ∗ l n ( H ∗ W ) x m a x − x m i n ≈ σ b n = H ∗ W ∗ B ∑ ( x i − μ ) 2 < < σ f r n = H ∗ W ∑ x i 2
非常有意思的點是，當我們把方差σ \sigma σ γ , β \gamma,\beta γ , β
σ f r n \sigma_{frn} σ f r n 有高斯分佈性質，大體可以認爲，σ f r n ≈ μ 2 + σ 2 \sigma_{frn}\approx\sqrt{\mu^2+\sigma^2} σ f r n ≈ μ 2 + σ 2 μ , σ \mu,\sigma μ , σ σ r a n g e \sigma_{range} σ r a n g e ( x m a x + x m i n ) / 2 (x_{max} + x_{min})/2 ( x m a x + x m i n ) / 2
M a x _ M i n _ F R N _ V 2 : x ^ = m a x ( γ x σ r a n g e _ v 2 + β , τ ) , σ r a n g e _ v 2 = ( x m a x − x m i n ) 2 2 ∗ l n ( H ∗ W ) + ( x m a x + x m i n ) 2 4 Max\_Min\_FRN\_V2:\hat{x}=max(\gamma\frac{x}{\sigma_{range\_v2}}+\beta,\tau),\sigma_{range\_v2}=\sqrt{\frac{(x_{max}-x_{min})^2}{2*ln(H*W)}+\frac{(x_{max}+x_{min})^2}{4}} M a x _ M i n _ F R N _ V 2 : x ^ = m a x ( γ σ r a n g e _ v 2 x + β , τ ) , σ r a n g e _ v 2 = 2 ∗ l n ( H ∗ W ) ( x m a x − x m i n ) 2 + 4 ( x m a x + x m i n ) 2
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.01%
FRN
92.30%
92.31%
Max_Min_FRN
91.25%
91.25%
Max_Min_FRN _V2
91.80%
91.81%
L e a r n a b l e _ V 1 : y = m a x ( γ ∗ m i n ( x , α ) + β , τ ) Learnable\_V1:y=max(\gamma *min(x,\alpha)+\beta,\tau) L e a r n a b l e _ V 1 : y = m a x ( γ ∗ m i n ( x , α ) + β , τ )
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.40%
FRN
92.30%
92.31%
Max_Min_FRN
91.25%
91.25%
Max_Min_FRN _V2
91.80%
91.81%
Learnable_V1
90.55%
L e a r n a b l e _ V 2 : y = m a x ( γ ∗ m i n ( x , α ) + β , τ ) / ( α − β ) Learnable\_V2:y=max(\gamma *min(x,\alpha)+\beta,\tau)/(\alpha - \beta) L e a r n a b l e _ V 2 : y = m a x ( γ ∗ m i n ( x , α ) + β , τ ) / ( α − β )
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.01%
FRN
92.30%
92.31%
Max_Min_FRN
91.25%
91.25%
Max_Min_FRN _V2
91.80%
91.81%
Learnable_V1
90.55%
Learnable_V2
不收斂，效果很差
L e a r n a b l e _ V 3 : y = 2 ∗ m a x ( γ ∗ m i n ( x , α ) + β , τ ) / ( α + β ) Learnable\_V3:y=2*max(\gamma *min(x,\alpha)+\beta,\tau)/(\alpha + \beta) L e a r n a b l e _ V 3 : y = 2 ∗ m a x ( γ ∗ m i n ( x , α ) + β , τ ) / ( α + β )
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.01%
FRN
92.30%
92.31%
Max_Min_FRN
91.25%
91.25%
Max_Min_FRN _V2
91.80%
91.81%
Learnable_V1
90.55%
Learnable_V2
不收斂，效果很差
Learnable_V3
不收斂，效果很差
L e a r n a b l e _ V 4 : y = m a x ( γ ∗ m i n ( x , α ) + β , τ ) / d e t a c h ( α − β ) Learnable\_V4:y=max(\gamma *min(x,\alpha)+\beta,\tau)/detach(\alpha - \beta) L e a r n a b l e _ V 4 : y = m a x ( γ ∗ m i n ( x , α ) + β , τ ) / d e t a c h ( α − β )
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.01%
FRN
92.30%
92.31%
Max_Min_FRN
91.25%
91.25%
Max_Min_FRN _V2
91.80%
91.81%
Learnable_V1
90.55%
Learnable_V2
不收斂，效果很差
Learnable_V3
不收斂，效果很差
Learnable_V4
不收斂，效果很差
L e a r n a b l e _ V 5 : y = m a x ( γ ∗ l e a k y _ m i n ( x , u p l i m , s l o p e ) + β , τ ) l e a k y _ m i n ( x , l i m , s ) = { x , if x< lim s ∗ x + ( 1 − s ) ∗ l i m , if x > lim Learnable\_V5:y=max(\gamma *leaky\_min(x,uplim,slope)+\beta,\tau)\\
leaky\_min(x,lim,s)= \begin{cases}x,&\text {if x< lim} \\
s*x + (1 - s)*lim,&\text{if x > lim}\end{cases} L e a r n a b l e _ V 5 : y = m a x ( γ ∗ l e a k y _ m i n ( x , u p l i m , s l o p e ) + β , τ ) l e a k y _ m i n ( x , l i m , s ) = { x , s ∗ x + ( 1 − s ) ∗ l i m , if x< lim if x > lim
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.01%
FRN
92.30%
92.31%
Max_Min_FRN
91.25%
91.25%
Max_Min_FRN _V2
91.80%
91.81%
Learnable_V1
90.55%
Learnable_V2
不收斂，效果很差
Learnable_V3
不收斂，效果很差
Learnable_V4
不收斂，效果很差
Learnable_V5
90.24%
L e a r n a b l e _ V 6 : y = m a x ( γ ∗ l e a k y _ m i n _ v 2 ( x , u p l i m , s l o p e ) + β , τ ) l e a k y _ m i n _ v 2 ( x , l i m , s ) = { x , if x< lim s ∗ x + ( 1 − s ) ∗ l i m , if x > lim s = r s q r t ( l n ( H ∗ W ) ) Learnable\_V6:y=max(\gamma *leaky\_min\_v2(x,uplim,slope)+\beta,\tau)\\
leaky\_min\_v2(x,lim,s)= \begin{cases}x,&\text {if x< lim} \\
s*x + (1 - s)*lim,&\text{if x > lim}\end{cases}\\
s = rsqrt(ln(H*W)) L e a r n a b l e _ V 6 : y = m a x ( γ ∗ l e a k y _ m i n _ v 2 ( x , u p l i m , s l o p e ) + β , τ ) l e a k y _ m i n _ v 2 ( x , l i m , s ) = { x , s ∗ x + ( 1 − s ) ∗ l i m , if x< lim if x > lim s = r s q r t ( l n ( H ∗ W ) )
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.01%
FRN
92.30%
92.31%
Max_Min_FRN
91.25%
91.25%
Max_Min_FRN _V2
91.80%
91.81%
Learnable_V1
90.55%
Learnable_V2
不收斂，效果很差
Learnable_V3
不收斂，效果很差
Learnable_V4
不收斂，效果很差
Learnable_V5
90.55%
Learnable_V6
90.50%
L e a r n a b l e _ V 7 : y = m a x ( γ ∗ l e a k y _ m i n _ v 4 ( x , s ) + β , τ ) l e a k y _ m i n _ v 3 ( x , s ) = x / ( ( 1 + s ) ∗ s q r t ( l n ( H ∗ W ) ) ) Learnable\_V7:y=max(\gamma *leaky\_min\_v4(x,s)+\beta,\tau)\\
leaky\_min\_v3(x,s)=x / ((1 + s)*sqrt(ln(H*W))) L e a r n a b l e _ V 7 : y = m a x ( γ ∗ l e a k y _ m i n _ v 4 ( x , s ) + β , τ ) l e a k y _ m i n _ v 3 ( x , s ) = x / ( ( 1 + s ) ∗ s q r t ( l n ( H ∗ W ) ) )
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.01%
FRN
92.30%
92.31%
Max_Min_FRN
91.25%
91.25%
Max_Min_FRN _V2
91.80%
91.81%
Learnable_V1
90.55%
Learnable_V2
不收斂，效果很差
Learnable_V3
不收斂，效果很差
Learnable_V4
不收斂，效果很差
Learnable_V5
90.55%
Learnable_V6
90.50%
Learnable_V7
效果很差
L e a r n a b l e _ V 8 : y = m a x ( γ ∗ l e a k y _ m i n _ v 6 ( x , u p l i m , s l o p e , s ) + β , τ ) l e a k y _ m i n _ v 8 ( x , l i m , s l o p e , s ) = { x , if x< lim ( 1 + a b s ( s l o p e ) ) s ∗ x + ( 1 − ( 1 + a b s ( s l o p e ) ) s ) ∗ l i m , if x > lim s = r s q r t ( l n ( H ∗ W ) ) Learnable\_V8:y=max(\gamma *leaky\_min\_v6(x,uplim,slope,s)+\beta,\tau)\\
leaky\_min\_v8(x,lim,slope,s)= \begin{cases}x,&\text {if x< lim} \\
(1+abs(slope))s*x + (1 - (1+abs(slope))s)*lim,&\text{if x > lim}\end{cases}\\
s = rsqrt(ln(H*W)) L e a r n a b l e _ V 8 : y = m a x ( γ ∗ l e a k y _ m i n _ v 6 ( x , u p l i m , s l o p e , s ) + β , τ ) l e a k y _ m i n _ v 8 ( x , l i m , s l o p e , s ) = { x , ( 1 + a b s ( s l o p e ) ) s ∗ x + ( 1 − ( 1 + a b s ( s l o p e ) ) s ) ∗ l i m , if x< lim if x > lim s = r s q r t ( l n ( H ∗ W ) )
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.01%
FRN
92.30%
92.31%
Max_Min_FRN
91.25%
91.25%
Max_Min_FRN _V2
91.80%
91.81%
Learnable_V1
90.55%
Learnable_V2
不收斂，效果很差
Learnable_V3
不收斂，效果很差
Learnable_V4
不收斂，效果很差
Learnable_V5
90.55%
Learnable_V6
90.50%
Learnable_V7
效果很差
F R N : y = m a x ( γ x σ f r n + β , τ ) , σ = ∑ x i 2 H ∗ W FRN:y=max(\gamma\frac{x}{\sigma_{frn}}+\beta,\tau),\sigma=\sqrt{\frac{\sum
x_i^2}{H*W}} F R N : y = m a x ( γ σ f r n x + β , τ ) , σ = H ∗ W ∑ x i 2 ∂ L ∂ x = ( σ 2 + ϵ ) − 0.5 ∗ [ ∂ L ∂ y 11 . . . ∂ L ∂ y 1 W . . . . . . ∂ L ∂ y H 1 . . ∂ L ∂ y H W ] − s u m ( d i a g n o a l ( [ ∂ L ∂ y 11 . . . ∂ L ∂ y 1 W . . . . . . ∂ L ∂ y H 1 . . ∂ L ∂ y H W ] ⨂ [ x 11 . . . x 1 W . . . . . . x H 1 . . x H W ] T ( σ 2 + ϵ ) 1.5 ( H ∗ W ) \frac {\partial L}{\partial x}= (\sigma^2+\epsilon)^{-0.5}*
\begin{bmatrix}
\frac{\partial L}{\partial y_{11}} & ... & \frac{\partial L}{\partial y_{1W}} \\
.. & .. & .. \\
\frac{\partial L}{\partial y_{H1}} & .. & \frac{\partial L}{\partial y_{HW}}
\end{bmatrix} -
\frac{sum(diagnoal(
\begin{bmatrix}
\frac{\partial L}{\partial y_{11}} & ... & \frac{\partial L}{\partial y_{1W}} \\
.. & .. & .. \\
\frac{\partial L}{\partial y_{H1}} & .. & \frac{\partial L}{\partial y_{HW}}
\end{bmatrix}
\bigotimes \begin{bmatrix}
x_{11} & ... & x_{1W} \\
.. & .. & .. \\
x_{H1} & .. & x_{HW}
\end{bmatrix}^{T}}{(\sigma^2+\epsilon)^{1.5}(H*W)} ∂ x ∂ L = ( σ 2 + ϵ ) − 0 . 5 ∗ ⎣ ⎡ ∂ y 1 1 ∂ L . . ∂ y H 1 ∂ L . . . . . . . ∂ y 1 W ∂ L . . ∂ y H W ∂ L ⎦ ⎤ − ( σ 2 + ϵ ) 1 . 5 ( H ∗ W ) s u m ( d i a g n o a l ( ⎣ ⎡ ∂ y 1 1 ∂ L . . ∂ y H 1 ∂ L . . . . . . . ∂ y 1 W ∂ L . . ∂ y H W ∂ L ⎦ ⎤ ⨂ ⎣ ⎡ x 1 1 . . x H 1 . . . . . . . x 1 W . . x H W ⎦ ⎤ T ⨂ [ x 11 . . . x 1 W . . . . . . x H 1 . . x H W ] \bigotimes
\begin{bmatrix}
x_{11} & ... & x_{1W} \\
.. & .. & .. \\
x_{H1} & .. & x_{HW}
\end{bmatrix} ⨂ ⎣ ⎡ x 1 1 . . x H 1 . . . . . . . x 1 W . . x H W ⎦ ⎤
我們將σ \sigma σ
σ v 1 2 = ( m a x − m i n ) 2 2 ∗ l n ( H ∗ W ) ， σ v 1.1 2 = ( m a x − m i n ) 2 , σ v 1.2 2 = ( m a x − m i n ) 2 / 2 , σ v 1.3 2 = σ r a n g e _ v 2 = ( x m a x − x m i n ) 2 2 ∗ l n ( H ∗ W ) + ( x m a x + x m i n ) 2 4 \sigma_{v1}^{2}=\frac{(max - min)^2}{2*ln(H*W)}，\sigma_{v1.1}^{2}=(max - min)^2,\sigma_{v1.2}^{2}=(max - min)^2/2,\sigma_{v1.3}^{2}=\sigma_{range\_v2}=\sqrt{\frac{(x_{max}-x_{min})^2}{2*ln(H*W)}+\frac{(x_{max}+x_{min})^2}{4}} σ v 1 2 = 2 ∗ l n ( H ∗ W ) ( m a x − m i n ) 2 ， σ v 1 . 1 2 = ( m a x − m i n ) 2 , σ v 1 . 2 2 = ( m a x − m i n ) 2 / 2 , σ v 1 . 3 2 = σ r a n g e _ v 2 = 2 ∗ l n ( H ∗ W ) ( x m a x − x m i n ) 2 + 4 ( x m a x + x m i n ) 2
σ v 1.4 2 = σ r a n g e _ v 2 = ( x m a x − x m i n ) 2 4 + ( x m a x + x m i n ) 2 4 , σ v 1.5 2 = σ r a n g e _ v 2 = ( x m a x − x m i n ) 2 2 + ( x m a x + x m i n ) 2 2 ， σ v 1.6 2 = a b s ( x ) m a x 2 / 2 \sigma_{v1.4}^{2}=\sigma_{range\_v2}=\sqrt{\frac{(x_{max}-x_{min})^2}{4}+\frac{(x_{max}+x_{min})^2}{4}},\sigma_{v1.5}^{2}=\sigma_{range\_v2}=\sqrt{\frac{(x_{max}-x_{min})^2}{2}+\frac{(x_{max}+x_{min})^2}{2}}，\sigma_{v1.6}^{2}=abs(x)_{max}^{2}/2 σ v 1 . 4 2 = σ r a n g e _ v 2 = 4 ( x m a x − x m i n ) 2 + 4 ( x m a x + x m i n ) 2 , σ v 1 . 5 2 = σ r a n g e _ v 2 = 2 ( x m a x − x m i n ) 2 + 2 ( x m a x + x m i n ) 2 ， σ v 1 . 6 2 = a b s ( x ) m a x 2 / 2
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.01%
FRN
92.30%
92.31%
FRN
92.30%
92.31%
σ v 1 \sigma_{v1} σ v 1 效果很差，收斂慢
σ v 1.1 \sigma_{v1.1} σ v 1 . 1 效果很差，收斂慢,但是比σ v 1 \sigma_{v1} σ v 1
σ v 1.2 \sigma_{v1.2} σ v 1 . 2 與σ v 1.1 \sigma_{v1.1} σ v 1 . 1 σ v 1 \sigma_{v1} σ v 1
σ v 1.3 \sigma_{v1.3} σ v 1 . 3 90.3%
σ v 1.4 \sigma_{v1.4} σ v 1 . 4 90.7%
σ v 1.5 \sigma_{v1.5} σ v 1 . 5 88%
分析σ v 1.1 \sigma_{v1.1} σ v 1 . 1 σ v 1 \sigma_{v1} σ v 1 σ v 1.2 \sigma_{v1.2} σ v 1 . 2 σ v 1.1 \sigma_{v1.1} σ v 1 . 1 σ v 1.2 \sigma_{v1.2} σ v 1 . 2 σ v 1.0 \sigma_{v1.0} σ v 1 . 0 σ v 1.3 \sigma_{v1.3} σ v 1 . 3 σ v 1.2 \sigma_{v1.2} σ v 1 . 2 σ v 1.3 \sigma_{v1.3} σ v 1 . 3 σ v 1.4 \sigma_{v1.4} σ v 1 . 4 σ v 1.4 \sigma_{v1.4} σ v 1 . 4 σ v 1.3 \sigma_{v1.3} σ v 1 . 3 σ v 1.4 \sigma_{v1.4} σ v 1 . 4 σ v 1.5 \sigma_{v1.5} σ v 1 . 5 σ v 1.4 \sigma_{v1.4} σ v 1 . 4 σ v 1.5 \sigma_{v1.5} σ v 1 . 5 σ v 1.4 \sigma_{v1.4} σ v 1 . 4
σ v 2 2 = ( m a x ) 2 \sigma_{v2}^{2}=(max )^2 σ v 2 2 = ( m a x ) 2
σ v 3 2 = C = 2 l n ( H ∗ W ) \sigma_{v3}^{2}=C= \sqrt{2ln(H*W)} σ v 3 2 = C = 2 l n ( H ∗ W )
並沒有發現一個比較好的參數形式，能夠達到FRN的精度。
σ ^ = 1 + l r m a x − l r l r m a x ∗ σ \hat{\sigma} = 1 + \frac{lr_{max - lr}}{lr_{max}}*\sigma σ ^ = 1 + l r m a x l r m a x − l r ∗ σ
測試1：σ = 1 \sigma=1 σ = 1 σ = 1 \sigma=1 σ = 1 σ = σ f r n = ∑ x i 2 H ∗ W \sigma=\sigma_{frn}=\sqrt{\frac{\sum
x_i^2}{H*W}} σ = σ f r n = H ∗ W ∑ x i 2
測試2：e p o c h t o t a l _ e p o c h < s t a r t = 0.1 \frac{epoch}{total\_epoch}<start = 0.1 t o t a l _ e p o c h e p o c h < s t a r t = 0 . 1 σ ^ = 1 \hat{\sigma}=1 σ ^ = 1 e p o c h t o t a l _ e p o c h < e n d = 0.9 \frac{epoch}{total\_epoch}<end= 0.9 t o t a l _ e p o c h e p o c h < e n d = 0 . 9 σ ^ = m a x ( 1 , ( e p o c h t o t a l _ e p o c h ∗ ( e n d − s t a r t ) − s t a r t e n d − s t a r t ) ∗ σ ) \hat{\sigma}=max(1,(\frac{epoch}{total\_epoch*(end-start)}-\frac{start}{end-start})*\sigma) σ ^ = m a x ( 1 , ( t o t a l _ e p o c h ∗ ( e n d − s t a r t ) e p o c h − e n d − s t a r t s t a r t ) ∗ σ ) e p o c h t o t a l _ e p o c h > e n d \frac{epoch}{total\_epoch}>end t o t a l _ e p o c h e p o c h > e n d σ ^ = m a x ( 1 , σ ) \hat{\sigma}=max(1,\sigma) σ ^ = m a x ( 1 , σ )
測試3e p o c h t o t a l _ e p o c h < s t a r t = 0.1 \frac{epoch}{total\_epoch}<start = 0.1 t o t a l _ e p o c h e p o c h < s t a r t = 0 . 1 σ ^ = 1 \hat{\sigma}=1 σ ^ = 1 e p o c h t o t a l _ e p o c h < e n d = 0.9 \frac{epoch}{total\_epoch}<end= 0.9 t o t a l _ e p o c h e p o c h < e n d = 0 . 9 σ ^ = m a x ( 1 , ( e p o c h t o t a l _ e p o c h ∗ ( e n d − s t a r t ) − s t a r t e n d − s t a r t ) ∗ σ ) \hat{\sigma}=max(1,(\frac{epoch}{total\_epoch*(end-start)}-\frac{start}{end-start})*\sigma) σ ^ = m a x ( 1 , ( t o t a l _ e p o c h ∗ ( e n d − s t a r t ) e p o c h − e n d − s t a r t s t a r t ) ∗ σ ) e p o c h t o t a l _ e p o c h > e n d \frac{epoch}{total\_epoch}>end t o t a l _ e p o c h e p o c h > e n d σ ^ = m a x ( 1 , ( e p o c h t o t a l _ e p o c h ∗ ( e n d − s t a r t ) − s t a r t e n d − s t a r t ) ∗ σ ) ) \hat{\sigma}=max(1,(\frac{epoch}{total\_epoch*(end-start)}-\frac{start}{end-start})*\sigma)) σ ^ = m a x ( 1 , ( t o t a l _ e p o c h ∗ ( e n d − s t a r t ) e p o c h − e n d − s t a r t s t a r t ) ∗ σ ) ) σ \sigma σ
測試4α = e p o c h t o t a l _ e p o c h ∗ ( s t a r t − e n d ) − e n d s t a r t − e n d \alpha = \frac{epoch}{total\_epoch*(start-end)}-\frac{end}{start-end} α = t o t a l _ e p o c h ∗ ( s t a r t − e n d ) e p o c h − s t a r t − e n d e n d e p o c h t o t a l _ e p o c h < s t a r t = 0.1 \frac{epoch}{total\_epoch}<start = 0.1 t o t a l _ e p o c h e p o c h < s t a r t = 0 . 1 σ ^ = 1 \hat{\sigma}=1 σ ^ = 1 e p o c h t o t a l _ e p o c h < e n d = 0.9 \frac{epoch}{total\_epoch}<end= 0.9 t o t a l _ e p o c h e p o c h < e n d = 0 . 9 σ ^ = m a x ( α , ( e p o c h t o t a l _ e p o c h ∗ ( e n d − s t a r t ) − s t a r t e n d − s t a r t ) ∗ σ ) \hat{\sigma}=max(\alpha ,(\frac{epoch}{total\_epoch*(end-start)}-\frac{start}{end-start})*\sigma) σ ^ = m a x ( α , ( t o t a l _ e p o c h ∗ ( e n d − s t a r t ) e p o c h − e n d − s t a r t s t a r t ) ∗ σ ) e p o c h t o t a l _ e p o c h > e n d \frac{epoch}{total\_epoch}>end t o t a l _ e p o c h e p o c h > e n d σ ^ = m a x ( 1 , σ ) \hat{\sigma}=max(1,\sigma) σ ^ = m a x ( 1 , σ )
總結
接下來，基於最初的FRN版本
F R N : y = m a x ( γ x σ f r n + β , τ ) , σ f r n = ∑ x i 2 H ∗ W FRN:y=max(\gamma\frac{x}{\sigma_{frn}}+\beta,\tau),\sigma_{frn}=\sqrt{\frac{\sum
x_i^2}{H*W}} F R N : y = m a x ( γ σ f r n x + β , τ ) , σ f r n = H ∗ W ∑ x i 2
y τ > 0 = τ ∗ m a x ( x / d e t a c h ( τ ) , 1 ) , ( τ > 0 ) = { τ ∗ x d e t a c h ( τ ) , if x > τ , τ > 0 τ , if x < τ , τ > 0 y_{\tau>0}=\tau *max(x/detach(\tau),1),(\tau>0)\\
= \begin{cases}\tau*\frac{x}{detach(\tau)}, & \text {if x > $\tau$ ,$\tau>0$} \\ \tau, & \text{if x < $\tau$ ,$\tau>0$}\end{cases} y τ > 0 = τ ∗ m a x ( x / d e t a c h ( τ ) , 1 ) , ( τ > 0 ) = { τ ∗ d e t a c h ( τ ) x , τ , if x > τ , τ > 0 if x < τ , τ > 0
y τ < 0 = τ ∗ m i n ( x / d e t a c h ( τ ) , 1 ) , ( τ < 0 ) = { τ ∗ x d e t a c h ( τ ) , if x > τ , τ < 0 τ , if x < τ , τ < 0 y_{\tau<0}=\tau *min(x/detach(\tau),1),(\tau<0)
\\= \begin{cases}\tau*\frac{x}{detach(\tau)}, & \text {if x > $\tau$ ,$\tau<0$} \\ \tau, & \text{if x < $\tau$ ,$\tau<0$}\end{cases} y τ < 0 = τ ∗ m i n ( x / d e t a c h ( τ ) , 1 ) , ( τ < 0 ) = { τ ∗ d e t a c h ( τ ) x , τ , if x > τ , τ < 0 if x < τ , τ < 0 ∂ y ∂ τ = { x d e t a c h ( τ ) if x > τ , τ > 0 1 , if x < τ , τ > 0 \frac{\partial y}{\partial \tau} = \begin{cases}\frac{x}{detach(\tau)}& \text {if x > $\tau$ ,$\tau>0$}\\1,& \text {if x <$\tau$ ,$\tau>0$}\end{cases} ∂ τ ∂ y = { d e t a c h ( τ ) x 1 , if x > τ , τ > 0 if x < τ , τ > 0 ∂ L ∂ τ = ∂ L ∂ y ∂ y ∂ τ = ∂ L ∂ y ∗ { x d e t a c h ( τ ) if x > τ , τ > 0 1 , if x < τ , τ > 0 \frac{\partial L}{\partial \tau}=\frac{\partial L}{\partial y}\frac{\partial y}{\partial \tau}=\frac{\partial L}{\partial y}*\begin{cases}\frac{x}{detach(\tau)}& \text {if x > $\tau$ ,$\tau>0$}\\1,& \text {if x <$\tau$ ,$\tau>0$}\end{cases} ∂ τ ∂ L = ∂ y ∂ L ∂ τ ∂ y = ∂ y ∂ L ∗ { d e t a c h ( τ ) x 1 , if x > τ , τ > 0 if x < τ , τ > 0
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.01%
FRN
92.30%
92.31%
Max導數修正V1
90.49%
∂ L ∂ τ = ∂ L ∂ y ∂ y ∂ τ = ∂ L ∂ y ∗ { − 1 if x > τ , τ > 0 1 , if x < τ , τ > 0 \frac{\partial L}{\partial \tau}=\frac{\partial L}{\partial y}\frac{\partial y}{\partial \tau}=\frac{\partial L}{\partial y}*\begin{cases}-1& \text {if x > $\tau$ ,$\tau>0$}\\1,& \text {if x <$\tau$ ,$\tau>0$}\end{cases} ∂ τ ∂ L = ∂ y ∂ L ∂ τ ∂ y = ∂ y ∂ L ∗ { − 1 1 , if x > τ , τ > 0 if x < τ , τ > 0
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.01%
FRN
92.30%
92.31%
Max導數修正V1
90.49%
Max導數修正V2
92.02%
∂ L ∂ τ = ∂ L ∂ y ∂ y ∂ τ = ∂ L ∂ y ∗ { m e a n ( y ) − τ if x > τ , τ > 0 1 , if x < τ , τ > 0 \frac{\partial L}{\partial \tau}=\frac{\partial L}{\partial y}\frac{\partial y}{\partial \tau}=\frac{\partial L}{\partial y}*\begin{cases}mean(y) - \tau& \text {if x > $\tau$ ,$\tau>0$}\\1,& \text {if x <$\tau$ ,$\tau>0$}\end{cases} ∂ τ ∂ L = ∂ y ∂ L ∂ τ ∂ y = ∂ y ∂ L ∗ { m e a n ( y ) − τ 1 , if x > τ , τ > 0 if x < τ , τ > 0
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.01%
FRN
92.30%
92.31%
Max導數修正V1
90.49%
Max導數修正V2
92.02%
Max導數修正V3
91.47%
∂ L ∂ τ = ∂ L ∂ y ∂ y ∂ τ = ∂ L ∂ y ∗ { 1 if x > τ , τ > 0 1 , if x < τ , τ > 0 \frac{\partial L}{\partial \tau}=\frac{\partial L}{\partial y}\frac{\partial y}{\partial \tau}=\frac{\partial L}{\partial y}*\begin{cases}1& \text {if x > $\tau$ ,$\tau>0$}\\1,& \text {if x <$\tau$ ,$\tau>0$}\end{cases} ∂ τ ∂ L = ∂ y ∂ L ∂ τ ∂ y = ∂ y ∂ L ∗ { 1 1 , if x > τ , τ > 0 if x < τ , τ > 0
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.01%
FRN
92.30%
92.31%
Max導數修正V1
90.49%
Max導數修正V2
92.02%
Max導數修正V3
91.47%
Max導數修正V4
92.13%
首先修改正向傳播值
y = m a x ( γ x σ f r n + β , τ ) − τ , σ f r n = ∑ x i 2 H ∗ W y=max(\gamma\frac{x}{\sigma_{frn}}+\beta,\tau) - \tau,\sigma_{frn}=\sqrt{\frac{\sum
x_i^2}{H*W}} y = m a x ( γ σ f r n x + β , τ ) − τ , σ f r n = H ∗ W ∑ x i 2 τ \tau τ τ \tau τ ∂ L ∂ τ = ∂ L ∂ y ∂ y ∂ τ = ∂ L ∂ y ∗ { m e a n ( y ) − τ if x > τ , τ > 0 1 , if x < τ , τ > 0 \frac{\partial L}{\partial \tau}=\frac{\partial L}{\partial y}\frac{\partial y}{\partial \tau}=\frac{\partial L}{\partial y}*\begin{cases}mean(y) - \tau& \text {if x > $\tau$ ,$\tau>0$}\\1,& \text {if x <$\tau$ ,$\tau>0$}\end{cases} ∂ τ ∂ L = ∂ y ∂ L ∂ τ ∂ y = ∂ y ∂ L ∗ { m e a n ( y ) − τ 1 , if x > τ , τ > 0 if x < τ , τ > 0
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.01%
FRN
92.30%
92.31%
Max導數修正V1
90.49%
Max導數修正V2
92.02%
Max導數修正V3
91.47%
Max導數修正V4
92.13%
Max導數修正V5
89.35%
首先修改正向傳播值
y = m a x ( γ x σ f r n + β , τ ) − τ , σ f r n = ∑ x i 2 H ∗ W y=max(\gamma\frac{x}{\sigma_{frn}}+\beta,\tau) - \tau,\sigma_{frn}=\sqrt{\frac{\sum
x_i^2}{H*W}} y = m a x ( γ σ f r n x + β , τ ) − τ , σ f r n = H ∗ W ∑ x i 2 τ \tau τ τ \tau τ ∂ L ∂ τ = ∂ L ∂ y ∂ y ∂ τ = ∂ L ∂ y ∗ { m e a n ( y ) − τ − 1 if x > τ , τ > 0 0 , if x < τ , τ > 0 \frac{\partial L}{\partial \tau}=\frac{\partial L}{\partial y}\frac{\partial y}{\partial \tau}=\frac{\partial L}{\partial y}*\begin{cases}mean(y) - \tau - 1& \text {if x > $\tau$ ,$\tau>0$}\\0,& \text {if x <$\tau$ ,$\tau>0$}\end{cases} ∂ τ ∂ L = ∂ y ∂ L ∂ τ ∂ y = ∂ y ∂ L ∗ { m e a n ( y ) − τ − 1 0 , if x > τ , τ > 0 if x < τ , τ > 0
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.01%
FRN
92.30%
92.31%
Max導數修正V1
90.49%
Max導數修正V2
92.02%
Max導數修正V3
91.47%
Max導數修正V4
92.13%
Max導數修正V5
89.35%
Max導數修正V6
91.00%
經過實驗發現，在BN，LN上應用lr測試3的方法，並不work。BN會大大降低訓練速度，LN的數值會急劇增大，出現nan值。GN上沒有測試，估計也不行。
經過實驗測試，在沒有warm up的時候，FRN是不收斂的，lr測試3的方法也是，所以warm up對新方法來說，仍然是必須的
FRN中除以σ f r n \sigma_{frn} σ f r n σ f r n \sigma_{frn} σ f r n ∂ f ∂ x = 1 σ f r n 2 + ϵ ∗ ∂ f ∂ x ^ \frac{\partial f}{\partial x}=\frac{1}{\sqrt{\sigma_{frn}^{2}+\epsilon}}*\frac{\partial f}{\partial \hat{x}} ∂ x ∂ f = σ f r n 2 + ϵ 1 ∗ ∂ x ^ ∂ f
Image per GPU
128
32(batchsize:64,GPU:2)
BN
93.01%
92.01%
FRN
92.30%
92.31%
No_sigma_FRN_V1
91.51%
91.50%
No_sigma_FRN_V2
91.71%
91.71%
No_grad_FRN
91.0%
如果沒有導數，甚至還不如直接不除σ f r n \sigma_{frn} σ f r n
先檢查模塊的正確性，再做實驗
比較了paper中tensorflow版本的輸出和自己實現的pytorch版本的輸出。
import tensorflow as tf
tf. __version__
def FRNLayer ( x, tau, beta, gamma, eps= 1e - 6 ) :
nu2 = tf. reduce_mean( tf. square( x) , axis= [ 1 , 2 ] , keepdims= True )
x = x / tf. sqrt( nu2 + tf. abs ( eps) )
return tf. maximum( gamma * x + beta, tau)
x = tf. reshape( tf. range ( 2 , 34 . ) , [ 1 , 4 , 4 , 2 ] )
print ( x[ 0 , : , : , 0 ] )
gamma = tf. Variable( tf. ones( [ 1 , 1 , 1 , 2 ] ) )
beta = tf. Variable( tf. zeros( [ 1 , 1 , 1 , 2 ] ) )
tau = tf. Variable( tf. zeros( [ 1 , 1 , 1 , 2 ] ) )
with tf. GradientTape( ) as g:
g. watch( x)
y = FRNLayer( x, tau, beta, gamma)
loss = tf. reduce_sum( y)
print ( 'output:' , y[ 0 , : , : , 0 ] )
print ( 'loss:' , loss)
dl_dtau = g. gradient( loss, tau)
print ( dl_dtau)
class FilterResponseNormalization ( nn. Module) :
def __init__ ( self, num_features, eps= 1e - 6 ) :
"""
Input Variables:
----------------
beta, gamma, tau: Variables of shape [1, C, 1, 1].
eps: A scalar constant or learnable variable.
"""
super ( FilterResponseNormalization, self) . __init__( )
self. beta = nn. parameter. Parameter(
torch. Tensor( 1 , num_features, 1 , 1 ) , requires_grad= True )
self. gamma = nn. parameter. Parameter(
torch. Tensor( 1 , num_features, 1 , 1 ) , requires_grad= True )
self. tau = nn. parameter. Parameter(
torch. Tensor( 1 , num_features, 1 , 1 ) , requires_grad= True )
self. eps = nn. parameter. Parameter( torch. Tensor( [ eps] ) )
self. reset_parameters( )
def reset_parameters ( self) :
nn. init. ones_( self. gamma)
nn. init. zeros_( self. beta)
nn. init. zeros_( self. tau)
def forward ( self, x) :
"""
Input Variables:
----------------
x: Input tensor of shape [NxCxHxW]
"""
n, c, h, w = x. shape
assert ( self. gamma. shape[ 1 ] , self. beta. shape[ 1 ] , self. tau. shape[ 1 ] ) == ( c, c, c)
nu2 = x. pow ( 2 ) . mean( dim= ( 2 , 3 ) , keepdim= True )
x = x * torch. rsqrt( nu2 + torch. abs ( self. eps) )
return torch. max ( self. gamma * x + self. beta, self. tau)
x = torch. transpose( torch. transpose( torch. arange( 2 , 34 . ) . reshape( [ 1 , 4 , 4 , 2 ] ) , 1 , 3 ) , 2 , 3 )
print ( x[ 0 , 0 , : , : ] )
frn = FilterResponseNormalization( 2 )
y = frn( x)
print ( 'output:' , y[ 0 , 0 , : , : ] )
print ( "loss:" , loss)
loss = torch. sum ( y)
loss. backward( )
print ( frn. tau. grad)
import torch
x = torch. arange( 0 , 16 , dtype= torch. float32) . reshape( 1 , 1 , 4 , 4 )
n, c, h, w = x. shape
print ( x. shape)
channel_max = torch. max ( torch. max ( x, dim= 2 , keepdim= True ) [ 0 ] , dim= 3 , keepdim= True ) [ 0 ]
channel_min = torch. min ( torch. min ( x, dim= 2 , keepdim= True ) [ 0 ] , dim= 3 , keepdim= True ) [ 0 ]
print ( channel_max)
print ( channel_min)
Cn = torch. log( torch. tensor( h * w + 0.000001 ) ) * 2
print ( Cn)
import torch
class MyMax ( torch. autograd. Function) :
@staticmethod
def forward ( self, x, tau) :
self. save_for_backward( x, tau)
output = torch. max ( x, tau)
return output
@staticmethod
def backward ( self, grad_output) :
x, tau = self. saved_tensors
dl_dx = grad_output. clone( )
dl_dx[ x < tau] = 0
dl_dtau = grad_output. clone( )
A = ( x > tau) . float ( ) * ( x / ( tau + 1e - 6 ) ) * dl_dtau
print ( "A" )
print ( A)
dl_dtau = dl_dtau* ( x < tau) . float ( ) + A
return dl_dx, dl_dtau
x = torch. nn. parameter. Parameter( torch. reshape( torch. range ( 1 , 8 . ) , [ 2 , 2 , 2 , 1 ] ) )
print ( x[ 0 , 0 , : , : ] )
tau = torch. nn. parameter. Parameter( 4 * torch. ones( [ 1 , 2 , 1 , 1 ] ) )
max_ = MyMax. apply
y = max_( x, tau)
print ( "y" , y. shape)
loss = torch. sum ( y)
loss. backward( )
print ( "tau:" )
print ( tau. grad)
print ( "x:" )
print ( x. grad[ 1 , 1 , : , : ] )
