文章鏈接 2017年11月30日
視頻鏈接 (youtube)
主要的工作：
1：使用多種binary weight base進行線性組合來接近全精度的權值
2：引入多種binary activations。這個將BNNs在Image上的精度提升了將近5%
We relied on the idea of finding the best approximation of full-precision convolution using multiple binary operations, and employing multiple binary activations to allow more information passing through.
用( w , h , c i n , c o u t ) (w,h,c_{in},c_{out}) ( w , h , c i n , c o u t )
Approximate weights as a whole
使用M M M B 1 , B 2 , ⋯ , B M ∈ { − 1 , + 1 } w × h × c i n × c o u t B_1,B_2,\cdots,B_M \in \{-1, +1\}^{w\times h\times c_{in}\times c_{out}} B 1 , B 2 , ⋯ , B M ∈ { − 1 , + 1 } w × h × c i n × c o u t W ∈ R w × h × c i n × c o u t W\in \mathbb{R}^{w\times h\times c_{in}\times c_{out}} W ∈ R w × h × c i n × c o u t W ≈ α 1 B 1 + α 2 B 2 + ⋯ + α M B M W \approx \alpha_1B_1+\alpha_2B_2+\dots+\alpha_MB_M W ≈ α 1 B 1 + α 2 B 2 + ⋯ + α M B M min α , B J ( α , B ) = ∣ ∣ w − B α ∣ ∣ 2 s.t. B i j ∈ { − 1 , + 1 } (1)
\min _{\boldsymbol{\alpha}, \boldsymbol{B}} J(\boldsymbol{\alpha}, \boldsymbol{B})=
{{||\boldsymbol{w}-\boldsymbol{B}\boldsymbol{\alpha}||}^2
\text{ s.t. } \boldsymbol{B}_{i j} \in\{-1,+1\} }\tag{1}
α , B min J ( α , B ) = ∣ ∣ w − B α ∣ ∣ 2 s.t. B i j ∈ { − 1 , + 1 } ( 1 ) B = [ vec ( B 1 ) , vec ( B 2 ) , ⋯ , vec ( B M ) ] , w = vec ( W ) and α = [ α 1 , α 2 , ⋯ , α M ] T \boldsymbol{B}=\left[\operatorname{vec}\left(\boldsymbol{B}_{1}\right), \operatorname{vec}\left(\boldsymbol{B}_{2}\right), \cdots, \operatorname{vec}\left(\boldsymbol{B}_{M}\right)\right], \boldsymbol{w}=\operatorname{vec}(\boldsymbol{W}) \text { and } \boldsymbol{\alpha}=\left[\alpha_{1}, \alpha_{2}, \cdots, \alpha_{M}\right]^{\mathrm{T}} B = [ v e c ( B 1 ) , v e c ( B 2 ) , ⋯ , v e c ( B M ) ] , w = v e c ( W ) and α = [ α 1 , α 2 , ⋯ , α M ] T vec ( ⋅ ) \operatorname{vec}(\cdot) v e c ( ⋅ ) mean ( W ) \operatorname{mean}(\boldsymbol{W}) m e a n ( W ) std ( W ) \operatorname{std}(\boldsymbol{W}) s t d ( W ) W \boldsymbol{W} W B i B_i B i B i = F u i ( W ) : = sign ( W ‾ + u i std ( W ) ) , i = 1 , 2 , ⋯ , M (2)
\boldsymbol{B}_{i}=F_{u_{i}}(\boldsymbol{W}):=\operatorname{sign}\left(\overline{\boldsymbol{W}}+u_{i} \operatorname{std}(\boldsymbol{W})\right), i=1,2, \cdots, M\tag{2} B i = F u i ( W ) : = s i g n ( W + u i s t d ( W ) ) , i = 1 , 2 , ⋯ , M ( 2 ) W ‾ = W − mean ( W ) \overline{\boldsymbol{W}}=\boldsymbol{W}-\operatorname{mean}(\boldsymbol{W}) W = W − m e a n ( W ) u i u_i u i u i u_i u i u i = − 1 + ( i − 1 ) 2 M − 1 , i = 1 , 2 , ⋯ , M u_i=-1+(i-1){2 \over M-1},i=1,2,\cdots,M u i = − 1 + ( i − 1 ) M − 1 2 , i = 1 , 2 , ⋯ , M [ − std ( W ) , std ( W ) ] [-\operatorname{std}(\boldsymbol{W}),\operatorname{std}(\boldsymbol{W})] [ − s t d ( W ) , s t d ( W ) ]
一旦B i \boldsymbol{B}_i B i min α J ( α ) = ∥ w − B α ∥ 2 (3)
\min _{\boldsymbol{\alpha}} J(\boldsymbol{\alpha})=\|\boldsymbol{w}-\boldsymbol{B} \boldsymbol{\alpha}\|^{2}\tag{3}
α min J ( α ) = ∥ w − B α ∥ 2 ( 3 ) B i \boldsymbol{B}_i B i B i \boldsymbol{B}_i B i c c c A \boldsymbol{A} A O \boldsymbol{O} O Forward: B 1 , B 2 , ⋯ , B M = F u 1 ( W ) , F u 2 ( W ) , ⋯ , F u M ( W ) Solve ( 3 ) for α O = ∑ m = 1 M α m Conv ( B m , A ) Backward: ∂ c ∂ W = ∂ c ∂ O ( ∑ m = 1 M α m ∂ O ∂ B m ∂ B m ∂ W ) = sTE ∂ c ∂ O ( ∑ m = 1 M α m ∂ O ∂ B m ) = ∑ m = 1 M α m ∂ c ∂ B m
\begin{array}{l}{\text { Forward: } B_{1}, B_{2}, \cdots, B_{M}=F_{u_{1}}(W), F_{u_{2}}(W), \cdots, F_{u_{M}}(W)} \\ {\text { Solve }(3) \text { for } \alpha} \\ {\qquad \begin{aligned} O=& \sum_{m=1}^{M} \alpha_{m} \operatorname{Conv}\left(B_{m}, A\right) \\ \text { Backward: } \frac{\partial c}{\partial W} &=\frac{\partial c}{\partial O}\left(\sum_{m=1}^{M} \alpha_{m} \frac{\partial O}{\partial B_{m}} \frac{\partial B_{m}}{\partial W}\right) \stackrel{\text { sTE }}{=} \frac{\partial c}{\partial O}\left(\sum_{m=1}^{M} \alpha_{m} \frac{\partial O}{\partial B_{m}}\right)=\sum_{m=1}^{M} \alpha_{m} \frac{\partial c}{\partial B_{m}} \end{aligned}}\end{array}
Forward: B 1 , B 2 , ⋯ , B M = F u 1 ( W ) , F u 2 ( W ) , ⋯ , F u M ( W ) Solve ( 3 ) for α O = Backward: ∂ W ∂ c m = 1 ∑ M α m C o n v ( B m , A ) = ∂ O ∂ c ( m = 1 ∑ M α m ∂ B m ∂ O ∂ W ∂ B m ) = sTE ∂ O ∂ c ( m = 1 ∑ M α m ∂ B m ∂ O ) = m = 1 ∑ M α m ∂ B m ∂ c
爲了實現bitwise操作，必須將激活值也量化掉，因爲它們將作爲卷積的輸入。激活函數表示爲h ( x ) ∈ [ 0 , 1 ] h(x)\in [0,1] h ( x ) ∈ [ 0 , 1 ] h v ( x ) = clip ( x + v , 0 , 1 ) (4)
h_v(x)=\operatorname{clip}(x+v,0,1)\tag{4}
h v ( x ) = c l i p ( x + v , 0 , 1 ) ( 4 ) v v v H v ( R ) : = 2 I h v ( R ) ≥ 0.5 − 1 (5)
H_{v}(\boldsymbol{R}):=2 \mathbb{I}_{\boldsymbol{h}_{v}(\boldsymbol{R}) \geq 0.5}-1\tag{5}
H v ( R ) : = 2 I h v ( R ) ≥ 0 . 5 − 1 ( 5 ) I \mathbb{I} I Forward: A = H v ( R ) Backward: ∂ c ∂ R = ∂ c ∂ A ∘ I 0 ≤ R − v ≤ 1 (using STE)
\begin{array}{l}{\text { Forward: } A=H_{v}(\boldsymbol{R})} \\
\\
\\
{\text { Backward: } \frac{\partial c}{\partial \boldsymbol{R}}=\frac{\partial c}{\partial \boldsymbol{A}} \circ \mathbb{I}_{0 \leq \boldsymbol{R}-v \leq 1} \text { (using STE) }}\end{array}
Forward: A = H v ( R ) Backward: ∂ R ∂ c = ∂ A ∂ c ∘ I 0 ≤ R − v ≤ 1 (using STE) o \operatorname{o} o N N N R ≈ β 1 A 1 + β 2 A 2 + ⋯ + β N A N R\approx \beta_1\boldsymbol{A}_1+\beta_2\boldsymbol{A}_2+\dots+\beta_N\boldsymbol{A}_N R ≈ β 1 A 1 + β 2 A 2 + ⋯ + β N A N A 1 , A 2 , … , A N = H v 1 ( R ) , H v 2 ( R ) , … , H v N ( R ) (6)
\boldsymbol{A}_1,\boldsymbol{A}_2,\dots,\boldsymbol{A}_N=H_{v1}(\boldsymbol{R}),H_{v2}(\boldsymbol{R}),\dots,H_{vN}(\boldsymbol{R}) \tag{6}
A 1 , A 2 , … , A N = H v 1 ( R ) , H v 2 ( R ) , … , H v N ( R ) ( 6 ) β n \beta_n β n v n v_n v n Conv ( W , R ) ≈ Conv ( ∑ m = 1 M α m B m , ∑ n = 1 N β n A n ) = ∑ m = 1 M ∑ n = 1 N α m β n Conv ( B m , A n ) (7)
\operatorname{Conv}(\boldsymbol{W}, \boldsymbol{R}) \approx \operatorname{Conv}\left(\sum_{m=1}^{M} \alpha_{m} \boldsymbol{B}_{m}, \sum_{n=1}^{N} \beta_{n} \boldsymbol{A}_{n}\right)=\sum_{m=1}^{M} \sum_{n=1}^{N} \alpha_{m} \beta_{n} \operatorname{Conv}\left(\boldsymbol{B}_{m}, \boldsymbol{A}_{n}\right)\tag{7}
C o n v ( W , R ) ≈ C o n v ( m = 1 ∑ M α m B m , n = 1 ∑ N β n A n ) = m = 1 ∑ M n = 1 ∑ N α m β n C o n v ( B m , A n ) ( 7 ) M × N M\times N M × N
作者說一般的層的連接順序爲Conv → BN → Activation → Pooling \text{Conv}\rightarrow \text{BN}\rightarrow \text{Activation}\rightarrow \text{Pooling} Conv → BN → Activation → Pooling
使用Resnet作爲基網絡，圖片放縮成224*224大小。
Effect of weight approximation Comparison with the state-of-the-art
Why adding a shift parameter works?
Advantage over the fixed-point quantization scheme
看看視頻還是能更好地理解文章的想法的。
