這一期魔改Transformers主要關注對原始模型中位置信息的討論與優化，
一篇短文不是很難理解，文章要解決的痛點也非常清晰：self-attention機制在處理序列輸入時無法編碼位置信息。在原始Transformer裏是採取sin/cos函數顯示地引入位置信息，考慮的是絕對位置：P E ( pos , 2 i ) = sin ( pos 10000 0 2 i / d nodd ) P E ( pos , 2 i + 1 ) = cos ( pos 1000 0 2 i / d nodd )
\begin{aligned}
&P E(\text {pos}, 2 i)=\sin \left(\frac{\text {pos}}{100000^{2 i / d_{\text {nodd}}}}\right)\\
&P E(\text { pos }, 2 i+1)=\cos \left(\frac{\text { pos }}{10000^{2 i / d_{\text {nodd}}}}\right)
\end{aligned}
P E ( pos , 2 i ) = sin ( 1 0 0 0 0 0 2 i / d nodd pos ) P E ( pos , 2 i + 1 ) = cos ( 1 0 0 0 0 2 i / d nodd pos ) p o s pos p o s i i i i i i d m o d e l d_{model} d m o d e l 【淺談 Transformer-based 模型中的位置表示】 。
正對以上問題，這篇論文提出一種相對位置信息引入Transformer的方法。
爲了方便兩者的對比，給出原始Transformer裏self-attention的計算：z i = ∑ j = 1 n α i j ( x j W V )
z_{i}=\sum_{j=1}^{n} \alpha_{i j}\left(x_{j} W^{V}\right)
z i = j = 1 ∑ n α i j ( x j W V ) α i j = exp e i j ∑ k = 1 n exp e i k
\alpha_{i j}=\frac{\exp e_{i j}}{\sum_{k=1}^{n} \exp e_{i k}}
α i j = ∑ k = 1 n exp e i k exp e i j e i j = ( x i W Q ) ( x j W K ) T d z
e_{i j}=\frac{\left(x_{i} W^{Q}\right)\left(x_{j} W^{K}\right)^{T}}{\sqrt{d_{z}}}
e i j = d z ( x i W Q ) ( x j W K ) T
e i j e_{ij} e i j i i i j j j α i j \alpha_{ij} α i j i i i j j j z i z{i} z i i i i
輸入被看成是有向全連接圖，兩個token x i x_{i} x i x j x_{j} x j a i j V , a i j K ∈ R d a a_{i j}^{V}, a_{i j}^{K} \in \mathbb{R}^{d_{a}} a i j V , a i j K ∈ R d a k k k a i j K = w clip ( j − i , k ) K a i j V = w clip ( j − i , k ) V clip ( x , k ) = max ( − k , min ( k , x ) )
\begin{aligned}
a_{i j}^{K} &=w_{\operatorname{clip}(j-i, k)}^{K} \\
a_{i j}^{V} &=w_{\operatorname{clip}(j-i, k)}^{V} \\
\operatorname{clip}(x, k) &=\max (-k, \min (k, x))
\end{aligned}
a i j K a i j V c l i p ( x , k ) = w c l i p ( j − i , k ) K = w c l i p ( j − i , k ) V = max ( − k , min ( k , x ) ) a i j K a_{i j}^{K} a i j K a i j V a_{i j}^{V} a i j V w K = ( w − k K , … , w k K ) and w V = ( w − k V , … , w k V )
w^{K}=\left(w_{-k}^{K}, \ldots, w_{k}^{K}\right) \text { and } w^{V}=\left(w_{-k}^{V}, \ldots, w_{k}^{V}\right)
w K = ( w − k K , … , w k K ) and w V = ( w − k V , … , w k V ) 滑動窗口 有點像：w K = ( w − 4 K , … , w 4 K )
w^{K}=\left(w_{-4}^{K}, \ldots, w_{4}^{K}\right) w K = ( w − 4 K , … , w 4 K ) w V = ( w − 4 V , … , w 4 V ) w^{V}=\left(w_{-4}^{V}, \ldots, w_{4}^{V}\right)
w V = ( w − 4 V , … , w 4 V )
理解上面RPR之後，就可以對原始self-attention進行改寫，將相對位置信息融入進去：z i = ∑ j = 1 n α i j ( x j W V + a i j V )
z_{i}=\sum_{j=1}^{n} \alpha_{i j}\left(x_{j} W^{V}+a_{i j}^{V}\right)
z i = j = 1 ∑ n α i j ( x j W V + a i j V ) e i j = x i W Q ( x j W K + a i j K ) T d z
e_{i j}=\frac{x_{i} W^{Q}\left(x_{j} W^{K}+a_{i j}^{K}\right)^{T}}{\sqrt{d_{z}}}
e i j = d z x i W Q ( x j W K + a i j K ) T
不管是transformer原文的絕對位置編碼還是上文引入的相對位置編碼，都屬於sequential information 。作者從Hewitt和Manning發表的論文 中得到啓發：句子的潛在結構可以通過結構深度和距離來捕獲，於是他們提出了absolute structural position來編碼元素在句法樹種深度，relative structural position來編碼元素之間的距離。
Absolute Structural Position ：把句子的主要動詞(如上圖"held")作爲origin，然後通過計算句法樹中目標詞語origin之間的距離作爲絕對結構位置信息：abs str u ( x i ) = distance tree ( x i , origin )
\operatorname{abs}_{\operatorname{str} u}\left(x_{i}\right)=\operatorname{distance}_{\operatorname{tree}}\left(x_{i}, \text {origin}\right)
a b s s t r u ( x i ) = d i s t a n c e t r e e ( x i , origin ) Relative Structural Position ：相對位置信息考慮的單詞對之間的關係，
如果單詞x i x_{i} x i x j x_{j} x j r e l s t r u ( x i , x j ) = a b s s t r u ( x i ) − a b s s t r u ( x j )
r e l_{s t r u}\left(x_{i}, x_{j}\right)=a b s_{s t r u}\left(x_{i}\right)-a b s_{s t r u}\left(x_{j}\right)
r e l s t r u ( x i , x j ) = a b s s t r u ( x i ) − a b s s t r u ( x j )
如果在不同的邊上，相對位置爲絕對位置之和乘上一個係數f s t r u ( x ) f_{s t r u}(x) f s t r u ( x ) r e l s t r u ( x i , x j ) = f s t r u ( i − j ) ∗ ( a b s s t r u ( x i ) + a b s s t r u ( x j ) )
r e l_{s t r u}\left(x_{i}, x_{j}\right)=f_{s t r u}(i-j) *\left(a b s_{s t r u}\left(x_{i}\right)+a b s_{s t r u}\left(x_{j}\right)\right)
r e l s t r u ( x i , x j ) = f s t r u ( i − j ) ∗ ( a b s s t r u ( x i ) + a b s s t r u ( x j ) ) f s t r u ( x ) = { 1 x > 0 0 x = 0 − 1 x < 0
f_{s t r u}(x)=\left\{\begin{array}{ll}
{1} & {x>0} \\
{0} & {x=0} \\
{-1} & {x<0}
\end{array}\right.
f s t r u ( x ) = ⎩ ⎨ ⎧ 1 0 − 1 x > 0 x = 0 x < 0
接下去就是將結構位置信息整合進SAN，
對於絕對位置，通過一個非線性函數將序列位置和結構位置融合得到位置表示：asb ( x i ) = f a b s ( A B S P E ( a b s s e q ) , A B S P E ( a b s s t r u ) ) \operatorname{asb}\left(x_{i}\right)=f_{a b s}\left(\mathrm{ABSPE}\left(a b s_{s e q}\right)\right.,\left.\mathrm{ABSPE}\left(a b s_{s t r u}\right)\right) a s b ( x i ) = f a b s ( A B S P E ( a b s s e q ) , A B S P E ( a b s s t r u ) )
對於相對位置，採用同上一篇論文一致的方法
一點疑問 …《A Structural Probe for Finding Syntax in Word Representations》，也就是作者在文章裏說給了他們motivation的論文，大概說的是通過structural probe檢測像BERT這樣的模型學到的context represetation是否包含了syntax tree信息，結論是可以學到句法樹的結構信息。那這樣爲啥還要加structural position representation呢？
針對命名實體識別任務的一個工作，雖然Transformer系在NLP很多領域都取得了非常大的進步，但是在NER任務上表現不佳。作者分析了注意力機制與特定的NER任務，發現原始實現中的位置信息編碼 、注意力稀疏性 方面不太適用於NER。
距離性 P E t , 2 i = sin ( t 1000 0 2 i / d ) P E t , 2 i + 1 = cos ( t 1000 0 2 / d )
\begin{aligned}
P E_{t, 2 i}=& \sin \left(\frac{t}{10000^{2 i / d}}\right) \\
P E_{t, 2 i+1} &=\cos \left(\frac{t}{10000^{2 / d}}\right)
\end{aligned} P E t , 2 i = P E t , 2 i + 1 sin ( 1 0 0 0 0 2 i / d t ) = cos ( 1 0 0 0 0 2 / d t ) P E t T P E t + k = ∑ j = 0 d 2 − 1 [ sin ( c j t ) sin ( c j ( t + k ) ) + cos ( c j t ) cos ( c j ( t + k ) ) ] = ∑ j = 0 d 2 − 1 cos ( c j ( t − ( t + k ) ) ) = ∑ j = 0 d 2 − 1 cos ( c j k )
\begin{aligned}
P E_{t}^{T} P E_{t+k} &=\sum_{j=0}^{\frac{d}{2}-1}\left[\sin \left(c_{j} t\right) \sin \left(c_{j}(t+k)\right)+\cos \left(c_{j} t\right) \cos \left(c_{j}(t+k)\right)\right] \\
&=\sum_{j=0}^{\frac{d}{2}-1} \cos \left(c_{j}(t-(t+k))\right) \\
&=\sum_{j=0}^{\frac{d}{2}-1} \cos \left(c_{j} k\right)
\end{aligned}
P E t T P E t + k = j = 0 ∑ 2 d − 1 [ sin ( c j t ) sin ( c j ( t + k ) ) + cos ( c j t ) cos ( c j ( t + k ) ) ] = j = 0 ∑ 2 d − 1 cos ( c j ( t − ( t + k ) ) ) = j = 0 ∑ 2 d − 1 cos ( c j k ) k k k 距離敏感 ），但是進一步研究可以發現，當position embedding被映射到自注意力的鍵值對時，其就會失去距離敏感性，如下圖，最上面的曲線表示P E t T P E t + k P E_{t}^{T} P E_{t+k} P E t T P E t + k P E t T W P E t + k P E_{t}^{T}W PE_{t+k} P E t T W P E t + k
論文中說P E t T W P E t + k P E_{t}^{T}W PE_{t+k} P E t T W P E t + k W W W
方向性 j = t − k j=t-k j = t − k P E t T P E t + k = P E j T P E j + k = P E t − k T P E t
P E_{t}^{T} P E_{t+k}=P E_{j}^{T} P E_{j+k}=P E_{t-k}^{T} P E_{t}
P E t T P E t + k = P E j T P E j + k = P E t − k T P E t 方向不敏感 的
由上分析可知該種嵌入方式並不能反映方向性和距離性，但是對於NER任務而言距離和方向都是尤爲重要的。爲此本文對注意力分數計算進行改進，將絕對位置改成相對位置：Q , K , V = H W q , H d k , H W v Q, K, V=H W_{q}, H_{d_{k}}, H W_{v}
Q , K , V = H W q , H d k , H W v R t − j = [ … , sin ( t − j 1000 0 2 i d k ) cos ( t − j 1000 0 2 i d k ) , … ] T
{R_{t-j}=\left[\ldots, \sin \left(\frac{t-j}{10000^{2 i} d_{k}}\right) \cos \left(\frac{t-j}{10000^{2 i} d_{k}}\right), \ldots\right]^{T}}
R t − j = [ … , sin ( 1 0 0 0 0 2 i d k t − j ) cos ( 1 0 0 0 0 2 i d k t − j ) , … ] T A t j r e l = Q t T K j + Q t T R t − j u T K j + v T R t − j
{A_{t j}^{r e l}=Q_{t}^{T} K_{j}+Q_{t}^{T} R_{t-j} u^{T} K_{j}+v^{T} R_{t-j}}
A t j r e l = Q t T K j + Q t T R t − j u T K j + v T R t − j A t t n ( Q , K , V ) = softmax ( A r e l ) V
{A t tn(Q, K, V)=\operatorname{softmax}\left(A^{r e l}\right) V}
A t t n ( Q , K , V ) = s o f t m a x ( A r e l ) V
注意到修改後的Attention計算最後沒有用到scaling係數 d \sqrt{d} d
這篇論文感覺主要重點在工程意義，在NER任務達到最好的效果；針對NER任務做了專門的修改，除了上面的相對位置信息和係數注意力，由於NER數據集相對較小，也減少了模型的可學習參數避免過擬合。
比較有意思的工作，關注的點也是在序列建模的位置信息編碼。先前的方法通過引入額外的位置編碼，在embedding層將詞向量和位置向量通過加性編碼融合，f ( j , p o s ) = f w e ( j ) + f p e ( p o s )
f(j, p o s)=f_{w e}(j)+f_{p e}(p o s)
f ( j , p o s ) = f w e ( j ) + f p e ( p o s ) order relationship（例如鄰接或優先關係），作者將此稱爲the position independece problem。
針對該問題提出了一種新的位置編碼方式，將獨立的詞向量替換成自變量爲位置的函數，於是單詞表示會隨着位置的變化而平滑地移動，可以更好地建模單詞的絕對位置和順序信息。f ( j , pos ) = g j ( pos ) ∈ C D
f(j, \text { pos })=g_{j}(\text { pos }) \in \mathbb{C}^{D}
f ( j , pos ) = g j ( pos ) ∈ C D f ( j , pos ) f(j, \text { pos }) f ( j , pos ) j j j p o s pos p o s D D D g w e ( ⋅ ) : N → ( F ) D g_{w e}(\cdot):\mathbb{N} \rightarrow(\mathcal{F})^{D} g w e ( ⋅ ) : N → ( F ) D [ g j , 1 ( p o s ) , g j , 2 ( p o s ) , … , g j , D ( p o s ) ] ∈ C D
\left[g_{j, 1}(\mathrm{pos}), g_{j, 2}(\mathrm{pos}), \ldots, g_{j, D}(\mathrm{pos})\right] \in \mathbb{C}^{D}
[ g j , 1 ( p o s ) , g j , 2 ( p o s ) , … , g j , D ( p o s ) ] ∈ C D
Property 1. Position-free offset transformation 對於任意位置pos和n > 1 n>1 n > 1 n ( ⋅ ) = _{n}(\cdot)= n ( ⋅ ) = ( n , ⋅ ) (n, \cdot) ( n , ⋅ ) g ( p o s + n ) = Transform n ( g ( p o s ) )
g(\mathrm{pos}+n)=\text { Transform }_{n}(g(\mathrm{pos}))
g ( p o s + n ) = Transform n ( g ( p o s ) )
Property 2. Boundedness 函數應該是有界的，∃ δ ∈ R + , ∀ pos ∈ N , ∣ g ( pos ) ∣ ≤ δ
\exists \delta \in\mathbb{R}^{+}, \forall \text { pos } \in \mathbb{N},|g(\operatorname{pos})| \leq \delta
∃ δ ∈ R + , ∀ pos ∈ N , ∣ g ( p o s ) ∣ ≤ δ g ( p o s ) = z 2 z 1 p o s for z 1 , z 2 ∈ C with ∣ z 1 ∣ ≤ 1
g(p o s)=z_{2} z_{1}^{p o s} \text { for } z_{1}, z_{2} \in \mathbb{C} \text { with }\left|z_{1}\right| \leq 1
g ( p o s ) = z 2 z 1 p o s for z 1 , z 2 ∈ C with ∣ z 1 ∣ ≤ 1
貼一下論文給的證明：（看不看無所謂，能用就行haha）g g g n 1 , n 2 ∈ N n_{1}, n_{2} \in \mathbb{N} n 1 , n 2 ∈ N w ( n 1 ) w ( n 2 ) g ( pos ) = w ( n 2 ) g ( pos + n 1 ) = g ( pos + n 1 + n 2 ) = Transform n 1 + n 2 ( g ( pos ) ) = w ( n 1 + n 2 ) g ( pos )
\begin{aligned}
w\left(n_{1}\right) w\left(n_{2}\right) g(\text { pos }) &=w\left(n_{2}\right) g\left(\text { pos }+n_{1}\right)=g\left(\text { pos }+n_{1}+n_{2}\right) \\
&=\text { Transform }_{n_{1}+n_{2}}(g(\text { pos }))=w\left(n_{1}+n_{2}\right) g(\text { pos })
\end{aligned}
w ( n 1 ) w ( n 2 ) g ( pos ) = w ( n 2 ) g ( pos + n 1 ) = g ( pos + n 1 + n 2 ) = Transform n 1 + n 2 ( g ( pos ) ) = w ( n 1 + n 2 ) g ( pos ) w ( n 1 + n 2 ) = w ( n 1 ) w ( n 2 ) w\left(n_{1}+n_{2}\right)=w\left(n_{1}\right) w\left(n_{2}\right) w ( n 1 + n 2 ) = w ( n 1 ) w ( n 2 ) w ( 1 ) = z 1 w(1)=z_{1} w ( 1 ) = z 1 g ( 0 ) = z 2 g(0)=z_{2} g ( 0 ) = z 2 n 1 , n 2 ∈ N n_{1}, n_{2} \in \mathbb{N} n 1 , n 2 ∈ N w ( n ) = ( w ( 1 ) ) n = z 1 n w(n)=(w(1))^{n}=z_{1}^{n} w ( n ) = ( w ( 1 ) ) n = z 1 n g ( pos + n ) = w ( n ) g ( pos ) = z 1 n g ( pos )
g(\text { pos }+n)=w(n) g(\text { pos })=z_{1}^{n} g(\text { pos })
g ( pos + n ) = w ( n ) g ( pos ) = z 1 n g ( pos ) ≥ 1 \geq 1 ≥ 1 g ( pos ) = g ( 1 + pos − 1 ) = w ( pos ) g ( 0 ) = z 1 pos z 2 = z 2 z 1 pos g(\text { pos })=g(1+\text { pos }-1)=w(\text { pos }) g(0)=z_{1}^{\text {pos }} z_{2}=z_{2} z_{1}^{\text {pos }} g ( pos ) = g ( 1 + pos − 1 ) = w ( pos ) g ( 0 ) = z 1 pos z 2 = z 2 z 1 pos = 1 =1 = 1 g ( 0 ) = z 2 = z 2 z 1 0
g(0)=z_{2}=z_{2} z_{1}^{0}
g ( 0 ) = z 2 = z 2 z 1 0 g ( pos ) = z 2 z 1 pos
g(\text { pos })=z_{2} z_{1}^{\text {pos }}
g ( pos ) = z 2 z 1 pos ∣ z 1 ∣ > 1 \left|z_{1}\right|>1 ∣ z 1 ∣ > 1 g ( pos ) g(\text { pos }) g ( pos ) ∣ z 1 ∣ ≤ 1 \left|z_{1}\right| \leq 1 ∣ z 1 ∣ ≤ 1 ∣ g ( pos ) ∣ ≤ ∣ z 2 z 1 pos ∣ ≤ ∣ z 2 ∣ ∣ z 1 pos ∣ ≤ ∣ z 2 ∣
| g(\text { pos })|\leq| z_{2} z_{1}^{\text {pos }}|\leq| z_{2}|| z_{1}^{\text {pos }}|\leq| z_{2} |
∣ g ( pos ) ∣ ≤ ∣ z 2 z 1 pos ∣ ≤ ∣ z 2 ∣ ∣ z 1 pos ∣ ≤ ∣ z 2 ∣ w ( n ) = z 1 n w(n)=z_{1}^{n} w ( n ) = z 1 n n ( pos ) = w ( n ) _{n}(\text { pos })=w(n) n ( pos ) = w ( n ) g ( pos + n ) = z 2 z 1 p s + n = z 2 z 1 p o s z 1 n = g ( pos ) z 1 n = Transform n ( g ( pos ) ) g(\text { pos }+n)=z_{2} z_{1}^{\mathrm{ps}+n}=z_{2} z_{1}^{\mathrm{pos}} z_{1}^{n}=g(\text { pos }) z_{1}^{n}=\text{Transform}_{n}(g(\text { pos })) g ( pos + n ) = z 2 z 1 p s + n = z 2 z 1 p o s z 1 n = g ( pos ) z 1 n = Transform n ( g ( pos ) )
根據歐拉公式，可以將上式轉化爲，g ( pos ) = z 2 z 1 pos = r 2 e i θ 2 ( r 1 e i θ 1 ) pos = r 2 r 1 pos e i ( θ 2 + θ 1 pos ) subject to ∣ r 1 ∣ ≤ 1
g(\text { pos })=z_{2} z_{1}^{\text {pos }}=r_{2} e^{i \theta_{2}}\left(r_{1} e^{i \theta_{1}}\right)^{\text {pos }}=r_{2} r_{1}^{\text {pos }} e^{i\left(\theta_{2}+\theta_{1} \text { pos }\right)} \text { subject to }\left|r_{1}\right| \leq 1
g ( pos ) = z 2 z 1 pos = r 2 e i θ 2 ( r 1 e i θ 1 ) pos = r 2 r 1 pos e i ( θ 2 + θ 1 pos ) subject to ∣ r 1 ∣ ≤ 1 ∣ r 1 ∣ ≤ 1 \left|r_{1}\right| \leq 1 ∣ r 1 ∣ ≤ 1 r 1 = 1 r_{1}=1 r 1 = 1 g ( pos ) = r e i ( ω pos + θ )
g(\text { pos })=r e^{i(\omega \text { pos }+\theta)}
g ( pos ) = r e i ( ω pos + θ ) f ( j , pos ) = g j ( pos ) = r j e i ( ω j pos + θ j )
f(j, \text { pos })=\boldsymbol{g}_{j}(\text { pos })=\boldsymbol{r}_{j} e^{i\left(\boldsymbol{\omega}_{j} \text { pos }+\boldsymbol{\theta}_{j}\right)}
f ( j , pos ) = g j ( pos ) = r j e i ( ω j pos + θ j ) r j r_{j} r j w j w_{j} w j θ j \theta_{j} θ j
振幅r j r_{j} r j
角頻率w j w_{j} w j
要用的話，計算量以及參數量是不是會很大呀
好像也沒跟其他relative position的模型比較
