这一期魔改Transformers主要关注对原始模型中位置信息的讨论与优化，
一篇短文不是很难理解，文章要解决的痛点也非常清晰：self-attention机制在处理序列输入时无法编码位置信息。在原始Transformer里是采取sin/cos函数显示地引入位置信息，考虑的是绝对位置：P E ( pos , 2 i ) = sin ( pos 10000 0 2 i / d nodd ) P E ( pos , 2 i + 1 ) = cos ( pos 1000 0 2 i / d nodd )
\begin{aligned}
&P E(\text {pos}, 2 i)=\sin \left(\frac{\text {pos}}{100000^{2 i / d_{\text {nodd}}}}\right)\\
&P E(\text { pos }, 2 i+1)=\cos \left(\frac{\text { pos }}{10000^{2 i / d_{\text {nodd}}}}\right)
\end{aligned}
P E ( pos , 2 i ) = sin ( 1 0 0 0 0 0 2 i / d nodd pos ) P E ( pos , 2 i + 1 ) = cos ( 1 0 0 0 0 2 i / d nodd pos ) p o s pos p o s i i i i i i d m o d e l d_{model} d m o d e l 【浅谈 Transformer-based 模型中的位置表示】 。
正对以上问题，这篇论文提出一种相对位置信息引入Transformer的方法。
为了方便两者的对比，给出原始Transformer里self-attention的计算：z i = ∑ j = 1 n α i j ( x j W V )
z_{i}=\sum_{j=1}^{n} \alpha_{i j}\left(x_{j} W^{V}\right)
z i = j = 1 ∑ n α i j ( x j W V ) α i j = exp e i j ∑ k = 1 n exp e i k
\alpha_{i j}=\frac{\exp e_{i j}}{\sum_{k=1}^{n} \exp e_{i k}}
α i j = ∑ k = 1 n exp e i k exp e i j e i j = ( x i W Q ) ( x j W K ) T d z
e_{i j}=\frac{\left(x_{i} W^{Q}\right)\left(x_{j} W^{K}\right)^{T}}{\sqrt{d_{z}}}
e i j = d z ( x i W Q ) ( x j W K ) T
e i j e_{ij} e i j i i i j j j α i j \alpha_{ij} α i j i i i j j j z i z{i} z i i i i
输入被看成是有向全连接图，两个token x i x_{i} x i x j x_{j} x j a i j V , a i j K ∈ R d a a_{i j}^{V}, a_{i j}^{K} \in \mathbb{R}^{d_{a}} a i j V , a i j K ∈ R d a k k k a i j K = w clip ( j − i , k ) K a i j V = w clip ( j − i , k ) V clip ( x , k ) = max ( − k , min ( k , x ) )
\begin{aligned}
a_{i j}^{K} &=w_{\operatorname{clip}(j-i, k)}^{K} \\
a_{i j}^{V} &=w_{\operatorname{clip}(j-i, k)}^{V} \\
\operatorname{clip}(x, k) &=\max (-k, \min (k, x))
\end{aligned}
a i j K a i j V c l i p ( x , k ) = w c l i p ( j − i , k ) K = w c l i p ( j − i , k ) V = max ( − k , min ( k , x ) ) a i j K a_{i j}^{K} a i j K a i j V a_{i j}^{V} a i j V w K = ( w − k K , … , w k K ) and w V = ( w − k V , … , w k V )
w^{K}=\left(w_{-k}^{K}, \ldots, w_{k}^{K}\right) \text { and } w^{V}=\left(w_{-k}^{V}, \ldots, w_{k}^{V}\right)
w K = ( w − k K , … , w k K ) and w V = ( w − k V , … , w k V ) 滑动窗口 有点像：w K = ( w − 4 K , … , w 4 K )
w^{K}=\left(w_{-4}^{K}, \ldots, w_{4}^{K}\right) w K = ( w − 4 K , … , w 4 K ) w V = ( w − 4 V , … , w 4 V ) w^{V}=\left(w_{-4}^{V}, \ldots, w_{4}^{V}\right)
w V = ( w − 4 V , … , w 4 V )
理解上面RPR之后，就可以对原始self-attention进行改写，将相对位置信息融入进去：z i = ∑ j = 1 n α i j ( x j W V + a i j V )
z_{i}=\sum_{j=1}^{n} \alpha_{i j}\left(x_{j} W^{V}+a_{i j}^{V}\right)
z i = j = 1 ∑ n α i j ( x j W V + a i j V ) e i j = x i W Q ( x j W K + a i j K ) T d z
e_{i j}=\frac{x_{i} W^{Q}\left(x_{j} W^{K}+a_{i j}^{K}\right)^{T}}{\sqrt{d_{z}}}
e i j = d z x i W Q ( x j W K + a i j K ) T
不管是transformer原文的绝对位置编码还是上文引入的相对位置编码，都属于sequential information 。作者从Hewitt和Manning发表的论文 中得到启发：句子的潜在结构可以通过结构深度和距离来捕获，于是他们提出了absolute structural position来编码元素在句法树种深度，relative structural position来编码元素之间的距离。
Absolute Structural Position ：把句子的主要动词(如上图"held")作为origin，然后通过计算句法树中目标词语origin之间的距离作为绝对结构位置信息：abs str u ( x i ) = distance tree ( x i , origin )
\operatorname{abs}_{\operatorname{str} u}\left(x_{i}\right)=\operatorname{distance}_{\operatorname{tree}}\left(x_{i}, \text {origin}\right)
a b s s t r u ( x i ) = d i s t a n c e t r e e ( x i , origin ) Relative Structural Position ：相对位置信息考虑的单词对之间的关系，
如果单词x i x_{i} x i x j x_{j} x j r e l s t r u ( x i , x j ) = a b s s t r u ( x i ) − a b s s t r u ( x j )
r e l_{s t r u}\left(x_{i}, x_{j}\right)=a b s_{s t r u}\left(x_{i}\right)-a b s_{s t r u}\left(x_{j}\right)
r e l s t r u ( x i , x j ) = a b s s t r u ( x i ) − a b s s t r u ( x j )
如果在不同的边上，相对位置为绝对位置之和乘上一个系数f s t r u ( x ) f_{s t r u}(x) f s t r u ( x ) r e l s t r u ( x i , x j ) = f s t r u ( i − j ) ∗ ( a b s s t r u ( x i ) + a b s s t r u ( x j ) )
r e l_{s t r u}\left(x_{i}, x_{j}\right)=f_{s t r u}(i-j) *\left(a b s_{s t r u}\left(x_{i}\right)+a b s_{s t r u}\left(x_{j}\right)\right)
r e l s t r u ( x i , x j ) = f s t r u ( i − j ) ∗ ( a b s s t r u ( x i ) + a b s s t r u ( x j ) ) f s t r u ( x ) = { 1 x > 0 0 x = 0 − 1 x < 0
f_{s t r u}(x)=\left\{\begin{array}{ll}
{1} & {x>0} \\
{0} & {x=0} \\
{-1} & {x<0}
\end{array}\right.
f s t r u ( x ) = ⎩ ⎨ ⎧ 1 0 − 1 x > 0 x = 0 x < 0
接下去就是将结构位置信息整合进SAN，
对于绝对位置，通过一个非线性函数将序列位置和结构位置融合得到位置表示：asb ( x i ) = f a b s ( A B S P E ( a b s s e q ) , A B S P E ( a b s s t r u ) ) \operatorname{asb}\left(x_{i}\right)=f_{a b s}\left(\mathrm{ABSPE}\left(a b s_{s e q}\right)\right.,\left.\mathrm{ABSPE}\left(a b s_{s t r u}\right)\right) a s b ( x i ) = f a b s ( A B S P E ( a b s s e q ) , A B S P E ( a b s s t r u ) )
对于相对位置，采用同上一篇论文一致的方法
一点疑问 …《A Structural Probe for Finding Syntax in Word Representations》，也就是作者在文章里说给了他们motivation的论文，大概说的是通过structural probe检测像BERT这样的模型学到的context represetation是否包含了syntax tree信息，结论是可以学到句法树的结构信息。那这样为啥还要加structural position representation呢？
针对命名实体识别任务的一个工作，虽然Transformer系在NLP很多领域都取得了非常大的进步，但是在NER任务上表现不佳。作者分析了注意力机制与特定的NER任务，发现原始实现中的位置信息编码 、注意力稀疏性 方面不太适用于NER。
距离性 P E t , 2 i = sin ( t 1000 0 2 i / d ) P E t , 2 i + 1 = cos ( t 1000 0 2 / d )
\begin{aligned}
P E_{t, 2 i}=& \sin \left(\frac{t}{10000^{2 i / d}}\right) \\
P E_{t, 2 i+1} &=\cos \left(\frac{t}{10000^{2 / d}}\right)
\end{aligned} P E t , 2 i = P E t , 2 i + 1 sin ( 1 0 0 0 0 2 i / d t ) = cos ( 1 0 0 0 0 2 / d t ) P E t T P E t + k = ∑ j = 0 d 2 − 1 [ sin ( c j t ) sin ( c j ( t + k ) ) + cos ( c j t ) cos ( c j ( t + k ) ) ] = ∑ j = 0 d 2 − 1 cos ( c j ( t − ( t + k ) ) ) = ∑ j = 0 d 2 − 1 cos ( c j k )
\begin{aligned}
P E_{t}^{T} P E_{t+k} &=\sum_{j=0}^{\frac{d}{2}-1}\left[\sin \left(c_{j} t\right) \sin \left(c_{j}(t+k)\right)+\cos \left(c_{j} t\right) \cos \left(c_{j}(t+k)\right)\right] \\
&=\sum_{j=0}^{\frac{d}{2}-1} \cos \left(c_{j}(t-(t+k))\right) \\
&=\sum_{j=0}^{\frac{d}{2}-1} \cos \left(c_{j} k\right)
\end{aligned}
P E t T P E t + k = j = 0 ∑ 2 d − 1 [ sin ( c j t ) sin ( c j ( t + k ) ) + cos ( c j t ) cos ( c j ( t + k ) ) ] = j = 0 ∑ 2 d − 1 cos ( c j ( t − ( t + k ) ) ) = j = 0 ∑ 2 d − 1 cos ( c j k ) k k k 距离敏感 ），但是进一步研究可以发现，当position embedding被映射到自注意力的键值对时，其就会失去距离敏感性，如下图，最上面的曲线表示P E t T P E t + k P E_{t}^{T} P E_{t+k} P E t T P E t + k P E t T W P E t + k P E_{t}^{T}W PE_{t+k} P E t T W P E t + k
论文中说P E t T W P E t + k P E_{t}^{T}W PE_{t+k} P E t T W P E t + k W W W
方向性 j = t − k j=t-k j = t − k P E t T P E t + k = P E j T P E j + k = P E t − k T P E t
P E_{t}^{T} P E_{t+k}=P E_{j}^{T} P E_{j+k}=P E_{t-k}^{T} P E_{t}
P E t T P E t + k = P E j T P E j + k = P E t − k T P E t 方向不敏感 的
由上分析可知该种嵌入方式并不能反映方向性和距离性，但是对于NER任务而言距离和方向都是尤为重要的。为此本文对注意力分数计算进行改进，将绝对位置改成相对位置：Q , K , V = H W q , H d k , H W v Q, K, V=H W_{q}, H_{d_{k}}, H W_{v}
Q , K , V = H W q , H d k , H W v R t − j = [ … , sin ( t − j 1000 0 2 i d k ) cos ( t − j 1000 0 2 i d k ) , … ] T
{R_{t-j}=\left[\ldots, \sin \left(\frac{t-j}{10000^{2 i} d_{k}}\right) \cos \left(\frac{t-j}{10000^{2 i} d_{k}}\right), \ldots\right]^{T}}
R t − j = [ … , sin ( 1 0 0 0 0 2 i d k t − j ) cos ( 1 0 0 0 0 2 i d k t − j ) , … ] T A t j r e l = Q t T K j + Q t T R t − j u T K j + v T R t − j
{A_{t j}^{r e l}=Q_{t}^{T} K_{j}+Q_{t}^{T} R_{t-j} u^{T} K_{j}+v^{T} R_{t-j}}
A t j r e l = Q t T K j + Q t T R t − j u T K j + v T R t − j A t t n ( Q , K , V ) = softmax ( A r e l ) V
{A t tn(Q, K, V)=\operatorname{softmax}\left(A^{r e l}\right) V}
A t t n ( Q , K , V ) = s o f t m a x ( A r e l ) V
注意到修改后的Attention计算最后没有用到scaling系数 d \sqrt{d} d
这篇论文感觉主要重点在工程意义，在NER任务达到最好的效果；针对NER任务做了专门的修改，除了上面的相对位置信息和系数注意力，由于NER数据集相对较小，也减少了模型的可学习参数避免过拟合。
比较有意思的工作，关注的点也是在序列建模的位置信息编码。先前的方法通过引入额外的位置编码，在embedding层将词向量和位置向量通过加性编码融合，f ( j , p o s ) = f w e ( j ) + f p e ( p o s )
f(j, p o s)=f_{w e}(j)+f_{p e}(p o s)
f ( j , p o s ) = f w e ( j ) + f p e ( p o s ) order relationship（例如邻接或优先关系），作者将此称为the position independece problem。
针对该问题提出了一种新的位置编码方式，将独立的词向量替换成自变量为位置的函数，于是单词表示会随着位置的变化而平滑地移动，可以更好地建模单词的绝对位置和顺序信息。f ( j , pos ) = g j ( pos ) ∈ C D
f(j, \text { pos })=g_{j}(\text { pos }) \in \mathbb{C}^{D}
f ( j , pos ) = g j ( pos ) ∈ C D f ( j , pos ) f(j, \text { pos }) f ( j , pos ) j j j p o s pos p o s D D D g w e ( ⋅ ) : N → ( F ) D g_{w e}(\cdot):\mathbb{N} \rightarrow(\mathcal{F})^{D} g w e ( ⋅ ) : N → ( F ) D [ g j , 1 ( p o s ) , g j , 2 ( p o s ) , … , g j , D ( p o s ) ] ∈ C D
\left[g_{j, 1}(\mathrm{pos}), g_{j, 2}(\mathrm{pos}), \ldots, g_{j, D}(\mathrm{pos})\right] \in \mathbb{C}^{D}
[ g j , 1 ( p o s ) , g j , 2 ( p o s ) , … , g j , D ( p o s ) ] ∈ C D
Property 1. Position-free offset transformation 对于任意位置pos和n > 1 n>1 n > 1 n ( ⋅ ) = _{n}(\cdot)= n ( ⋅ ) = ( n , ⋅ ) (n, \cdot) ( n , ⋅ ) g ( p o s + n ) = Transform n ( g ( p o s ) )
g(\mathrm{pos}+n)=\text { Transform }_{n}(g(\mathrm{pos}))
g ( p o s + n ) = Transform n ( g ( p o s ) )
Property 2. Boundedness 函数应该是有界的，∃ δ ∈ R + , ∀ pos ∈ N , ∣ g ( pos ) ∣ ≤ δ
\exists \delta \in\mathbb{R}^{+}, \forall \text { pos } \in \mathbb{N},|g(\operatorname{pos})| \leq \delta
∃ δ ∈ R + , ∀ pos ∈ N , ∣ g ( p o s ) ∣ ≤ δ g ( p o s ) = z 2 z 1 p o s for z 1 , z 2 ∈ C with ∣ z 1 ∣ ≤ 1
g(p o s)=z_{2} z_{1}^{p o s} \text { for } z_{1}, z_{2} \in \mathbb{C} \text { with }\left|z_{1}\right| \leq 1
g ( p o s ) = z 2 z 1 p o s for z 1 , z 2 ∈ C with ∣ z 1 ∣ ≤ 1
贴一下论文给的证明：（看不看无所谓，能用就行haha）g g g n 1 , n 2 ∈ N n_{1}, n_{2} \in \mathbb{N} n 1 , n 2 ∈ N w ( n 1 ) w ( n 2 ) g ( pos ) = w ( n 2 ) g ( pos + n 1 ) = g ( pos + n 1 + n 2 ) = Transform n 1 + n 2 ( g ( pos ) ) = w ( n 1 + n 2 ) g ( pos )
\begin{aligned}
w\left(n_{1}\right) w\left(n_{2}\right) g(\text { pos }) &=w\left(n_{2}\right) g\left(\text { pos }+n_{1}\right)=g\left(\text { pos }+n_{1}+n_{2}\right) \\
&=\text { Transform }_{n_{1}+n_{2}}(g(\text { pos }))=w\left(n_{1}+n_{2}\right) g(\text { pos })
\end{aligned}
w ( n 1 ) w ( n 2 ) g ( pos ) = w ( n 2 ) g ( pos + n 1 ) = g ( pos + n 1 + n 2 ) = Transform n 1 + n 2 ( g ( pos ) ) = w ( n 1 + n 2 ) g ( pos ) w ( n 1 + n 2 ) = w ( n 1 ) w ( n 2 ) w\left(n_{1}+n_{2}\right)=w\left(n_{1}\right) w\left(n_{2}\right) w ( n 1 + n 2 ) = w ( n 1 ) w ( n 2 ) w ( 1 ) = z 1 w(1)=z_{1} w ( 1 ) = z 1 g ( 0 ) = z 2 g(0)=z_{2} g ( 0 ) = z 2 n 1 , n 2 ∈ N n_{1}, n_{2} \in \mathbb{N} n 1 , n 2 ∈ N w ( n ) = ( w ( 1 ) ) n = z 1 n w(n)=(w(1))^{n}=z_{1}^{n} w ( n ) = ( w ( 1 ) ) n = z 1 n g ( pos + n ) = w ( n ) g ( pos ) = z 1 n g ( pos )
g(\text { pos }+n)=w(n) g(\text { pos })=z_{1}^{n} g(\text { pos })
g ( pos + n ) = w ( n ) g ( pos ) = z 1 n g ( pos ) ≥ 1 \geq 1 ≥ 1 g ( pos ) = g ( 1 + pos − 1 ) = w ( pos ) g ( 0 ) = z 1 pos z 2 = z 2 z 1 pos g(\text { pos })=g(1+\text { pos }-1)=w(\text { pos }) g(0)=z_{1}^{\text {pos }} z_{2}=z_{2} z_{1}^{\text {pos }} g ( pos ) = g ( 1 + pos − 1 ) = w ( pos ) g ( 0 ) = z 1 pos z 2 = z 2 z 1 pos = 1 =1 = 1 g ( 0 ) = z 2 = z 2 z 1 0
g(0)=z_{2}=z_{2} z_{1}^{0}
g ( 0 ) = z 2 = z 2 z 1 0 g ( pos ) = z 2 z 1 pos
g(\text { pos })=z_{2} z_{1}^{\text {pos }}
g ( pos ) = z 2 z 1 pos ∣ z 1 ∣ > 1 \left|z_{1}\right|>1 ∣ z 1 ∣ > 1 g ( pos ) g(\text { pos }) g ( pos ) ∣ z 1 ∣ ≤ 1 \left|z_{1}\right| \leq 1 ∣ z 1 ∣ ≤ 1 ∣ g ( pos ) ∣ ≤ ∣ z 2 z 1 pos ∣ ≤ ∣ z 2 ∣ ∣ z 1 pos ∣ ≤ ∣ z 2 ∣
| g(\text { pos })|\leq| z_{2} z_{1}^{\text {pos }}|\leq| z_{2}|| z_{1}^{\text {pos }}|\leq| z_{2} |
∣ g ( pos ) ∣ ≤ ∣ z 2 z 1 pos ∣ ≤ ∣ z 2 ∣ ∣ z 1 pos ∣ ≤ ∣ z 2 ∣ w ( n ) = z 1 n w(n)=z_{1}^{n} w ( n ) = z 1 n n ( pos ) = w ( n ) _{n}(\text { pos })=w(n) n ( pos ) = w ( n ) g ( pos + n ) = z 2 z 1 p s + n = z 2 z 1 p o s z 1 n = g ( pos ) z 1 n = Transform n ( g ( pos ) ) g(\text { pos }+n)=z_{2} z_{1}^{\mathrm{ps}+n}=z_{2} z_{1}^{\mathrm{pos}} z_{1}^{n}=g(\text { pos }) z_{1}^{n}=\text{Transform}_{n}(g(\text { pos })) g ( pos + n ) = z 2 z 1 p s + n = z 2 z 1 p o s z 1 n = g ( pos ) z 1 n = Transform n ( g ( pos ) )
根据欧拉公式，可以将上式转化为，g ( pos ) = z 2 z 1 pos = r 2 e i θ 2 ( r 1 e i θ 1 ) pos = r 2 r 1 pos e i ( θ 2 + θ 1 pos ) subject to ∣ r 1 ∣ ≤ 1
g(\text { pos })=z_{2} z_{1}^{\text {pos }}=r_{2} e^{i \theta_{2}}\left(r_{1} e^{i \theta_{1}}\right)^{\text {pos }}=r_{2} r_{1}^{\text {pos }} e^{i\left(\theta_{2}+\theta_{1} \text { pos }\right)} \text { subject to }\left|r_{1}\right| \leq 1
g ( pos ) = z 2 z 1 pos = r 2 e i θ 2 ( r 1 e i θ 1 ) pos = r 2 r 1 pos e i ( θ 2 + θ 1 pos ) subject to ∣ r 1 ∣ ≤ 1 ∣ r 1 ∣ ≤ 1 \left|r_{1}\right| \leq 1 ∣ r 1 ∣ ≤ 1 r 1 = 1 r_{1}=1 r 1 = 1 g ( pos ) = r e i ( ω pos + θ )
g(\text { pos })=r e^{i(\omega \text { pos }+\theta)}
g ( pos ) = r e i ( ω pos + θ ) f ( j , pos ) = g j ( pos ) = r j e i ( ω j pos + θ j )
f(j, \text { pos })=\boldsymbol{g}_{j}(\text { pos })=\boldsymbol{r}_{j} e^{i\left(\boldsymbol{\omega}_{j} \text { pos }+\boldsymbol{\theta}_{j}\right)}
f ( j , pos ) = g j ( pos ) = r j e i ( ω j pos + θ j ) r j r_{j} r j w j w_{j} w j θ j \theta_{j} θ j
振幅r j r_{j} r j
角频率w j w_{j} w j
要用的话，计算量以及参数量是不是会很大呀
好像也没跟其他relative position的模型比较
