本文作爲第2部分，主要根據原始論文介紹幾篇基礎的工作，主要包括GNN，GCN及變體，DCNN，Tree-LSTM,包括模型的詳解和模型訓練，以及模型評價。
這篇論文是第一個提出Graph Neural Network模型的論文，它將神經網絡使用在圖結構數據上，並細述了神經網絡模型了結構組成、計算方法、優化算法、流程實現等等。論文後面還對模型的複雜度進行了評估，以及在現實任務上進行了實驗和比較(比較算法爲NL、L、FNN)。該報告暫時主要關注模型設計部分和實驗結果部分，忽略複雜性評估部分 。
對於圖領域問題，假設函數τ \tau τ G G G n n n τ ( G , n ) ∈ R m
\tau(G,n)\in{R^m}
τ ( G , n ) ∈ R m
圖領域的應用主要可以分爲兩種類型：專注於圖的應用(graph-focused)和專注於節點的應用(node-focused) 。對於graph-focused的應用，函數τ \tau τ τ ( G ) \tau(G) τ ( G ) τ \tau τ n n n τ ( G , n ) \tau(G,n) τ ( G , n )
圖 (a) 是一個化學分子結構，能夠使用圖 G G G τ ( G ) \tau(G) τ ( G )
圖 (b) 是一張城堡的圖片，圖片中的每一種結構都由節點表示，函數τ ( G , n ) \tau(G,n) τ ( G , n )
GNN模型基於信息傳播機制，每一個節點通過相互交換信息來更新自己的節點狀態，直到達到某一個穩定值 ，GNN的輸出就是在每個節點處，根據當前節點狀態分別計算輸出。
有如下定義：
一個圖G G G ( N , E ) (\boldsymbol{N}, \boldsymbol{E}) ( N , E ) N \boldsymbol{N} N E \boldsymbol{E} E
n e [ n ] ne[n] n e [ n ] n n n
c o [ n ] co[n] c o [ n ] n n n
l n ∈ R l N \boldsymbol{l}_{n} \in \mathbb{R}^{l_{N}} l n ∈ R l N n n n
l ( n 1 , n 2 ) ∈ R l E \boldsymbol{l}_{\left(n_{1}, n_{2}\right)} \in \mathbb{R}^{l_{E}} l ( n 1 , n 2 ) ∈ R l E ( n 1 , n 2 ) (n_1,n_2) ( n 1 , n 2 )
l \boldsymbol{l} l
注 ：原論文裏面l \boldsymbol{l} l features of objects related to nodes and features of the relationships between the objects，也就是相關特徵，所以這裏一律使用特徵向量翻譯。
論文將圖分爲positional graph和nonpositional graph，對於positional graph，對於每一個節點n n n u u u ν n ( u ) \nu_{n}(u) ν n ( u ) ν n : n e [ n ] → { 1 , … ∣ N ∣ } \nu_{n} : \mathrm{ne}[n] \rightarrow\{1, \ldots|\mathbf{N}|\} ν n : n e [ n ] → { 1 , … ∣ N ∣ }
假設存在一個圖-節點對的集合D = G × N \mathcal{D}=\mathcal{G} \times \mathcal{N} D = G × N G \mathcal{G} G N \mathcal{N} N 圖領域問題可以表示成一個有如下數據集的監督學習框架 L = { ( G i , n i , j , t i , j ) ∣ G i = ( N i , E i ) ∈ G ; n i , j ∈ N i ; t i , j ∈ R m , 1 ≤ i ≤ p , 1 ≤ j ≤ q i } \mathcal{L}=\left\{\left(\boldsymbol{G}_{i}, n_{i, j}, \boldsymbol{t}_{i, j}\right)| \boldsymbol{G}_{i}=\left(\boldsymbol{N}_{i}, \boldsymbol{E}_{i}\right) \in \mathcal{G}\right.;n_{i, j} \in \boldsymbol{N}_{i} ; \boldsymbol{t}_{i, j} \in \mathbb{R}^{m}, 1 \leq i \leq p, 1 \leq j \leq q_{i} \}
L = { ( G i , n i , j , t i , j ) ∣ G i = ( N i , E i ) ∈ G ; n i , j ∈ N i ; t i , j ∈ R m , 1 ≤ i ≤ p , 1 ≤ j ≤ q i } n i , j ∈ N i n_{i, j} \in \boldsymbol{N}_{i} n i , j ∈ N i N i ∈ N \boldsymbol{N}_{i} \in \mathcal{N} N i ∈ N j j j t i , j \boldsymbol{t}_{i, j} t i , j n i j n_{ij} n i j
節點n n n x n ∈ R s \boldsymbol{x}_{n} \in \mathbb{R}^{s} x n ∈ R s o n \boldsymbol{o}_{\boldsymbol{n}} o n f w f_{\boldsymbol{w}} f w local transition function ，g w g_{\boldsymbol{w}} g w local output function ，那麼x n \boldsymbol{x}_{n} x n o n \boldsymbol{o}_{\boldsymbol{n}} o n x n = f w ( l n , l c o [ n ] , x n e [ n ] , l n e [ n ] ) o n = g w ( x n , l n )
\begin{array}{l}{\boldsymbol{x}_{n}=f_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{\mathrm{co}[n]}, \boldsymbol{x}_{\mathrm{ne}[n]}, \boldsymbol{l}_{\mathrm{ne}\left[n\right]}\right)} \\ {\boldsymbol{o}_{n}=g_{\boldsymbol{w}}\left(\boldsymbol{x}_{n}, \boldsymbol{l}_{n}\right)}\end{array}
x n = f w ( l n , l c o [ n ] , x n e [ n ] , l n e [ n ] ) o n = g w ( x n , l n ) l n , l co [ n ] , x n e [ n ] , l n e [ n ] \boldsymbol{l}_{n}, \boldsymbol{l}_{\operatorname{co}[n]}, \boldsymbol{x}_{\mathrm{ne}[n]}, \boldsymbol{l}_{\mathrm{ne}[n]} l n , l c o [ n ] , x n e [ n ] , l n e [ n ] n n n n n n n n n n n n
假設x , o , l , l N \boldsymbol{x}, \boldsymbol{o}, \boldsymbol{l}, \boldsymbol{l}_{N} x , o , l , l N x = F w ( x , l ) o = G w ( x , l N )
\begin{array}{l}{\boldsymbol{x}=F_{\boldsymbol{w}}(\boldsymbol{x}, \boldsymbol{l})} \\ {\boldsymbol{o}=\boldsymbol{G}_{\boldsymbol{w}}\left(\boldsymbol{x}, \boldsymbol{l}_{\boldsymbol{N}}\right)}\end{array}
x = F w ( x , l ) o = G w ( x , l N ) F w F_{\boldsymbol{w}} F w global transition function ，G w G_{\boldsymbol{w}} G w global output function ，分別是f w f_{\boldsymbol{w}} f w g w g_{\boldsymbol{w}} g w
根據Banach的不動點理論，假設F w F_{\boldsymbol{w}} F w x ( t + 1 ) = F w ( x ( t ) , l )
\boldsymbol{x}(t+1)=F_{\boldsymbol{w}}(\boldsymbol{x}(t), \boldsymbol{l})
x ( t + 1 ) = F w ( x ( t ) , l ) x ( t ) \boldsymbol{x}(t) x ( t ) x \boldsymbol{x} x t t t x n ( t + 1 ) = f w ( l n , l c o [ n ] , x n e [ n ] ( t ) , l n e [ n ] ) o n ( t ) = g w ( x n ( t ) , l n ) , n ∈ N
\begin{aligned} \boldsymbol{x}_{n}(t+1) &=f_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{\mathrm{co}[n]}, \boldsymbol{x}_{\mathrm{ne}[n]}(t), \boldsymbol{l}_{\mathrm{ne}[n]}\right) \\ \boldsymbol{o}_{n}(t) &=g_{\boldsymbol{w}}\left(\boldsymbol{x}_{n}(t), \boldsymbol{l}_{n}\right), \quad n \in \boldsymbol{N} \end{aligned}
x n ( t + 1 ) o n ( t ) = f w ( l n , l c o [ n ] , x n e [ n ] ( t ) , l n e [ n ] ) = g w ( x n ( t ) , l n ) , n ∈ N
T T T
GNN的學習就是估計參數w \boldsymbol{w} w φ w \varphi_{\boldsymbol{w}} φ w L = { ( G i , n i , j , t i , j ) ∣ G i = ( N i , E i ) ∈ G ; n i , j ∈ N i ; t i , j ∈ R m , 1 ≤ i ≤ p , 1 ≤ j ≤ q i }
\mathcal{L}=\left\{\left(\boldsymbol{G}_{i}, n_{i, j}, \boldsymbol{t}_{i, j}\right)| \boldsymbol{G}_{i}=\left(\boldsymbol{N}_{i}, \boldsymbol{E}_{i}\right) \in \mathcal{G}\right.;n_{i, j} \in \boldsymbol{N}_{i} ; \boldsymbol{t}_{i, j} \in \mathbb{R}^{m}, 1 \leq i \leq p, 1 \leq j \leq q_{i} \}
L = { ( G i , n i , j , t i , j ) ∣ G i = ( N i , E i ) ∈ G ; n i , j ∈ N i ; t i , j ∈ R m , 1 ≤ i ≤ p , 1 ≤ j ≤ q i } q i q_i q i G i G_{i} G i e w = ∑ i = 1 p ∑ j = 1 q i ( t i , j − φ w ( G i , n i , j ) ) 2
e_{\boldsymbol{w}}=\sum_{i=1}^{p} \sum_{j=1}^{q_{i}}\left(\boldsymbol{t}_{i, j}-\varphi_{\boldsymbol{w}}\left(\boldsymbol{G}_{i}, n_{i, j}\right)\right)^{2}
e w = i = 1 ∑ p j = 1 ∑ q i ( t i , j − φ w ( G i , n i , j ) ) 2
按照迭代方程迭代T T T x n ( t ) x_{n}(t) x n ( t ) x ( T ) ≈ x \boldsymbol{x}(T) \approx \boldsymbol{x} x ( T ) ≈ x
計算參數權重的梯度∂ e w ( T ) / ∂ w \partial e_{\boldsymbol{w}}(T) / \partial \boldsymbol{w} ∂ e w ( T ) / ∂ w
使用該梯度來更新權重w \boldsymbol{w} w
這裏假設函數F w F_{\boldsymbol{w}} F w backpropagation-through-time algorithm 。
爲了表明前面的方法是可行的，論文接着證明了兩個結論
理論1(可微性)：令F w F_{\boldsymbol{w}} F w G w G_{\boldsymbol{w}} G w F w ( x , l ) F_{\boldsymbol{w}}(\boldsymbol{x}, \boldsymbol{l}) F w ( x , l ) G w ( x , l N ) G_{\boldsymbol{w}}\left(\boldsymbol{x}, \boldsymbol{l}_{\boldsymbol{N}}\right) G w ( x , l N ) x \boldsymbol{x} x w \boldsymbol{w} w φ w \varphi_{\boldsymbol{w}} φ w w \boldsymbol{w} w
理論2(反向傳播)：令F w F_{\boldsymbol{w}} F w G w G_{\boldsymbol{w}} G w F w ( x , l ) F_{\boldsymbol{w}}(\boldsymbol{x}, \boldsymbol{l}) F w ( x , l ) G w ( x , l N ) G_{\boldsymbol{w}}\left(\boldsymbol{x}, \boldsymbol{l}_{\boldsymbol{N}}\right) G w ( x , l N ) x \boldsymbol{x} x w \boldsymbol{w} w z ( t ) \boldsymbol{z}(t) z ( t ) z ( t ) = z ( t + 1 ) ⋅ ∂ F w ∂ x ( x , l ) + ∂ e w ∂ o ⋅ ∂ G w ∂ x ( x , l N )
z(t)=z(t+1) \cdot \frac{\partial F_{w}}{\partial x}(x, l)+\frac{\partial e_{w}}{\partial o} \cdot \frac{\partial G_{w}}{\partial x}\left(x, l_{N}\right)
z ( t ) = z ( t + 1 ) ⋅ ∂ x ∂ F w ( x , l ) + ∂ o ∂ e w ⋅ ∂ x ∂ G w ( x , l N ) z ( T ) , z ( T − 1 ) , … \boldsymbol{z}(T), \boldsymbol{z}(T-1), \ldots z ( T ) , z ( T − 1 ) , … z = lim t → − ∞ z ( t ) z=\lim _{t \rightarrow-\infty} z(t) z = lim t → − ∞ z ( t ) z ( T ) \boldsymbol{z}(T) z ( T ) ∂ e w ∂ w = ∂ e w ∂ o ⋅ ∂ G w ∂ w ( x , l N ) + z ⋅ ∂ F w ∂ w ( x , l )
\frac{\partial e_{w}}{\partial \boldsymbol{w}}=\frac{\partial e_{\boldsymbol{w}}}{\partial \boldsymbol{o}} \cdot \frac{\partial G_{\boldsymbol{w}}}{\partial \boldsymbol{w}}\left(\boldsymbol{x}, \boldsymbol{l}_{N}\right)+\boldsymbol{z} \cdot \frac{\partial F_{\boldsymbol{w}}}{\partial \boldsymbol{w}}(\boldsymbol{x}, \boldsymbol{l})
∂ w ∂ e w = ∂ o ∂ e w ⋅ ∂ w ∂ G w ( x , l N ) + z ⋅ ∂ w ∂ F w ( x , l ) x \boldsymbol{x} x
算法流程如下FORWARD 用於迭代計算出收斂點，BACKWARD 用於計算梯度。
在GNN中，函數g w g_{\boldsymbol{w}} g w f w f_{\boldsymbol{w}} f w f w f_{\boldsymbol{w}} f w
Linear(nonpositional) GNN ：
對於節點n n n f w f_{\boldsymbol{w}} f w x n = ∑ u ∈ ne ∣ n ] h w ( l n , l ( n , u ) , x u , l u ) , n ∈ N
\boldsymbol{x}_{n}=\sum_{u \in \text { ne } | n ]} h_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{(n, u)}, \boldsymbol{x}_{u}, \boldsymbol{l}_{u}\right), \quad n \in \boldsymbol{N}
x n = u ∈ ne ∣ n ] ∑ h w ( l n , l ( n , u ) , x u , l u ) , n ∈ N n n n h w h_{\boldsymbol{w}} h w n n n
由此，對上式中的函數h w h_{\boldsymbol{w}} h w h w ( l n , l ( n , a ) , x u , l u ) = A n , u x u + b n
h_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{(n, \mathfrak{a})}, \boldsymbol{x}_{u}, \boldsymbol{l}_{u}\right) = \boldsymbol{A}_{n, u} \boldsymbol{x}_{u}+\boldsymbol{b}_{n}
h w ( l n , l ( n , a ) , x u , l u ) = A n , u x u + b n b n ∈ R s \boldsymbol{b}_{n} \in \mathbb{R}^{s} b n ∈ R s A n , u ∈ R s × s \boldsymbol{A}_{n, u} \in \mathbb{R}^{s \times s} A n , u ∈ R s × s A n , u \boldsymbol{A}_{n, u} A n , u transition network ，產生向量b n \boldsymbol{b}_{n} b n forcing network
transition network表示爲ϕ w \phi_{\boldsymbol{w}} ϕ w ϕ w : R 2 l N + l E → R s 2
\phi_{\boldsymbol{w}} : \mathbb{R}^{2 l_{N}+l_{E}} \rightarrow \mathbb{R}^{s^{2}}
ϕ w : R 2 l N + l E → R s 2 ρ w \rho_{\boldsymbol{w}} ρ w ρ w : R l N → R s \rho_{\boldsymbol{w}} : \mathbb{R}^{l_{N}} \rightarrow \mathbb{R}^{s}
ρ w : R l N → R s A n , u \boldsymbol{A}_{n, u} A n , u b n \boldsymbol{b}_{n} b n A n , u = μ s ∣ ne [ u ] ∣ ⋅ Ξ b w = ρ w ( l n ) \begin{aligned} \boldsymbol{A}_{\boldsymbol{n}, \boldsymbol{u}} &=\frac{\mu}{s|\operatorname{ne}[u]|} \cdot \boldsymbol{\Xi} \\ \boldsymbol{b}_{\boldsymbol{w}} &=\rho_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}\right) \end{aligned}
A n , u b w = s ∣ n e [ u ] ∣ μ ⋅ Ξ = ρ w ( l n ) μ ∈ ( 0 , 1 ) \mu \in(0,1) μ ∈ ( 0 , 1 ) Ξ = resize ( ϕ w ( l n , l ( n , u ) , l u ) ) \Xi=\operatorname{resize}\left(\phi_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{(n, u)}, \boldsymbol{l}_{u}\right)\right) Ξ = r e s i z e ( ϕ w ( l n , l ( n , u ) , l u ) ) resize ( ⋅ ) \text{resize}(\cdot) resize ( ⋅ ) s 2 s^2 s 2 s × s s\times{s} s × s A n , u \boldsymbol{A}_{n, u} A n , u b n \boldsymbol{b}_{n} b n
在這裏，假定∥ ϕ w ( l n , l ( n , u ) , l u ) ∥ 1 ≤ s \left\|\phi_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{(\boldsymbol{n}, \boldsymbol{u})}, \boldsymbol{l}_{u}\right)\right\|_{1} \leq \boldsymbol{s} ∥ ∥ ϕ w ( l n , l ( n , u ) , l u ) ∥ ∥ 1 ≤ s t a n h ( ) tanh() t a n h ( ) F w ( x , l ) = A x + b F_{\boldsymbol{w}}(\boldsymbol{x}, \boldsymbol{l})=\boldsymbol{A} \boldsymbol{x}+\boldsymbol{b} F w ( x , l ) = A x + b A \boldsymbol{A} A b \boldsymbol{b} b A n , u \boldsymbol{A}_{n, u} A n , u b n \boldsymbol{b}_{n} b n ∥ ∂ F w ∂ x ∥ 1 = ∥ A ∥ 1 ≤ max u ∈ N ( ∑ n ∈ ne [ u ] ∥ A n , u ∥ 1 ) ≤ max u ∈ N ( μ s ∣ ne [ u ] ∣ ⋅ ∑ n ∈ n e [ u ] ∥ Ξ ∥ 1 ) ≤ μ
\begin{aligned}\left\|\frac{\partial F_{\boldsymbol{w}}}{\partial \boldsymbol{x}}\right\|_{1} &=\|\boldsymbol{A}\|_{1} \leq \max _{u \in \boldsymbol{N}}\left(\sum_{n \in \operatorname{ne}[u]}\left\|\boldsymbol{A}_{n, u}\right\|_{1}\right) \\ & \leq \max _{u \in N}\left(\frac{\mu}{s|\operatorname{ne}[u]|} \cdot \sum_{n \in \mathrm{ne}[u]}\|\mathbf{\Xi}\|_{1}\right) \leq \mu \end{aligned}
∥ ∥ ∥ ∥ ∂ x ∂ F w ∥ ∥ ∥ ∥ 1 = ∥ A ∥ 1 ≤ u ∈ N max ⎝ ⎛ n ∈ n e [ u ] ∑ ∥ A n , u ∥ 1 ⎠ ⎞ ≤ u ∈ N max ⎝ ⎛ s ∣ n e [ u ] ∣ μ ⋅ n ∈ n e [ u ] ∑ ∥ Ξ ∥ 1 ⎠ ⎞ ≤ μ F w F_{\boldsymbol{w}} F w w \boldsymbol{w} w
矩陣M M M ∥ M ∥ 1 = max j ∑ i ∣ m i , j ∣ \|M\|_{1}=\max _{j} \sum_{i}\left|m_{i, j}\right| ∥ M ∥ 1 = j max i ∑ ∣ m i , j ∣
Nonelinear(nonpositional) GNN ：在這個結構中，h w h_{\boldsymbol{w}} h w w \boldsymbol{w} w F w F_{\boldsymbol{w}} F w e w = ∑ i = 1 p ∑ j = 1 q i ( t i , j − φ w ( G i , n i , j ) ) 2 + β L ( ∥ ∂ F w ∂ x ∥ )
e_{\boldsymbol{w}}=\sum_{i=1}^{p} \sum_{j=1}^{q_{i}}\left(\boldsymbol{t}_{i, j}-\varphi_{\boldsymbol{w}}\left(\boldsymbol{G}_{i}, n_{i, j}\right)\right)^{2}+\beta L\left(\left\|\frac{\partial F_{\boldsymbol{w}}}{\partial \boldsymbol{x}}\right\|\right)
e w = i = 1 ∑ p j = 1 ∑ q i ( t i , j − φ w ( G i , n i , j ) ) 2 + β L ( ∥ ∥ ∥ ∥ ∂ x ∂ F w ∥ ∥ ∥ ∥ ) L ( y ) L(y) L ( y ) y > μ y>\mu y > μ ( y − μ ) 2 (y-\mu)^2 ( y − μ ) 2 y ≤ μ y\le{\mu} y ≤ μ μ ∈ ( 0 , 1 ) \mu\in(0,1) μ ∈ ( 0 , 1 ) F w F_{\boldsymbol{w}} F w
論文將GNN模型在三個任務上進行了實驗：子圖匹配(subgraph matching)任務，誘變(mutagenesis)任務和網頁排序(web page ranking)任務。在這些任務上使用linear和nonlinear的模型測試，其中nonlinear模型中的激活函數使用sigmoid函數。
子圖匹配任務爲在一個大圖G G G S S S τ \tau τ n i , j n_{i,j} n i , j G G G τ ( G i , n i , j ) = 1 \tau(G_i,n_{i,j})=1 τ ( G i , n i , j ) = 1 τ ( G i , n i , j ) = − 1 \tau(G_i,n_{i,j})=-1 τ ( G i , n i , j ) = − 1
誘變問題任務是對化學分子進行分類，識別出誘變化合物，採用二分類方法。實驗結果是nonlinear效果較好，但不是最好。
網頁排序任務是學會網頁排序。實驗表明雖然訓練集只包含50個網頁，但是仍然沒有產生過擬合的現象。
在模擬的節點分類任務上實現該論文的GNN模型。
任務要求 爲輸入一個graph，該graph的所有節點都有標籤，然後對部分節點進行訓練，在驗證階段使用另外一部分節點進行驗證。輸入的數據圖如下圖：
{ n 1 , n 2 , . . . , n 18 } \{n1,n2,...,n18\} { n 1 , n 2 , . . . , n 1 8 } { 0 , 1 , 2 } \{0,1,2\} { 0 , 1 , 2 } { n 1 , n 2 , n 3 , n 7 , n 8 , n 9 , n 13 , n 14 , n 15 } \{n1,n2,n3,n7,n8,n9,n13,n14,n15\} { n 1 , n 2 , n 3 , n 7 , n 8 , n 9 , n 1 3 , n 1 4 , n 1 5 }
輸入的數據爲(node,label)列表和(node1, node2)邊列表，表示如下
N = [ ( "n{}" . format ( i) , 0 ) for i in range ( 1 , 7 ) ] + \
[ ( "n{}" . format ( i) , 1 ) for i in range ( 7 , 13 ) ] + \
[ ( "n{}" . format ( i) , 2 ) for i in range ( 13 , 19 ) ]
E = [ ( "n1" , "n2" ) , ( "n1" , "n3" ) , ( "n1" , "n5" ) ,
( "n2" , "n4" ) ,
( "n3" , "n6" ) , ( "n3" , "n9" ) ,
( "n4" , "n5" ) , ( "n4" , "n6" ) , ( "n4" , "n8" ) ,
( "n5" , "n14" ) ,
( "n7" , "n8" ) , ( "n7" , "n9" ) , ( "n7" , "n11" ) ,
( "n8" , "n10" ) , ( "n8" , "n11" ) , ( "n8" , "n12" ) ,
( "n9" , "n10" ) , ( "n9" , "n14" ) ,
( "n10" , "n12" ) ,
( "n11" , "n18" ) ,
( "n13" , "n15" ) , ( "n13" , "n16" ) , ( "n13" , "n18" ) ,
( "n14" , "n16" ) , ( "n14" , "n18" ) ,
( "n15" , "n16" ) , ( "n15" , "n18" ) ,
( "n17" , "n18" ) ]
N N N E E E
模型部分使用論文的linear函數來設計f w f_w f w g w g_w g w Ξ \Xi Ξ ρ \rho ρ
class Xi ( nn. Module) :
def __init__ ( self, ln, s) :
. . .
def forward ( self, X) :
. . .
class Rou ( nn. Module) :
def __init__ ( self, ln, s) :
. . .
def forward ( self, X) :
. . .
class Hw ( nn. Module) :
def __init__ ( self, ln, s, mu= 0.9 ) :
. . .
def forward ( self, X, H, dg_list) :
. . .
class AggrSum ( nn. Module) :
def __init__ ( self, node_num) :
. . .
def forward ( self, H, X_node) :
. . .
class OriLinearGNN ( nn. Module) :
def __init__ ( self, node_num, feat_dim, stat_dim, T) :
. . .
def forward ( self, X_Node, X_Neis, dg_list) :
. . .
for t in range ( self. T) :
H = torch. index_select( self. node_states, 0 , X_Node)
H = self. Hw( X, H, dg_list)
self. node_states = self. Aggr( H, X_Node)
. . .
可以看出，在模型訓練階段，每次forward，都會直接循環計算T次f w f_w f w
模型訓練部分按照常規的分類模型進行訓練，採用Adam優化器，學習率保持爲0.01，權重衰減爲0.01，使用交叉熵作爲損失函數，模型訓練部分代碼如下
def CalAccuracy ( output, label) :
. . .
def train ( node_list, edge_list, label_list, T, ndict_path= "./node_dict.json" ) :
. . .
. . .
. . .
. . .
train_node_list = [ 0 , 1 , 2 , 6 , 7 , 8 , 12 , 13 , 14 ]
train_node_label = [ 0 , 0 , 0 , 1 , 1 , 1 , 2 , 2 , 2 ]
test_node_list = [ 3 , 4 , 5 , 9 , 10 , 11 , 15 , 16 , 17 ]
test_node_label = [ 0 , 0 , 0 , 1 , 1 , 1 , 2 , 2 , 2 ]
model = OriLinearGNN( node_num= len ( node_list) ,
feat_dim= 2 ,
stat_dim= 2 ,
T= T)
optimizer = torch. optim. Adam( model. parameters( ) , lr= 0.01 , weight_decay= 0.01 )
criterion = nn. CrossEntropyLoss( size_average= True )
min_loss = float ( 'inf' )
node_inds_tensor = Variable( torch. Tensor( node_inds) . long ( ) )
node_neis_tensor = Variable( torch. Tensor( node_neis) . long ( ) )
train_label = Variable( torch. Tensor( train_node_label) . long ( ) )
for ep in range ( 500 ) :
res = model( node_inds_tensor, node_neis_tensor, dg_list)
train_res = torch. index_select( res, 0 , torch. Tensor( train_node_list) . long ( ) )
test_res = torch. index_select( res, 0 , torch. Tensor( test_node_list) . long ( ) )
loss = criterion( input = train_res,
target= train_label)
loss_val = loss. item( )
train_acc = CalAccuracy( train_res. cpu( ) . detach( ) . numpy( ) , np. array( train_node_label) )
test_acc = CalAccuracy( test_res. cpu( ) . detach( ) . numpy( ) , np. array( test_node_label) )
optimizer. zero_grad( )
loss. backward( retain_graph= True )
optimizer. step( )
if loss_val < min_loss:
min_loss = loss_val
print ( "==> [Epoch {}] : loss {:.4f}, min_loss {:.4f}, train_acc {:.3f}, test_acc {:.3f}" . format ( ep, loss_val, min_loss, train_acc, test_acc) )
模型的訓練和評估結果如下圖:2 3 \frac{2}{3} 3 2
按照圖傅里葉變換的性質，可以得到如下圖卷積的定義( f ∗ h ) G = Φ diag [ h ^ ( λ 1 ) , … , h ^ ( λ n ) ] Φ T f
(\boldsymbol{f} * \boldsymbol{h})_{\mathcal{G}}=\boldsymbol{\Phi} \operatorname{diag}\left[\hat{h}\left(\lambda_{1}\right), \ldots, \hat{h}\left(\lambda_{n}\right)\right] \mathbf{\Phi}^{T} \boldsymbol{f}
( f ∗ h ) G = Φ d i a g [ h ^ ( λ 1 ) , … , h ^ ( λ n ) ] Φ T f
對於圖$ \boldsymbol{f}的 傅 裏 葉 變 換 爲 的傅里葉變換爲 的 傅 裏 葉 變 換 爲
對於卷積核的圖傅里葉變換：h ^ = ( h ^ 1 , … , h ^ n ) \hat{h}=\left(\hat{h}_{1}, \ldots, \hat{h}_{n}\right) h ^ = ( h ^ 1 , … , h ^ n ) h ^ k = ⟨ h , ϕ k ⟩ , k = 1 , 2 … , n
\hat{h}_{k}=\left\langle h, \phi_{k}\right\rangle, k=1,2 \ldots, n
h ^ k = ⟨ h , ϕ k ⟩ , k = 1 , 2 … , n h ^ = Φ T h \hat{\boldsymbol{h}}=\mathbf{\Phi}^{T} \boldsymbol{h} h ^ = Φ T h
對兩者的傅里葉變換向量f ^ ∈ R N × 1 \hat{f} \in \mathbb{R}^{N \times 1} f ^ ∈ R N × 1 h ^ ∈ R N × 1 \hat{h} \in \mathbb{R}^{N \times 1} h ^ ∈ R N × 1 等價於將h \boldsymbol{h} h ，即diag [ h ^ ( λ k ) ] ∈ R N × N \operatorname{diag}\left[\hat{h}\left(\lambda_{k}\right)\right] \in \mathbb{R}^{N \times N} d i a g [ h ^ ( λ k ) ] ∈ R N × N diag [ h ^ ( λ k ) ] \operatorname{diag}\left[\hat{h}\left(\lambda_{k}\right)\right] d i a g [ h ^ ( λ k ) ] f \boldsymbol{f} f
求上述結果的傅里葉逆變換，即左乘Φ \mathbf{\Phi} Φ
深度學習中的卷積就是要設計trainable的卷積核 ，從上式可以看出，就是要設計diag [ h ^ ( λ 1 ) , … , h ^ ( λ n ) ] \operatorname{diag}\left[\hat{h}\left(\lambda_{1}\right), \ldots, \hat{h}\left(\lambda_{n}\right)\right] d i a g [ h ^ ( λ 1 ) , … , h ^ ( λ n ) ] diag [ θ 1 , … , θ n ] \operatorname{diag}\left[\theta_{1}, \ldots, \theta_{n}\right] d i a g [ θ 1 , … , θ n ] 不需要再將卷積核進行傅里葉變換 ，由此，相當於直接將變換後的參量進行學習。
第一代GCN爲y output = σ ( Φ g θ Φ T x ) = σ ( Φ diag [ θ 1 , … , θ n ] Φ T x )
\boldsymbol{y}_{\text {output}}=\sigma\left(\mathbf{\Phi} \boldsymbol{g}_{\theta} \mathbf{\Phi}^{T} \boldsymbol{x}\right)=\sigma\left(\boldsymbol{\Phi} \operatorname{diag}\left[\theta_{1}, \ldots, \theta_{n}\right] \mathbf{\Phi}^{T} \boldsymbol{x}\right)
y output = σ ( Φ g θ Φ T x ) = σ ( Φ d i a g [ θ 1 , … , θ n ] Φ T x ) x \boldsymbol{x} x x = ( x 1 , x 2 , … , x n ) x=\left(x_{1}, x_{2}, \ldots, x_{n}\right) x = ( x 1 , x 2 , … , x n ) y output \boldsymbol{y}_{\text {output}} y output
一些缺點：
需要對拉普拉斯矩陣進行譜分解來求Φ \mathbf{\Phi} Φ O ( n 2 ) O(n^2) O ( n 2 )
卷積核參數爲n n n n n n
卷積核的spatial localization不好。
圖傅里葉變換是關於特徵值(相當於普通傅里葉變換的頻率)的函數，也就是F ( λ 1 ) , … , F ( λ n ) F\left(\lambda_{1}\right), \ldots, F\left(\lambda_{n}\right) F ( λ 1 ) , … , F ( λ n ) F ( Λ ) F(\mathbf{\Lambda}) F ( Λ ) g θ \boldsymbol{g}_{\theta} g θ g θ ( Λ ) \boldsymbol{g}_{\theta}(\Lambda) g θ ( Λ ) g θ ( Λ ) \boldsymbol{g}_{\theta}(\Lambda) g θ ( Λ ) k階多項式 g θ ′ ( Λ ) ≈ ∑ k = 0 K θ k ′ Λ k
g_{\theta^{\prime}}(\mathbf{\Lambda}) \approx \sum_{k=0}^{K} \theta_{k}^{\prime} \mathbf{\Lambda}^{k}
g θ ′ ( Λ ) ≈ k = 0 ∑ K θ k ′ Λ k g θ ′ ∗ x ≈ Φ ∑ k = 0 K θ k ′ Λ k Φ T x = ∑ k = 0 K θ k ′ ( Φ Λ k Φ T ) x = ∑ k = 0 K θ k ′ ( Φ Λ Φ T ) k x = ∑ k = 0 K θ k ′ L k x
\begin{aligned} g_{\theta^{\prime}} * x & \approx \Phi \sum_{k=0}^{K} \theta_{k}^{\prime} \mathbf{\Lambda}^{k} \mathbf{\Phi}^{T} \boldsymbol{x} \\ &=\sum_{k=0}^{K} \theta_{k}^{\prime}\left(\mathbf{\Phi} \mathbf{\Lambda}^{k} \mathbf{\Phi}^{T}\right) x \\ &=\sum_{k=0}^{K} \theta_{k}^{\prime}\left(\mathbf{\Phi} \mathbf{\Lambda} \mathbf{\Phi}^{T}\right)^{k} x \\ &=\sum_{k=0}^{K} \theta_{k}^{\prime} \boldsymbol{L}^{k} x \end{aligned}
g θ ′ ∗ x ≈ Φ k = 0 ∑ K θ k ′ Λ k Φ T x = k = 0 ∑ K θ k ′ ( Φ Λ k Φ T ) x = k = 0 ∑ K θ k ′ ( Φ Λ Φ T ) k x = k = 0 ∑ K θ k ′ L k x L k \boldsymbol{L}^{k} L k O ( K n 2 ) O(Kn^2) O ( K n 2 )
對於每一次Laplacian矩陣L \boldsymbol{L} L x \mathbf{x} x n n n n e [ n ] ne[n] n e [ n ] n n n k k k n n n 因此，實際上只利用了節點的K-Localized信息 。
另外，可以使用切比雪夫展開式來近似L k \boldsymbol{L}^{k} L k 任何k次多項式都可以使用切比雪夫展開式來近似 ，由此，引入切比雪夫多項式的K K K L k \boldsymbol{L}^{k} L k g θ ( Λ ) g_{\theta}(\mathbf{\Lambda}) g θ ( Λ ) g θ ′ ( Λ ) ≈ ∑ k = 0 K θ k ′ T k ( Λ ~ )
g_{\theta^{\prime}}(\mathbf{\Lambda}) \approx \sum_{k=0}^{K} \theta_{k}^{\prime} T_{k}(\tilde{\mathbf{\Lambda}})
g θ ′ ( Λ ) ≈ k = 0 ∑ K θ k ′ T k ( Λ ~ ) Λ ~ = 2 λ max Λ − I n \tilde{\mathbf{\Lambda}}=\frac{2}{\lambda_{\max }} \mathbf{\Lambda}-\boldsymbol{I}_{n} Λ ~ = λ max 2 Λ − I n θ ′ ∈ R K \boldsymbol{\theta}^{\prime} \in \mathbb{R}^{K} θ ′ ∈ R K θ k ′ \theta_{k}^{\prime} θ k ′ k k k T k ( x ) T_{k}(x) T k ( x ) T k ( x ) = 2 x T k − 1 ( x ) − T k − 2 ( x ) T_{k}(x)=2 x T_{k-1}(x)-T_{k-2}(x) T k ( x ) = 2 x T k − 1 ( x ) − T k − 2 ( x ) T 0 ( x ) = 1 , T 1 ( x ) = x T_{0}(x)=1, T_{1}(x)=x T 0 ( x ) = 1 , T 1 ( x ) = x
此時，帶入到卷積公式g θ ′ ∗ x ≈ Φ ∑ k = 0 K θ k ′ T k ( Λ ~ ) Φ T x ≈ ∑ k = 0 K θ k ′ ( Φ T k ( Λ ~ ) Φ T ) x = ∑ k = 0 K θ k ′ T k ( L ~ ) x
\begin{aligned} \boldsymbol{g}_{\boldsymbol{\theta}^{\prime}} * \boldsymbol{x} & \approx \mathbf{\Phi} \sum_{k=0}^{K} \theta_{k}^{\prime} T_{k}(\tilde{\boldsymbol{\Lambda}}) \mathbf{\Phi}^{T} \boldsymbol{x} \\
&\approx \sum_{k=0}^{K} \theta_{k}^{\prime}\left(\mathbf{\Phi} T_{k}(\tilde{\mathbf{\Lambda}}) \mathbf{\Phi}^{T}\right) x \\
&=\sum_{k=0}^{K} \theta_{k}^{\prime} T_{k}(\tilde{\boldsymbol{L}}) \boldsymbol{x} \end{aligned}
g θ ′ ∗ x ≈ Φ k = 0 ∑ K θ k ′ T k ( Λ ~ ) Φ T x ≈ k = 0 ∑ K θ k ′ ( Φ T k ( Λ ~ ) Φ T ) x = k = 0 ∑ K θ k ′ T k ( L ~ ) x L ~ = 2 λ max L − I n \tilde{\boldsymbol{L}}=\frac{2}{\lambda_{\max }} \boldsymbol{L}-\boldsymbol{I}_{n} L ~ = λ max 2 L − I n
因此，可以得到輸出爲y output = σ ( ∑ k = 0 K θ k ′ T k ( L ~ ) x )
\boldsymbol{y}_{\text {output}}=\sigma\left(\sum_{k=0}^{K} \theta_{k}^{\prime} T_{k}(\tilde{\boldsymbol{L}}) \boldsymbol{x}\right)
y output = σ ( k = 0 ∑ K θ k ′ T k ( L ~ ) x )
這一代GCN直接取切比雪夫多項式中K = 1 K=1 K = 1
將K = 1 K=1 K = 1 λ max = 2 \lambda_{\max }=2 λ max = 2 g θ ′ ∗ x ≈ θ 0 ′ x + θ 1 ′ ( L − I n ) x = θ 0 ′ x + θ 1 ′ ( L − I n ) x = θ 0 ′ x − θ 1 ′ ( D − 1 / 2 W D − 1 / 2 ) x
\begin{aligned} \boldsymbol{g}_{\boldsymbol{\theta}^{\prime}} * \boldsymbol{x} & \approx \boldsymbol{\theta}_{0}^{\prime} \boldsymbol{x}+\theta_{1}^{\prime}\left(\boldsymbol{L}-\boldsymbol{I}_{n}\right) \boldsymbol{x} \\ &=\boldsymbol{\theta}_{0}^{\prime} \boldsymbol{x}+\theta_{1}^{\prime}\left(\boldsymbol{L}-\boldsymbol{I}_{n}\right) \boldsymbol{x} \\ &=\theta_{0}^{\prime} \boldsymbol{x}-\theta_{1}^{\prime}\left(\boldsymbol{D}^{-1 / 2} \boldsymbol{W} \boldsymbol{D}^{-1 / 2}\right) \boldsymbol{x} \end{aligned}
g θ ′ ∗ x ≈ θ 0 ′ x + θ 1 ′ ( L − I n ) x = θ 0 ′ x + θ 1 ′ ( L − I n ) x = θ 0 ′ x − θ 1 ′ ( D − 1 / 2 W D − 1 / 2 ) x L = D − 1 / 2 ( D − W ) D − 1 / 2 = I n − D − 1 / 2 W D − 1 / 2 \boldsymbol{L}=\boldsymbol{D}^{-1 / 2}(\boldsymbol{D}-\boldsymbol{W}) \boldsymbol{D}^{-1 / 2}=\boldsymbol{I}_{n}-\boldsymbol{D}^{-1 / 2} \boldsymbol{W} \boldsymbol{D}^{-1 / 2} L = D − 1 / 2 ( D − W ) D − 1 / 2 = I n − D − 1 / 2 W D − 1 / 2 爲了進一步簡化，令θ 0 ′ = − θ 1 ′ \theta_{0}^{\prime}=-\theta_{1}^{\prime} θ 0 ′ = − θ 1 ′ θ \theta θ g θ ′ ∗ x = θ ( I n + D − 1 / 2 W D − 1 / 2 ) x
g_{\theta^{\prime}} * x=\theta\left(I_{n}+D^{-1 / 2} W D^{-1 / 2}\right) x
g θ ′ ∗ x = θ ( I n + D − 1 / 2 W D − 1 / 2 ) x I n + D − 1 / 2 W D − 1 / 2 \boldsymbol{I}_{n}+\boldsymbol{D}^{-1 / 2} \boldsymbol{W} \boldsymbol{D}^{-1 / 2} I n + D − 1 / 2 W D − 1 / 2 譜半徑[ 0 , 2 ] [0,2] [ 0 , 2 ] I n + D − 1 / 2 W D − 1 / 2 → D ~ − 1 / 2 W ~ D ~ − 1 / 2
\boldsymbol{I}_{n}+\boldsymbol{D}^{-1 / 2} \boldsymbol{W} \boldsymbol{D}^{-1 / 2} \rightarrow \tilde{\boldsymbol{D}}^{-1 / 2} \tilde{\boldsymbol{W}} \tilde{\boldsymbol{D}}^{-1 / 2}
I n + D − 1 / 2 W D − 1 / 2 → D ~ − 1 / 2 W ~ D ~ − 1 / 2 W ~ = W + I n \tilde{\boldsymbol{W}}=\boldsymbol{W}+\boldsymbol{I}_{n} W ~ = W + I n D ~ i j = Σ j W ~ i j \tilde{D}_{i j}=\Sigma_{j} \tilde{W}_{i j} D ~ i j = Σ j W ~ i j
由此，帶入卷積公式g θ ′ ∗ x ⏟ R n × 1 = θ ( D ~ − 1 / 2 W ~ D ~ − 1 / 2 ⏟ R n × n ) x ⏟ R n × 1
\underbrace{g_{\theta^{\prime}} * x}_{\mathbb{R}^{n \times 1}}=\theta\left(\underbrace{\tilde{D}^{-1 / 2} \tilde{W} \tilde{D}^{-1 / 2}}_{\mathbb{R}^{n \times n}}\right) \underbrace{x}_{\mathbb{R}^{n \times 1}}
R n × 1 g θ ′ ∗ x = θ ( R n × n D ~ − 1 / 2 W ~ D ~ − 1 / 2 ) R n × 1 x 推廣到多通道 ，相當於每一個節點的信息是向量x ∈ R N × 1 → X ∈ R N × C
x \in \mathbb{R}^{N \times 1} \rightarrow X \in \mathbb{R}^{N \times C}
x ∈ R N × 1 → X ∈ R N × C N N N 數量 ，C C C 維度數 。X \mathbf{X} X 特徵矩陣 。
相應的卷積核參數變化θ ∈ R → Θ ∈ R C × F
\theta \in \mathbb{R} \rightarrow \Theta \in \mathbb{R}^{C \times F}
θ ∈ R → Θ ∈ R C × F F F F
那麼卷積結果寫成矩陣形式爲Z ⏟ R N × F = D ~ − 1 / 2 W ~ D ~ − 1 / 2 ⏟ R N × N X ⏟ R N × C Θ ⏟ R C × F
\underbrace{Z}_{\mathbb{R}^{N \times F}}=\underbrace{\tilde{D}^{-1 / 2} \tilde{W} \tilde{D}^{-1 / 2}}_{\mathbb{R}^{N \times N}} \underbrace{X}_{\mathbb{R}^{N \times C}} \underbrace{\mathbf{\Theta}}_{\mathbb{R}^{C \times F}}
R N × F Z = R N × N D ~ − 1 / 2 W ~ D ~ − 1 / 2 R N × C X R C × F Θ 可以疊加多層，對上述輸出激活一下，就可以作爲下一層節點的特徵矩陣 。
這一代GCN特點：
取K = 1 K=1 K = 1 3 × 3 3\times{3} 3 × 3
由於卷積核寬度減小，可以通過增加捲積層數來擴大感受野，從而增強網絡的表達能力。
增加了參數約束，比如λ max ≈ 2 \lambda_{\max } \approx 2 λ max ≈ 2
論文采用兩層的GCN，用來在graph上進行半監督的節點分類任務，鄰接矩陣爲A A A A ^ = D ~ − 1 2 A ~ D ~ − 1 2 \hat{A}=\tilde{D}^{-\frac{1}{2}} \tilde{A} \tilde{D}^{-\frac{1}{2}} A ^ = D ~ − 2 1 A ~ D ~ − 2 1 Z = f ( X , A ) = softmax ( A ^ ReLU ( A ^ X W ( 0 ) ) W ( 1 ) )
Z=f(X, A)=\operatorname{softmax}\left(\hat{A} \operatorname{ReLU}\left(\hat{A} X W^{(0)}\right) W^{(1)}\right)
Z = f ( X , A ) = s o f t m a x ( A ^ R e L U ( A ^ X W ( 0 ) ) W ( 1 ) ) W ( 0 ) ∈ R C × H W^{(0)} \in \mathbb{R}^{C \times H} W ( 0 ) ∈ R C × H H H H W ( 1 ) ∈ R H × F W^{(1)} \in \mathbb{R}^{H \times F} W ( 1 ) ∈ R H × F softmax ( x i ) = 1 Z exp ( x i ) \operatorname{softmax}\left(x_{i}\right)=\frac{1}{\mathcal{Z}} \exp \left(x_{i}\right) s o f t m a x ( x i ) = Z 1 exp ( x i ) Z = ∑ i exp ( x i ) \mathcal{Z}=\sum_{i} \exp \left(x_{i}\right) Z = ∑ i exp ( x i ) 交叉熵損失函數 爲L = − ∑ l ∈ Y L ∑ f = 1 F Y l f ln Z l f
\mathcal{L}=-\sum_{l \in \mathcal{Y}_{L}} \sum_{f=1}^{F} Y_{l f} \ln Z_{l f}
L = − l ∈ Y L ∑ f = 1 ∑ F Y l f ln Z l f Y L \mathcal{Y}_{L} Y L
C C C F F F Y 1 Y_1 Y 1 Y 2 Y_2 Y 2
論文在如下幾個任務中進行實驗
在citation network中進行半監督的document classification。
在從knowledge graph中提取的bipartite graph中進行半監督的entity classification
實驗數據說明如下
結果如下：
可以看出，在比較的幾種算法中，論文GCN的在準確率和時間上都最好。
該模型對每一個節點(或邊、或圖)採用H個hop的矩陣進行表示，每一個hop都表示該鄰近範圍的鄰近信息，由此，對於局部信息的獲取效果比較好，得到的節點的representation的表示能力很強。
假設有如下定義
一個graph數據集G = { G t ∣ t ∈ 1 … T } \mathcal{G}=\left\{G_{t} | t \in 1 \ldots T\right\} G = { G t ∣ t ∈ 1 … T }
graph定義爲G t = ( V t , E t ) G_{t}=\left(V_{t}, E_{t}\right) G t = ( V t , E t ) V t V_t V t E t E_t E t
所有節點的特徵矩陣定義爲X t X_t X t N t × F N_t\times{F} N t × F N t N_t N t G t G_t G t F F F
邊信息E t E_t E t N t × N t N_t\times{}N_t N t × N t A t A_t A t 節點度(degree)歸一化 的轉移概率矩陣P t P_t P t i i i j j j
對於graph來說沒有任何限制，graph可以是帶權重的或不帶權重的，有向的或無向的。
模型的目標爲預測Y Y Y 在每一種情況中，模型輸入部分帶有標籤的數據集合，然後預測剩下的數據的標籤 。
DCNN模型輸入圖G \mathcal{G} G Y Y Y P ( Y ∣ X ) \mathbb{P}(Y|X) P ( Y ∣ X ) H × F H\times{}F H × F H H H t t t N t × H × F N_t\times{H}\times{F} N t × H × F Z t Z_t Z t Z t Z_t Z t H × F H\times{F} H × F Z t Z_t Z t M t × H × F M_t\times{H}\times{F} M t × H × F
節點分類任務 ，假設P t ∗ P_t^* P t ∗ P t P_t P t N t × H × N t N_t\times{H}\times{N_t} N t × H × N t t t t i i i j j j k k k Z t i j k Z_{tijk} Z t i j k Z t i j k = f ( W j k c ⋅ ∑ l = 1 N t P t i j l ∗ X t l k )
Z_{t i j k}=f\left(W_{j k}^{c} \cdot \sum_{l=1}^{N_{t}} P_{t i j l}^{*} X_{t l k}\right)
Z t i j k = f ( W j k c ⋅ l = 1 ∑ N t P t i j l ∗ X t l k ) Z t = f ( W c ⊙ P t ∗ X t )
Z_{t}=f\left(W^{c} \odot P_{t}^{*} X_{t}\right)
Z t = f ( W c ⊙ P t ∗ X t ) ⊙ \odot ⊙ H H H O ( H × F ) O(H\times{F}) O ( H × F ) 使得diffusion-convolutional representation不受輸入大小的限制 。
在計算出Z Z Z Y Y Y Y ^ \hat{Y} Y ^ P ( Y ∣ X ) \mathbb{P}(Y|X) P ( Y ∣ X ) Y ^ = arg max ( f ( W d ⊙ Z ) )
\hat{Y}=\arg \max \left(f\left(W^{d} \odot Z\right)\right)
Y ^ = arg max ( f ( W d ⊙ Z ) )
P ( Y ∣ X ) = softmax ( f ( W d ⊙ Z ) )
\mathbb{P}(Y | X)=\operatorname{softmax}\left(f\left(W^{d} \odot Z\right)\right)
P ( Y ∣ X ) = s o f t m a x ( f ( W d ⊙ Z ) )
對於圖分類任務 ，直接採用所有節點表示的均值作爲graph的representationZ t = f ( W c ⊙ 1 N t T P t ∗ X t / N t )
Z_{t}=f\left(W^{c} \odot 1_{N_{t}}^{T} P_{t}^{*} X_{t} / N_{t}\right)
Z t = f ( W c ⊙ 1 N t T P t ∗ X t / N t ) 1 N t 1_{N_t} 1 N t N t × 1 N_t\times{1} N t × 1
對於邊分類任務 ，通過將每一條邊轉化爲一個節點來進行訓練和預測 ，這個節點與原來的邊對應的首尾節點相連，轉化後的圖的鄰接矩陣A t ′ A_t' A t ′ A t A_t A t incidence matrix 得到A t ′ = ( A t B t T B t 0 )
A_{t}^{\prime}=\left(\begin{array}{cc}{A_{t}} & {B_{t}^{T}} \\ {B_{t}} & {0}\end{array}\right)
A t ′ = ( A t B t B t T 0 ) A t ′ A_t' A t ′ P t ′ P_t' P t ′ P t P_t P t
對於模型訓練 ，使用梯度下降法，並採用early-stop方式得到最終模型。
節點分類任務的實驗數據集使用Cora和Pubmed數據集，包含scientific papers(相當於node)、citations(相當於edge)和subjects(相當於label)。實驗評估標準使用分類準確率以及F1值。
節點分類的實驗結果 如下
可以看出使用各種評估標準，DCNN效果都是最好的。
圖分類任務 的實驗結果如下
優點 ：
缺點 ：
內存佔用大 ：DCNN建立在密集的張量計算上，需要存儲大量的張量，需要O ( N t 2 H ) O(N_t^2H) O ( N t 2 H ) 長距離信息傳播不足 ：模型對於局部的信息獲取較好，但是遠距離的信息傳播不足。
將序列型的LSTM模型擴展到樹型的LSTM模型 ，簡稱Tree-LSTM，並根據孩子節點是否有序，論文提出了兩個模型變體，Child-Sum Tree-LSTM模型和N-ary Tree-LSTM模型 。和序列型的LSTM模型的主要不同點在於，序列型的LSTM從前一時刻獲取隱藏狀態h t h_t h t
Tree-LSTM模型對於每一個孩子節點都會產生一個“遺忘門”f j k f_{jk} f j k 從所有的孩子節點選擇性地獲取信息和結合信息 。
該模型的更新方程如下h ~ j = ∑ k ∈ C ( j ) h k i j = σ ( W ( i ) x j + U ( i ) h ~ j + b ( i ) ) f j k = σ ( W ( f ) x j + U ( f ) h k + b ( f ) ) o j = σ ( W ( o ) x j + U ( o ) h ~ j + b ( o ) ) u j = tanh ( W ( u ) x j + U ( u ) h ~ j + b ( u ) ) c j = i j ⊙ u j + ∑ k ∈ C ( j ) f j k ⊙ c k h j = o j ⊙ tanh ( c j )
\begin{aligned} \tilde{h}_{j} &=\sum_{k \in C(j)} h_{k} \\ i_{j} &=\sigma\left(W^{(i)} x_{j}+U^{(i)} \tilde{h}_{j}+b^{(i)}\right) \\ f_{j k} &=\sigma\left(W^{(f)} x_{j}+U^{(f)} h_{k}+b^{(f)}\right) \\ o_{j} &=\sigma\left(W^{(o)} x_{j}+U^{(o)} \tilde{h}_{j}+b^{(o)}\right) \\ u_{j} &=\tanh \left(W^{(u)} x_{j}+U^{(u)} \tilde{h}_{j}+b^{(u)}\right) \\ c_{j} &=i_{j} \odot u_{j}+\sum_{k \in C(j)} f_{j k} \odot c_{k} \\ h_{j} &=o_{j} \odot \tanh \left(c_{j}\right) \end{aligned}
h ~ j i j f j k o j u j c j h j = k ∈ C ( j ) ∑ h k = σ ( W ( i ) x j + U ( i ) h ~ j + b ( i ) ) = σ ( W ( f ) x j + U ( f ) h k + b ( f ) ) = σ ( W ( o ) x j + U ( o ) h ~ j + b ( o ) ) = tanh ( W ( u ) x j + U ( u ) h ~ j + b ( u ) ) = i j ⊙ u j + k ∈ C ( j ) ∑ f j k ⊙ c k = o j ⊙ tanh ( c j ) C ( j ) C(j) C ( j ) j j j h k h_k h k k k k i j i_j i j j j j f j k f_{jk} f j k j j j k k k o j o_j o j j j j
這裏的關鍵點在於第三個公式的f j k f_{jk} f j k j j j k k k c j c_j c j
由於該模型是對所有的孩子節點求和，所以這個模型對於節點順序不敏感的 ，適合於孩子節點無序的情況。
假如一個樹的最大分支數爲N N N N N N j j j k k k h j k h_{jk} h j k c j k c_{jk} c j k i j = σ ( W ( i ) x j + ∑ ℓ = 1 N U ℓ ( i ) h j ℓ + b ( i ) ) f j k = σ ( W ( f ) x j + ∑ ℓ = 1 N U k ℓ ( f ) h j ℓ + b ( f ) ) o j = σ ( W ( o ) x j + ∑ ℓ = 1 N U ℓ ( o ) h j ℓ + b ( o ) ) u j = tanh ( W ( u ) x j + ∑ ℓ = 1 N U ℓ ( a ) h j ℓ + b ( a ) ) c j = i j ⊙ u j + ∑ ℓ = 1 N f j ℓ ⊙ c j ℓ h j = o j ⊙ tanh ( c j )
\begin{aligned} i_{j} &=\sigma\left(W^{(i)} x_{j}+\sum_{\ell=1}^{N} U_{\ell}^{(i)} h_{j \ell}+b^{(i)}\right) \\ f_{j k} &=\sigma\left(W^{(f)} x_{j}+\sum_{\ell=1}^{N} U_{k \ell}^{(f)} h_{j \ell}+b^{(f)}\right) \\ o_{j} &=\sigma\left(W^{(o)} x_{j}+\sum_{\ell=1}^{N} U_{\ell}^{(o)} h_{j \ell}+b^{(o)}\right) \\ u_{j} &=\tanh \left(W^{(u)} x_{j}+\sum_{\ell=1}^{N} U_{\ell}^{(a)} h_{j \ell}+b^{(a)}\right) \\ c_{j} &=i_{j} \odot u_{j}+\sum_{\ell=1}^{N} f_{j \ell} \odot c_{j \ell} \\ h_{j} &=o_{j} \odot \tanh \left(c_{j}\right) \end{aligned}
i j f j k o j u j c j h j = σ ( W ( i ) x j + ℓ = 1 ∑ N U ℓ ( i ) h j ℓ + b ( i ) ) = σ ( W ( f ) x j + ℓ = 1 ∑ N U k ℓ ( f ) h j ℓ + b ( f ) ) = σ ( W ( o ) x j + ℓ = 1 ∑ N U ℓ ( o ) h j ℓ + b ( o ) ) = tanh ( W ( u ) x j + ℓ = 1 ∑ N U ℓ ( a ) h j ℓ + b ( a ) ) = i j ⊙ u j + ℓ = 1 ∑ N f j ℓ ⊙ c j ℓ = o j ⊙ tanh ( c j ) 該模型爲每個孩子節點都單獨地設置了參數U l U_{l} U l 。
分類任務定義爲在類別集Y \mathcal{Y} Y y ^ \hat{y} y ^ j j j y ^ j \hat{y}_j y ^ j h j h_j h j p ^ θ ( y ∣ { x } j ) = softmax ( W ( s ) h j + b ( s ) ) y ^ j = arg max y p ^ θ ( y ∣ { x } j )
\begin{aligned} \hat{p}_{\theta}\left(y |\{x\}_{j}\right) &=\operatorname{softmax}\left(W^{(s)} h_{j}+b^{(s)}\right) \\ \hat{y}_{j} &=\arg \max _{y} \hat{p}_{\theta}\left(y |\{x\}_{j}\right) \end{aligned}
p ^ θ ( y ∣ { x } j ) y ^ j = s o f t m a x ( W ( s ) h j + b ( s ) ) = arg y max p ^ θ ( y ∣ { x } j ) J ( θ ) = − 1 m ∑ k = 1 m log p ^ θ ( y ( k ) ∣ { x } ( k ) ) + λ 2 ∥ θ ∥ 2 2
J(\theta)=-\frac{1}{m} \sum_{k=1}^{m} \log \hat{p}_{\theta}\left(y^{(k)} |\{x\}^{(k)}\right)+\frac{\lambda}{2}\|\theta\|_{2}^{2}
J ( θ ) = − m 1 k = 1 ∑ m log p ^ θ ( y ( k ) ∣ { x } ( k ) ) + 2 λ ∥ θ ∥ 2 2 m m m λ \lambda λ L 2 L2 L 2
該任務給定一個句子對(sentence pair)，模型需要預測出一個範圍在[ 1 , K ] [1,K] [ 1 , K ]
論文首先對每一個句子產生一個representation，兩個句子的表示分別用h L h_L h L h R h_R h R ( h L , h R ) (h_L,h_R) ( h L , h R ) h × = h L ⊙ h R h + = ∣ h L − h R ∣ h s = σ ( W ( × ) h × + W ( + ) h + + b ( h ) ) p ^ θ = softmax ( W ( p ) h s + b ( p ) ) y ^ = r T p ^ θ
\begin{aligned} h_{ \times} &=h_{L} \odot h_{R} \\ h_{+} &=\left|h_{L}-h_{R}\right| \\ h_{s} &=\sigma\left(W^{( \times)} h_{ \times}+W^{(+)} h_{+}+b^{(h)}\right) \\ \hat{p}_{\theta} &=\operatorname{softmax}\left(W^{(p)} h_{s}+b^{(p)}\right) \\ \hat{y} &=r^{T} \hat{p}_{\theta} \end{aligned}
h × h + h s p ^ θ y ^ = h L ⊙ h R = ∣ h L − h R ∣ = σ ( W ( × ) h × + W ( + ) h + + b ( h ) ) = s o f t m a x ( W ( p ) h s + b ( p ) ) = r T p ^ θ r T = [ 1 2 … K ] r^{T}=\left[\begin{array}{llll}{1} & {2} & {\ldots} & {K}\end{array}\right] r T = [ 1 2 … K ] θ \theta θ y ^ = r T p ^ θ ≈ y \hat{y}=r^{T} \hat{p}_{\theta} \approx y y ^ = r T p ^ θ ≈ y p p p p i = { y − ⌊ y ⌋ , i = ⌊ y ⌋ + 1 ⌊ y ⌋ − y + 1 , i = ⌊ y ⌋ 0 otherwise
p_{i}=\left\{\begin{array}{ll}{y-\lfloor y\rfloor,} & { i=\lfloor y\rfloor+ 1} \\ {\lfloor y\rfloor- y+1,} & {i=\lfloor y\rfloor} \\ { 0} & {\text { otherwise }}\end{array}\right.
p i = ⎩ ⎨ ⎧ y − ⌊ y ⌋ , ⌊ y ⌋ − y + 1 , 0 i = ⌊ y ⌋ + 1 i = ⌊ y ⌋ otherwise 1 ≤ i ≤ K 1\le{i}\le{K} 1 ≤ i ≤ K p p p p ^ θ \hat{p}_{\theta} p ^ θ J ( θ ) = 1 m ∑ k = 1 m K L ( p ( k ) ∥ p ^ θ ( k ) ) + λ 2 ∥ θ ∥ 2 2
J(\theta)=\frac{1}{m} \sum_{k=1}^{m} \mathrm{KL}\left(p^{(k)} \| \hat{p}_{\theta}^{(k)}\right)+\frac{\lambda}{2}\|\theta\|_{2}^{2}
J ( θ ) = m 1 k = 1 ∑ m K L ( p ( k ) ∥ p ^ θ ( k ) ) + 2 λ ∥ θ ∥ 2 2
對於分類任務，實驗數據使用Stanford Sentiment Treebank ，分爲五類：very negative, negative, neural, positive和very positive。在該數據集上的測試集上的準確度結果如下：
對於語義相似度度量，模型的任務是預測出兩個句子語義的相似度分數。在SICK的語義相似度子任務上的測試結果如下
根據上圖所示，頂端的圖是原始的Graph，中間的圖表示狀態向量和輸出向量的計算流圖，最下面的圖表示將更新流程迭代次，並展開之後得到等效網絡圖。
