騰訊AI Lab圖神經網絡研究結果已經被ICLR-2021收錄

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"圖神經網絡已經成爲分析圖結構數據的標準框架。騰訊 AI Lab 正努力探索更加快速、魯棒、具有可解釋性深度圖學習方法，以及在生物製藥、社交網絡分析上的應用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文即是其中的一項成果，研究用圖信息瓶頸理論識別圖結構數據中關鍵子圖，論文已被ICLR-2021接收。論文題目是Graph Information Bottleneck for Subgraph Recognition。該方法能有效識別關鍵子圖，同時濾除噪聲與無關結構。該方法在圖數據解釋，提升圖分類結果，以及圖去噪等任務上取得了較好的效果。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"論文核心信息摘錄如下："}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、問題提出"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"近年來，圖神經網絡的提出使得圖學習領域得到了巨大的發展。在圖分類和圖數據預測等任務中，圖神經網絡首先在節點層面上聚合鄰居信息得到節點表徵，然後通過readout函數將所有的節點表徵轉化爲圖數據表徵。此外，diffpool等方法通過利用圖數據的層級結構，將不規則的圖結構數據通過可學習的pooling方法得到圖數據的表徵。雖然現有的方法在圖分類等任務上取得了較好的效果，但是由於利用了所有節點的信息，因此容易受到圖結構數據中冗餘、噪聲信息的影響。此外，現有方法無法判斷圖結構中哪一部分子結構最能影響圖屬性，例如在藥物分子屬性預測中，基於圖神經網絡的預測模型僅能輸出藥物分子的屬性，而無法識別。因此需要在圖數據中高效地識別最能影響圖屬性\/類別的子結構，同時濾除冗餘和噪聲信息，我們稱之爲子圖識別問題。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、子圖識別的難點"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"子圖識別的主要難點是難以獲得成對的訓練數據。人工標註一方面費時費力，例如ZINC250K數據集中有25萬分子，需要相當長的時間進行標註；另一方面需要相應的專業知識，例如分子數據中官能團的標註需要具備生物化學專業知識的專家。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/56\/564fa2d7b36df3de6081ed72b2e44521.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如何在缺少子圖標註的情況下有效的識別影響原圖屬性的子圖？"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、圖信息瓶頸"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"子圖識別問題雖然缺少直接的子圖標註，然而該問題的設置和信息瓶頸理論非常相似。信息瓶頸理論在學習數據的表徵時，通過最小化表徵與原始數據的互信息，同時最大化表徵與數據標籤的互信息，能夠得到與噪聲無關的預測性表徵。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ad\/adceb397899c3859e4cb166112ca6721.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於信息瓶頸理論，我們提出了圖信息瓶頸理論："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/8a\/8ae10681247d489def1fc9117c84e7d3.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"圖信息瓶頸最小化輸入圖與子圖的互信息，同時最大化子圖與原圖標籤的互信息，從而得到濾除噪聲與冗餘信息且最能影響原圖屬性的子圖。我們將這種子圖定義爲信息瓶頸子圖。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、優化方法"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然而，互信息難以直接計算，因此導致了難以直接優化圖信息瓶頸的目標函數。對於目標函數中的第一項，我們通過尋找互信息的下界，將互信息的最大化問題轉化爲分類損失最小化問題："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/dd\/dd70067ec7ba40f3953c8fda2db0ca35.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於目標函數中的第二項，我們需要最小化子圖與原圖的互信息，[1]在表徵學習中通過變分的方式尋找到互信息的一個上界："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/63\/6339ca056068902f23c6f9bf3a794ebd.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然而，該方法需要假設表徵的先驗分佈，例如[1]中假設表徵的先驗分佈爲標準正態分佈。然而，在子圖識別場景中，我們難以對子圖的先驗分佈給出合理的假設，因此我們採用bilevel的優化策略，在內層優化過程中訓練參數網絡估計子圖和原圖的互信息，在外層通過更新子圖最小化子圖和原圖的互信息。具體的，在內層優化中，我們首先利用圖神經網絡得到原圖與子圖的表徵，而後訓練參數網絡最大化互信息的Donsker-varadhan表示形式估計當前訓練步數中原圖與子圖的互信息，隨後在外層優化中優化子圖最小化子圖和原圖的互信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/7c\/7c5f640182cbbf2a5a060d22bacf83a4.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，圖信息瓶頸的優化目標爲："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ca\/cae09ddd4f141b146fea3371916dccc7.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"五、連續化鬆弛"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於子圖識別需要選擇性刪除原圖的節點，而節點的刪除操作爲離散變量，難以直接通過梯度的方式進行優化，因此我們設計了子圖生成器並提出了連續化鬆弛方法。從而能夠利用梯度方法優化圖信息瓶頸目標函數。子圖生成器由一個二層的圖神經網絡和一個二層的全聯接網絡組成，圖神經網絡首先得到每個節點的表徵，隨後全聯接網絡通過輸出節點分配矩陣判斷該節點是否屬於信息瓶頸子圖。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/06\/06213e758e91023c8eebf386e68a115b.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/5b\/5b10001c0aa834d92ef10d7eab66c20c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了使子圖更加緊湊並且穩定連續化鬆弛帶來的訓練不穩定問題，我們提出了連接損失目標函數。該目標函數可以使節點分配矩陣中的元素趨近於0或1，從而使訓練更加穩定，同時也能約束相鄰的節點儘可能同時位於信息瓶頸子圖內。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"六、實驗結果"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們首先在四個圖分類數據集上進行了圖分類實驗，相比於GIB能夠有效的提高baseline的分類效果。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/f7\/f7f02177160b44fd4025ed99093a3d08.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨後，我們在zinc250k數據集上進行了圖解釋實驗，即尋找最能體現分子某種屬性的子結構，相比於基於注意力機制的方法，GIB能夠更準確的識別決定分子屬性的子結構。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/95\/95180341b15e1d50b6edef1d7747117a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/ca\/cabb7b1949f4c62e0fa4e786c6edb173.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後我們進行了圖去噪實驗，GIB能有效的去除圖數據中人爲添加的噪聲邊。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/23\/230c56a739af52f94ebcd148f7e651b6.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}