華爲雲AAAI 2021論文：一站式AI平臺ModelArts聯邦學習服務技術揭祕

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"人工智能頂級會議AAAI 2021將於2月2日-9 日在線上召開，本次會議，華爲雲AI最新聯邦學習成果“Personalized Cross-Silo Federated Learning on Non-IID Data”成功入選。這篇論文首創自分組個性化聯邦學習框架，該框架讓擁有相似數據分佈的客戶進行更多合作，並對每個客戶的模型進行個性化定製，從而有效處理普遍存在的數據分佈不一致問題，並大幅度提高聯邦學習性能。該框架已被集成至華爲雲一站式AI開發管理平臺ModelArts聯邦學習服務中。"}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"背景介紹"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"聯邦學習機制以其獨有的隱私保護機制受到很多擁有高質量數據的客戶青睞。通過聯邦學習，能有效地打破數據孤島，使數據發揮更大的作用，實現多方客戶在保證隱私的情況下共贏。但與此同時，在實際應用中各個客戶的數據分佈非常不一致，對模型的需求也不盡相同，這些在很大程度上制約了傳統聯邦學習方法的性能和應用範圍。爲此, 在客戶數據分佈不一致的情況下如何提高模型的魯棒性成爲了當前學術界與工業界對聯邦學習算法優化的核心目標，也就是希望通過聯邦學習得到的模型能滿足不同客戶的需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"傳統的聯邦學習的目的是爲了獲得一個全局共享的模型，供所有參與者使用。但當各個參與者數據分佈不一致時，全局模型卻無法滿足每個聯邦學習參與者對性能的需求，有的參與者甚至無法獲得一個比僅採用本地數據訓練模型更優的模型。這大大降低了部分用戶參與聯邦學習的積極性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了解決上述問題，讓每個參與方都在聯邦學習過程中獲益，個性化聯邦學習在最近獲得了極大的關注。與傳統聯邦學習要求所有參與方最終使用同一個模型不同，個性化聯邦學習允許每個參與方生成適合自己數據分佈的個性化模型。爲了生成這樣的個性化的模型，常見的方法是通過對一個統一的全局模型在本地進行定製化。而這樣的方法仍然依賴一個高效可泛化的全局模型，然而這樣的模型在面對每個客戶擁有不同分佈數據時卻是經常可遇而不可求的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲此，華爲雲EI溫哥華大數據與人工智能實驗室自研了一套個性化聯邦學習框架FedAMP。該框架使用獨特的自適應分組學習機制，讓擁有相似數據分佈的客戶進行更多的合作，並對每個客戶的模型進行個性化定製，從而有效地處理普遍存在的數據分佈不一致問題，並大幅度提高聯邦學習性能。下面我們來具體看下這一新的框架FedAMP是怎麼提升聯邦學習性能的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"論文地址：https:\/\/arxiv.org\/abs\/2007.03797"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/1c\/07\/1cd1a010869fb3f49a75ae3b47a73a07.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖一： FedAMP的注意消息傳遞機制"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"算法介紹"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在這個新的聯邦學習框架FedAMP中，作者首先引入了一種新穎的注意消息傳遞機制（Attentive message passing mechanism）。如圖一所示，這種機制允許每個客戶在擁有本地個性化模型, 同時在雲端維持一個個性化的雲端模型。FedAMP通過計算本地個性化模型兩兩之間的相似度來實現注意消息傳遞機制，從而使雲端可以利用注意消息傳遞機制聚合本地個性化模型，得到雲端個性化模型， 然後再通過本地個性化訓練拉近本地個性化模型與雲端個性化模型之間的距離。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/a7\/34\/a7e50fdf660d6e490081251c57043434.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖二：FedAMP僞代碼"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於上述描述，圖二給出了FedAMP僞代碼。不難看出，在FedAMP的迭代中實現了一種正反饋循環，即擁有相似模型參數的客戶將逐步形成越來越緊密合作。這樣的合作將自適應地隱性地將相似的客戶組合起來並因此形成更爲高效的合作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"文章在此基礎上給出了FedAMP框架的收斂性證明，並進一步針對深度學習網絡提出了一套啓發式個性化聯邦學習框架HeurFedAMP。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/98\/e3\/982cd785d06128a2fbc3bffdaf850ee3.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖三：最優平均測試準確率"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"結果展示"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了評估FedAMP及HeurFedAMP的性能，作者設計了一套更爲符合實際應用場景的非均勻數據分佈。如圖三所示，FedAMP及HeurFedAMP在四個常見數據集上展示了比現有七種SOTA算法更高的最優平均測試準確率。相比 Google 提出的原始聯邦學習框架FedAvg，FedAMP及HeurFedAMP所獲得的最優平均測試準確率更是大幅提升，表現非常亮眼。"}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/f7\/89\/f7f6cfd4f2da72570aee2d4dc4e85e89.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖四：所有客戶測試準確率分佈"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過分析進一步統計的結果（如圖四），作者發現通過FedAMP和HeurFedAMP所得到的模型對於每個客戶的測試精度在統計上顯著高於其他方法獲得的結果。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/89\/48\/89d70f99c00bf23ee219fd6660d2ac48.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖五：對於EMNIST數據集的可視化分組結果"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了更好的理解FedAMP及HeurFedAMP的機理， 作者進一步分析了注意消息傳遞機制（如圖五）。作者發現FedAMP和HeurFedAMP均成功發現了蘊含在客戶之間的真實分組關係。這一發現進一步解釋了FedAMP及HeurFedAMP在數據分佈不均勻時性能卓越的原因。聯邦學習三步驟，降低使用門檻基於華爲雲ModelArts平臺，實現聯邦學習僅需簡單的三步操作：第一步：發起者創建一個聯邦學習團隊，定義聯邦任務，並邀請參與者，如圖六所示(其中更新策略可配置FedAVG，FedAMP等)："}]},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/32\/80\/32cb8af8ba15eb2dd09804721be2a780.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖六：基於ModelArts的聯邦訓練任務創建"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第二步：參與者同意加入聯邦團隊，並配置數據及資源類型，如圖七所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/15\/9c\/15f9606e8a8107cff1ed3f1691e2319c.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/3b\/5e\/3b0e30d59da1dc7a7c2f3b74fb0c1f5e.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖七：基於ModelArts的聯邦學習團隊加入"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"第三步：聯邦訓練發起者啓動聯邦訓練，直至訓練完成，如圖八所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/a0\/95\/a004e5ce9cc8982fec11cc48e9a2b295.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"圖八：基於ModelArts的聯邦學習訓練"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"FedAMP\/HeurFedAMP是兩種簡單高效的個性化聯邦學習框架。通過注意消息傳遞機制，FedAMP\/HeurFedAMP還將天然擁有抗投毒潛力。其在數據分佈不均勻時的優異表現，將爲雲產商吸引更多擁有高質量數據的客戶參與聯邦學習。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於上述框架，華爲雲一站式AI開發 ModelArts提供聯邦學習特性，用戶各自利用本地數據訓練，不交換數據本身，只用加密方式交換更新的模型參數，實現聯合建模。"},{"type":"link","attrs":{"href":"https:\/\/console.huaweicloud.com\/modelarts\/?region=cn-north-4#\/notebook\/loading?share-url-b64=aHR0cHM6Ly9jbm5vcnRoNC1tb2RlbGFydHMtc2RrLm9icy5jbi1ub3J0aC00Lm15aHVhd2VpY2xvdWQuY29tL3NuYXBzaG90L3B5dG9yY2hfZmVkYW1wX2VtbmlzdF9jbGFzc2lmaWNhdGlvbi5pcHluYg==","title":"xxx","type":null},"content":[{"type":"text","text":"算法體驗鏈接"}]}]}]}