{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"目前，數據科學團隊在使用機器學習模型時正面臨着越來越大的壓力。雖然 AI 的採用和分析持續上升，但大約有87% 的數據科學項目從未投入生產。根據 Algorithmia 最近的調查顯示，22% 的公司需要一到三個月的時間來部署模型以實現業務價值，而 18% 的公司需要三個月以上的時間。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"爲了緩解這一現狀，近日，微軟宣佈了開源一款簡單的、多語言的、大規模並行的機器學習庫SynapseML（以前稱爲 MMLSpark），以幫助開發人員簡化機器學習管道的創建。微軟表示，藉助 SynapseML，開發人員可以構建“可擴展的智能”系統來解決跨領域的挑戰，包括文本分析、翻譯和語音處理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"構建可用於生產的分佈式 ML 管道通常困難，即使對於最有經驗的開發人員也是如此。而對於初學者來說，組合來自不同生態系統的工具需要大量代碼，而且許多框架在設計時並沒有考慮到服務器集羣。SynapseML 通過將多個現有的 ML 框架和新的 Microsoft 算法統一到一個可跨 Python、R、Scala 和 Java 使用的可擴展 API 中來解決這一挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"藉助 SynapseML，開發人員可以構建可擴展的智能系統來解決以下領域的挑戰："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 異常檢測"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 計算機視覺"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 深度學習"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 表格識別和人臉識別"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 梯度提升"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 微服務編排"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 模型可解釋性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 強化學習和個性化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 搜索和檢索"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 語音處理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 文本分析"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 翻譯"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"項目地址："},{"type":"link","attrs":{"href":"https:\/\/github.com\/microsoft\/SynapseML","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/github.com\/microsoft\/SynapseML"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"通過統一的 API 簡化分佈式機器學習"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"編寫容錯分佈式程序很複雜，而且是一個容易出錯的過程。我們來舉個深度網絡的分佈式評估的例子。第一步是將一個GB量級的模型發送到數百臺工作機器上，而不會使網絡不堪重負。第二步，必須協調數據讀取器以確保所有數據都排隊等待處理並且 GPU 處於滿負荷狀態。如果新計算機加入或離開集羣，新的工作機器必須接收模型的副本，數據讀取器需要適應與新機器共享工作並重新計算丟失的工作。最後，必須跟蹤進度以確保正確釋放資源。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"當然，像 Horovod 這樣的框架可以管理這一點，但是如果想要與不同的 ML 框架（例如 LightGBM、XGBoost 或 SparkML）進行比較，則需要新的環境和集羣。此外，這些訓練系統並非旨在服務或部署模型，因此需要單獨的推理和流媒體架構。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"SynapseML 通過將許多不同的 ML 學習框架與一個可擴展、與數據和語言無關且適用於批量處理、流媒體和服務應用程序的 API 統一起來，簡化了這種體驗。它旨在幫助開發人員專注於其數據和任務的高級結構上，而無需過多考慮不同 ML 生態系統和數據庫的實現細節和特性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"統一的 API 標準化了當下許多的工具、框架和算法，從而簡化了分佈式 ML 體驗。這使開發人員能夠爲需要多個框架的用例快速組合不同的 ML 框架，例如網絡監督學習、搜索引擎創建等。它還可以在單節點、多節點和可彈性調整大小的計算機集羣上訓練和評估模型，因此開發人員可以在不浪費資源的情況下擴展他們的工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"除了在幾種不同的編程語言中的可用性之外，該 API 還對各種數據庫、文件系統和雲數據存儲進行了抽象，以簡化實驗，如圖 1 所示。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/49\/78\/4980edf936fd84fe898ac3b81d067d78.gif","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"深耕五年，打造高普世化機器學習庫"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"正如微軟在項目網站上所解釋的那樣，SynapseML在幾個新方向上擴展了用於大規模數據處理的開源引擎Apache Spark：“SynapseML 中的工具允許用戶製作功能強大且高度可擴展的模型，這些模型可以跨多個機器學習生態系統完成工作。SynapseML 還爲 Spark 生態系統帶來了新的網絡功能。通過 HTTP on Spark 項目，用戶可以將任何 Web 服務嵌入到他們的 SparkML 模型中，並使用他們的 Spark 集羣進行大規模的網絡工作流。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"SynapseML 中的許多工具不需要大型標記好的訓練數據集。相反，SynapseML 爲預構建的智能服務（例如 Azure 認知服務）提供簡單的 API，以快速解決與業務和研究相關的大規模 AI 挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"SynapseML能夠讓開發者將超過 45 種不同的先進的機器學習服務直接嵌入到他們的系統和數據庫中。最新版本增加了對分佈式表單識別、對話轉錄和翻譯的支持，如圖 2 所示。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/yy\/d1\/yyd0fe064333130ca2790b2cfe4b74d1.gif","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"這些即用型算法可以解析各種文檔、實時轉錄多說話者對話，此外還能翻譯100多種不同語言的文本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"“在過去的五年中，我們一直致力於改進和穩定用於生產工作負載的 SynapseML 庫。使用 Azure Synapse Analytics 的開發人員將很高興地知曉了SynapseML 現在在這項服務上普遍可用，並提供企業服務。”微軟軟件工程師 Mark Hamilton 在一篇博客文章中寫道。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"使用 SynapseML 構建負責任的 AI 系統"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"SynapseML 使開發人員不僅可以使用現有的模型和服務，還可以構建和訓練自己的模型和服務。此版本的 SynapseML 引入了使用"},{"type":"link","attrs":{"href":"https:\/\/vowpalwabbit.org\/","title":null,"type":null},"content":[{"type":"text","text":"Vowpal Wabbit 框架"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"進行個性化推薦和上下文老虎機強化學習的新算法。這種 Vowpal Wabbit 集成可以爲單個模型分配模型訓練和預測，或者跨多個模型並行訓練。這對於快速調整策略優化和個性化系統的超參數非常有效。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Vowpal Wabbit項目地址："},{"type":"link","attrs":{"href":"https:\/\/microsoft.github.io\/SynapseML\/docs\/features\/vw\/Vowpal%20Wabbit%20-%20Overview\/","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/microsoft.github.io\/SynapseML\/docs\/features\/vw\/Vowpal%20Wabbit%20-%20Overview\/"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"構建模型後，研究人員和工程師需要在部署前瞭解其侷限性和行爲。SynapseML 通過引入新工具來幫助開發人員和研究人員構建負責任的 AI 系統，這些工具揭示了模型爲什麼會做出這樣的預測以及如何改進訓練數據集以消除偏見。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"更具體地說，SynapseML 包括 Shapley Additive Explanations (SHAP) 和 Locally Interpretable Model-Agnostic Explanations (LIME) 的分佈式實現，以解釋視覺、文本和表格模型的預測。通常，這些“黑盒”方法通常需要對每個解釋進行數千次模型評估，並且解釋大型數據集的每個預測可能需要數天時間。SynapseML 使開發人員能夠在數百臺機器上分配計算，從而顯著加快理解用戶訓練模型的過程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"除了有監督的模型可解釋性之外，SynapseML 還爲無監督的負責任 AI 引入了幾項新功能。藉助微軟的用於理解數據集不平衡的新工具，研究人員可以檢測敏感數據集特徵（例如種族或性別）是否被過度或不足表達，並採取措施提高模型公平性。此外，SynapseML 的分佈式孤立森林使研究人員能夠在不需要標記訓練數據的情況下檢測其數據集中的異常值和異常情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"“我們的目標是幫助開發人員簡化分佈式實現細節，並使他們能夠將分佈式部署到各種數據庫、集羣和語言中，而無需更改原來的代碼，”Hamilton 說道。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"參考鏈接："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/synapseml-a-simple-multilingual-and-massively-parallel-machine-learning-library\/","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/synapseml-a-simple-multilingual-and-massively-parallel-machine-learning-library\/"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/venturebeat.com\/2021\/11\/17\/microsoft-open-sources-synapseml-for-developing-ai-pipelines\/","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/venturebeat.com\/2021\/11\/17\/microsoft-open-sources-synapseml-for-developing-ai-pipelines\/"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]}]}]}
微軟開源深度學習庫 SynapseML：可直接在系統中嵌入45種不同機器學習服務、支持100多種語言文本翻譯
{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"目前，數據科學團隊在使用機器學習模型時正面臨着越來越大的壓力。雖然 AI 的採用和分析持續上升，但大約有87% 的數據科學項目從未投入生產。根據 Algorithmia 最近的調查顯示，22% 的公司需要一到三個月的時間來部署模型以實現業務價值，而 18% 的公司需要三個月以上的時間。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"爲了緩解這一現狀，近日，微軟宣佈了開源一款簡單的、多語言的、大規模並行的機器學習庫SynapseML（以前稱爲 MMLSpark），以幫助開發人員簡化機器學習管道的創建。微軟表示，藉助 SynapseML，開發人員可以構建“可擴展的智能”系統來解決跨領域的挑戰，包括文本分析、翻譯和語音處理。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"構建可用於生產的分佈式 ML 管道通常困難，即使對於最有經驗的開發人員也是如此。而對於初學者來說，組合來自不同生態系統的工具需要大量代碼，而且許多框架在設計時並沒有考慮到服務器集羣。SynapseML 通過將多個現有的 ML 框架和新的 Microsoft 算法統一到一個可跨 Python、R、Scala 和 Java 使用的可擴展 API 中來解決這一挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"藉助 SynapseML，開發人員可以構建可擴展的智能系統來解決以下領域的挑戰："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 異常檢測"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 計算機視覺"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 深度學習"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 表格識別和人臉識別"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 梯度提升"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 微服務編排"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 模型可解釋性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 強化學習和個性化"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 搜索和檢索"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 語音處理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 文本分析"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"• 翻譯"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}},{"type":"strong"}],"text":"項目地址："},{"type":"link","attrs":{"href":"https:\/\/github.com\/microsoft\/SynapseML","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/github.com\/microsoft\/SynapseML"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"通過統一的 API 簡化分佈式機器學習"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"編寫容錯分佈式程序很複雜，而且是一個容易出錯的過程。我們來舉個深度網絡的分佈式評估的例子。第一步是將一個GB量級的模型發送到數百臺工作機器上，而不會使網絡不堪重負。第二步，必須協調數據讀取器以確保所有數據都排隊等待處理並且 GPU 處於滿負荷狀態。如果新計算機加入或離開集羣，新的工作機器必須接收模型的副本，數據讀取器需要適應與新機器共享工作並重新計算丟失的工作。最後，必須跟蹤進度以確保正確釋放資源。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"當然，像 Horovod 這樣的框架可以管理這一點，但是如果想要與不同的 ML 框架（例如 LightGBM、XGBoost 或 SparkML）進行比較，則需要新的環境和集羣。此外，這些訓練系統並非旨在服務或部署模型，因此需要單獨的推理和流媒體架構。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"SynapseML 通過將許多不同的 ML 學習框架與一個可擴展、與數據和語言無關且適用於批量處理、流媒體和服務應用程序的 API 統一起來，簡化了這種體驗。它旨在幫助開發人員專注於其數據和任務的高級結構上，而無需過多考慮不同 ML 生態系統和數據庫的實現細節和特性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"統一的 API 標準化了當下許多的工具、框架和算法，從而簡化了分佈式 ML 體驗。這使開發人員能夠爲需要多個框架的用例快速組合不同的 ML 框架，例如網絡監督學習、搜索引擎創建等。它還可以在單節點、多節點和可彈性調整大小的計算機集羣上訓練和評估模型，因此開發人員可以在不浪費資源的情況下擴展他們的工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"除了在幾種不同的編程語言中的可用性之外，該 API 還對各種數據庫、文件系統和雲數據存儲進行了抽象，以簡化實驗，如圖 1 所示。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/49\/78\/4980edf936fd84fe898ac3b81d067d78.gif","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"深耕五年，打造高普世化機器學習庫"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"正如微軟在項目網站上所解釋的那樣，SynapseML在幾個新方向上擴展了用於大規模數據處理的開源引擎Apache Spark：“SynapseML 中的工具允許用戶製作功能強大且高度可擴展的模型，這些模型可以跨多個機器學習生態系統完成工作。SynapseML 還爲 Spark 生態系統帶來了新的網絡功能。通過 HTTP on Spark 項目，用戶可以將任何 Web 服務嵌入到他們的 SparkML 模型中，並使用他們的 Spark 集羣進行大規模的網絡工作流。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"SynapseML 中的許多工具不需要大型標記好的訓練數據集。相反，SynapseML 爲預構建的智能服務（例如 Azure 認知服務）提供簡單的 API，以快速解決與業務和研究相關的大規模 AI 挑戰。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"SynapseML能夠讓開發者將超過 45 種不同的先進的機器學習服務直接嵌入到他們的系統和數據庫中。最新版本增加了對分佈式表單識別、對話轉錄和翻譯的支持，如圖 2 所示。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/yy\/d1\/yyd0fe064333130ca2790b2cfe4b74d1.gif","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"這些即用型算法可以解析各種文檔、實時轉錄多說話者對話，此外還能翻譯100多種不同語言的文本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"“在過去的五年中，我們一直致力於改進和穩定用於生產工作負載的 SynapseML 庫。使用 Azure Synapse Analytics 的開發人員將很高興地知曉了SynapseML 現在在這項服務上普遍可用，並提供企業服務。”微軟軟件工程師 Mark Hamilton 在一篇博客文章中寫道。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"使用 SynapseML 構建負責任的 AI 系統"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"SynapseML 使開發人員不僅可以使用現有的模型和服務，還可以構建和訓練自己的模型和服務。此版本的 SynapseML 引入了使用"},{"type":"link","attrs":{"href":"https:\/\/vowpalwabbit.org\/","title":null,"type":null},"content":[{"type":"text","text":"Vowpal Wabbit 框架"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]},{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"進行個性化推薦和上下文老虎機強化學習的新算法。這種 Vowpal Wabbit 集成可以爲單個模型分配模型訓練和預測，或者跨多個模型並行訓練。這對於快速調整策略優化和個性化系統的超參數非常有效。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"Vowpal Wabbit項目地址："},{"type":"link","attrs":{"href":"https:\/\/microsoft.github.io\/SynapseML\/docs\/features\/vw\/Vowpal%20Wabbit%20-%20Overview\/","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/microsoft.github.io\/SynapseML\/docs\/features\/vw\/Vowpal%20Wabbit%20-%20Overview\/"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"構建模型後，研究人員和工程師需要在部署前瞭解其侷限性和行爲。SynapseML 通過引入新工具來幫助開發人員和研究人員構建負責任的 AI 系統，這些工具揭示了模型爲什麼會做出這樣的預測以及如何改進訓練數據集以消除偏見。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"更具體地說，SynapseML 包括 Shapley Additive Explanations (SHAP) 和 Locally Interpretable Model-Agnostic Explanations (LIME) 的分佈式實現，以解釋視覺、文本和表格模型的預測。通常，這些“黑盒”方法通常需要對每個解釋進行數千次模型評估，並且解釋大型數據集的每個預測可能需要數天時間。SynapseML 使開發人員能夠在數百臺機器上分配計算，從而顯著加快理解用戶訓練模型的過程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"除了有監督的模型可解釋性之外，SynapseML 還爲無監督的負責任 AI 引入了幾項新功能。藉助微軟的用於理解數據集不平衡的新工具，研究人員可以檢測敏感數據集特徵（例如種族或性別）是否被過度或不足表達，並採取措施提高模型公平性。此外，SynapseML 的分佈式孤立森林使研究人員能夠在不需要標記訓練數據的情況下檢測其數據集中的異常值和異常情況。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"“我們的目標是幫助開發人員簡化分佈式實現細節，並使他們能夠將分佈式部署到各種數據庫、集羣和語言中，而無需更改原來的代碼，”Hamilton 說道。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":"參考鏈接："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/synapseml-a-simple-multilingual-and-massively-parallel-machine-learning-library\/","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/synapseml-a-simple-multilingual-and-massively-parallel-machine-learning-library\/"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}],"text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/venturebeat.com\/2021\/11\/17\/microsoft-open-sources-synapseml-for-developing-ai-pipelines\/","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/venturebeat.com\/2021\/11\/17\/microsoft-open-sources-synapseml-for-developing-ai-pipelines\/"}],"marks":[{"type":"color","attrs":{"color":"#494949","name":"user"}}]}]}]}
發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章