端智能研發核心套件:MNN 工作臺深度剖析

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c7/c7c73a83994db10394f5998ded9b8b81.gif","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作者:鹿尤","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"背景","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着移動互聯網的快速發展,人工智能在移動端上的應用越來越廣泛,端智能在圖像識別、視頻檢測、數據計算等核心場景發揮着重要作用。衆所周知,Python 是算法進行端智能研發的首選語言,目前阿里巴巴內已經建立了端智能的研發生態,包含 Python 虛擬機,系列的數據/視覺等運行時 Python 擴展庫、Python 任務調度庫,以及配套的任務發佈系統等等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"端智能的主要場景在數據計算、視覺內容理解這兩個領域,如果算法研發完全在 PC 端的會簡單的多,因爲 PC 環境對於 Python 的研發有官方天然的支持,但在移動端上,進行算法的部署、調試、驗證,仍處在“刀耕火種”的時代,目前算法主要通過兩種途徑:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"日誌調試:完全脫離運行時代碼,通過在代碼中插入日誌,驗證程序的運行邏輯和結果。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"PC 模型端側環境:爲了能夠進行代碼的斷點調試,在 PC 側搭建和安裝端側 Python 代碼依賴的所有庫,讓 Python 程序能獨立地運行和在和調試在 PC 端通用 IDE(如 PyCharm )上,數據輸入則使用 Mock 的方式。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過打日誌當然也能驗證結果和定位問題,但一旦工程稍微複雜點,生產效率會非常低;而脫離移動端環境,在 PC 側獨立運行,並不能反應真實的運行環境(實時數據、運行性能等),無法保證 PC 和端側數據的一致性。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上面兩種研發方式的主要問題是脫離了代碼當前的運行環境,因而會產生調試信息的缺失。我們知道,每一種語言在開發者之間的流行都離不開配套的 IDE 調試工具,就像可以使用 Xcode 調試手機應用的 OC 代碼一樣,我們需要一種能真正進行端智能算法在端側部署和調試的工具和方案,來提高 AI 研發生產效率。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/77/775353cf76eb5e09e6be35cc2dfb89d9.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"MNN 工作臺","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MNN 是阿里巴巴開源的一個輕量級深度學習端側推理引擎,核心解決深度神經網絡模型在端側推理運行問題,涵蓋深度神經網絡模型的優化、轉換和推理。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"開源地址:https://github.com/alibaba/MNN","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"除 MNN 引擎建設外,MNN 團隊還深度參與了阿里內部的端AI應用實踐。AI 應用門檻高、算法模型部署鏈路長是長期困擾我們的問題。爲了解決這兩個問題,我們將上百次實踐中長期積累的方案沉澱爲一站式 AI 研發範式 —— MNN 工作臺。(www.mnn.zone 下載 MNN 工作臺)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MNN 工作臺提供了端 AI 的研發範式,同時它也是一個對移動端 Python 的通用調試編輯器,工作臺不僅包含標準的端側 Python 調試能力,還覆蓋了阿里集團內端智能算法常用的研發部署場景,是算法研發的效率工具。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/57/577731c0568b3bf53ad5c1a1e9b5b884.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"工作臺提供了 VSCode 風格的 Python 集成開發環境,它的使用過程很簡單,一般分爲三步:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0d/0d455f9b6e67b3ff6899e633e36e7f65.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"建立連接:應用通過掃描工作臺上的二維碼,建立局域網內手機和PC的連接","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"代碼推送:工作臺上的工程通過局域網推送到端側應用中","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"運行:端側應用觸發執行端側 Python 代碼,如果設置了斷點,就可以進行端側代碼的調試了","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8e/8eba665471102ba432e7f16b978ccc58.gif","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"","title":"","type":null}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"端計算研發","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MNN 工作臺同樣爲阿里內重要的端智能場景 - 端計算的研發提供了高效的本地調試部署方案,覆蓋了 Walle/Jarvis、CV 這幾個主要的端計算場景。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"研發流程","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ea/ea010e47ab32b009f64938c2c578e17a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在此之前,端計算的研發是依賴於平臺發佈的,在開發階段將調試日誌加到代碼中,通過預發/beta發佈到平臺,然後再掃碼連接到平臺將日誌輸出,通過日誌來定位和分析問題,以此往復。這種方式相對比較原始,開發調試周期也比較長,一次修改或驗證就是一輪上傳發布的操作,而且調試信息只有日誌,整體開發效率比較低。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f4/f4e88ecab97b6e41af9ca1570dbcad6e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖爲算法使用工作臺進行端計算研發的一般過程:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"新建端計算工程,或打開從平臺 clone 的端計算工程","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"應用掃碼連接工作臺,此時工作臺中會實時輸出算法的日誌","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"在工作臺 IDE 環境中開發代碼,修改配置,添加資源","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"運行任務,此時任務會通過本地推送部署到端上。線上/預發環境都可以,任何代碼都不會影響到線上,無需擔心穩定性問題","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"代碼斷點調試、查看堆棧、變量等","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":6,"align":null,"origin":null},"content":[{"type":"text","text":"工作臺本地測試通過後,提交代碼,再進行後續的平臺發佈操作","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"工作臺有效解決了算法在開發階段的調試部署問題,它的主要優勢在於:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本地部署,開發階段可以完全脫離平臺","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持端計算工程的實時 Python 調試","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"端上部署不受預發/線上環境,或窗口期的限制","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"實時查看端側算法的運行性能","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"持久化調試","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/33/33e9f9a8d13438eafca39bc140180176.gif","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"端計算任務的觸發一般都是 UT,也有工程主動調用的,不管觸發方式如何,代碼只要推送一次,後面每次觸發都會運行調試,除非代碼有修改才需要重新推送,這就是工作臺的持久化調試能力。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"實時日誌","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ad/ad41b1ebfb472ef9f714b5418f5d0003.gif","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"工作臺支持各種類型日誌的實時打印,在設備與工作臺連接成功後就可以開始使用,支持的日誌類型包含:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Current Task:本工程日誌,包含本工程普通 print 日誌和工程的運行日誌","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Python:應用中所有的 Python 日誌","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"C:通過端計算提供的 C 接口打印的日誌","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"OC/Java:通過端計算中 Java/OC 接口打印的日誌","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"端文件瀏覽器","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/f5/f56cdbbe89fb33f2f931b90159ee18d6.gif","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"端計算研發中,經常需要查看或處理設備的本地文件,如數據庫文件、深度學習模型、資源圖片、腳本等等。工作臺中提供了可視化的設備文件瀏覽器,iOS 設備上可以查看應用的沙盒目錄,Android 設備上的瀏覽和 Android Studio 一樣,同時支持常用的目錄/文件的本地保存和路徑拷貝。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"依賴二方庫聯調","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有時算法需要去查看依賴庫代碼的運行情況,工作臺同樣能支持依賴 Python 二方庫的研發,本地部署時會將算法指定的依賴庫和工程一起推送到端上,調試的時候也能斷點到庫代碼中。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/66/66fe86650ba58f72b7fca2558cae7763.gif","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖展示了一個 Jarvis 工程依賴 Jarvis 基礎庫中的代碼進行聯調的過程,log.log 爲 Jarvis 基礎庫中的打印方法。依賴二方庫聯調隻影響到本工程,對其他工程不會產生任何影響。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"視覺內容理解研發","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Walle/Jarvis 都屬於端計算的數據場景,除此之外很大一部分算法處理的都是視頻、圖片,最常見的是攝像頭/視頻流輸入、內存中的圖片,典型的算法如圖像分類、目標檢測等,他們都是算法在計算機視覺(CV,Computer Vision)場景中的具體應用,簡稱視覺算法。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"CV 部署方式","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"端計算框架爲數據算法提供了豐富了 Python 運行時擴展庫,算法可以基於這些庫開發各種 Python 計算任務,然後部署在端上執行。而針對視覺場景,圖片和視頻流的處理一般都涉及到計算機視覺庫 OpenCV 以及函數計算庫 Numpy,之前的端側 Python 運行時庫中並不包含,所以算法在 PC 上使用 Python 開發的視覺算法並不能直接運行在端上,所以一般都是使用 SDK 集成的部署方式:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8c/8c4161f6ae3fb9d6dc6fbb27e8ab46df.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"算法使用 Python 在 PC 環境開發,通常都會基於 OpenCV/Numpy 等基礎庫,移植到端上時,雖然端上這些基礎還未支持,但可以通過一些方式轉換爲類似的 C/C++ 實現,如格式轉換使用 MNN CV 替換 OpenCV 的實現,同時也要針對端側的運行環境做一些性能優化。後面藍色的部分都需要工程同學深度參與,封裝 Android/iOS 上的 SDK,然後搭建本地的 Demo 應用驗證,再集成到應用中。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其主要缺點在於:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"視覺算法需要針對端側部分重新實現和優化,成本較高","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"算法嚴重依賴工程協作,工程維護SDK的成本也比較大(以算法爲核心的SDK)","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"算法的測試驗證需要讓工程開發App Demo,且算法一般不能直接在手淘等應用環境中測試算法代碼","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"發佈後一旦出現缺陷,無論是算法還是工程問題,迭代週期依賴應用的版本計劃,一般超過1周","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MNNKit 就是視覺算法最典型的 SDK 集成方式。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ac/ac8b63f62c6f53d429e485822ed52a63.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以往視覺算法的部署只能通過算法 C/C++ 的形式集成到應用,不過目前端側運行時已經建設完善了CV 運行時能力(OpenCV/Numpy/MNN/MNNRuntime),這樣視覺算法就可以像一個普通 Walle 任務一樣,通過工作臺或發佈平臺動態地部署到端上,典型的業務如貓客中的白屏檢測,淘寶直播中的智能看點等,都是通過這種方式部署的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過運行時部署 CV 算法的主要優勢在於:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"算法可以自己搭建視覺驗證 Playground Demo(見下文三端一體)","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"和普通端計算任務一樣,CV 算法使用 Python 開發,動態發佈,迭代週期短","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以使用工作臺本地部署,在應用中調試算法,預發/線上環境都可以","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"CV 研發流程","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/49/493422a34fcb7b763bcd12116fbb41bc.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上圖爲使用工作臺進行 CV 算法研發的完整過程,算法初步產出後,會經歷三個研發測試階段:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"單元測試:算法在獨立 Demo 中驗證視覺效果的正確性;","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"預集成測試:算法在目標應用中和關聯業務一起進行本地測試,因爲算法還未使用平臺部署方式,所以稱爲預集成測試;","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"集成測試:算法通過平臺部署,預發回歸測試,然後線上灰度/正式發佈;","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"算法在上線前在平臺進行集成測試,而單元測試和預集成測試都可以在工作臺環境中進行。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"三端一體","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"與端計算數據算法有所不同的是,新開發的 CV 模型,一般都會先在本地進行效果驗證,它是獨立於集成的應用的,就相當於在 Demo 中的單元測試。比如你開發了一個圖像算法,如目標檢測,你想在端上看一下檢測的結果對不對,定位的座標準不準,那你就需要一個 Android/iOS 的應用,把模型和代碼集成進去,然後再去開發上層代碼(典型的如 MNNKit Demo)。作爲算法並不瞭解 Android/iOS 開發,往往需要工程同學協助,協同成本較高。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有沒有一種方式能讓算法脫離工程,自己去寫一個Android/iOS 上的 Playground 驗證應用呢?","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"跨平臺效果調試庫 DebugUI","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MNN工作臺針對算法在模型效果驗證中的痛點,提供了“三端一體”的解決方案。簡單來說,我們提供了一套跨平臺效果調試的 Python 擴展庫 - DebugUI,算法可以使用 DebugUI 的 Python API,快速搭建一個視覺驗證應用,並且能在一端編寫,三端運行。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/61/61abceb78f9337a51fbe192a1b03802c.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"DebugUI 擴展庫抽象和提取了視覺效果驗證的核心鏈路,爲算法提供了了精簡又實用的 Python 接口,覆蓋了目前視覺算法實際的驗證場景:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"輸入模式:一般數據來源只有圖片和視頻","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"交互組件:用於簡單的控制,比如可以通過滑塊調整閾值,通過選擇器切換使用的模型,通過開關打開/關閉某個功能,等等","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"渲染組件:用於展示結果,比如分類得分值可以用文本渲染,人臉點位可以使用關鍵點渲染,人像分割可以用圖像渲染,等等","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據回調:攝像頭、相冊選擇,交互組件的事件,均會產生數據回調交給算法處理","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"環境變量:算法可以獲取工程設置的變量,比如某個本地文件路徑,等等","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2a/2aa3f27ab22199e011e02041f697a21c.gif","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"工作臺模型市場內置了常用的一些開箱即用算法,如漫畫臉、人臉檢測、OCR、人像分割、卡通風格等等,它們都是算法使用 DebugUI 搭建的 Playground 示例應用。上圖中展示的就是人臉檢測算法的Playground示例,在 Mac、Android、iOS 上運行都能得到相同的檢測結果,這就是“一端編寫,三端運行”。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Playground 代碼調試","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用 DebugUI 搭建的 Playground 同樣支持 Python 代碼的斷點調試,這對於算法在模型單元測試階段的開發是很有幫助。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/bf/bff800c7e7d3c121bb0474eb49072349.gif","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"CV 算法調試","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用三端一體快速搭建視覺 Playground 應用驗證算法效果後,接下來就是將算法模塊通過工作臺本地部署到集成應用(如手淘)中,應用環境會包含業務模塊的主動調用觸發算法,這樣就可以在集成應用中進行算法調試了,這個就是預集成測試的階段。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"算法通過工作臺本地調試通過後,在平臺進行預發/beta的發佈測試,這個就是走平臺部署的方式了,只不過是在應用的預發環境部署,這就是集成測試階段。集成測試通過之後,算法正式發佈上線。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/2e/2e589f564e502d63ce1adfafc1ed487f.gif","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如圖示例淘寶直播在預集成測試階段,使用工作臺進行 CV 算法的本地部署,工程主動調用算法的初始化、攝像頭每隔幾秒執行推理,都會觸發進入到對應方法的調試斷點,這樣就可以方便的在應用中調試 CV 算法了。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"性能評測","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"算法經常需要評測代碼的運行性能以針對性的進行優化,尤其是對於 CV 實時算法。MNN工作臺提供了端上 Python 代碼運行實時的性能顯示,通過“profile”方式運行 Python 工程,可以看到代碼的執行路徑,以及路徑中每一行代碼的執行時間和次數。這對於在實際應用環境中評測算法的運行性能,分析性能優化的瓶頸很有幫助","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/90/90cac54879f6d338dce801a10a9e86ef.gif","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如圖示例了淘寶直播中使用 profile 來評測 CV 算法的性能,運行時間和次數會實時地顯示在對應的代碼行。profile 適用於所有能通過工作臺部署的 Python 工程,Walle、Jarvis、CV、Playground 等工程都可以進行性能評測。後續我們會在性能評測中加入更多的端側實時信息,如內存、CPU 佔用等,幫助算法更好的分析代碼性能。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"結語","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"端智能的研發過程中,算法的開發只是一小部分,大部分工作其實都在算法之外,MNN 工作臺不僅降低了普通開發者的 AI 研發的門檻,同時也是算法研發的提效工具,它有效解決了端 Python 的開發部署困難,讓算法可以脫離工程自己搭建 Demo 應用,也支持了阿里集團內主要的端智能的研發場景,這樣算法同學可以更加專注於算法本身的開發,從而提高 AI 的研發生產效率。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"招聘啦!","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"歡迎加入阿里巴巴淘系技術部-端智能團隊,負責構建行業領先的開源推理引擎 MNN 以及一站式機器學習軟件-MNN 工作臺,在阿里我們負責核心電商的 AR 平臺及新形式的商品導航,同時還有應用規模巨大的端雲協同的搜索推薦、用戶觸達、直播內容理解等創新應用及系統。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"招聘崗位:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"算法類:CV/CG/推薦/搜索/機器學習/模型壓縮工程類:iOS/Android/Java服務端/C++/端上高性能計算簡歷投至方式","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"簡歷投遞:[email protected]","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#F5222D","name":"red"}},{"type":"strong","attrs":{}}],"text":"關注我們,每週 3 篇移動技術實踐&乾貨給你思考!","attrs":{}}]}]}
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章