雲原生 AI 賦能自動駕駛的技術探索與毫末智行落地實踐

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"毫末智行是一家致力於自動駕駛，提供智能物流解決方案的人工智能技術公司。數據智能是毫末智行的核心能力，乘用車自動駕駛系統及解決方案、低速無人車生態系統及解決方案、自動駕駛相關產品研發與定製服務三大垂類產品爲數據智能服務，同時，數據智能也反哺三大垂類產品。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/dc\/4f\/dc359cbb3ee231cdd24224364cd8b84f.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"傳統機器學習訓練性能瓶頸"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"隨着機器學習在自動駕駛業務場景中得到廣泛使用，機器學習平臺扮演了非常核心的角色。該平臺採用了存儲與計算分離的架構，使得計算資源得以與存儲資源解耦， 從而實現了靈活的資源配比以及便捷的存儲擴展，並且降低了存儲資金和運維成本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/f3\/48\/f3caa6f3c0a41ccdb0c3ed2712e87448.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"然而，這種架構也帶來了一些挑戰，其中比較關鍵的問題體現在數據訪問性能和穩定性方面："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"計算存儲分離架構導致數據訪問高延時，導致訓練慢"},{"type":"text","text":"：業務團隊使用的機器學習任務在訓練過程中要實時頻繁訪問OSS上的數據，在OSS帶寬受限或者壓力較大時， 訪問OSS上數據速度比訪問本地文件速度要慢很多；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Kubernetes調度器數據緩存無感知，同一數據源多次運行訪問依舊慢"},{"type":"text","text":": 在現實應用中深度學習任務運行會不斷 重複訪問同一數據，包括相同模型不同超參的任務、微調模型相同輸入的任務、以及AutoML任務等。這種深度學習任務的重複數據訪問就產生了可以複用的數據緩存。然而，由於原生Kubernetes調度器無法感知緩存，導致應用調度的結果不佳，緩存無法重用，性能難以提升；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"OSS成爲數據併發訪問的瓶頸點，穩定性挑戰大"},{"type":"text","text":": 毫末機器學習平臺上的大量機器學習任務在同時訓練時都會併發訪問後端OSS存儲。這種併發機器學習訓練造成的IO壓力比較大， OSS服務成爲了性能單點，一旦OSS帶寬出現瓶頸則會影響所有機器學習任務；"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"訓練文件分散，元數據壓力大"},{"type":"text","text":": 機器學習任務的訓練數據文件通常會分散在不同路徑下，讀取文件需要耗費大量的時間在list操作上。對象存儲的list操作性能較差， 因此在進行大規模list時對OSS元數據壓力很大，經常出現超時或者list失敗的情況。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在現實應用中，通過對於毫末機器學習平臺的監控分析，我們發現IO性能問題會導致GPU等昂貴計算資源不能被充分利用。機器學習自身訓練的特點導致了數據文件訪問較分散，元數據壓力較大。如果能夠精細化地緩存元數據和文件數據，那麼一方面可以提高緩存效率和磁盤利用率，另一方面也可以解決文件查找操作帶來的元數據損耗。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"開源讓智能駕駛加速擁抱雲原生 AI"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了能更好地滿足大規模機器學習模型訓練的高效性需求，模型訓練過程中需要對數據訪問取得更好的數據本地化效果。因此，我們希望達到以下目標："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"計算能夠充分利用本地化數據訪問：這樣就不需通過網絡反覆讀取，從而加速機器學習模型的訓練速度，並提升集羣的GPU使用率。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"降低OSS負載壓力：通過應用對於部分數據的本地讀取，減小數據訪問延時和降低對底層OSS的帶寬壓力。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"充分發揮熱點數據集的緩存節點優勢：在對用戶無感知的前提下，智能地將任務調度到數據緩存節點上，從而使得常用模型訓練程序越來越快。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"自定義文件列表形式緩存指定文件：只緩存訓練需要的文件，極大提高緩存使用率和磁盤利用率。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"元數據緩存和數據緩存分離：可單獨對文件進行元數據緩存，緩存策略定製化。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":6,"align":null,"origin":null},"content":[{"type":"text","text":"通過POSIX接口讀取數據：這樣無需在模型開發和訓練階段使用不同的數據訪問接口，降低開發機器學習模型程序的成本。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了達到上述目標，我們迫切希望找到Kubernetes上具有分佈式緩存加速能力的系統平臺。我們發現CNCF Sandbox項目 "},{"type":"link","attrs":{"href":"https:\/\/github.com\/fluid-cloudnative\/fluid","title":null,"type":null},"content":[{"type":"text","text":"Fluid"}]},{"type":"text","text":" 正好可以滿足我們的訴求。於是，我們設計了基於Fluid的新架構方案，經過驗證比較，我們選擇 "},{"type":"link","attrs":{"href":"https:\/\/github.com\/aliyun\/alibabacloud-jindofs\/blob\/master\/docs\/jindo_fluid\/jindo_fluid_overview.md","title":null,"type":null},"content":[{"type":"text","text":"JindoRuntime"}]},{"type":"text","text":" 作爲加速運行時。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/2b\/9d\/2bbyyd4b7ab32de5b93b326a0a2f309d.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"技術方案"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"Fluid"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/github.com\/fluid-cloudnative\/fluid","title":null,"type":null},"content":[{"type":"text","text":"Fluid"}]},{"type":"text","text":" 是一個運行在 Kubernetes 上可擴展的分佈式數據編排和加速系統，它通過數據的編排和 使用數據的應用調度，解決雲原生編排框架運行此類應用面臨數據訪問延時高、多數據源聯合分析難、應用使用數據過程複雜等痛點。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"JindoRuntime"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/github.com\/aliyun\/alibabacloud-jindofs\/blob\/master\/docs\/jindo_fluid\/jindo_fluid_overview.md","title":null,"type":null},"content":[{"type":"text","text":"JindoRuntime"}]},{"type":"text","text":" 是 Fluid 一種分佈式緩存 Runtime 的實現，基於 "},{"type":"text","marks":[{"type":"strong"}],"text":"JindoFS 分佈式緩存加速引擎"},{"type":"text","text":"。"},{"type":"link","attrs":{"href":"https:\/\/github.com\/aliyun\/alibabacloud-jindofs","title":null,"type":null},"content":[{"type":"text","text":"JindoFS"}]},{"type":"text","text":" 是阿里雲開源大數據-數據湖存儲團隊自研大數據存儲優化引擎，完全兼容 Hadoop 文件系統接口，給客戶帶來更加靈活、高效的計算存儲方案。JindoRuntime 使用 JindoFS 的 Cache 模式進行遠端文件的訪問和緩存，支持 OSS、HDFS、標準 S3 協議等多種存儲產品的訪問和緩存加速。在 Fluid 上使用和部署 JindoRuntime 流程簡單、兼容原生 K8s 環境、可以開箱即用。深度結合對象存儲特性，使用 Navite 框架優化性能，並支持免密、checksum 校驗等雲上數據安全功能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之所於選型基於JindoRuntime 的 Fluid，主要是基於以下考慮："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"Fluid 可以將數據集編排在 Kubernetes 集羣中，實現數據和計算的同置，並且提供基於 Persistent Volume Claim 接口，實現 Kubernetes 上應用 的無縫對接。同時 JindoRuntime 提供對 OSS 上數據的訪問和緩存加速能力，並且可以利用 FUSE 的 POSIX 文件系統接口實現可以像本地磁盤一樣輕鬆使用 OSS 上的海量文件，pytorch 等深度學習訓練工具可利用 POSIX 文件接口讀取訓練數據。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"提供元數據和數據分佈式緩存，可單獨進行元數據緩存預熱。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"提供元數據緩存預熱，避免訓練文件在OSS上大量元數據操作、提供數據預熱機制，避免在訓練時刻拉取數據造成的數據訪問競爭"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","text":"提供文件列表形式的數據定製化預熱，精細化預熱數據。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":5,"align":null,"origin":null},"content":[{"type":"text","text":"通過 Fluid 的數據感知調度能力，用戶無需知道緩存節點信息就可以將任務放置到有緩存數據的節點，實現數據訪問性能的優勢最大化。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"落地實踐"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"numberedlist","attrs":{"start":null,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"選擇合適的緩存節點"},{"type":"text","text":": 使用 JindoRuntime 可以獲得更好的數據本地性能，在實際生產中我們發現不是所有節點都來做緩存性能就比較好。原因是有些 節點的磁盤和網絡 IO 性能不是很好，這個時候需要我們能夠把緩存節點儘量選擇到一些大容量磁盤和網絡較好的節點上。Fluid 支持 dataset 的可調度性，換言 之，就是緩存節點的可調度性，我們通過指定 dataset 的 nodeAffinity 來進行數據集緩存節點的調度，從而保證緩存節點可高效的提供緩存服務。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"配置緩存容量與路徑"},{"type":"text","text":": 通過 dataset 的 Mounts 和 JindoRuntime 的 tieredstore 可以設定數據的掛載目錄。同時，爲避免數據量過多而導致 緩存量過於龐大，可手動配置 JindoRuntime 的 tieredstore 來約束緩存的最大容量與水位線（超過水位線的數據會被自動丟棄），tieredstore 也包含對緩存 存放路徑的設定與存儲層（SSD\/MEM\/HDD）的設定，以滿足各種場景的需要。對於多節點的場景，使用dataset 的 replacement 可以支持在同一集羣上部署多個 dataset。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"設定緩存安全策略"},{"type":"text","text":": 在Fluid中創建Dataset時，有時候我們需要在mounts中配置一些敏感信息，如 oss 賬號的 accessKeyId、accessKeySecret 。 爲了保證安全，Fluid提供使用Secret來配置這些敏感信息的能力。通過創建Secret，dataset 以 EncryptOptions 字段指定 Secret 的 name，實現對敏感信息的綁定。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":4,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"數據預加載"},{"type":"text","text":": 對於已經創建完成的 dataset 和 jindoruntime，第一次訪問掛載的數據會經歷一次下載數據目錄下全部文件的過程，這就產生了一個問題： 若數據所在的目錄存在無需使用的其他數據，會造成無意義的空間資源與網絡資源浪費。爲避免這種問題，Fluid 既支持對數據的預加載，同時也支持元數據緩存。通 過創建 dataload讀取所要預加載數據路徑信息，可以動態將數據注入。dataload 支持緩存元數據與屏蔽非預加載數據的訪問，這樣就大大降低的數據訪問效率。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"帶來性能的顯著提升"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們使用了不同的模型，推理和訓練相同的數據，分別以使用 JindoRuntime 和不使用 JindoRuntime 的方式推理和訓練，對比訓練時長，發現性能帶來了顯示提升："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/4b\/7d\/4b3fd1082cb3febb44ba8c7d46812f7d.png","alt":null,"title":"模型在雲端推理1萬幀圖片的測試結果","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/7e\/46\/7e2600849e4a4caa8c3abfyy02204f46.png","alt":null,"title":" 另一稍的模型在雲端推理1萬幀圖片的測試結果","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/4b\/f4\/4bae3d7dc9b4423c5364f1de98665cf4.png","alt":null,"title":"模型使用 4 卡在雲端訓練1萬幀圖片幀耗時測試結果","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"集成了 Fluid+JindoRuntime 後，顯著提升了雲端訓練和推理的效率，尤其是一些小模型，在雲端做訓練和推理 JindoRuntime 可以有效解決 IO 瓶頸問題，訓練速度最高可提升約 300% 。同時也大幅度提升雲端 GPU 使用效率，加速了在雲端數據驅動的迭代效率。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"共建開源生態，讓更多行業更“聰明”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"毫末機器學習訓練場景對於數據讀取有較高的性能要求，且對於元數據和數據緩存的精細化控制要求較高， 通過Fluid + JindoRuntime的緩存能力可以靈活地緩存OSS訓練文件進行元數據和數據，提供高效的元數據和數據訪問性能。 基於這種方案，我們可以實現精細化控制緩存的內容，提高生產資源利用率，不僅有效緩解了OSS帶寬的壓力，也大大提高了訓練效率。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當前 Fluid + JindoRuntime 能夠滿足生產環境的基本需要，對OSS的加速效果也比較明顯，提供的精細化緩存策略更加高效。 我們期望能夠把彈性的數據加速作爲毫末機器學習平臺的差異化競爭能力，提升整體訓練任務速度和計算資源的利用率。在未來的工作中，我們也希望也幫助社區不斷演進，幫助到更多的開發者。具體來說，後面在平臺中計劃增加的功能包括："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持定時任務支持動態擴縮容"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提供性能監控控制檯"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持規模化 K8s 集羣中多數據集的全生命週期管理"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"支持對緩存數據和緩存元數據的動態刪減"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"李範：毫末智行服務端開發工程師，負責 AI 自動訓練平臺的研發與算法優化。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"陳鐵文：毫末智行服務端開發工程師，負責 AI 自動訓練平臺的上層研發。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"關於毫末智行"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"經近 10 年的積累與全棧自研，以及 90% 以上的研發投入，毫末智行在乘用車、低速無人車、智能硬件三個方面不斷積累相關數據，目前已孵化出小魔盒、小魔駝、小魔盤等 10 餘款成熟產品。毫末智行的快速發展也體現着更高級別的智能駕駛將在更廣泛的場景中發揮作用，自動駕駛的商業應用正在駛入快車道。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"致謝"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"感謝阿里雲 JindoFS 團隊的辰山、揚禮和容器團隊的車漾在整個方案設計和優化過程中的巨大幫助，對生產過程中的需求給予定製化的支持，對遇到的各種問題進行快速的幫助和解決。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"相關鏈接"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"[1] Fluid: https:\/\/github.com\/fluid-cloudnative\/fluid"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"[2] JindoFS: https:\/\/github.com\/aliyun\/alibabacloud-jindodata"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"更多 Fluid 和 JindoFS 相關介紹請參考以上鍊接。"}]}]}