嫌 OSS 查詢太慢？看我們如何將速度提升 10 倍！

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"內容轉載自我司官方博客 https://juicefs.com/blog/cn/posts/speed-up-oss/"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"HDFS 是 Hadoop 生態的默認存儲系統，很多數據分析和管理工具都是基於它的 API 設計和實現的。但 HDFS 是爲傳統機房設計的，在雲上維護 HDFS 一點也不輕鬆，需要投入不少人力進行監控、調優、擴容、故障恢復等一系列事情，而且還費用高昂，成本可能是對象存儲是十倍以上。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"在存儲與計算分離大趨勢下，很多人嘗試用對象存儲來構建數據湖方案，對象存儲也提供了用於 Hadoop 生態的 connector，但因爲對象存儲自身的侷限性，功能和性能都非常有限，在數據增長到一定規模後這些問題更加突出。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"JuiceFS 正是爲了解決這些問題而設計的，在保留對象存儲的雲原生特點的同時，更好地兼容 HDFS 的語義和功能，顯著提升整體性能。本文以阿里雲 OSS 爲例，給大家介紹一下 JuiceFS 是如何全面提升對象存儲在雲上大數據場景中的表現的。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"元數據性能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"爲了完整兼容 HDFS 並提供極致的元數據性能，JuiceFS 使用全內存的方式來管理元數據，將 OSS 作爲數據存儲使用，所有的元數據操作都不需要訪問 OSS 以保證極致的性能和一致性。絕大部分元數據操作的響應時間都在 1ms 以內，而 OSS 通常要幾十到一百毫秒以上。下面是使用 NNBench 進行元數據壓測的結果："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7a/7a0c15cca047909fd31c2ec095a2ccf9.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"上圖中的 rename 操作還只是針對單個文件的，因爲它要拷貝數據所以很慢。"},{"type":"text","marks":[{"type":"size","attrs":{"size":14}},{"type":"bgcolor","attrs":{"color":"#FDED8A","name":"yellow"}}],"text":"在大數據實際的任務中通常是對目錄做重命名，OSS 是 O(N) 複雜度，會隨着目錄裏文件數量的增多限制變慢，而 JuiceFS 的 rename 的複雜度是 O(1) 的, 只是服務器端的一個原子操作，不管目錄多大都可以一直這麼快。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"類似的還有 du 操作，它是要看一個目錄裏所有文件的總大小，在管理容量或者瞭解數據規模時非常有用。下圖是對一個 100GB 數據（包含3949個子目錄和文件）的目錄做 du 的時間對比，"},{"type":"text","marks":[{"type":"size","attrs":{"size":14}},{"type":"strong"}],"text":"JuiceFS 比 OSS 快 76倍！"},{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":" 這是因爲 JuiceFS 的 du 是基於服務器端內存中實時統計好的大小即時返回的，而 OSS 需要通過客戶端遍歷目錄下的所有文件再累加求和，如果目錄下的文件更多的話，性能差距會更大。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c4/c4d208b53221f2e003c2cd8401dbbd8b.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"順序讀寫性能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"大數據場景有很多原始數據是以文本格式存儲的，數據以追加方式寫入，讀取以順序讀爲主（或者是順序讀其中一個分塊）。在訪問這類文件時，吞吐能力是一個關鍵指標。爲了能夠更好地支持這樣的場景，JuiceFS 會先將它們切割成 64MB 的邏輯 Chunk，再分割成 4MB（可配置）的數據塊寫入對象存儲，這樣可以併發讀寫多個數據塊以提升吞吐量。OSS 也支持分塊上傳，但有分塊大小和分塊數量的限制，而 JuiceFS 沒有這些限制，單個文件可達 256PB。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"同時，這類文本格式的文件還非常容易被壓縮，JuiceFS 內置的 LZ4 或者 ZStandard 壓縮算法可以在並行讀寫的同時進行壓縮/解壓縮，不但可以降低存儲成本，還能減少網絡流量，進一步提升順序讀寫的性能。對於已經被壓縮過的數據，這兩個算法也能自動識別，避免重複的壓縮。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"再結合 JuiceFS 的智能預讀和回寫算法，很容易充分利用網絡帶寬和多核 CPU 的能力，將文本文件的處理性能推向極致。下圖是單線程順序 I/O 性能測試結果，顯示了 JuiceFS 對大文件（使用不能被壓縮的隨機數據）的讀寫提速是非常顯著的。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/09/099b9f8453c7da14ef2367ba8bd1aead.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"隨機讀性能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"對於分析型數倉，通常會將原始數據經過清洗後使用更爲高效的列存格式（Parquet 或者 ORC）來存儲，一方面大幅節省存儲空間，還能顯著提升分析的速度。這些列存格式的數據，在訪問模式上跟文本格式很不一樣，以隨機讀居多，對存儲系統的綜合性能有更高的要求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"JuiceFS 針對這些列存格式文件的訪問特點做了很多優化，將數據分塊緩存到計算節點的 SSD 盤上是其中最核心的一點。爲了保證緩存數據的正確性，JuiceFS 對所有寫入的數據都使用唯一的 ID 來標識 OSS 中的數據塊，並且永不修改，這樣緩存的數據就不需要失效，只在空間不足時按照 LRU 算法清理即可。Parquet 和 ORC 文件通常只有局部列是熱點，緩存整個文件或者一個 64MB 的 Chunk 會浪費空間，JuiceFS 採取的是以 1MB 分塊（可配置）爲單位的緩存機制。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}},{"type":"bgcolor","attrs":{"color":"#FDED8A","name":"yellow"}}],"text":"計算集羣中通常只會有一個緩存副本，通過一致性哈希算法來決定緩存的位置，並利用調度框架的本地優化機制來將計算任務調度到有數據緩存的節點，達到跟 HDFS 的數據本地化一樣甚至更好的效果，因爲 HDFS 的三個副本通常是隨機調度的，操作系統頁緩存的利用率會比較低，JuiceFS 的數據緩存會盡量調度到同一個節點，系統頁緩存的利用率會更高。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}},{"type":"bgcolor","attrs":{"color":"#FDED8A","name":"yellow"}}],"text":"當調度系統不能做本地化調度時，比如 SparkSQL 在讀小文件時，會隨機地把多個小文件合併到同一個任務中，就喪失了本地化特性，即使使用 HDFS 也是如此。JuiceFS 的分佈式緩存很好地解決了這個問題，當計算任務未能調度到緩存所在節點時，JuiceFS 客戶端會通過內部的 P2P 機制來訪問緩存的數據，大幅提高緩存命中率和性能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"我們選取查詢時間比較有代表性的 q2 來測試不同分塊大小和緩存設置情況的加速效果："}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4c/4cca7d49e239f5f1468d0900d2d25e52.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"當沒有啓用緩存時，使用 1MB 的分塊比 4MB 的分塊性能更好，因爲 4MB 的分塊會產生更多的讀放大，導致隨機讀變慢，也會浪費很多網絡帶寬導致網絡擁堵。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"啓用緩存後，Spark 可以直接從緩存的數據塊上做隨機讀，大大的提高了隨機讀性能。因爲 SparkSQL 會將小文件隨機合併到一個任務中，導致大部分文件沒辦法調度到有緩存的那個節點，緩存命中率很低，部分未命中緩存的讀請求只能讀對象存儲，嚴重拖慢了整個任務。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"在啓用了分佈式緩存後，不管計算任務調度到哪，JuiceFS 客戶端都能夠通過固定的節點讀到緩存的速度，緩存命中率非常高，速度也非常快（通常第二次查詢就能獲得顯著加速效果）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"JuiceFS 還支持隨機寫，但大數據場景不需要這個能力，OSS 也不支持，就不做對比了。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"綜合性能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"TPC-DS 是大數據分析場景的典型測試集，我們用它來測試一下 JuiceFS 對 OSS 的性能提升效果，包括不同數據格式和不同分析引擎。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"測試環境"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"我們在阿里雲上使用 CDH 5.16 （估計是使用最爲廣泛的版本）搭建了一個集羣，詳細配置和軟件版本如下："}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"Apache Spark 2.4.0.cloudera2\nApache Impala 2.12\nPresto 0.234\nOSS-Java-SDK 3.4.1\nJuiceFS Hadoop SDK 0.6-beta\n\nMaster: 4 CPU 32G 內存，1臺\nSlave: 4 CPU 16G 內存，200G 高效雲盤 x 2，3臺\n\nSpark 參數：\n master yarn\n driver-memory 3g\n executor-memory 9g\n executor-cores 3\n num-executors 3\n spark.locality.wait 100\n spark.dynamicAllocation.enabled false"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"測試數據集使用 100GB 的 TPC-DS 數據集，多種存儲格式和參數。完整跑完 99 條測試語句需要太多時間，我們選取了前面 10 條語句作爲代表，已經包括各種類型的查詢。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"寫入性能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"通過讀寫同一張表來測試寫入性能，使用的 SQL 語句是："}]},{"type":"codeblock","attrs":{"lang":"sql"},"content":[{"type":"text","text":"INSERT OVERWRITE store_sales SELECT * FROM store_sales;"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"我們對比了未分區的文本格式和按日期分區的 Parquet 格式，JuiceFS 都有顯著性能提升，尤其是分區的 Parquet 格式。通過分析發現，OSS 花了很多時間在 Rename 上，它需要拷貝數據，還不能併發，而 Rename 在 JuiceFS 裏是一個原子操作，瞬間完成。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/48/48a52be43353b62941e9708bf81bd58e.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"SparkSQL 查詢性能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"Apache Spark 的使用非常廣泛，我們使用 SparkSQL 來測試文本、Parquet 和 ORC 這 3 種文件格式下 JuiceFS 的提速效果，其中文本格式是未分區的，Parquet 和 ORC 格式是按照日期分區的。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7b/7b8df30fe81d6e2045a5cce5ac5b8ed4.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"對於未分區的文本格式，需要掃描全部文本數據，主要瓶頸在 CPU，JuiceFS 的提速效果有限，最高能提升 3 倍。需要注意的是，如果使用 HTTPS 訪問 OSS，Java 的 TLS 庫比 JuiceFS 使用的 Go 的 TLS 庫慢很多，同時 JuiceFS 對數據做了壓縮，網絡流量也會小很多，因此在兩者都啓用 HTTPS 來訪問 OSS 時，JuiceFS 效果更好。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/76/766bc7a7729b23f544b314f63896a236.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"上圖說明了在使用 HTTPS 的情況下，JuiceFS 的性能幾乎沒有變化，而 OSS 卻下降很多。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/12/12bf2c82b90eb256084a967657e178b2.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"對於交互式查詢，經常要對熱點數據做反覆查詢的，上圖是同一個查詢重複 3 次後的結果，JuiceFS 依靠緩存的熱點數據大幅提升性能，10 個查詢中的 8 個有幾倍的性能提升，提升幅度最少的 q4 也提升了 30%。"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e8/e82b6f2983062be0a2822d8f93518ff1.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"對 ORC 格式的數據集的提速效果跟 Parquet 格式類似，最高提速 11 倍，最少提速 40%。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}},{"type":"bgcolor","attrs":{"color":"#FDED8A","name":"yellow"}}],"text":"對所有的數據格式，JuiceFS 都能顯著提升 OSS 的查詢性能，最高超過 10 倍。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Impala 查詢性能"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"Impala 是性能非常好的交互分析引擎，對 I/O 本地化和 I/O 調度有非常好的優化，不需要使用 JuiceFS 的分佈式緩存就能夠獲得很好的效果："},{"type":"text","marks":[{"type":"size","attrs":{"size":14}},{"type":"strong"}],"text":"爲 OSS 提速 42倍！"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/62/621f4ba3f6979e8abf2088ad0df8a4fd.png","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}}],"text":"Presto 是與 Impala 類似的查詢引擎，但因爲測試環境下配置的 OSS 不能跟 Presto 工作（原因未知），JuiceFS 沒辦法與 OSS 做比較。"}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}},{"type":"bgcolor","attrs":{"color":"#FDED8A","name":"yellow"}}],"text":"彙總上面的測試結果，JuiceFS 在所有場景中都能爲 OSS 顯著提速，當存儲格式爲 Parquet 和 ORC 這類列存格式時提速尤爲明顯，寫入提升 8 倍，查詢提升可達 10 倍以上。這顯著的性能提升，不但節省了數據分析人員的寶貴時間，還能大幅減少計算資源的使用，降低成本。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"size","attrs":{"size":14}},{"type":"bgcolor","attrs":{"color":"#FDED8A","name":"yellow"}}],"text":"以上只是以阿里雲的 OSS 爲實例做了性能對比，JuiceFS 的提速能力適用於所有云的對象存儲，包括亞馬遜的 S3、谷歌雲的 GCS、騰訊雲的 COS 等，也包括各種私有云或者自研的對象存儲，JuiceFS 能顯著提升它們在數據湖場景下的性能。此外，JuiceFS 還提供了更好的 Hadoop 兼容性（比如權限控制、快照等）和完整的 POSIX 訪問能力，是雲上數據湖的理想選擇。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}