回顧經典，Netflix的推薦系統架構

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大家好，這篇文章我們回顧一篇經典博客，Netflix官方博客介紹的推薦系統架構，雖然文章發佈已有六年， 但是現在回看起來我自己還是蠻驚訝的，因爲Netflix的推薦系統架構居然到現在依然是主流。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當然，框架中的諸多技術在不斷的迭代更新，因爲近期與Netflix的很多同行有過交流，也可以更新一下框架中一些模塊的最新進展。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"Netflix推薦系統架構圖"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fd/fd43611c40828b0f23f547cf66a4ca95.jpeg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Netflix推薦系統架構圖"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Netflix的推薦系統從上至下依次分爲離線（offline）、近線（nearline），在線（online）三部分。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"離線（offline）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"存儲離線數據，利用大數據查詢工具進行數據查詢和處理，離線模型訓練。離線部分對於數據數量和算法複雜度限制很少，以批量方式完成數據處理，但是數據處理的實時性非常差，無法做到數據和模型的即使更新。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到當時還是hive，pig等工具的天下，現在spark 是主流，但也有越來越多的offline job被合併到near line之中，可以說當前的offline和nearline的界限日漸模糊了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"近線（near line）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於數據消息隊列，利用一些流計算平臺進行數據的準實時處理。它居於離線和在線之間，既可以以分鐘級別甚至秒級的延時來準實時地處理數據，也有一定的數據批量處理能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"nearline可以說是近幾年大數據架構發展的重中之重了。當時Netflix開發了自己的流處理框架Manhattan，但現在已經是Flink一統天下的時候，Netflix內部的Flink平臺每天會運行上千個不同的流處理任務。涵蓋了"},{"type":"text","marks":[{"type":"strong"}],"text":"特徵實時計算"},{"type":"text","text":"、"},{"type":"text","marks":[{"type":"strong"}],"text":"數據監控"},{"type":"text","text":"、"},{"type":"text","marks":[{"type":"strong"}],"text":"BI"},{"type":"text","text":"、"},{"type":"text","marks":[{"type":"strong"}],"text":"模型實時訓練"},{"type":"text","text":"等等。越來越多的offline任務被替代，也許Kappa架構徹底替代Lambda架構的日子不太遠了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"在線（online）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"online部分的主要任務是進行用戶請求的實時處理，模型的在線服務。在線部分需要更快地響應最近的事件和用戶交互，因此對於延遲的要求比較苛刻，一般都要求在100ms以內完成所有處理，這會限制所用算法的複雜性和可處理的數據量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"正是online部分極高的響應延遲要求和相比離線、近線較弱的數據處理能力，要求online部分採用不同的高效的model serving方法去支持個性化推薦服務。這也是前段時間我們專欄花了幾篇文章介紹model serving的原因。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://zhuanlan.zhihu.com/p/111929212","title":null},"content":[{"type":"text","text":"從阿里的User Interest Center看模型線上實時serving方法"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://zhuanlan.zhihu.com/p/77664408","title":null},"content":[{"type":"text","text":"如何解決推薦系統工程難題——深度學習推薦模型線上serving？"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"AWS"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大家要注意架構圖右上角的AWS的標誌，它意味着Netflix的所有服務器和大數據設施都是架構在amazon的雲平臺上的，而且一直沿用至今。作爲AWS的第一大用戶，Netflix服務的雲化還是非常徹底的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有的時候我也挺佩服美國這些互聯網公司的選擇，像Netflix、Pinterest這些公司，已經是不折不扣的互聯網巨頭，居然非常放心的使用AWS，AWS確實能夠提供非常專業安全的雲服務。這樣開放的精神還是讓我挺感慨的。國內的阿里雲發展當然也非常好，但是巨頭級別的公司完全依賴阿里雲的案例還是不多，從這一點上，國內和國外整個互聯網的氛圍還是有一些微妙的區別。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"不同層之間的配合與系統的整體性"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到，從離線到在線，"},{"type":"text","marks":[{"type":"strong"}],"text":"數據的實時性從上到下依次增強"},{"type":"text","text":"，而"},{"type":"text","marks":[{"type":"strong"}],"text":"數據規模和處理能力從上到下依次減弱"},{"type":"text","text":"。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但作爲同一個系統之中的不同功能層，只有整合發揮不同層的優勢，才能夠讓系統整體發揮出最大的作用。所以我們可以在架構圖中看到很多躍層的調用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e0/e0e7762a9a4606e00f3d06b9f6b0e529.jpeg","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"比如從online 到nearline和offline通過用戶消息隊列（User Event Queue，現在基本都使用Kafka）來緩存數據流，這是連接online和其他層的接口。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而從nearline和offline中連接online的接口則是algorithm service，online data service，以及model到online層的接口。他們分別存儲了算法結果，數據特徵和模型文件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在架構圖正中央的存儲部分，也有不同的數據庫作爲數據中心作爲不同模塊的數據交換接口。比如cassandra更適宜存儲大數據量的nosql數據，mysql當然是適合結構化的小數據量數據，而EVcache則作爲內存數據庫當作數據緩存使用。當然，技術的發展使得現在已經有更合適的技術選型，AWS的dynamoDB，以及redis都可以作爲更好的替代方案。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後的總結就直接用netflix官方博客的總結吧。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"We want the ability to use sophisticated machine learning algorithms that can grow to arbitrary complexity and can deal with huge amounts of data. We also want an architecture that allows for flexible and agile innovation where new approaches can be developed and plugged-in easily. Plus, we want our recommendation results to be fresh and respond quickly to new data and user actions. Finding the sweet spot between these desires is not trivial: it requires a thoughtful analysis of requirements, careful selection of technologies, and a strategic decomposition of recommendation algorithms to achieve the best outcomes for our members."}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們需要具備使用複雜機器學習算法的能力，這些算法要可以適應高度複雜性，可以處理大量數據。我們還要能夠提供靈活、敏捷創新的架構，新的方法可以很容易在其基礎上開發和插入。而且，我們需要我們的推薦結果足夠新，能快速響應新的數據和用戶行爲。"},{"type":"text","marks":[{"type":"strong"}],"text":"找到這些要求之間恰當的平衡並不容易，需要深思熟慮的需求分析，細心的技術選擇，戰略性的推薦算法分解，最終才能爲客戶達成最佳的結果。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"能讓我拍案叫絕的技術經驗不多，上面的標黑部分算是一句，寫的多好，我幾乎可以認爲這是一個工程師乃至架構師的最高境界，與大家共勉。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"照例跟大家討論一個問題："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Netflix的架構從大框架上看過時了嗎？業界還有其他的推薦系統工程架構方案嗎？"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"參考資料："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://link.zhihu.com/?target=https%3A//netflixtechblog.com/system-architectures-for-personalization-and-recommendation-e081aa94b5d8","title":null},"content":[{"type":"text","text":"https://netflixtechblog.com/system-architectures-for-personalization-and-recommendation-e081aa94b5d8"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://link.zhihu.com/?target=https%3A//www.infoq.cn/article/2013%252F04%252Fnetflix-ml-architecture","title":null},"content":[{"type":"text","text":"https://www.infoq.cn/article/2013%2F04%2Fnetflix-ml-architecture"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"horizontalrule"},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"最後歡迎大家關注我的"},{"type":"text","marks":[{"type":"strong"}],"text":"微信公衆號：王喆的機器學習筆記"},{"type":"text","text":"（"},{"type":"text","marks":[{"type":"strong"}],"text":"wangzhenotes"},{"type":"text","text":"），跟蹤計算廣告、推薦系統等機器學習領域前沿。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"也歡迎大家購買我的新書《"},{"type":"text","marks":[{"type":"strong"}],"text":"深度學習推薦系統"},{"type":"text","text":"》，下面的鏈接應該還是滿100減50，期待與大家交流，也歡迎大家的批評指正。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https://item.jd.com/12630209.html","title":null},"content":[{"type":"text","text":"https://item.jd.com/12630209.html"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}