乾貨 | 攜程酒店推薦模型優化

{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當用戶在線上瀏覽酒店時，作爲旅行平臺，如何挑選更合適的酒店推薦給用戶，降低其選擇的費力度，是需要考慮的一個問題。在攜程APP中，一般會觸發多種場景。在Figure 1中，我們列舉了幾種典型的場景：歡迎度排序，智能排序和搜索補償推薦。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/c3\/c3086d2cf6e60112aadada09befe5892.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"Figure1：用戶觸發的場景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"歡迎度排序"},{"type":"text","text":"：在異地場景下，當用戶沒有明確表達自己需求的情況下，默認按照歡迎度排序展示酒店；"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"智能排序"},{"type":"text","text":"：在同城或者當用戶在瀏覽上述排序結果的過程中，發現自己的興趣並沒有得到滿足的時候，往往會添加篩選條件。比如添加地標選項，從而獲得該地標附近的相關酒店。相比歡迎度排序，用戶表達了更多的個性化需求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"搜索補償推薦排序"},{"type":"text","text":"：如果用戶在當前搜索的展示結果中，還是沒有找到滿足自己需要的酒店，就會不斷往下翻頁查看更多的搜索結果。而當滿足搜索限制條件的酒店數目不足時，會觸底。這個時候就會觸發搜索補償推薦。前面兩個場景都沒有滿足用戶需求，所以在搜索補償場景中，我們需要深入挖掘用戶歷史行爲，提供更加個性化的推薦內容。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文將主要介紹我們在補償推薦場景中所做的算法優化工作。包含模型迭代、模型迭代過程中遇到的技術需求以及針對技術需求所做的一些基建等。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、推薦模型的迭代"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在酒店推薦的場景中，我們需要把滿足用戶需求的產品優先曝光給用戶，減少其使用產品的費力度。我們以用戶在酒店的TOP點擊（編者注：可以簡單理解爲用戶點擊排在TOP位置酒店的概率，TOP點擊命中率越高，用戶體驗越好）和轉化命中率（CR）作爲費力度指標；CR優化問題被建模成二分類問題，離線採取AUC和NDCG作爲模型效果評估標準。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分類問題的本質就是要找到一條決策邊界函數f(x)，來把正樣本（比如成單）和負樣本（比如沒有成單）數據分開。這裏，x就是模型特徵，好的特徵能更好地表徵正負樣本的差異，讓函數學習事半功倍；學習函數f(x)的過程就是建模過程。在本節中，我們分別介紹下我們在特徵和建模方面所嘗試的工作。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.1 特徵"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"推薦特徵使用的特徵可以分爲：用戶側特徵、物品側特徵以及用戶和物品的交互特徵。從特徵的數值特性上，又可以分成：連續值特徵和離散值特徵。算法剛開始接入的時候，我們的模型只有連續值特徵。後經歷了從完全連續值特徵，到離散特徵比重越來越重的過程。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"連續值特徵既有靜態的（比如用戶的性別，酒店價格等），也有基於用戶行爲的動態的（比如用戶點擊某個酒店的次數）。"},{"type":"text","marks":[{"type":"strong"}],"text":"連續值特徵的優點是具備良好的泛化能力。"},{"type":"text","text":"一個用戶對一個商圈的偏好可以泛化到另外一個對這個商圈有相同統計特性的用戶身上。"},{"type":"text","marks":[{"type":"strong"}],"text":"連續值特徵的缺點是缺乏記憶能力導致區分度不高。"},{"type":"text","text":"比如：在同一個商圈酒店列表中，一個用戶點擊了(A,B,C,D)，而另外一個用戶點擊了(W,X,Y,Z)。雖然兩個用戶的行爲序列不同，但是統計值特徵忘記了用戶具體點擊了哪些酒店，認爲兩個用戶都是點擊了4家酒店。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"離散值特徵是細粒度的特徵：設備ID，用戶ID，用戶點擊的item ID都可以做特徵。這樣一來不同的人，有不同的行爲就可以在特徵上有很好的區分度，因此離散值特徵是模型可以把千人千面做得更進一步的基礎。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"離散特徵的優點就是記憶力強區分度高。"},{"type":"text","text":"我們還可以讓離散特徵通過特徵組合的方式，挖掘用戶對於酒店更深層次的興趣偏好：比如點擊item A的人也喜歡item B，我們可以直接基於(A,B)生成一個組合離散特徵，來學習A和B的協同信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"離散特徵的缺點是泛化能力相對較弱。"},{"type":"text","text":"這是因爲特徵粒度太細，預測的時候命中率會偏低。同時在模型訓練的時候，細粒度的特徵相對粗粒度的特徵更容易獲得權重，讓泛化能力強的粗粒度的特徵學到的信息更少，進一步的惡化了模型的泛化能力。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"舉一個例子來說：每個樣本都有一個唯一的樣本id，如果我們以樣本id爲特徵，那麼我們模型訓練的時候，樣本id特徵可以完美的擬合label；但實際測試中，會發現因爲模型嚴重過擬合而效果非常差。這是設計離散特徵需要考慮的特徵記憶性和泛化性的tradeoff，也就是特徵儘可能細和特徵在測試數據中命中率儘可能高的tradeoff。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們結合業務場景，探索出技術方案：可以讓離散值特徵在線上也有良好的泛化能力，從而最終讓模型兼具較強的記憶能力和泛化能力。在工程方面：因爲組合特徵的存在，我們特徵空間會很大（e.g. 現在酒店推薦場景特徵可以輕鬆到億級別），對模型訓練和在線serving工程提出新的挑戰。這也是我們在推進大規模離散DNN訓練框架要解決的關鍵問題。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.2 模型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們模型經歷了從Logistic Regression (LR)，Gradient BoostingDecision Tree(GBDT)到Deep Neural Network（DNN）的迭代過程。在業務開始階段，數據量和特徵量都比較少，通常會採用LR模型。隨着算法的迭代，數據量和特徵規模越來越多的時候，基於XGBOOST或者LightGBM構建GBDT模型是業務成長期快速拿到收益的好的選擇。當數據量越來越大的時候，需要基於DNN的框架來把個性化模型做的更細。下面對三種模型的特點做一些簡單的介紹。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/bb\/bbccc8170a219c494314fbc16bb3c6c8.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"Figure2：模型決策邊界"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Figure 2中，展示了三個模型的決策邊界："}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"LR"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在LR裏，決策邊界函數是線性的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"模型的優點：可以通過模型的權重大小，解釋特徵的重要性；同時LR支持增量更新；在引入大規模離散特徵的情況下，業界在LR時代的經典做法是對LR加L1正則並通過OWLQN或者Coordinate Descent的方式進行優化，也可以通過FTRL算法讓模型稀疏避免過擬合。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"模型的缺點：線性決策邊界這個假設太強，會讓模型的精度受到限制；另外，模型的可擴展性程度低。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"GBDT"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在GBDT中，決策邊界是非線性的；模型通過將樣本空間分而治之的方式，來提高模型精度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"模型的優點：樹模型可以計算每個特徵的重要性程度，來獲得一些可解釋性；同時模型比LR有更高的精度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"模型的缺點：不支持大規模的離散特徵，不支持增量更新；模型可擴展性程度低。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們在酒店推薦場景中，嘗試了pointwise loss和pairwise loss，每次嘗試都獲得了不少的提升。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"DNN"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在DNN中，決策邊界是高度非線性的。我們知道計算機通過與或非這種簡單的邏輯，可以表達各種複雜的對象：音頻，視頻，網頁等。而DNN每一層網絡比與或非更加複雜，DNN通過多層神經元疊加，成爲一個萬能函數逼近器。在理想情況下，只要有足夠的數據量，不論我們實際的決策邊界如何複雜，我們都可以通過DNN來表達。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"同時DNN，支持增量更新，支持根據業務場景進行靈活定製各種網絡結構，支持大規模離散DNN，在離散模型中學習出來的Embedding向量還可以用在向量相似召回裏面。正因爲有這麼多的好處，DNN正在成爲業界推薦算法的標配。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個模型的缺點是：特徵經過不同層交叉，交互耦合關係過於複雜，而導致可解釋性不好；工程複雜度在我們用不同結構的時候所有不同。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面的表總結了三個模型的特點："}]},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"

模型	決策邊界	大規模離散特徵	增量\/在線學習	精度	可擴展性	可解釋性	工程複雜度
LR	線性	支持	支持	中	低	高	低
GBDT	非線性	不支持	不支持	高	低	中	中
DNN	高度非線性	支持	支持	很高	高	低	Depends"}}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"酒店推薦算法"}]},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/61\/612f26c2644ca4a428d2e99c104f7b94.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":"center","origin":null},"content":[{"type":"text","text":"Figure 3：模型結構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":"br"}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用戶在酒店的行爲包括點擊(click)和轉化(CR)。在LR和GBDT的算法迭代階段，我們優化目標只用到了CR信息。後來在DNN模型優化過程中，在模型建模中也引入了click信息，主要有兩方面的考慮："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）相對CR，click行爲更加稠密；同時click也能反映一定的用戶個性化偏好。所以合理的利用稠密的click信息，會對最終CR的學習有所幫助。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）在酒店列表中，用戶是先有點擊，再有轉化。對用戶的行爲漏斗建模，方便分析轉化不好是哪個階段導致的。對因爲點擊率偏低導致轉化低的酒店，可以做定向優化和機制上的細節調整。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在引入了click信息後，需要解決的是如何進行多目標建模的問題。因爲DNN結構靈活，我們嘗試了各種多目標學習的方式，最後採取了Figure 3所示的ESMM的模型結構。整個模型從下到上包括如下三個步驟。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）"},{"type":"text","marks":[{"type":"strong"}],"text":"離散特徵簽名"},{"type":"text","text":"：簽名是通過hash函數使“特徵-slot”映射到feasign，作爲特徵的唯一標示。每個feasign會通過模型學習得到一個K維embedding向量。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）"},{"type":"text","marks":[{"type":"strong"}],"text":"Pooling和Concat"},{"type":"text","text":"：離散特徵由單值離散和多值離散組成：單值離散是指一個slot只有一個唯一值，比如用戶id；多值離散則是一個slot會有多個值，比如用戶歷史點擊行爲。相同的slot會放到一起做sum pooling，這一操作把多值離散從N*K維向量映射爲1*K維embedding向量；再和單值離散的1*K維embedding concat連接。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3）"},{"type":"text","marks":[{"type":"strong"}],"text":"Multilayer Perceptron（MLP）和損失函數"},{"type":"text","text":"：通過pooling和concat將每條樣本映射爲固定長度的1*M維向量，通過全連接神經網絡，以交叉熵爲損失函數擬合是否點擊、轉化兩個二分類目標。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於以上的模型框架，我們相對pairwise的GBDT模型，有了顯著的提升。後來在以上模型結構的基礎上，我們針對離散特徵，做了更多模型交叉的嘗試，其中Deep Cross Network(DCN)模型結構，在用戶費力度指標和AUC\/NDCG這些模型指標上都獲得了非常大的提升。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在模型推進的過程中，我們也遇到了很多問題。包括如何做normalization、如何處理異常值和缺省值、在從LR到GBDT再到DNN升級中如何處理數據分佈變化，這些細節對最終模型線上效果都有直接的影響。當然還有一些很基礎性的困難，是跟工程團隊一起克服推進的。這個困難主要包括兩方面："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）DNN模型結構靈活，在離散特徵加成下，規模也很大。我們需要一個模型訓練平臺，支持我們定義any model並且支持any scale規模模型的訓練。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）在DNN這種複雜的非線性模型下，線上線下不一致問題會被放大。一個直接的影響是線下AUC提升很明顯，但是線上效果是負向的。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對這兩個問題，我們開發了一套大規模離散DNN模型訓練框架和統一特徵處理框架，分別解決以上兩個問題。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、推薦工程的演進"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對策略需求，推薦中臺做了很多的標準化服務。在本章節中，我們主要介紹推薦中臺裏面的大規模離散DNN訓練框架和特徵處理框架。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.1大規模離散DNN訓練框架"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大規模離散DNN模型因爲具備很高精度和靈活性，成爲目前業界做推薦算法的主流方法。但要推進大規模離散DNN在我們業務場景中落地，有一個前提：需要一個能訓練大規模離散DNN模型的框架。我們結合長期實踐經驗，開發了一套分佈式大規模離散DNN訓練平臺。這個平臺具備如下特點："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"支持任意形式的模型Any Model："},{"type":"text","text":"訓練框架前端採取tensorflow。Tensorflow允許我們定義任意在推薦中常用的網絡結構，並提供網絡結構梯度計算功能。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"支持任意規模的模型Any Scale："},{"type":"text","text":"訓練框架後端是一個異步參數服務器；服務器節點可以任意橫向擴展。服務器會做梯度融合和權重更新的工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"開箱即用的默認參數："},{"type":"text","text":"參數服務器中的參數（學習率，初始值範圍）和權重更新函數是經過長期實踐的結果；用這組默認的參數，不用調參，基本上在我們碰到的推薦業務場景都具備良好的效果。DNN調參是比較關鍵的，參數調整不合適會直接影響模型精度；而因爲DNN的複雜性，調參往往需要較長的時間摸索。我們調研的相對通用的默認參數，是目前上線快速迭代的前提。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"模型框架有如下幾個部分組成："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Front前端"},{"type":"text","text":"：採取Tensorflow，策略同學通過python編寫模型結構；tensorflow本身提供靈活的網絡結構定義和梯度計算能力。Tensorflow是我們策略擴展性的保障。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Client端"},{"type":"text","text":"：Front通過Client和Server進行通信；通信的內容就是（feasign, grads\/weights）。Client可以看作一個路由器：feasign可以看作是ip，我們根據feasign把相應的特徵分發到相應的server機器上；而grads\/weights可以看作通訊報文；而通信push\/pull可以看作http的post\/request請求。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"Server端"},{"type":"text","text":"：基於BRPC協議，用c++編寫的後臺。目前酒店推薦模型掃描一天只需要6分鐘，訓練效率高於xgb。Server端把每次請求裏相同的梯度信息做merge，並且根據自定義的優化器進行權重更新。Server可以通過增加節點來橫向擴展。Server是我們工程能力擴展性的保障。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.geekbang.org\/infoq\/3d\/3dc50cbd22dd2c1886e7528b2aa4c114.png","alt":"圖片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們構建了分佈式訓練集羣，開發了python API接口以支持用戶模型任務的創建，任務隊列的管理以及日誌監控等。現在這套大規模離散DNN的訓練框架以中臺服務的形式，同時支持着許多其他場景推薦模型的迭代。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.2 統一特徵處理框架"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"線上線下不一致是目前推薦算法應用常見的問題。出現這個問題的本質原因有兩個方面："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）輸入不一致：在業務的早期，我們並沒有完善的落日誌服務。離線訓練數據的輸入來自hive表的落盤數據；而線上的特徵是來自近線數據服務。雖然工程同學建立了一定的數據同步機制，但仍然難以保證線上和線下數據嚴格一致。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2）OP不一致：在業務早期，離線特徵處理OP的邏輯是由策略的同學在hive調度框架上以hive sql的形式實現的；而線上代碼則由策略同學跟工程的同學溝通後，由工程的同學開發的java OP。這裏面既存在策略工程兩邊同學溝通代價，也帶來了因爲不同人在實現OP細節的時候存在的不一致。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果沒有標準的框架來做特徵處理的話，每次上線都會花很多時間在分析、排查線上線下問題上。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"針對這個問題，推薦中臺專門開發了一套統一的特徵處理框架。在這個框架裏面，我們做到了數據跟框架的解耦，以方便框架做不同業務場景的擴展。數據以標準化的Protobuf（PB）協議定義，線上用戶請求的時候框架會填充PB裏面的上下文、用戶側和Item側的信息。線上用這個PB作爲特徵抽取的輸入，並同時異步落盤PB數據到hive表中。這樣一來線上線下特徵輸入都是同一份PB數據，保證輸入嚴格一致。同時我們線上線下用同一個特徵抽取JAR包，特徵配置也是同一份，這樣保證OP嚴格一致。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過這種方式，徹底地解決了線上線下不一致問題。特徵抽取框架作爲推薦中臺模型服務的重要組成部分，支持很多業務場景做標準化的特徵工程迭代。在這套框架下，工程同學專注優化框架性能，策略同學專注實現OP算子。這樣一來，策略工程同學各司其職，減少不必要的溝通代價，增加整體迭代效率。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文主要介紹了我們在推進酒店排序模型迭代過程中，在策略和工程方面做的嘗試，線上也取得了不錯的效果。實際酒店業務中，排序模型僅僅是整個系統中的一環。線上推薦系統還要綜合考慮用戶端、平臺方和酒店端的多方生態。但不論如何，準確的預估用戶的個性化偏好，是任何精細化機制策略的基礎。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在未來的工作中，我們還會面臨很多挑戰。比如酒店業務同學非常關注的黑盒模型的可解釋性問題，我們在DNN模型可解釋方面也做了一些探索性工作。隨着基建的愈加完善，我們在算法上面還會有更深入的探索。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"作者簡介："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Yorkey，攜程高級算法專家，主要從事大規模分佈式推薦系統設計和算法研發工作。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文轉載自：攜程技術中心（ID：ctriptech）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/mp.weixin.qq.com\/s\/lb6Ei_JPxhuTJ7bR8UyxgA","title":"xxx","type":null},"content":[{"type":"text","text":"乾貨 | 攜程酒店推薦模型優化"}]}]}]}