libFM 的大名如雷貫耳, 然而一直沒有機會看看它的具體實現; 週末查看了一下 FM 算法的原理以及源碼實現, 現在記錄下心得. 另外發現大佬很多, 像博主 zhiyong_will 寫了一系列的文章, 比如 機器學習算法實現解析——libFM之libFM的訓練過程之SGD的方法 來介紹 libFM 的源碼實現, 看後受益匪淺. 感覺以後要是看其他的源碼, 也應該這樣來寫文章, 提煉出框架中的核心功能, 將實現原理講透徹, 代碼應配合公式推導, 不需要太在意細枝末節. 另外, UML 圖之類的也應該畫一畫.
S. Rendle. Factorization machines. In ICDM, pages995–1000, 2010.
介紹 FM 之前, 需要了解一下它被用來解決什麼問題. 在推薦/搜索/CTR預估等場景, 經常會用到 Categorical 特徵(即類別特徵), 由於單特徵的表達能力比較弱, 特徵交叉是必要的, 用來增強類別特徵的表達能力. 爲了將類別特徵轉換爲數值特徵, 我們常用 OneHot 編碼, 但由於類別特徵的種類以及取值可能衆多, 常導致編碼後的特徵空間大且稀疏. FM 模型就是用來解決稀疏數據下的特徵組合問題.
作爲對比, 考慮在線性模型中加入組合特徵, 模型可以表示爲:
y ( x ) = w 0 + ∑ i = 1 n w i x i + ∑ i = 1 n ∑ j = i + 1 n w i j x i x j
y(x) = w_0 + \sum_{i=1}^{n} w_ix_i + \sum_{i=1}^{n}\sum_{j=i+1}^{n}w_{ij}x_ix_j
y ( x ) = w 0 + i = 1 ∑ n w i x i + i = 1 ∑ n j = i + 1 ∑ n w i j x i x j
其中 n n n x x x x i x_i x i x x x i i i x i x j x_ix_j x i x j w 0 w_0 w 0 w i w_i w i w i j w_{ij} w i j
模型的參數的總個數爲:
1 + n + C n 2 = 1 + n + n ( n − 1 ) 2
1 + n + C_n^2 = 1 + n + \frac{n(n - 1)}{2}
1 + n + C n 2 = 1 + n + 2 n ( n − 1 )
由於一般樣本量大且稀疏, 二次項的權重 w i j w_{ij} w i j x i x_i x i x j x_j x j
下圖來自美團技術團隊的 深入FFM原理與實踐
如何解決二次項參數的訓練問題呢？矩陣分解提供了一種解決思路。在model-based的協同過濾中，一個rating矩陣可以分解爲user矩陣和item矩陣，每個user和item都可以採用一個隱向量表示。比如在下圖中的例子中，我們把每個user表示成一個二維向量，同時把每個item表示成一個二維向量，兩個向量的點積就是矩陣中user對item的打分。
對於一個實對稱的正定矩陣 W W W W = V T V W = V^TV W = V T V v i ∈ R k v_i\in\mathbb{R}^{k} v i ∈ R k v i v_i v i V V V i i i v i v_i v i v j v_j v j ⟨ v i , v j ⟩ \langle v_i, v_j \rangle ⟨ v i , v j ⟩ w i j w_{ij} w i j
y ( x ) = w 0 + ∑ i = 1 n w i x i + ∑ i = 1 n ∑ j = i + 1 n ⟨ v i , v j ⟩ x i x j
y(x) = w_0 + \sum_{i=1}^{n} w_ix_i + \sum_{i=1}^{n}\sum_{j=i+1}^{n}\langle v_i, v_j \rangle x_ix_j
y ( x ) = w 0 + i = 1 ∑ n w i x i + i = 1 ∑ n j = i + 1 ∑ n ⟨ v i , v j ⟩ x i x j
由於每一個特徵都對應一個隱向量, V ∈ R k × n V\in\mathbb{R}^{k\times n} V ∈ R k × n k n kn k n k ≪ n k \ll n k ≪ n n ( n − 1 ) 2 \frac{n(n - 1)}{2} 2 n ( n − 1 ) v i v_i v i x i x_i x i j ≠ i j \neq i j = i x i x j ≠ 0 x_ix_j\neq 0 x i x j = 0 v i v_i v i
如果直接去計算上式, 二次項的計算複雜度爲 O ( k n 2 ) O(kn^2) O ( k n 2 ) O ( k n ) O(kn) O ( k n )
∑ i = 1 n ∑ j = i + 1 n ⟨ v i , v j ⟩ x i x j = 1 2 ∑ i = 1 n ∑ j = 1 n ⟨ v i , v j ⟩ x i x j − 1 2 ∑ i = 1 n ⟨ v i , v i ⟩ x i x i = 1 2 ( ∑ i = 1 n ∑ j = 1 n ∑ f = 1 k v i , f v j , f x i x j − ∑ i = 1 n ∑ f = 1 k v i , f v i , f x i x i ) = 1 2 ∑ f = 1 k ( ( ∑ i = 1 n v i , f x i ) ( ∑ j = 1 n v j , f x j ) − ∑ j = 1 n v j , f 2 x j 2 ) = 1 2 ∑ f = 1 k ( ( ∑ i = 1 n v i , f x i ) 2 − ∑ j = 1 n v j , f 2 x j 2 )
\begin{aligned}
& \sum_{i=1}^{n} \sum_{j=i+1}^{n}\left\langle\mathbf{v}_{i}, \mathbf{v}_{j}\right\rangle x_{i} x_{j} \\
=& \frac{1}{2} \sum_{i=1}^{n} \sum_{j=1}^{n}\left\langle\mathbf{v}_{i}, \mathbf{v}_{j}\right\rangle x_{i} x_{j}-\frac{1}{2} \sum_{i=1}^{n}\left\langle\mathbf{v}_{i}, \mathbf{v}_{i}\right\rangle x_{i} x_{i} \\
=& \frac{1}{2}\left(\sum_{i=1}^{n} \sum_{j=1}^{n} \sum_{f=1}^{k} v_{i, f} v_{j, f} x_{i} x_{j}-\sum_{i=1}^{n} \sum_{f=1}^{k} v_{i, f} v_{i, f} x_{i} x_{i}\right) \\
=& \frac{1}{2} \sum_{f=1}^{k}\left(\left(\sum_{i=1}^{n} v_{i, f}x_{i}\right)\left(\sum_{j=1}^{n} v_{j, f} x_{j}\right) - \sum_{j=1}^{n} v^2_{j, f} x^2_{j}\right) \\
=& \frac{1}{2} \sum_{f=1}^{k}\left(\left(\sum_{i=1}^{n} v_{i, f}x_{i}\right)^2 - \sum_{j=1}^{n} v^2_{j, f} x^2_{j}\right) \\
\end{aligned}
= = = = i = 1 ∑ n j = i + 1 ∑ n ⟨ v i , v j ⟩ x i x j 2 1 i = 1 ∑ n j = 1 ∑ n ⟨ v i , v j ⟩ x i x j − 2 1 i = 1 ∑ n ⟨ v i , v i ⟩ x i x i 2 1 ⎝ ⎛ i = 1 ∑ n j = 1 ∑ n f = 1 ∑ k v i , f v j , f x i x j − i = 1 ∑ n f = 1 ∑ k v i , f v i , f x i x i ⎠ ⎞ 2 1 f = 1 ∑ k ( ( i = 1 ∑ n v i , f x i ) ( j = 1 ∑ n v j , f x j ) − j = 1 ∑ n v j , f 2 x j 2 ) 2 1 f = 1 ∑ k ⎝ ⎛ ( i = 1 ∑ n v i , f x i ) 2 − j = 1 ∑ n v j , f 2 x j 2 ⎠ ⎞
上式對 k k k n n n O ( k n ) O(kn) O ( k n )
( ∑ i = 1 n a i ) ( ∑ j = 1 n b j ) = ∑ i = 1 n ∑ j = 1 n a i b j
\left(\sum_{i=1}^{n}a_i\right)\left(\sum_{j=1}^{n}b_j\right) = \sum_{i=1}^{n}\sum_{j=1}^{n}a_ib_j
( i = 1 ∑ n a i ) ( j = 1 ∑ n b j ) = i = 1 ∑ n j = 1 ∑ n a i b j
libFM 在 src/fm_core/fm_model.h 中的 predict 方法中完成對預測值 y ^ \widehat{y} y
double fm_model:: predict ( sparse_row< FM_FLOAT> & x, DVector< double > & sum,
DVector< double > & sum_sqr) {
double result = 0 ;
if ( k0) {
result + = w0;
}
if ( k1) {
for ( uint i = 0 ; i < x. size; i++ ) {
assert ( x. data[ i] . id < num_attribute) ;
result + = w ( x. data[ i] . id) * x. data[ i] . value;
}
}
for ( int f = 0 ; f < num_factor; f++ ) {
sum ( f) = 0 ;
sum_sqr ( f) = 0 ;
for ( uint i = 0 ; i < x. size; i++ ) {
double d = v ( f, x. data[ i] . id) * x. data[ i] . value;
sum ( f) + = d;
sum_sqr ( f) + = d* d;
}
result + = 0.5 * ( sum ( f) * sum ( f) - sum_sqr ( f) ) ;
}
return result;
}
代碼分爲三個部分, 分別用來處理偏置 w 0 w_0 w 0 w i w_i w i v i v_i v i sum(f) 來保存 ∑ i = 1 n v i , f x i \sum\limits_{i=1}^{n} v_{i, f}x_{i} i = 1 ∑ n v i , f x i sum_sqr(f) 來保存 ∑ j = 1 n v j , f 2 x j 2 \sum\limits_{j=1}^{n} v^2_{j, f} x^2_{j} j = 1 ∑ n v j , f 2 x j 2 注意前者在後面的參數更新中會被使用到.
以上是 FM 的基本思路, 下面介紹 FM 的目標函數以及參數(w 0 , w , V w_0, \bm{w}, V w 0 , w , V
FM 可以用來做迴歸 (Regression) 以及二分類 (Binary Classification) 任務, 其中迴歸的目標函數一般採用 MSE:
l = 1 2 n ∑ i = 1 n ( y i − y ^ i ) 2
l = \frac{1}{2n}\sum_{i=1}^{n}\left(y_i - \widehat{y}_i\right)^2
l = 2 n 1 i = 1 ∑ n ( y i − y i ) 2
而分類任務一般用交叉熵, 但 libFM 中在實現時採用的 Log 損失函數, 參考 機器學習中的基本問題——log損失與交叉熵的等價性 , 瞭解到這兩個損失函數是等價的, 具體的推導方法如下:
首先 Log 損失的基本形式爲:
log ( 1 + exp ( − y ⋅ y ^ ) ) , y ∈ { − 1 , 1 }
\log(1 + \exp(-y\cdot\widehat{y})), \quad y\in\{-1, 1\}
log ( 1 + exp ( − y ⋅ y ) ) , y ∈ { − 1 , 1 }
對樣本集 T = { ( x 1 , y 1 ) , … , ( x n , y n ) } \mathcal{T} = \{(x_1, y_1), \ldots, (x_n, y_n)\} T = { ( x 1 , y 1 ) , … , ( x n , y n ) }
1 n ∑ i = 1 n log ( 1 + exp ( − y i ⋅ y ^ i ) )
\frac{1}{n}\sum_{i=1}^{n}\log\left(1 + \exp(-y_i\cdot\widehat{y}_i)\right)
n 1 i = 1 ∑ n log ( 1 + exp ( − y i ⋅ y i ) )
另一方面, Sigmoid 函數 (詳見 邏輯迴歸模型 Logistic Regression 詳細推導 (含 Numpy 與PyTorch 實現) ) 被定義爲:
σ ( x ) = 1 1 + exp ( − x )
\sigma(x) = \frac{1}{1 + \exp(-x)}
σ ( x ) = 1 + exp ( − x ) 1
從上面公式可以推出兩點結論:
⇒ 1 + exp ( − x ) = 1 σ ( x ) ⇒ σ ( x ) = 1 − σ ( − x )
\Rightarrow 1 + \exp(-x) = \frac{1}{\sigma(x)} \\
\Rightarrow \sigma(x) = 1 - \sigma(-x)
⇒ 1 + exp ( − x ) = σ ( x ) 1 ⇒ σ ( x ) = 1 − σ ( − x )
因此 Log 損失可以進一步推導爲:
1 n ∑ i = 1 n log ( 1 + exp ( − y i ⋅ y ^ i ) ) = 1 n ∑ i = 1 n log ( 1 σ ( y i ⋅ y ^ i ) ) = − 1 n ∑ i = 1 n log ( σ ( y i ⋅ y ^ i ) )
\begin{aligned}
&\frac{1}{n}\sum_{i=1}^{n}\log\left(1 + \exp(-y_i\cdot\widehat{y}_i)\right) \\
&= \frac{1}{n}\sum_{i=1}^{n}\log\left(\frac{1}{\sigma\left(y_i\cdot\widehat{y}_i\right)}\right) \\
&= -\frac{1}{n}\sum_{i=1}^{n}\log\left(\sigma\left(y_i\cdot\widehat{y}_i\right)\right)
\end{aligned}
n 1 i = 1 ∑ n log ( 1 + exp ( − y i ⋅ y i ) ) = n 1 i = 1 ∑ n log ( σ ( y i ⋅ y i ) 1 ) = − n 1 i = 1 ∑ n log ( σ ( y i ⋅ y i ) )
上式爲 libFM 中用到的損失函數 , 下面再介紹交叉熵. 交叉熵定義爲:
H ( y , y ^ ) = − 1 n ∑ i = 1 n I { y i = 1 } ⋅ log σ ( y ^ i ) + I { y i = − 1 } ⋅ log ( 1 − σ ( y ^ i ) )
H(y, \widehat{y}) = -\frac{1}{n}\sum_{i=1}^{n}I\{y_i = 1\}\cdot\log\sigma(\widehat{y}_i) + I\{y_i = -1\}\cdot\log\left(1 - \sigma(\widehat{y}_i)\right)
H ( y , y ) = − n 1 i = 1 ∑ n I { y i = 1 } ⋅ log σ ( y i ) + I { y i = − 1 } ⋅ log ( 1 − σ ( y i ) )
由於 σ ( x ) = 1 − σ ( − x ) \sigma(x) = 1 - \sigma(-x) σ ( x ) = 1 − σ ( − x )
H ( y , y ^ ) = − 1 n ∑ i = 1 n I { y i = 1 } ⋅ log σ ( y ^ i ) + I { y i = − 1 } ⋅ log ( σ ( − y ^ i ) )
H(y, \widehat{y}) = -\frac{1}{n}\sum_{i=1}^{n}I\{y_i = 1\}\cdot\log\sigma(\widehat{y}_i) + I\{y_i = -1\}\cdot\log\left(\sigma(-\widehat{y}_i)\right)
H ( y , y ) = − n 1 i = 1 ∑ n I { y i = 1 } ⋅ log σ ( y i ) + I { y i = − 1 } ⋅ log ( σ ( − y i ) )
又因爲:
I { y ( i ) = k } = { 0 if y ( i ) ≠ k 1 if y ( i ) = k
I\left\{y^{(i)}=k\right\}=\left\{
\begin{array}{ll}
0 & \text { if } y^{(i)} \neq k \\
1 & \text { if } y^{(i)}=k
\end{array}\right.
I { y ( i ) = k } = { 0 1 if y ( i ) = k if y ( i ) = k
因此, 可以得到交叉熵爲:
H ( y , y ^ ) = − 1 n ∑ i = 1 n log σ ( y i ⋅ y ^ i )
H(y, \widehat{y}) = -\frac{1}{n}\sum_{i=1}^{n}\log\sigma(y_i\cdot\widehat{y}_i)
H ( y , y ) = − n 1 i = 1 ∑ n log σ ( y i ⋅ y i )
該式和 Log 損失相同, 因此得到了 Log 損失和交叉熵等價的結論.
這裏關注 libFM 實現的 SGD 算法, 由於每次只對一個樣本 ( x i , y i ) (x_i, y_i) ( x i , y i )
l = 1 2 ( y i − y ^ i ) 2
l = \frac{1}{2}\left(y_i - \widehat{y}_i\right)^2
l = 2 1 ( y i − y i ) 2
其對參數的梯度爲:
∂ l ∂ θ = − ( y i − y ^ i ) ⋅ ∂ y ^ i ∂ θ
\frac{\partial l}{\partial\theta} = -\left(y_i - \widehat{y}_i\right)\cdot\frac{\partial \widehat{y}_i}{\partial\theta}
∂ θ ∂ l = − ( y i − y i ) ⋅ ∂ θ ∂ y i
對於迴歸問題, 損失函數爲 Log 損失 (詳見上一節):
l = − log ( σ ( y i ⋅ y ^ i ) )
l = -\log(\sigma(y_i\cdot\widehat{y}_i))
l = − log ( σ ( y i ⋅ y i ) )
梯度更新的方法爲:
∂ l ∂ θ = − 1 σ ( y i ⋅ y ^ i ) ⋅ σ ′ ( y i ⋅ y ^ i ) ⋅ y i ⋅ ∂ y ^ i ∂ θ = − 1 σ ( y i ⋅ y ^ i ) ⋅ ( σ ( y i ⋅ y ^ i ) ( 1 − σ ( y i ⋅ y ^ i ) ) ) ⋅ y i ⋅ ∂ y ^ i ∂ θ = − ( 1 − σ ( y i ⋅ y ^ i ) ) ⋅ y i ⋅ ∂ y ^ i ∂ θ = − y i ⋅ ( 1 − σ ( y i ⋅ y ^ i ) ) ⋅ ∂ y ^ i ∂ θ
\begin{aligned}
\frac{\partial l}{\partial\theta} &= -\frac{1}{\sigma(y_i\cdot\widehat{y}_i)}\cdot\sigma^\prime(y_i\cdot\widehat{y}_i)\cdot y_i \cdot \frac{\partial \widehat{y}_i}{\partial\theta} \\
&= -\frac{1}{\sigma(y_i\cdot\widehat{y}_i)}\cdot\left(\sigma(y_i\cdot\widehat{y}_i)\left(1 - \sigma(y_i\cdot\widehat{y}_i)\right)\right)\cdot y_i \cdot \frac{\partial \widehat{y}_i}{\partial\theta} \\
&= -\left(1 - \sigma(y_i\cdot\widehat{y}_i)\right)\cdot y_i \cdot \frac{\partial \widehat{y}_i}{\partial\theta} \\
&= -y_i\cdot \left(1 - \sigma(y_i\cdot\widehat{y}_i)\right)\cdot \frac{\partial \widehat{y}_i}{\partial\theta}
\end{aligned}
∂ θ ∂ l = − σ ( y i ⋅ y i ) 1 ⋅ σ ′ ( y i ⋅ y i ) ⋅ y i ⋅ ∂ θ ∂ y i = − σ ( y i ⋅ y i ) 1 ⋅ ( σ ( y i ⋅ y i ) ( 1 − σ ( y i ⋅ y i ) ) ) ⋅ y i ⋅ ∂ θ ∂ y i = − ( 1 − σ ( y i ⋅ y i ) ) ⋅ y i ⋅ ∂ θ ∂ y i = − y i ⋅ ( 1 − σ ( y i ⋅ y i ) ) ⋅ ∂ θ ∂ y i
推導中用到了 Sigmoid 函數求導的性質:
σ ′ ( x ) = σ ( x ) ( 1 − σ ( x ) )
\sigma^\prime(x) = \sigma(x)(1 - \sigma(x))
σ ′ ( x ) = σ ( x ) ( 1 − σ ( x ) )
libFM 在 libfm/src/fm_learn_sgd_element.h 的 learn 方法中實現上面的更新步驟, 核心代碼如下 (無關代碼已刪去):
void fm_learn_sgd_element:: learn ( Data& train, Data& test) {
fm_learn_sgd:: learn ( train, test) ;
for ( int i = 0 ; i < num_iter; i++ ) {
double iteration_time = getusertime ( ) ;
for ( train. data- > begin ( ) ; ! train. data- > end ( ) ; train. data- > next ( ) ) {
double p = fm- > predict ( train. data- > getRow ( ) , sum, sum_sqr) ;
double mult = 0 ;
if ( task == 0 ) {
p = std:: min ( max_target, p) ;
p = std:: max ( min_target, p) ;
mult = - ( train. target ( train. data- > getRowIndex ( ) ) - p) ;
} else if ( task == 1 ) {
mult = - train. target ( train. data- > getRowIndex ( ) ) * ( 1.0 - 1.0 / ( 1.0 + exp ( - train. target ( train. data- > getRowIndex ( ) ) * p) ) ) ;
}
SGD ( train. data- > getRow ( ) , mult, sum) ;
}
}
}
其中
double p = fm- > predict ( train. data- > getRow ( ) , sum, sum_sqr) ;
完成預估值的計算 (前面在闡述 “基本思路” 時已經介紹過了):
y ^ ( x ) = w 0 + ∑ i = 1 n w i x i + ∑ i = 1 n ∑ j = i + 1 n ⟨ v i , v j ⟩ x i x j = w 0 + ∑ i = 1 n w i x i + 1 2 ∑ f = 1 k ( ( ∑ i = 1 n v i , f x i ) 2 − ∑ j = 1 n v j , f 2 x j 2 )
\begin{aligned}
\widehat{y}(x) &= w_0 + \sum_{i=1}^{n} w_ix_i + \sum_{i=1}^{n}\sum_{j=i+1}^{n}\langle v_i, v_j \rangle x_ix_j \\
&= w_0 + \sum_{i=1}^{n} w_ix_i + \frac{1}{2} \sum_{f=1}^{k}\left(\left(\sum_{i=1}^{n} v_{i, f}x_{i}\right)^2 - \sum_{j=1}^{n} v^2_{j, f} x^2_{j}\right)
\end{aligned}
y ( x ) = w 0 + i = 1 ∑ n w i x i + i = 1 ∑ n j = i + 1 ∑ n ⟨ v i , v j ⟩ x i x j = w 0 + i = 1 ∑ n w i x i + 2 1 f = 1 ∑ k ⎝ ⎛ ( i = 1 ∑ n v i , f x i ) 2 − j = 1 ∑ n v j , f 2 x j 2 ⎠ ⎞
p 即爲 y ^ i \widehat{y}_i y i sum(f) 來保存 ∑ i = 1 n v i , f x i \sum\limits_{i=1}^{n} v_{i, f}x_{i} i = 1 ∑ n v i , f x i mult 表示 − ( y i − y ^ i ) -\left(y_i - \widehat{y}_i\right) − ( y i − y i ) mult 表示 − y i ⋅ ( 1 − σ ( y i ⋅ y ^ i ) ) -y_i\cdot\left(1 - \sigma(y_i\cdot\widehat{y}_i)\right) − y i ⋅ ( 1 − σ ( y i ⋅ y i ) )
之後的 SGD 函數, 真正實現對參數 ( w 0 , w , V ) (w_0, \bm{w}, V) ( w 0 , w , V ) ∂ y ^ i ∂ θ \frac{\partial \widehat{y}_i}{\partial\theta} ∂ θ ∂ y i θ \theta θ
∂ ∂ θ y ^ i ( x ) = { 1 , if θ is w 0 x i , if θ is w i x i ∑ j = 1 n v j , f x j − v i , f x i 2 , if θ is v i , f
\frac{\partial}{\partial \theta} \hat{y}_i(\mathbf{x})=\left\{
\begin{array}{ll}
1, & \text { if } \theta \text { is } w_{0} \\
x_{i}, & \text { if } \theta \text { is } w_{i} \\
x_{i} \sum\limits_{j=1}^{n} v_{j, f} x_{j}-v_{i, f} x_{i}^{2}, & \text { if } \theta \text { is } v_{i, f}
\end{array}\right.
∂ θ ∂ y ^ i ( x ) = ⎩ ⎪ ⎪ ⎨ ⎪ ⎪ ⎧ 1 , x i , x i j = 1 ∑ n v j , f x j − v i , f x i 2 , if θ is w 0 if θ is w i if θ is v i , f
需要注意的是, 在前面計算 y ^ i ( x ) \widehat{y}_i(x) y i ( x ) sum(f) 保存了 ∑ j = 1 n v j , f x j \sum\limits_{j=1}^{n} v_{j, f} x_{j} j = 1 ∑ n v j , f x j θ \theta θ O ( 1 ) O(1) O ( 1 )
SGD 函數定義在 src/fm_core/fm_sgd.h 中, 具體爲:
void fm_SGD ( fm_model* fm, const double & learn_rate, sparse_row< DATA_FLOAT> & x,
const double multiplier, DVector< double > & sum) {
if ( fm- > k0) {
double & w0 = fm- > w0;
w0 - = learn_rate * ( multiplier + fm- > reg0 * w0) ;
}
if ( fm- > k1) {
for ( uint i = 0 ; i < x. size; i++ ) {
double & w = fm- > w ( x. data[ i] . id) ;
w - = learn_rate * ( multiplier * x. data[ i] . value + fm- > regw * w) ;
}
}
for ( int f = 0 ; f < fm- > num_factor; f++ ) {
for ( uint i = 0 ; i < x. size; i++ ) {
double & v = fm- > v ( f, x. data[ i] . id) ;
double grad = sum ( f) * x. data[ i] . value - v * x. data[ i] . value * x. data[ i] . value;
v - = learn_rate * ( multiplier * grad + fm- > regv * v) ;
}
}
}
代碼中還考慮了對各個參數的正則項. 其中 multiplier 就是前面 mult, 對於迴歸問題, 表示 − ( y i − y ^ i ) -\left(y_i - \widehat{y}_i\right) − ( y i − y i ) multiplier 表示 − y i ⋅ ( 1 − σ ( y i ⋅ y ^ i ) ) -y_i\cdot\left(1 - \sigma(y_i\cdot\widehat{y}_i)\right) − y i ⋅ ( 1 − σ ( y i ⋅ y i ) ) sum(f) 表示 ∑ j = 1 n v j , f x j \sum\limits_{j=1}^{n} v_{j, f} x_{j} j = 1 ∑ n v j , f x j
關於 libFM 就分析到這了~
