机器学习中的凸和非凸优化问题

题目（145）：机器学习中的优化问题，哪些是凸优化问题，哪些是非凸优化问题？请各举一个例子。

• 凸优化定义

• 凸优化问题

• 非凸优化问题
  
  

• 凸优化定义：公式、geometric insight

• 凸优化问题：逻辑回归；通过Hessian matrix的半正定性质判定；局部最优等价于全部最优

• 非凸优化问题：PCA；PCA求解方式
  
  

凸优化问题

逻辑回归

$L_i(\theta) = \log(1+\exp(-y_i \theta^T x_i))$

损失函数推导
logistic regression model:
$\log \frac{p}{1-p}=\theta^T x \Rightarrow p = \frac{\exp(\theta^T x)}{1+\exp(\theta^T x)}$

$\max \text{MLE} \simeq -\min \log \text{MLE}:= \min L(x,y;\theta)$

$\begin{aligned} L &= - (y \log p + (1-y) \log (1-p)) \\ &= - y \log \frac{1}{1+\exp(-\theta^T x)} - (1-y) \log \frac{1}{1+\exp(\theta^T x)}\\ &= y \log (1+\exp(-\theta^T x)) + (1-y) \log (1+\exp(\theta^T x))\\ &=\log (1+\exp(-\theta^T x \cdot y)), \end{aligned}$

where $Y \in \{0,1\}$ and $p=P(Y=1|X=x)$.

$\textcolor{red}{\text{\small 其它例子：SVM, linear regression}}$

非凸优化问题

PCA

$\min_{V V^T}L(V)= \| X-V^T V X\|_F^2$

$\textcolor{gray}{\textit{\small (minimise the reconstruction error)}}$

$\textcolor{red}{\text{\small Formulation from the perspective of maximising the variance}}$

验证该目标为非凸问题：检查定义
If $V^\ast$ is the minimum, then $-V^\ast$ is also the minimum as $L(V^\ast)=L(-V^\ast)$.
$\begin{aligned} L\large(\frac{1}{2} V^\ast + \frac{1}{2} (-V^\ast) \large)=L(0)&=\|X\|_F^2 \\ &> \| X-V^{\ast T} V^\ast X\|_F^2=\frac{1}{2} L(V^\ast) + \frac{1}{2} L(-V^\ast) \end{aligned}$

求解: $\textcolor{red}{\text{\small SVD}}$

$\textcolor{red}{\text{\small 其它例子：low-rank model (e.g. matrix decomposition), deep neural network}}$

参考文献：

1. 《百面机器学习》