前面講了梯度下降、次梯度下降、近似點梯度下降方法並分析了收斂性。一開始我們還講了對偶原理，那麼如果原問題比較難求解的時候，我們可不可以轉化爲對偶問題並應用梯度法求解呢？當然可以，不過有一個問題就是對偶函數的梯度或者次梯度怎麼計算呢？這就是這一節要關注的問題。
首先一個問題是哪些形式的問題，其對偶問題相比於原問題更簡單呢？可能有很多種，這一節主要關注一種：線性等式/不等式約束的優化問題 。之所以考慮此類問題是因爲如果有線性的等式/不等式約束，應用 Lagrange 對偶原理之後，我們可以自然的用原函數的共軛函數 來表示其對偶問題 ，而共軛函數的次梯度是容易求解的，因爲我們可以用 Legendre transformation 。那麼接下來就是詳細的原理和例子了。
假如我們考慮如下無約束優化問題，通過添加線性等式約束以及對偶原理可以容易獲得對偶問題的形式( P ) min f ( x ) + g ( A x ) ( D ) max − g ⋆ ( z ) − f ⋆ ( − A T z )
\begin{aligned}
(P)\quad\min \ & f(x)+g(Ax) \\
(D)\quad\max \ & -g^\star(z)-f^\star(-A^Tz)
\end{aligned}
( P ) min ( D ) max f ( x ) + g ( A x ) − g ⋆ ( z ) − f ⋆ ( − A T z ) f ⋆ , g ⋆ f^\star,g^\star f ⋆ , g ⋆
關於共軛函數有以下性質
若 f f f epi f \text{epi }f epi f f ∗ ∗ = f f^{**}=f f ∗ ∗ = f
(Fenchel’s inequality) f ( x ) + f ∗ ( y ) ≥ x T y f(x)+f^*(y)\ge x^Ty f ( x ) + f ∗ ( y ) ≥ x T y
(Legendre transform )如果 f f f 凸的、閉的 ，設 x ∗ = arg max { y T x − f ( x ) } x^*=\arg\max\{y^Tx-f(x)\} x ∗ = arg max { y T x − f ( x ) } x ∗ ∈ ∂ f ∗ ( y ) ⟺ y ∈ ∂ f ( x ∗ ) x^*\in\partial f^*(y)\iff y\in\partial f(x^*) x ∗ ∈ ∂ f ∗ ( y ) ⟺ y ∈ ∂ f ( x ∗ ) min f ( x ) ⟹ 0 ∈ ∂ f ( x ) ⟺ x ∈ ∂ f ∗ ( 0 ) \min f(x)\Longrightarrow 0\in\partial f(x)\iff x\in\partial f^*(0) min f ( x ) ⟹ 0 ∈ ∂ f ( x ) ⟺ x ∈ ∂ f ∗ ( 0 )
所以只需要找到x ^ = arg max x − z T A x − f ( x ) = arg min x z T A x + f ( x )
\hat{x}=\arg\max_x -z^TAx-f(x) = \arg\min_x z^TAx+f(x)
x ^ = arg x max − z T A x − f ( x ) = arg x min z T A x + f ( x ) x ^ ∈ ∂ f ⋆ ( − A T z ) \hat{x}\in\partial f^\star(-A^Tz) x ^ ∈ ∂ f ⋆ ( − A T z ) f f f 嚴格凸函數 ，意味着上面的 x ^ \hat{x} x ^ ∂ f ⋆ = ∇ f ⋆ \partial f^\star=\nabla f^\star ∂ f ⋆ = ∇ f ⋆ f f f μ \mu μ ∥ ∇ f ∗ ( y ) − ∇ f ∗ ( y ′ ) ∥ ≤ 1 μ ∥ y − y ′ ∥ ∗ for all y , y ′
\left\|\nabla f^{*}(y)-\nabla f^{*}\left(y^{\prime}\right)\right\| \leq \frac{1}{\mu}\left\|y-y^{\prime}\right\|_{*} \quad \text { for all } y, y^{\prime}
∥ ∇ f ∗ ( y ) − ∇ f ∗ ( y ′ ) ∥ ≤ μ 1 ∥ y − y ′ ∥ ∗ for all y , y ′ f ⋆ f^\star f ⋆ 1 / μ 1/\mu 1 / μ
例子 g g g ( P ) minimize f ( x ) subject to A x = b ( D ) minimize − b T z − f ⋆ ( − A T z )
\begin{aligned}
(P)\quad\text{minimize } \quad& f(x) \\
\text{subject to } \quad& Ax=b \\
(D)\quad\text{minimize } \quad& -b^Tz-f^\star(-A^Tz)
\end{aligned}
( P ) minimize subject to ( D ) minimize f ( x ) A x = b − b T z − f ⋆ ( − A T z ) x ^ = argmin x ( f ( x ) + z T A x ) z + = z + t ( A x ^ − b )
\begin{aligned}
\hat{x} &=\underset{x}{\operatorname{argmin}}\left(f(x)+z^{T} A x\right) \\
z^{+} &=z+t(A \hat{x}-b)
\end{aligned}
x ^ z + = x a r g m i n ( f ( x ) + z T A x ) = z + t ( A x ^ − b ) f ⋆ f^\star f ⋆ z z z L ( x , z ) = f ( x ) + z T ( A x − b ) = − b T z + ( f ( x ) + z T A x )
\begin{aligned}
L(x,z)&=f(x)+z^T(Ax-b) \\
&=-b^Tz+(f(x)+z^TAx)
\end{aligned}
L ( x , z ) = f ( x ) + z T ( A x − b ) = − b T z + ( f ( x ) + z T A x ) x x x z z z x x x x ^ \hat{x} x ^ z z z
好了，對偶問題的梯度下降方法原理就這些，但是標題裏面“對偶分解”還有一個“分解 ”是什麼意思呢？我們說如果原問題比較複雜就可以考慮解對偶問題，那麼具體是哪一種問題呢？看個例子minimize f 1 ( x 1 ) + f 2 ( x 2 ) subject to A 1 x 1 + A 2 x 2 ⪯ b
\begin{aligned}
\text{minimize } \quad& f_1(x_1)+f_2(x_2) \\
\text{subject to } \quad& A_1x_1+A_2x_2\preceq b \\
\end{aligned}
minimize subject to f 1 ( x 1 ) + f 2 ( x 2 ) A 1 x 1 + A 2 x 2 ⪯ b f 1 ( x 1 ) , f 2 ( x 2 ) f_1(x_1),f_2(x_2) f 1 ( x 1 ) , f 2 ( x 2 ) f 1 , f 2 f_1,f_2 f 1 , f 2 x 1 , x 2 x_1,x_2 x 1 , x 2 由於這個約束條件，這兩個問題耦合在一起了 ，這就有點麻煩了。那麼在對偶問題中能不能解耦合呢？好消息是可以！他們的對偶問題可以寫爲minimize − f 1 ⋆ ( − A 1 T z ) − f 2 ⋆ ( − A 2 T z ) − b T z subject to z ⪰ 0
\begin{aligned}
\text{minimize } \quad& -f_1^\star(-A_1^Tz)-f_2^\star(-A_2^Tz)-b^Tz \\
\text{subject to } \quad& z\succeq 0 \\
\end{aligned}
minimize subject to − f 1 ⋆ ( − A 1 T z ) − f 2 ⋆ ( − A 2 T z ) − b T z z ⪰ 0 f 1 ⋆ , f 2 ⋆ f_1^\star,f_2^\star f 1 ⋆ , f 2 ⋆ x ^ 1 , x ^ 2 \hat{x}_1,\hat{x}_2 x ^ 1 , x ^ 2 x ^ j = argmin x j ( f j ( x j ) + z T A j x j ) for j = 1 , 2 z + = ( z + t ( A 1 x ^ 1 + A 2 x ^ 2 − b ) ) +
\begin{aligned}
&\hat{x}_{j}=\underset{x_{j}}{\operatorname{argmin}}\left(f_{j}\left(x_{j}\right)+z^{T} A_{j} x_{j}\right) \quad \text { for } j=1,2\\
&z^{+}=\left(z+t\left(A_{1} \hat{x}_{1}+A_{2} \hat{x}_{2}-b\right)\right)_{+}
\end{aligned}
x ^ j = x j a r g m i n ( f j ( x j ) + z T A j x j ) for j = 1 , 2 z + = ( z + t ( A 1 x ^ 1 + A 2 x ^ 2 − b ) ) + x ^ j \hat{x}_j x ^ j z z z z + z^+ z + z ⪰ 0 z\succeq 0 z ⪰ 0 C = { z ∣ z ⪰ 0 } C=\{z|z\succeq0\} C = { z ∣ z ⪰ 0 }
例子 1 P j ≻ 0 P_j\succ0 P j ≻ 0 minimize ∑ j = 1 r ( 1 2 x j T P j x j + q j T x j ) subject to B j x j ⪯ d j , j = 1 , … , r ∑ j = 1 r A j x j ⪯ b
\begin{array}{ll}
\text { minimize } & \sum_{j=1}^{r}\left(\frac{1}{2} x_{j}^{T} P_{j} x_{j}+q_{j}^{T} x_{j}\right) \\
\text { subject to } & B_{j} x_{j} \preceq d_{j}, \quad j=1, \ldots, r \\
& \sum_{j=1}^{r} A_{j} x_{j} \preceq b
\end{array}
minimize subject to ∑ j = 1 r ( 2 1 x j T P j x j + q j T x j ) B j x j ⪯ d j , j = 1 , … , r ∑ j = 1 r A j x j ⪯ b f j ( x j ) = 1 2 x j T P j x j + q j T x j f_j(x_j)=\frac{1}{2} x_{j}^{T} P_{j} x_{j}+q_{j}^{T} x_{j} f j ( x j ) = 2 1 x j T P j x j + q j T x j { x j ∣ B j x j ⪯ d j } \{x_j|B_jx_j\preceq d_j\} { x j ∣ B j x j ⪯ d j } minimize − b T z − ∑ j = 1 r f j ⋆ ( − A j T z ) subject to z ⪰ 0
\begin{aligned}
\text{minimize } \quad& -b^Tz-\sum_{j=1}^r f_j^\star(-A_j^Tz) \\
\text{subject to } \quad& z\succeq 0 \\
\end{aligned}
minimize subject to − b T z − j = 1 ∑ r f j ⋆ ( − A j T z ) z ⪰ 0 h ( z ) = ∑ j f j ⋆ ( − A j T z ) h(z)=\sum_j f_j^\star(-A_j^Tz) h ( z ) = ∑ j f j ⋆ ( − A j T z ) ∥ ∇ h ( z ) − ∇ h ( z ′ ) ∥ 2 ≤ ∥ A ∥ 2 2 min j λ min ( P j ) ∥ z − z ′ ∥ 2
\left\|\nabla h(z)-\nabla h\left(z^{\prime}\right)\right\|_{2} \leq \frac{\|A\|_{2}^{2}}{\min _{j} \lambda_{\min }\left(P_{j}\right)}\left\|z-z^{\prime}\right\|_{2}
∥ ∇ h ( z ) − ∇ h ( z ′ ) ∥ 2 ≤ min j λ min ( P j ) ∥ A ∥ 2 2 ∥ z − z ′ ∥ 2 A = [ A 1 ⋯ A r ] A=[\begin{array}{ccc}A_1&\cdots&A_r\end{array}] A = [ A 1 ⋯ A r ] x ^ j = ∂ f j ⋆ ( − A j T z ) \hat{x}_j=\partial f_j^\star(-A_j^Tz) x ^ j = ∂ f j ⋆ ( − A j T z ) minimize 1 2 x j T P j x j + ( q j + A j T z ) T x j subject to B j x j ⪯ d j
\begin{array}{ll}
\text { minimize } & \frac{1}{2} x_{j}^{T} P_{j} x_{j}+(q_{j}+A_j^Tz)^T x_{j} \\
\text { subject to } & B_{j} x_{j} \preceq d_{j}
\end{array}
minimize subject to 2 1 x j T P j x j + ( q j + A j T z ) T x j B j x j ⪯ d j 例子 2 ( P ) maximize ∑ j = 1 n U j ( x j ) subject to R x ≤ c ( D ) minimize c T z + ∑ j = 1 n ( − U j ) ∗ ( − r j T z ) subject to z ≥ 0
\begin{aligned}
(P) \quad \text { maximize } \quad& \sum_{j=1}^{n} U_{j}\left(x_{j}\right)\\
\text { subject to } \quad& R x \leq c\\
(D) \quad \text { minimize } \quad& c^{T} z+\sum_{j=1}^{n}\left(-U_{j}\right)^{*}\left(-r_{j}^{T} z\right)\\
\text { subject to } \quad& z \geq 0
\end{aligned}
( P ) maximize subject to ( D ) minimize subject to j = 1 ∑ n U j ( x j ) R x ≤ c c T z + j = 1 ∑ n ( − U j ) ∗ ( − r j T z ) z ≥ 0 R R R
這節的一開始我們考慮的問題是( P ) min f ( x ) + g ( A x ) ( D ) min g ⋆ ( z ) + f ⋆ ( − A T z )
\begin{aligned}
(P)\quad\min \ & f(x)+g(Ax) \\
(D)\quad\min \ & g^\star(z)+f^\star(-A^Tz)
\end{aligned}
( P ) min ( D ) min f ( x ) + g ( A x ) g ⋆ ( z ) + f ⋆ ( − A T z ) g g g g = δ C ( x ) g=\delta_C(x) g = δ C ( x ) g g g z + = prox t g ∗ ( z + t A ∇ f ∗ ( − A T z ) )
z^{+}=\operatorname{prox}_{t g^{*}}\left(z+t A \nabla f^{*}\left(-A^{T} z\right)\right)
z + = p r o x t g ∗ ( z + t A ∇ f ∗ ( − A T z ) ) x ^ = argmin x ( f ( x ) + z T A x ) z + = prox t g ∗ ( z + t A x ^ )
\begin{aligned}
\hat{x} &=\underset{x}{\operatorname{argmin}}\left(f(x)+z^{T} A x\right) \\
z^{+} &=\operatorname{prox}_{t g^{*}}(z+t A \hat{x})
\end{aligned}
x ^ z + = x a r g m i n ( f ( x ) + z T A x ) = p r o x t g ∗ ( z + t A x ^ ) g ⋆ g^\star g ⋆ z + = z + t A x ^ − t prox t − 1 g ( t − 1 z + A x ^ )
z^{+} =z+tA\hat{x}-t\operatorname{prox}_{t^{-1} g}(t^{-1}z+A \hat{x})
z + = z + t A x ^ − t p r o x t − 1 g ( t − 1 z + A x ^ ) y ^ \hat{y} y ^ y ^ = prox t − 1 g ( t − 1 z + A x ^ ) = argmin y ( g ( y ) + t 2 ∥ A x ^ − t − 1 z − y ∥ 2 2 ) = argmin y ( g ( y ) + z T ( A x ^ − y ) + t 2 ∥ A x ^ − y ∥ 2 2 )
\begin{aligned}
\hat{y} &=\operatorname{prox}_{t^{-1} g}\left(t^{-1} z+A \hat{x}\right) \\
&=\underset{y}{\operatorname{argmin}}\left(g(y)+\frac{t}{2}\left\|A \hat{x}-t^{-1} z-y\right\|_{2}^{2}\right) \\
&=\underset{y}{\operatorname{argmin}}\left(g(y)+z^{T}(A \hat{x}-y)+\frac{t}{2}\|A \hat{x}-y\|_{2}^{2}\right)
\end{aligned}
y ^ = p r o x t − 1 g ( t − 1 z + A x ^ ) = y a r g m i n ( g ( y ) + 2 t ∥ ∥ A x ^ − t − 1 z − y ∥ ∥ 2 2 ) = y a r g m i n ( g ( y ) + z T ( A x ^ − y ) + 2 t ∥ A x ^ − y ∥ 2 2 ) z + = z + t ( A x ^ − y ^ ) z^{+}=z+t(A\hat{x}-\hat{y}) z + = z + t ( A x ^ − y ^ ) y ^ \hat{y} y ^ g ⋆ ( z ) g^\star(z) g ⋆ ( z ) x ^ = argmin x ( f ( x ) + z T A x ) y ^ = argmin y ( g ( y ) − z T y + t 2 ∥ A x ^ − y ∥ 2 2 ) z + = z + t ( A x ^ − y ^ )
\begin{aligned}
\hat{x} &=\underset{x}{\operatorname{argmin}}\left(f(x)+z^{T} A x\right) \\
\hat{y} &=\underset{y}{\operatorname{argmin}}\left(g(y)-z^{T} y+\frac{t}{2}\|A \hat{x}-y\|_{2}^{2}\right) \\
z^{+} &=z+t(A \hat{x}-\hat{y})
\end{aligned}
x ^ y ^ z + = x a r g m i n ( f ( x ) + z T A x ) = y a r g m i n ( g ( y ) − z T y + 2 t ∥ A x ^ − y ∥ 2 2 ) = z + t ( A x ^ − y ^ ) ( x ^ , y ^ ) = argmin x , y ( f ( x ) + g ( y ) + z T ( A x − y ) + t 2 ∥ A x − y ∥ 2 2 )
(\hat{x}, \hat{y})=\underset{x, y}{\operatorname{argmin}}\left(f(x)+g(y)+z^{T}(A x-y)+\frac{t}{2}\|A x-y\|_{2}^{2}\right)
( x ^ , y ^ ) = x , y a r g m i n ( f ( x ) + g ( y ) + z T ( A x − y ) + 2 t ∥ A x − y ∥ 2 2 ) x , y x,y x , y L ( x , y , z ) = ( f ( x ) + z T A x ) + ( g ( y ) − z T y + t 2 ∥ A x − y ∥ 2 2 )
L(x,y,z)=\left(f(x)+z^{T}A x\right)+\left(g(y)-z^Ty+\frac{t}{2}\|A x-y\|_{2}^{2}\right)
L ( x , y , z ) = ( f ( x ) + z T A x ) + ( g ( y ) − z T y + 2 t ∥ A x − y ∥ 2 2 ) x x x x ^ \hat{x} x ^ y y y y ^ \hat{y} y ^ z z z f f f f f f
基本的理論就完了，下面主要是一些例子。
例子 1 g g g ( P ) minimize f ( x ) + ∥ A x − b ∥ ( D ) maximize − b T z − f ∗ ( − A T z ) subject to ∥ z ∥ ∗ ≤ 1
\begin{aligned}
(P)\quad\text{minimize}\quad& f(x)+\|A x-b\| \\
(D)\quad\text{maximize} \quad& -b^{T} z-f^{*}\left(-A^{T} z\right)\\
\text{subject to} \quad& \|z\|_{*} \leq 1
\end{aligned}
( P ) minimize ( D ) maximize subject to f ( x ) + ∥ A x − b ∥ − b T z − f ∗ ( − A T z ) ∥ z ∥ ∗ ≤ 1 g ( y ) = ∥ y − b ∥ g(y)=\Vert y-b\Vert g ( y ) = ∥ y − b ∥ g ∗ ( z ) = { b T z ∥ z ∥ ∗ ≤ 1 + ∞ otherwise prox t g ∗ ( z ) = P C ( z − t b )
g^{*}(z)=\left\{\begin{array}{ll}
b^{T} z & \|z\|_{*} \leq 1 \\
+\infty & \text { otherwise }
\end{array} \quad \operatorname{prox}_{t g *}(z)=P_{C}(z-t b)\right.
g ∗ ( z ) = { b T z + ∞ ∥ z ∥ ∗ ≤ 1 otherwise p r o x t g ∗ ( z ) = P C ( z − t b ) x ^ = argmin x ( f ( x ) + z T A x ) z + = P C ( z + t ( A x ^ − b ) )
\begin{aligned}
\hat{x} &=\underset{x}{\operatorname{argmin}}\left(f(x)+z^{T} A x\right) \\
z^{+} &=P_{C}(z+t(A \hat{x}-b))
\end{aligned}
x ^ z + = x a r g m i n ( f ( x ) + z T A x ) = P C ( z + t ( A x ^ − b ) ) 例子 2(重要) g g g g g g g i g_i g i ( P ) minimize f ( x ) + ∑ i = 1 p ∥ B i x ∥ 2 ( D ) maximize − f ∗ ( − B 1 T z 1 − ⋯ − B p T z p ) subject to ∥ z i ∥ 2 ≤ 1 , i = 1 , … , p
\begin{aligned}
(P) \quad \text { minimize } \quad& f(x)+\sum_{i=1}^{p}\left\|B_{i} x\right\|_{2}\\
(D) \quad \text { maximize } \quad& -f^{*}\left(-B_{1}^{T} z_{1}-\cdots-B_{p}^{T} z_{p}\right)\\
\text { subject to }\quad& \left\|z_{i}\right\|_{2} \leq 1, \quad i=1, \ldots, p
\end{aligned}
( P ) minimize ( D ) maximize subject to f ( x ) + i = 1 ∑ p ∥ B i x ∥ 2 − f ∗ ( − B 1 T z 1 − ⋯ − B p T z p ) ∥ z i ∥ 2 ≤ 1 , i = 1 , … , p g ( x ) = g 1 ( B 1 x ) + . . . + g p ( B p x ) g(x)=g_1(B_1x)+...+g_p(B_px) g ( x ) = g 1 ( B 1 x ) + . . . + g p ( B p x ) B = [ B 1 ; . . . ; B p ] B=[B_1;...;B_p] B = [ B 1 ; . . . ; B p ] g ( x ) = g ^ ( B x ) g(x)=\hat{g}(Bx) g ( x ) = g ^ ( B x ) minimize f ( x ) + g ^ ( y ) subject to y = B x
\begin{aligned}
\text{minimize } \quad& f(x)+\hat{g}(y) \\
\text{subject to } \quad& y=Bx \\
\end{aligned}
minimize subject to f ( x ) + g ^ ( y ) y = B x L ( x , y , z ) = f ( x ) + ∑ i g i ( y i ) + ∑ i z i T ( B i x − y i ) L(x,y,z)=f(x)+\sum_i g_i(y_i) + \sum_i z_i^T(B_ix-y_i) L ( x , y , z ) = f ( x ) + ∑ i g i ( y i ) + ∑ i z i T ( B i x − y i ) maximize − f ⋆ ( − ∑ i B i T z i ) − ∑ i g i ⋆ ( z i )
\text{maximize}\quad -f^\star(-\sum_i B_i^Tz_i) - \sum_i g_i^\star(z_i)
maximize − f ⋆ ( − i ∑ B i T z i ) − i ∑ g i ⋆ ( z i ) g ⋆ ( z ) = ∑ i g i ⋆ ( z i ) g^\star(z)=\sum_i g^\star_i(z_i) g ⋆ ( z ) = ∑ i g i ⋆ ( z i ) prox t g ⋆ ( x ) = [ prox t g 1 ⋆ ( x 1 ) ⋮ prox t g p ⋆ ( x p ) ]
\operatorname{prox}_{tg^\star}(x)=\left[\begin{array}{c}\operatorname{prox}_{tg^\star_1}(x_1)\\ \vdots \\ \operatorname{prox}_{tg^\star_p}(x_p) \end{array}\right]
p r o x t g ⋆ ( x ) = ⎣ ⎢ ⎡ p r o x t g 1 ⋆ ( x 1 ) ⋮ p r o x t g p ⋆ ( x p ) ⎦ ⎥ ⎤ z i + z_i^+ z i + x ^ \hat{x} x ^ z i + z_i^+ z i + x ^ = argmin x ( f ( x ) + ( ∑ i = 1 p B i T z i ) T x ) z i + = P C i ( z i + t B i x ^ ) , i = 1 , … , p
\begin{aligned}
\hat{x} &=\underset{x}{\operatorname{argmin}}\left(f(x)+\left(\sum_{i=1}^{p} B_{i}^{T} z_{i}\right)^{T} x\right) \\
z_{i}^{+} &=P_{C_{i}}\left(z_{i}+t B_{i} \hat{x}\right), \quad i=1, \ldots, p
\end{aligned}
x ^ z i + = x a r g m i n ⎝ ⎛ f ( x ) + ( i = 1 ∑ p B i T z i ) T x ⎠ ⎞ = P C i ( z i + t B i x ^ ) , i = 1 , … , p f 1 ( x 1 ) + f 2 ( x 2 ) f_1(x_1)+f_2(x_2) f 1 ( x 1 ) + f 2 ( x 2 ) ∑ i f ⋆ ( − A i T z ) \sum_i f^\star(-A_i^Tz) ∑ i f ⋆ ( − A i T z ) x ^ i \hat{x}_i x ^ i ∑ i g ( x ) \sum_i g(x) ∑ i g ( x ) ∑ i g ⋆ ( z i ) \sum_i g^\star(z_i) ∑ i g ⋆ ( z i ) z i + z_i^+ z i +
最後給我的博客打個廣告，歡迎光臨https://glooow1024.github.io/ https://glooow.gitee.io/
前面的一些博客鏈接如下凸優化專欄 凸優化學習筆記 1：Convex Sets 凸優化學習筆記 2：超平面分離定理 凸優化學習筆記 3：廣義不等式 凸優化學習筆記 4：Convex Function 凸優化學習筆記 5：保凸變換 凸優化學習筆記 6：共軛函數 凸優化學習筆記 7：擬凸函數 Quasiconvex Function 凸優化學習筆記 8：對數凸函數 凸優化學習筆記 9：廣義凸函數 凸優化學習筆記 10：凸優化問題 凸優化學習筆記 11：對偶原理 凸優化學習筆記 12：KKT條件 凸優化學習筆記 13：KKT條件 & 互補性條件 & 強對偶性 凸優化學習筆記 14：SDP Representablity 最優化方法 15：梯度方法 最優化方法 16：次梯度 最優化方法 17：次梯度下降法 最優化方法 18：近似點算子 Proximal Mapping 最優化方法 19：近似梯度下降 最優化方法 20：對偶近似點梯度下降法 最優化方法 21：加速近似梯度下降方法 最優化方法 22：近似點算法 PPA 最優化方法 23：算子分裂法 & ADMM 最優化方法 24：ADMM
