Line Search Methods

重點

• Armijo condition的直觀理解

背景: In gradient descent algorithms, step size may be too large or too small, as shown in the figures below.

Backtracking line search

 - Initialization: alpha (=1), tau (decay rate)
 - while f(x^t + alpha p^t) ">" f(x^t)
	 alpha = tau*alpha
	end
 - Update x^{t+1} = x^t

• $\alpha = \alpha * \tau$的作用是防止step length過小
• $\textcolor{blue}{f(x^t + \alpha p^t) “>” f(x^t): \text{ prevent steps that are too long relative to the decrease in } f \text{；通過Armijo condition實現}}$

Wolfe conditions

Armijo condition

$f(x^t + \alpha^t p^t) \leq f(x^t) + \alpha^t c_1 \cdot [g^t]^T p^k,$
where $g^t$ denotes the first derivative of $f$; $p$ denotes the direction, $g^T p <0$ (remark: gradient descent $-g \Leftrightarrow p$).
In practice, $c_1$ is chosen quite small, say $c_1=10^{-4}$.
In the case that $p=g$, the Armijo condition in the 2nd step of pseudo-code step can be simplified as follows:
$f(x - \alpha \nabla(f)) > f(x) - c_1\alpha \|\nabla(f) \|_2^2 .$
*B&V book, $c_1 \in [0.01,0.03], \tau \in [0.1,0.8]$

$\textcolor{blue}{直觀理解}$

• require the reduction in $f$ to be at least a fixed fraction $\beta$ of the reduction promised by the first-order Taylor approximation of $f$ at $x^t$.
• aka significant decrease condition: require $\alpha$ to decrease the objective function by a significant amount.

Curvature condition

The curvature condition rules out small steps.
$\nabla f(x^t + \alpha^t p^t)^T p^t \geq c_2 \nabla f(x^t)^T p^t,$
where $c_2 \in (c_1,1)$.
$\textcolor{gray}{\text{The condition requires that the new slope is at least } c_2 \text{ times the original gradient.}}$

圖片來源

• step sizes: https://people.maths.ox.ac.uk/hauser/hauser_lecture2.pdf
• Figs 3.3, 3.4:
  Numerical Optimization

參考文獻：

1. https://people.maths.ox.ac.uk/hauser/hauser_lecture2.pdf
2. https://optimization.mccormick.northwestern.edu/index.php/Line_search_methods
3. Numerical Optimization