Lam R, Willcox K, Wolpert D H, et al. Bayesian Optimization with a Finite Budget: An Approximate Dynamic Programming Approach[C]. neural information processing systems, 2016: 883-891.
@article{lam2016bayesian,
貝葉斯優化中的多步優化策略. 像經典的EI方法, 就是隻考慮一步, 即希望找到r ( S k , x k + 1 , f k + 1 ) = max { 0 , f m i n S k − f k + 1 }
r(\mathcal{S}_k, x_{k+1},f_{k+1})=\max \{0, f_{min}^{\mathcal{S}_k}-f_{k+1}\}
r ( S k , x k + 1 , f k + 1 ) = max { 0 , f m i n S k − f k + 1 } x k + 1 x_{k+1} x k + 1
上式中的f m i n S k f_{min}^{\mathcal{S}_k} f m i n S k S k \mathcal{S}_k S k
考慮如下動態規劃, 第k步的S k \mathcal{S}_k S k u k u_k u k u k ( S k ) = x k + 1 u_k(\mathcal{S}_k)=x_{k+1} u k ( S k ) = x k + 1 w k : = f k + 1 ∼ p ( f ( x k + 1 ) ∣ S k ) w_k:=f_{k+1} \sim p(f(x_{k+1})|\mathcal{S}_k) w k : = f k + 1 ∼ p ( f ( x k + 1 ) ∣ S k )
設狀態轉移爲:S k + 1 = F k ( S k , x k + 1 , f k + 1 ) = S k ∪ { ( x k + 1 , f k + 1 ) } .
\mathcal{S}_{k+1} = \mathcal{F}_k (\mathcal{S}_{k}, x_{k+1}, f_{k+1}) = \mathcal{S}_{k}\cup \{(x_{k+1}, f_{k+1})\}.
S k + 1 = F k ( S k , x k + 1 , f k + 1 ) = S k ∪ { ( x k + 1 , f k + 1 ) } .
收益(效用函數):U k ( x k + 1 ; S k ) = E w k [ r k ( S k , x k + 1 , f k + 1 ) + J k + 1 ( F k ( S k , x k + 1 , f k + 1 ) ) ] , J k ( x k + 1 ) = max x k + 1 U k , J N = r N ( x N + 1 ) .
U_k(x_{k+1}; \mathcal{S} _k) = \mathbb{E}_{w_k}[r_k(\mathcal{S}_k, x_{k+1}, f_{k+1})+J_{k+1}(\mathcal{F}_k (\mathcal{S}_{k}, x_{k+1}, f_{k+1}))], \\
J_k(x_{k+1}) = \max_{x_{k+1}} U_k,\\
J_N=r_N(x_{N+1}).
U k ( x k + 1 ; S k ) = E w k [ r k ( S k , x k + 1 , f k + 1 ) + J k + 1 ( F k ( S k , x k + 1 , f k + 1 ) ) ] , J k ( x k + 1 ) = x k + 1 max U k , J N = r N ( x N + 1 ) .
很自然的想法是, 我們最大化U 1 U_1 U 1
本文采用rollout 算法來估計U k U_k U k
給定基本的決策控制π = ( π 1 , … , π N ) \pi = (\pi_1, \ldots, \pi_N) π = ( π 1 , … , π N ) U k U_k U k H k + 1 H_{k+1} H k + 1 J k + 1 J_{k+1} J k + 1
n ∈ { k + 1 , … , N − 1 } n \in \{k+1, \ldots, N-1\} n ∈ { k + 1 , … , N − 1 } γ ∈ [ 0 , 1 ] \gamma \in [0, 1] γ ∈ [ 0 , 1 ]
H n H_n H n
其中N ~ = min { k + h , N } \tilde{N} = \min \{k+h, N\} N ~ = min { k + h , N } α ( q ) \alpha^{(q)} α ( q ) f n + 1 ( q ) f_{n+1}^{(q)} f n + 1 ( q )
於是, U k ( x k + 1 , S k ) U_k(x_{k+1},\mathcal{S}_k) U k ( x k + 1 , S k )
算法如下:
Input: h , γ , N , S 1 h, \gamma, N, \mathcal{S}_1 h , γ , N , S 1
根據(20）近似最大化U k U_k U k
更新S k + 1 = S k ∪ { ( x k + 1 , f k + 1 ) } \mathcal{S}_{k+1}=\mathcal{S}_k \cup \{(x_{k+1},f_{k+1})\} S k + 1 = S k ∪ { ( x k + 1 , f k + 1 ) }
out: f m i n S N + 1 f_{min}^{S_{N+1}} f m i n S N + 1
