四參數擬合之LMF法（無約束)

前言

 寫到這裏，已經發現了前面兩篇文章的重大bug。那就是牛頓法也好，LM法也好，都是針對無約束的問題，而四參數擬合問題是一個有約束的問題，參數一般設置爲0到正無窮。這也解釋了爲何之前的計算結果，總是和L4P的結果不同。根本原因在於完全沒搞懂四參數擬合的參數意義。所以這篇重點介紹LM算法，四參數擬合結果仍然有問題
 事到如今，將錯就錯， 把Levenberg-Marquardt算法寫完。Levenberg-Marquardt簡稱L-M，它的主要目的是克服了高斯牛頓法必須要求雅可比矩陣秩滿列的缺點，爲此，它引入了新的目標函數。

LM 算法

高斯牛頓法目標
$arg \ \underset{w}{min} (F(w)) = arg \ \underset{w}{min}\sum_{i=1}^{m}||y_i-f(w;x_i)||^2 = arg \ \underset{w}{min}\frac{1}{2}r(w)^Tr(w)$
$r(w)$是殘差向量,$r_i(w) = y_i-f(w;x_i)$，$w = [A,B,C,D]$爲帶求參數
$f(w;x_i) = D+\frac{A-D}{1+(x_i/C)^B}$
L-M法目標

$arg \ \underset{w}{min}\sum_{i=1}^{m}||y_i-f(w;x_i)||^2+\lambda||\Delta w ||^2$

這個目標式對$||\Delta w ||^2$的進行約束，防止$||\Delta w ||^2$，使得泰勒展開式誤差過大。這個目標跟嶺迴歸的目標式如出一轍
$\Delta w = - (J_f^TJ_f+\lambda I)^{-1}J_f^Tr(w)$
從上式可以看出，$\lambda$很大的時候，下降梯度接近最速下降法，很小時，則接近於高斯牛頓法。爲了得到合適的$\lambda$值需要引入一個評價指標$q$。
令$s =\Delta w$,$L(s) = F(w+\Delta w)\approx F(w)+g(w)^Ts+\frac{1}{2}s^TGs$
這裏的$g(w),G$分爲$F$的梯度以及海賽矩陣，其中，$G$只是取一個近似值
$q = \frac{F(w)-F(w+\Delta w)}{L(0)-L(s)}$
從上式可以看出，$q$越接近$1$，泰勒展開的估計值越準確。$q$只要大於0，至少說明方向是正確的，如果$q$小於0，則要調節步長了。我們已經能看到步長$s =\Delta w$實際是通過$\lambda$控制，調節範圍在最速下降和高斯牛頓法之間。這種步長的調節方法也稱爲稱爲信賴域的方法。
具體調整策略很多，簡單的做法如下:
設置參數如下$\epsilon=0.01,\eta_1=0.01,\eta_2=0.75,\gamma_1= 0.5,\gamma _2= 2,\lambda=1$
$\left\{\begin{matrix} q<\eta _1, & \lambda =\gamma_2 \lambda \\ \eta_2\leq q\leq \eta _1, & \lambda = \lambda \\ q>\eta_2, & \lambda = \gamma_1\lambda \end{matrix}\right.$
設置收斂條件：

1. 決定係數R>0.997
2. $F_{k+1}-F_k<10e-6$
3. $||J_f*r(w)||^2<\epsilon$
4. 超過最大運算次數

可以多選或者選其一

Matlab代碼

%Levenberg-Marquardt

clear;
load census;
x1 = cdate ;
y1 = pop ;
m = length(x1);
%parameters
eps = 0.01;
eta1 = 0.01;
eta2 = 0.75;
gama1 = 0.5;
gama2 = 2;
lamda = 1;
%init a,b,c,d
d = max(y1)+1;
a = min(y1)-0.1;
y2 = log((y1-d)./(a-y1));
x2 = log(x1);
[curve2,gof2] = fit(x2,y2, 'poly1');
b = -curve2.p1;
c = exp(curve2.p2/b);
%LMF
w=[a,b,c,d]'; 
[res,R,fit] = evaluateFit(y1,x1,w);
mse = 0.5*sum((y1-fit).^2); 
r = y1-fit;
for k = 1:1:1000
    JacobiMatrix = getJaccobiMatrix(x1,a,b,c,d);
    HessenMatrix = (JacobiMatrix)'*(JacobiMatrix)+lamda*eye(4);
    delta_w = inv(HessenMatrix)*JacobiMatrix'*r;
    w_new = w+delta_w;
    [res,R,fit_new] = evaluateFit(y1,x1,w_new);
    mse_new =0.5*sum( (y1-fit_new).^2);
    q = (mse - mse_new)/(r'*JacobiMatrix*delta_w+0.5*delta_w'*HessenMatrix*delta_w);
     if q<eta1
        lamda = lamda * gama2;
        continue;
    elseif  q>eta2
         lamda = lamda * gama1;
     end
    %coverage
    de = abs(norm(JacobiMatrix'*r));
    if de<eps
        break;
    end
    %update compution result
    fit = fit_new;
    mse = mse_new;
    r  = y1-fit;
    %update a b c d
    a = w_new(1);b = w_new(2);c = w_new(3);d= w_new(4);
    w=w_new; 

end
function [JacobiMatrix] = getJaccobiMatrix(x1,a,b,c,d)
   JacobiMatrix = ones(length(x1),4);
   for i = 1:1:length(x1)
        JacobiMatrix(i,1) = calc_pA(x1(i),a,b,c,d);
        JacobiMatrix(i,2) = calc_pB(x1(i),a,b,c,d);
        JacobiMatrix(i,3) = calc_pC(x1(i),a,b,c,d);
        JacobiMatrix(i,4) = calc_pD(x1(i),a,b,c,d);
    end
end
function [res,R2,fit] = evaluateFit(y,x,w)
fit = getFittingValue(x,w);
res =  norm(y-fit)/sqrt(length(fit));
yu =  mean(y);
R2 = 1 - norm(y-fit)^2/norm(y - yu)^2;
end
function fit = getFittingValue(x,w)
len = length(x);
fit = ones(len,1);
for i = 1:1:len
    fit(i)  = hypothesis(x(i),w);
end
end
function val  = hypothesis(x,w)
a = w(1);b= w(2);c= w(3);d= w(4);
val = d+(a-d)/(1+(x/c)^b);
end
function val = calc_pA(x,A,B,C,D)
val = 1/((x/C)^B + 1);
end
function val = calc_pB(x,A,B,C,D)
val = -(log(x/C)*(A - D)*(x/C)^B)/((x/C)^B + 1)^2;
end
function val = calc_pC(x,A,B,C,D)
val = (B*x*(A - D)*(x/C)^(B - 1))/(C^2*((x/C)^B + 1)^2);
end
function val = calc_pD(x,A,B,C,D)
val = 1 - 1/((x/C)^B + 1);
end

後面，研究一下四參數擬合的約束情況