最近闲着没事，想把coursera上斯坦福ML课程里面的练习，用Python来实现一下，一是加深ML的基础，二是熟悉一下numpy，matplotlib，scipy这些库。
在EX2中，优化theta使用了matlab里面的fminunc函数，不知道Python里面如何实现。搜索之后，发现stackflow上有人提到用scipy库里面的minimize函数来替代。我尝试直接调用我的costfunction和grad，程序报错，提示(3,)和(100,1)dim维度不等，gradient vector不对之类的，试了N多次后，终于发现问题何在。。
首先来看看使用np.info(minimize)查看函数的介绍，传入的参数有：
fun : callable
The objective function to be minimized.
``fun(x, *args) -> float``
where x is an 1-D array with shape (n,) and `args`
is a tuple of the fixed parameters needed to completely
specify the function.
x0 : ndarray, shape (n,)
Initial guess. Array of real elements of size (n,),
where 'n' is the number of independent variables.
args : tuple, optional
Extra arguments passed to the objective function and its
derivatives (`fun`, `jac` and `hess` functions).
method : str or callable, optional
Type of solver. Should be one of
- 'Nelder-Mead' :ref:`(see here) <optimize.minimize-neldermead>`
- 'Powell' :ref:`(see here) <optimize.minimize-powell>`
- 'CG' :ref:`(see here) <optimize.minimize-cg>`
- 'BFGS' :ref:`(see here) <optimize.minimize-bfgs>`
- 'Newton-CG' :ref:`(see here) <optimize.minimize-newtoncg>`
- 'L-BFGS-B' :ref:`(see here) <optimize.minimize-lbfgsb>`
- 'TNC' :ref:`(see here) <optimize.minimize-tnc>`
- 'COBYLA' :ref:`(see here) <optimize.minimize-cobyla>`
- 'SLSQP' :ref:`(see here) <optimize.minimize-slsqp>`
- 'trust-constr':ref:`(see here) <optimize.minimize-trustconstr>`
- 'dogleg' :ref:`(see here) <optimize.minimize-dogleg>`
- 'trust-ncg' :ref:`(see here) <optimize.minimize-trustncg>`
- 'trust-exact' :ref:`(see here) <optimize.minimize-trustexact>`
- 'trust-krylov' :ref:`(see here) <optimize.minimize-trustkrylov>`
- custom - a callable object (added in version 0.14.0),
see below for description.
If not given, chosen to be one of ``BFGS``, ``L-BFGS-B``, ``SLSQP``,
depending if the problem has constraints or bounds.
jac : {callable, '2-point', '3-point', 'cs', bool}, optional
Method for computing the gradient vector. Only for CG, BFGS,
Newton-CG, L-BFGS-B, TNC, SLSQP, dogleg, trust-ncg, trust-krylov,
trust-exact and trust-constr. If it is a callable, it should be a
function that returns the gradient vector:
``jac(x, *args) -> array_like, shape (n,)``
where x is an array with shape (n,) and `args` is a tuple with
the fixed parameters. Alternatively, the keywords
{'2-point', '3-point', 'cs'} select a finite
difference scheme for numerical estimation of the gradient. Options
'3-point' and 'cs' are available only to 'trust-constr'.
If `jac` is a Boolean and is True, `fun` is assumed to return the
gradient along with the objective function. If False, the gradient
will be estimated using '2-point' finite difference estimation.
需要注意的是fun关键词参数里面的函数，需要把优化的theta放在第一个位置，X,y，放到后面。并且，theta在传入的时候一定要是一个一维shape（n,）的数组，不然会出错。
然后jac是梯度，这里的有两个地方要注意，第一个是传入的theta依然要是一个一维shape(n,)，第二个是返回的梯度也要是一个一维shape(n,)的数组。
总之，关键在于传入的theta一定要是一个1D shape(n,)的，不然就不行。我之前为了方便已经把theta塑造成了一个（n,1）的列向量，导致使用minimize时会报错。所以，学会用help看说明可谓是相当重要啊~
import numpy as np
import pandas as pd
import scipy.optimize as op
def LoadData(filename):
data=pd.read_csv(filename,header=None)
data=np.array(data)
return data
def ReshapeData(data):
m=np.size(data,0)
X=data[:,0:2]
Y=data[:,2]
Y=Y.reshape((m,1))
return X,Y
def InitData(X):
m,n=X.shape
initial_theta = np.zeros(n + 1)
VecOnes = np.ones((m, 1))
X = np.column_stack((VecOnes, X))
return X,initial_theta
def sigmoid(x):
z=1/(1+np.exp(-x))
return z
def costFunction(theta,X,Y):
m=X.shape[0]
J = (-np.dot(Y.T, np.log(sigmoid(X.dot(theta)))) - \
np.dot((1 - Y).T, np.log(1 - sigmoid(X.dot(theta))))) / m
return J
def gradient(theta,X,Y):
m,n=X.shape
theta=theta.reshape((n,1))
grad=np.dot(X.T,sigmoid(X.dot(theta))-Y)/m
return grad.flatten()
if __name__=='__main__':
data = LoadData('ex2data1csv.csv')
X, Y = ReshapeData(data)
X, initial_theta = InitData(X)
result = op.minimize(fun=costFunction, x0=initial_theta, args=(X, Y), method='TNC', jac=gradient)
print(result)
最后结果如下，符合MATLAB里面用fminunc优化的结果（fminunc:cost:0.203,theta:-25.161,0.206,0.201）
fun: array([0.2034977])
jac: array([8.95038682e-09, 8.16149951e-08, 4.74505693e-07])
message: 'Local minimum reached (|pg| ~= 0)'
nfev: 36
nit: 17
status: 0
success: True
x: array([-25.16131858, 0.20623159, 0.20147149])
此外，由于知道cost在0.203左右，所以我用最笨的梯度下降试了一下，由于后面实在是太慢了，所以设置while J>0.21，循环了大概13W次。。可见，使用集成好的优化算法是多么重要。。。还有，在以前的理解中，如果一个学习速率不合适，J会一直发散，但是昨天的实验发现，有的速率开始会发散，后面还是会收敛。