1.在需要使用bool矩陣選取元素數值時，應該將bool類型轉換爲float類型（此處的錯誤導致我debug了很久）：
a = np.array([[1,-1,3],[2,3,5],[-1,-3,4]])
bool_m = a > 0
b = np.sum(bool_m,axis=1)
bool_m[1,:] = b #此時bool_m中的元素還是true/false
print(bool_m)
bool_m = bool_m*1.
bool_m[1,:] = b #變爲數值
print(bool_m)2,通過兩個列表作爲座標來選取矩陣中的元素，還可以使用
a = [1,1,3]
b = [2,3,1]
c = np.arange(16).reshape(4,4)
print(c[a,b])
c[a,b] = np.arange(3)* 兩層NN的fine-tune 的訓練
hyperparameter :
- learning_rate
- layer_size（hidden_layer_node_num）
- traning_epoch
- regularization
測試情況：lr = 1e-4 ,node_num=50 , traning_time = 2000*100,reg=0.25
（紅色代表training acc,紫色代表testing acc(training acc僅指training batch的acc) )
存在的問題：
1. loss冷啓動，表明loss沒有初始化好
2. loss下降過慢，表明lr過小
3. training和testing的acc曲線交疊在一起，表明model capcity不夠。
fine-tune過程
step1,Preprocess the data,zero center就行
step2,choose a architecture:
1,. double check the loss the reasonable(有reg和沒有reg的情況)
2, try to overfitting a small portion of the training data (reg=0)
3, coarse -> fine
1) corase
iter_num = 100
for it in range(iter_num):
new_net = TwoLayerNet(input_size, hidden_size, num_classes)
lr = 10**np.random.uniform(-3,-6)
reg = 10**np.random.uniform(-5,5)
stats = new_net.train(X_train, y_train, X_val, y_val,num_iters=5000, batch_size=200,
learning_rate=lr, learning_rate_decay=0.95,reg=reg, verbose=False)
val_acc = (new_net.predict(X_val) == y_val).mean()
print('lr: %e reg: %e Validation accuracy:%f ' %(lr,reg,val_acc))結果如下：（因爲時間原因，沒有運行到100次就break了）
lr: 6.510157e-04 reg: 3.669642e+04 Validation accuracy:0.087000
lr: 2.643562e-06 reg: 1.044038e-01 Validation accuracy:0.235000
lr: 1.558228e-04 reg: 1.763366e-04 Validation accuracy:0.465000
lr: 3.269025e-05 reg: 1.024827e+00 Validation accuracy:0.309000
lr: 3.788300e-04 reg: 1.748592e-01 Validation accuracy:0.511000
lr: 8.619492e-06 reg: 2.045766e+00 Validation accuracy:0.180000
lr: 4.993109e-06 reg: 1.745723e+04 Validation accuracy:0.102000
lr: 4.883415e-04 reg: 1.855065e-01 Validation accuracy:0.521000
lr: 2.214773e-04 reg: 4.860300e-02 Validation accuracy:0.491000
lr: 2.573395e-06 reg: 3.769904e-03 Validation accuracy:0.246000
lr: 2.109817e-05 reg: 3.371226e-02 Validation accuracy:0.274000
lr: 2.708635e-04 reg: 2.103766e+03 Validation accuracy:0.098000
lr: 8.793248e-04 reg: 2.777957e+02 Validation accuracy:0.078000
lr: 7.318431e-04 reg: 4.973982e-04 Validation accuracy:0.540000
lr: 3.145155e-04 reg: 2.485930e-05 Validation accuracy:0.505000
lr: 2.975205e-04 reg: 6.164198e+00 Validation accuracy:0.451000
lr: 7.481808e-05 reg: 6.600768e+02 Validation accuracy:0.078000
lr: 9.190149e-06 reg: 1.595968e+00 Validation accuracy:0.193000
lr: 2.421630e-04 reg: 1.449934e+03 Validation accuracy:0.079000
lr: 1.483436e-06 reg: 7.021175e-03 Validation accuracy:0.205000
lr: 4.497050e-06 reg: 8.490002e+00 Validation accuracy:0.213000
lr: 1.298748e-04 reg: 5.249712e+01 Validation accuracy:0.238000
lr: 2.802625e-04 reg: 7.045640e+03 Validation accuracy:0.078000
lr: 6.116626e-06 reg: 1.250906e+01 Validation accuracy:0.205000
lr: 5.290487e-05 reg: 1.282979e-01 Validation accuracy:0.369000
lr: 4.644485e-06 reg: 2.415987e-04 Validation accuracy:0.215000
lr: 4.887054e-04 reg: 1.509442e-04 Validation accuracy:0.527000
lr: 5.136713e-05 reg: 3.751147e+01 Validation accuracy:0.258000
lr: 4.018885e-05 reg: 1.130037e-04 Validation accuracy:0.337000
lr: 2.501210e-05 reg: 1.218030e+00 Validation accuracy:0.278000
lr: 5.686478e-06 reg: 4.445485e+03 Validation accuracy:0.087000
lr: 2.776303e-04 reg: 4.061512e-01 Validation accuracy:0.491000
lr: 7.090708e-04 reg: 2.382944e+02 Validation accuracy:0.087000
lr: 1.683200e-04 reg: 9.081885e-04 Validation accuracy:0.473000
lr: 3.538319e-05 reg: 2.933638e-05 Validation accuracy:0.329000
lr: 2.639026e-05 reg: 1.571852e+02 Validation accuracy:0.087000 找到相對比較好的結果：
lr: 4.887054e-04 reg: 1.509442e-04 Validation accuracy:0.527000
lr: 7.318431e-04 reg: 4.973982e-04 Validation accuracy:0.540000
lr: 3.145155e-04 reg: 2.485930e-05 Validation accuracy:0.505000
lr: 4.883415e-04 reg: 1.855065e-01 Validation accuracy:0.521000
lr: 3.788300e-04 reg: 1.748592e-01 Validation accuracy:0.511000**2)**fine
大致可以將lr範圍確定到 e-4 ~e-3 ,reg的範圍確定到 1e-4~1e-1，
sample20次的結果：
lr: 4.058940e-04 reg: 1.929913e-02 Validation accuracy:0.534000
lr: 4.222605e-04 reg: 1.200906e-02 Validation accuracy:0.503000
lr: 2.211911e-04 reg: 1.255418e-02 Validation accuracy:0.485000
lr: 2.588576e-04 reg: 8.566199e-04 Validation accuracy:0.488000
lr: 1.229843e-04 reg: 2.898743e-04 Validation accuracy:0.466000
lr: 1.985153e-04 reg: 4.509311e-04 Validation accuracy:0.473000
lr: 1.175998e-04 reg: 1.462556e-03 Validation accuracy:0.453000
lr: 6.840469e-04 reg: 2.621410e-04 Validation accuracy:0.520000
lr: 5.093372e-04 reg: 1.555551e-03 Validation accuracy:0.525000
lr: 2.212163e-04 reg: 2.355791e-03 Validation accuracy:0.495000
lr: 2.239080e-04 reg: 2.024601e-02 Validation accuracy:0.495000
lr: 2.805192e-04 reg: 4.729087e-02 Validation accuracy:0.519000
lr: 2.400963e-04 reg: 2.614076e-03 Validation accuracy:0.484000
lr: 2.626168e-04 reg: 3.921650e-02 Validation accuracy:0.498000
lr: 4.159985e-04 reg: 1.809018e-04 Validation accuracy:0.515000
lr: 2.357435e-04 reg: 6.627438e-03 Validation accuracy:0.496000
lr: 6.429720e-04 reg: 5.428195e-04 Validation accuracy:0.507000
lr: 1.941110e-04 reg: 5.121491e-04 Validation accuracy:0.470000
lr: 2.136214e-04 reg: 1.695283e-03 Validation accuracy:0.482000
lr: 4.427498e-04 reg: 9.225977e-03 Validation accuracy:0.527000 以及上一次的最好結果
lr: 7.318431e-04 reg: 4.973982e-04 Validation accuracy:0.540000 相比上一次sample，這次明顯更好，比較好的選擇(0,7,8,19)從左到右，從上到下的四副圖像如下：
最後選擇第19個組合，因爲上升空間較大，還沒有趨於平緩。
最終結果：
lr: 4.427498e-04 reg: 9.225977e-03 Validation accuracy:0.527000 
batch=200,node_num=100 訓練15000次後，在validation上的結果爲52.1%,在test上測試,acc爲52.9%。但是由圖可知，存在overfitting現象所以我們再嘗試訓練7500次，10000次左右。結果如下：
綜上選擇取得最好的validation acc的model :batch=200,node_num=100,training_time=7500
改進方向：
- W的初始化採用Xavier
- 使用更深的網絡結構
Note:Step2中try to overfitting a small portion of the training data (reg=0)
但是奇怪的是在後續過程中，我不小心使用了std*randn的方式，也得到了很好的結果，而且我又重新復現了一次，究其原因我也不是太清楚。這會不會表明使用Xavier會得到更好的結果呢？
於是決定使用Xavier初始化方法再重新做一次實驗
lr: 6.341384e-05 reg: 1.769403e-03 Validation accuracy:0.335000 0
lr: 3.852774e-04 reg: 3.193370e-04 Validation accuracy:0.384000
lr: 2.094796e-06 reg: 5.271914e-05 Validation accuracy:0.260000
lr: 3.409431e-04 reg: 6.048618e-02 Validation accuracy:0.378000
lr: 2.334389e-06 reg: 1.421406e-04 Validation accuracy:0.245000
lr: 1.074157e-06 reg: 4.755761e-01 Validation accuracy:0.222000
lr: 4.507436e-05 reg: 3.534975e-03 Validation accuracy:0.304000
lr: 5.701219e-05 reg: 2.793682e+01 Validation accuracy:0.337000
lr: 1.424817e-04 reg: 1.072272e-05 Validation accuracy:0.337000
lr: 3.476407e-06 reg: 2.268886e+01 Validation accuracy:0.257000
lr: 3.107524e-05 reg: 3.554090e-03 Validation accuracy:0.302000
lr: 7.066331e-05 reg: 8.943052e-04 Validation accuracy:0.294000
lr: 7.147195e-06 reg: 6.824755e+01 Validation accuracy:0.335000
lr: 2.840902e-06 reg: 4.868638e-02 Validation accuracy:0.272000
lr: 3.801703e-05 reg: 2.096270e+01 Validation accuracy:0.442000 14
lr: 4.093321e-06 reg: 2.500971e-01 Validation accuracy:0.269000
lr: 2.001587e-05 reg: 5.745243e+01 Validation accuracy:0.349000
lr: 6.730932e-04 reg: 1.897193e+00 Validation accuracy:0.509000 17
lr: 6.049389e-04 reg: 4.889209e-04 Validation accuracy:0.455000
lr: 1.512126e-05 reg: 1.814202e+00 Validation accuracy:0.320000
lr: 3.772819e-05 reg: 2.384045e-01 Validation accuracy:0.313000
lr: 8.231728e-04 reg: 5.110095e+00 Validation accuracy:0.462000
lr: 6.948852e-05 reg: 1.765428e-02 Validation accuracy:0.367000
lr: 5.072889e-05 reg: 1.018637e-01 Validation accuracy:0.355000
lr: 2.877371e-05 reg: 7.400674e-02 Validation accuracy:0.312000
lr: 2.735792e-04 reg: 4.399957e-04 Validation accuracy:0.371000
lr: 2.983375e-05 reg: 3.060068e-01 Validation accuracy:0.315000
lr: 5.892237e-05 reg: 2.432885e+00 Validation accuracy:0.335000
lr: 1.810665e-04 reg: 3.647910e-01 Validation accuracy:0.381000
lr: 1.017972e-05 reg: 4.084320e-01 Validation accuracy:0.308000
30th lr: 4.322279e-06 reg: 2.016450e-03 Validation accuracy:0.267000
31th lr: 1.052033e-04 reg: 8.066398e-02 Validation accuracy:0.329000
32th lr: 1.940260e-04 reg: 3.866905e-01 Validation accuracy:0.380000
33th lr: 2.908686e-04 reg: 2.485808e+01 Validation accuracy:0.334000
34th lr: 6.938006e-04 reg: 3.538238e-02 Validation accuracy:0.431000
35th lr: 1.897502e-04 reg: 2.381476e-04 Validation accuracy:0.354000
36th lr: 1.028106e-04 reg: 2.249541e-01 Validation accuracy:0.365000
37th lr: 9.773299e-06 reg: 4.400711e-04 Validation accuracy:0.319000
38th lr: 8.901756e-05 reg: 6.154393e+01 Validation accuracy:0.237000
39th lr: 2.382143e-04 reg: 3.235230e-02 Validation accuracy:0.376000
40th lr: 6.426793e-04 reg: 2.050400e-01 Validation accuracy:0.480000
41th lr: 3.168815e-04 reg: 5.118725e+00 Validation accuracy:0.463000
42th lr: 6.664813e-04 reg: 4.848741e-01 Validation accuracy:0.469000
43th lr: 8.270327e-04 reg: 2.412895e-03 Validation accuracy:0.431000
44th lr: 3.473371e-05 reg: 1.298962e-02 Validation accuracy:0.321000
45th lr: 5.198179e-05 reg: 3.302815e+01 Validation accuracy:0.321000
46th lr: 2.981851e-05 reg: 4.685633e-05 Validation accuracy:0.327000
47th lr: 1.331836e-06 reg: 1.185023e-03 Validation accuracy:0.225000
48th lr: 2.388945e-05 reg: 1.057670e+01 Validation accuracy:0.337000
49th lr: 1.171363e-04 reg: 1.908012e-01 Validation accuracy:0.354000
50th lr: 5.786395e-06 reg: 4.114150e-05 Validation accuracy:0.270000
51th lr: 4.908611e-04 reg: 1.332642e-04 Validation accuracy:0.440000
52th lr: 1.825870e-06 reg: 1.507947e-01 Validation accuracy:0.248000
53th lr: 8.423663e-06 reg: 1.810298e-05 Validation accuracy:0.304000
54th lr: 3.539308e-06 reg: 1.211471e-05 Validation accuracy:0.271000
55th lr: 1.768898e-06 reg: 1.316256e-02 Validation accuracy:0.238000
56th lr: 1.910679e-05 reg: 4.767425e-03 Validation accuracy:0.328000
57th lr: 2.361348e-04 reg: 1.944416e-05 Validation accuracy:0.383000
58th lr: 6.626397e-05 reg: 5.926622e+01 Validation accuracy:0.244000
59th lr: 7.005012e-04 reg: 5.000929e-03 Validation accuracy:0.434000 相對比較好的點
lr: 3.801703e-05 reg: 2.096270e+01 Validation accuracy:0.442000
lr: 6.730932e-04 reg: 1.897193e+00 Validation accuracy:0.509000 ok 17
lr: 6.049389e-04 reg: 4.889209e-04 Validation accuracy:0.455000 ok 18
lr: 8.231728e-04 reg: 5.110095e+00 Validation accuracy:0.462000
40th lr: 6.426793e-04 reg: 2.050400e-01 Validation accuracy:0.480000 ok
41th lr: 3.168815e-04 reg: 5.118725e+00 Validation accuracy:0.463000
42th lr: 6.664813e-04 reg: 4.848741e-01 Validation accuracy:0.469000 ok
51th lr: 4.908611e-04 reg: 1.332642e-04 Validation accuracy:0.440000 oklr=1e-4~1e-3,1e-4~1e-3
60th lr: 4.709525e-04 reg: 8.095851e-04 Validation accuracy:0.439000
61th lr: 2.726294e-04 reg: 1.949236e-04 Validation accuracy:0.377000
62th lr: 4.297156e-04 reg: 4.621484e-04 Validation accuracy:0.443000
63th lr: 2.489500e-04 reg: 6.292587e-04 Validation accuracy:0.375000
64th lr: 2.331295e-04 reg: 2.676282e-04 Validation accuracy:0.369000
65th lr: 2.781939e-04 reg: 1.396553e-04 Validation accuracy:0.365000
66th lr: 9.496852e-04 reg: 4.296325e-04 Validation accuracy:0.438000
67th lr: 7.280289e-04 reg: 4.663365e-04 Validation accuracy:0.459000
68th lr: 5.814480e-04 reg: 2.177770e-04 Validation accuracy:0.457000
69th lr: 5.976867e-04 reg: 3.075948e-04 Validation accuracy:0.455000
70th lr: 1.489146e-04 reg: 4.210730e-04 Validation accuracy:0.337000
71th lr: 5.585008e-04 reg: 6.217684e-04 Validation accuracy:0.437000
72th lr: 4.433013e-04 reg: 1.457733e-04 Validation accuracy:0.450000
73th lr: 2.387401e-04 reg: 8.833313e-04 Validation accuracy:0.371000
74th lr: 2.781313e-04 reg: 6.748215e-04 Validation accuracy:0.396000
75th lr: 1.204839e-04 reg: 2.052842e-04 Validation accuracy:0.344000
76th lr: 8.505226e-04 reg: 2.182052e-04 Validation accuracy:0.446000
77th lr: 3.224040e-04 reg: 1.535878e-04 Validation accuracy:0.406000
78th lr: 5.363216e-04 reg: 1.629328e-04 Validation accuracy:0.437000
79th lr: 3.087325e-04 reg: 7.169442e-04 Validation accuracy:0.420000
80th lr: 1.664995e-04 reg: 9.278335e-04 Validation accuracy:0.392000
81th lr: 6.931702e-04 reg: 1.442553e-04 Validation accuracy:0.443000
82th lr: 3.088621e-04 reg: 2.243105e-04 Validation accuracy:0.384000
83th lr: 4.278311e-04 reg: 1.159277e-04 Validation accuracy:0.409000
84th lr: 1.001049e-04 reg: 4.231650e-04 Validation accuracy:0.321000
85th lr: 3.351125e-04 reg: 8.224279e-04 Validation accuracy:0.410000
86th lr: 9.641541e-04 reg: 9.183995e-04 Validation accuracy:0.465000
87th lr: 1.260288e-04 reg: 4.936542e-04 Validation accuracy:0.331000
88th lr: 7.027471e-04 reg: 4.970635e-04 Validation accuracy:0.459000
89th lr: 1.710624e-04 reg: 1.329212e-04 Validation accuracy:0.341000
lr=1e-4~1e-3,reg=1e-1~1e1
90th lr: 1.887898e-04 reg: 5.492338e-01 Validation accuracy:0.369000
91th lr: 6.789602e-04 reg: 2.696616e-01 Validation accuracy:0.485000
92th lr: 9.224811e-04 reg: 8.581825e-01 Validation accuracy:0.522000
93th lr: 1.916202e-04 reg: 2.981688e+00 Validation accuracy:0.502000
94th lr: 4.389360e-04 reg: 2.801716e+00 Validation accuracy:0.491000
95th lr: 5.546313e-04 reg: 4.809618e-01 Validation accuracy:0.485000
96th lr: 2.543303e-04 reg: 1.886383e+00 Validation accuracy:0.485000
97th lr: 5.447017e-04 reg: 5.978606e-01 Validation accuracy:0.485000
98th lr: 7.483245e-04 reg: 8.129600e+00 Validation accuracy:0.439000
99th lr: 1.272105e-04 reg: 2.349757e+00 Validation accuracy:0.404000
100th lr: 5.265308e-04 reg: 2.310304e-01 Validation accuracy:0.468000
101th lr: 2.381013e-04 reg: 1.190625e-01 Validation accuracy:0.392000
102th lr: 3.632505e-04 reg: 1.754443e+00 Validation accuracy:0.493000
103th lr: 6.462288e-04 reg: 7.055405e-01 Validation accuracy:0.496000
104th lr: 6.799179e-04 reg: 1.057823e-01 Validation accuracy:0.455000
105th lr: 1.349350e-04 reg: 1.536159e+00 Validation accuracy:0.377000
106th lr: 5.687849e-04 reg: 1.006980e+00 Validation accuracy:0.506000
107th lr: 2.301599e-04 reg: 5.041087e+00 Validation accuracy:0.457000
108th lr: 1.208513e-04 reg: 4.028463e-01 Validation accuracy:0.316000
109th lr: 1.854946e-04 reg: 5.846844e+00 Validation accuracy:0.443000
110th lr: 3.002745e-04 reg: 2.760922e+00 Validation accuracy:0.495000
111th lr: 2.479276e-04 reg: 6.671733e-01 Validation accuracy:0.415000
112th lr: 1.645383e-04 reg: 3.215961e-01 Validation accuracy:0.347000
113th lr: 2.039684e-04 reg: 1.474140e-01 Validation accuracy:0.360000
114th lr: 1.500013e-04 reg: 2.389144e-01 Validation accuracy:0.333000
115th lr: 3.748992e-04 reg: 3.976034e-01 Validation accuracy:0.440000
116th lr: 6.600451e-04 reg: 2.083297e-01 Validation accuracy:0.488000
117th lr: 5.128263e-04 reg: 8.254960e+00 Validation accuracy:0.431000
118th lr: 1.038475e-04 reg: 2.729728e-01 Validation accuracy:0.305000
119th lr: 3.601892e-04 reg: 4.204359e-01 Validation accuracy:0.422000 選擇
92th lr: 9.224811e-04 reg: 8.581825e-01 Validation accuracy:0.522000 (batch_size=200)
訓練20,000次後，
Validation accuracy:0.554000 Test accuracy: 0.551 
總結：
*做實驗時及時記下代碼的改變，在sublime中改變代碼時要保存
