K-Nearest Neighbor KNN算法
如果一個樣本在特徵空間中的K個最相似（即特徵空間中最鄰近）的樣本中的大多數屬於某一個類別，則該樣本也屬於這個類別。
所選擇的鄰居都是已經正確分類的對象。
如果K=3，則綠色圓形的類別與紅色三角形相同
如果K=5，則綠色圓形的類別與藍色正方形相同
The choice of distance is a hyperparameter.
K的選擇： 不同問題不同對待，一般是嘗試很多不同的K值，找到最好的。
- Choose hyperparameters that work best on the data.
BAD：K=1 always works perfectly on training data
- Split data into train and test, choose hyperparameters that work best on test data.
BAD: test set is a proxy for the generalization performance, using only at the end.
(交叉驗證）用四個fold作爲訓練，一個fold作爲驗證，循環。
距離選擇：
K-Nearest Neighbor on images never used.
效率低
並不提供詳細的信息
KNN算法matlab實現（一次預測一個點）
function ytest = KNN(X, y, Xtest, k)
% X = [1,3; 2,2; 1,1; 3,1; 3,0.5; 2,0.5]
% y = [0;0;1;1;1;1]
% Xtest = [1,2]
% k =3 k =5
m = size(X,1);
n = size(X,2);
mtest = size(Xtest,1);
dis = zeros(m,1);
for i = 1:m,
temp = 0;
for j = 1:n
temp = temp + (Xtest(1,j) - X(i,j))^2;
end;
temp = temp.^0.5;
dis(i,1) = temp;
end;
ordered_dis = sort(dis);
disp(ordered_dis);
max_dis = ordered_dis(k);
index = find(dis<=max_dis);
num = size(index,1);
tar_y = y(index);
count = zeros(num,1);
for i = 1:num,
count(i) = size(find(tar_y == tar_y(i)),1);
end;
tar_index = find(count==max(count));
ytest = tar_y(tar_index(1));
Python實現
import numpy
def KNN(X, y, Xtest, k):
m = X.shape[0]
n = X.shape[1]
print(m)
print(n)
ytest = 0
temp = (numpy.tile(Xtest, (m, 1)) - X)**2
dis = []
for z in range(m):
s = 0
for l in range(n):
s = s + temp[z][l]
dis.append(s ** 0.5)
print(dis)
index = numpy.argsort(dis)
index = index[0:k]
print(index)
tar_y = []
for i in index:
tar_y.append(y[i])
print(tar_y)
count = 0
class_index = 0
for j in tar_y:
print('j =',j)
if count < tar_y.count(j):
count = tar_y.count(j)
print(count)
ytest = tar_y[class_index]
class_index = class_index + 1
return ytest
X = numpy.array([[1, 3], [2, 2], [1, 1], [3, 1], [3, 0.5], [2, 0.5]])
y = [0, 0, 1, 1, 1, 1]
k = 3
Xtest = [1, 2]
ytest = KNN(X, y, Xtest, k)
print(ytest)
