Algorithm: K-Means

K-Means

The K-Means is  an unsupervised learning algorithm which has the input sample data without label.

Sometimes we use the CRM system to manage the relationship between the customer. The concept is clustering

 

 

The application of clustering: 

 

It can also be used to compress the images

 

The concept of K-mean:

1. rearange each sample to the nearest category by compare the distances.

2. for each category we calculate the center point.

For K = 2

We choose two center point randomly

We clustering each example to each category respect to the center points.

Then we recalculate the center point by the calculating the mean coordinate of each points of the respect cluster(category.)

We use the new center points for clustering.

Then we recalculate the center point again.

And we do the cluster again:

If the new center point is the same as the previous iteration, then we can stop the calculation for converge.

 

Python Implementation for K-Mean

# import package
from copy import deepcopy
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# set paramter k for K-means
k = 3

# randomize the center point. and save the result into C
X = np.random.random((200, 2)) * 10
C_x = np.random.choice(range(0, int(np.max(X[:, 0]))), size = k, replace = False)
C_y = np.random.choice(range(0, int(np.max(X[:, 1]))), size = k, replace = False)
C = np.array(list(zip(C_x, C_y)), dtype = np.float32)

print("The init center point is :")
print(C)

# plot the center point
plt.scatter(X[:, 0], X[:, 1], c = '#050505', s = 7)
plt.scatter(C[:, 0], C[:, 1], marker = '*', s = 300, c = 'g')
plt.show()

 

# store the previous center point
C_old = np.zeros(C.shape)
clusters = np.zeros(len(X))

# calculate the distance
def dist(a, b, ax = 1):
    return np.linalg.norm(a - b, axis = ax)

error = dist(C, C_old, None)
# iteration for K-mean clustering until converge(that is the error = 0)
while error != 0:
    # Assigning each value to its closest cluster
    for i in range(len(X)):
        distances = dist(X[i], C)
        category = np.argmin(distances)
        clusters[i] = category
    
    # We save the old center points
    C_old = deepcopy(C)
    # and calculate the new center points
    for i in range(k):
        points = [X[j] for j in range(len(X)) if clusters[j] == i]
        C[i] = np.mean(points, axis = 0)
    error = dist(C, C_old, None)

# plot the clusters
colors = ['r', 'g', 'b', 'y', 'c', 'm']
fig, ax = plt.subplots()
for i in range(k):
    points = np.array([X[j] for j in range(len(X)) if clusters[j] == i])
    ax.scatter(points[:, 0], points[:, 1], s = 7, c = colors[i])
ax.scatter(C[:, 0], C[:, 1], marker = '*', s = 200, c = '#050505')
plt.show()

 

K-Means in detail

 

What is the object function os K-mean？

At first ,we don't known the cluster and the center point, how do we define the loss function?

we obtain two parameters γ and μ from the object function of K-mean

We can optimize the parameter separately，the approach is set one parameters as known and we optimize the other one.

 

Does the K-means must converge？

Alternative Optimization

1）fix {uk} to solve {γik}

calculate the distance between sample to the center points

tag each sample to the specific cluster

2) Fix{γik} to recalculate center{uk}

It is an optimization problem, the step 1 well let our object function become small.

the step 2 will let our object function become small.

Coordinate Descent

EM Algorithm(GMM)

Gaussian Mixer Model

K-Means named hard cluster, GMM - soft cluster

 

The different start center point will result different result

Because we could only obtain the local optima due to the object function of k-mean is not convex

 

How to choose K for K-mean？

Recall the loss function

base on the change of the L to choose the K

 

Vector Qualization

This method can be used to compress the image data. The core concept is that we use the k-mean to present the similary color pixels

#import packages
from pylab import imread, imshow, figure, show, subplot
import numpy as np
from sklearn.cluster import KMeans
from copy import deepcopy

# read the image data
img = imread('Tulips.jpg')
imshow(img)
show()
# convert three dimension tensor into two dimension matrix
pixel = img.reshape(img.shape[0] * img.shape[1], 3)
pixel_new = deepcopy(pixel)

print (img.shape)

# construct K-means model
model = KMeans(n_clusters = 3)
labels = model.fit_predict(pixel)
palette = model.cluster_centers_

for i in range(len(pixel)):
    pixel_new[i,:] = palette[labels[i]]

# reshow the compressed image
imshow(pixel_new.reshape(img.shape[0], img.shape[1], 3))
show()

 

原始圖像，

進行三色壓縮後的效果（K = 3）：

進行十六色 （K-means for K = 16）壓縮後的效果：