Table of Contents
autoencoder理解
-
學習訓練：前向傳播+後向誤差傳播
-
激活函數：隱藏層relu+輸出層sigmoid
-
目標函數：
-
算法：梯度下降求解
-
學習過程面臨問題：梯度消失——緩解方法(pretraining，Hinton等提出)
-
主要應用：數據壓縮（圖像中的效果不佳） 數據去噪。具體應用：
- 異常值檢測
- 數據去噪（圖像、音頻）
- 圖像修復
- 信息檢索
-
幾種類型
- Denoising AutoEncoder
- Sparse AutoEncoder
- Constructive AutoEncoder
- Variational AutoEncoder
-
發展歷程：
1986——1989——1994 -
特點
- 數據相關性。
- 數據有損性。
- 自動學習能力。
-
模型構成
encoder: X→F
decoder: F→X’
目標函數:argmin L2(X - X’)
一、autoencoder demo練習1
http://machinelearninguru.com/deep_learning/tensorflow/neural_networks/autoencoder/autoencoder.html
# To support both python 2 and python 3
from __future__ import division, print_function, unicode_literals
# Common imports
import numpy as np
import os
import sys
# to make this notebook's output stable across runs
def reset_graph(seed=42):
tf.reset_default_graph()
tf.set_random_seed(seed)
np.random.seed(seed)
# To plot pretty figures
get_ipython().run_line_magic('matplotlib', 'inline')
import matplotlib
import matplotlib.pyplot as plt
plt.rcParams['axes.labelsize'] = 14
plt.rcParams['xtick.labelsize'] = 12
plt.rcParams['ytick.labelsize'] = 12
# Where to save the figures
PROJECT_ROOT_DIR = "."
CHAPTER_ID = "autoencoders"
# 忽視warning
import warnings
warnings.filterwarnings("ignore")
1. autoencoder函數
import tensorflow.contrib.layers as lays
import tensorflow as tf
def autoencoder(inputs):
net = lays.conv2d(inputs, 32, [5,5], stride=2, padding='SAME')
net = lays.conv2d(net, 16, [5,5], stride=2, padding='SAME')
net = lays.conv2d(net, 8, [5,5], stride=4, padding='SAME')
net = lays.conv2d_transpose(net, 16, [5,5], stride=4, padding='SAME')
net = lays.conv2d_transpose(net, 32, [5,5], stride=2, padding='SAME')
net = lays.conv2d_transpose(net, 1, [5,5], stride=2, padding='SAME')
return net
2. 數據處理-2828變爲3232
# resize the shape of MNIST image from 28*28 to 32*32
import numpy as np
from skimage import transform
def resize_batch(imgs):
imgs = imgs.reshape((-1, 28, 28, 1))
resized_imgs = np.zeros((imgs.shape[0], 32, 32, 1))
for i in range(imgs.shape[0]):
resized_imgs[i, ..., 0] = transform.resize(imgs[i, ..., 0], (32, 32))
return resized_imgs
3. 圖關係網絡創建
batch_size = 500
epoch_num = 5
lr = 0.1
ae_inputs = tf.placeholder(tf.float32, (None, 32, 32, 1))
ae_outputs = autoencoder(ae_inputs)
loss = tf.reduce_mean(tf.square(ae_outputs - ae_inputs))
train_op = tf.train.AdamOptimizer(learning_rate=lr).minimize(loss)
# initialize the network
init = tf.global_variables_initializer()
4. 數據讀入與訓練
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets(r'C:\Users\Administrator\Documents\MNIST', one_hot=True)
batch_per_ep = mnist.train.num_examples//batch_size
with tf.Session() as sess:
sess.run(init)
for ep in range(epoch_num):
for batch_n in range(5):
batch_img, batch_label = mnist.train.next_batch(batch_size)
batch_img = batch_img.reshape((-1, 28, 28, 1))
batch_img = resize_batch(batch_img)
_, c = sess.run([train_op, loss], feed_dict={ae_inputs:batch_img})
print('*'*100)
print('Epoch: {} - cost = {:.5f}'.format((ep +1), c))
batch_img, batch_label = mnist.test.next_batch(50)
batch_img = resize_batch(batch_img)
recon_img = sess.run([ae_outputs], feed_dict={ae_inputs:batch_img})[0]
plt.figure(1)
plt.suptitle('reconstructed images')
for i in range(50):
plt.subplot(5, 10, i+1)
plt.imshow(recon_img[i,..., 0], cmap='Blues')
plt.figure(2)
plt.title('input images')
for i in range(50):
plt.subplot(5, 10, i+1)
plt.imshow(batch_img[i,...,0], cmap='gray')
plt.show()
Extracting C:\Users\Administrator\Documents\MNIST\train-images-idx3-ubyte.gz
Extracting C:\Users\Administrator\Documents\MNIST\train-labels-idx1-ubyte.gz
Extracting C:\Users\Administrator\Documents\MNIST\t10k-images-idx3-ubyte.gz
Extracting C:\Users\Administrator\Documents\MNIST\t10k-labels-idx1-ubyte.gz
****************************************************************************************************
Epoch: 1 - cost = 0.10160
****************************************************************************************************
Epoch: 1 - cost = 37594284.00000
****************************************************************************************************
Epoch: 1 - cost = 0.08536
****************************************************************************************************
Epoch: 1 - cost = 0.09980
****************************************************************************************************
Epoch: 1 - cost = 0.09825
****************************************************************************************************
Epoch: 2 - cost = 0.10323
****************************************************************************************************
Epoch: 2 - cost = 0.10384
****************************************************************************************************
Epoch: 2 - cost = 0.09964
****************************************************************************************************
Epoch: 2 - cost = 0.09995
****************************************************************************************************
Epoch: 2 - cost = 0.10047
****************************************************************************************************
Epoch: 3 - cost = 0.10135
****************************************************************************************************
Epoch: 3 - cost = 0.10175
****************************************************************************************************
Epoch: 3 - cost = 0.10085
****************************************************************************************************
Epoch: 3 - cost = 0.09877
****************************************************************************************************
Epoch: 3 - cost = 0.09804
****************************************************************************************************
Epoch: 4 - cost = 0.10006
****************************************************************************************************
Epoch: 4 - cost = 0.10135
****************************************************************************************************
Epoch: 4 - cost = 0.10089
****************************************************************************************************
Epoch: 4 - cost = 0.10399
****************************************************************************************************
Epoch: 4 - cost = 0.09836
****************************************************************************************************
Epoch: 5 - cost = 0.10019
****************************************************************************************************
Epoch: 5 - cost = 0.10136
****************************************************************************************************
Epoch: 5 - cost = 0.10194
****************************************************************************************************
Epoch: 5 - cost = 0.09951
****************************************************************************************************
Epoch: 5 - cost = 0.10232
二、autoencoder 練習2
理解encoder和decoder的過程
https://blog.keras.io/building-autoencoders-in-keras.html
import numpy as np
from keras.layers import Input, Dense
from keras.models import Model
# 網絡搭建-全網模型
encoding_dim = 32
input_img = Input(shape=(784, ))
encoded = Dense(encoding_dim, activation='relu')(input_img)
decoded = Dense(784, activation='sigmoid')(encoded)
autoencoder = Model(input_img, decoded)
網絡搭建-encoder和decoder單獨拆分
便於獲取單獨的encoder模型，類如轉換特徵
encoder = Model(input_img, encoded)
encoded_input = Input(shape=(encoding_dim, ))
decoded_layer = autoencoder.layers[-1]
decoder = Model(encoded_input, decoded_layer(encoded_input))
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')
數據準備
from keras.datasets import mnist
(x_train, _), (x_test, _) = mnist.load_data()
x_train = x_train.astype('float32')/255.
x_test = x_test.astype('float32')/255.
x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))
print(x_train.shape)
print(x_test.shape)
(60000, 784)
(10000, 784)
訓練
autoencoder.fit(x_train, x_train, verbose=2, epochs=20, batch_size=256, shuffle=True, validation_data=(x_test, x_test))
Train on 60000 samples, validate on 10000 samples
Epoch 1/20
- 15s - loss: 0.2631 - val_loss: 0.2517
Epoch 2/20
- 14s - loss: 0.2410 - val_loss: 0.2284
Epoch 3/20
- 13s - loss: 0.2209 - val_loss: 0.2113
Epoch 4/20
- 14s - loss: 0.2065 - val_loss: 0.1992
Epoch 5/20
- 14s - loss: 0.1960 - val_loss: 0.1900
Epoch 6/20
- 14s - loss: 0.1879 - val_loss: 0.1830
Epoch 7/20
- 15s - loss: 0.1814 - val_loss: 0.1771
Epoch 8/20
- 16s - loss: 0.1757 - val_loss: 0.1717
Epoch 9/20
- 16s - loss: 0.1706 - val_loss: 0.1669
Epoch 10/20
- 14s - loss: 0.1660 - val_loss: 0.1623
Epoch 11/20
- 14s - loss: 0.1617 - val_loss: 0.1583
Epoch 12/20
- 14s - loss: 0.1578 - val_loss: 0.1546
Epoch 13/20
- 16s - loss: 0.1543 - val_loss: 0.1514
Epoch 14/20
- 14s - loss: 0.1512 - val_loss: 0.1483
Epoch 15/20
- 14s - loss: 0.1483 - val_loss: 0.1455
Epoch 16/20
- 14s - loss: 0.1455 - val_loss: 0.1428
Epoch 17/20
- 14s - loss: 0.1429 - val_loss: 0.1402
Epoch 18/20
- 13s - loss: 0.1404 - val_loss: 0.1379
Epoch 19/20
- 14s - loss: 0.1381 - val_loss: 0.1354
Epoch 20/20
- 14s - loss: 0.1358 - val_loss: 0.1332
<keras.callbacks.History at 0x16422780>
模型應用結果
encoded_imgs = encoder.predict(x_test)
decoded_imgs = decoder.predict(encoded_imgs)
import matplotlib.pyplot as plt
def plt_plot(x_test, decoded):
n = 10 # how many digits we will display
plt.figure(figsize=(20, 4))
for i in range(n):
# display original
ax = plt.subplot(2, n, i + 1)
plt.imshow(x_test[i].reshape(28, 28))
plt.gray()
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
# display reconstruction
ax = plt.subplot(2, n, i + 1 + n)
plt.imshow(decoded[i].reshape(28, 28),)
plt.gray()
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
plt.suptitle('upper is original, under is reconstruction',fontsize=20)
plt.show()
plt_plot(x_test, decoded_imgs)
關於由拆分的decoder對新數據預測的結果是否與全網絡訓練得到的autoencoder預測結果是否相同？？
print(autoencoder.layers,'\n',
autoencoder.summary)
print('模型結果：\n {}'.format(autoencoder.predict(x_test[1:3,])
))
print('\n')
print('拆分的解編碼結果：\n {}'.format(decoder.predict(encoder.predict(x_test[1:3,]))))
模型結果：
[[ 4.77199501e-05 1.71007414e-04 1.32167725e-05 ..., 5.18762681e-05
1.27580452e-05 1.14900060e-04]
[ 1.05760782e-03 2.56783795e-04 7.73304608e-04 ..., 2.13044506e-04
4.67815640e-04 5.78384090e-04]]
拆分的解編碼結果：
[[ 4.77199501e-05 1.71007414e-04 1.32167725e-05 ..., 5.18762681e-05
1.27580452e-05 1.14900060e-04]
[ 1.05760782e-03 2.56783795e-04 7.73304608e-04 ..., 2.13044506e-04
4.67815640e-04 5.78384090e-04]]
兩者一樣的