上週發現3.4給漏了，家裏小孩突然發燒，醫院裏待了快一週，所以現在補上~~這幾天來回第三章看了好幾遍，終於對自編碼器有一點了解，這個知乎網頁說得很全：https://www.zhihu.com/question/41490383
import tensorflow as tf
import tensorlayer as tl
import numpy as np
learning_rate = 0.0001
lambda_l2_w = 0.01
n_epochs = 200
batch_size =128
print_interval = 200
hidden_size = 196
input_size = 784
image_width = 28
model = 'sigmoid'
# model = 'relu'
x = tf.placeholder(tf.float32, shape=[None, 784], name='x')
print('~~~~~~~~~~~~~~~~build network~~~~~~~~~~~~~~~~~')
if model == 'relu':
network = tl.layers.InputLayer(x, name='input')
network = tl.layers.DropoutLayer(network, keep=0.5, is_fix=True, name='denoising1') #這裏要加is_fix=True，否則默認是False會報錯
network = tl.layers.DenseLayer(network, hidden_size, tf.nn.relu, name='relu1')
encoded_img = network.outputs
recon_layer1 = tl.layers.DenseLayer(network, input_size, tf.nn.softplus, name='recon_layer1')
elif model == 'sigmoid':
network = tl.layers.InputLayer(x, name='input')
network = tl.layers.DropoutLayer(network, keep=0.5, is_fix=True, name='denoising1') #這裏要加is_fix=True，否則默認是False會報錯
network = tl.layers.DenseLayer(network, hidden_size, tf.nn.sigmoid, name='sigmoid1')
encoded_img = network.outputs
recon_layer1 = tl.layers.DenseLayer(network, input_size, tf.nn.sigmoid, name='recon_layer1')
y = recon_layer1.outputs
train_params = recon_layer1.all_params[-4:]
mse = tf.reduce_sum(tf.squared_difference(y, x), 1)
mse = tf.reduce_mean(mse)
L2_w = tf.contrib.layers.l2_regularizer(lambda_l2_w)(train_params[0] + tf.contrib.layers.l2_regularizer(lambda_l2_w)(train_params[2]))
activation_out = recon_layer1.all_layers[-2]
L1_a = 0.001 * tf.reduce_mean(activation_out)
beta = 5
rho = 0.15
p_hat = tf.reduce_mean(activation_out, 0)
KLD = beta * tf.reduce_sum(rho * tf.log(tf.divide(rho, p_hat)) + (1-rho) * tf.log((1-rho)/(tf.subtract(float(1), p_hat))))
if model == 'sigmoid':
cost = mse + L2_w + KLD
if model == 'relu':
cost = mse +L2_w +L1_a
train_op = tf.train.AdamOptimizer(learning_rate).minimize(cost)
saver = tf.train.Saver()
print('~~~~~~~~~~~模型訓練~~~~~~~~~~~~~~~~~~')
X_train, y_train, X_val, y_val, X_test, y_test = tl.files.load_mnist_dataset(shape=(-1, 784))
total_batch = X_train.shape[0] // batch_size
with tf.Session() as sess:
tl.layers.initialize_global_variables(sess)
for epoch in range(n_epochs):
avg_cost = 0.
for i in range(total_batch):
batch_x, batch_y = X_train[i*batch_size:(i+1)*batch_size], y_train[i*batch_size:(i+1)*batch_size]
batch_x = np.array(batch_x).astype(np.float32)
batch_cost,_ = sess.run([cost,train_op], feed_dict={x:batch_x})
avg_cost += batch_cost
if not i % print_interval:
print('Minibatch: %03d | Cost: %.3f' %(i+1, batch_cost))
print('Epoch: %03d | AvgCost: %.3f' %(epoch+1, avg_cost/(i+1)) )
saver.save(sess, save_path='./ae_tl_3.4/autoencoder-sigmoid.ckpt')
# saver.save(sess, save_path='./ae_tl_3.4/autoencoder-relu.ckpt')
print('~~~~~~~~~~~~~~~image_show~~~~~~~~~~~~~~~~')
import matplotlib.pyplot as plt
n_images = 15
fig, axes = plt.subplots(nrows=2, ncols=n_images, sharex=True, sharey=True, figsize=(20, 2.5))
test_images = X_test[:n_images]
with tf.Session() as sess:
saver.restore(sess, save_path='./ae_tl_3.4/autoencoder-sigmoid.ckpt')
# saver.restore(sess, save_path='./ae_tl_3.4/autoencoder-relu.ckpt')
decoded = sess.run(recon_layer1.outputs, feed_dict={x:test_images})
if model =='relu':
weights = sess.run(tl.layers.get_variables_with_name('relu/W:0', train_only=False, printable=True))
elif model == 'sigmoid':
weights = sess.run(tl.layers.get_variables_with_name('sigmoid/W:0', train_only=False, printable=True))
recon_weights = sess.run(tl.layers.get_variables_with_name('recon_layer1/W:0', train_only=False, printable=True))
recon_bias = sess.run(tl.layers.get_variables_with_name('recon_layer1/b:0', train_only=False, printable=True))
for i in range(n_images):
for ax, img in zip(axes, [test_images, decoded]):
ax[i].imshow(img[i].reshape((image_width, image_width)), cmap='binary')
plt.show()
輸出如下：
~~~~~~~~~~~~~~~~build network~~~~~~~~~~~~~~~~~
[TL] InputLayer input: (?, 784)
[TL] DropoutLayer denoising1: keep:0.500000 is_fix:True
[TL] DenseLayer sigmoid1: 196 sigmoid
[TL] DenseLayer recon_layer1: 784 sigmoid
~~~~~~~~~~~模型訓練~~~~~~~~~~~~~~~~~~
[TL] Load or Download MNIST > data\mnist
[TL] data\mnist\train-images-idx3-ubyte.gz
[TL] data\mnist\t10k-images-idx3-ubyte.gz
Minibatch: 001 | Cost: 27641.484
Minibatch: 201 | Cost: 13699.196
Epoch: 001 | AvgCost: 15177.753
Minibatch: 001 | Cost: 7864.798
Minibatch: 201 | Cost: 4753.522
Epoch: 002 | AvgCost: 5038.634
Minibatch: 001 | Cost: 3118.037
Minibatch: 201 | Cost: 2102.419
Epoch: 003 | AvgCost: 2185.543
Minibatch: 001 | Cost: 1498.473
Minibatch: 201 | Cost: 1089.769
Epoch: 004 | AvgCost: 1120.570
Minibatch: 001 | Cost: 826.790
Minibatch: 201 | Cost: 638.751
Epoch: 005 | AvgCost: 651.981
~~~~~~~
~~~~~~~
Minibatch: 001 | Cost: 47.045
Minibatch: 201 | Cost: 50.873
Epoch: 196 | AvgCost: 48.965
Minibatch: 001 | Cost: 47.466
Minibatch: 201 | Cost: 50.894
Epoch: 197 | AvgCost: 48.939
Minibatch: 001 | Cost: 46.919
Minibatch: 201 | Cost: 50.641
Epoch: 198 | AvgCost: 48.950
Minibatch: 001 | Cost: 47.575
Minibatch: 201 | Cost: 51.019
Epoch: 199 | AvgCost: 48.927
Minibatch: 001 | Cost: 46.787
Minibatch: 201 | Cost: 50.829
Epoch: 200 | AvgCost: 48.949
~~~~~~~~~~~~~~~image_show~~~~~~~~~~~~~~~~
INFO:tensorflow:Restoring parameters from ./ae_tl_3.4/autoencoder-relu.ckpt
[TL] Restoring parameters from ./ae_tl_3.4/autoencoder-relu.ckpt
[TL] [*] geting variables with sigmoid/W:0
[TL] [*] geting variables with recon_layer1/W:0
[TL] got 0: recon_layer1/W:0 (196, 784)
[TL] [*] geting variables with recon_layer1/b:0
[TL] got 0: recon_layer1/b:0 (784,)
QWindowsWindow::setGeometry: Unable to set geometry 2000x317+4+23 on QWidgetWindow/'MainWindowClassWindow'. Resulting geometry: 1284x317+4+23 (frame: 4, 23, 4, 4, custom margin: 0, 0, 0, 0, minimum size: 69x67, maximum size: 16777215x16777215).
[Finished in 2003.1s]
還有個relu版的，可以自己試試~~