一、保存和恢復模型
保存和恢復模型：
http://cv-tricks.com/tensorflow-tutorial/save-restore-tensorflow-models-quick-complete-tutorial/
二、區分好variable和tensor
靈活利用dir()函數來解析對象的屬性和方法
tf.layers.dense和tf.layer.cov2d返回的都是tensorflow.python.framework.ops.Tensor，也就是tensor，而type(tf.Variable([1]))返回的是tensorflow.python.ops.variables.Variable
在variable的創建過程中，
# Create two variables.
weights = tf.Variable(tf.random_normal([784, 200], stddev=0.35), name="weights")
biases = tf.Variable(tf.zeros([200]), name="biases")
其實有多個ops（對應TF中的nodes）被創建
Calling tf.Variable() adds several ops to the graph:
- A variable op that holds the variable value.
- An initializer op that sets the variable to its initial value. This is actually a tf.assign op.
- The ops for the initial value, such as the zeros op for the biases variable in the example are also added to the graph.The value returned by tf.Variable() value is an instance of the Python class tf.Variable.
最後返回的是一個tf.Variable類的實例化對象
根據文檔所說：
Just like any Tensor, variables created with Variable() can be used as inputs for other Ops in the graph. Additionally, all the operators overloaded for the Tensor class are carried over to variables, so you can also add nodes to the graph by just doing arithmetic on variables.
也就是說和tensor一樣，variable（一種區別於tensor的對象）也可以作爲其他ops的輸入，並且所有能夠重載tensor的操作都適用於variable，所以可以直接像tensor一樣拿variable來做計算，如下：
import tensorflow as tf
# Create a variable.
w = tf.Variable(<initial-value>, name=<optional-name>)
# Use the variable in the graph like any Tensor.
y = tf.matmul(w, ...another variable or tensor...)
# The overloaded operators are available too.
z = tf.sigmoid(w + y)
# Assign a new value to the variable with `assign()` or a related method.
w.assign(w + 1.0)
w.assign_add(1.0)
三、Debug TensorFlow
A Practical Guide for Debugging TensorFlow Codes
https://github.com/wookayin/tensorflow-talk-debugging
要意識到細心地解剖神經網絡是一門藝術，因此要耐心地考慮，首先要可以從簡單的查看acc和loss入手：
I recommend to take simple and essential scalar summaries only (e.g. train/validation loss, overall accuracy, etc.), and to include debugging stuffs only on demand
其他的一些建議：
養成給tensor/variable命名的好習慣
Use proper names (prefixed or scoped) for tensors and variables (specifying name=… to tensor/variable declaration)
會使用tf.Print()函數，每當指定的op被evaluated的時候就會打印出相應的信息，它本身是一個 identity op with the side effect of printing data, when this op is evaluated.
Debug Trick:
獲取graph中的ops和對應的tensors，參考https://stackoverflow.com/questions/35336648/list-of-tensor-names-in-graph-in-tensorflow
sess.graph.get_operations() gives you a list of operations. For an op, op.name gives you the name and op.values() gives you a list of tensors it produces (in the inception-v3 model, all tensor names are the op name with a “:0” appended to it, so pool_3:0 is the tensor produced by the final pooling op.)
爲了獲得模型的參數，可以直接訪問名字含有kernel或者bias字符串的tensor，參考https://stackoverflow.com/questions/43244446/how-to-get-cnn-kernel-values-in-tensorflow
獲取variables
可以通過tf.global_variables()來獲取所有的變量
可以通過tf.trainable_variables()來獲取所有可以訓練的變量，爲了區分開不需要訓練的變量，可以在定義變量的時候設置好trainable的參數，如下：
global_step = tf.Variable(0, dtype=tf.int32, trainable=False)
train_op = tf.train.AdamOptimizer(learning_rate=0.001)\
.minimize(loss, global_step=global_step)更多variables相關的helper functions請見：
https://www.tensorflow.org/versions/r0.12/api_docs/python/state_ops/variable_helper_functions
以上獲取ops/tensors和variables的方法再結合下圖就很完整了：
當weight未初始化的時候，默認的初始化方式：
https://stackoverflow.com/questions/43284047/what-is-the-default-kernel-initializer-in-tf-layers-conv2d-and-tf-layers-dense