AlexNet_代碼詳細

問題背景：昨天瞭解清除了AlexNet的原理部分，今天着手看看代碼。BTW，我是參考別人的文章，不算原創，不過文中會有很多我自己的見解與補充，算是做個筆記吧。
原文鏈接：
https://blog.csdn.net/taoyanqi8932/article/details/71081390 @yqtaowhu的

---

首先還是PO一下AlexNet的網絡結構：

---

@yqtaowhu使用的是CPU運行，怕耗時太久，因此只定義網絡結構,並進行前向後向的測試.

# 此函數爲輸出當前層的參數（層名稱、輸出尺寸）
def print_activations(t):   #輸入爲tensor
  print(t.op.name, ' ', t.get_shape().as_list())

#此函數爲搭建AlexNet的函數
def inference(images):
  """Build the AlexNet model.
  Args:
    images: Images Tensor
  Returns:
    pool5: the last Tensor in the convolutional component of AlexNet.     #返回conv_5中池化後的結果
    parameters: a list of Tensors corresponding to the weights and biases of the AlexNet model.  #返回權重和偏差列表
  """

  parameters = []  
  # conv1
  with tf.name_scope('conv1') as scope:# 將scope內生成的Variable自動命名爲conv1/xxx，區分不同卷積層之間的組件。
    kernel = tf.Variable(tf.truncated_normal([11, 11, 3, 96],dtype=tf.float32,stddev=1e-1), name='weights') #conv1的核是11*11，3是輸入channel數（後面的函數run_benchmark()中指定了image的channel是3）,96是輸出channel數.stddev標準差0.1
    conv = tf.nn.conv2d(images, kernel, [1, 4, 4, 1], padding='SAME')
    biases = tf.Variable(tf.constant(0.0, shape=[96],dtype=tf.float32),trainable=True, name='biases')  #初始化biases，一維向量，96個元素，全爲0
    bias = tf.nn.bias_add(conv, biases)  #把初始biases加到conv
    conv1 = tf.nn.relu(bias, name=scope)  #計算relu
    print_activations(conv1)  #輸出當前層參數
    parameters += [kernel, biases]    #更新權重和偏差

  # lrn1 沒有用LRN層
  # TODO(shlens, jiayq): Add a GPU version of local response normalization.
  # 考慮到LRN層效果不明顯，而且會讓forward和backwood的速度大大下降所以在此沒有添加

  # pool1 
  pool1 = tf.nn.max_pool(conv1,
                         ksize=[1, 3, 3, 1], 
                         strides=[1, 2, 2, 1], #各個維度上的移動步長
                         padding='VALID',
                         name='pool1')
  print_activations(pool1)

  # conv2 第二個卷積層，和conv1類似
  with tf.name_scope('conv2') as scope:
    kernel = tf.Variable(tf.truncated_normal([5, 5, 96, 256], dtype=tf.float32,stddev=1e-1), name='weights')
    conv = tf.nn.conv2d(pool1, kernel, [1, 1, 1, 1], padding='SAME')
    biases = tf.Variable(tf.constant(0.0, shape=[256], dtype=tf.float32),trainable=True, name='biases')
    bias = tf.nn.bias_add(conv, biases)
    conv2 = tf.nn.relu(bias, name=scope)
    parameters += [kernel, biases]
  print_activations(conv2)

  # pool2
  pool2 = tf.nn.max_pool(conv2,
                         ksize=[1, 3, 3, 1],
                         strides=[1, 2, 2, 1],
                         padding='VALID',
                         name='pool2')
  print_activations(pool2)

  # conv3
  with tf.name_scope('conv3') as scope:
    kernel = tf.Variable(tf.truncated_normal([3, 3, 256, 384],
                                             dtype=tf.float32,
                                             stddev=1e-1), name='weights')
    conv = tf.nn.conv2d(pool2, kernel, [1, 1, 1, 1], padding='SAME')
    biases = tf.Variable(tf.constant(0.0, shape=[384], dtype=tf.float32),trainable=True, name='biases')
    bias = tf.nn.bias_add(conv, biases)
    conv3 = tf.nn.relu(bias, name=scope)
    parameters += [kernel, biases]
    print_activations(conv3)

  # conv4
  with tf.name_scope('conv4') as scope:
    kernel = tf.Variable(tf.truncated_normal([3, 3, 384, 384],dtype=tf.float32,stddev=1e-1),name='weights')
    conv = tf.nn.conv2d(conv3, kernel, [1, 1, 1, 1], padding='SAME')
    biases = tf.Variable(tf.constant(0.0, shape=[384], dtype=tf.float32),trainable=True, name='biases')
    bias = tf.nn.bias_add(conv, biases)
    conv4 = tf.nn.relu(bias, name=scope)
    parameters += [kernel, biases]
    print_activations(conv4)

  # conv5
  with tf.name_scope('conv5') as scope:
    kernel = tf.Variable(tf.truncated_normal([3, 3, 384, 256],dtype=tf.float32,stddev=1e-1),name='weights')
    conv = tf.nn.conv2d(conv4, kernel, [1, 1, 1, 1], padding='SAME')
    biases = tf.Variable(tf.constant(0.0, shape=[256], dtype=tf.float32),trainable=True, name='biases')
    bias = tf.nn.bias_add(conv, biases)
    conv5 = tf.nn.relu(bias, name=scope)
    parameters += [kernel, biases]
    print_activations(conv5)

  # pool5
  pool5 = tf.nn.max_pool(conv5,
                         ksize=[1, 3, 3, 1],
                         strides=[1, 2, 2, 1],
                         padding='VALID',
                         name='pool5')
  print_activations(pool5)
  return pool5, parameters

關於代碼裏的一些函數：
1.tf.truncated_normal() :產生截斷正太分佈。參考鏈接：
https://blog.csdn.net/uestc_c2_403/article/details/72235565
2.tf.nn.conv2d():實現卷積。參考鏈接：
https://www.cnblogs.com/qggg/p/6832342.html
這個鏈接寫得很好很詳細！強烈推薦！
3.關於tensor中的batch.參考鏈接：
http://hp.stuhome.net/index.php/2016/09/20/tensorflow_batch_minibatch/
4.有一個關於部分Tensorflow Python API 的集合：
https://www.jianshu.com/p/e3a79eac554f
5.tf.Variable()：定義變量，參考鏈接：
https://blog.csdn.net/gg_18826075157/article/details/78368924

#此函數爲測試函數，image是隨機生成的數據,不是真實的image
def run_benchmark():
  """Run the benchmark on AlexNet."""
  with tf.Graph().as_default():
    # Generate some dummy images. 生成一些假image
    image_size = 224
    # Note that our padding definition is slightly different the cuda-convnet.
    # In order to force the model to start with the same activations sizes,
    # we add 3 to the image_size and employ VALID padding above.
    images = tf.Variable(tf.random_normal([FLAGS.batch_size,
                                           image_size,
                                           image_size, 3],
                                          dtype=tf.float32,
                                          stddev=1e-1))

    # Build a Graph that computes the logits predictions from the
    # inference model.
    pool5, parameters = inference(images)

    # Build an initialization operation.所有變量初始化
    init = tf.global_variables_initializer()

    # Start running operations on the Graph.
    config = tf.ConfigProto()
    config.gpu_options.allocator_type = 'BFC' #BFC算法，防止溢出
    sess = tf.Session(config=config)
    sess.run(init)

    # Run the forward benchmark.
    time_tensorflow_run(sess, pool5, "Forward") #前饋，這個函數是自定義的，在後面講解

    # Add a simple objective so we can calculate the backward pass.
    objective = tf.nn.l2_loss(pool5)
    # Compute the gradient with respect to all the parameters.根據梯度更新參數
    grad = tf.gradients(objective, parameters)
    # Run the backward benchmark.
    time_tensorflow_run(sess, grad, "Forward-backward")

關於代碼裏的一些函數：
1.tf.ConfigProto():配置硬件參數，參考鏈接
https://blog.csdn.net/u012436149/article/details/53837651
2.tf.nn.l2_loss() ：L2損失函數，參考鏈接：
http://www.cnblogs.com/lovephysics/p/7222488.html

#此函數爲測試程序跑起來用時多久
def time_tensorflow_run(session, target, info_string):
  """Run the computation to obtain the target tensor and print timing stats.
  Args:
    session: the TensorFlow session to run the computation under.
    target: the target Tensor that is passed to the session's run() function.
    info_string: a string summarizing this run, to be printed with the stats.
  Returns:
    None
  """
  num_steps_burn_in = 10   #頭幾輪迭代有顯存加載、cache命中等問題因此可以跳過，只考量10輪迭代之後的計算時間
  total_duration = 0.0   #總耗時
  total_duration_squared = 0.0 #總耗時的平方，用來後面計算方差
  for i in xrange(FLAGS.num_batches + num_steps_burn_in):#每一輪迭代
    start_time = time.time()  #開始時間是當前時間
    _ = session.run(target) #跑起來
    duration = time.time() - start_time  #耗時=當前（結束）時間-開始時間（單位是毫秒）
    if i >= num_steps_burn_in: #超過10步
      if not i % 10:  #是10的倍數
        print ('%s: step %d, duration = %.3f' %
               (datetime.now(), i - num_steps_burn_in, duration))
      total_duration += duration #更新總耗時
      total_duration_squared += duration * duration #更新總耗時的平方
  mn = total_duration / FLAGS.num_batches # 每輪迭代的平均耗時
  vr = total_duration_squared / FLAGS.num_batches - mn * mn #方差，衡量離散程度，即離均值遠不遠
  sd = math.sqrt(vr) #標準差
  print ('%s: %s across %d steps, %.3f +/- %.3f sec / batch' %
         (datetime.now(), info_string, FLAGS.num_batches, mn, sd))

、
在我的電腦上跑這個程序：

各層尺寸，沒有問題
這裏提個小問題：每一層尺寸中最前面的128是什麼？
答：batch_size，這個是在main中指定了的。


可以看到 後者比前者耗時久了好多。。。
另外，對比了一下我和其它博主的結果，發現我的電腦好差勁。。。耗時最久嗚嗚嗚