AlexNet_代码详细

问题背景：昨天了解清除了AlexNet的原理部分，今天着手看看代码。BTW，我是参考别人的文章，不算原创，不过文中会有很多我自己的见解与补充，算是做个笔记吧。
原文链接：
https://blog.csdn.net/taoyanqi8932/article/details/71081390 @yqtaowhu的

---

首先还是PO一下AlexNet的网络结构：

---

@yqtaowhu使用的是CPU运行，怕耗时太久，因此只定义网络结构,并进行前向后向的测试.

# 此函数为输出当前层的参数（层名称、输出尺寸）
def print_activations(t):   #输入为tensor
  print(t.op.name, ' ', t.get_shape().as_list())

#此函数为搭建AlexNet的函数
def inference(images):
  """Build the AlexNet model.
  Args:
    images: Images Tensor
  Returns:
    pool5: the last Tensor in the convolutional component of AlexNet.     #返回conv_5中池化后的结果
    parameters: a list of Tensors corresponding to the weights and biases of the AlexNet model.  #返回权重和偏差列表
  """

  parameters = []  
  # conv1
  with tf.name_scope('conv1') as scope:# 将scope内生成的Variable自动命名为conv1/xxx，区分不同卷积层之间的组件。
    kernel = tf.Variable(tf.truncated_normal([11, 11, 3, 96],dtype=tf.float32,stddev=1e-1), name='weights') #conv1的核是11*11，3是输入channel数（后面的函数run_benchmark()中指定了image的channel是3）,96是输出channel数.stddev标准差0.1
    conv = tf.nn.conv2d(images, kernel, [1, 4, 4, 1], padding='SAME')
    biases = tf.Variable(tf.constant(0.0, shape=[96],dtype=tf.float32),trainable=True, name='biases')  #初始化biases，一维向量，96个元素，全为0
    bias = tf.nn.bias_add(conv, biases)  #把初始biases加到conv
    conv1 = tf.nn.relu(bias, name=scope)  #计算relu
    print_activations(conv1)  #输出当前层参数
    parameters += [kernel, biases]    #更新权重和偏差

  # lrn1 没有用LRN层
  # TODO(shlens, jiayq): Add a GPU version of local response normalization.
  # 考虑到LRN层效果不明显，而且会让forward和backwood的速度大大下降所以在此没有添加

  # pool1 
  pool1 = tf.nn.max_pool(conv1,
                         ksize=[1, 3, 3, 1], 
                         strides=[1, 2, 2, 1], #各个维度上的移动步长
                         padding='VALID',
                         name='pool1')
  print_activations(pool1)

  # conv2 第二个卷积层，和conv1类似
  with tf.name_scope('conv2') as scope:
    kernel = tf.Variable(tf.truncated_normal([5, 5, 96, 256], dtype=tf.float32,stddev=1e-1), name='weights')
    conv = tf.nn.conv2d(pool1, kernel, [1, 1, 1, 1], padding='SAME')
    biases = tf.Variable(tf.constant(0.0, shape=[256], dtype=tf.float32),trainable=True, name='biases')
    bias = tf.nn.bias_add(conv, biases)
    conv2 = tf.nn.relu(bias, name=scope)
    parameters += [kernel, biases]
  print_activations(conv2)

  # pool2
  pool2 = tf.nn.max_pool(conv2,
                         ksize=[1, 3, 3, 1],
                         strides=[1, 2, 2, 1],
                         padding='VALID',
                         name='pool2')
  print_activations(pool2)

  # conv3
  with tf.name_scope('conv3') as scope:
    kernel = tf.Variable(tf.truncated_normal([3, 3, 256, 384],
                                             dtype=tf.float32,
                                             stddev=1e-1), name='weights')
    conv = tf.nn.conv2d(pool2, kernel, [1, 1, 1, 1], padding='SAME')
    biases = tf.Variable(tf.constant(0.0, shape=[384], dtype=tf.float32),trainable=True, name='biases')
    bias = tf.nn.bias_add(conv, biases)
    conv3 = tf.nn.relu(bias, name=scope)
    parameters += [kernel, biases]
    print_activations(conv3)

  # conv4
  with tf.name_scope('conv4') as scope:
    kernel = tf.Variable(tf.truncated_normal([3, 3, 384, 384],dtype=tf.float32,stddev=1e-1),name='weights')
    conv = tf.nn.conv2d(conv3, kernel, [1, 1, 1, 1], padding='SAME')
    biases = tf.Variable(tf.constant(0.0, shape=[384], dtype=tf.float32),trainable=True, name='biases')
    bias = tf.nn.bias_add(conv, biases)
    conv4 = tf.nn.relu(bias, name=scope)
    parameters += [kernel, biases]
    print_activations(conv4)

  # conv5
  with tf.name_scope('conv5') as scope:
    kernel = tf.Variable(tf.truncated_normal([3, 3, 384, 256],dtype=tf.float32,stddev=1e-1),name='weights')
    conv = tf.nn.conv2d(conv4, kernel, [1, 1, 1, 1], padding='SAME')
    biases = tf.Variable(tf.constant(0.0, shape=[256], dtype=tf.float32),trainable=True, name='biases')
    bias = tf.nn.bias_add(conv, biases)
    conv5 = tf.nn.relu(bias, name=scope)
    parameters += [kernel, biases]
    print_activations(conv5)

  # pool5
  pool5 = tf.nn.max_pool(conv5,
                         ksize=[1, 3, 3, 1],
                         strides=[1, 2, 2, 1],
                         padding='VALID',
                         name='pool5')
  print_activations(pool5)
  return pool5, parameters

关于代码里的一些函数：
1.tf.truncated_normal() :产生截断正太分布。参考链接：
https://blog.csdn.net/uestc_c2_403/article/details/72235565
2.tf.nn.conv2d():实现卷积。参考链接：
https://www.cnblogs.com/qggg/p/6832342.html
这个链接写得很好很详细！强烈推荐！
3.关于tensor中的batch.参考链接：
http://hp.stuhome.net/index.php/2016/09/20/tensorflow_batch_minibatch/
4.有一个关于部分Tensorflow Python API 的集合：
https://www.jianshu.com/p/e3a79eac554f
5.tf.Variable()：定义变量，参考链接：
https://blog.csdn.net/gg_18826075157/article/details/78368924

#此函数为测试函数，image是随机生成的数据,不是真实的image
def run_benchmark():
  """Run the benchmark on AlexNet."""
  with tf.Graph().as_default():
    # Generate some dummy images. 生成一些假image
    image_size = 224
    # Note that our padding definition is slightly different the cuda-convnet.
    # In order to force the model to start with the same activations sizes,
    # we add 3 to the image_size and employ VALID padding above.
    images = tf.Variable(tf.random_normal([FLAGS.batch_size,
                                           image_size,
                                           image_size, 3],
                                          dtype=tf.float32,
                                          stddev=1e-1))

    # Build a Graph that computes the logits predictions from the
    # inference model.
    pool5, parameters = inference(images)

    # Build an initialization operation.所有变量初始化
    init = tf.global_variables_initializer()

    # Start running operations on the Graph.
    config = tf.ConfigProto()
    config.gpu_options.allocator_type = 'BFC' #BFC算法，防止溢出
    sess = tf.Session(config=config)
    sess.run(init)

    # Run the forward benchmark.
    time_tensorflow_run(sess, pool5, "Forward") #前馈，这个函数是自定义的，在后面讲解

    # Add a simple objective so we can calculate the backward pass.
    objective = tf.nn.l2_loss(pool5)
    # Compute the gradient with respect to all the parameters.根据梯度更新参数
    grad = tf.gradients(objective, parameters)
    # Run the backward benchmark.
    time_tensorflow_run(sess, grad, "Forward-backward")

关于代码里的一些函数：
1.tf.ConfigProto():配置硬件参数，参考链接
https://blog.csdn.net/u012436149/article/details/53837651
2.tf.nn.l2_loss() ：L2损失函数，参考链接：
http://www.cnblogs.com/lovephysics/p/7222488.html

#此函数为测试程序跑起来用时多久
def time_tensorflow_run(session, target, info_string):
  """Run the computation to obtain the target tensor and print timing stats.
  Args:
    session: the TensorFlow session to run the computation under.
    target: the target Tensor that is passed to the session's run() function.
    info_string: a string summarizing this run, to be printed with the stats.
  Returns:
    None
  """
  num_steps_burn_in = 10   #头几轮迭代有显存加载、cache命中等问题因此可以跳过，只考量10轮迭代之后的计算时间
  total_duration = 0.0   #总耗时
  total_duration_squared = 0.0 #总耗时的平方，用来后面计算方差
  for i in xrange(FLAGS.num_batches + num_steps_burn_in):#每一轮迭代
    start_time = time.time()  #开始时间是当前时间
    _ = session.run(target) #跑起来
    duration = time.time() - start_time  #耗时=当前（结束）时间-开始时间（单位是毫秒）
    if i >= num_steps_burn_in: #超过10步
      if not i % 10:  #是10的倍数
        print ('%s: step %d, duration = %.3f' %
               (datetime.now(), i - num_steps_burn_in, duration))
      total_duration += duration #更新总耗时
      total_duration_squared += duration * duration #更新总耗时的平方
  mn = total_duration / FLAGS.num_batches # 每轮迭代的平均耗时
  vr = total_duration_squared / FLAGS.num_batches - mn * mn #方差，衡量离散程度，即离均值远不远
  sd = math.sqrt(vr) #标准差
  print ('%s: %s across %d steps, %.3f +/- %.3f sec / batch' %
         (datetime.now(), info_string, FLAGS.num_batches, mn, sd))

、
在我的电脑上跑这个程序：

各层尺寸，没有问题
这里提个小问题：每一层尺寸中最前面的128是什么？
答：batch_size，这个是在main中指定了的。


可以看到 后者比前者耗时久了好多。。。
另外，对比了一下我和其它博主的结果，发现我的电脑好差劲。。。耗时最久呜呜呜