接上文，處理好輸入圖片後即可輸入ENet的網絡模型進行訓練。
#Create the model inference
with slim.arg_scope(ENet_arg_scope(weight_decay=weight_decay)):
logits, probabilities = ENet(images,
num_classes,
batch_size=batch_size,
is_training=True,
reuse=None,
num_initial_blocks=num_initial_blocks,
stage_two_repeat=stage_two_repeat,
skip_connections=skip_connections)
其中，slim.arg_scope是對函數進行修飾，修改已經定義函數中的某個參數值。在這裏，修改了ENet_arg_scope函數中的weight_decay值爲我們定義的值。而ENet_arg_scope函數又長下面這個模樣：
def ENet_arg_scope(weight_decay=2e-4,
batch_norm_decay=0.1,
batch_norm_epsilon=0.001):
'''
The arg scope for enet model. The weight decay is 2e-4 as seen in the paper.
Batch_norm decay is 0.1 (momentum 0.1) according to official implementation.
INPUTS:
- weight_decay(float): the weight decay for weights variables in conv2d and separable conv2d
- batch_norm_decay(float): decay for the moving average of batch_norm momentums.
- batch_norm_epsilon(float): small float added to variance to avoid dividing by zero.
OUTPUTS:
- scope(arg_scope): a tf-slim arg_scope with the parameters needed for xception.
'''
# Set weight_decay for weights in conv2d and separable_conv2d layers.
with slim.arg_scope([slim.conv2d], # 使用slim.arg_scope對 slim.conv2d函數進行修飾，設置默認參數
weights_regularizer=slim.l2_regularizer(weight_decay), # 修改了conv2d的weights_regularizer l2正則化
biases_regularizer=slim.l2_regularizer(weight_decay)): # 修改了conv2d的biases_regularizer l2正則化
# Set parameters for batch_norm.
with slim.arg_scope([slim.batch_norm], # 同上 設置batchnorm的參數
decay=batch_norm_decay,
epsilon=batch_norm_epsilon) as scope:
return scope
這個函數通過再次嵌套slim.arg_scope函數來修改slim.conv2d函數的參數，修改conv2d權重及偏置正則化事l2正則化的權重衰減值。同時修改了slim.batch_norm函數中decay及epsilon的值。梳理一下就是，首先使用slim.arg_scope修改ENet_arg_scope的傳入參數，然ENet_arg_scope內部再調用slim.arg_scope來修改slim.conv2d和slim.batch_norm的參數。
隨後，調整好了ENet的參數，我們就可以把圖像輸入ENet進行訓練。
logits, probabilities = ENet(images, # 輸入訓練圖像
num_classes,
batch_size=batch_size,
is_training=True, # 訓練過程是否使用PReLu 和 batch normalization
reuse=None,
num_initial_blocks=num_initial_blocks,
stage_two_repeat=stage_two_repeat,
skip_connections=skip_connections)
ENet的傳入參數爲（原始圖像，類別數，batch_size，是否使用PReLu和bn，reuse這個還不太理解，ENet中初始化模塊的個數，ENet中第二部分模塊的個數，是否使用跳躍連接）。
現在我們看一下ENet的網絡結構，首先有個整體的概念（截圖來自ENet原論文）：
除了initial模塊和fullconv模塊，共計五大模塊。且模塊三中除了缺少一個下采樣層外，同模塊二完全相同。在ENet中，initial模塊的定義如下：
圖像輸入後，一邊通過13個3*3的卷積核，以步長2進行卷積；另一邊使用2*2的核以步長2進行池化，因爲輸入爲3通道的圖片，池化後channel爲3，將兩個輸出結合，得到channel爲16的輸出。
而bottleneck模塊是由ResNet得到的啓發，共定義了三種，分別是“普通卷積”，“空洞卷積”，“非對稱卷積”三種。而bottleneck模塊的總體定義爲：
結合代碼，對上述模塊分別進行理解。
initial模塊代碼如下：
def initial_block(inputs, is_training=True, scope='initial_block'):
'''
The initial block for Enet has 2 branches: The convolution branch and Maxpool branch.
The conv branch has 13 layers, while the maxpool branch gives 3 layers corresponding to the RGB channels.
Both output layers are then concatenated to give an output of 16 layers.
NOTE: Does not need to store pooling indices since it won't be used later for the final upsampling.
INPUTS:
- inputs(Tensor): A 4D tensor of shape [batch_size, height, width, channels]
OUTPUTS:
- net_concatenated(Tensor): a 4D Tensor that contains the
'''
#Convolutional branch
net_conv = slim.conv2d(inputs, 13, [3,3], stride=2, activation_fn=None, scope=scope+'_conv') # 3x3卷積，13個卷積核，步長2
net_conv = slim.batch_norm(net_conv, is_training=is_training, fused=True, scope=scope+'_batchnorm')
net_conv = prelu(net_conv, scope=scope+'_prelu')
#Max pool branch
net_pool = slim.max_pool2d(inputs, [2,2], stride=2, scope=scope+'_max_pool')
#Concatenated output - does it matter max pool comes first or conv comes first? probably not.
net_concatenated = tf.concat([net_conv, net_pool], axis=3, name=scope+'_concat')
return net_concatenated
卷積分支中，首先將輸入使用13個3*3的卷積核，按步長爲2進行卷積，然後輸出結果輸入bn層做歸一化處理，然後使用PReLU作爲激活函數。
而PReLU的函數代碼爲：
def prelu(x, scope, decoder=False):
'''
Performs the parametric relu operation. This implementation is based on:
https://stackoverflow.com/questions/39975676/how-to-implement-prelu-activation-in-tensorflow
For the decoder portion, prelu becomes just a normal prelu
INPUTS:
- x(Tensor): a 4D Tensor that undergoes prelu
- scope(str): the string to name your prelu operation's alpha variable.
- decoder(bool): if True, prelu becomes a normal relu.
OUTPUTS:
- pos + neg / x (Tensor): gives prelu output only during training; otherwise, just return x.
'''
#If decoder, then perform relu and just return the output
if decoder:
return tf.nn.relu(x, name=scope)
#tf.get_variable(name,shape=None,dtype=None,initializer=None,regularizer=None.
#trainable=True,collections=None,caching_device=None,partitioner=None,validate_shape=Ture
#use_resource=None,custom_getter=None,constraint=None)
# name:新變量或現有變量的名稱，shape:...的形狀，initializer:用來初始化變量
# dtype:....的類型
alpha= tf.get_variable(scope + 'alpha', x.get_shape()[-1], # 獲取一個已經存在的變量或者創建一個新的變量
initializer=tf.constant_initializer(0.0),# ----存疑
dtype=tf.float32)
pos = tf.nn.relu(x)
neg = alpha * (x - abs(x)) * 0.5
return pos + neg
完完全全對PReLU的復現，大於0時就是relu函數，小於0時，乘以一個很小的alpha係數再乘以0.5。再記錄一個疑問，就是tf.constant_initializer(0.0)豈不是將alpha設爲了0？這不還是ReLU嗎？？現在還是不明白。
然後，就是最大池化分支，用了2*2的核，步長爲2進行池化，最終使用concat將兩個分支的輸出進行合併，並輸出最終結果。
接下來介紹五種種不同的bottleneck模塊(原論文中是三種，普通、空洞卷積和非對稱卷積三種）。在作者代碼中統一定義了bottleneck函數，函數體中使用if條件分支來判斷是何種bottleneck模塊。
def bottleneck(inputs,
output_depth,
filter_size,
regularizer_prob,
projection_ratio=4,
seed=0,
is_training=True,
downsampling=False,
upsampling=False,
pooling_indices=None,
output_shape=None,
dilated=False,
dilation_rate=None,
asymmetric=False,
decoder=False,
scope='bottleneck'):
'''
'''
#Calculate the depth reduction based on the projection ratio used in 1x1 convolution. #？？投影降維？？ 不理解
reduced_depth = int(inputs.get_shape().as_list()[3] / projection_ratio)
with slim.arg_scope([prelu], decoder=decoder):
#=============DOWNSAMPLING BOTTLENECK====================
if downsampling:
...
#============DILATION CONVOLUTION BOTTLENECK==================== 空洞卷積模塊
#Everything is the same as a regular bottleneck except for the dilation rate argument
elif dilated:
...
#===========ASYMMETRIC CONVOLUTION BOTTLENECK============== 非對稱卷積
#Everything is the same as a regular bottleneck except for a [5,5] kernel decomposed into two [5,1] then [1,5]
elif asymmetric:
...
#============UPSAMPLING BOTTLENECK================
#Everything is the same as a regular one, except convolution becomes transposed.
elif upsampling:
...
#OTHERWISE, just perform a regular bottleneck!
#==============REGULAR BOTTLENECK==================
#Save the main branch for addition later
net_main = inputs
....
先看帶下采樣的bottleneck模塊，代碼已經很清晰了：
#=============DOWNSAMPLING BOTTLENECK====================
if downsampling:
#=============MAIN BRANCH=============
#Just perform a max pooling
net_main, pooling_indices = tf.nn.max_pool_with_argmax(inputs,
ksize=[1,2,2,1], # 池化的結果就是 batch size和 num channels不變，輸入圖片寬高減半
strides=[1,2,2,1],
padding='SAME',
name=scope+'_main_max_pool')
#First get the difference in depth to pad, then pad with zeros only on the last dimension.
inputs_shape = inputs.get_shape().as_list()
depth_to_pad = abs(inputs_shape[3] - output_depth)
paddings = tf.convert_to_tensor([[0,0], [0,0], [0,0], [0, depth_to_pad]]) # shape(4,2)
net_main = tf.pad(net_main, paddings=paddings, name=scope+'_main_padding')
# tf.pad(tensor, paddings, mode="CONSTANT", name=None, constant_values=0)
# 其中 tensor爲輸入 paddings指出要給tensor的哪個維度進行填充,以及填充多少,要注意的是paddings的rank必須和tensor的rank相同
# mode指填充方式,"CONSTANT"表示用常數進行填充(默認爲0，在需要的情況下可以用constant_value賦值)
# name 節點名稱
#=============SUB BRANCH==============
#First projection that has a 2x2 kernel and stride 2
net = slim.conv2d(inputs, reduced_depth, [2,2], stride=2, scope=scope+'_conv1')
net = slim.batch_norm(net, is_training=is_training, scope=scope+'_batch_norm1')
net = prelu(net, scope=scope+'_prelu1')
#Second conv block
net = slim.conv2d(net, reduced_depth, [filter_size, filter_size], scope=scope+'_conv2')
net = slim.batch_norm(net, is_training=is_training, scope=scope+'_batch_norm2')
net = prelu(net, scope=scope+'_prelu2')
#Final projection with 1x1 kernel
net = slim.conv2d(net, output_depth, [1,1], scope=scope+'_conv3')
net = slim.batch_norm(net, is_training=is_training, scope=scope+'_batch_norm3')
net = prelu(net, scope=scope+'_prelu3')
#Regularizer # 正則化 # 使用spatial_dropout 隨機將某些區域置0
net = spatial_dropout(net, p=regularizer_prob, seed=seed, scope=scope+'_spatial_dropout')
#Finally, combine the two branches together via an element-wise addition
net = tf.add(net, net_main, name=scope+'_add')
net = prelu(net, scope=scope+'_last_prelu')
#also return inputs shape for convenience later
return net, pooling_indices, inputs_shape
主分支首先進行2×2的最大池化，池化後因爲前面有個投影降維，0padding這一步再將維度還原，然後輸出到net_main。
次分支中，降維後的輸入進行2×2，步長爲2的卷積，卷積核個數等於計算出的reduced_depth用於替代原本的1×1的卷積；然後進行bn，pReLU；緊接着進行指定卷積核大小的卷積，隨後又是bn，pReLU，然後1×1的卷積，bn，pReLU，最後使用spatial_dropout正則化。
主從分支求和，使用pReLU進行處理，帶有downsampling功能的bottleneck模塊定義完畢。
接下來是空洞卷積（dilated）bottleneck模塊，代碼如下：
elif dilated:
#Check if dilation rate is given
if not dilation_rate:
raise ValueError('Dilation rate is not given.')
#Save the main branch for addition later
net_main = inputs
#First projection with 1x1 kernel (dimensionality reduction)
net = slim.conv2d(inputs, reduced_depth, [1,1], scope=scope+'_conv1')
net = slim.batch_norm(net, is_training=is_training, scope=scope+'_batch_norm1')
net = prelu(net, scope=scope+'_prelu1')
#Second conv block --- apply dilated convolution here
net = slim.conv2d(net, reduced_depth, [filter_size, filter_size], rate=dilation_rate, scope=scope+'_dilated_conv2') #----空洞卷積
net = slim.batch_norm(net, is_training=is_training, scope=scope+'_batch_norm2')
net = prelu(net, scope=scope+'_prelu2')
#Final projection with 1x1 kernel (Expansion)
net = slim.conv2d(net, output_depth, [1,1], scope=scope+'_conv3')
net = slim.batch_norm(net, is_training=is_training, scope=scope+'_batch_norm3')
net = prelu(net, scope=scope+'_prelu3')
#Regularizer
net = spatial_dropout(net, p=regularizer_prob, seed=seed, scope=scope+'_spatial_dropout')
net = prelu(net, scope=scope+'_prelu4')
#Add the main branch
net = tf.add(net_main, net, name=scope+'_add_dilated')
net = prelu(net, scope=scope+'_last_prelu')
return net
主分支就是原始輸入，從分支在第二次卷積時使用了空洞卷積，空洞卷積的“間隔率”需要後面調用函數時指定，其他均相同。
隨後就是非對稱卷積模塊：
elif asymmetric:
#Save the main branch for addition later
net_main = inputs
#First projection with 1x1 kernel (dimensionality reduction)
net = slim.conv2d(inputs, reduced_depth, [1,1], scope=scope+'_conv1')
net = slim.batch_norm(net, is_training=is_training, scope=scope+'_batch_norm1')
net = prelu(net, scope=scope+'_prelu1')
#Second conv block --- apply asymmetric conv here
net = slim.conv2d(net, reduced_depth, [filter_size, 1], scope=scope+'_asymmetric_conv2a')
net = slim.conv2d(net, reduced_depth, [1, filter_size], scope=scope+'_asymmetric_conv2b')
net = slim.batch_norm(net, is_training=is_training, scope=scope+'_batch_norm2')
net = prelu(net, scope=scope+'_prelu2')
#Final projection with 1x1 kernel
net = slim.conv2d(net, output_depth, [1,1], scope=scope+'_conv3')
net = slim.batch_norm(net, is_training=is_training, scope=scope+'_batch_norm3')
net = prelu(net, scope=scope+'_prelu3')
#Regularizer
net = spatial_dropout(net, p=regularizer_prob, seed=seed, scope=scope+'_spatial_dropout')
net = prelu(net, scope=scope+'_prelu4')
#Add the main branch
net = tf.add(net_main, net, name=scope+'_add_asymmetric')
net = prelu(net, scope=scope+'_last_prelu')
return net
主分支還是原始輸入，從分支中在1×1卷積後，用了f×1,1×f的兩個卷積核分別進行卷積。對於非對稱卷積如何計算還是不太理解。隨後沒有什麼變化。
接下來就是帶上採樣的bottleneck模塊，代碼如下：
#============UPSAMPLING BOTTLENECK================
#Everything is the same as a regular one, except convolution becomes transposed.
elif upsampling:
#Check if pooling indices is given
if pooling_indices == None:
raise ValueError('Pooling indices are not given.')
#Check output_shape given or not
if output_shape == None:
raise ValueError('Output depth is not given')
#=======MAIN BRANCH=======
#Main branch to upsample. output shape must match with the shape of the layer that was pooled initially, in order
#for the pooling indices to work correctly. However, the initial pooled layer was padded, so need to reduce dimension
#before unpooling. In the paper, padding is replaced with convolution for this purpose of reducing the depth!
net_unpool = slim.conv2d(inputs, output_depth, [1,1], scope=scope+'_main_conv1')
net_unpool = slim.batch_norm(net_unpool, is_training=is_training, scope=scope+'batch_norm1')
net_unpool = unpool(net_unpool, pooling_indices, output_shape=output_shape, scope='unpool')
#======SUB BRANCH=======
#First 1x1 projection to reduce depth
net = slim.conv2d(inputs, reduced_depth, [1,1], scope=scope+'_conv1')
net = slim.batch_norm(net, is_training=is_training, scope=scope+'_batch_norm2')
net = prelu(net, scope=scope+'_prelu1')
#Second conv block -----------------------------> NOTE: using tf.nn.conv2d_transpose for variable input shape.
net_unpool_shape = net_unpool.get_shape().as_list()
output_shape = [net_unpool_shape[0], net_unpool_shape[1], net_unpool_shape[2], reduced_depth]
output_shape = tf.convert_to_tensor(output_shape)
filter_size = [filter_size, filter_size, reduced_depth, reduced_depth]
filters = tf.get_variable(shape=filter_size, initializer=initializers.xavier_initializer(), dtype=tf.float32, name=scope+'_transposed_conv2_filters')
# net = slim.conv2d_transpose(net, reduced_depth, [filter_size, filter_size], stride=2, scope=scope+'_transposed_conv2')
net = tf.nn.conv2d_transpose(net, filter=filters, strides=[1,2,2,1], output_shape=output_shape, name=scope+'_transposed_conv2')
net = slim.batch_norm(net, is_training=is_training, scope=scope+'_batch_norm3')
net = prelu(net, scope=scope+'_prelu2')
#Final projection with 1x1 kernel
net = slim.conv2d(net, output_depth, [1,1], scope=scope+'_conv3')
net = slim.batch_norm(net, is_training=is_training, scope=scope+'_batch_norm4')
net = prelu(net, scope=scope+'_prelu3')
#Regularizer
net = spatial_dropout(net, p=regularizer_prob, seed=seed, scope=scope+'_spatial_dropout')
net = prelu(net, scope=scope+'_prelu4')
#Finally, add the unpooling layer and the sub branch together
net = tf.add(net, net_unpool, name=scope+'_add_upsample')
net = prelu(net, scope=scope+'_last_prelu')
return net
其中，用到了上池化操作，作者自己編寫了函數，完全看不懂是如何操作的。
def unpool(updates, mask, k_size=[1, 2, 2, 1], output_shape=None, scope=''):
'''
Unpooling function based on the implementation by Panaetius at https://github.com/tensorflow/tensorflow/issues/2169
INPUTS:
- inputs(Tensor): a 4D tensor of shape [batch_size, height, width, num_channels] that represents the input block to be upsampled
- mask(Tensor): a 4D tensor that represents the argmax values/pooling indices of the previously max-pooled layer
- k_size(list): a list of values representing the dimensions of the unpooling filter.
- output_shape(list): a list of values to indicate what the final output shape should be after unpooling
- scope(str): the string name to name your scope
OUTPUTS:
- ret(Tensor): the returned 4D tensor that has the shape of output_shape.
'''
with tf.variable_scope(scope):
mask = tf.cast(mask, tf.int32) # 數據格式轉換，將mask轉換爲tf.int32 mask就是max pooling時的最大值索引
input_shape = tf.shape(updates, out_type=tf.int32)
# calculation new shape
if output_shape is None:
output_shape = (input_shape[0], input_shape[1] * ksize[1], input_shape[2] * ksize[2], input_shape[3])
# calculation indices for batch, height, width and feature maps
one_like_mask = tf.ones_like(mask, dtype=tf.int32) # 將mask中所元素都變成1
batch_shape = tf.concat([[input_shape[0]], [1], [1], [1]], 0) # [[input_shape[0],[1],[1],[1]]
batch_range = tf.reshape(tf.range(output_shape[0], dtype=tf.int32), shape=batch_shape)
b = one_like_mask * batch_range
y = mask // (output_shape[2] * output_shape[3]) #mask 向下取整
x = (mask // output_shape[3]) % output_shape[2] #mask % (output_shape[2] * output_shape[3]) // output_shape[3]
feature_range = tf.range(output_shape[3], dtype=tf.int32)
f = one_like_mask * feature_range
# transpose indices & reshape update values to one dimension
updates_size = tf.size(updates)
indices = tf.transpose(tf.reshape(tf.stack([b, y, x, f]), [4, updates_size]))
values = tf.reshape(updates, [updates_size])
ret = tf.scatter_nd(indices, values, output_shape)
return ret
最後，定義了最平常的 bottleneck模塊：
#OTHERWISE, just perform a regular bottleneck!
#==============REGULAR BOTTLENECK==================
#Save the main branch for addition later
net_main = inputs
#First projection with 1x1 kernel
net = slim.conv2d(inputs, reduced_depth, [1,1], scope=scope+'_conv1')
net = slim.batch_norm(net, is_training=is_training, scope=scope+'_batch_norm1')
net = prelu(net, scope=scope+'_prelu1')
#Second conv block
net = slim.conv2d(net, reduced_depth, [filter_size, filter_size], scope=scope+'_conv2')
net = slim.batch_norm(net, is_training=is_training, scope=scope+'_batch_norm2')
net = prelu(net, scope=scope+'_prelu2')
#Final projection with 1x1 kernel
net = slim.conv2d(net, output_depth, [1,1], scope=scope+'_conv3')
net = slim.batch_norm(net, is_training=is_training, scope=scope+'_batch_norm3')
net = prelu(net, scope=scope+'_prelu3')
#Regularizer
net = spatial_dropout(net, p=regularizer_prob, seed=seed, scope=scope+'_spatial_dropout')
net = prelu(net, scope=scope+'_prelu4')
#Add the main branch
net = tf.add(net_main, net, name=scope+'_add_regular')
net = prelu(net, scope=scope+'_last_prelu')
return net
對比ENet論文結構，作者將網絡結構完全復現。
ENET的代碼如下：
inputs_shape = inputs.get_shape().as_list()
inputs.set_shape(shape=(batch_size, inputs_shape[1], inputs_shape[2], inputs_shape[3]))
with tf.variable_scope(scope, reuse=reuse): # 設置變量作用域？ 暫未理解
#Set the primary arg scopes. Fused batch_norm is faster than normal batch norm.
with slim.arg_scope([initial_block, bottleneck], is_training=is_training),\
slim.arg_scope([slim.batch_norm], fused=True), \
slim.arg_scope([slim.conv2d, slim.conv2d_transpose], activation_fn=None):
#=================INITIAL BLOCK=================
net = initial_block(inputs, scope='initial_block_1')
for i in range(2, max(num_initial_blocks, 1) + 1): # range(2,2) 空列表 for循環不執行
net = initial_block(net, scope='initial_block_' + str(i))
#Save for skip connection later
if skip_connections:
net_one = net
#===================STAGE ONE=======================
net, pooling_indices_1, inputs_shape_1 = bottleneck(net, output_depth=64, filter_size=3, regularizer_prob=0.01, downsampling=True, scope='bottleneck1_0')
net = bottleneck(net, output_depth=64, filter_size=3, regularizer_prob=0.01, scope='bottleneck1_1')
net = bottleneck(net, output_depth=64, filter_size=3, regularizer_prob=0.01, scope='bottleneck1_2')
net = bottleneck(net, output_depth=64, filter_size=3, regularizer_prob=0.01, scope='bottleneck1_3')
net = bottleneck(net, output_depth=64, filter_size=3, regularizer_prob=0.01, scope='bottleneck1_4')
#Save for skip connection later
if skip_connections:
net_two = net
#regularization prob is 0.1 from bottleneck 2.0 onwards
with slim.arg_scope([bottleneck], regularizer_prob=0.1):
net, pooling_indices_2, inputs_shape_2 = bottleneck(net, output_depth=128, filter_size=3, downsampling=True, scope='bottleneck2_0')
#Repeat the stage two at least twice to get stage 2 and 3:
for i in range(2, max(stage_two_repeat, 2) + 2):
net = bottleneck(net, output_depth=128, filter_size=3, scope='bottleneck'+str(i)+'_1')
net = bottleneck(net, output_depth=128, filter_size=3, dilated=True, dilation_rate=2, scope='bottleneck'+str(i)+'_2')
net = bottleneck(net, output_depth=128, filter_size=5, asymmetric=True, scope='bottleneck'+str(i)+'_3')
net = bottleneck(net, output_depth=128, filter_size=3, dilated=True, dilation_rate=4, scope='bottleneck'+str(i)+'_4')
net = bottleneck(net, output_depth=128, filter_size=3, scope='bottleneck'+str(i)+'_5')
net = bottleneck(net, output_depth=128, filter_size=3, dilated=True, dilation_rate=8, scope='bottleneck'+str(i)+'_6')
net = bottleneck(net, output_depth=128, filter_size=5, asymmetric=True, scope='bottleneck'+str(i)+'_7')
net = bottleneck(net, output_depth=128, filter_size=3, dilated=True, dilation_rate=16, scope='bottleneck'+str(i)+'_8') # 以2\4\6\8空洞卷積,（是否更改參數提高效果？）
with slim.arg_scope([bottleneck], regularizer_prob=0.1, decoder=True):
#===================STAGE FOUR========================
bottleneck_scope_name = "bottleneck" + str(i + 1)
#The decoder section, so start to upsample.
net = bottleneck(net, output_depth=64, filter_size=3, upsampling=True,
pooling_indices=pooling_indices_2, output_shape=inputs_shape_2, scope=bottleneck_scope_name+'_0')
#Perform skip connections here
if skip_connections:
net = tf.add(net, net_two, name=bottleneck_scope_name+'_skip_connection')
net = bottleneck(net, output_depth=64, filter_size=3, scope=bottleneck_scope_name+'_1')
net = bottleneck(net, output_depth=64, filter_size=3, scope=bottleneck_scope_name+'_2')
#===================STAGE FIVE========================
bottleneck_scope_name = "bottleneck" + str(i + 2)
net = bottleneck(net, output_depth=16, filter_size=3, upsampling=True,
pooling_indices=pooling_indices_1, output_shape=inputs_shape_1, scope=bottleneck_scope_name+'_0')
#perform skip connections here
if skip_connections:
net = tf.add(net, net_one, name=bottleneck_scope_name+'_skip_connection')
net = bottleneck(net, output_depth=16, filter_size=3, scope=bottleneck_scope_name+'_1')
#=============FINAL CONVOLUTION=============
logits = slim.conv2d_transpose(net, num_classes, [2,2], stride=2, scope='fullconv') # 反捲積
probabilities = tf.nn.softmax(logits, name='logits_to_softmax')
return logits, probabilities