GAN網絡中的Normalization彙總

引用¹

$\hat{x_i} = \frac{1}{\sigma_i}(x_i-\mu_i) \\ {y_i} = \gamma\hat{x_i} + \beta \\ \mu_i = \frac{1}{m}\Sigma_{k\in S_i}x_k \\ \sigma_i = \frac{1}{m}\sqrt{\Sigma_{x_k\in S_i}(x_k-\mu_i)^2+\epsilon}\\$

Batch Normalization

對於BN來說，其中$S_i$是沿着$N, H, W$三個方向的。

def Batchnorm_simple_for_train(x, gamma, beta, bn_param):
"""
param:x    : 輸入數據，設shape(B,H,W,C)
param:gama : 縮放因子  γ
param:beta : 平移因子  β
param:bn_param   : batchnorm所需要的一些參數
	eps      : 接近0的數，防止分母出現0
	momentum : 動量參數，一般爲0.9， 0.99， 0.999
	running_mean ：滑動平均的方式計算新的均值，訓練時計算，爲測試數據做準備
	running_var  : 滑動平均的方式計算新的方差，訓練時計算，爲測試數據做準備
"""
	running_mean = bn_param['running_mean']  #shape = [B]
    running_var = bn_param['running_var']    #shape = [B]
	results = 0. # 建立一個新的變量
    
	x_mean=np.mean(x, axis=(0,1,2), keepdim=True)  # 計算x的均值
    x_var=np.var(x, axis=(0,1,2), keepdim=True)    # 計算方差
    x_normalized=(x-x_mean)/np.sqrt(x_var+eps)       # 歸一化
    results = gamma * x_normalized + beta            # 縮放平移

    running_mean = momentum * running_mean + (1 - momentum) * x_mean
    running_var = momentum * running_var + (1 - momentum) * x_var
    
    #記錄新的值
    bn_param['running_mean'] = running_mean
    bn_param['running_var'] = running_var 
    
	return results , bn_param
	
# https://blog.csdn.net/qq_25737169/article/details/79048516

Layer Normalization

對於LN來說，其中$S_i$是沿着$C, H, W$三個方向。LN在RNN的效果比較理想。在CNN的效果不如BN

def Layernorm(x, gamma, beta):

    # x_shape:[B, H, W， C]
    results = 0.
    eps = 1e-5

    x_mean = np.mean(x, axis=(1, 2, 3), keepdims=True)
    x_var = np.var(x, axis=(1, 2, 3), keepdims=True)
    x_normalized = (x - x_mean) / np.sqrt(x_var + eps)
    results = gamma * x_normalized + beta
    return results

Instance Normalization

對於IN來說，其中$S_i$是沿着$H, W$兩個方向。
IN主要用於計算風格遷移。

def Instancenorm(x, gamma, beta):

    # x_shape:[B, H, W, C]
    results = 0.
    eps = 1e-5

    x_mean = np.mean(x, axis=(2, 3), keepdims=True)
    x_var = np.var(x, axis=(2, 3), keepdims=True)
    x_normalized = (x - x_mean) / np.sqrt(x_var + eps)
    results = gamma * x_normalized + beta
    return results

Group Normalization

對於LN來說，其中$S_i$是沿着$C, H, W$三個方向。是LN和IN的一般化表示方法。LN和IN分別是GN的兩種特殊情況。
GN對於batchsize較小的情況下表現仍然穩定。

def GroupNorm(x, gamma, beta, G=32):

    # x_shape:[B, H, W, C]
    results = 0.
    eps = 1e-5
    shape = np.shape(x)
    x = np.reshape(x, (x.shape[0], x.shape[2], x.shape[3], G, x.shape[-1]/32))

    x_mean = np.mean(x, axis=(1, 2, 4), keepdims=True)
    x_var = np.var(x, axis=(1, 2, 4), keepdims=True)
    x_normalized = (x - x_mean) / np.sqrt(x_var + eps)
    results = gamma * x_normalized + beta
    results = np.reshape(results, shape)
    return results

Spectral Normalization²

我們先說一下Spectral Normalization的實現方式，之後再說其推理過程以及作用。
SN是針對超參數W進行的處理。

def spectral_norm(w, iteration=1):
    w_shape = w.shape.as_list()
    w = tf.reshape(w, [-1, w_shape[-1]])

    # 
    u = tf.get_variable("u", [1, w_shape[-1]], 
        initializer=tf.random_normal_initializer(), trainable=False)

    u_hat = u
    v_hat = None
    for i in range(iteration):
        """
        power iteration
        Usually iteration = 1 will be enough
        """
        v_ = tf.matmul(u_hat, tf.transpose(w))
        v_hat = tf.nn.l2_normalize(v_)

        u_ = tf.matmul(v_hat, w)
        u_hat = tf.nn.l2_normalize(u_)

    u_hat = tf.stop_gradient(u_hat)
    v_hat = tf.stop_gradient(v_hat)

    sigma = tf.matmul(tf.matmul(v_hat, w), tf.transpose(u_hat))

    with tf.control_dependencies([u.assign(u_hat)]):
        w_norm = w / sigma
        w_norm = tf.reshape(w_norm, w_shape)

    return w_norm

爲啥要對卷積使用的參數進行SN，這就要回到最初討論GAN的不足上了。
Iran GoodFellow發明GAN網絡之後，大家就對這網絡進行了深入的研究，發現這個玩意兒容易蹦，而且參數稍微設置的不順心，訓練結果就跟沒訓練差不多。這到底是啥原因呢。
³

AdaIN（自適應實例歸一化，StyleGANs的方法）

Total_Variation Regularization⁴

這種歸一化的作用是減少椒鹽噪聲，使得畫面平滑。
下面採用手寫和調用Tensorflow的API來使用上述過程：

import tensorflow as tf
from PIL import Image
import numpy as np

img_path = '' # 圖片地址
img = Image.open(img_paht)
image = np.array(img)
image = tf.cast(image, tf.int32)

# 調用API
tv = tf.reduce_mean(tf.image.total_variation(image))

# 手動實現
tv_h = tf.reduce_sum(tf.abs(image[1:,:,:] - image[:-1,:,:]))
tv_w = tf.reduce_sum(tf.abs(image[:,1:,:] - image[:,:-1,:]))
tv2 = (tv_h + tv_w)

with tf.Session() as sess:
    tv, tv2 = sess.run([tv, tv2])
    print('tv = ', tv)
    print('tv2 = ', tv2)

結果如下：
tv = 34654796
tv2 = 34654796

---

1. Group Normalization ↩︎

2. SPECTRAL NORMALIZATION FOR GENERATIVE ADVERSARIAL NETWORKS ↩︎

3. Spectral Normalization 譜歸一化/https://zhuanlan.zhihu.com/p/55393813 ↩︎

4. Image Up-Sampling Using Total-Variation Regularization With a New Observation Model ↩︎