theano學習指南5（翻譯）- 降噪自動編碼器

降噪自動編碼器是經典的自動編碼器的一種擴展，它最初被當作深度網絡的一個模塊使用 [Vincent08]。這篇指南中，我們首先也簡單的討論一下自動編碼器。

自動編碼器

文獻[Bengio09] 給出了自動編碼器的一個簡介。在編碼過程，它可以把輸入x∈[0,1]dx∈[0,1]d映射到一個隱式表達y∈[0,1]d′y∈[0,1]d′。映射關係定義如下：

y=s(Wx+b)y=s(Wx+b)

其中，ss爲一個非線性函數，比如sigmoid。在解碼過程，映射後的yy可以重新映射成和輸入具有相同維度的zz，這個過程和編碼過程非常類似。

z=s(W′y+b′)z=s(W′y+b′)

這裏需要注意，上標並不表示轉置。zz可以看作對於給定y的條件下，xx的預測。當然，逆映射的權重W′W′也可以是映射種權重WW的轉置。W′=WTW′=WT，這種情況被成爲綁定的權重。模型的係數（WW,bb,b′b′,以及W′W′如果兩個映射矩陣不綁定的話）可以通過最小化平均重建錯誤的方法求解。

這裏的重建錯誤可以通過很多不同的方法量化。經典的方差L(x,z)=||x−z||2L(x,z)=||x−z||2也可以採用。如果輸入爲位向量，或者是位概率向量，重建的交叉熵也可以使用。

LH(x,z)=−∑k=1d[xklogzk+(1−xk)log(1−zk)]LH(x,z)=−∑k=1d[xklog⁡zk+(1−xk)log⁡(1−zk)]

這裏希望編碼後的yy爲能夠捕獲數據中主要變量的變化的一種分佈式表示。這個和主分量分析PCA中的原理是一樣的。事實上，如果編碼過程採用線性映射，並採用均方差準則訓練網絡，那麼這kk個隠單元可以把輸入在PCA意義下進行投影。如果隱層位非線性函數，模型會捕獲輸入的多模態信息，這和PCA有些區別。

因爲yy可以看作輸入xx的有損壓縮，它不能對所有的輸入都達到最佳壓縮的目的。優化過程可以對訓練數據達到好的壓縮目的，並希望可以應用到其他數據上面，但不能是任意數據上。這就是自動編碼器泛化的道理：它可以在和輸入數據具有相同分佈的測試數據達到較低的重構錯誤，但是在隨機數據上效果不佳。

爲了複用的方便，我們採用theano實現自動編碼器類。首先，創建模型參數的共享變量WW,bb,b′b′（這裏W′=WTW′=WT)。

 

def __init__(

    self,

    numpy_rng,

    theano_rng=None,

    input=None,

    n_visible=784,

    n_hidden=500,

    W=None,

    bhid=None,

    bvis=None

):

    """

    Initialize the dA class by specifying the number of visible units (the

    dimension d of the input ), the number of hidden units ( the dimension

    d' of the latent or hidden space ) and the corruption level. The

    constructor also receives symbolic variables for the input, weights and

    bias. Such a symbolic variables are useful when, for example the input

    is the result of some computations, or when weights are shared between

    the dA and an MLP layer. When dealing with SdAs this always happens,

    the dA on layer 2 gets as input the output of the dA on layer 1,

    and the weights of the dA are used in the second stage of training

    to construct an MLP.

 

    :type numpy_rng: numpy.random.RandomState

    :param numpy_rng: number random generator used to generate weights

 

    :type theano_rng: theano.tensor.shared_randomstreams.RandomStreams

    :param theano_rng: Theano random generator; if None is given one is

                 generated based on a seed drawn from `rng`

 

    :type input: theano.tensor.TensorType

    :param input: a symbolic description of the input or None for

                  standalone dA

 

    :type n_visible: int

    :param n_visible: number of visible units

 

    :type n_hidden: int

    :param n_hidden:  number of hidden units

 

    :type W: theano.tensor.TensorType

    :param W: Theano variable pointing to a set of weights that should be

              shared belong the dA and another architecture; if dA should

              be standalone set this to None

 

    :type bhid: theano.tensor.TensorType

    :param bhid: Theano variable pointing to a set of biases values (for

                 hidden units) that should be shared belong dA and another

                 architecture; if dA should be standalone set this to None

 

    :type bvis: theano.tensor.TensorType

    :param bvis: Theano variable pointing to a set of biases values (for

                 visible units) that should be shared belong dA and another

                 architecture; if dA should be standalone set this to None

 

 

    """

    self.n_visible = n_visible

    self.n_hidden = n_hidden

 

    # create a Theano random generator that gives symbolic random values

    if not theano_rng:

        theano_rng = RandomStreams(numpy_rng.randint(2 ** 30))

 

    # note : W' was written as `W_prime` and b' as `b_prime`

    if not W:

        # W is initialized with `initial_W` which is uniformely sampled

        # from -4*sqrt(6./(n_visible+n_hidden)) and

        # 4*sqrt(6./(n_hidden+n_visible))the output of uniform if

        # converted using asarray to dtype

        # theano.config.floatX so that the code is runable on GPU

        initial_W = numpy.asarray(

            numpy_rng.uniform(

                low=-4 * numpy.sqrt(6. / (n_hidden + n_visible)),

                high=4 * numpy.sqrt(6. / (n_hidden + n_visible)),

                size=(n_visible, n_hidden)

            ),

            dtype=theano.config.floatX

        )

        W = theano.shared(value=initial_W, name='W', borrow=True)

 

    if not bvis:

        bvis = theano.shared(

            value=numpy.zeros(

                n_visible,

                dtype=theano.config.floatX

            ),

            borrow=True

        )

 

    if not bhid:

        bhid = theano.shared(

            value=numpy.zeros(

                n_hidden,

                dtype=theano.config.floatX

            ),

            name='b',

            borrow=True

        )

 

    self.W = W

    # b corresponds to the bias of the hidden

    self.b = bhid

    # b_prime corresponds to the bias of the visible

    self.b_prime = bvis

    # tied weights, therefore W_prime is W transpose

    self.W_prime = self.W.T

    self.theano_rng = theano_rng

    # if no input is given, generate a variable representing the input

    if input is None:

        # we use a matrix because we expect a minibatch of several

        # examples, each example being a row

        self.x = T.dmatrix(name='input')

    else:

        self.x = input

 

    self.params = [self.W, self.b, self.b_prime]

這裏我們把符號inputinput作爲輸入傳給模型，這樣可以把幾個自動編碼器的層組合起來，構建深度網絡，把第kk層的輸出，當作kk+1層的輸入。

潛在表示和重建信號可以通過一下方式進行計算：

1 2 3

def get_hidden_values(self, input):     """ Computes the values of the hidden layer """     return T.nnet.sigmoid(T.dot(input, self.W) + self.b)

　　

    def get_reconstructed_input(self, hidden):
        """Computes the reconstructed input given the values of the
        hidden layer

        """
        return T.nnet.sigmoid(T.dot(hidden, self.W_prime) + self.b_prime)

接下來，我們計算損失函數，並採用SGD算法求解參數。

def get_cost_updates(self, corruption_level, learning_rate):
        """ This function computes the cost and the updates for one trainng
        step of the dA """

        tilde_x = self.get_corrupted_input(self.x, corruption_level)
        y = self.get_hidden_values(tilde_x)
        z = self.get_reconstructed_input(y)
        # note : we sum over the size of a datapoint; if we are using
        #        minibatches, L will be a vector, with one entry per
        #        example in minibatch
        L = - T.sum(self.x * T.log(z) + (1 - self.x) * T.log(1 - z), axis=1)
        # note : L is now a vector, where each element is the
        #        cross-entropy cost of the reconstruction of the
        #        corresponding example of the minibatch. We need to
        #        compute the average of all these to get the cost of
        #        the minibatch
        cost = T.mean(L)

        # compute the gradients of the cost of the `dA` with respect
        # to its parameters
        gparams = T.grad(cost, self.params)
        # generate the list of updates
        updates = [
            (param, param - learning_rate * gparam)
            for param, gparam in zip(self.params, gparams)
        ]

        return (cost, updates)

然後，我們定義一個函數來迭代模型參數以最小化重建誤差。

da = dA(
        numpy_rng=rng,
        theano_rng=theano_rng,
        input=x,
        n_visible=28 * 28,
        n_hidden=500
    )

    cost, updates = da.get_cost_updates(
        corruption_level=0.,
        learning_rate=learning_rate
    )

    train_da = theano.function(
        [index],
        cost,
        updates=updates,
        givens={
            x: train_set_x[index * batch_size: (index + 1) * batch_size]
        }
    )

    start_time = timeit.default_timer()

    ############
    # TRAINING #
    ############

    # go through training epochs
    for epoch in range(training_epochs):
        # go through trainng set
        c = []
        for batch_index in range(n_train_batches):
            c.append(train_da(batch_index))

        print('Training epoch %d, cost ' % epoch, numpy.mean(c))

    end_time = timeit.default_timer()

    training_time = (end_time - start_time)

    print(('The no corruption code for file ' +
           os.path.split(__file__)[1] +
           ' ran for %.2fm' % ((training_time) / 60.)), file=sys.stderr)
    image = Image.fromarray(
        tile_raster_images(X=da.W.get_value(borrow=True).T,
                           img_shape=(28, 28), tile_shape=(10, 10),
                           tile_spacing=(1, 1)))
    image.save('filters_corruption_0.png')

    # start-snippet-3
    #####################################
    # BUILDING THE MODEL CORRUPTION 30% #
    #####################################

    rng = numpy.random.RandomState(123)
    theano_rng = RandomStreams(rng.randint(2 ** 30))

    da = dA(
        numpy_rng=rng,
        theano_rng=theano_rng,
        input=x,
        n_visible=28 * 28,
        n_hidden=500
    )

    cost, updates = da.get_cost_updates(
        corruption_level=0.3,
        learning_rate=learning_rate
    )

    train_da = theano.function(
        [index],
        cost,
        updates=updates,
        givens={
            x: train_set_x[index * batch_size: (index + 1) * batch_size]
        }
    )

    start_time = timeit.default_timer()

    ############
    # TRAINING #
    ############

    # go through training epochs
    for epoch in range(training_epochs):
        # go through trainng set
        c = []
        for batch_index in range(n_train_batches):
            c.append(train_da(batch_index))

        print('Training epoch %d, cost ' % epoch, numpy.mean(c))

    end_time = timeit.default_timer()

    training_time = (end_time - start_time)

    print(('The 30% corruption code for file ' +
           os.path.split(__file__)[1] +
           ' ran for %.2fm' % (training_time / 60.)), file=sys.stderr)
    # end-snippet-3

    # start-snippet-4
    image = Image.fromarray(tile_raster_images(
        X=da.W.get_value(borrow=True).T,
        img_shape=(28, 28), tile_shape=(10, 10),
        tile_spacing=(1, 1)))
    image.save('filters_corruption_30.png')
    # end-snippet-4

    os.chdir('../')


if __name__ == '__main__':
    test_dA()

 降噪自動編碼器

降噪自動編碼器的動機其實很簡單。爲了強制隱層發掘更魯棒的特徵，我們將污染後的數據作爲輸入來訓練自動編碼器。

降噪自動編碼器是自動編碼器的一個隨機版本。直覺上，這個模型主要解決了兩個問題，首先，它可以對輸入進行編碼並儘量保持其信息，其次它儘量地從污染數據中，恢復輸入數據。後者主要通過分析輸入數據的統計依賴性實現的。降噪自動編碼器可以從流形學習、隨機分析等不同方面進行解釋[Vincent08]。

爲了實現降噪自動編碼器，我們只需要在自動編碼器的基礎上對輸入數據添加一個隨機污染的過程。這個有很多實現方法，這裏我們採用隨機地對輸入進行掩膜處理，使每個實例的和爲0，相應代碼如下：

 def get_corrupted_input(self, input, corruption_level):
        """This function keeps ``1-corruption_level`` entries of the inputs the
        same and zero-out randomly selected subset of size ``coruption_level``
        Note : first argument of theano.rng.binomial is the shape(size) of
               random numbers that it should produce
               second argument is the number of trials
               third argument is the probability of success of any trial

                this will produce an array of 0s and 1s where 1 has a
                probability of 1 - ``corruption_level`` and 0 with
                ``corruption_level``

                The binomial function return int64 data type by
                default.  int64 multiplicated by the input
                type(floatX) always return float64.  To keep all data
                in floatX when floatX is float32, we set the dtype of
                the binomial to floatX. As in our case the value of
                the binomial is always 0 or 1, this don't change the
                result. This is needed to allow the gpu to work
                correctly as it only support float32 for now.

        """
        return self.theano_rng.binomial(size=input.shape, n=1,
                                        p=1 - corruption_level,
                                        dtype=theano.config.floatX) * input

這樣，我們就可以得到一個完整的降噪自動編碼器類：

 

class dA(object):
    """Denoising Auto-Encoder class (dA)

    A denoising autoencoders tries to reconstruct the input from a corrupted
    version of it by projecting it first in a latent space and reprojecting
    it afterwards back in the input space. Please refer to Vincent et al.,2008
    for more details. If x is the input then equation (1) computes a partially
    destroyed version of x by means of a stochastic mapping q_D. Equation (2)
    computes the projection of the input into the latent space. Equation (3)
    computes the reconstruction of the input, while equation (4) computes the
    reconstruction error.

    .. math::

        \tilde{x} ~ q_D(\tilde{x}|x)                                     (1)

        y = s(W \tilde{x} + b)                                           (2)

        x = s(W' y  + b')                                                (3)

        L(x,z) = -sum_{k=1}^d [x_k \log z_k + (1-x_k) \log( 1-z_k)]      (4)

    """

    def __init__(
        self,
        numpy_rng,
        theano_rng=None,
        input=None,
        n_visible=784,
        n_hidden=500,
        W=None,
        bhid=None,
        bvis=None
    ):
        """
        Initialize the dA class by specifying the number of visible units (the
        dimension d of the input ), the number of hidden units ( the dimension
        d' of the latent or hidden space ) and the corruption level. The
        constructor also receives symbolic variables for the input, weights and
        bias. Such a symbolic variables are useful when, for example the input
        is the result of some computations, or when weights are shared between
        the dA and an MLP layer. When dealing with SdAs this always happens,
        the dA on layer 2 gets as input the output of the dA on layer 1,
        and the weights of the dA are used in the second stage of training
        to construct an MLP.

        :type numpy_rng: numpy.random.RandomState
        :param numpy_rng: number random generator used to generate weights

        :type theano_rng: theano.tensor.shared_randomstreams.RandomStreams
        :param theano_rng: Theano random generator; if None is given one is
                     generated based on a seed drawn from `rng`

        :type input: theano.tensor.TensorType
        :param input: a symbolic description of the input or None for
                      standalone dA

        :type n_visible: int
        :param n_visible: number of visible units

        :type n_hidden: int
        :param n_hidden:  number of hidden units

        :type W: theano.tensor.TensorType
        :param W: Theano variable pointing to a set of weights that should be
                  shared belong the dA and another architecture; if dA should
                  be standalone set this to None

        :type bhid: theano.tensor.TensorType
        :param bhid: Theano variable pointing to a set of biases values (for
                     hidden units) that should be shared belong dA and another
                     architecture; if dA should be standalone set this to None

        :type bvis: theano.tensor.TensorType
        :param bvis: Theano variable pointing to a set of biases values (for
                     visible units) that should be shared belong dA and another
                     architecture; if dA should be standalone set this to None


        """
        self.n_visible = n_visible
        self.n_hidden = n_hidden

        # create a Theano random generator that gives symbolic random values
        if not theano_rng:
            theano_rng = RandomStreams(numpy_rng.randint(2 ** 30))

        # note : W' was written as `W_prime` and b' as `b_prime`
        if not W:
            # W is initialized with `initial_W` which is uniformely sampled
            # from -4*sqrt(6./(n_visible+n_hidden)) and
            # 4*sqrt(6./(n_hidden+n_visible))the output of uniform if
            # converted using asarray to dtype
            # theano.config.floatX so that the code is runable on GPU
            initial_W = numpy.asarray(
                numpy_rng.uniform(
                    low=-4 * numpy.sqrt(6. / (n_hidden + n_visible)),
                    high=4 * numpy.sqrt(6. / (n_hidden + n_visible)),
                    size=(n_visible, n_hidden)
                ),
                dtype=theano.config.floatX
            )
            W = theano.shared(value=initial_W, name='W', borrow=True)

        if not bvis:
            bvis = theano.shared(
                value=numpy.zeros(
                    n_visible,
                    dtype=theano.config.floatX
                ),
                borrow=True
            )

        if not bhid:
            bhid = theano.shared(
                value=numpy.zeros(
                    n_hidden,
                    dtype=theano.config.floatX
                ),
                name='b',
                borrow=True
            )

        self.W = W
        # b corresponds to the bias of the hidden
        self.b = bhid
        # b_prime corresponds to the bias of the visible
        self.b_prime = bvis
        # tied weights, therefore W_prime is W transpose
        self.W_prime = self.W.T
        self.theano_rng = theano_rng
        # if no input is given, generate a variable representing the input
        if input is None:
            # we use a matrix because we expect a minibatch of several
            # examples, each example being a row
            self.x = T.dmatrix(name='input')
        else:
            self.x = input

        self.params = [self.W, self.b, self.b_prime]

    def get_corrupted_input(self, input, corruption_level):
        """This function keeps ``1-corruption_level`` entries of the inputs the
        same and zero-out randomly selected subset of size ``coruption_level``
        Note : first argument of theano.rng.binomial is the shape(size) of
               random numbers that it should produce
               second argument is the number of trials
               third argument is the probability of success of any trial

                this will produce an array of 0s and 1s where 1 has a
                probability of 1 - ``corruption_level`` and 0 with
                ``corruption_level``

                The binomial function return int64 data type by
                default.  int64 multiplicated by the input
                type(floatX) always return float64.  To keep all data
                in floatX when floatX is float32, we set the dtype of
                the binomial to floatX. As in our case the value of
                the binomial is always 0 or 1, this don't change the
                result. This is needed to allow the gpu to work
                correctly as it only support float32 for now.

        """
        return self.theano_rng.binomial(size=input.shape, n=1,
                                        p=1 - corruption_level,
                                        dtype=theano.config.floatX) * input

    def get_hidden_values(self, input):
        """ Computes the values of the hidden layer """
        return T.nnet.sigmoid(T.dot(input, self.W) + self.b)

    def get_reconstructed_input(self, hidden):
        """Computes the reconstructed input given the values of the
        hidden layer

        """
        return T.nnet.sigmoid(T.dot(hidden, self.W_prime) + self.b_prime)

    def get_cost_updates(self, corruption_level, learning_rate):
        """ This function computes the cost and the updates for one trainng
        step of the dA """

        tilde_x = self.get_corrupted_input(self.x, corruption_level)
        y = self.get_hidden_values(tilde_x)
        z = self.get_reconstructed_input(y)
        # note : we sum over the size of a datapoint; if we are using
        #        minibatches, L will be a vector, with one entry per
        #        example in minibatch
        L = - T.sum(self.x * T.log(z) + (1 - self.x) * T.log(1 - z), axis=1)
        # note : L is now a vector, where each element is the
        #        cross-entropy cost of the reconstruction of the
        #        corresponding example of the minibatch. We need to
        #        compute the average of all these to get the cost of
        #        the minibatch
        cost = T.mean(L)

        # compute the gradients of the cost of the `dA` with respect
        # to its parameters
        gparams = T.grad(cost, self.params)
        # generate the list of updates
        updates = [
            (param, param - learning_rate * gparam)
            for param, gparam in zip(self.params, gparams)
        ]

        return (cost, updates)

整合

我們可以非常容易的構建一個實例並進行訓練：

 # allocate symbolic variables for the data
    index = T.lscalar()    # index to a [mini]batch
    x = T.matrix('x')  # the data is presented as rasterized images
    #####################################
    # BUILDING THE MODEL CORRUPTION 30% #
    #####################################

    rng = numpy.random.RandomState(123)
    theano_rng = RandomStreams(rng.randint(2 ** 30))

    da = dA(
        numpy_rng=rng,
        theano_rng=theano_rng,
        input=x,
        n_visible=28 * 28,
        n_hidden=500
    )

    cost, updates = da.get_cost_updates(
        corruption_level=0.3,
        learning_rate=learning_rate
    )

    train_da = theano.function(
        [index],
        cost,
        updates=updates,
        givens={
            x: train_set_x[index * batch_size: (index + 1) * batch_size]
        }
    )

    start_time = timeit.default_timer()

    ############
    # TRAINING #
    ############

    # go through training epochs
    for epoch in range(training_epochs):
        # go through trainng set
        c = []
        for batch_index in range(n_train_batches):
            c.append(train_da(batch_index))

        print('Training epoch %d, cost ' % epoch, numpy.mean(c))

    end_time = timeit.default_timer()

    training_time = (end_time - start_time)

    print(('The 30% corruption code for file ' +
           os.path.split(__file__)[1] +
           ' ran for %.2fm' % (training_time / 60.)), file=sys.stderr)

最後，爲了對模型有一個直觀的瞭解，我們可以藉助tile_raster_images函數，將訓練得到的權重畫出來。

    image = Image.fromarray(tile_raster_images(
        X=da.W.get_value(borrow=True).T,
        img_shape=(28, 28), tile_shape=(10, 10),
        tile_spacing=(1, 1)))
    image.save('filters_corruption_30.png')

如果允許上述代碼，我們可以得到如下結果：

1. 沒有加入噪聲的模型：

2. 加入30%噪聲的模型