Defining your own Neural Net Module

Modules are bricks to build neural networks. A Module is a neural network by itself, but it can be combined with other networks using container classes to create complex neural networks. Module is an abstract class which defines fundamental methods necessary for a training a neural network. All modules are serializable.

Modules contain two states variables: output and gradInput. Here we review the set of basic functions that a Module has to implement:

[output] forward(input)
Takes an input object, and computes the corresponding output of the module. In general input and output are Tensors. However, some special sub-classes like table layers might expect something else.

After a forward(), the output state variable should have been updated to the new value.

It is not advised to override this function. Instead, one should implement updateOutput(input) function. The forward module in the abstract parent class Module will call updateOutput(input).

[gradInput] backward(input, gradOutput)
Performs a backpropagation step through the module, with respect to the given input. In general this method makes the assumption forward(input) has been called before, with the same input. This is necessary for optimization reasons. If you do not respect this rule, backward() will compute incorrect gradients.

A backpropagation step consist in computing two kind of gradients at input given gradOutput (gradients with respect to the output of the module). This function simply performs this task using two function calls:

• A function call to updateGradInput(input, gradOutput).
• A function call to accGradParameters(input,gradOutput).

It is not advised to override this function call in custom classes. It is better to override updateGradInput(input, gradOutput) and accGradParameters(input, gradOutput) functions.

[output] updateOutput(input, gradOutput)
When defining a new module, this method should be overloaded.

Computes the output using the current parameter set of the class and input. This function returns the result which is stored in the output field.

[gradInput] updateGradInput(input, gradOutput)
When defining a new module, this method should be overloaded.

Computing the gradient of the module with respect to its own input. This is returned in gradInput. Also, the gradInput state variable is updated accordingly.

[gradInput] accGradParameters(input, gradOutput)
When defining a new module, this method may need to be overloaded, if the module has trainable parameters.

Computing the gradient of the module with respect to its own parameters. Many modules do not perform this step as they do not have any parameters. The state variable name for the parameters is module dependent. The module is expected to accumulate the gradients with respect to the parameters in some variable.

Zeroing this accumulation is achieved with zeroGradParameters() and updating the parameters according to this accumulation is done with updateParameters().

reset()
This method defines how the trainable parameters are reset, i.e. initialized before training.

---

Modules provide a few other methods that you might want to define, if you are not planing to use the optim package. These methods help zero() the parameters, and update them using very basic techniques.

In terms of code structure, Torch provides a class model, which we use for inheritance, and in general for the definition of all the modules in nn. Here is an empty holder for a typical new class:

local NewClass, Parent = torch.class('nn.NewClass', 'nn.Module')

function NewClass:__init()
   parent.__init(self)
end

function NewClass:updateOutput(input)
end

function NewClass:updateGradInput(input, gradOutput)
end

function NewClass:accGradParameters(input, gradOutput)
end

function NewClass:reset()
end

When defining a new class, all we need to do is fill in these empty functions. Note that when defining the constructor __init(), we always call the parent’s constructor first.

Dropout Activation Units

local Dropout, Parent = torch.class('nn.Dropout', 'nn.Module')

function Dropout:__init(percentage)
   Parent.__init(self)
   self.p = percentage or 0.5
   if self.p > 1 or self.p < 0 then
      error('<Dropout> illegal percentage, must be 0 <= p <= 1')
   end
end

function Dropout:updateOutput(input)
   self.noise = torch.rand(input:size()) -- uniform noise between 0 and 1
   self.noise:add(1 - self.p):floor()  -- a percentage of noise
   self.output:resizeAs(input):copy(input)
   self.output:cmul(self.noise)
   return self.output
end

function Dropout:updateGradInput(input, gradOutput)
   self.gradInput:resizeAs(gradOutput):copy(gradOutput)
   self.gradInput:cmul(self.noise) -- simply mask the gradients with the noise vector
   return self.gradInput
end

The file is provided in this directory, in Dropout.lua. The script 1_dropout.lua demonstrates how to create an instance of this module, and test it on some data (lena):

-- in this file, we test the dropout module we've defined:
require 'nn'
require 'Dropout'
require 'image'

-- define a dropout object:
n = nn.Dropout(0.5)

-- load an image:
i = image.lena()

-- process the image:
result = n:forward(i)

-- display results:
image.display{image=i, legend='original image'}
image.display{image=result, legend='dropout-processed image'}

-- some stats:
mse = i:dist(result)
print('mse between original imgae and dropout-processed image: ' .. mse)