Recurrent neural nets

Another tool that has been used for analysing temporal classification problems is recurrent neural networks. A neural network consists of a set of units, an example of which is shown in figure 2.2. The unit has a weight associated with each input. A function of the weights and inputs (typically, a squashing function applied to the sum of the weight-input products) is then generated as an output.

  
Figure 2.2: A unit

These individual units are connected together as shown in figure 2.3, with an input layer, an output layer and usually a hidden layer. Typically, the input layer consists of one unit per attribute, and the output layer of one unit per class. The number of units in the hidden layer is arbitrary. Through algorithms such as backpropagation, the weights of the neural net can be adjusted so as to produce an output on the appropriate unit when a particular pattern at the input is observed. The interested reader may wish to find out more in [RM86].

  
Figure 2.3: A typical feedforward neural network

A recurrent neural network (RNN) is a modification to this architecture to allow for temporal classification, as shown in figure 2.4. In this case, a ``context'' layer is added to the structure, which retains information between observations. At each timestep, new inputs are fed into the RNN. The previous contents of the hidden layer are passed into the context layer. These then feed back into the hidden layer in the next time step.

In an algorithm similar to the backpropagation algorithm, called back propagation through time (BPTT), the weights of the hidden layers and context layers are set.

To do classification, postprocessing of the outputs from the RNN is performed; so, for example, when a threshold on the output from one of the nodes is observed, we register that a particular class has been observed.

  
Figure 2.4: Recurrent neural network architecture

Recurrent neural networks suffer from many of the same problems as hidden Markov models, namely:

• There are many control variables. Many different control variables need to be set in a neural network system, such as the number of units in the hidden layer, the appropriate structure, the correct parameter adjustment algorithm (there are many alternatives to backpropagation), the learning rate, the encoding and more. As of yet, well-formed techniques for deciding appropriate values for these parameters are still developing.
• Their efficacy for learning long connected sequences is unproven. Attempts to learn long sequences of events, for example, [Ben96], lasting for hundreds of samples, have not shown good results. They do seem capable of learning short sequences (tens of frames), such as learning individual phonemes, rather than whole words.
• Extracting meaning from a normal feedforward neural networks has proved difficult. Extracting it from recurrent neural networks is even more so.
• They do not have fast learning curves. They usually require many examples before converging, especially if there are many inputs or the hidden layer contains many nodes.