Recurrent Neural Networks

Another tool that has been used for temporal classification problems is recurrent neural networks. A neural network is made of individual units termed neurons. Each neuron has a weight associated with each input. A function of the weights and inputs (typically, a squashing function applied to the sum of the weight-input products) is then generated as an output.

These individual units are connected together as shown in Figure 3.2, with an input layer, an output layer and usually one or more hidden layers. Typically, the input layer consists of one unit per attribute, and the output layer of one unit per class. The number of units and topology within the hidden layer is defined by the user. Through algorithms such as backpropagation, the weights of the neural net can be adjusted so as to produce an output on the appropriate unit when a particular pattern at the input is observed. The backpropagation algorithm works by running the training instance through the neural network, and calculating the difference between the desired and actual outputs. These differences are then ``propagated back'' from the output layer to the hidden and input layers in the form of modifications to the weights of each of the component neurons. This modification is done in a manner proportional to the contribution to the difference in output, so that the weights most responsible for the difference are modified the most. The interested reader may wish to find out more in [RM86].

Figure 3.2: A typical feedforward neural network.

A recurrent neural network (RNN) is a modification to this architecture to allow for temporal classification, as shown in Figure 3.3. In this case, a ``context'' layer is added to the structure, which retains information between observations. At each timestep, new inputs are fed into the RNN. The previous contents of the hidden layer are passed into the context layer. These then feed back into the hidden layer in the next time step.

In an algorithm similar to the backpropagation algorithm, called back propagation through time (BPTT), the weights of the hidden layers and context layers are set.

To do classification, postprocessing of the outputs from the RNN is performed; so, for example, when a threshold on the output from one of the nodes is observed, we register that a particular class has been observed.

Figure 3.3: Recurrent neural network architecture.

Recurrent neural networks suffer from many of the same problems as HMMs, namely:

• There are many parameters. These include parameters such as the number of units in the hidden layer, the appropriate structure, the correct parameter adjustment algorithm (there are many alternatives to backpropagation), the learning rate, the encoding and more. Well-formed techniques for deciding appropriate values for these parameters are in the development stage.

• Their efficacy for learning long connected sequences is doubtful. Attempts to learn long sequences of events, for example, [Ben96], lasting for hundreds of samples, have not shown good results. They do seem capable of learning short sequences (tens of frames), such as learning individual phonemes, rather than whole words.

• There is work on extracting meaning from neural networks [Thr95,TAGD98], and there is also some work on extracting meaning from recurrent neural networks for learning very simple noise-free discrete finite state automata [OG95,VO99] with mixed results. However, whether these techniques can be applied to recurrent neural networks used to classify long-term, noisy, time-series is an open question.