COMP9444 Neural Networks

Solutions to Exercises on Hopfield Networks

1. Can the vector [1, 0, –1, 0, 1] be stored in a 5-neuron discrete Hopfield network? If so, what would be the weight matrix for a Hopfield network with just that vector stored in it? If not, why not?

   Answer: No. Components of vectors in discrete Hopfield nets must be +1 or –1.

2. 1. Compute the weight matrix for a Hopfield network with the two memory vectors [1, –1, 1, –1, 1, 1] and [1, 1, 1, –1, –1, –1] stored in it.
   2. Confirm that both these vectors are stable states of the network.

   Answer:

   1. The outer product W₁ of [1, –1, 1, –1, 1, 1] with itself is

      0	–1	1	–1	1	1
      –1	0	–1	1	–1	–1
      1	–1	0	–1	1	1
      –1	1	–1	0	–1	–1
      1	–1	1	–1	0	1
      1	–1	1	–1	1	0

      The outer product W₂ of [1, 1, 1, –1, –1, –1] with itself is

      0	1	1	–1	–1	–1
      1	0	1	–1	–1	–1
      1	1	0	–1	–1	–1
      –1	–1	–1	0	1	1
      –1	–1	–1	1	0	1
      –1	–1	–1	1	1	0

      The weight matrix W is (1/6)×(W₁ + W₂) = (1/3)×

      0	0	1	–1	0	0
      0	0	0	0	–1	–1
      1	0	0	–1	0	0
      –1	0	–1	0	0	0
      0	–1	0	0	0	1
      0	–1	0	0	1	0

   2. We have to show that the original two vectors are stable states.

      sgn(W.[1, –1, 1, –1, 1, 1])	=	sgn((2/3)×[1, –1, 1, –1, 1, 1])
      	=	[1, –1, 1, –1, 1, 1])
      so this one is stable. Similarly,
      sgn(W.[1, 1, 1, –1, –1, –1])	=	sgn((2/3)×[1, 1, 1, –1, –1, –1])
      	=	[1, 1, 1, –1, –1, –1]
      so this one is stable too.

3. Consider the following weight matrix W:

   0.0	–0.2	0.2	–0.2	–0.2
   –0.2	0.0	–0.2	0.2	0.2
   0.2	–0.2	0.0	–0.2	–0.2
   –0.2	0.2	–0.2	0.0	0.2
   –0.2	0.2	–0.2	0.2	0.0

   1. Starting in the state [1, 1, 1, 1, –1], compute the state flow to the stable state using asynchronous updates.
      
   2. Starting in the (same) state [1, 1, 1, 1, –1], compute the next state using synchronous updates.

   Answer:

   1. Asynchronous updates, starting at [1, 1, 1, 1, –1]:
      W.[1, 1, 1, 1, –1] = [0, -0.4, 0, -0.4, 0]. Hence ...
      If neuron 1, 3, or 5 updates first, its total net input is 0, so it does not change state;
      If neuron 2 updates first, its total net input is -0.4, and it's current value is one, so it changes state to –1, and the new state is [1, –1, 1, 1, –1]. Call this case A.
      If neuron 4 updates first, its total net input is -0.4, and it's current value is one, so it changes state to –1, and the new state is [1, 1, 1, –1, –1]. Call this case B.
      

      Case A: W.[1, –1, 1, 1, –1] = [0.4, 0, 0.4, –0.8, 0]. Hence, if neurons 1, 2, 3, or 5 update first, there is no stage change. If neuron 4 updates first, it flips, and the new state is [1, –1, 1, –1, –1].

      W.[1, –1, 1, –1, –1] = [0.8, –0.8, 0.8, –0.8, –0.8]. So no matter which neuron updates, there is no change. This is a stable state.

      Case B: W.[1, 1, 1, –1, –1] = [0, –0.8, 0.4, –0.4, –0.4]. Hence, if neurons 1, 3, 4 or 5 update first, there is no stage change. If neuron 2 updates first, it flips, and the new state is [1, –1, 1, –1, –1].

      This is the same state as that reached in case A, and as seen in case A, it is a stable state.

   2. Synchronous updates, starting at [1, 1, 1, 1, –1]:
      W.[1, 1, 1, 1, –1] = [0, -0.4, 0, -0.4, 0], so neurons 2 and 4 flip, resulting in a stage of [1, –1, 1, –1, –1]. (We know from the previous part that this is a stable state.)

Copyright © Bill Wilson, 2008.
Bill Wilson's contact info

UNSW's CRICOS Provider No. is 00098G