Interactive Activation and Competition

Aim:

Interactive activation and competition nets are a class of neural network different from backpropagation networks. They tend to model choice and clustering phenomena in biological neural networks

Reference: Chapter 2 of McClelland, JL and Rumelhart, DE, Explorations in Parallel Distributed Processing, MIT Press, 1989.

Plan:

network architecture update rule network dynamics how competition works Jets and Sharks dataset retrieval from partial descriptions graceful degradation generalization

Intro to Interactive Activation and Competition (IAC) Nets

• an IAC network is a collection of processing units organised into a number of competitive pools

• there are excitatory connections between units in different pools and inhibitory connections between units in the same pool

• excitatory connections are usually bi-directional

• inhibitory connections usually interconnect all units in a pool

IAC Network Diagram

[If you are looking at a black-and-white printout of this diagram, the (red) inhibitory connections are the ones entirely inside pools, and the (blue) excitatory connections are the ones between pools.]

IAC Unit Diagram

IAC Network Activation, Threshold, and Weights

• units take on continuous activation values between a maximum max and minimum min

• connections between units are weighted - the weight from unit j to unit i is denoted w_ij

• an connection is excitatory if its weight w_ij > 0
  an connection is inhibitory if its weight w_ij < 0

• units have a resting value rest: if they get no input from other units, their activation level will decay to this resting value at a rate determined by a decay parameter.

Net Input, and Change of Activation

• Time is discrete in an IAC model - that is, it changes in finite steps rather than continuously. In each time step, the unit activations are recalculated based on the values in the previous time step, and using formulae given below.

• the net input of unit i is calculated as

  net_i = Σ_j w_ij a_j + extinput_i

• the change in activation resulting from the net input is as follows:

  if (net_i > 0)
     Δa_i = (max – a_i)net_i – decay(a_i – rest)
  else
     Δa_i = (a_i – min)net_i – decay(a_i – rest)

The Update Rule

• (repeated): the change in activation resulting from the net input is as follows:

  if (net_i > 0)
     Δa_i = (max – a_i)net_i – decay(a_i – rest)
  else
     Δa_i = (a_i – min)net_i – decay(a_i – rest)

• decay is the rate at which activation will decay in the absence of input

• rest is the resting value of the unit activation, when it decays in the absence of input

• Typical values for the parameters are:

  max = 1		min ≤ rest ≤ 0
  0 ≤ decay ≤ 1		initially, min ≤ ai ≤ max

Activation Dynamics

• (repeated): the change in activation resulting from the net input is as follows:

  if (net_i > 0)
     Δa_i = (max – a_i)net_i – decay(a_i – rest)
  else
     Δa_i = (a_i – min)net_i – decay(a_i – rest)

• If net input to a particular unit remains constant but positive, then Δa_i will get smaller and smaller as the activation of the unit (a_i) gets larger and larger.

• Δa_i can in fact be negative, if a_i is larger than rest and net_i is not large enough to balance the decay.

• Exercise: Figure out the expression for Δa_i if net_i > 0 and
  (i) activation a_i = max;
  (ii) activation a_i = rest
  Is Δa_i positive or negative in these cases?

Activation Stable Point

• Another point of interest is the circumstances under which Δa_i = 0 (i.e. when the activation value is stable). Solving Δa_i = 0 when net_i > 0 gives:

  a_i = ( max × net_i + rest × decay ) / ( net_i + decay )

  If max = 1 and rest = 0, then comes down to:

  a_i = net_i / ( net_i + decay )

  which is close to 1 if net_i is large compared with decay.

• Exercise: Calculate the condition for stability if net_i is negative.

How Competition Works

• In a real situation, net input does not stay fixed, and in fact normally all activations can change each time step.

• Consider two units a and b that are in competition, and assume that the total excitatory input to a (e_a) is larger than the total excitatory input to b (e_b).

• Let γ represent the strength of the mutual inhibition between a and b. Then the net inputs to a and b are:

  net_a = e_a – γ×output_b
  net_b = e_b – γ×output_a

• While the activations are still positive, output_i = a_i, so:

  net_a = e_a – γa_b
  net_b = e_b – γa_a

• Thus b, which starts smaller, gets smaller still, and so on in a vicious circle, so ultimately a "wins" - units with slight initial advantages, in terms of external inputs, amplify this advantage over their competitors.

Resonance

• If unit a and unit b have mutually excitatory connections, then if say a becomes active, it will activate b, which will activate a right back, and so on - so that they will keep each other active (other things being equal).

• The resonance can in some situations counteract decay.

• This has been called resonance by Grossberg¹.

1. Grossberg, S., A theory of visual coding, memory, and development, in E.L.J. Leeuwenberg & H.F.J.M. Buffart (eds), Formal theories of visual perception. New York: Wiley, 1978.

Example of Resonance in the Presence of Decay

• Suppose two units, a and b, have bidirectional, excitatory connections with strengths equal to 2 × decay. Suppose each unit's activation is initially ½ and that there is no other input.

• Then a_a = a_b = ½
  and net_a = connection_strength × a_b = 2×decay × ½ = decay, so

  Δaa	= (1 – aa)neta – decay×aa
  	= (1 – ½)×decay – decay×½
  	= ½×decay – decay×½
  	= 0.

  and the same for b. So the activations will remain the same forever.

Retrieval and Generalisation in Human Memory

An IAC system can be used to illustrate the following properties of human memory:

• Retrieval by name and content

• Retrieval with noisy cues

• Assignment of plausible default values when stored information is incomplete

• Spontaneous generalization over a set of familiar items

The "Jets and Sharks" Database

Characteristics of a number of individuals belonging to two gangs, the Jets and the Sharks. (From "Retrieving General and Specific Knowledge From Stored Knowledge of Specifics" by J. L. McClelland, 1981, Proceedings of the Third Annual Conference of the Cognitive Science Society.)

Some Jets+Sharks Connections

After a diagram in the McClelland paper cited above.

PDP++ Software

• You can get software that runs IAC and a range of other neural network models from the PDP++ home page at http://www.cnbc.cmu.edu/Resources/PDP++/PDP++.html.

• (The actual code is found by clicking on the USA-Colorado link under the heading "FTP". I haven't tried to compile the code recently.)

Retrieving an Individual from their Name

• Arrange for internal input to the instance unit corresponding to Ken;

• Let the system run for several time steps;

• Inspect the activations;

• We would find that the units for Sharks, 20s, High School, Single, and Burglar, and the name unit for Ken were all activated.

Retrieval from Partial Description

• Ken happens to be the only gang member who is a Shark and in his 20s

• We can activate both Shark and 20s, and let the system run for several time steps

• Then we would find that Ken's instance node and his attributes are activated.

• Other nodes will be activated to a much lesser degree, because other nodes are connected to Sharks, or to 20s. But Ken's stuff dominates.

Graceful Degradation

• Brains can operate reasonably well even in the presence of wrong or inconsistent information.

• We can activate all of Ken's properties except that JuniorHigh is activated instead of HighSchool.

• After running the system for several time steps, we find that the Ken instance node is correctly activated.

• This might not work if there was someone in the gangs who had the same attributes as Ken except for Name and Education and whose Education was in fact Junior High.

Spontaneous Generalisation

• What happens if we activate the "Jets" unit only and run for several timesteps?

• What you get is an activation pattern that provides a general description of what Jets are like.

• You can of course do similar things with other attribute units, like 20s or JuniorHigh

CRICOS Provider Code No. 00098G
Copyright (C) Bill Wilson, 2012, except where another source is acknowledged.