Dynamic Nets - Solutions

COMP9844 Extended Neural Networks

Dynamic Nets - Solutions

It is best not to read the answers until you've tried to answer the questions yourself.

What three problems with backprop does cascade-correlation try to solve?

Answer: The stepsize problem, the moving target problem, and the "herd effect" problem.
How does cascade-correlation address each of the problems mentioned in the previous question?

Answer:
stepsize problem: Cascade correlation per se does not address this, though the use of quickprop (second-derivative approximations to gradient descent) as the learning algorithm does in fact help. (See the cited paper for more details).
moving target problem: in backprop, the output neurons are trying to produce correct outputs based on hidden layer neuron performance that is changing at the same time. Cascade correlation trains one neuron at a time with the previously trained neurons' weights frozen, so this problem doesn't arise.
herd effect problem: training one neuron at a time means that parts of the problem that are already solved stay solved, and each new neurons then tackles a different subproblem.
How does the use of a pool of candidate neurons improve cascade correlation?

Answer: Stochastic effects mean that different neurons trained to solve the same problem will do so with different degrees of success. In backprop, with all the neurons being trained at once, the only way to deal with this is to re-run the entire algorithm from a new random starting point. With cascade correlation, it is possible to train several neurons to solve a subproblem, pick the best, then move on to train candidate neurons to solve the next subproblem.