COMP9444/9844 Neural Networks

Feedback Quiz on Tensor Product Networks - Solutions

It is best not read the answers until you've tried to answer the questions yourself.

1. What does one-shot learning mean?

   Answer: learning which happens in a single change to weights or similar structures, as opposed to a sequence of small incremental changes.

2. Describe the role of binding units in a rank 2 tensor product network.

   Answer: the binding units store the facts ("bindings") known to the network; each fact is provided in the form of an outer product, one component of which is added to each binding units. (The number of binding units is equal to the product of the dimensions of the role and filler spaces.)

3. Define a distributed representation.

   Answer: A distributed representation is one in which many of the components of the representation vectors are non-zero.

4. What does Σ_iΣ_i b_ijv_if_j = 1 signify in a rank 2 tensor product network.

   Answer: that the variable v is bound to the filler f.

5. Define a Hadamard matrix.

   Answer: A Hadamard matrix H is a square matrix whose entries are all either +1 or –1, and such that HH^T = nI_n, where n is the length of the size of the matrix (and I_n is the n×n identity matrix).

6. How can a rank 3 tensor product network be used to store a relational fact like hit(max, frank)?

   Answer: Let h, m, and f be the vectors representing hit, max, and frank. Form the outer product of h, m, and f, and add it to the binding unit tensor for the network.

7. Explain how a rank 3 tensor product network storing suitable relational facts can be used to solve a proportional analogy problem like dog : kennel :: rabbit : What?

   Answer: Let the vectors representing dog and kennel be input to the ARG1 and ARG2 sides of the tensor. Take the "relational bundle" that is output by the REL side of the tensor, together with the vector representing rabbit, and input them to the REL and ARG1 one side of the tensor. The output from the ARG2 side of the tensor should be a weighted sum of ARG2 concepts. The concept with the largest weight should be the solution to the problem.

8. Explain the accessibility property of tensor product networks.

   Answer: Concepts can be input to any combination of one or more sides of the tensor, and a vector or tensor of will be output from the remaining sides (or if there are none, then a discriminant value of +1 or –1 is output). For example, for a rank 3 tensor, with "axes" labelled REL, ARG1 and ARG2, the possible inputs and corresponding outputs are:

   Inputs	Outputs
   REL, ARG1, ARG2	Discriminant
   REL, ARG1	ARG2
   REL, ARG2	ARG1
   ARG1, ARG2	REL
   REL	tensor ARG1⊗ARG2
   ARG1	tensor REL⊗ARG2
   ARG2	tensor REL⊗ARG1

9. What is a representable non-fact?

   Answer: A representable non-fact is one all of whose atomic concepts have representation vectors, but which has not been taught to the tensor product network (or other system for storing relational knowledge).

10. What is a (dense) random representation?

    Answer: A dense random representation is one formed by creating represenation vectors by generating random components from a uniform distribution over the interval [–r, +r], and then normalising the resulting vectors. Such vectors will have an average inner product of zero, and are in this sense quasi-orthogonal.

Bill Wilson's contact info

UNSW's CRICOS Provider No. is 00098G