COMP572

Neural Networks

Ludovic Hilde

Summer 2008

07/23/08

Assignment 5

NETWORK ARCHITECTURE

The network is made of a grid of 120 neurons (10 columns and 12 rows). All neurons are interconnected and there are no self-connections. Each neuron is activated with either +1 or -1.

During the training phase, all weights are initialized to zero, and then they are updated when adding a new pattern as follows:

wij +=1 if ai * aj > 0

wij -=1 if ai * aj < 0

SIMULATION

The simulation first trains the network by inputting digit 0 to 3. It then has to recognize identical and noisy patterns.

Click on the link below to access the source code of the simulation:

Binary Hopfield Network

TRAINING

During the first test, I trained the network with digits represented by a single line. As a result, the activation value of these patterns contained a lot more -1 than +1, and the network was unable to reproduce those digits without distortion (see pictures below).

Original activations for digit 0 are:

- - X X X X X X - -

- X - - - - - - X -

X - - - - - - - - X

- X - - - - - - X -

- - X X X X X X - -

Updated activations for digit 0 are:

- - X X X X X X - -

- X - - - - - - X -

- - - - - - - - - -

- - - - - - - - X -

- - X X X X X X - -

Original activations for digit 1 are:

- - - - - X - - - -

- - - - X X - - - -

- - - X - X - - - -

- - - - - X - - - -

- - X X X X X X - -

Updated activations for digit 1 are:

- - X X X X X X - -

- - - - - - - - - -

- - - - - X - - - -

- - - - - - - - - -

- - - - - X - - - -

- - - - - - - - - -

- - - - - X - - - -

- - - - - - - - - -

- - X X X X X X - -

Original activations for digit 2 are:

- - X X X X X X - -

- X - - - - - - X -

- - - - - - - X - -

- - - - - - X - - -

- - - - - X - - - -

- - - - X - - - - -

- - - X - - - - - -

- - X - - - - - X -

- X X X X X X X X -

Updated activations for digit 2 are:

- - X X X X X X - -

- X - - - - - - X -

- - - - - - - - - -

- - - - - X - - - -

- - - - - - - - - -

- - - - - - - - X -

- X X X X X X X - -

Original activations for digit 3 are:

- X X X X X X X X -

- - - - - - - X - -

- - - - - - X - - -

- - - - - X - - - -

- - - - X - - - - -

- - - X X X X - - -

- - - - - - - X - -

- - - - - - - - X -

- - - - - - - X - -

- X X X X X X - - -

Updated activations for digit 3 are:

- - X X X X X X - -

- - - - - - - - - -

- - - - - X - - - -

- - - - - - - - - -

- - - - - X - - - -

- - - - - - - - - -

- X X X X X X X - -

The second test gave much better results. After increasing the size of the line as shown in the following drawings, the initial and final states were identical after convergence of the network.

Original activations for digit 0 are:

- - X X X X X X - -

- X X X X X X X X -

X X - - - - - - X X

- X X X X X X X X -

- - X X X X X X - -

Updated activations for digit 0 are:

- - X X X X X X - -

- X X X X X X X X -

X X - - - - - - X X

- X X X X X X X X -

- - X X X X X X - -

Original activations for digit 1 are:

- - - - X X - - - -

- - - X X X - - - -

- - X X X X - - - -

- - - - X X - - - -

- - X X X X X X - -

Updated activations for digit 1 are:

- - - - X X - - - -

- - - X X X - - - -

- - X X X X - - - -

- - - - X X - - - -

- - X X X X X X - -

Original activations for digit 2 are:

- - X X X X X X - -

- X X X X X X X X -

- X X - - - - X X -

- - - - - - X X - -

- - - - - X X - - -

- - - - X X - - - -

- - - X X - - - - -

- - X X - - - - - -

- X X X X X X X X -

Updated activations for digit 2 are:

- - X X X X X X - -

- X X X X X X X X -

- X X - - - - X X -

- - - - - - X X - -

- - - - - X X - - -

- - - - X X - - - -

- - - X X - - - - -

- - X X - - - - - -

- X X X X X X X X -

Original activations for digit 3 are:

- X X X X X X X X -

- X X X X X X X - -

- - - - - X X - - -

- - - - X X - - - -

- - - X X - - - - -

- - X X X X X - - -

- - X X X X X X - -

- - - - - - - X X -

- X X X X X X X - -

- X X X X X X - - -

Updated activations for digit 3 are:

- X X X X X X X X -

- X X X X X X X - -

- - - - - X X - - -

- - - - X X - - - -

- - - X X - - - - -

- - X X X X X - - -

- - X X X X X X - -

- - - - - - - X X -

- X X X X X X X - -

- X X X X X X - - -

NOISY PATTERN

The performance of the binary network has also been tested with noisy pattern.

Digit 1 with 10 percent was perfectly recalled (see drawing below).

Original activations for digit 1 with 10 percent noise are:

- - X - X X - - - -

- - - X X X - X - -

- - X - X X - - - -

- - - - X X - - - X

- - X - X X - - - -

- - - X X X - - - -

- - - - X X - - X -

- - - - - X - - - -

X - - - X X - - - -

- - X - X X - - - -

- - X X X - X X - -

- - X X X X X - - -

Updated activations for digit 1 with 10 percent noise are:

- - - - X X - - - -

- - - X X X - - - -

- - X X X X - - - -

- - - - X X - - - -

- - X X X X X X - -

Digit 2 with 20 percent is perfectly recalled (see drawing below).

Original activations for digit 2 with 20 percent noise are:

X - X X X X - X - -

- X - X - X X X X -

- X X - X X - X X -

- - X - - - - X X X

- - X - - X - X X -

- X X - - - X X - -

- - - - X - X - - -

- - X - X X X - - -

X - - X - - - - - -

- - - X - - - X - -

- - X X X X X X - -

- X - X X X - X X -

Updated activations for digit 2 with 20 percent noise are:

- - X X X X X X - -

- X X X X X X X X -

- X X - - - - X X -

- - - - - - X X - -

- - - - - X X - - -

- - - - X X - - - -

- - - X X - - - - -

- - X X - - - - - -

- X X X X X X X X -

Digit 2 with 30 percent is perfectly recalled (see drawing below). (This is impressive because the input pattern hardly looks like a 2.)

Original activations for digit 2 with 30 percent noise are:

X - X - X X X X X -

- - X X - X X X - -

X X X - - - X X - -

- X X X X X - X X -

- X X - - - X - - -

- X - - - - X - - X

X - - X - X X X - -

- - X - X - - - X -

- - - - X - X - - X

X - X X - X - - - X

- X - X X X X - - -

- - X X - X X X X X

Updated activations for digit 2 with 30 percent noise are:

- - X X X X X X - -

- X X X X X X X X -

- X X - - - - X X -

- - - - - - X X - -

- - - - - X X - - -

- - - - X X - - - -

- - - X X - - - - -

- - X X - - - - - -

- X X X X X X X X -

I was also interested in finding out the kind of pattern the network would output after presenting an unknown pattern. The network did not associate any of its canned patterns to the unknown pattern (see drawing below).

Original activations for unknown pattern are:

- - - - X X - - X -

- X X X X X X X X -

X X - - - X - X X -

X X - X - X - - X X

- X - X - - - - - X

- X X X X X X X X -

X - - - - - - - X -

- - X - - - - X - X

- - X - X X - X X -

- X X - - - X X - -

- - - - X X - - - -

- - - X - - - - - -

Updated activations for unknown pattern are:

- - X X X X X X - -

- X X X X X X X - -

- - - - - - - - - -

- - - - X - - - - -

- - - - X - - - X -

- - - - - - X - - -

- - - - X X X - - -

- - - - - - - - X -

- X X X X X X X X -

- X X X X X X X - -

At last, I inverted all activation values for pattern 0. Interestingly, this pattern is stable as it was recalled by the network correctly.

Original activations for inverted digit 0 are:

X X - - - - - - X X

X - - - - - - - - X

- - X X X X X X - -

X - - - - - - - - X

X X - - - - - - X X

Updated activations for inverted digit 0 are:

X X - - - - - - X X

X - - - - - - - - X

- - X X X X X X - -

X - - - - - - - - X

X X - - - - - - X X