Dustin Stevens-Baer
Comp 572 Neural
Networks
Assignment #1
The source code I used can be found here. and
here..
1. Does the Energy decrease at each iteration?
The Energy function should decrease in value at
each step of the iteration -- if you observe that
this is not true then there is something wrong with
your implementation.
Plot the Energy as a function of t for:
cases: step1, step2, step3, step4 and ext1
That's 4 plots, but it would be nice to have all 4
on the same axes for comparison purposes.
Yes, the Energy function decreases at each iteration you can see how the lines become more and more l shaped losing there slow slopes
and gaining rather drastic slopes to start out with.
2. Does the network converge to a state that corresponds to "k-winners"?
For example, in the case e = 1.5 there should be 2 "winners", and the 2
winners should correspond to the two neurons that had the highest initial
activations.
Demonstrate this by showing what the initial activations were, and what
the final convergent state was for a single run (random initial activations)
of each of the parameter cases.
(Note: that's 12 cases)
The initial activiations and final
convergent states can be seen here.
Yes the network converges to a state where there are k winners at each
iteration. The winners correspond to the nerons with the highest
activation. In some situations you can see that the neurons stablize
quickly and in others it takes forever.
3. Try a case where the initial activations are all
equal (non-zero).
a. Is there convergence to winners?
There is no convergence to the
winner state. All values converge to the low state
b.
Does the Energy decrease at each iteration?
Yes the energy does decrease at each
iteration.
c. Demonstrate your responses by showing a specific simulation
run:
-- Initial activations
-- Activations at convergence
-- Energy plot
Here
is the simulation statistics. The graph of the energy decrease is shown
below.
4. Show (mathematical proof) that the k-winner
states are stable equilibrium when the external input satisfies: k-1 < ext
< k.
Hint: K-winner states are the corners of the hypercube
corresponding to k neurons at maximum activation and the rest at minimum
activation. To show that these states are stable equilibrium you must imagine
the activation vector as being very close to, but not on, the hypercube corner.
Then show that the dynamics of the network will "drive" the activation vector
into the corner, as opposed to away from the corner. That is, if an activation
is near the maximum, then the update will add a positive amount, and if an
activation is near the minimum the update will add a negative amount.
The activations are updated by the
following equation, which was given as part of the problem statement: ai(t+1) = ai(t) +step*(M-ai(t))* (ai(t) - m) * neti(t), Assuming that M=1.0 and m=0.0 The step is positive and we
also know that ai(t) is between M and m. So, increase or decrease in a
neuron's activation is controlled by its net input, neti(t). A neuron's net
input is defined as: net i(t) = [Wa(t) + e(t)]i
neti(t) = Wa(t) + e(t), where Wa(t) is the weight sums of all
incoming connections to a node, and e(t) is the external input to the
network. Therfore, neti(t) will only be positive when
Wa(t) > -e(t). The (M-ai(t))* (ai(t) - m) will always be positive unless one of the thresholds is
actually reached, then the value will be 0 and the equation will result in:
ai(t+1) = ai(t)
For a stable
state with k winners eachwinner will receive input from the other k-1
winners, and since the input from all connecting nodes is -1. neti(t) = (-1)(k-1) + e(t). And for all the non-winners, they
will receive input from k winners, so at equilibrium they will look like:
neti(t) = (-1)(k) + e(t).
If k>e(t), then all the
losers will be driven towards the value m, since this will cause neti(t) to be negative. And if (k-1)<e(t), then the winners will
be driven towards M, since this causes neti(t) to be
positive.
Therefore we can conclude that k-1 < ext < k