Fann for Mathematica User Guide

Fann for Mathematica User Guide

The Fann for Mathematica user guide shows how to use the Fann for Mathematica neural network package. It is meant to be read interactively in Mathematica, evaluating each cell in turn. It can also be viewed in static form as a web page. It does not describe the Fann C library by S. Nissen on which it is based, nor neural network theory or how to use a neural network for a particular task. See the Fann C library documentation at fann.sourceforge.net and a good textbook for information on those subjects. The Fann for Mathematica package is intended for educational purposes and come without any warranties. See the enclosed license for more information © 2004 freegoldbar (http://www.geocities.com/freegoldbar/).

Loading the Fann package

The Fann package is internally a MathLink module but it is used as a standard Mathematica package. The included Readme.txt file explains how to install the Fann package.

Contents of the Fann package

The Fann for Mathematica package contains the following functions: NeuralNet, RandomizeWeights, InitializeWeights, Train, Test, Execute, SaveFloat, SaveFixed, NeuralNetData, Shuffle and Discard. The FannGraph package contains the NeuralNetGraph function. FannGraph requires the Combinatorica standard package. The Combinatorica graph functions can be used with the Fann neural networks in graph form. A list of all symbols in the package can be obtained with the Names function. Online usage information is available for each symbol. Use ?Fann`* for quick reference.

A palette is provided so creating neural networks with Fann for Mathematica does not require memorizing any of the symbol names. Simply click in the palette to insert function calls with options and properties. All functions are included, properties and options are in collapsible sections. A partial sample of the palette with two open sections can be seen here.

Generating training data

The main example in the following shows how to use a neural network with the Fann package to mimic a simple function. First some training data has to be generated. The indata contains the training input values to be presented to the neural network. The outdata contains the desired output for each of the input values. As can be seen the function the network has to learn  is quite simple. The NeuralNetData function converts the data into the structure used by Fann.

Creating a neural network

The NeuralNet function initializes a symbol as a neural net object with a set of default properties.The properties of the net can be changed with options to the NeuralNet function or later by setting properties on the object. The difference is that NeuralNet resets all properties on the neural net to their defaults before assigning the new properties, whereas changing properties on the object directly preserves the existing properties. Because the network topology is changed by NeuralNetType, ConnectionRate and Layers, changing them will change the weights as well.

A fully connected neural network is set up using the incremental backpropagation algorithm. The network layout is given as a list with the number of neurons in each layer. Here the input and output requires one neuron each and five neurons are created in a hidden layer.

The connection weights are initialized according to the range of input values present in the training data.

Training a neural network

The neural network is ready to be trained on the training data. The desired output values are given so the error for each input can be measured and the weights adjusted. Training will stop when either the mean square error drops below 0.0005 or when 10000 iterations have been performed. The mean square errors are collected during training at regular intervals.

The mean square error can be visualized to see how the network performed during training.The error list is in a form that can directly be used with ListPlot. The LogListPlot function from the Graphics package can be used to enhance the plot.

Testing and using a neural network

The neural network can now be tested to see how well it performs against known values. The obtained output here deviates only slightly from the desired output.

To see if the neural network picked up on the underlying function in the training data it can be run over the entire input range. Plot picks input values in the given x range and where needed subdivides the x values to obtain a smooth plot. It calls on Fann` Execute function to obtain the output values. The training data and the obtained output are overlaid to illustrate how well the network performs.

Inspecting a neural network

To inspect the neural net topology and the state of the network in detail its properties can be viewed individually or all at once.

The weights are shown abbreviated when viewing all the network properties since for complex networks the number of connections and weights can be very large. The weights can be viewed by requesting the property individually.

Changing a neural network

Most properties on a neural net can be changed. The properties are given as rules. For instance to reset the mean square error, set it to zero.

Some properties such as TotalNeuronCount only provide summary information about the network, so they can not be changed directly. Attempts to change read-only properties are ignored. To change the number of neurons in the network use the Layers property where the number of neurons in each layer can be specified. All properties and the symbolic values they can take have online help so their usage and effect can be judged. The Threshold activation function can for example only be used when testing and executing a neural net.

Neural networks as graphs

A weighted, directed graph can be created from a neural network with the NeuralNetGraph function. The graph is in the form used by the standard Combinatorica package so it can be used to show the graph and study it further. The layout of the graph can be based on a grid showing the layers or based on Hasse diagrams.

In the graph neurons are shown as black dots. Bias are shown in green. The weights are shown as arrows, thicker arrows show larger relative weights. Positive weights are in blue, negative in red. Here a diagram of the net is shown with numbered neurons in HasseStyle. Hasse diagrams order the graph from input at the bottom to output at the top.

Graph algoritms can be used as well. For example BellmanFord gives a shortest-path spanning tree and associated distances from the input neuron in the sine example.

For larger networks the style of the arrows and weights are changed to only show if they are reinforcing (positive) or prohibitory (negative). The defaults can be changed with options to the NeuralNetGraph function. Here a new network is created and shown in two styles in a GraphicsArray. The processing time in ShowGraph and in producing Hasse diagrams can be substantial for very complex neural networks due to the rapidly rising number of connections. The LayerStyle on the left shows the neurons ordered by their layer from left to right.

On the lighter side, the following code snippet shows how a graph can be manipulated. It finds the neurons that are internal in the graph with the complement of the ConvexHull and animates them. After evaluating the cell, collapse the graphs and double click to see the animation. An understanding of this code snippet is not required to use the Fann package.

Visualizing neural network training

To see how a neural network learns from its errors by adjusting its weights, the neural net graph and the results of executing the network can be shown together as an animation. The following cell repeats the setup of the sine example to make it easy to try this example without evaluating the previous cells in the notebook. The DisplayFunction→Identity suppresses output of the graphics so they are only shown in the animation.

The first line in the following cell defines the number of frames to be displayed, the number of training epochs between each frame and a fixed scale used to keep the visual style of the relative weights consistent from frame to frame. Then a neural net is set up and initialized. Each frame is shown as a graphics array with a label showing how many epochs the net has been trained and its mean square error. The graphics in the array are the neural net graph and the result it produces when it is executed at that point. The NeuralNetGraph uses the StyleRange option to keep the display of the relative weights fixed to a common scale in all the frames. The standard Plot function samples the graph it is plotting at a number of points, subdividing where needed to obtain a good plot. The Execute function returns the neural net output corresponding to the input without changing the net.

Evaluate the cells to produce the animation. A series of frames will be generated, collapse them and double-click the graph to see the animation. The animation is different every time it is executed because of the randomness of the initial weight values. It is instructive to see which weights and neurons are involved in producing the result and how the neural net adjusts the weights to gradually produce a better result. Changing properties such as TrainingAlgorithm, LearningRate, SteepnessHiddenLayer show their effect on the neural net for this particular problem. Changing the number of frames and training epochs can be helpful depending on the learning ability of the neural net. Functions such as BesselJ[0, indata*10] can also be used.

Saving and restoring in Mathematica files

Neural networks can be saved and restored in Mathematica form with Put and Get. All properties are saved as a Mathematica expression. Change the path before evaluating the cell.

NeuralNetData can be saved and restored equivalently as a Mathematica expression.

Saving and restoring in Fann files

Fann files can be in floating point or fixed point format. The floating point format is suitable for use on PC's. To save a neural net in Fann floating point format use SaveFloat. Neural networks can be restored by giving a filename to a NeuralNet object. Note that the Fann file format does not store training related information such as Training algorithm and ErrorFunction, to save all properties use the Mathematica format. Due to the random connections used in sparsely connected networks, the weights can not be restored in the Mathematica format. Use the Fann format for sparsely connected networks.

The fixed point format is intended for use on devices without a floating point processor such as robots and embedded controllers. After training and testing a neural network in Mathematica, it can be saved in fixed point format and since the Fann neural network library is written in portable C, the network can readily be used in such projects. To save a neural net in Fann fixed point format use SaveFixed. Neural networks can not be restored into Fann for Mathematica from fixed point files.

Neural net data can also be saved with SaveFloat in Fann format files. It is restored with the NeuralNetData function.

When saving data in fixed point format the place of the decimal point has to be given. Data can not be restored into Fann for Mathematica from fixed point files.

Cleanup

To reclaim the resources used by a neural network in the Fann MathLink process, it can be removed with the Discard function. When the NeuralNet function is called without any arguments, it returns a list of the neural networks that have been created.

When Discard is called with a neural net as its argument, it only unloads the resources used by the given net so other neural networks can continue to be used.

Discard can also be used to unload all neural networks and close the link to the Fann package. To use the Fann package again load it with <<Fann`.

Created by freegoldbar  (September 16, 2004)