Summary
of the Thesis written by
Pietro Mele
Analysis and Development of a Modular Neural Network based on "Mixture of Experts", and comparison with Boosting Algorithms
Computer Science Department
Università degli Studi di Milano
2001
Contents
1. Introduction.
2. Introduction to learning algorithms and neural networks.
3. General description of a modular neural network.
- Ensemble averaging.
- Mixture of Experts.
4. Comparison with other neural networks.
5. Comparison with "Boosting" algorithms.
6. Parallelisms with the central nervous system.
7. Analytic description of a network with the "Mixture of Experts" architecture.
8. Analysis of its behavior.
9. Description of a possible implementation with the UML modeling language.
10. Description of a possible implementation with the Standard C++ programming language.
11. Use of wavelets for input pre-elaboration.
12. Use of genetic algorithms for the structure development.
13. Conclusions.
Appendices:
A. Software used.
B. Introduction to UML.
C. Notes about Standard C++.
References.
2. Introduction to learning algorithms and neural networks.
3. General description of a modular neural network.
- Ensemble averaging.
- Mixture of Experts.
4. Comparison with other neural networks.
5. Comparison with "Boosting" algorithms.
6. Parallelisms with the central nervous system.
7. Analytic description of a network with the "Mixture of Experts" architecture.
8. Analysis of its behavior.
9. Description of a possible implementation with the UML modeling language.
10. Description of a possible implementation with the Standard C++ programming language.
11. Use of wavelets for input pre-elaboration.
12. Use of genetic algorithms for the structure development.
13. Conclusions.
Appendices:
A. Software used.
B. Introduction to UML.
C. Notes about Standard C++.
References.
Introduction
In this thesis I
will describe a particular kind of neural network, known as «Mixture of
Experts». I will treat this network model under a theoretical
point of view, I will describe a real implementation, and I will make a
comparison with other algorithms that try to solve the same class of
problems in a similar way.
In particular, in chapter 2 I will introduce the concept of learning algorithm, starting from a generic point of view and then focusing on neural networks. In chapter 3 I will describe the basic concepts of a modular neural network. In chapter 4 I will make a comparison between this model and the most common networks, underlining their similarities and differences. In chapter 5 I will make a comparison with the so called "Boosting" algorithm. In chapter 6 I will try to find a parallelism between the mixture model and the central nervous system. In chapters 7 and 8 I will analytically describe how a particular kind of modular network works, the so called “Mixture of Experts”. In chapter 9 I will represent a possible implementation using some UML diagrams, that will give you a view of the system with a graphical notation. In chapter 10 I will show the implementation I am developing using the Standard C++ programming language. In chapter 11 I will make an analysis of the results and a comparison with the most used algorithms. In the two following chapters I will discuss two possible extensions of the basic algorithm that would allow an efficient pre-elaboration of the input signal through wavelets, and an autonomous development of the network architecture using genetic algorithms.
Introduction to learning algorithms and neural networks
General description of a modular neural network
The chapter starts with a description of the concept of "modularity": with the term "module" we identify a part of a more complex system, that is usually represented as a "black box" the inner workings of which are unknown.
The neural networks seen so far can be considered modular at the level of their neurons and their layers.
In a modular network, this concept of module is extended. Each of these units encloses one or more complete and independent networks; the module is defined as an "expert" that gives his answer to the current input signal.
Generalizing, a module is not forced to contain a neural network, and it can be a generic "container" of any kind of algorithm.
It is possible to follow different strategies to get an algorithm that realizes such a system:
Static algorithms can be grouped in different categories, two of which, discussed in this thesis, are:
The dynamic algorithms considered here are the so called "mixture of experts".
Ensemble Averaging
This algorithm is the simplest one between those that realize a modular neural network; we have a fixed number of modules, called expert networks, that have a common input and that, through a possibly unknown internal mechanism, generate an output signal. Subsequently, the outputs of the single modules are combined in some way to produce the output of the whole network.
The reason why the use of a set of experts can be convenient compared to a single neural network depends on the fact that (1) the number of parameters (e.g. synaptic weights) to be considered can be smaller and (2) each expert will converge, in general, to different local minima on the error surface, improving the global output as a combination of the single experts' output.
Mixture of Experts
This model can be derived from the ensemble averaging one, because the basic structure of the network is substantially the same. The only difference is the presence of another module, called gating network, that controls the output of the expert networks. Even this module, like the others, can be considered as a black-box, which receives the same input vector the other network modules receive, and generates an output vector whose elements can be seen as coefficients that weight the output of the expert networks.
The gating network does not change its output according to the current input, only; it also changes its behaviour during a learning phase, through which it can learn how to control the output of the other modules.
Another fundamental role the gating network has is to control the learning rates of the expert networks, which depend on the current input.
Comparison with "Boosting" algorithms
"Boosting" is a static method; however it is very different from ensemble averaging, even if they both are in the same class of algorithms. Infact, while in the ensemble averaging method all the experts are trained on the same training set, in the boosting method the experts are trained on data sets that can be completely distinct.
Boosting can also be seen as a general methodology to improve the performance of any learning algorithm.
Parallelisms with the central nervous system
The central nervous system can be seen both as a highly modular and hierarchical structure: as a consequence, at different observation scales, you can note different kinds of modules, "nested" in higher level modules, and built of simpler sub-models.
Considering the nervous system as a whole, we can identify the following main components:
Observing the brain from the point of view of the learning mechanisms it employs, we can split it into three main areas, corresponding to the three classes of algorithms previously described:
In particular, in chapter 2 I will introduce the concept of learning algorithm, starting from a generic point of view and then focusing on neural networks. In chapter 3 I will describe the basic concepts of a modular neural network. In chapter 4 I will make a comparison between this model and the most common networks, underlining their similarities and differences. In chapter 5 I will make a comparison with the so called "Boosting" algorithm. In chapter 6 I will try to find a parallelism between the mixture model and the central nervous system. In chapters 7 and 8 I will analytically describe how a particular kind of modular network works, the so called “Mixture of Experts”. In chapter 9 I will represent a possible implementation using some UML diagrams, that will give you a view of the system with a graphical notation. In chapter 10 I will show the implementation I am developing using the Standard C++ programming language. In chapter 11 I will make an analysis of the results and a comparison with the most used algorithms. In the two following chapters I will discuss two possible extensions of the basic algorithm that would allow an efficient pre-elaboration of the input signal through wavelets, and an autonomous development of the network architecture using genetic algorithms.
Introduction to learning algorithms and neural networks
In this chapter I
will give an overview on different kinds of learning algorithms. I will
make a distinction between the different points of view you can adopt
to interpret or to realize and "intelligent object": a top-down view,
inspired by the way we think, and a bottom-up view that comes from how
the brain works on a small dimensional scale.
A general discussion on generic learning algorithms will follow: these can be described as modules that receive some input data and return some output data, and they must be able to establish some kind of relation between the first and the second. A distinction between supervised, reinforcement and unsupervised learning will follow.
A general discussion on generic learning algorithms will follow: these can be described as modules that receive some input data and return some output data, and they must be able to establish some kind of relation between the first and the second. A distinction between supervised, reinforcement and unsupervised learning will follow.
General description of a modular neural network
The chapter starts with a description of the concept of "modularity": with the term "module" we identify a part of a more complex system, that is usually represented as a "black box" the inner workings of which are unknown.
The neural networks seen so far can be considered modular at the level of their neurons and their layers.
In a modular network, this concept of module is extended. Each of these units encloses one or more complete and independent networks; the module is defined as an "expert" that gives his answer to the current input signal.
Generalizing, a module is not forced to contain a neural network, and it can be a generic "container" of any kind of algorithm.
It is possible to follow different strategies to get an algorithm that realizes such a system:
- Static
algorithms: the modules' outputs are
combined in a way independent from
the current input.
- Dynamic algorithms: the modules' outputs are combined in a way dependent from the current input.
- Dynamic algorithms: the modules' outputs are combined in a way dependent from the current input.
Static algorithms can be grouped in different categories, two of which, discussed in this thesis, are:
- Ensemble
averaging: the modules' outputs are
linearly combined.
- Boosting: a weak learning algorithm is converted in an algorithm that can reach an arbitrary accuracy.
- Boosting: a weak learning algorithm is converted in an algorithm that can reach an arbitrary accuracy.
The dynamic algorithms considered here are the so called "mixture of experts".
Ensemble Averaging
This algorithm is the simplest one between those that realize a modular neural network; we have a fixed number of modules, called expert networks, that have a common input and that, through a possibly unknown internal mechanism, generate an output signal. Subsequently, the outputs of the single modules are combined in some way to produce the output of the whole network.
The reason why the use of a set of experts can be convenient compared to a single neural network depends on the fact that (1) the number of parameters (e.g. synaptic weights) to be considered can be smaller and (2) each expert will converge, in general, to different local minima on the error surface, improving the global output as a combination of the single experts' output.
Mixture of Experts
This model can be derived from the ensemble averaging one, because the basic structure of the network is substantially the same. The only difference is the presence of another module, called gating network, that controls the output of the expert networks. Even this module, like the others, can be considered as a black-box, which receives the same input vector the other network modules receive, and generates an output vector whose elements can be seen as coefficients that weight the output of the expert networks.
The gating network does not change its output according to the current input, only; it also changes its behaviour during a learning phase, through which it can learn how to control the output of the other modules.
Another fundamental role the gating network has is to control the learning rates of the expert networks, which depend on the current input.
Comparison with "Boosting" algorithms
"Boosting" is a static method; however it is very different from ensemble averaging, even if they both are in the same class of algorithms. Infact, while in the ensemble averaging method all the experts are trained on the same training set, in the boosting method the experts are trained on data sets that can be completely distinct.
Boosting can also be seen as a general methodology to improve the performance of any learning algorithm.
Parallelisms with the central nervous system
The central nervous system can be seen both as a highly modular and hierarchical structure: as a consequence, at different observation scales, you can note different kinds of modules, "nested" in higher level modules, and built of simpler sub-models.
Considering the nervous system as a whole, we can identify the following main components:
- the
two
cerebral hemispheres, involved in the higher cognitive, perceptive and
motor functions;
- the cerebellum, which regulates the moveements' force and precision, controls their learning, equilibrium and ocular movements;
- the basal ganglia, involved in high leveel aspects of movement and in involuntary movements;
- the diencephalon, which is a link structture between different parts of the nervous system;
- the brain stem, which receives sensory iinformation about equilibrium, taste and sounds, and controls facial muscles and ocular bulbs;
- the spinal cord, which is both the commuunication channel with the body's periphery, and a place where signals are treated in a reflexive and, hence, involuntary way.
- the cerebellum, which regulates the moveements' force and precision, controls their learning, equilibrium and ocular movements;
- the basal ganglia, involved in high leveel aspects of movement and in involuntary movements;
- the diencephalon, which is a link structture between different parts of the nervous system;
- the brain stem, which receives sensory iinformation about equilibrium, taste and sounds, and controls facial muscles and ocular bulbs;
- the spinal cord, which is both the commuunication channel with the body's periphery, and a place where signals are treated in a reflexive and, hence, involuntary way.
All these components are
themselves made of lower level structures, different according to their
function.
Observing the brain from the point of view of the learning mechanisms it employs, we can split it into three main areas, corresponding to the three classes of algorithms previously described:
- The cerebellum would be
specialized
in a kind of supervised learning (through the interaction between
parallel fibers and climbing fibers). One of its tasks would be the
reconstruction of an internal model of the environment.
- The basal ganglia seem to use reinforcemment learning, to evaluate a set of situations and to select a proper consequent action.
- The cerebral cortex, on the other hand, would work in an unsupervised way, representing the external environment status and the internal context. Furthermore, since they are not directly connected, it lets the cerebellum and the basal ganglia communicate through the thalamus.
- The basal ganglia seem to use reinforcemment learning, to evaluate a set of situations and to select a proper consequent action.
- The cerebral cortex, on the other hand, would work in an unsupervised way, representing the external environment status and the internal context. Furthermore, since they are not directly connected, it lets the cerebellum and the basal ganglia communicate through the thalamus.
