ActivationFunctionHiddenLayer

ActivationFunctionHiddenLayer

ActivationFunctionHiddenLayer is a property of a NeuralNet object. It specifies the activation function used in the hidden layer(s). It can be one of Linear, Threshold, ThresholdSymmetric, Sigmoid, SigmoidStepwise, SigmoidSymmetric, SigmoidSymmetricStepwise. See each function for further information. The default is SigmoidSymmetric.

Linear

Linear is a value of the NeuralNet properties ActivationFunctionHiddenLayer and ActivationFunctionOutputLayer. The linear activation function can produce any output values.

Linear is defined as : -∞<x<∞, f (x) = x s, f^′ (x) = 1 s, f (x)  -∞ ... ∞ where s is steepness .

Sigmoid

Sigmoid is a value of the NeuralNet properties ActivationFunctionHiddenLayer and ActivationFunctionOutputLayer. The sigmoid activation function is the usual sigmoid function with output between zero and one.

RowBox[{Sigmoid is defined as : 0<x<1, ,, f (x) = 1 / (1 - ^(2 s x)), ,, f^R ... s x (1 - x), ,, RowBox[{f (x), , RowBox[{RowBox[{0., ..}], 1}]}]}] where s is steepness .

SigmoidStepwise

SigmoidStepwise is a value of the NeuralNet properties ActivationFunctionHiddenLayer and ActivationFunctionOutputLayer. The stepwise sigmoid activation function is a piecewise linear approximation of the usual sigmoid function with output between zero and one.

RowBox[{Sigmoid Stepwise is an approximation of : 0<x<1, ,, f (x) = 1/(1 - ^(2 s ... s x (1 - x), ,, RowBox[{f (x), , RowBox[{RowBox[{0., ..}], 1}]}]}] where s is steepness .

SigmoidSymmetric

SigmoidSymmetric is a value of the NeuralNet properties ActivationFunctionHiddenLayer and ActivationFunctionOutputLayer. The symmetrical sigmoid activation function is the usual tanh sigmoid function with output between minus one and one. It is the default activation function.

RowBox[{SigmoidSymmetric is defined as : 0<x<1, ,, f (x) = 2/(1 + ^(-2 s x)), ,, ... s (1 - x^2), ,, RowBox[{f (x), , RowBox[{RowBox[{0., ..}], 1}]}]}] where s is steepness .

SigmoidSymmetricStepwise

SigmoidSymmetricStepwise is a value of the NeuralNet properties ActivationFunctionHiddenLayer and ActivationFunctionOutputLayer. The symmetrical sigmoid activation function is a piecewise linear approximation o the usual tanh sigmoid function with output between minus one and one.

RowBox[{Sigmoid Stepwise is an approximation of : 0<x<1, ,, f (x) = 2/(1 + ^(-2 ... s (1 - x^2), ,, RowBox[{f (x), , RowBox[{RowBox[{0., ..}], 1}]}]}] where s is steepness .

Threshold

Threshold is a value of the NeuralNet properties ActivationFunctionHiddenLayer and ActivationFunctionOutputLayer. The threshold activation function acts as a binary switch. Threshold is not differentiable and can therefore not be used for training.

RowBox[{Threshold is defined as : -∞<x<∞, ,, ∀x<0 : f (x) = 0∧∀x≥0 : f (x) = 1, ,, RowBox[{f (x), , RowBox[{0, ∨, 1.}]}]}]

ThresholdSymmetric

ThresholdSymmetric is a value of the NeuralNet properties ActivationFunctionHiddenLayer and ActivationFunctionOutputLayer. The symmetric threshold activation function acts as a binary switch with values of -1 and 1. ThresholdSymmetric is not differentiable and can therefore not be used for training.

RowBox[{ThresholdSymmetric is defined as : -∞<x<∞, ,, ∀x<0 : f (x) = -1∧∀x≥0 : f (x) = 1, ,, RowBox[{f (x), , RowBox[{-1, ∨, 1.}]}]}]

Created by freegoldbar  (September 16, 2004)

Hosted by www.Geocities.ws

1