Non-linear Feature Extraction by Redundancy Reduction in an Unsupervised Stochastic Neural Network.
Unsupervised feature extraction by a stochastic neural network can be defined as a minimization of the redundancy between the elements of the output layer, given complete information transfer from input to output. Redundancy minimization can be achieved by minimization of the mutual information between the units of the output layer. Complete information transfer is enforced by maximizing the mutual information of the input and output. With these two conditions we define a novel learning algorithm for stochastic recurrent networks. The minimum of redundancy corresponds to the extraction of statistically independent features, leading to a factorial representation of the environment. The resulting learning rule includes Hebbian and anti-Hebbian learning terms. These two terms are weighted by the amount of information transmitted in the learning synapse minus the grade of redundant information in the corresponding output neuron, giving thus, an information-theoretic interpretation of the proportionality constant of Hebb's biological rule. Simulations demonstrate the performance of this method. When a retina is simulated, the learning algorithm forms decorrelated receptive fields. This represents the first experiment that extends the results of the linear principle component analysis to the nonlinear case by a direct implementation of Barlow's principle of redundancy reduction for unsupervised features extraction by receptive fields formation in a retina model. Copyright 1997 Elsevier Science Ltd.