SEARCH · PubMed Health

Results for “recurrent neural network”

Explore indexed PubMed citations for clinical trials, systematic reviews and public health research. Read source abstracts and follow each citation to its original PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2Linked to original sources

PubMed2006

Improved conditions for global exponential stability of recurrent neural networks with time-varying delays.

This paper presents new theoretical results on global exponential stability of recurrent neural networks with bounded activation functions and time-varying delays. The stability conditions depend on external inputs, connection weights, and time delays of recurrent neural networks. Using these results, the global exponential stability of recurrent neural networks can be derived, and the estimated location of the equilibrium point can be obtained. As typical representatives, the Hopfield neural network (HNN) and the cellular neural network (CNN) are examined in detail.

Algorithms↗

PubMed2004

Inaccessibility in online learning of recurrent neural networks.

We apply nonlinear dynamical system techniques to recurrent neural networks. In particular, we numerically analyze the dynamical system characteristics of the online learning process. By introducing the notion of inaccessibility, we show that the learning process is well characterized by strong nonhyperbolicity and inaccessibility, which is a greater uncertainty than chaotic unpredictability. These results are clearly contrasted with a gradient descent dynamics, or ordinary chaos.

Artificial Intelligence↗

PubMed2004

A one-layer recurrent neural network for support vector machine learning.

This paper presents a one-layer recurrent neural network for support vector machine (SVM) learning in pattern classification and regression. The SVM learning problem is first converted into an equivalent formulation, and then a one-layer recurrent neural network for SVM learning is proposed. The proposed neural network is guaranteed to obtain the optimal solution of support vector classification and regression. Compared with the existing two-layer neural network for the SVM classification, the proposed neural network has a low complexity for implementation. Moreover, the proposed neural network can converge exponentially to the optimal solution of SVM learning. The rate of the exponential convergence can be made arbitrarily high by simply turning up a scaling parameter. Simulation examples based on benchmark problems are discussed to show the good performance of the proposed neural network for SVM learning.

Journal Article↗

PubMed2004

Markovian architectural bias of recurrent neural networks.

In this paper, we elaborate upon the claim that clustering in the recurrent layer of recurrent neural networks (RNNs) reflects meaningful information processing states even prior to training [1], [2]. By concentrating on activation clusters in RNNs, while not throwing away the continuous state space network dynamics, we extract predictive models that we call neural prediction machines (NPMs). When RNNs with sigmoid activation functions are initialized with small weights (a common technique in the RNN community), the clusters of recurrent activations emerging prior to training are indeed meaningful and correspond to Markov prediction contexts. In this case, the extracted NPMs correspond to a class of Markov models, called variable memory length Markov models (VLMMs). In order to appreciate how much information has really been induced during the training, the RNN performance should always be compared with that of VLMMs and NPMs extracted before training as the "null" base models. Our arguments are supported by experiments on a chaotic symbolic sequence and a context-free language with a deep recursive structure. Index Terms-Complex symbolic sequences, information latching problem, iterative function systems, Markov models, recurrent neural networks (RNNs).

Markov Chains↗

PubMed2003

A local training-pruning approach for recurrent neural networks.

The global extended Kalman filtering (EKF) algorithm for recurrent neural networks (RNNs) is plagued by the drawback of high computational cost and storage requirement. In this paper, we present a local EKF training-pruning approach that can solve this problem. In particular, the by-products, obtained along with the local EKF training, can be utilized to measure the importance of the network weights. Comparing with the original global approach, the proposed local approach results in much lower computational cost and storage requirement. Hence, it is more practical in solving real world problems. Simulation showed that our approach is an effective joint-training-pruning method for RNNs under online operation.

Algorithms↗

PubMed2005

Global exponential periodicity of a class of recurrent neural networks with oscillating parameters and time-varying delays.

In this paper, we present the analytical results on the global exponential periodicity of a class of recurrent neural networks with oscillating parameters and time-varying delays. Sufficient conditions are derived for ascertaining the existence, uniqueness and global exponential periodicity of the oscillatory solution of such recurrent neural networks by using the comparison principle and mixed monotone operator method. The periodicity results extend or improve existing stability results for the class of recurrent neural networks with and without time delays.

Animals↗

PubMed2003

Global dissipativity of continuous-time recurrent neural networks with time delay.

This paper addresses the global dissipativity of a general class of continuous-time recurrent neural networks. First, the concepts of global dissipation and global exponential dissipation are defined and elaborated. Next, the sets of global dissipativity and global exponentially dissipativity are characterized using the parameters of recurrent neural network models. In particular, it is shown that the Hopfield network and cellular neural networks with or without time delays are dissipative systems.

Journal Article↗

PubMedDate not supplied

[Dynamic paradigm in psychopathology: "chaos theory", from physics to psychiatry].

For the last thirty years, progress in the field of physics, known as "Chaos theory"--or more precisely: non-linear dynamical systems theory--has increased our understanding of complex systems dynamics. This framework's formalism is general enough to be applied in other domains, such as biology or psychology, where complex systems are the rule rather than the exception. Our goal is to show here that this framework can become a valuable tool in scientific fields such as neuroscience and psychiatry where objects possess natural time dependency (i.e. dynamical properties) and non-linear characteristics. The application of non-linear dynamics concepts on these topics is more precise than a loose metaphor and can throw a new light on mental functioning and dysfunctioning. A class of neural networks (recurrent neural networks) constitutes an example of the implementation of the dynamical system concept and provides models of cognitive processes (15). The state of activity of the network is represented in its state space and the time evolution of this state is a trajectory in this space. After a period of time those networks settle on an equilibrium (a kind of attractor). The strength of connections between neurons define the number and relations between those attractors. The attractors of the network are usually interpreted as "mental representations". When an initial condition is imposed to the network, the evolution towards an attractor is considered as a model of information processing (27). This information processing is not defined in a symbolic manner but is a result of the interaction between distributed elements. Several properties of dynamical models can be used to define a way where the symbolic properties emerge from physical and dynamical properties (28) and thus they can be candidates for the definition of the emergence of mental properties on the basis of neuronal dynamics (42). Nevertheless, mental properties can also be considered as the result of an underlying dynamics without explicit mention of the neuronal one (47). In that case, dynamical tools can be used to elucidate the Freudian psychodynamics (34, 35). Recurrent neuronal networks have been used to propose interpretation of several mental dysfunctions (12). For example in the case of schizophrenia, it has been proposed that troubles in the cortical pruning during development (13) may cause a decrease in neural network storage ability and lead to the creation of spurious attractors. Those attractors do not correspond to stored memories and attract a large amount of initial conditions: they were thus associated to reality distorsion observed in schizophrenia (14). Nevertheless, the behavior of these models are too simple to be directly compared with real physiological data. In fact, equilibrium attractors are hardly met in biological dynamics. More complex behaviors (such as oscillations or chaos) should thus to be taken into account. The study of chaotic behavior have lead to the development of numerical methods devoted to the analysis of complex time series (17). These methods may be used to characterise the dynamical processes at the time-scales of both the cerebral dynamics and the clinical symptoms variations. The application of these methods to physiological signals have shown that complex behaviors are related to healthy states whereas simple dynamics are related to pathology (8). These studies have thus confirmed the notion of "dynamical disease" (20, 21) which denotes pathological conditions characterised by changes in physiological rhythms. Depression has been studied within this framework (25, 32) in order to define possible changes in brain electrical rhythms related to this trouble and its evolution. It has been shown that controls' brain dynamics is more complex than depressive one and that the recovery of a complex brain activity depends on the number of previous episodes. In the case of the symptoms time evolution, several studies have demonstrated that non-linear dynamical process may be involved in the recurrence of symptoms in troubles such as manic-depressive illness (9) or schizophrenia (51). These observations can contribute to more parcimonious interpretation of the time course of these illnesses than usual theories. In the search of a relationship between brain dynamics and mental troubles, it has been shown in three depressed patients an important correlation between the characteristics of brain dynamics and the intensity of depressive mood (49). This preliminary observation is in accordance with the emergence hypothesis according which changes in neuronal dynamics should be related to changes in mental processes. We reviewed here some theoretical and experimental results related to the use of "physical" dynamical theory in the field of psychopathology. It has been argued that these applications go beyond metaphor and that they are empirically founded. Nevertheless, these studies only constitute first steps on the way of a cautious development and definition of a "dynamical paradigm" in psychopathology. The introduction of concepts from dynamics such as complexity and dynamical changes (i.e. bifurcations) permits a new perspective on function and dysfunction of the mind/brain and the time evolution of symptoms. Moreover, it offers a ground for the hypothesis of the emergence of mental properties on the basis of neuronal dynamics (42). Since this theory can help to throw light on classical problems in psychopathology, we consider that a precise examination of both its theoretical and empirical consequences is requested to define its validity on this topic.

Brain↗

PubMed2001

Recurrent neural networks for predicting outcomes after liver transplantation: representing temporal sequence of clinical observations.

OBJECTIVES: This paper investigates a version of recurrent neural network with the backpropagation through time (BPTT) algorithm for predicting liver transplant graft failure based on a time series sequence of clinical observations. The objective is to improve upon the current approaches to liver transplant outcome prediction by developing a more complete model that takes into account not only the preoperative risk assessment, but also the early postoperative history. METHODS: A 6-fold cross-validation procedure was used to measure the performance of the networks. The data set was divided into a learning set and a test set by maintaining the same proportion of positive and negative cases in the original set. The effects of network complexity on overfitting were investigated by constructing two types of networks with different numbers of hidden units. For each type of network, 10 individual networks were trained on the learning set and used to form a committee. The performance of the networks was measured exhaustively with respect to both the entire training and test sets. RESULTS: The networks were capable of learning the time series problem and achieved good performances of 90% correct classification on the learning set and 78% on the test set. The prediction accuracy increases as more information becomes progressively available after the operation with the daily improvement of 10% on the learning set and 5% on the test set. CONCLUSIONS: Recurrent neural networks trained with BPTT algorithm are capable of learning to represent temporal behavior of the time series prediction task. This model is an improvement upon the current model that does not take into account postoperative temporal information.

Adult↗

PubMed2005

A bayesian recurrent neural network for unsupervised pattern recognition in large incomplete data sets.

A recurrent neural network, modified to handle highly incomplete training data is described. Unsupervised pattern recognition is demonstrated in the WHO database of adverse drug reactions. Comparison is made to a well established method, AutoClass, and the performances of both methods is investigated on simulated data. The neural network method performs comparably to AutoClass in simulated data, and better than AutoClass in real world data. With its better scaling properties, the neural network is a promising tool for unsupervised pattern recognition in huge databases of incomplete observations.

Algorithms↗

PubMed2002

Center-crossing recurrent neural networks for the evolution of rhythmic behavior.

A center-crossing recurrent neural network is one in which the null-(hyper)surfaces of each neuron intersect at their exact centers of symmetry, ensuring that each neuron's activation function is centered over the range of net inputs that it receives. We demonstrate that relative to a random initial population, seeding the initial population of an evolutionary search with center-crossing networks significantly improves both the frequency and the speed with which high-fitness oscillatory circuits evolve on a simple walking task. The improvement is especially striking at low mutation variances. Our results suggest that seeding with center-crossing networks may often be beneficial, since a wider range of dynamics is more likely to be easily accessible from a population of center-crossing networks than from a population of random networks.

Algorithms↗

PubMed2004

A complex-valued RTRL algorithm for recurrent neural networks.

A complex-valued real-time recurrent learning (CRTRL) algorithm for the class of nonlinear adaptive filters realized as fully connected recurrent neural networks is introduced. The proposed CRTRL is derived for a general complex activation function of a neuron, which makes it suitable for nonlinear adaptive filtering of complex-valued nonlinear and nonstationary signals and complex signals with strong component correlations. In addition, this algorithm is generic and represents a natural extension of the real-valued RTRL. Simulations on benchmark and real-world complex-valued signals support the approach.

Algorithms↗

PubMed1999

Embedding recurrent neural networks into predator-prey models.

We study changes of coordinates that allow the embedding of ordinary differential equations describing continuous-time recurrent neural networks into differential equations describing predator-prey models-also called Lotka-Volterra systems. We transform the equations for the neural network first into quasi-monomial form (Brenig, L. (1988). Complete factorization and analytic solutions of generalized Lotka-Volterra equations. Physics Letters A, 133(7-8), 378-382), where we express the vector field of the dynamical system as a linear combination of products of powers of the variables. In practice, this transformation is possible only if the activation function is the hyperbolic tangent or the logistic sigmoid. From this quasi-monomial form, we can directly transform the system further into Lotka-Volterra equations. The resulting Lotka-Volterra system is of higher dimension than the original system, but the behavior of its first variables is equivalent to the behavior of the original neural network. We expect that this transformation will permit the application of existing techniques for the analysis of Lotka-Volterra systems to recurrent neural networks. Furthermore, our results show that Lotka-Volterra systems are universal approximators of dynamical systems, just as are continuous-time neural networks.

Journal Article↗

PubMed2002

Fast computation with spikes in a recurrent neural network.

Neural networks with recurrent connections are sometimes regarded as too slow at computation to serve as models of the brain. Here we analytically study a counterexample, a network consisting of N integrate-and-fire neurons with self excitation, all-to-all inhibition, instantaneous synaptic coupling, and constant external driving inputs. When the inhibition and/or excitation are large enough, the network performs a winner-take-all computation for all possible external inputs and initial states of the network. The computation is done very quickly: As soon as the winner spikes once, the computation is completed since no other neurons will spike. For some initial states, the winner is the first neuron to spike, and the computation is done at the first spike of the network. In general, there are M potential winners, corresponding to the top M external inputs. When the external inputs are close in magnitude, M tends to be larger. If M>1, the selection of the actual winner is strongly influenced by the initial states. If a special relation between the excitation and inhibition is satisfied, the network always selects the neuron with the maximum external input as the winner.

Journal Article↗

PubMed2006

Short-term memory for serial order: a recurrent neural network model.

Despite a century of research, the mechanisms underlying short-term or working memory for serial order remain uncertain. Recent theoretical models have converged on a particular account, based on transient associations between independent item and context representations. In the present article, the authors present an alternative model, according to which sequence information is encoded through sustained patterns of activation within a recurrent neural network architecture. As demonstrated through a series of computer simulations, the model provides a parsimonious account for numerous benchmark characteristics of immediate serial recall, including data that have been considered to preclude the application of recurrent neural networks in this domain. Unlike most competing accounts, the model deals naturally with findings concerning the role of background knowledge in serial recall and makes contact with relevant neuroscientific data. Furthermore, the model gives rise to numerous testable predictions that differentiate it from competing theories. Taken together, the results presented indicate that recurrent neural networks may offer a useful framework for understanding short-term memory for serial order.

Artificial Intelligence↗

PubMed2006

Global asymptotical stability of recurrent neural networks with multiple discrete delays and distributed delays.

By employing the Lyapunov-Krasovskii functional and linear matrix inequality (LMI) approach, the problem of global asymptotical stability is studied for recurrent neural networks with both discrete time-varying delays and distributed time-varying delays. Some sufficient conditions are given for checking the global asymptotical stability of recurrent neural networks with mixed time-varying delay. The proposed LMI result is computationally efficient as it can be solved numerically using standard commercial software. Two examples are given to show the usefulness of the results.

Algorithms↗

PubMed2006

Recurrent neural network architecture with pre-synaptic inhibition for incremental learning.

We propose a recurrent neural network architecture that is capable of incremental learning and test the performance of the network. In incremental learning, the consistency between the existing internal representation and a new sequence is unknown, so it is not appropriate to overwrite the existing internal representation on each new sequence. In the proposed model, the parallel pathways from input to output are preserved as possible, and the pathway which has emitted the wrong output is inhibited by the previously fired pathway. Accordingly, the network begins to try other pathways ad hoc. This modeling approach is based on the concept of the parallel pathways from input to output, instead of the view of the brain as the integration of the state spaces. We discuss the extension of this approach to building a model of the higher functions such as decision making.

Affect↗

PubMed1997

Training Spatially Homogeneous Fully Recurrent Neural Networks in Eigenvalue Space.

A new design method for spatially-homogeneous, fully recurrent neural networks is presented. In our approach the eigenvalues of the synaptic matrix, rather than the weights, are learned from the examples. When the learning process is carried out, the connection weights are easily computed from the eigenvalues by inverse discrete Fourier transform. The adaptation is performed in the eigenvalue space in order to simply incorporate in the training algorithm the conditions for the uniqueness of the steady-state. As a consequence, the trained networks are insensitive to initial conditions. The method is illustrated by computer simulations concerning two specific feature extraction examples. Copyright 1996 Elsevier Science Ltd.

Journal Article↗