SEARCH · PubMed Health
Explore indexed PubMed citations for clinical trials, systematic reviews and public health research. Read source abstracts and follow each citation to its original PubMed record.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
The uniform asymptotic stability of recurrent neural networks (RNNs) with distributed delay is analyzed by comparing RNNs to linear Volterra integro-differential systems under Lipschitz continuity of activation functions. The stability criteria obtained have unified and extended many existing results on RNNs.
An adaptive amplitude real time recurrent learning (AARTRL) algorithm for fully connected recurrent neural networks (RNNs) employed as nonlinear adaptive filters is proposed. Such an algorithm is beneficial when dealing with signals that have rich and unknown dynamical characteristics. Following the approach from, three different cases for the algorithm are considered; a common adaptive amplitude shared among all the neurons; each layer has its own adaptive amplitude; different adaptive amplitude for each neuron. Experimental results show the AARTRL outperforms the standard RTRL algorithm.
In this paper, we present a new approach for chaos reproduction using variable structure recurrent neural networks (VSRNN). A neural network identifier is designed, with a variable structure that will change according to its output performance as compared to the given orbits of an unknown chaotic systems. A tradeoff between identification errors and computational complexity is discussed.
This letter aims at studying the impact of iterative Hebbian learning algorithms on the recurrent neural network's underlying dynamics. First, an iterative supervised learning algorithm is discussed. An essential improvement of this algorithm consists of indexing the attractor information items by means of external stimuli rather than by using only initial conditions, as Hopfield originally proposed. Modifying the stimuli mainly results in a change of the entire internal dynamics, leading to an enlargement of the set of attractors and potential memory bags. The impact of the learning on the network's dynamics is the following: the more information to be stored as limit cycle attractors of the neural network, the more chaos prevails as the background dynamical regime of the network. In fact, the background chaos spreads widely and adopts a very unstructured shape similar to white noise. Next, we introduce a new form of supervised learning that is more plausible from a biological point of view: the network has to learn to react to an external stimulus by cycling through a sequence that is no longer specified a priori. Based on its spontaneous dynamics, the network decides "on its own" the dynamical patterns to be associated with the stimuli. Compared with classical supervised learning, huge enhancements in storing capacity and computational cost have been observed. Moreover, this new form of supervised learning, by being more "respectful" of the network intrinsic dynamics, maintains much more structure in the obtained chaos. It is still possible to observe the traces of the learned attractors in the chaotic regime. This complex but still very informative regime is referred to as "frustrated chaos."
Neural networks do not readily provide an explanation of the knowledge stored in their weights as part of their information processing. Until recently, neural networks were considered to be black boxes, with the knowledge stored in their weights not readily accessible. Since then, research has resulted in a number of algorithms for extracting knowledge in symbolic form from trained neural networks. This article addresses the extraction of knowledge in symbolic form from recurrent neural networks trained to behave like deterministic finite-state automata (DFAs). To date, methods used to extract knowledge from such networks have relied on the hypothesis that networks' states tend to cluster and that clusters of network states correspond to DFA states. The computational complexity of such a cluster analysis has led to heuristics that either limit the number of clusters that may form during training or limit the exploration of the space of hidden recurrent state neurons. These limitations, while necessary, may lead to decreased fidelity, in which the extracted knowledge may not model the true behavior of a trained network, perhaps not even for the training set. The method proposed here uses a polynomial time, symbolic learning algorithm to infer DFAs solely from the observation of a trained network's input-output behavior. Thus, this method has the potential to increase the fidelity of the extracted knowledge.
Selection of machine learning techniques requires a certain sensitivity to the requirements of the problem. In particular, the problem can be made more tractable by deliberately using algorithms that are biased toward solutions of the requisite kind. In this paper, we argue that recurrent neural networks have a natural bias toward a problem domain of which biological sequence analysis tasks are a subset. We use experiments with synthetic data to illustrate this bias. We then demonstrate that this bias can be exploitable using a data set of protein sequences containing several classes of subcellular localization targeting peptides. The results show that, compared with feed forward, recurrent neural networks will generally perform better on sequence analysis tasks. Furthermore, as the patterns within the sequence become more ambiguous, the choice of specific recurrent architecture becomes more critical.
B-cell epitopes play a vital role in the development of peptide vaccines, in diagnosis of diseases, and also for allergy research. Experimental methods used for characterizing epitopes are time consuming and demand large resources. The availability of epitope prediction method(s) can rapidly aid experimenters in simplifying this problem. The standard feed-forward (FNN) and recurrent neural network (RNN) have been used in this study for predicting B-cell epitopes in an antigenic sequence. The networks have been trained and tested on a clean data set, which consists of 700 non-redundant B-cell epitopes obtained from Bcipep database and equal number of non-epitopes obtained randomly from Swiss-Prot database. The networks have been trained and tested at different input window length and hidden units. Maximum accuracy has been obtained using recurrent neural network (Jordan network) with a single hidden layer of 35 hidden units for window length of 16. The final network yields an overall prediction accuracy of 65.93% when tested by fivefold cross-validation. The corresponding sensitivity, specificity, and positive prediction values are 67.14, 64.71, and 65.61%, respectively. It has been observed that RNN (JE) was more successful than FNN in the prediction of B-cell epitopes. The length of the peptide is also important in the prediction of B-cell epitopes from antigenic sequences. The webserver ABCpred is freely available at www.imtech.res.in/raghava/abcpred/.
It has been shown that dynamic recurrent neural networks are successful in identifying the complex mapping relationship between full-wave-rectified electromyographic (EMG) signals and limb trajectories during complex movements. These connectionist models include two types of adaptive parameters: the interconnection weights between the units and the time constants associated to each neuron-like unit; they are governed by continuous-time equations. Due to their internal structure, these models are particularly appropriate to solve dynamical tasks (with time-varying input and output signals). We show in this paper that the introduction of a modular organization dedicated to different aspects of the dynamical mapping including privileged communication channels can refine the architecture of these recurrent networks. We first divide the initial individual network into two communicating subnetworks. These two modules receive the same EMG signals as input but are involved in different identification tasks related to position and acceleration. We then show that the introduction of an artificial distance in the model (using a Gaussian modulation factor of weights) induces a reduced modular architecture based on a self-elimination of null synaptic weights. Moreover, this self-selected reduced model based on two subnetworks performs the identification task better than the original single network while using fewer free parameters (better learning curve and better identification quality). We also show that this modular network exhibits several features that can be considered as biologically plausible after the learning process: self-selection of a specific inhibitory communicating path between both subnetworks after the learning process, appearance of tonic and phasic neurons, and coherent distribution of the values of the time constants within each subnetwork.
This paper investigates the problem of approximating a dynamical system (DS) by a recurrent neural network (RNN) as one extension of the problem of approximating orbits by an RNN. We systematically investigate how an RNN can produce a DS on the visible state space to approximate a given DS and as a first step to the generalization problem for RNNs, we also investigate whether or not a DS produced by some RNN can be identified from several observed orbits of the DS. First, it is proved that RNNs without hidden units uniquely produce a certain class of DS. Next, neural dynamical systems (NDSs) are proposed as DSs produced by RNNs with hidden units. Moreover, affine neural dynamial systems (A-NDSs) are provided as nontrivial examples of NDSs and it is proved that any DS can be finitely approximated by an A-NDS with any precision. We propose an A-NDS as a DS that an RNN can actually produce on the visible state space to approximate the target DS. For the generalization problem of RNNs, a geometric criterion is derived in the case of RNNs without hidden units. This theory is also extended to the case of RNNs with hidden units for learning A-NDSs.
In this paper, we propose a recurrent neural network for solving nonlinear convex programming problems with linear constraints. The proposed neural network has a simpler structure and a lower complexity for implementation than the existing neural networks for solving such problems. It is shown here that the proposed neural network is stable in the sense of Lyapunov and globally convergent to an optimal solution within a finite time under the condition that the objective function is strictly convex. Compared with the existing convergence results, the present results do not require Lipschitz continuity condition on the objective function. Finally, examples are provided to show the applicability of the proposed neural network.
This paper studies the output convergence of a class of recurrent neural networks with time-varying inputs. The model of the studied neural networks has different dynamic structure from that in the well known Hopfield model, it does not contain linear terms. Since different structures of differential equations usually result in quite different dynamic behaviors, the convergence of this model is quite different from that of Hopfield model. This class of neural networks has been found many successful applications in solving some optimization problems. Some sufficient conditions to guarantee output convergence of the networks are derived.
In recent years it has been shown that first order recurrent neural networks trained by gradient-descent can learn not only regular but also simple context-free and context-sensitive languages. However, the success rate was generally low and severe instability issues were encountered. The present study examines the hypothesis that a combination of evolutionary hill climbing with incremental learning and a well-balanced training set enables first order recurrent networks to reliably learn context-free and mildly context-sensitive languages. In particular, we trained the networks to predict symbols in string sequences of the context-sensitive language [a(n)b(n)c(n); n > or = 1. Comparative experiments with and without incremental learning indicated that incremental learning can accelerate and facilitate training. Furthermore, incrementally trained networks generally resulted in monotonic trajectories in hidden unit activation space, while the trajectories of non-incrementally trained networks were oscillating. The non-incrementally trained networks were more likely to generalise.
This paper studies the global output convergence of a class of recurrent neural networks with globally Lipschitz continuous and monotone nondecreasing activation functions and locally Lipschitz continuous time-varying inputs. We establish two sufficient conditions for global output convergence of this class of neural networks. Symmetry in the connection weight matrix is not required in the present results which extend the existing ones.
For the learning of a finite state machine (FSM) by a recurrent neural network (RNN), we think about how to train an RNN so as to stably mimic an FSM even for sequences having a long length. First, we consider the relationship between the stable behavior and the internal representation of states, that is, clusters of the internal units' outputs. As for this relationship, we prove that an RNN can get the stable cluster transitions when a neuron activation parameter is larger than a certain finite value micro0. Secondly, to acquire the stable behavior, we regard the internal representation for the stable behavior as prior knowledge. This produces a new target function of learning with internal representation term. We derive a Bayesian style method to estimate coefficients of the terms in the function, corresponding to hyperparameters. Finally, experiments show that RNNs readily acquire stable behavior by using our proposed method.
In this paper, an adaptive recurrent-neural-network (ARNN) motion control system for a biaxial motion mechanism driven by two field-oriented control permanent magnet synchronous motors (PMSMs) in the computer numerical control (CNC) machine is proposed. In the proposed ARNN control system, a RNN with accurate approximation capability is employed to approximate an unknown dynamic function, and the adaptive learning algorithms that can learn the parameters of the RNN on line are derived using Lyapunov stability theorem. Moreover, a robust controller is proposed to confront the uncertainties including approximation error, optimal parameter vectors, higher-order terms in Taylor series, external disturbances, cross-coupled interference and friction torque of the system. To relax the requirement for the value of lumped uncertainty in the robust controller, an adaptive lumped uncertainty estimation law is investigated. Using the proposed control, the position tracking performance is substantially improved and the robustness to uncertainties including cross-coupled interference and friction torque can be obtained as well. Finally, some experimental results of the tracking of various reference contours demonstrate the validity of the proposed design for practical applications.
The paper presents an adaptive noise canceller (ANC) filter using an artificial neural network for real-time removal of electro-oculogram (EOG) interference from electro-encephalogram (EEG) signals. Conventional ANC filters are based on linear models of interference. Such linear models provide poorer prediction for biomedical signals. In this work, a recurrent neural network was employed for modelling the interference signals. The eye movement and eye blink artifacts were recorded by the placing of an electrode on the forehead above the left eye and an electrode on the left temple. The reference signal was then generated by the data collected from the forehead electrode being added to data recorded from the temple electrode. The reference signal was also contaminated by the EEG. To reduce the EEG interference, the reference signal was first low-pass filtered by a moving averaged filter and then applied to the ANC. Matlab Simulink was used for real-time data acquisition, filtering and ocular artifact suppression. Simulation results show the validity and effectiveness of the technique with different signal-to-noise ratios (SNRs) of the primary signal. On average, a significant improvement in SNR up to 27 dB was achieved with the recurrent neural network. The results from real data demonstrate that the proposed scheme removes ocular artifacts from contaminated EEG signals and is suitable for real-time and short-time EEG recordings.
This paper presents a recurrent neural network for solving strict convex quadratic programming problems and related linear piecewise equations. Compared with the existing neural networks for quadratic program, the proposed neural network has a one-layer structure with a low model complexity. Moreover, the proposed neural network is shown to have a finite-time convergence and exponential convergence. Illustrative examples further show the good performance of the proposed neural network in real-time applications.
This paper studies the multistability of a class of discrete-time recurrent neural networks with unsaturating piecewise linear activation functions. It addresses the nondivergence, global attractivity, and complete stability of the networks. Using the local inhibition, conditions for nondivergence are derived, which not only guarantee nondivergence, but also allow for the existence of multiequilibrium points. Under these nondivergence conditions, global attractive compact sets are obtained. Complete stability is studied via constructing novel energy functions and using the well-known Cauchy Convergence Principle. Examples and simulation results are used to illustrate the theory.