SEARCH · PubMed Health

Results for “recurrent neural network”

Explore indexed PubMed citations for clinical trials, systematic reviews and public health research. Read source abstracts and follow each citation to its original PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 recordsLinked to original sources

PubMed2006

[Simultaneous quantitative analysis of three kinds of phenols by Elman recurrent neural network].

Elman recurrent neural network ( ERNN) was applied to study the simultaneous quantitative analysis of seriously overlapped spectra of a p-nitrophenol, o-nitrophenol and 2,4-dinitrophenol system. The multivariate linear regression (MLR) method was also applied in this study for comparison. Two programs (PERNN and PMLR) were designed to perform the calculations. By optimization, the structure and parameters of Elman recurrent neural network were defined. The relative standard errors of prediction (RSEP) for all components with ERNN and MLR were 3.1% and 2 027.3%, respectively. Experimental results showed the method to be successful even where there was a severe overlap of spectra. The ERNN method is a valuable tool in solving the minimum problem and improving the convergence rate, and can be used to analyze the whole spectra rather than just picking out a few characteristic values. The method provides a new way of simultaneous determination of severely overlapped molecular spectra without prior separation.

English Abstract↗

PubMed1993

Protein secondary structure prediction with partially recurrent neural networks.

Partially recurrent neural networks with different topologies are applied for secondary structure prediction of proteins. The state of some activations in the network is available after a pattern presentation via feedback connections as additional input during the processing of the next pattern in a sequence. A reference data set containing 91 proteins in the training set and 15 non-homologous proteins in the test set is used for training and testing a network with a modified, hierarchical Elman architecture. The network predicts the secondary structures alpha-helix, beta-sheet, and "coil" for each amino acid. The percentage of correctly classified amino acids is 67.83% on the training set and 63.98% on the test set. The best performance of a three-layer feedforward network is 62.7% on the same test set. A cascaded network, where the outputs of the recurrent network are processed by a second net with 13 x 3 inputs, four hidden and three output units has a predictive performance of 64.49%. The best corresponding feedforward net has a performance of 64.3%.

Amino Acid Sequence↗

PubMed2001

A real-coded genetic algorithm for training recurrent neural networks.

The use of Recurrent Neural Networks is not as extensive as Feedforward Neural Networks. Training algorithms for Recurrent Neural Networks, based on the error gradient, are very unstable in their search for a minimum and require much computational time when the number of neurons is high. The problems surrounding the application of these methods have driven us to develop new training tools. In this paper, we present a Real-Coded Genetic Algorithm that uses the appropriate operators for this encoding type to train Recurrent Neural Networks. We describe the algorithm and we also experimentally compare our Genetic Algorithm with the Real-Time Recurrent Learning algorithm to perform the fuzzy grammatical inference.

Algorithms↗

PubMedDate not supplied

Recurrent neural networks of integrate-and-fire cells simulating short-term memory and wrist movement tasks derived from continuous dynamic networks.

Dynamic recurrent neural networks composed of units with continuous activation functions provide a powerful tool for simulating a wide range of behaviors, since the requisite interconnections can be readily derived by gradient descent methods. However, it is not clear whether more realistic integrate-and-fire cells with comparable connection weights would perform the same functions. We therefore investigated methods to convert dynamic recurrent neural networks of continuous units into networks with integrate-and-fire cells. The transforms were tested on two recurrent networks derived by backpropagation. The first simulates a short-term memory task with units that mimic neural activity observed in cortex of monkeys performing instructed delay tasks. The network utilizes recurrent connections to generate sustained activity that codes the remembered value of a transient cue. The second network simulates patterns of neural activity observed in monkeys performing a step-tracking task with flexion/extension wrist movements. This more complicated network provides a working model of the interactions between multiple spinal and supraspinal centers controlling motoneurons. Our conversion algorithm replaced each continuous unit with multiple integrate-and-fire cells that interact through delayed "synaptic potentials". Successful transformation depends on obtaining an appropriate fit between the activation function of the continuous units and the input-output relation of the spiking cells. This fit can be achieved by adapting the parameters of the synaptic potentials to replicate the input-output behavior of a standard sigmoidal activation function (shown for the short-term memory network). Alternatively, a customized activation function can be derived from the input-output relation of the spiking cells for a chosen set of parameters (demonstrated for the wrist flexion/extension network). In both cases the resulting networks of spiking cells exhibited activity that replicated the activity of corresponding continuous units. This confirms that the network solutions obtained through backpropagation apply to spiking networks and provides a useful method for deriving recurrent spiking networks performing a wide range of functions.

Animals↗

PubMed2005

Simultaneous perturbation learning rule for recurrent neural networks and its FPGA implementation.

Recurrent neural networks have interesting properties and can handle dynamic information processing unlike ordinary feedforward neural networks. However, they are generally difficult to use because there is no convenient learning scheme. In this paper, a recursive learning scheme for recurrent neural networks using the simultaneous perturbation method is described. The detailed procedure of the scheme for recurrent neural networks is explained. Unlike ordinary correlation learning, this method is applicable to analog learning and the learning of oscillatory solutions of recurrent neural networks. Moreover, as a typical example of recurrent neural networks, we consider the hardware implementation of Hopfield neural networks using a field-programmable gate array (FPGA). The details of the implementation are described. Two examples of a Hopfield neural network system for analog and oscillatory targets are shown. These results show that the learning scheme proposed here is feasible.

Algorithms↗

PubMed2006

Homeostatic plasticity improves signal propagation in continuous-time recurrent neural networks.

Continuous-time recurrent neural networks (CTRNNs) are potentially an excellent substrate for the generation of adaptive behaviour in artificial autonomous agents. However, node saturation effects in these networks can leave them insensitive to input and stop signals from propagating. Node saturation is related to the problems of hyper-excitation and quiescence in biological nervous systems, which are thought to be avoided through the existence of homeostatic plastic mechanisms. Analogous mechanisms are here implemented in a variety of CTRNN architectures and are shown to increase node sensitivity and improve signal propagation, with implications for robotics. These results lend support to the view that homeostatic plasticity may prevent quiescence and hyper-excitation in biological nervous systems.

Homeostasis↗

PubMed1997

Efficient Training of Recurrent Neural Network with Time Delays.

Training recurrent neural networks to perform certain tasks is known to be difficult. The possibility of adding synaptic delays to the network properties makes the training task more difficult. However, the disadvantage of tough training procedure is diminished by the improved network performance. During our research of training neural networks with time delays we encountered a robust method for accomplishing the training task. The method is based on adaptive simulated annealing algorithm (ASA) which was found to be superior to other training algorithms. It requires no tuning and is fast enough to enable training to be held on low end platforms such as personal computers. The implementation of the algorithm is presented over a set of typical benchmark tests of training recurrent neural networks with time delays. Copyright 1996 Elsevier Science Ltd.

Journal Article↗

PubMed2006

Prediction of C alpha-H...O and C alpha-H...pi interactions in proteins using recurrent neural network.

In this study, an attempt has been made to develop a method for predicting weak hydrogen bonding interactions, namely, C alpha-H...O and C alpha-H...pi interactions in proteins using artificial neural network. Both standard feed-forward neural network (FNN) and recurrent neural networks (RNN) have been trained and tested using five-fold cross-validation on a non-homologous dataset of 2298 protein chains where no pair of sequences has more than 25% sequence identity. It has been found that the prediction accuracy varies with the separation distance between donor and acceptor residues. The maximum sensitivity achieved with RNN for C alpha-H...O is 51.2% when donor and acceptor residues are four residues apart (i.e. at delta D-A = 4) and for C alpha-H...pi is 82.1% at delta D-A = 3. The performance of RNN is increased by 1-3% for both types of interactions when PSIPRED predicted protein secondary structure is used. Overall, RNN performs better than feed-forward networks at all separation distances between donor-acceptor pair for both types of interactions. Based on the observations, a web server CHpredict (available at http://www.imtech.res.in/raghava/chpredict/) has been developed for predicting donor and acceptor residues in C alpha-H...O and C alpha-H...pi interactions in proteins.

Algorithms↗

PubMed1998

How embedded memory in recurrent neural network architectures helps learning long-term temporal dependencies.

Learning long-term temporal dependencies with recurrent neural networks can be a difficult problem. It has recently been shown that a class of recurrent neural networks called NARX networks perform much better than conventional recurrent neural networks for learning certain simple long-term dependency problems. The intuitive explanation for this behavior is that the output memories of a NARX network can be manifested as jump-ahead connections in the time-unfolded network. These jump-ahead connections can propagate gradient information more efficiently, thus reducing the sensitivity of the network to long-term dependencies. This work gives empirical justification to our hypothesis that similar improvements in learning long-term dependencies can be achieved with other classes of recurrent neural network axchitectures simply by increasing the order of the embedded memory. In particular we explore the impact of learning simple long-term dependency problems on three classes of recurrent neural network architectures: globally recurrent networks, locally recurrent networks, and NARX (output feedback) networks.Comparing the performance of these architectures with different orders of embedded memory on two simple long-term dependencies problems shows that all of these classes of network architectures demonstrate significant improvement on learning long-term dependencies when the orders of embedded memory are increased. These results can be important to a user comfortable with a specific recurrent neural network architecture because simply increasing the embedding memory order of that architecture will make it more robust to the problem of long-term dependency learning.

Journal Article↗

PubMed2001

The simultaneous recurrent neural network for addressing the scaling problem in static optimization.

A trainable recurrent neural network, Simultaneous Recurrent Neural network, is proposed to address the scaling problem faced by neural network algorithms in static optimization. The proposed algorithm derives its computational power to address the scaling problem through its ability to "learn" compared to existing recurrent neural algorithms, which are not trainable. Recurrent backpropagation algorithm is employed to train the recurrent, relaxation-based neural network in order to associate fixed points of the network dynamics with locally optimal solutions of the static optimization problems. Performance of the algorithm is tested on the NP-hard Traveling Salesman Problem in the range of 100 to 600 cities. Simulation results indicate that the proposed algorithm is able to consistently locate high-quality solutions for all problem sizes tested. In other words, the proposed algorithm scales demonstrably well with the problem size with respect to quality of solutions and at the expense of increased computational cost for large problem sizes.

Algorithms↗

PubMed1998

A multilayer recurrent neural network for solving continuous-time algebraic Riccati equations.

A multilayer recurrent neural network is proposed for solving continuous-time algebraic matrix Riccati equations in real time. The proposed recurrent neural network consists of four bidirectionally connected layers. Each layer consists of an array of neurons. The proposed recurrent neural network is shown to be capable of solving algebraic Riccati equations and synthesizing linear-quadratic control systems in real time. Analytical results on stability of the recurrent neural network and solvability of algebraic Riccati equations by use of the recurrent neural network are discussed. The operating characteristics of the recurrent neural network are also demonstrated through three illustrative examples.

Journal Article↗

PubMed2004

Software reliability prediction using recurrent neural network with Bayesian regularization.

A recurrent neural network modeling approach for software reliability prediction with respect to cumulative failure time is proposed. Our proposed network structure has the capability of learning and recognizing the inherent internal temporal property of cumulative failure time sequence. Further, by adding a penalty term of sum of network connection weights, Bayesian regularization is applied to our network training scheme to improve the generalization capability and lower the susceptibility of overfitting. The performance of our proposed approach has been tested using four real-time control and flight dynamic application data sets. Numerical results show that our proposed approach is robust across different software projects, and has a better performance with respect to both goodness-of-fit and next-step-predictability compared to existing neural network models for failure time prediction.

Algorithms↗

PubMedDate not supplied

Training simultaneous recurrent neural network with resilient propagation for static optimization.

This paper proposes a non-recurrent training algorithm, resilient propagation, for the Simultaneous Recurrent Neural network operating in relaxation-mode for computing high quality solutions of static optimization problems. Implementation details related to adaptation of the recurrent neural network weights through the non-recurrent training algorithm, resilient backpropagation, are formulated through an algebraic approach. Performance of the proposed neuro-optimizer on a well-known static combinatorial optimization problem, the Traveling Salesman Problem, is evaluated on the basis of computational complexity measures and, subsequently, compared to performance of the Simultaneous Recurrent Neural network trained with the standard backpropagation, and recurrent backpropagation for the same static optimization problem. Simulation results indicate that the Simultaneous Recurrent Neural network trained with the resilient backpropagation algorithm is able to locate superior quality solutions through comparable amount of computational effort for the Traveling Salesman Problem.

Algorithms↗

PubMed1996

The dynamics of discrete-time computation, with application to recurrent neural networks and finite state machine extraction.

Recurrent neural networks (RNNs) can learn to perform finite state computations. It is shown that an RNN performing a finite state computation must organize its state space to mimic the states in the minimal deterministic finite state machine that can perform that computation, and a precise description of the attractor structure of such systems is given. This knowledge effectively predicts activation space dynamics, which allows one to understand RNN computation dynamics in spite of complexity in activation dynamics. This theory provides a theoretical framework for understanding finite state machine (FSM) extraction techniques and can be used to improve training methods for RNNs performing FSM computations. This provides an example of a successful approach to understanding a general class of complex systems that has not been explicitly designed, e.g., systems that have evolved or learned their internal structure.

Neural Networks, Computer↗

PubMed1995

Dynamical recurrent neural networks--towards environmental time series prediction.

Dynamical Recurrent Neural Networks (DRNN) (Aussem 1995a) are a class of fully recurrent networks obtained by modeling synapses as autoregressive filters. By virtue of their internal dynamic, these networks approximate the underlying law governing the time series by a system of nonlinear difference equations of internal variables. They therefore provide history-sensitive forecasts without having to be explicitly fed with external memory. The model is trained by a local and recursive error propagation algorithm called temporal-recurrent-backpropagation. The efficiency of the procedure benefits from the exponential decay of the gradient terms backpropagated through the adjoint network. We assess the predictive ability of the DRNN model with meterological and astronomical time series recorded around the candidate observation sites for the future VLT telescope. The hope is that reliable environmental forecasts provided with the model will allow the modern telescopes to be preset, a few hours in advance, in the most suited instrumental mode. In this perspective, the model is first appraised on precipitation measurements with traditional nonlinear AR and ARMA techniques using feedforward networks. Then we tackle a complex problem, namely the prediction of astronomical seeing, known to be a very erratic time series. A fuzzy coding approach is used to reduce the complexity of the underlying laws governing the seeing. Then, a fuzzy correspondence analysis is carried out to explore the internal relationships in the data. Based on a carefully selected set of meteorological variables at the same time-point, a nonlinear multiple regression, termed nowcasting (Murtagh et al. 1993, 1995), is carried out on the fuzzily coded seeing records. The DRNN is shown to outperform the fuzzy k-nearest neighbors method.

Environmental Monitoring↗

PubMed2002

Sufficient conditions for error backflow convergence in dynamical recurrent neural networks.

This article extends previous analysis of the gradient decay to a class of discrete-time fully recurrent networks, called dynamical recurrent neural networks, obtained by modeling synapses as finite impulse response (FIR) filters instead of multiplicative scalars. Using elementary matrix manipulations, we provide an upper bound on the norm of the weight matrix, ensuring that the gradient vector, when propagated in a reverse manner in time through the error-propagation network, decays exponentially to zero. This bound applies to all recurrent FIR architecture proposals, as well as fixed-point recurrent networks, regardless of delay and connectivity. In addition, we show that the computational overhead of the learning algorithm can be reduced drastically by taking advantage of the exponential decay of the gradient.

Journal Article↗

PubMed2006

A stable learning algorithm for block-diagonal recurrent neural networks: application to the analysis of lung sounds.

A novel learning algorithm, the Recurrent Neural Network Constrained Optimization Method (RENNCOM) is suggested in this paper, for training block-diagonal recurrent neural networks. The training task is formulated as a constrained optimization problem, whose objective is twofold: (1) minimization of an error measure, leading to successful approximation of the input/output mapping and (2) optimization of an additional functional, the payoff function, which aims at ensuring network stability throughout the learning process. Having assured the network and training stability conditions, the payoff function is switched to an alternative form with the scope to accelerate learning. Simulation results on a benchmark identification problem demonstrate that, compared to other learning schemes with stabilizing attributes, the RENNCOM algorithm has enhanced qualities, including, improved speed of convergence, accuracy and robustness. The proposed algorithm is also applied to the problem of the analysis of lung sounds. Particularly, a filter based on block-diagonal recurrent neural networks is developed, trained with the RENNCOM method. Extensive experimental results are given and performance comparisons with a series of other models are conducted, underlining the effectiveness of the proposed filter.

Algorithms↗

PubMedDate not supplied

On a recurrent neural network producing oscillations.

A recurrent two-node neural network producing oscillations is analyzed. The network has no true inputs and the outputs from the network exhibit a circular phase portrait. The weight configuration of the network is investigated, resulting in analytical weight expressions, which are compared with numerical weight estimates obtained by training the network on the desired trajectories. The values predicted by the analytical expressions agree well with the findings from the numerical study, and can also explain the asymptotic properties of the networks studied.

Algorithms↗