PubMed Health⌕ Search

SEARCH · PubMed Health

Results for “REINFORCEMENT LEARNING”

Explore indexed PubMed citations for clinical trials, systematic reviews and public health research. Read source abstracts and follow each citation to its original PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 recordsLinked to original sources
PubMed2003

Meta-learning in reinforcement learning.

Meta-parameters in reinforcement learning should be tuned to the environmental dynamics and the animal performance. Here, we propose a biologically plausible meta-reinforcement learning algorithm for tuning these meta-parameters in a dynamic, adaptive manner. We tested our algorithm in both a simulation of a Markov decision task and in a non-linear control task. Our results show that the algorithm robustly finds appropriate meta-parameter values, and controls the meta-parameter time course, in both static and dynamic environments. We suggest that the phasic and tonic components of dopamine neuron firing can encode the signal required for meta-learning of reinforcement learning.

Algorithms↗
PubMed1999

A unified analysis of value-function-based reinforcement- learning algorithms.

Reinforcement learning is the problem of generating optimal behavior in a sequential decision-making environment given the opportunity of interacting with it. Many algorithms for solving reinforcement-learning problems work by computing improved estimates of the optimal value function. We extend prior analyses of reinforcement-learning algorithms and present a powerful new theorem that can provide a unified analysis of such value-function-based reinforcement-learning algorithms. The usefulness of the theorem lies in how it allows the convergence of a complex asynchronous reinforcement-learning algorithm to be proved by verifying that a simpler synchronous algorithm converges. We illustrate the application of the theorem by analyzing the convergence of Q-learning, model-based reinforcement learning, Q-learning with multistate updates, Q-learning for Markov games, and risk-sensitive reinforcement learning.

Algorithms↗
PubMed1994

Reinforcement learning control.

Reinforcement learning refers to improving performance through trial-and-error. Despite recent progress in developing artificial learning systems, including new learning methods for artificial neural networks, most of these systems learn under the tutelage of a knowledgeable 'teacher' able to tell them how to respond to a set of training stimuli. Learning under these conditions is not adequate, however, when it is costly, or even impossible, to obtain this kind of training information. Reinforcement learning is attracting increasing attention in computer science and engineering because it can be used by autonomous systems to learn from their experiences instead of from knowledgeable teachers, and it is attracting attention in computational neuroscience because it is consonant with biological principles. Recent research has improved the efficiency of reinforcement learning and has provided some striking examples of its capabilities.

Animals↗
PubMedDate not supplied

Control of exploitation-exploration meta-parameter in reinforcement learning.

In reinforcement learning (RL), the duality between exploitation and exploration has long been an important issue. This paper presents a new method that controls the balance between exploitation and exploration. Our learning scheme is based on model-based RL, in which the Bayes inference with forgetting effect estimates the state-transition probability of the environment. The balance parameter, which corresponds to the randomness in action selection, is controlled based on variation of action results and perception of environmental change. When applied to maze tasks, our method successfully obtains good controls by adapting to environmental changes. Recently, Usher et al. [Science 283 (1999) 549] has suggested that noradrenergic neurons in the locus coeruleus may control the exploitation-exploration balance in a real brain and that the balance may correspond to the level of animal's selective attention. According to this scenario, we also discuss a possible implementation in the brain.

Animals↗
PubMed2004

Combining Hebbian and reinforcement learning in a minibrain model.

A toy model of a neural network in which both Hebbian learning and reinforcement learning occur is studied. The problem of 'path interference', which makes that the neural net quickly forgets previously learned input-output relations is tackled by adding a Hebbian term (proportional to the learning rate nu) to the reinforcement term (proportional to delta) in the learning rule. It is shown that the number of learning steps is reduced considerably if 1/4<nu/delta<1/2, i.e. if the Hebbian term is neither too small nor too large compared to the reinforcement term.

Animals↗
PubMed1999

A reinforcement learning approach to online clustering.

A general technique is proposed for embedding online clustering algorithms based on competitive learning in a reinforcement learning framework. The basic idea is that the clustering system can be viewed as a reinforcement learning system that learns through reinforcements to follow the clustering strategy we wish to implement. In this sense, the reinforcement guided competitive learning (RGCL) algorithm is proposed that constitutes a reinforcement-based adaptation of learning vector quantization (LVQ) with enhanced clustering capabilities. In addition, we suggest extensions of RGCL and LVQ that are characterized by the property of sustained exploration and significantly improve the performance of those algorithms, as indicated by experimental tests on well-known data sets.

Algorithms↗
PubMed2002

Multiple model-based reinforcement learning.

We propose a modular reinforcement learning architecture for nonlinear, nonstationary control tasks, which we call multiple model-based reinforcement learning (MMRL). The basic idea is to decompose a complex task into multiple domains in space and time based on the predictability of the environmental dynamics. The system is composed of multiple modules, each of which consists of a state prediction model and a reinforcement learning controller. The "responsibility signal," which is given by the softmax function of the prediction errors, is used to weight the outputs of multiple modules, as well as to gate the learning of the prediction models and the reinforcement learning controllers. We formulate MMRL for both discrete-time, finite-state case and continuous-time, continuous-state case. The performance of MMRL was demonstrated for discrete case in a nonstationary hunting task in a grid world and for continuous case in a nonlinear, nonstationary control task of swinging up a pendulum with variable physical parameters.

Humans↗
PubMed2003

Convergence of reinforcement learning algorithms and acceleration of learning.

The techniques of reinforcement learning have been gaining increasing popularity recently. However, the question of their convergence rate is still open. We consider the problem of choosing the learning steps alpha(n), and their relation with discount gamma and exploration degree epsilon. Appropriate choices of these parameters may drastically influence the convergence rate of the techniques. From analytical examples, we conjecture optimal values of alpha(n) and then use numerical examples to verify our conjectures.

Journal Article↗
PubMed2003

Inter-module credit assignment in modular reinforcement learning.

Critical issues in modular or hierarchical reinforcement learning (RL) are (i) how to decompose a task into sub-tasks, (ii) how to achieve independence of learning of sub-tasks, and (iii) how to assure optimality of the composite policy for the entire task. The second and last requirements are often under trade-off. We propose a method for propagating the reward for the entire task achievement between modules. This is done in the form of a 'modular reward', which is calculated from the temporal difference of the module gating signal and the value of the succeeding module. We implement modular reward for a multiple model-based reinforcement learning (MMRL) architecture and show its effectiveness in simulations of a pursuit task with hidden states and a continuous-time non-linear control task.

Learning↗
PubMed1996

Multiagent reinforcement learning in the Iterated Prisoner's Dilemma.

Reinforcement learning (RL) is based on the idea that the tendency to produce an action should be strengthened (reinforced) if it produces favorable results, and weakened if it produces unfavorable results. Q-learning is a recent RL algorithm that does not need a model of its environment and can be used on-line. Therefore, it is well suited for use in repeated games against an unknown opponent. Most RL research has been confined to single-agent settings or to multiagent settings where the agents have totally positively correlated payoffs (team problems) or totally negatively correlated payoffs (zero-sum games). This paper is an empirical study of reinforcement learning in the Iterated Prisoner's Dilemma (IPD), where the agents' payoffs are neither totally positively nor totally negatively correlated. RL is considerably more difficult in such a domain. This paper investigates the ability of a variety of Q-learning agents to play the IPD game against an unknown opponent. In some experiments, the opponent is the fixed strategy Tit-For-Tat, while in others it is another Q-learner. All the Q-learners learned to play optimally against Tit-For-Tat. Playing against another learner was more difficult because the adaptation of the other learner created a non-stationary environment, and because the other learner was not endowed with any a priori knowledge about the IPD game such as a policy designed to encourage cooperation. The learners that were studied varied along three dimensions: the length of history they received as context, the type of memory they employed (lookup tables based on restricted history windows or recurrent neural networks that can theoretically store features from arbitrarily deep in the past), and the exploration schedule they followed. Although all the learners faced difficulties when playing against other learners, agents with longer history windows, lookup table memories, and longer exploration schedules fared best in the IPD games.

Algorithms↗
PubMed1999

Multi-agent reinforcement learning: weighting and partitioning.

This article addresses weighting and partitioning, in complex reinforcement learning tasks, with the aim of facilitating learning. The article presents some ideas regarding weighting of multiple agents and extends them into partitioning an input/state space into multiple regions with differential weighting in these regions, to exploit differential characteristics of regions and differential characteristics of agents to reduce the learning complexity of agents (and their function approximators) and thus to facilitate the learning overall. It analyzes, in reinforcement learning tasks, different ways of partitioning a task and using agents selectively based on partitioning. Based on the analysis, some heuristic methods are described and experimentally tested. We find that some off-line heuristic methods perform the best, significantly better than single-agent models.

Journal Article↗
PubMed1979

Aspects of the reinforcer learned in second-order Pavlovian conditioning.

Four experiments used an autoshaping procedure in pigeons to explore learning about the reinforcer in a second-order conditioning paradigm. Experiment 1 conditioned two visual second-order stimuli (S2), using as reinforcers two visual first-order stimuli (S1), each of which had previously been paired with food. Animals for which the S2 stimuli were each consistently paired with one particular S1 developed second-order responding more rapidly than did animals for which the identity of S1 varied from trial to trial. Moreover, following consistent pairings, extinction of an S1 had a depressive effect upon second-order responding which was peculiar to the S2 with which it had been paired. Both results suggest that in this preparation the organism identifies a particular S1 as the reinforcer for each S2. The remaining experiments examined the details of that identification. A compound S1, itself composed of two separable elements, was used to reinforce an S2. Subsequent extinction of either element of S1 led to a depression in the responding to S2, which indicates that both elements were involved in the second-order conditioning. Moreover, the use of several complex discriminations, which produced different behavior to S1 and to its elements, suggested that the organism had associated the S2 with the compound S1 rather than with its separate elements. However, even complete extinction of the response to S1 left some residual behavior to S2, which indicates that a portion of the second-order conditioning is independent of the current state of the reinforcer. These results demonstrate that in some situations the organism associates a conditioned stimulus with a rich representation of the reinforcer.

Animals↗
PubMedDate not supplied

Reinforcement learning with internal expectation in the random neural networks for cascaded decisions.

The reinforcement learning scheme proposed in Halici (J. Biosystems 40 (1997) 83) for the random neural network (RNN) (Neural Computation 1 (1989) 502) is based on reward and performs well for stationary environments. However, when the environment is not stationary it suffers from getting stuck to the previously learned action and extinction is not possible. To overcome the problem, the reinforcement scheme is extended in Halici (Eur. J. Oper. Res., 126(2000) 288) by introducing a new weight update rule (E-rule) which takes into consideration the internal expectation of reinforcement. Although the E-rule is proposed for the RNN, it can be used for training learning automata or other intelligent systems based on reinforcement learning. This paper looks into the behavior of the learning scheme with internal expectation for the environments where the reinforcement is obtained after a sequence of cascaded decisions. The simulation results have shown that the RNN learns well and extinction is possible even for the cases with several decision steps and with hundreds of possible decision paths.

Computer Simulation↗
PubMedDate not supplied

Applications of the self-organising map to reinforcement learning.

This article is concerned with the representation and generalisation of continuous action spaces in reinforcement learning (RL) problems. A model is proposed based on the self-organising map (SOM) of Kohonen [Self Organisation and Associative Memory, 1987] which allows either the one-to-one, many-to-one or one-to-many structure of the desired state-action mapping to be captured. Although presented here for tasks involving immediate reward, the approach is easily extended to delayed reward. We conclude that the SOM is a useful tool for providing real-time, on-line generalisation in RL problems in which the latent dimensionalities of the state and action spaces are small. Scalability issues are also discussed.

Neural Networks, Computer↗
PubMedDate not supplied

Blood Clot: gaming to reinforce learning about disseminated intravascular coagulation.

BACKGROUND: The game, Blood Clot, is an educational tool to reinforce learning following a lecture about disseminated intravascular coagulation (DIC). METHOD: The game was implemented as an educational update for thirty Registered Nurses in a hospital setting. RESULTS: The game reinforced learning of lecture material and stimulated group involvement and communication. CONCLUSION: The self-report evaluations indicate Blood Clot enhanced communication within the group and application of lecture material to solve problems.

Attitude of Health Personnel↗
PubMed2002

The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity.

The authors present a unified account of 2 neural systems concerned with the development and expression of adaptive behaviors: a mesencephalic dopamine system for reinforcement learning and a "generic" error-processing system associated with the anterior cingulate cortex. The existence of the error-processing system has been inferred from the error-related negativity (ERN), a component of the event-related brain potential elicited when human participants commit errors in reaction-time tasks. The authors propose that the ERN is generated when a negative reinforcement learning signal is conveyed to the anterior cingulate cortex via the mesencephalic dopamine system and that this signal is used by the anterior cingulate cortex to modify performance on the task at hand. They provide support for this proposal using both computational modeling and psychophysiological experimentation.

Analysis of Variance↗
PubMed2000

Reinforcement learning in continuous time and space.

This article presents a reinforcement learning framework for continuous-time dynamical systems without a priori discretization of time, state, and action. Based on the Hamilton-Jacobi-Bellman (HJB) equation for infinite-horizon, discounted reward problems, we derive algorithms for estimating value functions and improving policies with the use of function approximators. The process of value function estimation is formulated as the minimization of a continuous-time form of the temporal difference (TD) error. Update methods based on backward Euler approximation and exponential eligibility traces are derived, and their correspondences with the conventional residual gradient, TD(0), and TD(lambda) algorithms are shown. For policy improvement, two methods-a continuous actor-critic method and a value-gradient-based greedy policy-are formulated. As a special case of the latter, a nonlinear feedback control law using the value gradient and the model of the input gain is derived. The advantage updating, a model-free algorithm derived previously, is also formulated in the HJB-based framework. The performance of the proposed algorithms is first tested in a nonlinear control task of swinging a pendulum up with limited torque. It is shown in the simulations that (1) the task is accomplished by the continuous actor-critic method in a number of trials several times fewer than by the conventional discrete actor-critic method; (2) among the continuous policy update methods, the value-gradient-based policy with a known or learned dynamic model performs several times better than the actor-critic method; and (3) a value function update using exponential eligibility traces is more efficient and stable than that based on Euler approximation. The algorithms are then tested in a higher-dimensional task: cart-pole swing-up. This task is accomplished in several hundred trials using the value-gradient-based policy with a learned dynamic model.

Algorithms↗