SEARCH · PubMed Health

Results for “REINFORCEMENT LEARNING”

Explore indexed PubMed citations for clinical trials, systematic reviews and public health research. Read source abstracts and follow each citation to its original PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2Linked to original sources

PubMed1999

Multi-agent reinforcement learning: weighting and partitioning.

This article addresses weighting and partitioning, in complex reinforcement learning tasks, with the aim of facilitating learning. The article presents some ideas regarding weighting of multiple agents and extends them into partitioning an input/state space into multiple regions with differential weighting in these regions, to exploit differential characteristics of regions and differential characteristics of agents to reduce the learning complexity of agents (and their function approximators) and thus to facilitate the learning overall. It analyzes, in reinforcement learning tasks, different ways of partitioning a task and using agents selectively based on partitioning. Based on the analysis, some heuristic methods are described and experimentally tested. We find that some off-line heuristic methods perform the best, significantly better than single-agent models.

Journal Article↗

PubMed2004

A new criterion using information gain for action selection strategy in reinforcement learning.

In this paper, we regard the sequence of returns as outputs from a parametric compound source. Utilizing the fact that the coding rate of the source shows the amount of information about the return, we describe l-learning algorithms based on the predictive coding idea for estimating an expected information gain concerning future information and give a convergence proof of the information gain. Using the information gain, we propose the ratio w of return loss to information gain as a new criterion to be used in probabilistic action-selection strategies. In experimental results, we found that our w-based strategy performs well compared with the conventional Q-based strategy.

Algorithms↗

PubMed2007

Using the XCS classifier system for multi-objective reinforcement learning problems.

We investigate the performance of a learning classifier system in some simple multi-objective, multi-step maze problems, using both random and biased action-selection policies for exploration. Results show that the choice of action-selection policy can significantly affect the performance of the system in such environments. Further, this effect is directly related to population size, and we relate this finding to recent theoretical studies of learning classifier systems in single-step problems.

Artificial Intelligence↗

PubMed1979

Aspects of the reinforcer learned in second-order Pavlovian conditioning.

Four experiments used an autoshaping procedure in pigeons to explore learning about the reinforcer in a second-order conditioning paradigm. Experiment 1 conditioned two visual second-order stimuli (S2), using as reinforcers two visual first-order stimuli (S1), each of which had previously been paired with food. Animals for which the S2 stimuli were each consistently paired with one particular S1 developed second-order responding more rapidly than did animals for which the identity of S1 varied from trial to trial. Moreover, following consistent pairings, extinction of an S1 had a depressive effect upon second-order responding which was peculiar to the S2 with which it had been paired. Both results suggest that in this preparation the organism identifies a particular S1 as the reinforcer for each S2. The remaining experiments examined the details of that identification. A compound S1, itself composed of two separable elements, was used to reinforce an S2. Subsequent extinction of either element of S1 led to a depression in the responding to S2, which indicates that both elements were involved in the second-order conditioning. Moreover, the use of several complex discriminations, which produced different behavior to S1 and to its elements, suggested that the organism had associated the S2 with the compound S1 rather than with its separate elements. However, even complete extinction of the response to S1 left some residual behavior to S2, which indicates that a portion of the second-order conditioning is independent of the current state of the reinforcer. These results demonstrate that in some situations the organism associates a conditioned stimulus with a rich representation of the reinforcer.

Animals↗

PubMed2007

Reinforcement learning signals predict future decisions.

Optimal behavior in a competitive world requires the flexibility to adapt decision strategies based on recent outcomes. In the present study, we tested the hypothesis that this flexibility emerges through a reinforcement learning process, in which reward prediction errors are used dynamically to adjust representations of decision options. We recorded event-related brain potentials (ERPs) while subjects played a strategic economic game against a computer opponent to evaluate how neural responses to outcomes related to subsequent decision-making. Analyses of ERP data focused on the feedback-related negativity (FRN), an outcome-locked potential thought to reflect a neural prediction error signal. Consistent with predictions of a computational reinforcement learning model, we found that the magnitude of ERPs after losing to the computer opponent predicted whether subjects would change decision behavior on the subsequent trial. Furthermore, FRNs to decision outcomes were disproportionately larger over the motor cortex contralateral to the response hand that was used to make the decision. These findings provide novel evidence that humans engage a reinforcement learning process to adjust representations of competing decision options.

Adult↗

PubMedDate not supplied

Reinforcement learning with internal expectation in the random neural networks for cascaded decisions.

The reinforcement learning scheme proposed in Halici (J. Biosystems 40 (1997) 83) for the random neural network (RNN) (Neural Computation 1 (1989) 502) is based on reward and performs well for stationary environments. However, when the environment is not stationary it suffers from getting stuck to the previously learned action and extinction is not possible. To overcome the problem, the reinforcement scheme is extended in Halici (Eur. J. Oper. Res., 126(2000) 288) by introducing a new weight update rule (E-rule) which takes into consideration the internal expectation of reinforcement. Although the E-rule is proposed for the RNN, it can be used for training learning automata or other intelligent systems based on reinforcement learning. This paper looks into the behavior of the learning scheme with internal expectation for the environments where the reinforcement is obtained after a sequence of cascaded decisions. The simulation results have shown that the RNN learns well and extinction is possible even for the cases with several decision steps and with hundreds of possible decision paths.

Computer Simulation↗

PubMed2004

Reinforcement learning with via-point representation.

In this paper, we propose a new learning framework for motor control. This framework consists of two components: reinforcement learning and via-point representation. In the field of motor control, conventional reinforcement learning has been used to acquire control sequences such as cart-pole or stand-up robot control. Recently, researchers have become interested in hierarchical architecture, such as multiple levels, and multiple temporal and spatial scales. Our new framework contains two levels of hierarchical architecture. The higher level is implemented using via-point representation, which corresponds to macro-actions or multiple time scales. The lower level is implemented using a trajectory generator that produces primitive actions. Our framework can modify the ongoing movement by means of temporally localized via-points and trajectory generation. Successful results are obtained in computer simulation of the cart-pole swing up task.

Computer Simulation↗

PubMedDate not supplied

Applications of the self-organising map to reinforcement learning.

This article is concerned with the representation and generalisation of continuous action spaces in reinforcement learning (RL) problems. A model is proposed based on the self-organising map (SOM) of Kohonen [Self Organisation and Associative Memory, 1987] which allows either the one-to-one, many-to-one or one-to-many structure of the desired state-action mapping to be captured. Although presented here for tasks involving immediate reward, the approach is easily extended to delayed reward. We conclude that the SOM is a useful tool for providing real-time, on-line generalisation in RL problems in which the latent dimensionalities of the state and action spaces are small. Scalability issues are also discussed.

Neural Networks, Computer↗

PubMedDate not supplied

Blood Clot: gaming to reinforce learning about disseminated intravascular coagulation.

BACKGROUND: The game, Blood Clot, is an educational tool to reinforce learning following a lecture about disseminated intravascular coagulation (DIC). METHOD: The game was implemented as an educational update for thirty Registered Nurses in a hospital setting. RESULTS: The game reinforced learning of lecture material and stimulated group involvement and communication. CONCLUSION: The self-report evaluations indicate Blood Clot enhanced communication within the group and application of lecture material to solve problems.

Attitude of Health Personnel↗

PubMed2002

The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity.

The authors present a unified account of 2 neural systems concerned with the development and expression of adaptive behaviors: a mesencephalic dopamine system for reinforcement learning and a "generic" error-processing system associated with the anterior cingulate cortex. The existence of the error-processing system has been inferred from the error-related negativity (ERN), a component of the event-related brain potential elicited when human participants commit errors in reaction-time tasks. The authors propose that the ERN is generated when a negative reinforcement learning signal is conveyed to the anterior cingulate cortex via the mesencephalic dopamine system and that this signal is used by the anterior cingulate cortex to modify performance on the task at hand. They provide support for this proposal using both computational modeling and psychophysiological experimentation.

Analysis of Variance↗

PubMed2004

The role of multisensor data fusion in neuromuscular control of a sagittal arm with a pair of muscles using actor-critic reinforcement learning method.

In this study, we consider the role of multisensor data fusion in neuromuscular control using an actor-critic reinforcement learning method. The model we use is a single link system actuated by a pair of muscles that are excited with alpha and gamma signals. Various physiological sensor information such as proprioception, spindle sensors, and Golgi tendon organs have been integrated to achieve an oscillatory movement with variable amplitude and frequency, while achieving a stable movement with minimum metabolic cost and coactivation. The system is highly nonlinear in all its physical and physiological attributes. Transmission delays are included in the afferent and efferent neural paths to account for a more accurate representation of the reflex loops. This paper proposes a reinforcement learning method with an Actor-Critic architecture instead of middle and low level of central nervous system (CNS). The Actor in this structure is a two layer feedforward neural network and the Critic is a model of the cerebellum. The Critic is trained by the State-Action-Reward-State-Action (SARSA) method. The Critic will train the Actor by supervisory learning based on previous experiences. The reinforcement signal in SARSA is evaluated based on available alternatives concerning the concept of multisensor data fusion. The effectiveness and the biological plausibility of the present model are demonstrated by several simulations. The system showed excellent tracking capability when we integrated the available sensor information. Addition of a penalty for activation of muscles resulted in much lower muscle coactivation while keeping the movement stable.

Algorithms↗

PubMed2004

A parameter control method in reinforcement learning to rapidly follow unexpected environmental changes.

In order to rapidly follow unexpected environmental changes, we propose a parameter control method in reinforcement learning that changes each of learning parameters in appropriate directions. We determine each appropriate direction on the basis of relationships between behaviors and neuromodulators by considering an emergency as a key word. Computer experiments show that the agents using our proposed method could rapidly respond to unexpected environmental changes, not depending on either two reinforcement learning algorithms (Q-learning and actor-critic (AC) architecture) or two learning problems (discontinuous and continuous state-action problems).

Adaptation, Physiological↗

PubMed2000

Reinforcement learning in continuous time and space.

This article presents a reinforcement learning framework for continuous-time dynamical systems without a priori discretization of time, state, and action. Based on the Hamilton-Jacobi-Bellman (HJB) equation for infinite-horizon, discounted reward problems, we derive algorithms for estimating value functions and improving policies with the use of function approximators. The process of value function estimation is formulated as the minimization of a continuous-time form of the temporal difference (TD) error. Update methods based on backward Euler approximation and exponential eligibility traces are derived, and their correspondences with the conventional residual gradient, TD(0), and TD(lambda) algorithms are shown. For policy improvement, two methods-a continuous actor-critic method and a value-gradient-based greedy policy-are formulated. As a special case of the latter, a nonlinear feedback control law using the value gradient and the model of the input gain is derived. The advantage updating, a model-free algorithm derived previously, is also formulated in the HJB-based framework. The performance of the proposed algorithms is first tested in a nonlinear control task of swinging a pendulum up with limited torque. It is shown in the simulations that (1) the task is accomplished by the continuous actor-critic method in a number of trials several times fewer than by the conventional discrete actor-critic method; (2) among the continuous policy update methods, the value-gradient-based policy with a known or learned dynamic model performs several times better than the actor-critic method; and (3) a value function update using exponential eligibility traces is more efficient and stable than that based on Euler approximation. The algorithms are then tested in a higher-dimensional task: cart-pole swing-up. This task is accomplished in several hundred trials using the value-gradient-based policy with a learned dynamic model.

Algorithms↗

PubMed1997

Response-reinforcement learning is dependent on N-methyl-D-aspartate receptor activation in the nucleus accumbens core.

The nucleus accumbens, a site within the ventral striatum, is best known for its prominent role in mediating the reinforcing effects of drugs of abuse such as cocaine, alcohol, and nicotine. Indeed, it is generally believed that this structure subserves motivated behaviors, such as feeding, drinking, sexual behavior, and exploratory locomotion, which are elicited by natural rewards or incentive stimuli. A basic rule of positive reinforcement is that motor responses will increase in magnitude and vigor if followed by a rewarding event. It is likely, therefore, that the nucleus accumbens may serve as a substrate for reinforcement learning. However, there is surprisingly little information concerning the neural mechanisms by which appetitive responses are learned. In the present study, we report that treatment of the nucleus accumbens core with the selective competitive N-methyl-D-aspartate (NMDA) antagonist 2-amino-5-phosphonopentanoic acid (AP-5; 5 nmol/0.5 microl bilaterally) impairs response-reinforcement learning in the acquisition of a simple lever-press task to obtain food. Once the rats learned the task, AP-5 had no effect, demonstrating the requirement of NMDA receptor-dependent plasticity in the early stages of learning. Infusion of AP-5 into the accumbens shell produced a much smaller impairment of learning. Additional experiments showed that AP-5 core-treated rats had normal feeding and locomotor responses and were capable of acquiring stimulus-reward associations. We hypothesize that stimulation of NMDA receptors within the accumbens core is a key process through which motor responses become established in response to reinforcing stimuli. Further, this mechanism, may also play a critical role in the motivational and addictive properties of drugs of abuse.

2-Amino-5-phosphonovalerate↗

PubMed1999

Optimal chaos control through reinforcement learning.

A general purpose chaos control algorithm based on reinforcement learning is introduced and applied to the stabilization of unstable periodic orbits in various chaotic systems and to the targeting problem. The algorithm does not require any information about the dynamical system nor about the location of periodic orbits. Numerical tests demonstrate good and fast performance under noisy and nonstationary conditions. (c) 1999 American Institute of Physics.

Journal Article↗

PubMed2006

The role of the basal ganglia in exploration in a neural model based on reinforcement learning.

We present a computational model of basal ganglia as a key player in exploratory behavior. The model describes exploration of a virtual rat in a simulated water pool experiment. The virtual rat is trained using a reward-based or reinforcement learning paradigm which requires units with stochastic behavior for exploration of the system's state space. We model the Subthalamic Nucleus-Globus Pallidus externa (STN-GPe) segment of the basal ganglia as a pair of neuronal layers with oscillatory dynamics, exhibiting a variety of dynamic regimes such as chaos, traveling waves and clustering. Invoking the property of chaotic systems to explore state-space, we suggest that the complex exploratory dynamics of STN-GPe system in conjunction with dopamine-based reward signaling from the Substantia Nigra pars compacta (SNc) present the two key ingredients of a reinforcement learning system.

Animals↗

PubMed2005

Reinforcement learning for a stochastic automaton modelling predation in stationary model-mimic environments.

In this paper we propose a mathematical learning model for the feeding behaviour of a specialist predator operating in a random environment occupied by two types of prey, palatable mimics and unpalatable models, and a generalist predator with additional alternative prey at its disposal. A well known linear reinforcement learning algorithm and its special cases are considered for updating the probabilities of the two actions, eat prey or ignore prey. Each action elicits a probabilistic response from the environment that can be favorable or unfavourable. To assess the performance of the predator a payoff function is constructed that captures the energetic benefit from consuming acceptable prey, the energetic cost from consuming unacceptable prey, and lost benefit from ignoring acceptable prey. Conditions for an improving predator payoff are also explicitly formulated.

Algorithms↗

PubMed2004

Reinforcement learning and decision making in monkeys during a competitive game.

Animals living in a dynamic environment must adjust their decision-making strategies through experience. To gain insights into the neural basis of such adaptive decision-making processes, we trained monkeys to play a competitive game against a computer in an oculomotor free-choice task. The animal selected one of two visual targets in each trial and was rewarded only when it selected the same target as the computer opponent. To determine how the animal's decision-making strategy can be affected by the opponent's strategy, the computer opponent was programmed with three different algorithms that exploited different aspects of the animal's choice and reward history. When the computer selected its targets randomly with equal probabilities, animals selected one of the targets more often, violating the prediction of probability matching, and their choices were systematically influenced by the choice history of the two players. When the computer exploited only the animal's choice history but not its reward history, animal's choice became more independent of its own choice history but was still related to the choice history of the opponent. This bias was substantially reduced, but not completely eliminated, when the computer used the choice history of both players in making its predictions. These biases were consistent with the predictions of reinforcement learning, suggesting that the animals sought optimal decision-making strategies using reinforcement learning algorithms.

Algorithms↗