Hierarchical Reinforcement Learning: a Hybrid Approach

Proceedings of the International Conference on Automated Planning and Scheduling

Reinforcement learning (RL) agents seek to maximize the cumulative reward obtained when interacting with their environment. Users define tasks or goals for RL agents by designing specialized reward functions such that maximization aligns with task satisfaction. This work explores the use of high-level symbolic action models as a framework for defining final-state goal tasks and automatically producing their corresponding reward functions. We also show how automated planning can be used to synthesize high-level plans that can guide hierarchical RL (HRL) techniques towards efficiently learning adequate policies. We provide a formal characterization of taskable RL environments and describe sufficient conditions that guarantee we can satisfy various notions of optimality (e.g., minimize total cost, maximize probability of reaching the goal). In addition, we do an empirical evaluation that shows that our approach converges to near-optimal solutions faster than standard RL and HRL methods...

Proceedings of the second international joint conference on Autonomous agents and multiagent systems - AAMAS '03, 2003

The agent approach, as seen by , intends to design "intelligent" behaviors. Yet, Reinforcement Learning (RL) methods often fail when confronted with complex tasks. We are therefore trying to develop a methodology for the automated design of agents (in the framework of Markov Decision Processes) in the case where the global task can be decomposed into simpler -possibly concurrentsub-tasks. Our main idea is to automatically combine basic behaviors using RL methods. This led us to propose two complementary mechanisms presented in the current paper. The first mechanism builds a global policy using a weighted combination of basic policies (which are reusable), the weights being learned by the agent (using Simulated Annealing in our case). An agent designed this way is highly scalable as, without further refinement of the global behavior, it can automatically combine several instances of the same basic behavior to take into account concurrent occurences of the same subtask. The second mechanism aims at creating new basic behaviors for combination. It is based on an incremental learning method that builds on the approximate solution obtained through the combination of older behaviors.

2008

Rational, autonomous agents that are able to achieve their goals in dynamic, partially observable environments are the ultimate dream of Artificial Intelligence research since its beginning. The goal of this PhD thesis is to propose, develop and evaluate a framework well suited for creating intelligent agents that would be able to learn from experience, thus becoming more efficient at solving their tasks.

Based upon fundamentally diflerent ideas about the nature of knowledge and intelligence, the behavioural approach to artificial intelligence is an alternative to traditional representational AI paradigms that overcomes specific limitations of traditional approaches. Behavioural control systems for autonomous mobile robots have demonstrated robust and effective performance. However, a comp. rehensive methodology for the behavioural paradigm has not yet been fully developed. Concentrating upon a behavioural methodology called the subsumption architecture, this paper describes methodological enhancements to the behavioural paradigm based upon the use of logic. The logical definition of behaviours allows many techniques from traditional art@cial intelligence to be applied to the analysis, definition, and verification of behavioural systems, and supports strategies for achieving robustness, without incurring the problems associated with traditional AI.

Current trends in software development show a move towards supporting autonomous, rational components (agents). One of the most interesting issues in agent technology has always been the modeling and enhancement of agent behavior. In this paper we are focused in the intersection of agent technology and machine learning techniques for producing intelligent agents. Our application shows that using neural network techniques we improve the reasoning mechanism of our agent supplying to it a new behavior which it did not possess from the beginning. The learning process can be applied initially to train 'dummy' agent to further improve agent reasoning. The machine learning algorithms allow for an agent to adequately respond to environment changes and improve the behavioral rules or acquire intelligent behavior. A case study will be given to demonstrate such enhancement. We simulate the behavior of a robot moving in an environment with random obstacles. Learning techniques that are added to the reasoning mechanism of this robot enrich his behavior in the dynamic environment, displaying a rational and intelligent behavior.

Adaptive Behavior, 1994

Systems, Man, and Cybernetics, 1997.' …, 1997

2007

The use of reinforcement learning to guide action selection of cognitive agents has been shown to be a powerful technique for stochastic environments. Standard Reinforcement learning techniques used to provide decision theoretic policies rely, however, on explicit state-based computations of value for each state-action pair. This requires the computation of a number of values exponential to the number of state variables and actions in the system. This research extends existing work with an acquired probabilistic rule representation of an agent environment by developing an algorithm to apply reinforcement learning to values attached to the rules themselves. Structure captured by the rules is then used to learn a policy directly. The resulting value attached to each rule represents the utility of taking an action if the conditions of the rule are present in the agent's current set of percepts. This has several advantages for planning purposes: generalization over many states and over unseen states; effective decisions can therefore be made with less training data than state based modelling systems (e.g. Dyna Q-Learning); and the problem of computation in an exponential state-action space is alleviated. The results of application of this algorithm to rules in a specific environment are presented, with comparison to standard reinforcement learning policies developed from related work.

2006

This paper will discuss three arguments for a multilevel heterogeneous approach to Artificial Intelligence (AI) hard problems. By a heterogeneous approach, we mean the use of multiple methodologies (symbolic, sub-symbolic, subsumption) to solve AI problems. First, if one accepts the postulate that cognitive psychological principles can be beneficial to AI, then one must look at the heterogeneous nature of the human cognitive system. The brain is not homogeneous; it is a collection of different cellular organizations performing different functions. Secondly, there are several examples from the cognitive systems literature that show hybrid approaches provide effective solutions to complex problems. In some cases, these approaches have been better than a single approach. Finally, cognition is so complex, so full of subtle nuance and interwoven interdependencies, that a multiple level heterogeneous approach is the only approach that will prove to be successful in the long term. In other words, the complexity of perceiving and understanding the environment in a human manner necessitates a multilevel approach.

Applied Intelligence, 1999

In developing autonomous agents, one usually emphasizes only (situated) procedural knowledge, ignoring more explicit declarative knowledge. On the other hand, in developing symbolic reasoning models, one usually emphasizes only declarative knowledge, ignoring procedural knowledge. In contrast, we have developed a learning model Clarion, which is a hybrid connectionist model consisting of both localist and distributed representations, based on the two-level approach proposed in . Clarion learns and utilizes both procedural and declarative knowledge, tapping into the synergy of the two types of processes, and enables an agent to learn in situated contexts and generalize resulting knowledge to different scenarios. It unifies connectionist, reinforcement, and symbolic learning in a synergistic way, to perform on-line, bottom-up learning. This summary paper presents one version of the architecture and some results of the experiments.