Robot Shaping: Developing Autonomous Agents Through Learning

2005

Proceeding of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction - HRI '06, 2006

Programming robots to carry out useful tasks is both a complex and non-trivial exercise. A simple and intuitive method to allow humans to train and shape robot behaviour is clearly a key goal in making this task easier. This paper describes an approach to this problem based on studies of social animals where two teaching strategies are applied to allow a human teacher to train a robot by moulding its actions within a carefully scaffolded environment. Within these enviroments sets of competences can be built by building state/action memory maps of the robot's interaction within that environment. These memory maps are then polled using a k-nearest neighbour based algorithm to provide a generalised competence. We take a novel approach in building the memory models by allowing the human teacher to construct them in a hierarchical manner. This mechanism allows a human trainer to build and extend an action-selection mechanism into which new skills can be added to the robot's repertoire of existing competencies. These techniques are implemented on physical Khepera miniature robots and validated on a variety of tasks.

In this paper we overviews the ongoing project that we called POPSICLE (Patterns in Orientation: Pattern-Aided Simulated Interaction Context Learning Experiment). We overview the experimental setup, and give some preliminary results.

Learning Systems used on robots require either a-priori knowledge in the form of models, rules of thumb or databases or require that robot to physically execute multitudes of trial solutions. The first requirement limits the robot's ability to operate in unstructured changing environments, and the second limits the robot's service life and resources. In this research a generalized approach to learning was developed through a series of algorithms that can be used for construction of behaviors that are able to cope with unstructured environments through adaptation of both internal parameters and system structure as a result of a goal based supervisory mechanism. Four main learning algorithms have been developed, along with a goal directed random exploration routine. These algorithms all use the concept of learning from a recent memory in order to save the robot/agent from having to exhaustively execute all trial solutions. The first algorithm is a reactive online learning algorithm that uses a supervised learning to find the sensor/action combinations that promote realization of a preprogrammed goal. It produces a feed forward neural network controller that is used to control the robot. The second algorithm is similar to first in that it uses a supervised learning strategy, but it produces a neural network that considers past values, thus providing a non-reactive solution. The third algorithm is a departure from the first two in that uses a nonsupervised learning technique to learn the best actions for each situation the robot encounters. The last algorithm builds a graph of the situations encountered by agent/robot in order to learn to associate the best actions with sensor inputs. It uses an unsupervised learning approach based on shortest paths to a goal situation in the graph in order to generate a non-reactive feed forward neural network. Test results were good, the first and third algorithms were tested in a formation maneuvering task in both simulation and onboard mobile robots, while the second and fourth were tested simulation. v

Adaptive Behavior, 2005

This paper introduces an integration of reinforcement learning and behavior-based control designed to produce real-time learning in situated agents. The model layers a distributed and asynchronous reinforcement learning algorithm over a learned topological map and standard behavioral substrate to create a reinforcement learning complex. The topological map creates a small and task-relevant state space that aims to make learning feasible, while the distributed and asynchronous aspects of the architecture make it compatible with behavior-based design principles. We present the design, implementation and results of an experiment that requires a mobile robot to perform puck foraging in three artificial arenas using the new model, random decision making, and layered standard reinforcement learning. The results show that our model is able to learn rapidly on a real robot in a real environment, learning and adapting to change more quickly than both alternatives. We show that the robot is a...

To enable autonomous systems to learn basic skills for unknown and changing environments and stay robust in case of change, Organic Computing principles have to be applied at all layers. In this work an architecture is presented that can be used at the lowest layer providing robust skills to higher-levelstrategy layers, that depend on encapsulated actions. With emphasis on robustness it is able to learn to control its actors without apriori informationabout their meaning. This is made possible by skill modules that are learned together with their action-effect dependencies and their enabling preconditions by proactively carrying out experiments within their environment. The architecture is evaluated by simulating adifferentially drivenrobot. 1I ntroduction When some form of adaptation is needed typically atinypartinthe overall control architecture is identified and substituted by e. g. neural nets or solved by other statistical learning methods. However, theyall assume astatic training set, which hampers the ability to adapt appropriately to suddenly changing environments. Furthermore, theyexpect the designer to foresee all possible changes the system might undergo. In this work wepresent an architecture that is able to detect changes in the environment that render previously learned skills useless, and react in aw ay that relearns the obsolete parts by proactively carrying out experiments. Thereby,the designer does not have to foresee every possible change the system might undergo. With skills we understand low-levelblocks of behavior that can be triggered by some higher-levelstrategy process. We do this by coupling learned skills with their enabling conditions that have been observed while experimenting and the effects of the action. Thereby the system can monitor progress via manyfi ne-grained cause-effect schemata it has learned, and trigger relearning of the previously learned skill. By developing and finding basic skills the robot drastically reduces the exploration space the higher levels otherwise had to consider. Let us assume an upper strategy layer requesting some behavior that has ac ertain effect on its environment. The skill learning layer then consults its skill database (Skill DB) for appropriate skill modules consisting of aset of preconditions, the action and the predicted effect (similar to start condition, action type, end condition in [Bis05]). If an adequate skill is found, meaning that there is some behavior that resulted in the desired effects

2000

A method for evolving behavior-based robot controllers using genetic programming is presented. Due to their hierarchical nature, genetic programs are useful representing high-level knowledge for robot controllers. One drawback is the difficulty of incorporating sensory inputs. To overcome the gap between symbolic representation and direct sensor values, the elements of the function set in genetic programming is implemented as a single-layer perceptron. Each perceptron is composed of senory input nodes and a decision output node. The robot learns proper behavior rules based on local, limited sensory information without using an internal map. First, it learns how to discriminate the target using single-layer perceptrons. Then, the learned perceptrons are applied to the function nodes of the genetic program tree which represents a robot controller. Experiments have been performed using Khepera robots. The presented method successfully evolved high-level genetic programs that control the robot to find the light source from sensory inputs.

IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics), 1996

|This work describes a control architecture based on a hierarchical classi er system. This system, which learns both reactive and planning rules, implements a motivationally autonomous animat that chooses the actions it performs according to its perception of the external environment, to its physiological or internal state, to the consequences of its current behavior, and to the expected consequences of its future behavior. The adaptive faculties of this architecture are illustrated within the context of a navigation task, through various experiments with a simulated and a real robot.

Neurocomputing, 1999

The development of high potential for out-door or hostile environment ability necessitates an adaptive and versatile control system in order to avoid the difficulties of complex and unpredictable behaviour modelling. Auto-organisation allows artificial machines to approach these goals. For that, reinforcement methods are investigated: considering that the relations between the task to perform and the environment may act as a supervisor, efficient learning is performed. Starting from a very simple structure inspired by insect behaviour, the study presented in this paper is devoted to a neural network based control system which allows a simulated six legged robot to walk and avoid obstacles even when it is partially damaged.

2008

Abstract We describe a learning algorithm that generates behaviors by self-organization of sensorimotor loops in an autonomous robot. The behavior of the robot is analyzed by a multi-expert architecture, where a number of controllers compete for the data from the physical robot. Each expert stabilizes the representation of the acquired sensorimotor mapping in dependence of the achieved prediction error and forms eventually a behavioral primitive.

1995

One of the most general forms of representing and specifying behavior is by using a computer language. We have evaluated the use of the evolutionary technique of Genetic Programming (GP) to directly control a miniature robot. The goal of the GP-system was to evolve real-time obstacle avoiding behavior from sensorial. The evolved programs are used in a sense-think-act context. We employed a novel technique to enable real time learning with a real robot using genetic programming. To our knowledge, this is the rst use of GP with a real robot. The method uses a probabilistic sampling of the environment where each individual is tested on a new real-time tness case in a tournament selection procedure. The robots behavior is evolved without any knowledge of the task except for the feed-back from a tness function. The tness has a pain and a pleasure part. The negative part of tness, the pain, is simply the sum of the proximity sensor values. In order to keep the robot from standing still or gyrating, it has a pleasure component to its tness. It gets pleasure from going straight and fast. The evolved algorithm shows robust performance even if the robot is lifted and placed in a completely di erent environment or if obstacles are moved around.

Journal of Physical Agents (JoPha), 2012

This article describes a proposal to achieve fast robot learning from its interaction with the environment. Our proposal will be suitable for continuous learning procedures as it tries to limit the instability that appears every time the robot encounters a new situation it had not seen before. On the other hand, the user will not have to establish a degree of exploration (usual in reinforcement learning) and that would prevent continual learning procedures. Our proposal will use an ensemble of learners able to combine dynamic programming and reinforcement learning to predict when a robot will make a mistake. This information will be used to dynamically evolve a set of control policies that determine the robot actions.

Connection Science, 2006

Future application areas for humanoid robots range from the household, to agriculture, to the military, and to the exploration of space. Service applications such as these must address a changing, unstructured environment, a collaboration with human clients, and the integration of manual dexterity and mobility. Control frameworks for service-oriented humanoid robots must, therefore, accommodate many independently challenging issues including: techniques for configuring networks of sensorimotor resources; modeling tasks and constructing behavior in partially observable environments; and integrated control paradigms for mobile manipulators. Our approach advocates actively gathering salient information, modeling the environment, reasoning about solutions to new problems, and coordinating ad hoc interactions between multiple degrees of freedom to do mechanical work. Representations that encode control knowledge are a primary concern. Individual robots must exploit declarative structure for planning and must learn procedural strategies that work in recognizable contexts. We present several pieces of an overall framework in which a robot learns situated policies for control that exploit existing control knowledge and extend its scope. Several examples drawn from the research agenda at the Laboratory for Perceptual Robotics are used to illustrate the ideas.

Künstliche Intell., 2021