Controlling robots with two cameras: how to do it properly

2001

Most visual servoing applications are concerned with geometrically modeled objects. In this paper, the problem of controlling a motion by visual servoing around an unknown object with a stereovision system is addressed. The main goal is to move the end-effector around the object in order to observe several viewpoints of the object for other tasks, e.g. inspection or grasping. The present work uses the well-known imagebased visual servoing approach with a point, but the importance of the relationship between the end-effector and camera frames is clarified and emphasized. This relationship is needed for properly stacking the Jacobians or interaction matrices of each camera. A comparison with a visual servoing approach with a direct stacking of the Jacobians is presented. The centroid of a region, obtained by color segmentation, is used to move around the observed object. Experiments are developed on a PA-10 robot connected to a real time stereovision system, with two cameras mounted on the end-effector. Experimental results demonstrate the importance of a proper definition of the stacked Jacobians, to avoid undesired motions in the servoing task. Particularly, when turning around an unknown object, undesired motions on roll angle of the stereovision system can be avoided.

IEEE ISR 2013, 2013

This paper introduces a new comprehensive solution for the open problem of uncalibrated 3D image-based stereo visual servoing for robot manipulators. One of the main contributions of this article is a novel 3D stereo camera model to map positions in the task space to positions in a new 3D Visual Cartesian Space (a visual feature space where 3D positions are measured in pixels). This model is used to compute a full-rank Image Jacobian Matrix (J img), which solves several common problems presented on the classical image Jacobians, e.g., image space singularities and local minima. This Jacobian is a fundamental key for the image-based control design, where uncalibrated stereo camera systems can be used to drive a robot manipulator. Furthermore, an adaptive second order sliding mode control is designed to track 3D visual motions using the 3D trajectory errors defined in the Visual Cartesian Space, where a Torque to Position Model is designed to allow the implementation of joint torque control techniques on joint position-controlled robots. This approach has been experimentally implemented on a real industrial robot where exponential convergence of errors in the Visual Cartesian Space and Task space without local minima are demonstrated. This approach offers a proper solution for the common problem of visual occlusion, since the stereo system can be moved manually to obtain a clear view of the task at any time.

Robot …, 2008

This chapter deals with direct visual servoing of robot manipulators under monocular fixedcamera configuration. Considering the image-based approach, the complete nonlinear system dynamics is utilized into the control system design. Multiple coplanar object feature ...

2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013

This paper introduces a new comprehensive solution for the open problem of uncalibrated 3D image-based stereo visual servoing for robot manipulators. One of the main contributions of this article is a novel 3D stereo camera model to map positions in the task space to positions in a new 3D Visual Cartesian Space (a visual feature space where 3D positions are measured in pixel). This model is used to compute a full-rank Image Jacobian Matrix (J img ), which solves several common problems presented on the classical image Jacobians, e.g., image space singularities and local minima. This Jacobian is a fundamental key for the image-based control design, where uncalibrated stereo camera systems can be used to drive a robot manipulator. Furthermore, an adaptive second order sliding mode visual servo control is designed to track 3D visual motions using the 3D trajectory errors defined in the Visual Cartesian Space. The stability of the control in closed loop with a dynamic robot system is formally analyzed and proved, where exponential convergence of errors in the Visual Cartesian Space and task space without local minima are demonstrated. The complete control system is evaluated both in simulation and on a real industrial robot. The robustness of the control scheme is evaluated for cases where the extrinsic parameters of the stereo camera system change on-line and the kinematic/dynamic robot parameters are considered as unknown. This approach offers a proper solution for the common problem of visual occlusion, since the stereo system can be moved to obtain a clear view of the task at any time.

2006

This paper describes a innovative algorithm for the on-line estimation of the image jacobian based on previous movements. It has been utilized for the tracking of an uncalibrated 3 DOF joint system, using two weakly calibrated fixed cameras. The algorithm incorporates the epipolar restriction of the two cameras conveyed in the fundamental matrix, and a reliability factor has been defined for the previous movements. The tests carried out show that the proposed algorithm is more robust in presence of noise than those obtained from already existing algorithms in specialized literature.

The main goal of this paper is the study of image-based stereo visual servoing. A pair of cameras is mounted on the end-effector of the manipulator arm. The visual features are the pair of images of an unknown object. The developed control laws use either the raw image points, or the estimated 3D coordinates. The experimental setup is challenging: large rotations are involved, images are noisy, and cameras are coarsely calibrated. In this setup, the trajectory of the end-effector differs notably, sometimes leading the arm near its joint range limits. Experimental results demonstrate that using pixel coordinates is disadvantageous, compared with 3D coordinates estimated from the same pixel data.

2003

Abstract In this paper, the 3-dimensional (3D) position and orientation of a camera held by the end-effector of a robot manipulator is regulated to a constant desired position and orientation despite (i) the lack of depth information of the actual or desired camera position from a target,(ii) the lack of a geometric model of the target object, and (iii) uncertainty regarding both the angle and axis of rotation of the camera with respect to the robot end-effector (ie, the orientation extrinsic camera parameters).

2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2014

This paper introduces 6 new image features to provide a solution to the open problem of uncalibrated 6D image-based visual servoing for robot manipulators, where the goal is to control the 3D position and orientation of the robot end-effector using visual feedback. One of the main contributions of this article is a novel stereo camera model which employs virtual orthogonal cameras to map 6D Cartesian poses defined in the Task space to 6D visual poses defined in a Virtual Visual space (Image space). This new model is used to compute a full-rank square Image Jacobian matrix (J img), which solves several common problems exhibited by the classical image Jacobians, e.g., Image space singularities and local minima. This Jacobian is a fundamental key for the imagebased controller design, where a chattering-free adaptive second order sliding mode is employed to track 6D visual motions for a robot manipulator. Exponential convergence of errors in both spaces without local minima are demonstrated. The complete control system is experimentally evaluated on a real industrial robot. The robustness of the control scheme is evaluated for cases where the extrinsic parameters of the uncalibrated stereo camera system are changed on-line and unknown when the stereo system is manually moved to obtain a clearer view of the task.

IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004, 2004

This paper presents a novel visual servoing approach, aimed at controlling the so-called extended-2D (EZD) coordinates of the points constituting a tracked target. This approach does not q u i r e any pose reconstruction. Rather, the only informations required to build the EZD coordinates are the estimated depth distribution of the target points, and the estimated camera model. Several implementations of the controller are considered, which are inspired from conventional image based visual servoing from points. In spite of their simplicity, the proposed control laws exhibit remarkable stability robustness properties. A key issue is that only three configurations of undesired equilibrium exist, and they are all proven to be unstable even in the uncalibrated case. In other words, contrarily to existing 2D methods, there are no local minima. The paper details lbe control design and analysis and provides simulation results, emphasizing the remarkable robustness with respect to camera calibration errors. 0-7803-8232-3/04/$17.00 @22004 IEEE