PUDIIC reoortina ouraen for tnts collection of information is estimated to average i hour oer res... more PUDIIC reoortina ouraen for tnts collection of information is estimated to average i hour oer resoorse, including the time tor reviewing instructions, searcning existing data sources, gatnenna and maintaining the oata needed, ana comoteting ano reviewing the collection of information. Send comments regaroing triis burden estimate or any other asoect of this collection" of information, including suggestions for reducing this ouroen. to Washington Meaoauarters Services. Direaorate for information Ooerations and fteooas. 1215 Jefferson Davis Highway. Suite 1204. Arlington, vA 22202-4302. and to the Office of Management and Budget. Paperwork Reduction Proiea(07C4-0188), Washington. DC 20503.
Identifying the functionality in objects means to be able to associate a purpose with them in a s... more Identifying the functionality in objects means to be able to associate a purpose with them in a specific environment. The purpose depends on the intention of the agent and on the applicability of the object in a particular task. In our investigation of functionality we focus on functionalities which involve changes of physical relation and properties between objects in the environment. A formal model, based on Discrete Event Dynamic System Theory (DEDS), is introduced to define an interactive task for recovering and describing functionality. To observe and control the recovery process we introduce the notion of piecewise observability of a task by different sensors. This allows the description of a dynamic system in which neither all events nor the time of their occurrence may be predicted in advance. We have developed an experimental system consisting of actuators and both force and position sensors, for carrying out the interactive recovery of functionality. In particular, we demonstrate how this approach can be used by carrying out some experiments investigating the functionality of piercing. Furthermore, we discuss the importance of a multisensory approach for the observation and interpretation of functionality.
Despite the recent successes in robotics, artificial intelligence and computer vision, a complete... more Despite the recent successes in robotics, artificial intelligence and computer vision, a complete artificial agent necessarily must include active perception. A multitude of ideas and methods for how to accomplish this have already appeared in the past, their broader utility perhaps impeded by insufficient computational power or costly hardware. The history of these ideas, perhaps selective due to our perspectives, is presented with the goal of organizing the past literature and highlighting the seminal contributions. We argue that those contributions are as relevant today as they were decades ago and, with the state of modern computational tools, are poised to find new life in the robotic perception systems of the next decade. Keywords Sensing • Perception • Attention • Control This is one of several papers published in Autonomous Robots comprising the Special Issue on Active Perception.
A new approach to telepresence is presented in which a m ultitude of stationary cameras are used ... more A new approach to telepresence is presented in which a m ultitude of stationary cameras are used to acquire both photometric and depth information. A virtual environment is constructed by displaying the acquired data from the remote site in accordance with the head position and orientation of a local participant. Shown are preliminary results of a depth image of a human subject calculated from 11 closely spaced video camera positions. A user wearing a head-mounted display walks around this 3D data that has been inserted into a 3D model of a simple room. Future systems based on this approach m a y exhibit more natural and intuitive i n teraction among participants than current 2 D teleconferencing systems.
This report describes progress made during the past year on the UGV RSTA project being conducted ... more This report describes progress made during the past year on the UGV RSTA project being conducted by a consortium led by t h e U n i v ersity of Maryland and including the University o f P ennsylvania, the University o f R o c hester, and the National Institute of Standards and Technology. W e rst review work done on the design, implementation and integration of real time vision algorithms for image stabilization, detection of moving objects from a moving platform and camera control. We then present brief descriptions of a number of supporting basic research projects being conducted by the members of the consortium.
We present a computational model and algorithm for detecting diffuse and specular interface refle... more We present a computational model and algorithm for detecting diffuse and specular interface reflections and some inter-reflections. Our color reflection model is based on the dichromatic model for dielectric materials and on a color space, called S space, formed with three orthogonal basis functions. We transform color pixels measured in RGB into the S space and analyze color variations on objects in terms of brightness, hue and saturation which are defined in the S space. When transforming the original RGB data into the S space, we discount the scene illumination color that is estimated using a white reference plate as an active probe. As a result, the color image appears as if the scene illumination is white. Under the whitened illumination, the interface reflection clusters in the S space are all aligned with the brightness direction. The brightness, hue and saturation values exhibit a more direct correspondence to body colors and to diffuse and specular interface reflections, shading, shadows and interreflections than the RGB coordinates. We exploit these relationships to segment the color image, and to separate specular and diffuse interface reflections and some inter-reflections from body reflections. The proposed algorithm is efficacious for uniformly colored dielectric surfaces under singly colored scene illumination. Experimental results conform to our model and algorithm within the limitations discussed.
2010 IEEE International Conference on Multimedia and Expo, 2010
Though the variety of desktop real time stereo vision systems has grown considerably in the past ... more Though the variety of desktop real time stereo vision systems has grown considerably in the past several years, few make any verifiable claims about the accuracy of the algorithms used to construct 3D data or describe how the data generated by such systems, which is large in size, can be effectively distributed. In this paper, we describe a system that creates an accurate (on the order of a centimeter), 3D reconstruction of an environment in real time (under 30 ms) that also allows for remote interaction between users. This paper addresses how to reconstruct, compress, and visualize the 3D environment. In contrast to most commercial desktop real time stereo vision systems our algorithm produces 3D meshes instead of dense point clouds, which we show allows for better quality visualizations. The chosen representation of the data also allows for high compression ratios for transfer to remote sites. We demonstrate the accuracy and speed of our results on a variety of benchmarks.
When visual behaviors are combined to provide a speci c functionality needed for a task, the comb... more When visual behaviors are combined to provide a speci c functionality needed for a task, the combination is often based on heuristic rules. In this paper we s h o w that by adopting the Discrete Event Systems DES formalism for describing the interaction between visual behaviors it is possible to provide systems which h a ve w ell de ned properties in terms of observability and controllability. The method is in particular suited for describing the coupling between action and perception. An introduction to the use of DES is provided and it is demonstrated how DES are used for modeling behaviors and controlling a mobile robot equipped with a binocular camera head and some additional sensors.
Studies in health technology and informatics, 2013
We present mobile health (mHealth) applications utilizing embedded phone sensors as an angle-meas... more We present mobile health (mHealth) applications utilizing embedded phone sensors as an angle-measuring device for upper-limb range of motion (ROM) and estimation of reachable workspace to assist in evaluation of upper limb functional capacity. Our results show that the phone can record accurate measurements, as well as provide additional functionalities for clinicians.
Studies in health technology and informatics, 2013
We propose a novel low-cost method for quantitative assessment of upper extremity workspace envel... more We propose a novel low-cost method for quantitative assessment of upper extremity workspace envelope using Microsoft Kinect camera. In clinical environment there are currently no practical and cost-effective methods available to provide arm-function evaluation in three-dimensional space. In this paper we examine the accuracy of the proposed technique for workspace estimation using Kinect in comparison with a motion capture system. The experimental results show that the developed system is capable of capturing the workspace with sufficient accuracy and robustness.
Studies in health technology and informatics, 2012
We present a real-time algorithm for human pose detection and tracking from vision-based 3D data ... more We present a real-time algorithm for human pose detection and tracking from vision-based 3D data and its application to tele-rehabilitation in virtual environments. We employ stereo camera(s) to capture 3D avatars of geographically dislocated patient and therapist in real-time, while sending the data remotely and displaying it in a virtual scene. A pose detection and tracking algorithm extracts kinematic parameters from each participant and determines pose similarity. The pose similarity score is used to quantify patient's performance and provide real-time feedback for remote rehabilitation.
During semi-autonomous driving, threat assessment is used to determine when controller interventi... more During semi-autonomous driving, threat assessment is used to determine when controller intervention that overwrites or corrects the driver's input is required. Since today's semi-autonomous systems perform threat assessment by predicting the vehicle's future state while treating the driver's input as a disturbance, controller intervention is limited to just emergency maneuvers. In order to improve vehicle safety and reduce the aggressiveness of maneuvers, threat assessment must occur over longer prediction horizons where driver's behavior cannot be neglected. We propose a framework that divides the problem of semi-autonomous control into two components. The first component reliably predicts the vehicle's potential behavior by using empirical observations of the driver's pose. The second component determines when the semi-autonomous controller should intervene. To quantitatively measure the performance of the proposed approach, we define metrics to evaluate the informativeness of the prediction and the utility of the intervention procedure. A multi-subject driving experiment illustrates the usefulness, with respect to these metrics, of incorporating the driver's pose while designing a semi-autonomous system.
2015 International Conference on Healthcare Informatics, 2015
Microsoft Kinect camera and its skeletal tracking capabilities have been embraced by many researc... more Microsoft Kinect camera and its skeletal tracking capabilities have been embraced by many researchers and commercial developers in various applications of real-time human movement analysis. In this paper, we evaluate the accuracy of the human kinematic motion data in the first and second generation of the Kinect system, and compare the results with an optical motion capture system. We collected motion data in 12 exercises for 10 different subjects and from three different viewpoints. We report on the accuracy of the joint localization and bone length estimation of Kinect skeletons in comparison to the motion capture. We also analyze the distribution of the joint localization offsets by fitting a mixture of Gaussian and uniform distribution models to determine the outliers in the Kinect motion data. Our analysis shows that overall Kinect 2 has more robust and more accurate tracking of human pose as compared to Kinect 1.
We present work-in-progress on framework for 3D Kinect-based remote motion and function assessmen... more We present work-in-progress on framework for 3D Kinect-based remote motion and function assessment. Tele-MFAsT facilities streaming of data (video, depth, skeleton, and audio) from network-connected Microsoft Kinect cameras that can be utilized to observe a patient remotely during function assessment by a physician or therapist, measure motion trajectories, and joint angles, while taking advantage of tele-presence in a 3D virtual environment. The network architecture also allows for integration with wireless sensors in the future. In this paper we describe Tele-MFAsT application for upper-body range of motion measurements.
2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2013
Over the last few years, with the immense popularity of the Kinect, there has been renewed intere... more Over the last few years, with the immense popularity of the Kinect, there has been renewed interest in developing methods for human gesture and action recognition from 3D data. A number of approaches have been proposed that extract representative features from 3D depth data, a reconstructed 3D surface mesh or more commonly from the recovered estimate of the human skeleton. Recent advances in neuroscience have discovered a neural encoding of static 3D shapes in primate infero-temporal cortex that can be represented as a hierarchy of medial axis and surface features. We hypothesize a similar neural encoding might also exist for 3D shapes in motion and propose a hierarchy of dynamic medial axis structures at several spatio-temporal scales that can be modeled using a set of Linear Dynamical Systems (LDSs). We then propose novel discriminative metrics for comparing these sets of LDSs for the task of human activity recognition. Combined with simple classification frameworks, our proposed features and corresponding hierarchical dynamical models provide the highest human activity recognition rates as compared to state-of-the-art methods on several skeletal datasets.
Proceedings of the 3rd International Conference on PErvasive Technologies Related to Assistive Environments - PETRA '10, 2010
We present a mobile platform for body sensor networking based on a smartphone for lightweight sig... more We present a mobile platform for body sensor networking based on a smartphone for lightweight signal processing of sensor mote data. The platform allows for local processing of data at both the sensor mote and smartphone levels, reducing the overhead of data transmission to remote services. We discuss how the smartphone platform not only provides the ability for wearable signal processing, but it allows for opportunistic sensing strategies, in which many of the onboard sensors and capabilities of modern smartphones may be collected and fused with body sensor data to provide environmental and social context. We propose that this can help refine data reduction at the local level. We describe three examples related to health and wellness, to which our system has been applied.
PUDIIC reoortina ouraen for tnts collection of information is estimated to average i hour oer res... more PUDIIC reoortina ouraen for tnts collection of information is estimated to average i hour oer resoorse, including the time tor reviewing instructions, searcning existing data sources, gatnenna and maintaining the oata needed, ana comoteting ano reviewing the collection of information. Send comments regaroing triis burden estimate or any other asoect of this collection" of information, including suggestions for reducing this ouroen. to Washington Meaoauarters Services. Direaorate for information Ooerations and fteooas. 1215 Jefferson Davis Highway. Suite 1204. Arlington, vA 22202-4302. and to the Office of Management and Budget. Paperwork Reduction Proiea(07C4-0188), Washington. DC 20503.
Identifying the functionality in objects means to be able to associate a purpose with them in a s... more Identifying the functionality in objects means to be able to associate a purpose with them in a specific environment. The purpose depends on the intention of the agent and on the applicability of the object in a particular task. In our investigation of functionality we focus on functionalities which involve changes of physical relation and properties between objects in the environment. A formal model, based on Discrete Event Dynamic System Theory (DEDS), is introduced to define an interactive task for recovering and describing functionality. To observe and control the recovery process we introduce the notion of piecewise observability of a task by different sensors. This allows the description of a dynamic system in which neither all events nor the time of their occurrence may be predicted in advance. We have developed an experimental system consisting of actuators and both force and position sensors, for carrying out the interactive recovery of functionality. In particular, we demonstrate how this approach can be used by carrying out some experiments investigating the functionality of piercing. Furthermore, we discuss the importance of a multisensory approach for the observation and interpretation of functionality.
Despite the recent successes in robotics, artificial intelligence and computer vision, a complete... more Despite the recent successes in robotics, artificial intelligence and computer vision, a complete artificial agent necessarily must include active perception. A multitude of ideas and methods for how to accomplish this have already appeared in the past, their broader utility perhaps impeded by insufficient computational power or costly hardware. The history of these ideas, perhaps selective due to our perspectives, is presented with the goal of organizing the past literature and highlighting the seminal contributions. We argue that those contributions are as relevant today as they were decades ago and, with the state of modern computational tools, are poised to find new life in the robotic perception systems of the next decade. Keywords Sensing • Perception • Attention • Control This is one of several papers published in Autonomous Robots comprising the Special Issue on Active Perception.
A new approach to telepresence is presented in which a m ultitude of stationary cameras are used ... more A new approach to telepresence is presented in which a m ultitude of stationary cameras are used to acquire both photometric and depth information. A virtual environment is constructed by displaying the acquired data from the remote site in accordance with the head position and orientation of a local participant. Shown are preliminary results of a depth image of a human subject calculated from 11 closely spaced video camera positions. A user wearing a head-mounted display walks around this 3D data that has been inserted into a 3D model of a simple room. Future systems based on this approach m a y exhibit more natural and intuitive i n teraction among participants than current 2 D teleconferencing systems.
This report describes progress made during the past year on the UGV RSTA project being conducted ... more This report describes progress made during the past year on the UGV RSTA project being conducted by a consortium led by t h e U n i v ersity of Maryland and including the University o f P ennsylvania, the University o f R o c hester, and the National Institute of Standards and Technology. W e rst review work done on the design, implementation and integration of real time vision algorithms for image stabilization, detection of moving objects from a moving platform and camera control. We then present brief descriptions of a number of supporting basic research projects being conducted by the members of the consortium.
We present a computational model and algorithm for detecting diffuse and specular interface refle... more We present a computational model and algorithm for detecting diffuse and specular interface reflections and some inter-reflections. Our color reflection model is based on the dichromatic model for dielectric materials and on a color space, called S space, formed with three orthogonal basis functions. We transform color pixels measured in RGB into the S space and analyze color variations on objects in terms of brightness, hue and saturation which are defined in the S space. When transforming the original RGB data into the S space, we discount the scene illumination color that is estimated using a white reference plate as an active probe. As a result, the color image appears as if the scene illumination is white. Under the whitened illumination, the interface reflection clusters in the S space are all aligned with the brightness direction. The brightness, hue and saturation values exhibit a more direct correspondence to body colors and to diffuse and specular interface reflections, shading, shadows and interreflections than the RGB coordinates. We exploit these relationships to segment the color image, and to separate specular and diffuse interface reflections and some inter-reflections from body reflections. The proposed algorithm is efficacious for uniformly colored dielectric surfaces under singly colored scene illumination. Experimental results conform to our model and algorithm within the limitations discussed.
2010 IEEE International Conference on Multimedia and Expo, 2010
Though the variety of desktop real time stereo vision systems has grown considerably in the past ... more Though the variety of desktop real time stereo vision systems has grown considerably in the past several years, few make any verifiable claims about the accuracy of the algorithms used to construct 3D data or describe how the data generated by such systems, which is large in size, can be effectively distributed. In this paper, we describe a system that creates an accurate (on the order of a centimeter), 3D reconstruction of an environment in real time (under 30 ms) that also allows for remote interaction between users. This paper addresses how to reconstruct, compress, and visualize the 3D environment. In contrast to most commercial desktop real time stereo vision systems our algorithm produces 3D meshes instead of dense point clouds, which we show allows for better quality visualizations. The chosen representation of the data also allows for high compression ratios for transfer to remote sites. We demonstrate the accuracy and speed of our results on a variety of benchmarks.
When visual behaviors are combined to provide a speci c functionality needed for a task, the comb... more When visual behaviors are combined to provide a speci c functionality needed for a task, the combination is often based on heuristic rules. In this paper we s h o w that by adopting the Discrete Event Systems DES formalism for describing the interaction between visual behaviors it is possible to provide systems which h a ve w ell de ned properties in terms of observability and controllability. The method is in particular suited for describing the coupling between action and perception. An introduction to the use of DES is provided and it is demonstrated how DES are used for modeling behaviors and controlling a mobile robot equipped with a binocular camera head and some additional sensors.
Studies in health technology and informatics, 2013
We present mobile health (mHealth) applications utilizing embedded phone sensors as an angle-meas... more We present mobile health (mHealth) applications utilizing embedded phone sensors as an angle-measuring device for upper-limb range of motion (ROM) and estimation of reachable workspace to assist in evaluation of upper limb functional capacity. Our results show that the phone can record accurate measurements, as well as provide additional functionalities for clinicians.
Studies in health technology and informatics, 2013
We propose a novel low-cost method for quantitative assessment of upper extremity workspace envel... more We propose a novel low-cost method for quantitative assessment of upper extremity workspace envelope using Microsoft Kinect camera. In clinical environment there are currently no practical and cost-effective methods available to provide arm-function evaluation in three-dimensional space. In this paper we examine the accuracy of the proposed technique for workspace estimation using Kinect in comparison with a motion capture system. The experimental results show that the developed system is capable of capturing the workspace with sufficient accuracy and robustness.
Studies in health technology and informatics, 2012
We present a real-time algorithm for human pose detection and tracking from vision-based 3D data ... more We present a real-time algorithm for human pose detection and tracking from vision-based 3D data and its application to tele-rehabilitation in virtual environments. We employ stereo camera(s) to capture 3D avatars of geographically dislocated patient and therapist in real-time, while sending the data remotely and displaying it in a virtual scene. A pose detection and tracking algorithm extracts kinematic parameters from each participant and determines pose similarity. The pose similarity score is used to quantify patient's performance and provide real-time feedback for remote rehabilitation.
During semi-autonomous driving, threat assessment is used to determine when controller interventi... more During semi-autonomous driving, threat assessment is used to determine when controller intervention that overwrites or corrects the driver's input is required. Since today's semi-autonomous systems perform threat assessment by predicting the vehicle's future state while treating the driver's input as a disturbance, controller intervention is limited to just emergency maneuvers. In order to improve vehicle safety and reduce the aggressiveness of maneuvers, threat assessment must occur over longer prediction horizons where driver's behavior cannot be neglected. We propose a framework that divides the problem of semi-autonomous control into two components. The first component reliably predicts the vehicle's potential behavior by using empirical observations of the driver's pose. The second component determines when the semi-autonomous controller should intervene. To quantitatively measure the performance of the proposed approach, we define metrics to evaluate the informativeness of the prediction and the utility of the intervention procedure. A multi-subject driving experiment illustrates the usefulness, with respect to these metrics, of incorporating the driver's pose while designing a semi-autonomous system.
2015 International Conference on Healthcare Informatics, 2015
Microsoft Kinect camera and its skeletal tracking capabilities have been embraced by many researc... more Microsoft Kinect camera and its skeletal tracking capabilities have been embraced by many researchers and commercial developers in various applications of real-time human movement analysis. In this paper, we evaluate the accuracy of the human kinematic motion data in the first and second generation of the Kinect system, and compare the results with an optical motion capture system. We collected motion data in 12 exercises for 10 different subjects and from three different viewpoints. We report on the accuracy of the joint localization and bone length estimation of Kinect skeletons in comparison to the motion capture. We also analyze the distribution of the joint localization offsets by fitting a mixture of Gaussian and uniform distribution models to determine the outliers in the Kinect motion data. Our analysis shows that overall Kinect 2 has more robust and more accurate tracking of human pose as compared to Kinect 1.
We present work-in-progress on framework for 3D Kinect-based remote motion and function assessmen... more We present work-in-progress on framework for 3D Kinect-based remote motion and function assessment. Tele-MFAsT facilities streaming of data (video, depth, skeleton, and audio) from network-connected Microsoft Kinect cameras that can be utilized to observe a patient remotely during function assessment by a physician or therapist, measure motion trajectories, and joint angles, while taking advantage of tele-presence in a 3D virtual environment. The network architecture also allows for integration with wireless sensors in the future. In this paper we describe Tele-MFAsT application for upper-body range of motion measurements.
2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2013
Over the last few years, with the immense popularity of the Kinect, there has been renewed intere... more Over the last few years, with the immense popularity of the Kinect, there has been renewed interest in developing methods for human gesture and action recognition from 3D data. A number of approaches have been proposed that extract representative features from 3D depth data, a reconstructed 3D surface mesh or more commonly from the recovered estimate of the human skeleton. Recent advances in neuroscience have discovered a neural encoding of static 3D shapes in primate infero-temporal cortex that can be represented as a hierarchy of medial axis and surface features. We hypothesize a similar neural encoding might also exist for 3D shapes in motion and propose a hierarchy of dynamic medial axis structures at several spatio-temporal scales that can be modeled using a set of Linear Dynamical Systems (LDSs). We then propose novel discriminative metrics for comparing these sets of LDSs for the task of human activity recognition. Combined with simple classification frameworks, our proposed features and corresponding hierarchical dynamical models provide the highest human activity recognition rates as compared to state-of-the-art methods on several skeletal datasets.
Proceedings of the 3rd International Conference on PErvasive Technologies Related to Assistive Environments - PETRA '10, 2010
We present a mobile platform for body sensor networking based on a smartphone for lightweight sig... more We present a mobile platform for body sensor networking based on a smartphone for lightweight signal processing of sensor mote data. The platform allows for local processing of data at both the sensor mote and smartphone levels, reducing the overhead of data transmission to remote services. We discuss how the smartphone platform not only provides the ability for wearable signal processing, but it allows for opportunistic sensing strategies, in which many of the onboard sensors and capabilities of modern smartphones may be collected and fused with body sensor data to provide environmental and social context. We propose that this can help refine data reduction at the local level. We describe three examples related to health and wellness, to which our system has been applied.
In this work we establish a framework for the general problem of observation, which may be applie... more In this work we establish a framework for the general problem of observation, which may be applied to different kinds of visual tasks. We construct ‘intelligent’ high-level control mechanisms for active visual recognition of different processes within a hybrid dynamic system. We address the problem of observing a manipulation process in order to illustrate the ideas and motive behind our framework. We use a discrete event dynamic system as a high-level structuring technique to model the manipulation system. The formulation utilizes the knowledge about the system and the different actions in order to solve the observer problem in an efficient, stable and practical manner. The model uses different tracking mechanisms so that the observer can ‘see’ the workspace of the manipulating robot. An automaton is developed for the hand/object interaction over time and a stabilizing observer is constructed. Low-level modules are developed for recognizing the visual ‘events’ that causes state transitions within the dynamic manipulation system in real time. A coarse quantization of the manipulation actions is used in order to attain an active, adaptive and goaldirected sensing mechanism. The formulation provides high-level symbolic interpretations of the scene under observation. The discrete event framework is augmented with mechanisms for recovering the continuous parametric evolution of the scene under observation and for asserting the state of the manipulation agent. This work examines closely the possibilities for errors, mistakes and uncertainties in the manipulation system, observer construction process and event identification mechanisms. We identify and suggest techniques for modeling these uncertainties. Ambiguities are allowed to develop and are resolved after finite time. Error recovery mechanisms are also devised. The computed uncertainties are utilized for navigating the observer automaton state space, asserting state transitions and developing a suitable tracking
Uploads
Papers by Ruzena Bajcsy