Skip to main content

Li Guan

University of North Carolina at Chapel Hill, Computer Science, Graduate Student

Followers

15

Following

1

Co-author

1

Public Views

Harvard University

School of the Art Institute of Chicago

Duke University

Kansas State University

Armando Marques-Guedes

UNL - New University of Lisbon

Oxford Brookes University

Roshan Chitrakar

Nepal College of Information Technology

Graduate Center of the City University of New York

PALIMOTE JUSTICE

RIVERS STATE POLYTECHNIC

Punjab Technical University

Interests

Uploads

Papers by Li Guan

Multi-view dynamic scene modeling

Modeling dynamic scenes/events from multiple fixed-location vision sensors, such as video camcord... more Modeling dynamic scenes/events from multiple fixed-location vision sensors, such as video camcorders, infrared cameras, Time-of-Flight sensors etc, is of broad interest in computer vision society, with many applications including 3D TV, virtual reality, medical surgery, markerless motion capture, video games, and security surveillance. However, most of the existing multi-view systems are set up in a strictly controlled indoor environment, with fixed lighting conditions and simple background views. Many challenges are preventing the technology to an outdoor natural environment. These include varying sunlight, shadows, reflections, background motion and visual occlusion. In this thesis, I address different aspects to overcome all of the aforementioned difficulties, so as to reduce human preparation and manipulation, and to make a robust outdoor system as automatic as possible. In particular, the main novel technical contributions of this thesis are as follows: a generic heterogeneous ...

Robocup field distortion calibration with variational image registration

Proceedings of 2004 International Symposium on Intelligent Multimedia, Video and Speech Processing, 2004.

Global Vision System is widely used among teams of Robocup F180 (Small Size League). This archite... more Global Vision System is widely used among teams of Robocup F180 (Small Size League). This architecture as well as the 1-hour preparation time limit before each match poses strict requirements to the accuracy and efficiency of the calibration method against pitch field distortions. Because of the curvature-preserving feature of these distortions, we propose a brand-new image registration approach, which does not require calibrating the camera, but directly maps the distorted image to the real pitch. While this method substantially simplifies the pre-match manipulation, it also maintains high accuracyour experimental result shows that, with double camera, the mean error ratio is limited within 0.22%, accurate enough for Robocup matches.

Smart instrumented training ranges: bringing automated system solutions to support critical domain needs

The Journal of Defense Modeling and Simulation: Applications, Methodology, Technology, 2013

The training objective for urban warfare includes acquisition and perfection of a set of diverse ... more The training objective for urban warfare includes acquisition and perfection of a set of diverse skills in support of kinetic and non-kinetic operations. The US Marines (USMC) employ long-duration acted scenarios with verbal training feedback provided sporadically throughout the training session and at the end in a form of an after-action review (AAR). The inherent characteristic of training ranges for urban warfare is that they are the environments with a high level of physical occlusion, which causes many performances not to be seen by a group of instructors who oversee the training. We describe BASE-IT (Behavioral Analysis and Synthesis for Intelligent Training), a system in development that aims to automate capture of training data and their analysis, performance evaluation, and AAR report generation. The goal of this effort is to greatly increase the amount of observed behavior and improve the quality of the AAR. The system observes training with stationary cameras and personal tracking devices. It then analyzes movement and body postures, measures individual and squad-level performance, and compares it to standards and levels of performance expected in given situations. An interactive visualization component delivers live views augmented with real-time analytics and alerts; it also generates a personalized AAR review in a three-dimensional virtual or mixed reality environment, indexed by automatically extracted salient events and accompanied by summary statistics of unit performance. The approaches presented in the system have the potential to radically change the analysis and performance assessment on physical training ranges and ultimately this type of training itself.

Visual Shapes of Silhouette Sets

Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06), 2006

Shape from silhouette methods are extensively used to model dynamic and non-rigid objects using b... more Shape from silhouette methods are extensively used to model dynamic and non-rigid objects using binary foreground-background images. Since the problem of reconstructing shapes from silhouettes is ambiguous, a number of solutions exist and several approaches only consider the one with a maximal volume, called the visual hull. However, the visual hull is not always a good approximation of shapes, in particular when observing smooth surfaces with few cameras. In this paper, we consider instead a class of solutions to the silhouette reconstruction problem that we call visual shapes. Such a class includes the visual hull, but also better approximations of the observed shapes which can take into account local assumptions such as smoothness, among others. Our contributions with respect to existing works is first to identify silhouette consistent shapes different from the visual hull, and second to give a practical way to estimate such shapes in real time. Experiments on various sets of data including human body silhouettes are shown to illustrate the principle and the interests of visual shapes.

A Real-Time Video Tracking System

IEEE Transactions on Pattern Analysis and Machine Intelligence, 1980

Object identification and tracking applications of pattern recognition at video rates is a proble... more Object identification and tracking applications of pattern recognition at video rates is a problem of wide interest, with previous attempts limited to very simple threshold or correlation (restricted window) methods. New high-speed algorithms together with fast digital hardware have produced a system for missile and aircraft identification and tracking that possesses a degree of "intelligence" not previously implemented in a real-time tracking system. Adaptive statistical clustering and projection-based classification algorithms are applied in real time to identify and track objects that change in appearance through complex and nonstationary background/foreground situations. Fast estimation and prediction algorithms combine linear and quadratic estimators to provide speed and sensitivity. Weights are determined to provide a measure of confi'dence in the data and resulting decisions. Strategies based on maximizing the probability of maintaining track are developed. This paper emphasizes the theoretical aspects of the system and discusses the techniques used to achieve realtime implementation. Index Terms-Image processing, intensity histograms, object identification, optical tracking, projections, tracking system, video data compression, video processing, video tracking. velopment of the real-time videotheodolite (RTV). U.S. Army White Sands Missile Range began the development of the RTV in 1974, and the system is being deployed at this time. Design philosophy called for a system capable of discriminatory judgment in identifying the object to be tracked with 60 independent observations/s, capable of locating the center of mass of the object projection on the image plane within about 2 per-Manuscript

Sensor-Based Cooperative Multi-Robot 3D Environment Reconstruction

Master Thesis, Department of Computer Science, University of North Carolina-Chapel Hill, Chapel Hill, NC, USA, Apr 1, 2006

In this paper, we discuss the task of unknown environment exploration with a team of robots. The ... more In this paper, we discuss the task of unknown environment exploration with a team of robots. The goal is to reconstruct the 3D scene from onboard sensor data. There are mainly three problems involved: reconstruction, motion planning, and multiple robot collaboration. First we introduce the principal algorithms of reconstruction 3D models from different kinds of vision sensors. Given the 3D model, feature-based localization combined with probability framework can be used to locate each of the robots. Then the problem is to ...

A unified approach to calibrate a network of camcorders and tof cameras

Abstract. In this paper, we propose a unified calibration technique for a heterogeneous sensor ne... more Abstract. In this paper, we propose a unified calibration technique for a heterogeneous sensor network of video camcorders and Time-of-Flight (ToF) cameras. By moving a spherical calibration target around the commonly observed scene, we can robustly and conveniently extract the sphere centers in the observed images and recover the geometric extrinsics for both types of sensors. The approach is then evaluated with a real dataset of two HD camcorders and two ToF cameras, and 3D shapes are reconstructed from this ...

Multi-object shape estimation and tracking from silhouette cues

2008 IEEE Conference on Computer Vision and Pattern Recognition, 2008

This paper deals with the 3D shape estimation from silhouette cues of multiple moving objects in ... more This paper deals with the 3D shape estimation from silhouette cues of multiple moving objects in general indoor or outdoor 3D scenes with potential static obstacles, using multiple calibrated video streams. Most shape-fromsilhouette techniques use a two-classification of space occupancy and silhouettes, based on image regions that match or disagree with a static background appearance model. Binary silhouette information becomes insufficient to unambiguously carve 3D space regions as the number and density of dynamic objects increases. In such difficult scenes, multi-view stereo methods suffer from visibility problems, and rely on color calibration procedures tedious to achieve outdoors. We propose a new algorithm to automatically detect and reconstruct scenes with a variable number of dynamic objects. Our formulation distinguishes between m different shapes in the scene by using automatically learnt view-specific appearance models, eliminating the color calibration requirement. Bayesian reasoning is then applied to solve the m-shape occupancy problem, with m updated as objects enter or leave the scene. Results show that this method yields multiple silhouette-based estimates that drastically improve scene reconstructions over traditional twolabel silhouette scene analysis. This enables the method to also efficiently deal with multi-person tracking problems.

3D Occlusion Inference from Silhouette Cues

2007 IEEE Conference on Computer Vision and Pattern Recognition, 2007

We consider the problem of detecting and accounting for the presence of occluders in a 3D scene b... more We consider the problem of detecting and accounting for the presence of occluders in a 3D scene based on silhouette cues in video streams obtained from multiple, calibrated views. While well studied and robust in controlled environments, silhouette-based reconstruction of dynamic objects fails in general environments where uncontrolled occlusions are commonplace, due to inherent silhouette corruption by occluders. We show that occluders in the interaction space of dynamic objects can be detected and their 3D shape fully recovered as a byproduct of shape-from-silhouette analysis. We provide a Bayesian sensor fusion formulation to process all occlusion cues occurring in a multi-view sequence. Results show that the shape of static occluders can be robustly recovered from pure dynamic object motion, and that this information can be used for online self-correction and consolidation of dynamic object shape reconstruction.

Multi-Object Shape Estimation from Silhouette Cues

This paper deals with the 3D shape estimation from silhouette cues of multiple moving objects in ... more This paper deals with the 3D shape estimation from silhouette cues of multiple moving objects in general 3D scenes, using video streams obtained from multiple calibrated cameras. Most silhouette-based methods usually assume that silhouette information corresponds to an underlying two-classification, based on image regions that match or disagree with a static background appearance model. However this reasoning becomes limited as the number of observed objects increases, because silhouette reasoning then ...

Flot d'occupation 3D à partir de silhouettes latentes

Résumé Nous examinons l'extraction simultanée d'informations de forme et de mouvement, ... more Résumé Nous examinons l'extraction simultanée d'informations de forme et de mouvement, dans le contexte des systemes multi-caméra calibrés. Nous proposons une nouvelle analyse basée sur l'information de silhouette latente dans les images. De nombreuses méthodes utilisent un modele explicite de surface pour une telle analyse. Nous montrons qu'il est possible d'extraire une information pertinente sans modele explicite, en utilisant une méthode EM pour simultanément extraire un ensemble de probabilités sur une grille de ...

3D Object Reconstruction with Heterogeneous Sensor Data

4th International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT), Atlanta, GA, USA, Jun 18, 2008

In this paper, we reconstruct 3D objects with a heterogeneous sensor network of Time of Flight (T... more In this paper, we reconstruct 3D objects with a heterogeneous sensor network of Time of Flight (ToF) Range Imaging (RIM) sensors and high-res camcorders. With this setup, we first carry out a simple but effective depth calibration for the RIM cameras. We then combine the camcorder silhouette cues and RIM camera depth information, for the reconstruction. Our main contribution is the proposal of a sensor fusion framework so that the computation is general, simple and scalable. Although we only discuss the fusion of ...

Pano Popups: Indoor 3D Reconstruction with a Plane-Aware Network

2019 International Conference on 3D Vision (3DV), 2019

In this work we present a method to train a planeaware convolutional neural network for dense dep... more In this work we present a method to train a planeaware convolutional neural network for dense depth and surface normal estimation as well as plane boundaries from a single indoor 360 • image. Using our proposed loss function, our network outperforms existing methods for singleview, indoor, omnidirectional depth estimation and provides an initial benchmark for surface normal prediction from 360 • images. Our improvements are due to the use of a novel plane-aware loss that leverages principal curvature as an indicator of planar boundaries. We also show that including geodesic coordinate maps as network priors provides a significant boost in surface normal prediction accuracy. Finally, we demonstrate how we can combine our network's outputs to generate high quality 3D "pop-up" models of indoor scenes.

Exploring High-Level Plane Primitives for Indoor 3D Reconstruction with a Hand-held RGB-D Camera

Given a hand-held RGB-D camera (e.g. Kinect), methods such as Structure from Motion (SfM) and Ite... more Given a hand-held RGB-D camera (e.g. Kinect), methods such as Structure from Motion (SfM) and Iterative Closest Point (ICP), perform poorly when reconstructing indoor scenes with few image features or little geometric structure information. In this paper, we propose to extract high level primitivesplanes-from an RGB-D camera, in addition to low level image features (e.g. SIFT), to better constrain the problem and help improve indoor 3D reconstruction. Our work has two major contributions: first, for frame to frame matching, we propose a new scheme which takes into account both low-level appearance feature correspondences in RGB image and high-level plane correspondences in depth image. Second, in the global bundle adjustment step, we formulate a novel error measurement that not only takes into account the traditional 3D point re-projection errors, but also the planar surface alignment errors. We demonstrate with real datasets that our method with plane constraints achieves more accurate and more appealing results comparing with other state-of-the-art scene reconstruction algorithms in aforementioned challenging indoor scenarios.

Multi-view Occlusion Reasoning for Probabilistic Silhouette-Based Dynamic Scene Reconstruction

International journal of computer vision, Dec 1, 2010

In this paper, we present an algorithm to probabilistically estimate object shapes in a 3D dynami... more In this paper, we present an algorithm to probabilistically estimate object shapes in a 3D dynamic scene using their silhouette information derived from multiple geometrically calibrated video camcorders. The scene is represented by a 3D volume. Every object in the scene is associated with a distinctive label to represent its existence at every voxel location. The label links together automatically-learned view-specific appearance models of the respective object, so as to avoid the photometric calibration of the cameras. Generative ...

A Real-Time Video Tracking System

by Li Guan and Kedar Patwardhan

IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000

Object identification and tracking applications of pattern recognition at video rates is a proble... more Object identification and tracking applications of pattern recognition at video rates is a problem of wide interest, with previous attempts limited to very simple threshold or correlation (restricted window) methods. New high-speed algorithms together with fast digital hardware have produced a system for missile and aircraft identification and tracking that possesses a degree of "intelligence" not previously implemented in a real-time tracking system. Adaptive statistical clustering and projection-based classification algorithms are applied in real time to identify and track objects that change in appearance through complex and nonstationary background/foreground situations. Fast estimation and prediction algorithms combine linear and quadratic estimators to provide speed and sensitivity. Weights are determined to provide a measure of confi'dence in the data and resulting decisions. Strategies based on maximizing the probability of maintaining track are developed. This paper emphasizes the theoretical aspects of the system and discusses the techniques used to achieve realtime implementation.

Simultaneous Image Segmentation and 3D Plane Fitting for RGB-D Sensors - An Iterative Framework

Abstract In this paper, we segment RGB-D sensor (eg Microsoft Kinect camera) images into 3D plana... more Abstract In this paper, we segment RGB-D sensor (eg Microsoft Kinect camera) images into 3D planar surfaces. We initialize a set of plane equations based solely from the depth (point cloud) information. We then iteratively refine the pixel-to-plane assignment and plane equations. During this process, the number of planes are also reduced by merging adjacent local planes with similar orientations.

Computer Vision Aided Target Linked Radiation Imaging

Abstract In this paper, we demonstrated an application of video tracking to radiation detection, ... more Abstract In this paper, we demonstrated an application of video tracking to radiation detection, where a vision-based tracking system enables a traditional CZT (cadmium zinc telluride)-based radiation imaging device to detect radioactive targets that are in motion. An integrated real-time system consisting of multiple fixed cameras and radiation detectors was implemented and tested.

Sensor-Based Cooperative Multi-Robot 3D Environment Reconstruction

Abstract In this paper, we discuss the task of unknown environment exploration with a team of rob... more Abstract In this paper, we discuss the task of unknown environment exploration with a team of robots. The goal is to reconstruct the 3D scene from onboard sensor data. There are mainly three problems involved: reconstruction, motion planning, and multiple robot collaboration. First we introduce the principal algorithms of reconstruction 3D models from different kinds of vision sensors. Given the 3D model, feature-based localization combined with probability framework can be used to locate each of the robots.

Flot d'occupation 3D à partir de silhouettes latentes

Résumé Nous examinons l'extraction simultanée d'informations de forme et de mouvement, dans le co... more Résumé Nous examinons l'extraction simultanée d'informations de forme et de mouvement, dans le contexte des systemes multi-caméra calibrés. Nous proposons une nouvelle analyse basée sur l'information de silhouette latente dans les images. De nombreuses méthodes utilisent un modele explicite de surface pour une telle analyse.

Multi-view dynamic scene modeling

Modeling dynamic scenes/events from multiple fixed-location vision sensors, such as video camcord... more Modeling dynamic scenes/events from multiple fixed-location vision sensors, such as video camcorders, infrared cameras, Time-of-Flight sensors etc, is of broad interest in computer vision society, with many applications including 3D TV, virtual reality, medical surgery, markerless motion capture, video games, and security surveillance. However, most of the existing multi-view systems are set up in a strictly controlled indoor environment, with fixed lighting conditions and simple background views. Many challenges are preventing the technology to an outdoor natural environment. These include varying sunlight, shadows, reflections, background motion and visual occlusion. In this thesis, I address different aspects to overcome all of the aforementioned difficulties, so as to reduce human preparation and manipulation, and to make a robust outdoor system as automatic as possible. In particular, the main novel technical contributions of this thesis are as follows: a generic heterogeneous ...

Robocup field distortion calibration with variational image registration

Proceedings of 2004 International Symposium on Intelligent Multimedia, Video and Speech Processing, 2004.

Global Vision System is widely used among teams of Robocup F180 (Small Size League). This archite... more Global Vision System is widely used among teams of Robocup F180 (Small Size League). This architecture as well as the 1-hour preparation time limit before each match poses strict requirements to the accuracy and efficiency of the calibration method against pitch field distortions. Because of the curvature-preserving feature of these distortions, we propose a brand-new image registration approach, which does not require calibrating the camera, but directly maps the distorted image to the real pitch. While this method substantially simplifies the pre-match manipulation, it also maintains high accuracyour experimental result shows that, with double camera, the mean error ratio is limited within 0.22%, accurate enough for Robocup matches.

Smart instrumented training ranges: bringing automated system solutions to support critical domain needs

The Journal of Defense Modeling and Simulation: Applications, Methodology, Technology, 2013

The training objective for urban warfare includes acquisition and perfection of a set of diverse ... more The training objective for urban warfare includes acquisition and perfection of a set of diverse skills in support of kinetic and non-kinetic operations. The US Marines (USMC) employ long-duration acted scenarios with verbal training feedback provided sporadically throughout the training session and at the end in a form of an after-action review (AAR). The inherent characteristic of training ranges for urban warfare is that they are the environments with a high level of physical occlusion, which causes many performances not to be seen by a group of instructors who oversee the training. We describe BASE-IT (Behavioral Analysis and Synthesis for Intelligent Training), a system in development that aims to automate capture of training data and their analysis, performance evaluation, and AAR report generation. The goal of this effort is to greatly increase the amount of observed behavior and improve the quality of the AAR. The system observes training with stationary cameras and personal tracking devices. It then analyzes movement and body postures, measures individual and squad-level performance, and compares it to standards and levels of performance expected in given situations. An interactive visualization component delivers live views augmented with real-time analytics and alerts; it also generates a personalized AAR review in a three-dimensional virtual or mixed reality environment, indexed by automatically extracted salient events and accompanied by summary statistics of unit performance. The approaches presented in the system have the potential to radically change the analysis and performance assessment on physical training ranges and ultimately this type of training itself.

Visual Shapes of Silhouette Sets

Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06), 2006

Shape from silhouette methods are extensively used to model dynamic and non-rigid objects using b... more Shape from silhouette methods are extensively used to model dynamic and non-rigid objects using binary foreground-background images. Since the problem of reconstructing shapes from silhouettes is ambiguous, a number of solutions exist and several approaches only consider the one with a maximal volume, called the visual hull. However, the visual hull is not always a good approximation of shapes, in particular when observing smooth surfaces with few cameras. In this paper, we consider instead a class of solutions to the silhouette reconstruction problem that we call visual shapes. Such a class includes the visual hull, but also better approximations of the observed shapes which can take into account local assumptions such as smoothness, among others. Our contributions with respect to existing works is first to identify silhouette consistent shapes different from the visual hull, and second to give a practical way to estimate such shapes in real time. Experiments on various sets of data including human body silhouettes are shown to illustrate the principle and the interests of visual shapes.

A Real-Time Video Tracking System

IEEE Transactions on Pattern Analysis and Machine Intelligence, 1980

Object identification and tracking applications of pattern recognition at video rates is a proble... more Object identification and tracking applications of pattern recognition at video rates is a problem of wide interest, with previous attempts limited to very simple threshold or correlation (restricted window) methods. New high-speed algorithms together with fast digital hardware have produced a system for missile and aircraft identification and tracking that possesses a degree of "intelligence" not previously implemented in a real-time tracking system. Adaptive statistical clustering and projection-based classification algorithms are applied in real time to identify and track objects that change in appearance through complex and nonstationary background/foreground situations. Fast estimation and prediction algorithms combine linear and quadratic estimators to provide speed and sensitivity. Weights are determined to provide a measure of confi'dence in the data and resulting decisions. Strategies based on maximizing the probability of maintaining track are developed. This paper emphasizes the theoretical aspects of the system and discusses the techniques used to achieve realtime implementation. Index Terms-Image processing, intensity histograms, object identification, optical tracking, projections, tracking system, video data compression, video processing, video tracking. velopment of the real-time videotheodolite (RTV). U.S. Army White Sands Missile Range began the development of the RTV in 1974, and the system is being deployed at this time. Design philosophy called for a system capable of discriminatory judgment in identifying the object to be tracked with 60 independent observations/s, capable of locating the center of mass of the object projection on the image plane within about 2 per-Manuscript

Sensor-Based Cooperative Multi-Robot 3D Environment Reconstruction

Master Thesis, Department of Computer Science, University of North Carolina-Chapel Hill, Chapel Hill, NC, USA, Apr 1, 2006

In this paper, we discuss the task of unknown environment exploration with a team of robots. The ... more In this paper, we discuss the task of unknown environment exploration with a team of robots. The goal is to reconstruct the 3D scene from onboard sensor data. There are mainly three problems involved: reconstruction, motion planning, and multiple robot collaboration. First we introduce the principal algorithms of reconstruction 3D models from different kinds of vision sensors. Given the 3D model, feature-based localization combined with probability framework can be used to locate each of the robots. Then the problem is to ...

A unified approach to calibrate a network of camcorders and tof cameras

Abstract. In this paper, we propose a unified calibration technique for a heterogeneous sensor ne... more Abstract. In this paper, we propose a unified calibration technique for a heterogeneous sensor network of video camcorders and Time-of-Flight (ToF) cameras. By moving a spherical calibration target around the commonly observed scene, we can robustly and conveniently extract the sphere centers in the observed images and recover the geometric extrinsics for both types of sensors. The approach is then evaluated with a real dataset of two HD camcorders and two ToF cameras, and 3D shapes are reconstructed from this ...

Multi-object shape estimation and tracking from silhouette cues

2008 IEEE Conference on Computer Vision and Pattern Recognition, 2008

This paper deals with the 3D shape estimation from silhouette cues of multiple moving objects in ... more This paper deals with the 3D shape estimation from silhouette cues of multiple moving objects in general indoor or outdoor 3D scenes with potential static obstacles, using multiple calibrated video streams. Most shape-fromsilhouette techniques use a two-classification of space occupancy and silhouettes, based on image regions that match or disagree with a static background appearance model. Binary silhouette information becomes insufficient to unambiguously carve 3D space regions as the number and density of dynamic objects increases. In such difficult scenes, multi-view stereo methods suffer from visibility problems, and rely on color calibration procedures tedious to achieve outdoors. We propose a new algorithm to automatically detect and reconstruct scenes with a variable number of dynamic objects. Our formulation distinguishes between m different shapes in the scene by using automatically learnt view-specific appearance models, eliminating the color calibration requirement. Bayesian reasoning is then applied to solve the m-shape occupancy problem, with m updated as objects enter or leave the scene. Results show that this method yields multiple silhouette-based estimates that drastically improve scene reconstructions over traditional twolabel silhouette scene analysis. This enables the method to also efficiently deal with multi-person tracking problems.

3D Occlusion Inference from Silhouette Cues

2007 IEEE Conference on Computer Vision and Pattern Recognition, 2007

We consider the problem of detecting and accounting for the presence of occluders in a 3D scene b... more We consider the problem of detecting and accounting for the presence of occluders in a 3D scene based on silhouette cues in video streams obtained from multiple, calibrated views. While well studied and robust in controlled environments, silhouette-based reconstruction of dynamic objects fails in general environments where uncontrolled occlusions are commonplace, due to inherent silhouette corruption by occluders. We show that occluders in the interaction space of dynamic objects can be detected and their 3D shape fully recovered as a byproduct of shape-from-silhouette analysis. We provide a Bayesian sensor fusion formulation to process all occlusion cues occurring in a multi-view sequence. Results show that the shape of static occluders can be robustly recovered from pure dynamic object motion, and that this information can be used for online self-correction and consolidation of dynamic object shape reconstruction.

Multi-Object Shape Estimation from Silhouette Cues

This paper deals with the 3D shape estimation from silhouette cues of multiple moving objects in ... more This paper deals with the 3D shape estimation from silhouette cues of multiple moving objects in general 3D scenes, using video streams obtained from multiple calibrated cameras. Most silhouette-based methods usually assume that silhouette information corresponds to an underlying two-classification, based on image regions that match or disagree with a static background appearance model. However this reasoning becomes limited as the number of observed objects increases, because silhouette reasoning then ...

Flot d'occupation 3D à partir de silhouettes latentes

Résumé Nous examinons l'extraction simultanée d'informations de forme et de mouvement, ... more Résumé Nous examinons l'extraction simultanée d'informations de forme et de mouvement, dans le contexte des systemes multi-caméra calibrés. Nous proposons une nouvelle analyse basée sur l'information de silhouette latente dans les images. De nombreuses méthodes utilisent un modele explicite de surface pour une telle analyse. Nous montrons qu'il est possible d'extraire une information pertinente sans modele explicite, en utilisant une méthode EM pour simultanément extraire un ensemble de probabilités sur une grille de ...

3D Object Reconstruction with Heterogeneous Sensor Data

4th International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT), Atlanta, GA, USA, Jun 18, 2008

In this paper, we reconstruct 3D objects with a heterogeneous sensor network of Time of Flight (T... more In this paper, we reconstruct 3D objects with a heterogeneous sensor network of Time of Flight (ToF) Range Imaging (RIM) sensors and high-res camcorders. With this setup, we first carry out a simple but effective depth calibration for the RIM cameras. We then combine the camcorder silhouette cues and RIM camera depth information, for the reconstruction. Our main contribution is the proposal of a sensor fusion framework so that the computation is general, simple and scalable. Although we only discuss the fusion of ...

Pano Popups: Indoor 3D Reconstruction with a Plane-Aware Network

2019 International Conference on 3D Vision (3DV), 2019

In this work we present a method to train a planeaware convolutional neural network for dense dep... more In this work we present a method to train a planeaware convolutional neural network for dense depth and surface normal estimation as well as plane boundaries from a single indoor 360 • image. Using our proposed loss function, our network outperforms existing methods for singleview, indoor, omnidirectional depth estimation and provides an initial benchmark for surface normal prediction from 360 • images. Our improvements are due to the use of a novel plane-aware loss that leverages principal curvature as an indicator of planar boundaries. We also show that including geodesic coordinate maps as network priors provides a significant boost in surface normal prediction accuracy. Finally, we demonstrate how we can combine our network's outputs to generate high quality 3D "pop-up" models of indoor scenes.

Exploring High-Level Plane Primitives for Indoor 3D Reconstruction with a Hand-held RGB-D Camera

Given a hand-held RGB-D camera (e.g. Kinect), methods such as Structure from Motion (SfM) and Ite... more Given a hand-held RGB-D camera (e.g. Kinect), methods such as Structure from Motion (SfM) and Iterative Closest Point (ICP), perform poorly when reconstructing indoor scenes with few image features or little geometric structure information. In this paper, we propose to extract high level primitivesplanes-from an RGB-D camera, in addition to low level image features (e.g. SIFT), to better constrain the problem and help improve indoor 3D reconstruction. Our work has two major contributions: first, for frame to frame matching, we propose a new scheme which takes into account both low-level appearance feature correspondences in RGB image and high-level plane correspondences in depth image. Second, in the global bundle adjustment step, we formulate a novel error measurement that not only takes into account the traditional 3D point re-projection errors, but also the planar surface alignment errors. We demonstrate with real datasets that our method with plane constraints achieves more accurate and more appealing results comparing with other state-of-the-art scene reconstruction algorithms in aforementioned challenging indoor scenarios.

Multi-view Occlusion Reasoning for Probabilistic Silhouette-Based Dynamic Scene Reconstruction

International journal of computer vision, Dec 1, 2010

In this paper, we present an algorithm to probabilistically estimate object shapes in a 3D dynami... more In this paper, we present an algorithm to probabilistically estimate object shapes in a 3D dynamic scene using their silhouette information derived from multiple geometrically calibrated video camcorders. The scene is represented by a 3D volume. Every object in the scene is associated with a distinctive label to represent its existence at every voxel location. The label links together automatically-learned view-specific appearance models of the respective object, so as to avoid the photometric calibration of the cameras. Generative ...

A Real-Time Video Tracking System

by Li Guan and Kedar Patwardhan

IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000

Object identification and tracking applications of pattern recognition at video rates is a proble... more Object identification and tracking applications of pattern recognition at video rates is a problem of wide interest, with previous attempts limited to very simple threshold or correlation (restricted window) methods. New high-speed algorithms together with fast digital hardware have produced a system for missile and aircraft identification and tracking that possesses a degree of "intelligence" not previously implemented in a real-time tracking system. Adaptive statistical clustering and projection-based classification algorithms are applied in real time to identify and track objects that change in appearance through complex and nonstationary background/foreground situations. Fast estimation and prediction algorithms combine linear and quadratic estimators to provide speed and sensitivity. Weights are determined to provide a measure of confi'dence in the data and resulting decisions. Strategies based on maximizing the probability of maintaining track are developed. This paper emphasizes the theoretical aspects of the system and discusses the techniques used to achieve realtime implementation.

Simultaneous Image Segmentation and 3D Plane Fitting for RGB-D Sensors - An Iterative Framework

Abstract In this paper, we segment RGB-D sensor (eg Microsoft Kinect camera) images into 3D plana... more Abstract In this paper, we segment RGB-D sensor (eg Microsoft Kinect camera) images into 3D planar surfaces. We initialize a set of plane equations based solely from the depth (point cloud) information. We then iteratively refine the pixel-to-plane assignment and plane equations. During this process, the number of planes are also reduced by merging adjacent local planes with similar orientations.

Computer Vision Aided Target Linked Radiation Imaging

Abstract In this paper, we demonstrated an application of video tracking to radiation detection, ... more Abstract In this paper, we demonstrated an application of video tracking to radiation detection, where a vision-based tracking system enables a traditional CZT (cadmium zinc telluride)-based radiation imaging device to detect radioactive targets that are in motion. An integrated real-time system consisting of multiple fixed cameras and radiation detectors was implemented and tested.

Sensor-Based Cooperative Multi-Robot 3D Environment Reconstruction

Abstract In this paper, we discuss the task of unknown environment exploration with a team of rob... more Abstract In this paper, we discuss the task of unknown environment exploration with a team of robots. The goal is to reconstruct the 3D scene from onboard sensor data. There are mainly three problems involved: reconstruction, motion planning, and multiple robot collaboration. First we introduce the principal algorithms of reconstruction 3D models from different kinds of vision sensors. Given the 3D model, feature-based localization combined with probability framework can be used to locate each of the robots.

Flot d'occupation 3D à partir de silhouettes latentes

Résumé Nous examinons l'extraction simultanée d'informations de forme et de mouvement, dans le co... more Résumé Nous examinons l'extraction simultanée d'informations de forme et de mouvement, dans le contexte des systemes multi-caméra calibrés. Nous proposons une nouvelle analyse basée sur l'information de silhouette latente dans les images. De nombreuses méthodes utilisent un modele explicite de surface pour une telle analyse.