Skip to main content

Zilong Dong

Followers

0

Following

1

Public Views

Vasileios Gkitsas

Petros Drakoulis

University of NSW at Canberra

Université Laval

University of North Carolina at Chapel Hill

University of Surrey

Interests

Uploads

Papers by Zilong Dong

$Research paper thumbnail of $\mathcal {S}^{2}$Net: Accurate Panorama Depth Estimation on Spherical Surface$

$\mathcal {S}^{2}$Net: Accurate Panorama Depth Estimation on Spherical Surface

IEEE Robotics and Automation Letters

Monocular depth estimation is an ambiguous problem, thus global structural cues play an important... more Monocular depth estimation is an ambiguous problem, thus global structural cues play an important role in current data-driven single-view depth estimation methods. Panorama images capture the complete spatial information of their surroundings utilizing the equirectangular projection which introduces large distortion. This requires the depth estimation method to be able to handle the distortion and extract global context information from the image. In this paper, we propose an end-to-end deep network for monocular panorama depth estimation on a unit spherical surface. Specifically, we project the feature maps extracted from equirectangular images onto unit spherical surface sampled by uniformly distributed grids, where the decoder network can aggregate the information from the distortion-reduced feature maps. Meanwhile, we propose a global cross-attention-based fusion module to fuse the feature maps from skip connection and enhance the ability to obtain global context. Experiments are conducted on five panorama depth estimation datasets, and the results demonstrate that the proposed method substantially outperforms previous state-ofthe-art methods. All related codes will be open-sourced in the upcoming days.

RenderNet: Visual Relocalization Using Virtual Viewpoints in Large-Scale Indoor Environments

arXiv (Cornell University), Jul 25, 2022

Visual relocalization has been a widely discussed problem in 3D vision: given a pre-constructed 3... more Visual relocalization has been a widely discussed problem in 3D vision: given a pre-constructed 3D visual map, the 6 DoF (Degrees-of-Freedom) pose of a query image is estimated. Relocalization in large-scale indoor environments enables attractive applications such as augmented reality and robot navigation. However, appearance changes fast in such environments when the camera moves, which is challenging for the relocalization system. To address this problem, we propose a virtual view synthesis-based approach, RenderNet, to enrich the database and refine poses regarding this particular scenario. Instead of rendering real images which requires high-quality 3D models, we opt to directly render the needed global and local features of virtual viewpoints and apply them in the subsequent image retrieval and feature matching operations respectively. The proposed method can largely improve the performance in large-scale indoor environments, e.g., achieving an improvement of 7.1% and 12.2% on the Inloc dataset.

AR Mapping: Accurate and Efficient Mapping for Augmented Reality

arXiv (Cornell University), Mar 27, 2021

Augmented reality (AR) has gained increasingly attention from both research and industry communit... more Augmented reality (AR) has gained increasingly attention from both research and industry communities. By overlaying digital information and content onto the physical world, AR enables users to experience the world in a more informative and efficient manner. As a major building block for AR systems, localization aims at determining the device's pose from a pre-built "map" consisting of visual and depth information in a known environment. While the localization problem has been widely studied in the literature, the "map" for AR systems is rarely discussed. In this paper, we introduce the AR Map for a specific scene to be composed of 1) color images with 6-DOF poses; 2) dense depth maps for each image and 3) a complete point cloud map. We then propose an efficient end-to-end solution to generating and evaluating AR Maps. Firstly, for efficient data capture, a backpack scanning device is presented with a unified calibration pipeline. Secondly, we propose an AR mapping pipeline which takes the input from the scanning device and produces accurate AR Maps. Finally, we present an approach to evaluating the accuracy of AR Maps with the help of the highly accurate reconstruction result from a high-end laser scanner. To the best of our knowledge, it is the first time to present an end-to-end solution to efficient and accurate mapping for AR applications.

ENFT: Efficient Non-Consecutive Feature Tracking for Robust Structure-from-Motion

Structure-from-motion (SfM) largely relies on feature tracking. In image sequences, if disjointed... more Structure-from-motion (SfM) largely relies on feature tracking. In image sequences, if disjointed tracks caused by objects moving in and out of the field of view, occasional occlusion, or image noise, are not handled well, corresponding SfM could be affected. This problem becomes severer for large-scale scenes, which typically requires to capture multiple sequences to cover the whole scene. In this paper, we propose an efficient non-consecutive feature tracking (ENFT) framework to match interrupted tracks distributed in different subsequences or even in different videos. Our framework consists of steps of solving the feature `dropout' problem when indistinctive structures, noise or large image distortion exists, and of rapidly recognizing and joining common features located in different subsequences. In addition, we contribute an effective segment-based coarse-to-fine SfM algorithm for robustly handling large datasets. Experimental results on challenging video data demonstrate t...

Single-Shot is Enough: Panoramic Infrastructure Based Calibration of Multiple Cameras and 3D LiDARs

2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021

The integration of multiple cameras and 3D Li-DARs has become basic configuration of augmented re... more The integration of multiple cameras and 3D Li-DARs has become basic configuration of augmented reality devices, robotics, and autonomous vehicles. The calibration of multi-modal sensors is crucial for a system to properly function, but it remains tedious and impractical for mass production. Moreover, most devices require re-calibration after usage for certain period of time. In this paper, we propose a single-shot solution for calibrating extrinsic transformations among multiple cameras and 3D LiDARs. We establish a panoramic infrastructure, in which a camera or LiDAR can be robustly localized using data from single frame. Experiments are conducted on three devices with different camera-LiDAR configurations, showing that our approach achieved comparable calibration accuracy with the state-of-the-art approaches but with much greater efficiency.

UniFuse: Unidirectional Fusion for 360° Panorama Depth Estimation

IEEE Robotics and Automation Letters, 2021

Learning depth from spherical panoramas is becoming a popular research topic because a panorama h... more Learning depth from spherical panoramas is becoming a popular research topic because a panorama has a full field-of-view of the environment and provides a relatively complete description of a scene. However, applying well-studied CNNs for perspective images to the standard representation of spherical panoramas, i.e., the equirectangular projection, is suboptimal, as it becomes distorted towards the poles. Another representation is the cubemap projection, which is distortionfree but discontinued on edges and limited in the field-ofview. This paper introduces a new framework to fuse features from the two projections, unidirectionally feeding the cubemap features to the equirectangular features only at the decoding stage. Unlike the recent bidirectional fusion approach operating at both the encoding and decoding stages, our fusion scheme is much more efficient. Besides, we also designed a more effective fusion module for our fusion scheme. Experiments verify the effectiveness of our proposed fusion strategy and module, and our model achieves state-of-the-art performance on four popular datasets. Additional experiments show that our model also has the advantages of model complexity and generalization capability. The code is available at https://github.com/ alibaba/UniFuse-Unidirectional-Fusion.

Keyframe-based real-time camera tracking

2009 IEEE 12th International Conference on Computer Vision, 2009

We present a novel keyframe selection and recognition method for robust markerless real-time came... more We present a novel keyframe selection and recognition method for robust markerless real-time camera tracking. Our system contains an offline module to select features from a group of reference images and an online module to match them to the input live video in order to quickly estimate the camera pose. The main contribution lies in constructing an optimal set of keyframes from the input reference images, which are required to approximately cover the entire space and at the same time minimize the content redundancy amongst the selected frames. This strategy not only greatly saves the computation, but also helps significantly reduce the number of repeated features so as to improve the camera tracking quality. Our system also employs a parallel-computing scheme with multi-CPU hardware architecture. Experimental results show that our method dramatically enhances the computation efficiency and eliminates the jittering artifacts.

MRStudio: A mixed reality display system for aircraft cockpit

2011 IEEE International Symposium on VR Innovation, 2011

... As a result, the user's head position and orientation is tracked. 5.1 Hardware settings ... more

Efficient Non-consecutive Feature Tracking for Structure-from-Motion

Computer Vision – ECCV 2010, 2010

Structure-from-motion (SfM) is an important computer vision problem and largely relies on the qua... more Structure-from-motion (SfM) is an important computer vision problem and largely relies on the quality of feature tracking. In image sequences, if disjointed tracks caused by objects moving in and out of the view, occasional occlusion, or image noise, are not handled well, the corresponding SfM could be significantly affected. In this paper, we address the non-consecutive feature point tracking problem and propose an effective method to match interrupted tracks. Our framework consists of steps of solving the feature 'dropout' problem when indistinctive structures, noise or even large image distortion exist, and of rapidly recognizing and joining common features located in different subsequences. Experimental results on several challenging and large-scale video sets show that our method notably improves SfM.

Interactive weathering of depth-inferred videos

Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry, 2011

Aging or weathering is an important technique for generating natural images in computer graphics.... more Aging or weathering is an important technique for generating natural images in computer graphics. In this paper, we propose a novel video weathering method which can synthesize the weathering effects for the real captured videos. We first recover the depth maps of the input video, and employ a depth-based interactive video object cutout method to efficiently extract the target objects. Then, we sample the 3D point cloud from the recovered depth maps within the specified object mask, and reconstruct the 3D geometry model. In order to make the obtained 3D point cloud accurate with small redundancy, we select the pixels with the priority of depth accuracy and project each sample to other frames to avoid reselection. In the weathering stage, with the reconstructed 3D mesh, we employ γ-ton tracing to create the weathering map, and then compute the correspondence point for each object pixel on the weathering map by intersecting with the 3D mesh, which is used to resemble the weathering value to blend the texture with the original pixel color. We also introduce two interactive tools which allow the user to guide the process of weathering. A number of weathering effects are shown in the experimental results to demonstrate the robustness and efficiency of the proposed method.

Synthesizing trees from samples

In this paper, we propose a novel tree modeling ap- proach, synthesizing new trees from samples. ... more In this paper, we propose a novel tree modeling ap- proach, synthesizing new trees from samples. First, we capture real world trees as samples by image-based modeling or laser scanning techniques. Then, we pre- sent a two-level statistical tree model and design a maximum likelihood estimation algorithm to extract it from samples. At the low level in the tree model, groups of similar organs are clustered to depict tree organ de- tails statistically. At the high level, a set of transitions between clusters is outlined to describe the stochastic distribution of organs. Experimental results show that our two-level model extracted from samples is capable of synthesizing new trees similar, yet visually different.

Depth-Varying Human Video Sprite Synthesis

Lecture Notes in Computer Science, 2012

Video texture is an appealing method to extract and replay natural human motion from video shots.... more Video texture is an appealing method to extract and replay natural human motion from video shots. There have been much research on video texture analysis, generation and interactive control. However, the video sprites created by existing methods are typically restricted to constant depths, so that the motion diversity is strongly limited. In this paper, we propose a novel depth-varying human video sprite synthesis method, which significantly increases the degrees of freedom of human video sprite. A novel image distance function encoding scale variation is proposed, which can effectively measure the human snapshots with different depths/scales and poses, so that aligning similar poses with different depths is possible. The transitions among non-consecutive frames are modeled as a 2D transformation matrix, which can effectively avoid drifting without leveraging markers or user intervention. The synthesized depth-varying human video sprites can be seamlessly inserted into new scenes for realistic video composition. A variety of challenging examples demonstrate the effectiveness of our method.

Robust monocular SLAM in dynamic environments

2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2013

We present a novel real-time monocular SLAM system which can robustly work in dynamic environment... more We present a novel real-time monocular SLAM system which can robustly work in dynamic environments. Different to the traditional methods, our system allows parts of the scene to be dynamic or the whole scene to gradually change. The key contribution is that we propose a novel online keyframe representation and updating method to adaptively model the dynamic environments, where the appearance or structure changes can be effectively detected and handled. We reliably detect the changed features by projecting them from the keyframes to current frame for appearance and structure comparison. The appearance change due to occlusions also can be reliably detected and handled. The keyframes with large changed areas will be replaced by newly selected frames. In addition, we propose a novel prior-based adaptive RANSAC algorithm (PARSAC) to efficiently remove outliers even when the inlier ratio is rather low, so that the camera pose can be reliably estimated even in very challenging situations. Experimental results demonstrate that the proposed system can robustly work in dynamic environments and outperforms the state-of-the-art SLAM systems (e.g. PTAM).

Adaptive voxels: interactive rendering of massive 3D models

The Visual Computer, 2010

We present a novel approach for interactive rendering of massive 3D models. Our approach integrat... more We present a novel approach for interactive rendering of massive 3D models. Our approach integrates adaptive sampling-based simplification, visibility culling, out-ofcore data management and level-of-detail. We use a unified scene graph representation for acceleration techniques. In preprocessing, we subdivide large objects, and build a BVH clustering hierarchy. We make use of a novel adaptive sampling method to generate LOD models: AdaptiveVoxels. The AdaptiveVoxels reduces the preprocessing cost and our outof-core rendering algorithm improves rendering efficiency. We have implemented our algorithm on a desktop PC. We can render massive CAD and isosurface models, consisting of hundreds of millions of triangles interactively with little loss in image quality.

Synthesizing trees by plantons

The Visual Computer, 2006

In this paper, we present a two-level statistical model for characterizing the stochastic and spe... more In this paper, we present a two-level statistical model for characterizing the stochastic and specific nature of trees. At the low level, we define plantons, which are a group of similar organs, to depict tree organ details statistically. At the high level, a set of transitions between plantons is provided to describe the stochastic distribution of organs. Based on such a tree model, we propose a novel tree modeling approach, synthesizing trees by plantons, which are extracted from tree samples. All tree samples are captured from the real world. We have designed a maximum likelihood estimation algorithm to acquire the two-level statistical tree model from single samples or multi-samples. Experimental results show that our new model is capable of synthesizing new trees with similar, yet visually different shapes. Keywords Tree modeling • Modeling from samples • Markov model • Image-based modeling HUJUN BAO received his Bachelor and PhD degrees in Applied Mathematics from Zhejiang University in 1987 and 1993. His research interests include modeling and rendering techniques for large scale of virtual environments and their applications. He is currently the director of State Key Laboratory of CAD & CG of Zhejiang University. He is also the principal investigator of a virtual reality project sponsored by Ministry of Science and Technology of China.

Refilming with Depth-Inferred Videos

IEEE Transactions on Visualization and Computer Graphics, 2009

Compared to still image editing, content-based video editing faces the additional challenges of m... more Compared to still image editing, content-based video editing faces the additional challenges of maintaining the spatiotemporal consistency with respect to geometry. This brings up difficulties of seamlessly modifying video content, for instance, inserting or removing an object. In this paper, we present a new video editing system for creating spatio-temporally consistent and visually appealing re-filming effects. Unlike the typical filming practice, our system requires no labor-intensive construction of 3D models/surfaces mimicking the real scene. Instead, it is based on an unsupervised inference of view-dependent depth maps for all video frames. We provide interactive tools requiring only a small amount of user input to perform elementary video content editing, such as separating video layers, completing background scene, and extracting moving objects. These tools can be utilized to produce a variety of visual effects in our system, including but not limited to video composition, "predator" effect, bullet-time, depth-of-field, and fog synthesis. Some of the effects can be achieved in real-time.

Live Video Montage with a Rotating Camera

Computer Graphics Forum, 2009

High-quality video editing usually requires accurate layer separation in order to resolve occlusi... more High-quality video editing usually requires accurate layer separation in order to resolve occlusions. However, most of the existing bilayer segmentation algorithms require either considerable user intervention or a simple stationary camera configuration with known background, which is difficult to meet for many real world online applications. This paper demonstrates that various visually appealing montage effects can be online created from a live video captured by a rotating camera, by accurately retrieving the camera state and segmenting out the dynamic foreground. The key contribution is that a novel fast bilayer segmentation method is proposed which can effectively extract the dynamic foreground under rotational camera configuration, and is robust to imperfect background estimation and complex background colors. Our system can create a variety of live visual effects, including but not limited to, realistic virtual object insertion, background substitution and blurring, non-photorealistic rendering and camouflage effect. A variety of challenging examples demonstrate the effectiveness of our method.

Chinese-character-marker Based Augmented Reality System

Augmented reality is a technique to fuse virtual and real worlds. Using an off-the-shelf web came... more Augmented reality is a technique to fuse virtual and real worlds. Using an off-the-shelf web camera, the proposed augmented reality system can augment related virtual media information onto the Chinese-Character-Markers surrounded by black rectangles, which provides a new learning experience for Chinese character. The proposed method enhances the marker detection method with edge detection, and performs better under complicated shading condition. To reliably solve the camera parameters, the proposed system first uses the homography to estimate the initial values, and then incorporate adaptive smoothness constraint to reduce the image noise. This constraint makes the solved camera parameters more accurate and stable. The experimental results demonstrate the effectiveness and robustness of the proposed system.

Camera tracking based on structure and motion recovery for augmented video

Robust camera tracking plays a key role in augmented video. This paper proposes an efficient and ... more Robust camera tracking plays a key role in augmented video. This paper proposes an efficient and robust approach to structure and motion recovery for long video sequences with varying and unknown focal length. In this approach, a long sequence is abstracted as a sequence of key frames in-between which have long baselines in order to assure the preciseness of the solution. The sequence of the key frames are resolved incrementally in order to recover the structure of 3D points, by which the camera motion of all frames of the sequence is retrieved. The algorithm begins with three key frames suitable for initializing the sequential structure and motion computation, and the projective structure is upgraded to metric one in time though self-calibration. The implemented examples demonstrate very precise structure and motion recovery, and prove the efficiency and robustness of the proposed method.

Efficient keyframe-based real-time camera tracking

Computer Vision and Image Understanding, 2014

We present a novel keyframe-based global localization method for markerless real-time camera trac... more We present a novel keyframe-based global localization method for markerless real-time camera tracking. Our system contains an offline module to select features from a group of reference images and an online module to match them to the input live video for quickly estimating the camera pose. The main contribution lies in constructing an optimal set of keyframes from the input reference images, which are required to approximately cover the entire space and at the same time to minimize the content redundancy among the selected frames. This strategy not only greatly saves computation, but also helps significantly reduce the number of repeated features. For a large-scale scene, it requires a significant effort to capture sufficient reference images and reconstruct the 3D environment. In order to alleviate the effort of offline preprocessing and enhance the tracking ability in a larger scale scene, we also propose an online reference map extension module, which can real-time reconstruct new 3D features and select online keyframes to extend the keyframe set. In addition, we develop a parallel-computing framework that employs both GPUs and multi-threading for speedup. Experimental results show that our method dramatically enhances the computing efficiency and eliminates the jittering artifacts in real-time camera tracking.

$Research paper thumbnail of $\mathcal {S}^{2}$Net: Accurate Panorama Depth Estimation on Spherical Surface$

$\mathcal {S}^{2}$Net: Accurate Panorama Depth Estimation on Spherical Surface

IEEE Robotics and Automation Letters

Monocular depth estimation is an ambiguous problem, thus global structural cues play an important... more Monocular depth estimation is an ambiguous problem, thus global structural cues play an important role in current data-driven single-view depth estimation methods. Panorama images capture the complete spatial information of their surroundings utilizing the equirectangular projection which introduces large distortion. This requires the depth estimation method to be able to handle the distortion and extract global context information from the image. In this paper, we propose an end-to-end deep network for monocular panorama depth estimation on a unit spherical surface. Specifically, we project the feature maps extracted from equirectangular images onto unit spherical surface sampled by uniformly distributed grids, where the decoder network can aggregate the information from the distortion-reduced feature maps. Meanwhile, we propose a global cross-attention-based fusion module to fuse the feature maps from skip connection and enhance the ability to obtain global context. Experiments are conducted on five panorama depth estimation datasets, and the results demonstrate that the proposed method substantially outperforms previous state-ofthe-art methods. All related codes will be open-sourced in the upcoming days.

RenderNet: Visual Relocalization Using Virtual Viewpoints in Large-Scale Indoor Environments

arXiv (Cornell University), Jul 25, 2022

Visual relocalization has been a widely discussed problem in 3D vision: given a pre-constructed 3... more Visual relocalization has been a widely discussed problem in 3D vision: given a pre-constructed 3D visual map, the 6 DoF (Degrees-of-Freedom) pose of a query image is estimated. Relocalization in large-scale indoor environments enables attractive applications such as augmented reality and robot navigation. However, appearance changes fast in such environments when the camera moves, which is challenging for the relocalization system. To address this problem, we propose a virtual view synthesis-based approach, RenderNet, to enrich the database and refine poses regarding this particular scenario. Instead of rendering real images which requires high-quality 3D models, we opt to directly render the needed global and local features of virtual viewpoints and apply them in the subsequent image retrieval and feature matching operations respectively. The proposed method can largely improve the performance in large-scale indoor environments, e.g., achieving an improvement of 7.1% and 12.2% on the Inloc dataset.

AR Mapping: Accurate and Efficient Mapping for Augmented Reality

arXiv (Cornell University), Mar 27, 2021

Augmented reality (AR) has gained increasingly attention from both research and industry communit... more Augmented reality (AR) has gained increasingly attention from both research and industry communities. By overlaying digital information and content onto the physical world, AR enables users to experience the world in a more informative and efficient manner. As a major building block for AR systems, localization aims at determining the device's pose from a pre-built "map" consisting of visual and depth information in a known environment. While the localization problem has been widely studied in the literature, the "map" for AR systems is rarely discussed. In this paper, we introduce the AR Map for a specific scene to be composed of 1) color images with 6-DOF poses; 2) dense depth maps for each image and 3) a complete point cloud map. We then propose an efficient end-to-end solution to generating and evaluating AR Maps. Firstly, for efficient data capture, a backpack scanning device is presented with a unified calibration pipeline. Secondly, we propose an AR mapping pipeline which takes the input from the scanning device and produces accurate AR Maps. Finally, we present an approach to evaluating the accuracy of AR Maps with the help of the highly accurate reconstruction result from a high-end laser scanner. To the best of our knowledge, it is the first time to present an end-to-end solution to efficient and accurate mapping for AR applications.

ENFT: Efficient Non-Consecutive Feature Tracking for Robust Structure-from-Motion

Structure-from-motion (SfM) largely relies on feature tracking. In image sequences, if disjointed... more Structure-from-motion (SfM) largely relies on feature tracking. In image sequences, if disjointed tracks caused by objects moving in and out of the field of view, occasional occlusion, or image noise, are not handled well, corresponding SfM could be affected. This problem becomes severer for large-scale scenes, which typically requires to capture multiple sequences to cover the whole scene. In this paper, we propose an efficient non-consecutive feature tracking (ENFT) framework to match interrupted tracks distributed in different subsequences or even in different videos. Our framework consists of steps of solving the feature `dropout' problem when indistinctive structures, noise or large image distortion exists, and of rapidly recognizing and joining common features located in different subsequences. In addition, we contribute an effective segment-based coarse-to-fine SfM algorithm for robustly handling large datasets. Experimental results on challenging video data demonstrate t...

Single-Shot is Enough: Panoramic Infrastructure Based Calibration of Multiple Cameras and 3D LiDARs

2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021

The integration of multiple cameras and 3D Li-DARs has become basic configuration of augmented re... more The integration of multiple cameras and 3D Li-DARs has become basic configuration of augmented reality devices, robotics, and autonomous vehicles. The calibration of multi-modal sensors is crucial for a system to properly function, but it remains tedious and impractical for mass production. Moreover, most devices require re-calibration after usage for certain period of time. In this paper, we propose a single-shot solution for calibrating extrinsic transformations among multiple cameras and 3D LiDARs. We establish a panoramic infrastructure, in which a camera or LiDAR can be robustly localized using data from single frame. Experiments are conducted on three devices with different camera-LiDAR configurations, showing that our approach achieved comparable calibration accuracy with the state-of-the-art approaches but with much greater efficiency.

UniFuse: Unidirectional Fusion for 360° Panorama Depth Estimation

IEEE Robotics and Automation Letters, 2021

Learning depth from spherical panoramas is becoming a popular research topic because a panorama h... more Learning depth from spherical panoramas is becoming a popular research topic because a panorama has a full field-of-view of the environment and provides a relatively complete description of a scene. However, applying well-studied CNNs for perspective images to the standard representation of spherical panoramas, i.e., the equirectangular projection, is suboptimal, as it becomes distorted towards the poles. Another representation is the cubemap projection, which is distortionfree but discontinued on edges and limited in the field-ofview. This paper introduces a new framework to fuse features from the two projections, unidirectionally feeding the cubemap features to the equirectangular features only at the decoding stage. Unlike the recent bidirectional fusion approach operating at both the encoding and decoding stages, our fusion scheme is much more efficient. Besides, we also designed a more effective fusion module for our fusion scheme. Experiments verify the effectiveness of our proposed fusion strategy and module, and our model achieves state-of-the-art performance on four popular datasets. Additional experiments show that our model also has the advantages of model complexity and generalization capability. The code is available at https://github.com/ alibaba/UniFuse-Unidirectional-Fusion.

Keyframe-based real-time camera tracking

2009 IEEE 12th International Conference on Computer Vision, 2009

We present a novel keyframe selection and recognition method for robust markerless real-time came... more We present a novel keyframe selection and recognition method for robust markerless real-time camera tracking. Our system contains an offline module to select features from a group of reference images and an online module to match them to the input live video in order to quickly estimate the camera pose. The main contribution lies in constructing an optimal set of keyframes from the input reference images, which are required to approximately cover the entire space and at the same time minimize the content redundancy amongst the selected frames. This strategy not only greatly saves the computation, but also helps significantly reduce the number of repeated features so as to improve the camera tracking quality. Our system also employs a parallel-computing scheme with multi-CPU hardware architecture. Experimental results show that our method dramatically enhances the computation efficiency and eliminates the jittering artifacts.

MRStudio: A mixed reality display system for aircraft cockpit

2011 IEEE International Symposium on VR Innovation, 2011

... As a result, the user's head position and orientation is tracked. 5.1 Hardware settings ... more

Efficient Non-consecutive Feature Tracking for Structure-from-Motion

Computer Vision – ECCV 2010, 2010

Structure-from-motion (SfM) is an important computer vision problem and largely relies on the qua... more Structure-from-motion (SfM) is an important computer vision problem and largely relies on the quality of feature tracking. In image sequences, if disjointed tracks caused by objects moving in and out of the view, occasional occlusion, or image noise, are not handled well, the corresponding SfM could be significantly affected. In this paper, we address the non-consecutive feature point tracking problem and propose an effective method to match interrupted tracks. Our framework consists of steps of solving the feature 'dropout' problem when indistinctive structures, noise or even large image distortion exist, and of rapidly recognizing and joining common features located in different subsequences. Experimental results on several challenging and large-scale video sets show that our method notably improves SfM.

Interactive weathering of depth-inferred videos

Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry, 2011

Aging or weathering is an important technique for generating natural images in computer graphics.... more Aging or weathering is an important technique for generating natural images in computer graphics. In this paper, we propose a novel video weathering method which can synthesize the weathering effects for the real captured videos. We first recover the depth maps of the input video, and employ a depth-based interactive video object cutout method to efficiently extract the target objects. Then, we sample the 3D point cloud from the recovered depth maps within the specified object mask, and reconstruct the 3D geometry model. In order to make the obtained 3D point cloud accurate with small redundancy, we select the pixels with the priority of depth accuracy and project each sample to other frames to avoid reselection. In the weathering stage, with the reconstructed 3D mesh, we employ γ-ton tracing to create the weathering map, and then compute the correspondence point for each object pixel on the weathering map by intersecting with the 3D mesh, which is used to resemble the weathering value to blend the texture with the original pixel color. We also introduce two interactive tools which allow the user to guide the process of weathering. A number of weathering effects are shown in the experimental results to demonstrate the robustness and efficiency of the proposed method.

Synthesizing trees from samples

In this paper, we propose a novel tree modeling ap- proach, synthesizing new trees from samples. ... more In this paper, we propose a novel tree modeling ap- proach, synthesizing new trees from samples. First, we capture real world trees as samples by image-based modeling or laser scanning techniques. Then, we pre- sent a two-level statistical tree model and design a maximum likelihood estimation algorithm to extract it from samples. At the low level in the tree model, groups of similar organs are clustered to depict tree organ de- tails statistically. At the high level, a set of transitions between clusters is outlined to describe the stochastic distribution of organs. Experimental results show that our two-level model extracted from samples is capable of synthesizing new trees similar, yet visually different.

Depth-Varying Human Video Sprite Synthesis

Lecture Notes in Computer Science, 2012

Video texture is an appealing method to extract and replay natural human motion from video shots.... more Video texture is an appealing method to extract and replay natural human motion from video shots. There have been much research on video texture analysis, generation and interactive control. However, the video sprites created by existing methods are typically restricted to constant depths, so that the motion diversity is strongly limited. In this paper, we propose a novel depth-varying human video sprite synthesis method, which significantly increases the degrees of freedom of human video sprite. A novel image distance function encoding scale variation is proposed, which can effectively measure the human snapshots with different depths/scales and poses, so that aligning similar poses with different depths is possible. The transitions among non-consecutive frames are modeled as a 2D transformation matrix, which can effectively avoid drifting without leveraging markers or user intervention. The synthesized depth-varying human video sprites can be seamlessly inserted into new scenes for realistic video composition. A variety of challenging examples demonstrate the effectiveness of our method.

Robust monocular SLAM in dynamic environments

2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2013

We present a novel real-time monocular SLAM system which can robustly work in dynamic environment... more We present a novel real-time monocular SLAM system which can robustly work in dynamic environments. Different to the traditional methods, our system allows parts of the scene to be dynamic or the whole scene to gradually change. The key contribution is that we propose a novel online keyframe representation and updating method to adaptively model the dynamic environments, where the appearance or structure changes can be effectively detected and handled. We reliably detect the changed features by projecting them from the keyframes to current frame for appearance and structure comparison. The appearance change due to occlusions also can be reliably detected and handled. The keyframes with large changed areas will be replaced by newly selected frames. In addition, we propose a novel prior-based adaptive RANSAC algorithm (PARSAC) to efficiently remove outliers even when the inlier ratio is rather low, so that the camera pose can be reliably estimated even in very challenging situations. Experimental results demonstrate that the proposed system can robustly work in dynamic environments and outperforms the state-of-the-art SLAM systems (e.g. PTAM).

Adaptive voxels: interactive rendering of massive 3D models

The Visual Computer, 2010

We present a novel approach for interactive rendering of massive 3D models. Our approach integrat... more We present a novel approach for interactive rendering of massive 3D models. Our approach integrates adaptive sampling-based simplification, visibility culling, out-ofcore data management and level-of-detail. We use a unified scene graph representation for acceleration techniques. In preprocessing, we subdivide large objects, and build a BVH clustering hierarchy. We make use of a novel adaptive sampling method to generate LOD models: AdaptiveVoxels. The AdaptiveVoxels reduces the preprocessing cost and our outof-core rendering algorithm improves rendering efficiency. We have implemented our algorithm on a desktop PC. We can render massive CAD and isosurface models, consisting of hundreds of millions of triangles interactively with little loss in image quality.

Synthesizing trees by plantons

The Visual Computer, 2006

In this paper, we present a two-level statistical model for characterizing the stochastic and spe... more In this paper, we present a two-level statistical model for characterizing the stochastic and specific nature of trees. At the low level, we define plantons, which are a group of similar organs, to depict tree organ details statistically. At the high level, a set of transitions between plantons is provided to describe the stochastic distribution of organs. Based on such a tree model, we propose a novel tree modeling approach, synthesizing trees by plantons, which are extracted from tree samples. All tree samples are captured from the real world. We have designed a maximum likelihood estimation algorithm to acquire the two-level statistical tree model from single samples or multi-samples. Experimental results show that our new model is capable of synthesizing new trees with similar, yet visually different shapes. Keywords Tree modeling • Modeling from samples • Markov model • Image-based modeling HUJUN BAO received his Bachelor and PhD degrees in Applied Mathematics from Zhejiang University in 1987 and 1993. His research interests include modeling and rendering techniques for large scale of virtual environments and their applications. He is currently the director of State Key Laboratory of CAD & CG of Zhejiang University. He is also the principal investigator of a virtual reality project sponsored by Ministry of Science and Technology of China.

Refilming with Depth-Inferred Videos

IEEE Transactions on Visualization and Computer Graphics, 2009

Compared to still image editing, content-based video editing faces the additional challenges of m... more Compared to still image editing, content-based video editing faces the additional challenges of maintaining the spatiotemporal consistency with respect to geometry. This brings up difficulties of seamlessly modifying video content, for instance, inserting or removing an object. In this paper, we present a new video editing system for creating spatio-temporally consistent and visually appealing re-filming effects. Unlike the typical filming practice, our system requires no labor-intensive construction of 3D models/surfaces mimicking the real scene. Instead, it is based on an unsupervised inference of view-dependent depth maps for all video frames. We provide interactive tools requiring only a small amount of user input to perform elementary video content editing, such as separating video layers, completing background scene, and extracting moving objects. These tools can be utilized to produce a variety of visual effects in our system, including but not limited to video composition, "predator" effect, bullet-time, depth-of-field, and fog synthesis. Some of the effects can be achieved in real-time.

Live Video Montage with a Rotating Camera

Computer Graphics Forum, 2009

High-quality video editing usually requires accurate layer separation in order to resolve occlusi... more High-quality video editing usually requires accurate layer separation in order to resolve occlusions. However, most of the existing bilayer segmentation algorithms require either considerable user intervention or a simple stationary camera configuration with known background, which is difficult to meet for many real world online applications. This paper demonstrates that various visually appealing montage effects can be online created from a live video captured by a rotating camera, by accurately retrieving the camera state and segmenting out the dynamic foreground. The key contribution is that a novel fast bilayer segmentation method is proposed which can effectively extract the dynamic foreground under rotational camera configuration, and is robust to imperfect background estimation and complex background colors. Our system can create a variety of live visual effects, including but not limited to, realistic virtual object insertion, background substitution and blurring, non-photorealistic rendering and camouflage effect. A variety of challenging examples demonstrate the effectiveness of our method.

Chinese-character-marker Based Augmented Reality System

Augmented reality is a technique to fuse virtual and real worlds. Using an off-the-shelf web came... more Augmented reality is a technique to fuse virtual and real worlds. Using an off-the-shelf web camera, the proposed augmented reality system can augment related virtual media information onto the Chinese-Character-Markers surrounded by black rectangles, which provides a new learning experience for Chinese character. The proposed method enhances the marker detection method with edge detection, and performs better under complicated shading condition. To reliably solve the camera parameters, the proposed system first uses the homography to estimate the initial values, and then incorporate adaptive smoothness constraint to reduce the image noise. This constraint makes the solved camera parameters more accurate and stable. The experimental results demonstrate the effectiveness and robustness of the proposed system.

Camera tracking based on structure and motion recovery for augmented video

Robust camera tracking plays a key role in augmented video. This paper proposes an efficient and ... more Robust camera tracking plays a key role in augmented video. This paper proposes an efficient and robust approach to structure and motion recovery for long video sequences with varying and unknown focal length. In this approach, a long sequence is abstracted as a sequence of key frames in-between which have long baselines in order to assure the preciseness of the solution. The sequence of the key frames are resolved incrementally in order to recover the structure of 3D points, by which the camera motion of all frames of the sequence is retrieved. The algorithm begins with three key frames suitable for initializing the sequential structure and motion computation, and the projective structure is upgraded to metric one in time though self-calibration. The implemented examples demonstrate very precise structure and motion recovery, and prove the efficiency and robustness of the proposed method.

Efficient keyframe-based real-time camera tracking

Computer Vision and Image Understanding, 2014

We present a novel keyframe-based global localization method for markerless real-time camera trac... more We present a novel keyframe-based global localization method for markerless real-time camera tracking. Our system contains an offline module to select features from a group of reference images and an online module to match them to the input live video for quickly estimating the camera pose. The main contribution lies in constructing an optimal set of keyframes from the input reference images, which are required to approximately cover the entire space and at the same time to minimize the content redundancy among the selected frames. This strategy not only greatly saves computation, but also helps significantly reduce the number of repeated features. For a large-scale scene, it requires a significant effort to capture sufficient reference images and reconstruct the 3D environment. In order to alleviate the effort of offline preprocessing and enhance the tracking ability in a larger scale scene, we also propose an online reference map extension module, which can real-time reconstruct new 3D features and select online keyframes to extend the keyframe set. In addition, we develop a parallel-computing framework that employs both GPUs and multi-threading for speedup. Experimental results show that our method dramatically enhances the computing efficiency and eliminates the jittering artifacts in real-time camera tracking.