Skip to main content

Ruofei Du

Google, Google.inc, Research Scientist

University of Maryland, College Park, Computer Science, Graduate Student

Followers

8

Following

30

Co-authors

2

Public Views

Ruofei Du is a Research Scientist at Google and works on creating novel interactive technologies for VR and AR. Ruofei holds a Ph.D. in Computer Science from University of Maryland, College Park, advised by Prof. Varshney. Their research covers a wide range of topics in VR and AR, including mixed-reality social platforms (Geollery and Social Street View), 4D video-based rendering (Montage4D and VideoFields), foveated rendering (KFR and EFR), and deep learning in graphics (SketchySceneColorization). I am passionate about inventing interactive technologies with computer graphics, 3D vision, and HCI. Feel free to visit my research, arts, blog, youtube, github, and shadertoy demos for fun!
Supervisors: Amitabh Varshney

less

Marco Di Benedetto

Fabio Ganovelli

Consiglio Nazionale delle Ricerche (CNR)

Marcos Balsa Rodriguez

Consiglio Nazionale delle Ricerche (CNR)

Fraunhofer IGD

Dieter Schmalstieg

Rudolph Pienaar

Harvard University

Swiss Federal Institute of Technology (ETH)

Interests

Uploads

Papers by Ruofei Du

Project Geollery.com: Reconstructing A Live Mirrored World With Geotagged Social Media

Proceedings of the 24th International Conference on Web3D Technology, 2020

Social media in virtual reality is in a high-growth market segment with influential products and ... more Social media in virtual reality is in a high-growth market segment with influential products and services in virtual tourism, remote education, and business meetings. Nevertheless, previous systems have never achieved an online platform which renders a 6DoF mirrored world with geotagged social media in real time. In this paper, we introduce the technical detail behind Geollery.com which reconstructs a mirrored world at two levels of detail. Given a pair of latitude and longitude coordinates, our pipeline streams and caches depth maps, street view panoramas, and building polygons from Google Maps and OpenStreetMap APIs. At a fine level of detail for close-up views, we render textured meshes using adjacent local street views and depth maps. When viewed from afar, we apply projection mappings to 3D geometries extruded from building polygons for a coarse level of detail. In contrast to teleportation, our system allows users to virtually walk through the mirrored world at the street level. Our system integrates geotagged social media from both internal users and external sources such as Twitter, Yelp, and Flicker. We validate our real-time strategies of Geollery.com on various platforms including mobile phones, workstations, and head-mounted displays.

Geollery: A Mixed Reality Social Media Platform

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019

3D buildings with 360°images geotagged social media virtual avatars and live chats geotagged virt... more 3D buildings with 360°images geotagged social media virtual avatars and live chats geotagged virtual gifts geotagged street art geotagged framed photos Geollery.com Figure 1: Geollery creates an interactive mirrored world where users are immersed with 3D buildings, live chats, and geotagged social media. The social media are visualized as balloons, billboards, framed photos, and gift boxes in real time. ABSTRACT We present Geollery, an interactive mixed reality social media platform for creating, sharing, and exploring geotagged information. Geollery introduces a real-time pipeline to progressively render an interactive mirrored world with three-dimensional (3D) buildings, internal user-generated content, and external geotagged social media. This mirrored world allows users to see, chat, and collaborate with remote participants with the same spatial context in an immersive virtual environment. We describe the system architecture of Permission to make digital Geollery, its key interactive capabilities, and our design decisions. Finally, we conduct a user study with 20 participants to qualitatively compare Geollery with another social media system, Social Street View. Based on the participants' responses, we discuss the benefits and drawbacks of each system and derive key insights for designing an interactive mirrored world with geotagged social media. User feedback from our study reveals several use cases for Geollery including travel planning, virtual meetings, and family gathering. CCS CONCEPTS • Human-centered computing Mixed / augmented reality; Virtual reality; Web-based interaction.

Montage4D: Real-time Seamless Fusion and Stylization of Multiview Video Textures

by Ruofei Du, Wayne Chang, and Hugues Hoppe

Journal of Computer Graphics Techniques, 2019

The commoditization of virtual and augmented reality devices and the availability of inexpensive ... more The commoditization of virtual and augmented reality devices and the availability of inexpensive consumer depth cameras have catalyzed a resurgence of interest in spatiotemporal performance capture. Recent systems like Fusion4D and Holoportation address several crucial problems in the real-time fusion of multiview depth maps into volumetric and deformable representations. Nonetheless, stitching multiview video textures onto dynamic meshes remains challenging due to imprecise geometries, occlusion seams, and critical time constraints. In this paper, we present a practical solution for real-time seamless texture montage for dynamic multiview reconstruction. We build on the ideas of dilated depth discontinuities and majority voting from Holoportation to reduce ghosting effects when blending textures. In contrast to that approach, we determine the appropriate blend of textures per vertex using view-dependent rendering techniques, so as to avert fuzziness caused by the ubiquitous normal-weighted blending. By leveraging geodesics-guided diffusion and temporal texture fields, our algorithm mitigates spatial occlusion seams while preserving temporal consistency. Experiments demonstrate significant enhancement in rendering quality, especially in detailed regions such as faces. Furthermore, we present our preliminary exploration of real-time stylization and relighting to empower Holoportation users to interactively stylize live 3D content. We envision a wide range of applications for Montage4D, including immersive telepresence for business, training, and live entertainment.

LUCSS: Language-based User-customized Colorization of Scene Sketches

We introduce LUCSS, a language-based system for interactive col- orization of scene sketches, bas... more We introduce LUCSS, a language-based system for interactive col- orization of scene sketches, based on their semantic understanding. LUCSS is built upon deep neural networks trained via a large-scale repository of scene sketches and cartoon-style color images with text descriptions. It con- sists of three sequential modules. First, given a scene sketch, the segmenta- tion module automatically partitions an input sketch into individual object instances. Next, the captioning module generates the text description with spatial relationships based on the instance-level segmentation results. Fi- nally, the interactive colorization module allows users to edit the caption and produce colored images based on the altered caption. Our experiments show the effectiveness of our approach and the desirability of its compo- nents to alternative choices.

SketchyScene: Richly-Annotated Scene Sketches

European Conference on Computer Vision, 2018

We contribute the first large-scale dataset of scene sketches, SketchyScene, with the goal of adv... more We contribute the first large-scale dataset of scene sketches, SketchyScene, with the goal of advancing research on sketch understanding at both the object and scene level. The dataset is created through a novel and carefully designed crowdsourcing pipeline, enabling users to efficiently generate large quantities of realistic and diverse scene sketches. SketchyScene contains more than 29,000 scene-level sketches, 7,000+ pairs of scene templates and photos, and 11,000+ object sketches. All objects in the scene sketches have ground-truth semantic and instance masks. The dataset is also highly scalable and extensible, easily allowing augmenting and/or changing scene composition. We demonstrate the potential impact of SketchyScene by training new computational models for semantic segmentation of scene sketches and showing how the new dataset enables several applications including image retrieval, sketch colorization, editing, and captioning, etc. The dataset and code can be found at https://github.com/SketchyScene/SketchyScene.

UistViz: 26 Years of UIST Coauthor Network Visualization

In this report, the author built a UIST coauthor network from 1988 to 2013 based on NodeXL. The p... more In this report, the author built a UIST coauthor network from 1988 to 2013 based on NodeXL. The publication data comes from DBLP and the affiliation data comes from Google Scholar. The author proposed four headlines according to the visualization of UIST coauthor network. Finally, the author provides several suggestions to improve NodeXL.

A Pilot Study of Spherical Harmonics for Saliency Computation and Navigation in 360°Videos

A) The input 360°video frame (B) Saliency map by Itti et al.'s model (C) Saliency map by our SSR ... more

Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment

ideo treams amera orld atri tatic 3D odels and atellite mage Video Fields Mapping utomatic egment... more

Project Geollery.com: Reconstructing A Live Mirrored World With Geotagged Social Media

Proceedings of the 24th International Conference on Web3D Technology, 2020

Social media in virtual reality is in a high-growth market segment with influential products and ... more Social media in virtual reality is in a high-growth market segment with influential products and services in virtual tourism, remote education, and business meetings. Nevertheless, previous systems have never achieved an online platform which renders a 6DoF mirrored world with geotagged social media in real time. In this paper, we introduce the technical detail behind Geollery.com which reconstructs a mirrored world at two levels of detail. Given a pair of latitude and longitude coordinates, our pipeline streams and caches depth maps, street view panoramas, and building polygons from Google Maps and OpenStreetMap APIs. At a fine level of detail for close-up views, we render textured meshes using adjacent local street views and depth maps. When viewed from afar, we apply projection mappings to 3D geometries extruded from building polygons for a coarse level of detail. In contrast to teleportation, our system allows users to virtually walk through the mirrored world at the street level. Our system integrates geotagged social media from both internal users and external sources such as Twitter, Yelp, and Flicker. We validate our real-time strategies of Geollery.com on various platforms including mobile phones, workstations, and head-mounted displays.

Geollery: A Mixed Reality Social Media Platform

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019

3D buildings with 360°images geotagged social media virtual avatars and live chats geotagged virt... more 3D buildings with 360°images geotagged social media virtual avatars and live chats geotagged virtual gifts geotagged street art geotagged framed photos Geollery.com Figure 1: Geollery creates an interactive mirrored world where users are immersed with 3D buildings, live chats, and geotagged social media. The social media are visualized as balloons, billboards, framed photos, and gift boxes in real time. ABSTRACT We present Geollery, an interactive mixed reality social media platform for creating, sharing, and exploring geotagged information. Geollery introduces a real-time pipeline to progressively render an interactive mirrored world with three-dimensional (3D) buildings, internal user-generated content, and external geotagged social media. This mirrored world allows users to see, chat, and collaborate with remote participants with the same spatial context in an immersive virtual environment. We describe the system architecture of Permission to make digital Geollery, its key interactive capabilities, and our design decisions. Finally, we conduct a user study with 20 participants to qualitatively compare Geollery with another social media system, Social Street View. Based on the participants' responses, we discuss the benefits and drawbacks of each system and derive key insights for designing an interactive mirrored world with geotagged social media. User feedback from our study reveals several use cases for Geollery including travel planning, virtual meetings, and family gathering. CCS CONCEPTS • Human-centered computing Mixed / augmented reality; Virtual reality; Web-based interaction.

Montage4D: Real-time Seamless Fusion and Stylization of Multiview Video Textures

by Ruofei Du, Wayne Chang, and Hugues Hoppe

Journal of Computer Graphics Techniques, 2019

The commoditization of virtual and augmented reality devices and the availability of inexpensive ... more The commoditization of virtual and augmented reality devices and the availability of inexpensive consumer depth cameras have catalyzed a resurgence of interest in spatiotemporal performance capture. Recent systems like Fusion4D and Holoportation address several crucial problems in the real-time fusion of multiview depth maps into volumetric and deformable representations. Nonetheless, stitching multiview video textures onto dynamic meshes remains challenging due to imprecise geometries, occlusion seams, and critical time constraints. In this paper, we present a practical solution for real-time seamless texture montage for dynamic multiview reconstruction. We build on the ideas of dilated depth discontinuities and majority voting from Holoportation to reduce ghosting effects when blending textures. In contrast to that approach, we determine the appropriate blend of textures per vertex using view-dependent rendering techniques, so as to avert fuzziness caused by the ubiquitous normal-weighted blending. By leveraging geodesics-guided diffusion and temporal texture fields, our algorithm mitigates spatial occlusion seams while preserving temporal consistency. Experiments demonstrate significant enhancement in rendering quality, especially in detailed regions such as faces. Furthermore, we present our preliminary exploration of real-time stylization and relighting to empower Holoportation users to interactively stylize live 3D content. We envision a wide range of applications for Montage4D, including immersive telepresence for business, training, and live entertainment.

LUCSS: Language-based User-customized Colorization of Scene Sketches

We introduce LUCSS, a language-based system for interactive col- orization of scene sketches, bas... more We introduce LUCSS, a language-based system for interactive col- orization of scene sketches, based on their semantic understanding. LUCSS is built upon deep neural networks trained via a large-scale repository of scene sketches and cartoon-style color images with text descriptions. It con- sists of three sequential modules. First, given a scene sketch, the segmenta- tion module automatically partitions an input sketch into individual object instances. Next, the captioning module generates the text description with spatial relationships based on the instance-level segmentation results. Fi- nally, the interactive colorization module allows users to edit the caption and produce colored images based on the altered caption. Our experiments show the effectiveness of our approach and the desirability of its compo- nents to alternative choices.

SketchyScene: Richly-Annotated Scene Sketches

European Conference on Computer Vision, 2018

We contribute the first large-scale dataset of scene sketches, SketchyScene, with the goal of adv... more We contribute the first large-scale dataset of scene sketches, SketchyScene, with the goal of advancing research on sketch understanding at both the object and scene level. The dataset is created through a novel and carefully designed crowdsourcing pipeline, enabling users to efficiently generate large quantities of realistic and diverse scene sketches. SketchyScene contains more than 29,000 scene-level sketches, 7,000+ pairs of scene templates and photos, and 11,000+ object sketches. All objects in the scene sketches have ground-truth semantic and instance masks. The dataset is also highly scalable and extensible, easily allowing augmenting and/or changing scene composition. We demonstrate the potential impact of SketchyScene by training new computational models for semantic segmentation of scene sketches and showing how the new dataset enables several applications including image retrieval, sketch colorization, editing, and captioning, etc. The dataset and code can be found at https://github.com/SketchyScene/SketchyScene.

UistViz: 26 Years of UIST Coauthor Network Visualization

In this report, the author built a UIST coauthor network from 1988 to 2013 based on NodeXL. The p... more In this report, the author built a UIST coauthor network from 1988 to 2013 based on NodeXL. The publication data comes from DBLP and the affiliation data comes from Google Scholar. The author proposed four headlines according to the visualization of UIST coauthor network. Finally, the author provides several suggestions to improve NodeXL.

A Pilot Study of Spherical Harmonics for Saliency Computation and Navigation in 360°Videos

A) The input 360°video frame (B) Saliency map by Itti et al.'s model (C) Saliency map by our SSR ... more

Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment

ideo treams amera orld atri tatic 3D odels and atellite mage Video Fields Mapping utomatic egment... more