Streaming Volumetric Video IOS Viewer for V3: Viewing Volumetric Videos on Mobiles via Streamable 2D Dynamic Gaussians

Official implementation of the streaming volumetric video viewer for V^3: Viewing Volumetric Videos on Mobiles via Streamable 2D Dynamic Gaussians.

Penghao Wang*, Zhirui Zhang*, Liao Wang*, Kaixin Yao, Siyuan Xie, Jingyi Yu†, Minye Wu†, Lan Xu†

SIGGRAPH Asia 2024 (ACM Transactions on Graphics)

| Webpage | Paper | Video | Training Code | SIBR Viewer Code | IOS Viewer Code |

Installation

Dependencies

You need to download opencv2.framework and put it in the project folder. You can get the latest version in https://opencv.org/releases/.

Project setup

First, you need to put the data folder in the folder of MetalSplat. We provide one checkpoint for testing. We have put two json files in MetalSplat folder and you need to download the video checkpoint from this link and put it to the same folder.

Usage

Please compile the project in Xcode and build the project to your IOS device.

Control

First, click the cover to load the volumetric video. Then you can use the following gestures to control the viewer:

• click the play button to play the video
• click the pause button to pause the video
• one finger for rotation
• two fingers for scaling and translation
• drag the slider to change the frame index
• click the back button to go back to the cover

Code illustration

We implement a compute shader in SplatShaders.metal for fast conversion from 2D image to 3D splats.

Streamable version

You can put the data folder on a server in the same network and change the path of data for loading streamable volumetric video.

Acknowledgement

This viewer is based on the metal splat viewer.

If you found our work useful, please kindly cite our paper:

@article{wang2024v,
  title={V\^{} 3: Viewing Volumetric Videos on Mobiles via Streamable 2D Dynamic Gaussians},
  author={Wang, Penghao and Zhang, Zhirui and Wang, Liao and Yao, Kaixin and Xie, Siyuan and Yu, Jingyi and Wu, Minye and Xu, Lan},
  journal={ACM Transactions on Graphics (TOG)},
  volume={43},
  number={6},
  pages={1--13},
  year={2024},
  publisher={ACM New York, NY, USA}
}