This repository provides the official implementation of Painting with 3D Gaussian Splat Brushes. The first tool for painting in real-time with real-world texture-geometry content, represented as 3D Gaussian splats. It allows artists to source both their brushes and canvases from in-the-wild captures, and swiftly remix highly realistic content through direct and controllable painting.
Painting with 3D Gaussian Splat Brushes
Karran Pandey, Anita Hu, Or Perel, Clement Fuji-Tsang, Karan Singh, Masha Shugrina
SIGGRAPH North America 2025 (Conference Track)
Project page / Paper / Video
Our creative tool supports painting with 3D gaussian splats. Specifically, our UI allows users to:
- Interactively create, save and load 3D brushes with portions of gsplat scenes
- Paint coherent 3D brush strokes, or cloned stamps, on surfaces in the scene
First set up a conda environment:
git clone https://github.com/JonathonLuiten/diff-gaussian-rasterization-w-depth.git
cd diff-gaussian-rasterization-w-depth
python setup.py install
conda create -n splat-painting python=3.9
conda activate splat-painting
Then install pytorch 2.x for your cuda version via pip, following instructions on https://pytorch.org/get-started/locally/. For e.g, for CUDA 12.1
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121
Then install remaining requirements via pip
pip install --no-build-isolation -r requirements.txt
In addition to this, we need a modified differential gaussian rasterization that outputs depth, currently sourced from here: https://github.com/JonathonLuiten/diff-gaussian-rasterization-w-depth
One can follow the installation instructions in the repo to set it up, i.e
git clone https://github.com/JonathonLuiten/diff-gaussian-rasterization-w-depth.git
cd diff-gaussian-rasterization-w-depth
python setup.py install
To run the interface do:
python main.py --scene <path-to-gsplat-scene-ply> --brush <path-to-brush-folder>
The scene ply file here is the 'canvas' or the scene on which we wish to paint, and the brush folder is the folder containing all the brushes we would like to try.
Given pre-loaded brushes, one can select a brush from the drop down and proceed to paint in the scene by holding CTRL as they hover the mouse over a surface. There are two painting modes currently:
- Stroke - This mode deforms the brush stamp and creates a coherent stroke with the stamps as the user paints.
- Clone - This mode clones and places the stamps following the users mouse.
The blend percentage sets the maximum overlap a new stamp can have with the previous one as the user paints.
Jitter controls introduce scale and in-plane rotation jitter to the brush stamp while painting.
Creating a brush is currently based on selecting a portion of the scene with a middle-mouse drag.
There are two modes for selection
- conn-comps: This attempts to select the closest connected component based on the dragged selection. One can set the threshold for connectedness using the associated slider in the UI.
- iterative-bbox: This is an iterative bounding box selection with modifiers to perform set operations (ALT - remove, SHIFT - add) to select precise portions with if its difficult to define a nice connectivity threshold interactively.
Having selected the portion one can click on setup brush to set it up as the current brush.
The brush has two transformations associated with it which can be modified in the setup window.
- Scale: The scale of the brush patch itself - one can press 'Scale Brush' to enter a transform mode and then use the gizmo to edit the canonical 3D transform of the brush
- Orientation: The three directions that define the stroke tangent, binormal and normal when painting strokes - press 'Orient Brush' to adjust these directions - the green arrows in the gizmo correspond to the normal, blue to the binormal and red to the tangent.
Brushes can be saved and loaded with their canonical transform and orientation (defined by the three directions mentioned above). Enter the file path relative to the current working directory in the designated text box and click on 'save brush' / 'load brush' to save or load brushes.
An invisible plane proxy can also be created in a scene with normal and center at the current hover position (with the brush enabled) using the 'shift' button when key bindings are enabled and surface proxy is enabled.
Some sample scenes and brushes to load with the program can be found here: https://drive.google.com/drive/folders/1pKtOlOniDksSGTztmfWg9VfILAHlRmtt?usp=sharing
This repository is currently not accepting contributions.
All code under deps/gsplats is under the Inria and MPII non-commercial license. All code in the repository not under deps/gsplats is under the Apache 2.0 License.
Thus, the entire repository is limited to non-commercial research and evaluation use.
@article{pandey2025painting,
author = {Pandey, Karran and Hu, Anita and Fuji-Tsang, Clement and Perel, Or and Singh, Karan and Shugrina, Maria},
title = {Painting with 3D Gaussian Splat Brushes},
booktitle = {ACM SIGGRAPH 2025 Conference Papers},
pages = {}
year = {2025},
}