*Equal Contribution †Corresponding Author
  |
  |
  |
  |
  |
  |
  |
  |
  |
GeneMAN is a generalizable framework for single-view-to-3D human reconstruction, built on a collection of multi-source human data. Given a single in-the-wild image of a person, GeneMAN could reconstruct a high-quality 3D human model, regardless of its clothing, pose, or body proportions (e.g., a full-body, a half-body, or a close-up shot) in the given image.
This part is the same as original threestudio. Skip it if you already have installed the environment.
See installation.md for additional information, including installation via Docker.
- You must have an NVIDIA graphics card with at least 20GB VRAM and have CUDA installed.
- Install
Python >= 3.8. - (Optional, Recommended) Create a virtual environment:
conda create -n geneman python==3.10
conda activate geneman- Install
PyTorch >= 1.12. We have tested ontorch1.12.1+cu113andtorch2.0.0+cu118, but other versions should also work fine.
# torch1.12.1+cu113
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113
# or torch2.0.0+cu118
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118- (Optional, Recommended) Install ninja to speed up the compilation of CUDA extensions:
pip install ninja- Install dependencies:
git clone https://github.com/3DTopia/GeneMAN
cd GeneMAN
pip install -r requirements.txt-
Download our pre-trained GeneMAN models from HuggingFace.
Copy thepretrained_modelsfolder intoGeneMAN/pretrained_models. Copy thetetsfolder intoGeneMAN/extern. -
Download HumanNorm pretrained models on HuggingFace: Normal-adapted-model, Depth-adapted-model, Normal-aligned-model and ControlNet. Place HumanNorm pretrained models into
GeneMAN/pretrained_models. -
Download required model checkpoints for pre-processing:
- For background removal, download ViT-H SAM model to
GeneMAN/pretrained_models/seg. - YOLO11, BLIP2 and Sapiens models will be downloaded automatically on first use.
After downloading, the GeneMAN folder is structured like:
GeneMAN/
├── extern/
│ └── tets/
├── pretrained_models/
│ ├── normal-adapted-sd1.5/
│ ├── depth-adapted-sd1.5/
│ ├── normal-aligned-sd1.5/
│ ├── controlnet-normal-sd1.5/
│ ├── geneman-prior2d/
│ ├── geneman-prior3d/
│ ├── sapiens/
│ └── seg/
│ ├── sam_vit_h_4b8939.pth
│ └── yolo11x.pt
└── …
Pre-process the human images to remove background and obtain normals, depths, keypoints, and text prompts:
sh script/preprocess.shOur model is trained in multiple stages. Run 3D human reconstruction from a single image:
sh script/run.sh[Note]: We have now switched to the Stage-3 strategy proposed in HumanNorm, based on our prior model.
In our original pipeline, the texture-refinement stage was carried out by refining the UV map. We observed that a small fraction of the results exhibited instability and visible artifacts introduced by our original UV-map texture refinement. As an alternative, we have now switched to the Stage-3 strategy proposed in HumanNorm. If you would like to use UV map texture refine, you can refer to the Stage-3 scheme described in DreamGaussian. We will further enhance this stage in future updates.sh script/export_mesh.sh- Release the code.
- Upload pretrained models.
- Enhance texture refine stage.
Our project benefits from the amazing open-source projects:
We are grateful for their contribution.
If you find this work useful for your research, please consider citing our paper:
@article{wang2024geneman,
title={GeneMAN: Generalizable Single-Image 3D Human Reconstruction from Multi-Source Human Data},
author={Wang, Wentao and Ye, Hang and Hong, Fangzhou and Yang, Xue and Zhang, Jianfu and Wang, Yizhou and Liu, Ziwei and Pan, Liang},
journal={arXiv preprint arXiv:2411.18624},
year={2024}
}