@ichsan2895 Use the dataset download script to download the fisheye dataset and then run the benchmarking script.

python datasets/download_dataset.py --dataset zipnerf
bash benchmarks/fisheye/mcmc_zipnerf.sh

Amazing work, can't wait to try!

How can we remap our own fisheye images to an ideal fisheye equidistant camera?

I use a Samyang 8mm lens on APS-C, which is very similar to the ZipNeRF dataset (12mm fisheye on full-frame I think) and also circular fisheyes from 360 cameras, which need the black edges masked out.

Thanks!

The dataloader will do it automatically. You just need to use COLMAP to provide the fisheye distortion parameters.
https://colmap.github.io/cameras.html

Support for masks right now is not great. You will need to follow the code and feed your own masks into the training process.

@jefequien I follow your comments,
python datasets/download_dataset.py --dataset zipnerf
bash benchmarks/fisheye/mcmc_zipnerf.sh
The video background after my training and rendering is very dim.
What's going on? Obviously not as bright as the video you provided above, but the rendering result of the image looks normal
alameda_ellipse
nyc_ellipse
berlin_ellipse
london_ellipse
All rendering results look like very dim,

@szhang963 sorry just saw this. It feels normal to me to have black surroundings. That is just how fisheye barrel distortion looks like. Illustration on this page might be helpful:
https://commonlands.com/blogs/technical/wide-angle-lenses-and-fisheye-camera-lens-distortion

@szhang963 I probably don't get why would you like to render with negative z, and unfortunately I probably don't have capacity recently too look into this.

But a quick thought is that we assume z is always positive here

If consider z might be negative (tho i don't know why that is a valid case), we should update that line of code to

 float theta = glm::atan(xy_len, std::abs(z) + eps); 

Maybe you could try this and let me know if it resolve your issue?

are you using the impl from _torch_impl to render it instead of the cuda impl? If you use the rasterization function you need to fix the CUDA side of code.

what would it look like if you render with pinhole?

I don't have any idea on what might cause this. Maybe could start with checking if the same issue exists with pinhole camera by setting camera_model to pinhole?

Thanks for doing the testing! This helps me to identify the issue: converting the depth value into bits for radixsort needs some extra care on the sign bit. should have fixed in #638