Issue with TestCamFromImgToImg in sensor/models_test.cc for Fisheye models #2802
Description
Describe the bug
In models_test.cc, line 171, we test the unprojection and projection of pixels for various camera models. However, this test fails under certain conditions for Fisheye camera models. For pixel coordinates, whose theta values are greater than pi/4, the test fails.
To Reproduce
Consider the following code snippet. I have selected some camera params for the OpenCVFisheye model. This mini test generates a grid - the pixel coordinates in the white area represent a match between original value and reprojected value. The black region represents a mismatch after reprojection. It also prints the minimum 10 theta outliers where the forward and inverse transformations of the camera model do not match.
import pycolmap
import math
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from tqdm import tqdm
def test_fisheye_model():
camera_id = 1
width = 640
height = 480
opencv_fisheye_params = [
241.996, 252.123, 323.024, 240.497, -0.0232, 0.0924, -0.0591, 0.003
]
fx, fy = opencv_fisheye_params[0], opencv_fisheye_params[1]
cx, cy = opencv_fisheye_params[2], opencv_fisheye_params[3]
ocv_camera = pycolmap.Camera(model='OPENCV_FISHEYE',
width=width,
height=height,
params=opencv_fisheye_params,
camera_id=camera_id)
xs, ys = np.meshgrid(np.arange(0, 800), np.arange(0, 800))
dists = np.stack([(xs - cx) / fx, (ys - cy) / fy], axis=2)
thetas = np.linalg.norm(dists, axis=2).transpose()
bw_grid = np.zeros((800, 800)).astype(bool)
for i in tqdm(range(0, 800, 1)):
for j in range(0, 800, 1):
if thetas[i, j] >= math.pi / 2:
continue
pixel = np.array([i, j])
point_colmap = ocv_camera.cam_from_img(pixel)
reproj_point = ocv_camera.img_from_cam(point_colmap)
if np.allclose(pixel, reproj_point):
bw_grid[i, j] = True
else:
bw_grid[i, j] = False
print("Passed: {0:d} / {1:d}".format(bw_grid.sum(), np.array(thetas < math.pi / 2).sum()))
theta_outliers = np.sort(thetas[~bw_grid])
print("Outliers (min 10): ", theta_outliers[:10])
plt.imshow(bw_grid, cmap='gray', origin='lower', vmin=0, vmax=1)
plt.title('Black = mismatch, White = match')
plt.savefig('./grid.png')
def main():
test_fisheye_model()
if __name__ == '__main__':
main()Expected behavior
The camera model's forward and inverse transformations remain consistent up to an angular distance of about 45 degrees.
Screenshots
