In our prototype transforms tests, we have the ability to set closeness_kwargs
|
# Additional parameters, e.g. `rtol=1e-3`, passed to `assert_close`. |
|
self.closeness_kwargs = closeness_kwargs or dict() |
that will be passed to assert_close in addition to the values to be compared:
|
assert_close(actual, expected, **info.closeness_kwargs) |
However, there is currently no option to specify the closeness_kwargs more specifically. Meaning, they will be applied to all tests regardless of the dtype and device. Especially for the dtype, this is not sufficient: images in torch.uint8 might be allowed an absolute tolerance of 1 since that is a minor difference. However, applying the same tolerance to an torch.float32 image would mean that any difference in the valid value range of [0, 1] would be fine and thus rendering the test almost useless. Thus, we use the stricter torch.float32 tolerance here, but that leads to flaky tests #6741.
We already have the TestMark functionality in, that allows us to apply pytest.mark.* to specific tests based on the inputs. We should extend this to also specify the tolerances or even the whole closeness_kwargs.
cc @vfdev-5 @datumbox @bjuncek
In our prototype transforms tests, we have the ability to set
closeness_kwargsvision/test/prototype_common_utils.py
Lines 637 to 638 in 11a2eed
that will be passed to
assert_closein addition to the values to be compared:vision/test/test_prototype_transforms_functional.py
Line 94 in 11a2eed
However, there is currently no option to specify the
closeness_kwargsmore specifically. Meaning, they will be applied to all tests regardless of thedtypeanddevice. Especially for thedtype, this is not sufficient: images intorch.uint8might be allowed an absolute tolerance of1since that is a minor difference. However, applying the same tolerance to antorch.float32image would mean that any difference in the valid value range of[0, 1]would be fine and thus rendering the test almost useless. Thus, we use the strictertorch.float32tolerance here, but that leads to flaky tests #6741.We already have the
TestMarkfunctionality in, that allows us to applypytest.mark.*to specific tests based on the inputs. We should extend this to also specify the tolerances or even the wholecloseness_kwargs.cc @vfdev-5 @datumbox @bjuncek