Skip to content

convert _upsample_bilinear2d_aa with the antialias filter - #2907

Open
HuzaifaChaudary wants to merge 4 commits into
apple:mainfrom
HuzaifaChaudary:fix/bilinear-antialias
Open

HuzaifaChaudary wants to merge 4 commits into
apple:mainfrom
HuzaifaChaudary:fix/bilinear-antialias

Conversation

@HuzaifaChaudary

Copy link
Copy Markdown

_upsample_bilinear2d_aa was registered as an alias of upsample_bilinear2d , so antialias=True was dropped without a warning . torchvision's Resize uses it by default , so most image models that resize on the way in get it . in torch the two only agree on an upsample without align_corners , every downsample and everything with align_corners=True is different .

this sends the _aa kinds through the separable path from #2818 , with aten's triangle filter in place of the cubic one . the weight code is shared , only the filter and the tap count change .

that was not enough on its own . at an image size the separable path itself comes out wrong on the core ml runtime , for bicubic on main as well . a 512x384 to 224x168 resize was off by up to 0.87 for bicubic , while the same weights applied in numpy match torch to 1e-5 . i narrowed it to the gather per tap , then mul and reduce_sum over the taps . mul alone and reduce_sum alone are exact , the two together are not once the tensor gets past about 65k elements with a short last axis , filed as #2906 :

@mb.program(input_specs=[mb.TensorSpec(shape=(98, 168, 4)), mb.TensorSpec(shape=(98, 168, 4))], opset_version=ct.target.macOS15)
def prog(x, y):
    return mb.reduce_sum(x=mb.mul(x=x, y=y), axes=[-1])

(96, 168, 4) is exact and (98, 168, 4) is off by about 3 , cpu only , fp32 , empty pass pipeline . so the third commit scatters the taps into a dense (out, in) matrix and applies it with one matmul per axis , which is also fewer ops .

torchvision.transforms.v2.Resize((224, 168)) on a 512x384 image in [0, 1] was off by 0.116 on average and up to 0.497 on main . with this it is within 7.3e-06 of torch .

a dynamic shape upsample without align_corners is the same with or without antialias , so it still goes to the plain bilinear lowering , which handles dynamic shapes . a dynamic downsample raises now instead of converting to the wrong answer .

44 of the new antialias cases fail without the converter change , and the image size test fails 4 of 9 with the gather version , plain bicubic included . TestUpsample with INCLUDE_MIL_FP32_UNIT_TESTS=1 is 945 passed , 3 skipped , 99 xfailed .

it was an alias of upsample_bilinear2d , so antialias=True was dropped and the
result was wrong whenever an axis is downsampled , and with align_corners=True
always . it now goes through the same separable _aa path as bicubic , with
ATen's triangle filter . a dynamic shape upsample without align_corners still
takes the plain bilinear lowering , which gives the same result there
output sizes that upsample , downsample and do both , scale factors with and
without align_corners , and a dynamic shape . without the converter change 44
of them fail , all the ones where antialiasing changes the result in torch
the gather per tap , then mul and reduce_sum over the taps , came out wrong on
the core ml runtime once the input is image sized . a 512x384 to 224x168 resize
was off by up to 0.83 for bilinear with antialias and 0.87 for bicubic , while
the same weights applied in numpy match torch to 1e-5 . the taps are scattered
into a dense (out, in) matrix and applied with matmul , which matches torch at
those sizes and is fewer ops
the other cases use 8x8 inputs , which are too small to show the runtime
problem . with the gather version 4 of these fail on the fp32 mlprogram
backend , including plain bicubic

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant