The documentation is not available anymore as the PR was closed or merged.

Thanks for working on this! Does the new implementation in Pytorch produce the exact same results as gelu_fast? If that is the case, I would prefer we just replace the current gelu_fast with this when PyTorch is 1.12 or above.

Thanks for working on this! Does the new implementation in Pytorch produce the exact same results as gelu_fast? If that is the case, I would prefer we just replace the current gelu_fast with this when PyTorch is 1.12 or above.

The results are similar but there are still rounding errors, see my analysis in the related issue #21344. I would also be in favor of replacing the existing implementation / using it as default, but I would introduce small numerical differences in some models, is that a problem?

Ah yes, the difference is quite significant sadly, so this will probably introduce a difference that is too big :-/
So let's go with a new activation. Maybe gelu_pytorch is a better name?

Ah yes, the difference is quite significant sadly, so this will probably introduce a difference that is too big :-/ So let's go with a new activation. Maybe gelu_pytorch is a better name?

Wouldn't it cause confusion with the default pytorch implementation? That one is currently named "gelu". (And the one named "gelu_python").

Also should I add an explicit pytorch version check?

Ok for the name then. For the version check, you will need to create a function that returns the instance of GELU and issues an import error if the PyTorch version is too low, then put that function in the mappinh.

Ok for the name then. For the version check, you will need to create a function that returns the instance of GELU and issues an import error if the PyTorch version is too low, then put that function in the mappinh.

Made a class to match the other activations, and raising a NotImplementedError (I don't think an ImportError is the best here since the function exists in earlier versions.) Also added to test_get_activation.

Failure is unrelated so merging. Thanks again for your contribution!