It is not possible to use the Fn::ImportValue Cloud Formation functionality for authorizers.
The feature would be helpful because of the Cloud Formation deletion policy.

There have been some discussions about the import functionality:
#3442
#3475
#3575