Hi,

inspired by huggingface/transformers#4257 I just had a look at the original ELECTRA implementation, and it seems that they're using gelu instead of tanh for getting a pooled output:

https://github.com/google-research/electra/blob/79111328070e491b287c307906701ebc61091eb2/model/modeling.py#L247

and:

https://github.com/google-research/electra/blob/79111328070e491b287c307906701ebc61091eb2/model/modeling.py#L45

Unfortunately, I used tanh in the ELECTRA model integration:

I will open a PR to fix that :)