Hi,

this PR adds the previously introduced ELECTRA pre-training approach into FARM:

The ELECTRA model was proposed in the paper: ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. ELECTRA is a new pre-training approach which trains two transformer models: the generator and the discriminator. The generator’s role is to replace tokens in a sequence, and is therefore trained as a masked language model. The discriminator, which is the model we’re interested in, tries to identify which tokens were replaced by the generator in the sequence.

Implementation notes

The Hugging Face Transformer library was updated to the latest 2.8 version to support ELECTRA model. Like DistilBERT, an additional pooler needs to be defined to get a one vector per sequence representation.

Experiments

I did pre-liminary experiments with CoNLL-2003 for NER. The configuration can be found under experiments/electra_eval/conll2003_en_config.json.

Result for one run on using the ELECTRA base model: 94.30% (dev) and 89.86% (test).