Thanks for the PR. CodeBert looks really interesting to us. Especially the code search functionality.

But does the code run for all cases of codebert?
I see model = RobertaForMaskedLM.from_pretrained('microsoft/codebert-base-mlm') which is different than RobertaModel.

So the code search functionality seems like textpairclassification, which is possible with current FARM and for this use case we should be able to load codebert.

The MLM objective for "code docstring generation" is something we would need to implement in FARM.

What are your thoughts on that?

I only checked the model for classification because I was interested to use it for retrieval (actually I used haystack).

The docstring generation would be pretty limited given that there is only model for MLM (I don't think it can handle variable length output), and Microsoft doesn't plan to release model that can be actually used for text generation (see this issue)

If you have spare compute I someone (for example me) could actually finetune CodeBERT for docstring generation, but for now at least it's pretty hard given that it was trained on 4 P40 GPUs and the data preprocessing step doesn't fit in 32GB RAM.

Nice, I am fine with just going for the textpairclassification case.

Will it create problems if we load the microsoft/codebert-base-mlm model without the mlm head? If so I would rather exclude this model from being loaded, e.g.

elif 'codebert' in pretrained_model_name_or_path.lower():
    if "mlm" in pretrained_model_name_or_path.lower():
        raise NotImplementedError("MLM part of codebert is currently not supported in FARM")
    else:
        ....

I've checked the microsoft/codebert-base-mlm model and it works fine.

Hey, thanks a lot for checking. I would still prefer to exclude the mlm model from loading, because it will confuse people loading mlm and doing classification.
Would you like to adjust the code or should I do it?

Changed.

I guess something went wrong with mail reply?

In my experience it is nearly always best to use the github UI for answering...

I removed AutoModel functionalities.