Thanks @tholor for PR, we need latest changes because of many bug fixes related to RAG.
https://github.com/huggingface/transformers/releases/tag/v3.4.0

Seems that there's a bug in transformers for saving/loading do_lower_case attribute of the Tokenizers: huggingface/transformers#8001

We should not upgrade before we have a clear fix / workaround.

@tholor possible to use BertTokenizerFast

from transformers import BertTokenizerFast

tokenizer = BertTokenizerFast.from_pretrained("bert-base-cased")
print(tokenizer.do_lower_case)


tokenizer.save_pretrained("debug_tokenizer")

tokenizer_loaded = BertTokenizerFast.from_pretrained("debug_tokenizer")
print(tokenizer_loaded.do_lower_case)

# Print
# False
# False

And this one

from transformers import BertTokenizerFast, BertConfig

tokenizer = BertTokenizerFast.from_pretrained("bert-base-cased", do_lower_case=True)
print(tokenizer.do_lower_case)


tokenizer.save_pretrained("debug_tokenizer")

tokenizer_loaded = BertTokenizerFast.from_pretrained("debug_tokenizer")
print(tokenizer_loaded.do_lower_case)

# Print
# True
# True

yeah, but the slow ones are failing! We can release FARM with such a nasty bug. Many deployments still rely on the slow tokenizers and not everybody can install the fast Tokenizers due to rust issues.

Agree, I think better wait for fix or workaround.
Anyway Haystack RAG can work with 3.3.1

It's fixed now upstream in transformers: huggingface/transformers#8001

What do you suggest about release of RAG -

• Wait for new transformers release ( >3.4.0) then release new FARM version
  OR
• Release FARM with transformers (3.3.1) then release another FARM version later.

We are currently preparing the release. Just a few more PRs we want to get in.
The plan is to release tomorrow or Friday. If transformers doesn't do a release before that, we'll go with 3.3.1 for now

We'll skip transformers 3.4.0 and wait for the next release that has a fix for the above mentioned lowercase bug.