Skip to content
This repository was archived by the owner on Apr 8, 2025. It is now read-only.

Refactor Regression and inference processors#702

Merged
Timoeller merged 6 commits intomasterfrom
refactor_processor_dpr-regression-embeddings
Feb 4, 2021
Merged

Refactor Regression and inference processors#702
Timoeller merged 6 commits intomasterfrom
refactor_processor_dpr-regression-embeddings

Conversation

@Timoeller
Copy link
Copy Markdown
Contributor

@Timoeller Timoeller commented Feb 2, 2021

There are quite some changed lines of code, but the overall changes are rather simple:

  1. We are now using inside Regression and InferenceProcessor the TextClassificationProcessor.dataset_from_dicts method and only adjust the way labels are converted
  2. A bugfix for conversion of python float labels to torch tensors. Previously they were silently converted from float to long, we now have to explicitly check for a correct conversion. This might affect other parts in our code as well. @tholor could you check if we have other float values that are converted to torch tensors?
  3. Smaller change in dataset.py for the inferenceprocessing, we now take the whole text as context and not tokens.
  4. I moved QAProcessor into NQProcessor, since it was not used elsewhere

partly closes #676

@Timoeller Timoeller self-assigned this Feb 2, 2021
@Timoeller Timoeller requested a review from tholor February 2, 2021 18:41
@Timoeller Timoeller changed the title WIP: Refactor Regression and inference processors Refactor Regression and inference processors Feb 3, 2021
@Timoeller
Copy link
Copy Markdown
Contributor Author

I now added checks weather data is silently converted to torch.long and tested most of our example scripts. I also added a couple of issues because some example scripts are currently not working.

@tholor please review

Copy link
Copy Markdown
Member

@tholor tholor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good! I would only suggest changing one warning message

@Timoeller Timoeller merged commit f560caf into master Feb 4, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refactor Processor for regression, DPR, plain embeddings

2 participants