Describe the bug
Given a QA_input dictionary with more than one question e.g.:
QA_input = [
{
"qas": ["Did GameTrailers rated Twilight Princess as one of the best games ever created?", "Did GameTrailers rated Twilight Princess as one of the worst games ever created?"],
"context": "Twilight Princess was released to universal critical acclaim and commercial success. It received perfect scores from major publications such as 1UP.com, Computer and Video Games, Electronic Gaming Monthly, Game Informer, GamesRadar, and GameSpy. On the review aggregators GameRankings and Metacritic, Twilight Princess has average scores of 95% and 95 for the Wii version and scores of 95% and 96 for the GameCube version. GameTrailers in their review called it one of the greatest games ever created."
}
]
the returned inference object:
result = model.inference_from_dicts(dicts=QA_input, return_json=False)
is a list that contains just one QApred object:
[<farm.modeling.predictions.QAPred object at 0x7f50dcb73e20>]
that contains just the results for the first question.
Expected behavior
Do return a list of QAPred objects of the same size of the number questions listed in qas (like the "normal" Squad model does).
Additional context
I understand that probably the reason of that is because the original NQ dataset contains just a question for each document, but I think that the API user it expect to receive answers for each question (or at least to be warned that Natural Questions Inference return just one prediction also of there are more questions).
To Reproduce
Modify the QA_input in examples/natural_questions.py adding one or more questions (like above).
Then run examples/natural_questions.py (to save time: the same error is obtained running the script just from point 9 (
|
# 9. Since training on the whole NQ corpus requires substantial compute resources we trained and uploaded a model on s3 |
)
System:
- OS:
- GPU/CPU:
- FARM version: