Skip to content
This repository was archived by the owner on Apr 8, 2025. It is now read-only.
This repository was archived by the owner on Apr 8, 2025. It is now read-only.

Natural Questions Inference return just one prediction result if there are more than one questions  #551

@ftesser

Description

@ftesser

Describe the bug
Given a QA_input dictionary with more than one question e.g.:

 QA_input = [
        {
            "qas": ["Did GameTrailers rated Twilight Princess as one of the best games ever created?", "Did GameTrailers rated Twilight Princess as one of the worst games ever created?"],
            "context":  "Twilight Princess was released to universal critical acclaim and commercial success. It received perfect scores from major publications such as 1UP.com, Computer and Video Games, Electronic Gaming Monthly, Game Informer, GamesRadar, and GameSpy. On the review aggregators GameRankings and Metacritic, Twilight Princess has average scores of 95% and 95 for the Wii version and scores of 95% and 96 for the GameCube version. GameTrailers in their review called it one of the greatest games ever created."
        }
    ]

the returned inference object:

result = model.inference_from_dicts(dicts=QA_input, return_json=False)

is a list that contains just one QApred object:

[<farm.modeling.predictions.QAPred object at 0x7f50dcb73e20>]

that contains just the results for the first question.

Expected behavior
Do return a list of QAPred objects of the same size of the number questions listed in qas (like the "normal" Squad model does).

Additional context
I understand that probably the reason of that is because the original NQ dataset contains just a question for each document, but I think that the API user it expect to receive answers for each question (or at least to be warned that Natural Questions Inference return just one prediction also of there are more questions).

To Reproduce
Modify the QA_input in examples/natural_questions.py adding one or more questions (like above).

Then run examples/natural_questions.py (to save time: the same error is obtained running the script just from point 9 (

# 9. Since training on the whole NQ corpus requires substantial compute resources we trained and uploaded a model on s3
)

System:

  • OS:
  • GPU/CPU:
  • FARM version:

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingenhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions