Better messaging and fix for incorrect shape when collating data.#18119
Conversation
…sting from DatasetDict
|
The documentation is not available anymore as the PR was closed or merged. |
…nd updated docs accordingly
sgugger
left a comment
sgugger
left a comment
There was a problem hiding this comment.
Thanks for your PR. I left some comments, as some of the asserts introduced will break training at batch size 1 which people use for large models.
| np_input_ids = np.array(feature["input_ids"]) | ||
| assert ( | ||
| np_input_ids.shape[0] != 1 | ||
| ), f""" | ||
| Feature shape cannot be collated: expected shape=( > 1, ... ), found shape={str(np_input_ids.shape)}. | ||
| You may want to remove excessively nested layers from your features. | ||
| """ |
There was a problem hiding this comment.
This will prevent any user from running training at batch size 1, so this should not be added.
| self[key] = tensor | ||
| except: # noqa E722 | ||
| if key == "overflowing_tokens": | ||
| if key == "overflowing_tokens" or key == "input_ids" or key == "attention_mask": |
There was a problem hiding this comment.
Please do not touch this part of the test or the error raised. You can dd the information on the key in the error raised at line 736.
There was a problem hiding this comment.
I am trying to formulate an explanation to add at line 736 but I'm led back to saying what is already written at line 731. The reason that excessive nesting is an issue at collating is that even with padding the features are tokenized at different lengths. This is due to the additional wrapping of the actual feature list like so: [[1,2]] and [[1,2,3]]. They both share the shape (1, but after that they are different shapes so what np.array does is: [[ list ]] and [[ list ]] thus returning shape(1, 1).
That is why I added the code at 730 because it technically is the perfect explanation at this point in the process.
3 participants
What does this PR do?
I ran into an error related to an incorrect shape of inputs when using DataCollatorForSeq2Seq. I learned it had to do with having excessively nested inputs for my features. The error message was not particularly useful.
This PR adds an assertion checking for incorrectly shaped inputs to be collated. The assertion also provides a solution by suggesting to use
remove_excess_nestingutil.remove_excess_nestingremoves excessive nesting from features within aDatasetDict.Fixes #15505
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@stas00