Add possibility to do nested cross validation#508
Add possibility to do nested cross validation#508Timoeller merged 14 commits intodeepset-ai:masterfrom
Conversation
|
After some offline discussion with @PhilipMay we are going forward with the nested Crossvalidation. We need a way to handle the variance in performance that comes from different restarts (be it different seeds or slightly different training data). This PR adds a solution in a structured way, so glad to have it in FARM. |
|
An other plus for this nested cross validation is that the train and dev datasets are also created in a stratified way. |
PhilipMay
force-pushed
the
nested_xval
branch
2 times, most recently
from
9c9a968 to
d694bb4
Compare
|
Rebased on master. |
Timoeller
left a comment
Timoeller
left a comment
There was a problem hiding this comment.
This is looking well but could be a bit better integrated into existing code.
I made some comments along the way - open to discussion
PhilipMay
force-pushed
the
nested_xval
branch
from
d694bb4 to
776d5b8
Compare
|
@Timoeller I made your suggested changes. I had a look on nested cross validation for question answering. Maybe you could just add this later when you / someone else wants / needs it? IMO this is good to go to be merged. |
Timoeller
left a comment
Timoeller
left a comment
There was a problem hiding this comment.
Looking good, thanks for integrating the proposed changes.
2 participants
This adds the possibility to do nested cross validation.
The PR introduces a new class
DataSiloForNestedCrossValthat inherits fromDataSiloForCrossVal.It does nested cross validation:
This is related to #507
Todo
add quest. answering