Skip to content

Added support for RankingMetrics with CrossValSummaryRunner#5386

Merged
mstfbl merged 6 commits intodotnet:masterfrom
mstfbl:Issue_5381
Sep 10, 2020
Merged

Added support for RankingMetrics with CrossValSummaryRunner#5386
mstfbl merged 6 commits intodotnet:masterfrom
mstfbl:Issue_5381

Conversation

@mstfbl
Copy link
Copy Markdown
Contributor

@mstfbl mstfbl commented Sep 4, 2020

Fix #5381

Previously, AutoFitRankingTest was never testing RankingMetrics with CrossValSummaryRunner due to the row limit for cross validation testing, and as the MLSR dataset that trainDataView is pulling from is large:

machinelearning/test/Microsoft.ML.AutoML.Tests/AutoFitTests.cs

Lines 123 to 141 in f87a3bb

[LightGBMFact]
public void AutoFitRankingTest()
{
string labelColumnName = "Label";
string scoreColumnName = "Score";
string groupIdColumnName = "GroupId";
string featuresColumnVectorNameA = "FeatureVectorA";
string featuresColumnVectorNameB = "FeatureVectorB";
var mlContext = new MLContext(1);
// STEP 1: Load data
var reader = new TextLoader(mlContext, GetLoaderArgsRank(labelColumnName, groupIdColumnName, featuresColumnVectorNameA, featuresColumnVectorNameB));
var trainDataView = reader.Load(new MultiFileSource(DatasetUtil.GetMLSRDataset()));
var testDataView = mlContext.Data.TakeRows(trainDataView, 500);
trainDataView = mlContext.Data.SkipRows(trainDataView, 500);
// STEP 2: Run AutoML experiment
var experiment = mlContext.Auto()
.CreateRankingExperiment(5);

machinelearning/src/Microsoft.ML.AutoML/API/ExperimentBase.cs

Lines 109 to 130 in 6bae29f

public ExperimentResult<TMetrics> Execute(IDataView trainData, ColumnInformation columnInformation,
IEstimator<ITransformer> preFeaturizer = null, IProgress<RunDetail<TMetrics>> progressHandler = null)
{
// Cross val threshold for # of dataset rows --
// If dataset has < threshold # of rows, use cross val.
// Else, run experiment using train-validate split.
const int crossValRowCountThreshold = 15000;
var rowCount = DatasetDimensionsUtil.CountRows(trainData, crossValRowCountThreshold);
var samplingKeyColumnName = GetSamplingKey(columnInformation?.GroupIdColumnName, columnInformation?.SamplingKeyColumnName);
if (rowCount < crossValRowCountThreshold)
{
const int numCrossValFolds = 10;
var splitResult = SplitUtil.CrossValSplit(Context, trainData, numCrossValFolds, samplingKeyColumnName);
return ExecuteCrossValSummary(splitResult.trainDatasets, columnInformation, splitResult.validationDatasets, preFeaturizer, progressHandler);
}
else
{
var splitResult = SplitUtil.TrainValidateSplit(Context, trainData, samplingKeyColumnName);
return ExecuteTrainValidate(splitResult.trainData, columnInformation, splitResult.validationData, preFeaturizer, progressHandler);
}
}

As a result, this lacking of support for RankingMetrics with CrossValSummaryRunner was not noticed until Issue #5381. This PR adds support for this case, and edits the RankingMetrics AutoML unit tests to include testing of RankingMetrics with CrossValSummaryRunner as well.

@mstfbl mstfbl requested a review from a team as a code owner September 4, 2020 11:00
@mstfbl mstfbl mentioned this pull request Sep 4, 2020
Closed
@mstfbl mstfbl requested a review from justinormont September 4, 2020 11:02
@codecov
Copy link
Copy Markdown

codecov Bot commented Sep 4, 2020 •
edited
Loading

Codecov Report

Merging #5386 into master will increase coverage by 0.06%.
The diff coverage is 61.66%.

@@            Coverage Diff             @@
##           master    #5386      +/-   ##
==========================================
+ Coverage   74.02%   74.09%   +0.06%     
==========================================
  Files        1019     1019              
  Lines      190308   190355      +47     
  Branches    20462    20469       +7     
==========================================
+ Hits       140877   141039     +162     
+ Misses      43888    43789      -99     
+ Partials     5543     5527      -16
Flag Coverage Δ
#Debug 74.09% <61.66%> (+0.06%) ⬆️
#production 69.86% <48.78%> (+0.08%) ⬆️
#test 87.71% <89.47%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...AutoML/Experiment/Runners/CrossValSummaryRunner.cs 56.73% <0.00%> (-7.78%) ⬇️
...soft.ML.Data/DataLoadSave/DataOperationsCatalog.cs 85.50% <ø> (ø)
src/Microsoft.ML.TimeSeries/STL/InnerStl.cs 90.50% <0.00%> (ø)
src/Microsoft.ML.TimeSeries/STL/Loess.cs 84.61% <0.00%> (ø)
src/Microsoft.ML.TimeSeries/Deseasonality.cs 92.42% <50.00%> (+4.18%) ⬆️
...crosoft.ML.TimeSeries.Tests/TimeSeriesDirectApi.cs 99.41% <85.71%> (-0.29%) ⬇️
...rosoft.ML.TimeSeries/SrCnnEntireAnomalyDetector.cs 91.85% <100.00%> (+0.22%) ⬆️
test/Microsoft.ML.AutoML.Tests/AutoFitTests.cs 93.79% <100.00%> (+0.02%) ⬆️
...c/Microsoft.ML.FastTree/Utils/ThreadTaskManager.cs 79.48% <0.00%> (-20.52%) ⬇️
...icrosoft.ML.AutoML/Experiment/SuggestedPipeline.cs 88.65% <0.00%> (-4.13%) ⬇️
... and 13 more

justinormont
justinormont reviewed Sep 4, 2020
View reviewed changes
Comment thread test/Microsoft.ML.AutoML.Tests/AutoFitTests.cs Outdated
justinormont
justinormont reviewed Sep 4, 2020
View reviewed changes
Comment thread test/Microsoft.ML.AutoML.Tests/AutoFitTests.cs Outdated
justinormont
justinormont reviewed Sep 4, 2020
View reviewed changes
Comment thread src/Microsoft.ML.AutoML/Experiment/Runners/CrossValSummaryRunner.cs
justinormont
justinormont reviewed Sep 4, 2020
View reviewed changes
Comment thread test/Microsoft.ML.AutoML.Tests/AutoFitTests.cs Outdated
@mstfbl mstfbl requested a review from justinormont September 4, 2020 20:58
@mstfbl mstfbl mentioned this pull request Sep 4, 2020
Closed
@mstfbl mstfbl linked an issue Sep 4, 2020 that may be closed by this pull request
Typo - anMicrosoft #5388
Closed
@mstfbl mstfbl requested a review from a team as a code owner September 4, 2020 22:51
LittleLittleCloud
LittleLittleCloud reviewed Sep 10, 2020
View reviewed changes
Comment thread src/Microsoft.ML.Data/DataLoadSave/DataOperationsCatalog.cs
LittleLittleCloud
LittleLittleCloud reviewed Sep 10, 2020
View reviewed changes
Comment thread test/Microsoft.ML.AutoML.Tests/AutoFitTests.cs Outdated
LittleLittleCloud
LittleLittleCloud reviewed Sep 10, 2020
View reviewed changes
Comment thread test/Microsoft.ML.AutoML.Tests/AutoFitTests.cs Outdated
LittleLittleCloud
LittleLittleCloud reviewed Sep 10, 2020
View reviewed changes
Comment thread test/Microsoft.ML.AutoML.Tests/AutoFitTests.cs Outdated
@LittleLittleCloud
Copy link
Copy Markdown
Member

LittleLittleCloud commented Sep 10, 2020

LGTM

@mstfbl mstfbl merged commit 68ea969 into dotnet:master Sep 10, 2020
@mstfbl mstfbl mentioned this pull request Sep 11, 2020
Merged
@ghost ghost locked as resolved and limited conversation to collaborators Mar 17, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@LittleLittleCloud LittleLittleCloud LittleLittleCloud approved these changes

@justinormont justinormont Awaiting requested review from justinormont

@kere-nel kere-nel Awaiting requested review from kere-nel

+1 more reviewer

@harishsk harishsk harishsk approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Typo - anMicrosoft AutoML Ranking Error: "RankingMetrics Not Implemented"

5 participants

@mstfbl @LittleLittleCloud @justinormont @harishsk @kere-nel