ddp precision recall by fco-dv · Pull Request #1646 · pytorch/ignite

fco-dv · 2021-02-16T15:31:42Z

Description:

Enable the case for Recall/Precision in multi_label, not averaged configuration for DDP
Remove associated warnings, tests updated by scaling output size with idist.get_world_size()
usage of metric_device should be checked in

ignite/tests/ignite/metrics/test_precision.py

Line 771 in 002b595

def _test_distrib_integration_multilabel(device):

Check list:
New tests are added (if a new feature is added)
New doc strings: description and/or example code are in RST format
Documentation is updated (if required)

… true

pytorch#1612) * For v0.4.3 - Add more versionadded, versionchanged tags - Change v0.5.0 to v0.4.3 * Update ignite/contrib/metrics/regression/canberra_metric.py Co-authored-by: vfdev <[email protected]> * Update ignite/contrib/metrics/regression/manhattan_distance.py Co-authored-by: vfdev <[email protected]> * Update ignite/contrib/metrics/regression/r2_score.py Co-authored-by: vfdev <[email protected]> * Update ignite/handlers/checkpoint.py Co-authored-by: vfdev <[email protected]> * address PR comments Co-authored-by: vfdev <[email protected]>

* added TimeLimit handler with its test and doc * fixed documentation * fixed docstring and formatting * flake8 fix trailing whitespace :) * modified class logger , default value and tests * changed rounding to nearest integer * tests refactored , docs modified * fixed default value , removed global logger * fixing formatting * Added versionadded * added test for engine termination Co-authored-by: vfdev <[email protected]>

* Fixes pytorch#1614 - Updated handlers EarlyStopping and TerminateOnNan - Replaced `logging.getLogger` with `setup_logger` in the mentioned handlers * Updated `TimeLimit` handler. Replaced use of `logger.getLogger` with `setup_logger` from `ignite.utils` Co-authored-by: Pradyumna Rahul K <[email protected]> Co-authored-by: Sylvain Desroziers <[email protected]>

@deprecated

* Starter code for managing deprecation * Make functions deprecated using the `@deprecated` decorator * Add arguments to the @deprecated decorator to customize it for each function * Improve `@deprecated` decorator and add tests * Replaced the `raise` keyword with added `warnings` * Added tests several possibilities of the decorator usage * Removing the test deprecation to check tests * Add static typing, fix mypy errors * Make `@deprecated` to raise Exceptions or Warning * The `@deprecated` decorator will now always emit warning unless explicitly asked to raise an Exception * Fix mypy errors * Fix mypy errors (hopefully) * Fix the test `test_deprecated_setup_any_logging` * Change the test to work with the `@deprecated` decorator * Change to snake_case, handle mypy ignores * Improve Type Annotations * Update common.py * For v0.4.3 - Add more versionadded, versionchanged tags - Change v0.5… (pytorch#1612) * For v0.4.3 - Add more versionadded, versionchanged tags - Change v0.5.0 to v0.4.3 * Update ignite/contrib/metrics/regression/canberra_metric.py Co-authored-by: vfdev <[email protected]> * Update ignite/contrib/metrics/regression/manhattan_distance.py Co-authored-by: vfdev <[email protected]> * Update ignite/contrib/metrics/regression/r2_score.py Co-authored-by: vfdev <[email protected]> * Update ignite/handlers/checkpoint.py Co-authored-by: vfdev <[email protected]> * address PR comments Co-authored-by: vfdev <[email protected]> * `version` -> version Co-authored-by: vfdev <[email protected]> Co-authored-by: François COKELAER <[email protected]> Co-authored-by: Sylvain Desroziers <[email protected]>

…1620) * modified CONTRIBUTING.md * bash instead of sh

* Added Checkpoint.get_default_score_fn to simplify best_model_handler creation * Added score_sign argument * Updated docs

* Change pre-commit config and CONTRIBUTING.md - Update hook versions - Remove seed-isort-config - Add black profile to isort * Fix files based on new pre-commit config * Add meaningful exclusions to prettier - Also update actions workflow files to match local pre-commit

* added requirements.txt and updated readme.md * Update examples/contrib/cifar10/README.md Co-authored-by: vfdev <[email protected]> * Update examples/contrib/cifar10/requirements.txt Co-authored-by: vfdev <[email protected]> Co-authored-by: vfdev <[email protected]>

* Updates for cifar10 example * Updates for cifar10 example * More updates * Updated code * Fixed code-formatting

* Updates for cifar10 example * Updates for cifar10 example * More updates * Updated code * Fixed code-formatting * Fixed typo and failing CI * Fixed hvd spawn fail and better synced qat code

- updated default pth image for gpu tests - updated TORCH_CUDA_ARCH_LIST - fixed /merge -> /head in trigger ci pipeline

* [docker] Pillow -> Pillow-SIMD (pytorch#1509) * [docker] Pillow -> Pillow-SIMD * replace pillow with pillow-simd in base docker files * chore(docker): apt-get autoremove after pillow-simd installation * apt-get install at once, autoremove g++ * install g++ in pillow installation layer Co-authored-by: Sylvain Desroziers <[email protected]> * Fix g++ install issue Co-authored-by: Jeff Yang <[email protected]> Co-authored-by: Sylvain Desroziers <[email protected]>

* fix run_multinode_tests_in_docker.sh : run tests with docker python version * add missing modules * build an image with test env and add 'nnodes' 'nproc_per_node' 'gpu' as parameters * pytorch#1615 : change nproc_per_node default to 4 * pytorch#1615 : fix for gpu enabled tests / container rm step at the end of the script * add xfail decorator for tests/ignite/engine/test_deterministic.py::test_multinode_distrib_cpu * fix script gpu_options * add default tol=1e-6 for _test_distrib_compute_on_criterion * fix for "RuntimeError: trying to initialize the default process group twice!" * tolerance for test_multinode_distrib_cpu case only * fix assert None error * autopep8 fix Co-authored-by: vfdev <[email protected]> Co-authored-by: Sylvain Desroziers <[email protected]> Co-authored-by: fco-dv <[email protected]>

sdesrozis · 2021-02-16T21:34:55Z

@fco-dv well done !

Please could you update docstring ?

ignite/ignite/metrics/recall.py

Line 53 in 63a6f1b

    
                   In multilabel cases, if average is False, current implementation does not work with distributed computations.

ignite/ignite/metrics/precision.py

Line 112 in 63a6f1b

    
                   In multilabel cases, if average is False, current implementation does not work with distributed computations.

vfdev-5

Thanks for the PR @fco-dv ! Please, take a look at the comment on how to improve the test.

vfdev-5 · 2021-02-16T23:37:13Z

tests/ignite/metrics/test_precision.py

-        ):
-            pr = Precision(average=False, is_multilabel=True)
-
+        pr = Precision(average=False, is_multilabel=True)


First of all, could you please fix the issue we spotted during the call with unused metric_device that should go to Precision definition (line 795).

Ultimately, we would like to enable the tests in the same way as test_multilabel_input_NCHW where we run _test(average=True, ...) and _test(average=False, ...). In case of average False, there is no equivalent option in sklearn (as probably it does not make much sense neither). So, please, take a look how we test things in test_multilabel_input_NCHW and do same thing here. Then we can simply remove this dummy test below.

PS.: average=False and is_multilabel=True gives as a result for precision = multilabel precision per sample

import torch from ignite.metrics import Precision pr = Precision(average=False, is_multilabel=True) pr.update((torch.tensor(y_pred[:5, :]), torch.tensor(y_true[:5, :]))) pr.update((torch.tensor(y_pred[5:, :]), torch.tensor(y_true[5:, :]))) print(pr._true_positives, pr._positives, pr.compute(), pr.compute().mean().item()) > (tensor([2., 2., 2., 2., 2., 2., 1., 0.], dtype=torch.float64), tensor([2., 3., 2., 2., 2., 2., 1., 1.], dtype=torch.float64), tensor([1.0000, 0.6667, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 0.0000], dtype=torch.float64), 0.8333333333333333)

vfdev-5 · 2021-02-20T23:13:56Z

@fco-dv could you please udpate the tests according the suggestion: #1646 (comment)

…ilabel_input_NCHW

vfdev-5

LGTM! Thanks @fco-dv !

fco-dv and others added 19 commits February 3, 2021 22:16

Recall/Precision metrics for ddp : average == false and multilabel ==…

e6691bc

… true

Create documentation.md

3c0b68f

Distributed tests on Windows should be skipped until fixed. (pytorch#…

4a52ebc

…1620) * modified CONTRIBUTING.md * bash instead of sh

Added Checkpoint.get_default_score_fn (pytorch#1621)

e4571ae

* Added Checkpoint.get_default_score_fn to simplify best_model_handler creation * Added score_sign argument * Updated docs

Update about.rst

6e8dd3d

Replace relative paths with raw.githubusercontent (pytorch#1629)

bd4ab8c

Updated cifar10 example (pytorch#1632)

944afab

* Updates for cifar10 example * Updates for cifar10 example * More updates * Updated code * Fixed code-formatting

Fixed failling CI and typos for cifar10 examples (pytorch#1633)

02e767e

* Updates for cifar10 example * Updates for cifar10 example * More updates * Updated code * Fixed code-formatting * Fixed typo and failing CI * Fixed hvd spawn fail and better synced qat code

Removed temporary hack to install pth 1.7.1 (pytorch#1638)

61d8c2f

- updated default pth image for gpu tests - updated TORCH_CUDA_ARCH_LIST - fixed /merge -> /head in trigger ci pipeline

remove warning for average=False and is_multilabel=True

04f8fd8

Merge branch 'master' into WIP_ddp_precision_recall

611ea97

fco-dv requested a review from sdesrozis February 16, 2021 17:51

vfdev-5 reviewed Feb 16, 2021

View reviewed changes

sdesrozis added 3 commits February 17, 2021 08:24

Merge branch 'master' into WIP_ddp_precision_recall

dd07cbd

Merge branch 'master' into WIP_ddp_precision_recall

d9a16ee

Merge branch 'master' into WIP_ddp_precision_recall

1e3e5d3

update docstring and {precision, recall} tests according to test_mult…

f3998cb

…ilabel_input_NCHW

vfdev-5 approved these changes Feb 21, 2021

View reviewed changes

vfdev-5 merged commit 7753eab into pytorch:master Feb 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ddp precision recall#1646

ddp precision recall#1646
vfdev-5 merged 23 commits intopytorch:masterfrom
fco-dv:WIP_ddp_precision_recall

fco-dv commented Feb 16, 2021

Uh oh!

sdesrozis commented Feb 16, 2021 •

edited

Loading

Uh oh!

vfdev-5 left a comment

Uh oh!

vfdev-5 Feb 16, 2021

Uh oh!

vfdev-5 commented Feb 20, 2021

Uh oh!

vfdev-5 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

fco-dv commented Feb 16, 2021

Uh oh!

sdesrozis commented Feb 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vfdev-5 left a comment

Choose a reason for hiding this comment

Uh oh!

vfdev-5 Feb 16, 2021

Choose a reason for hiding this comment

Uh oh!

vfdev-5 commented Feb 20, 2021

Uh oh!

vfdev-5 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

sdesrozis commented Feb 16, 2021 •

edited

Loading