Skip to content

Fixed test in test_precision_recall_curve and tests/ignite/contrib/metrics/regression #2511

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sayantan1410 wants to merge 10 commits into from
Closed

Fixed test in test_precision_recall_curve and tests/ignite/contrib/metrics/regression #2511

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
ea71d42
Fixed a test in test_precision_recall_curve.py
sayantan1410 Mar 11, 2022
5c6d863
Fixed integration tests
sayantan1410 Mar 13, 2022
080c49c
fixed mypy error
sayantan1410 Mar 13, 2022
76fd0f7
fixed pytest.approx relative errors
sayantan1410 Mar 13, 2022
42f5660
added manual_seed to _test function
sayantan1410 Mar 20, 2022
89d14d8
autopep8 fix
sayantan1410 Mar 20, 2022
9285ca5
Merge branch 'master' into tests
vfdev-5 Mar 20, 2022
de96850
fixed tolerance of pytest.approx
sayantan1410 Mar 20, 2022
d578dde
Added torch.manual_seed inside _test function
sayantan1410 Mar 23, 2022
a05cda1
reduced tolerance to default
sayantan1410 Mar 23, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions tests/ignite/contrib/metrics/regression/test_median_absolute_error.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,21 +150,21 @@ def _test(metric_device):


def _test_distrib_integration(device):

rank = idist.get_rank()
torch.manual_seed(12)

def _test(n_epochs, metric_device):
metric_device = torch.device(metric_device)
rank = idist.get_rank()
torch.manual_seed(rank)
n_iters = 80
size = 105
y_true = torch.rand(size=(size,)).to(device)
y_preds = torch.rand(size=(size,)).to(device)

offset = n_iters * size
y_true = torch.rand(size=(offset * idist.get_world_size(),)).to(device)
y_preds = torch.rand(size=(offset * idist.get_world_size(),)).to(device)
Copy link
Copy Markdown
Collaborator

@vfdev-5 vfdev-5 Mar 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sayantan1410 either revert these modifications or make them coherent to others and rename the PR.

Copy link
Copy Markdown
Contributor Author

@sayantan1410 sayantan1410 Mar 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, making them coherent with others, also some tests are failing, fixing them as well

Copy link
Copy Markdown
Collaborator

@vfdev-5 vfdev-5 Mar 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you updated others ? Do not forget to rename the PR title to reflect that you change not only test_precision_recall_curve.py

Copy link
Copy Markdown
Contributor Author

@sayantan1410 sayantan1410 Mar 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, in the other tests this has been changed and I have changed the PR title as well.

Copy link
Copy Markdown
Collaborator

@vfdev-5 vfdev-5 Mar 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't you add

        rank = idist.get_rank()
        torch.manual_seed(rank)

here and other regression files ?

Copy link
Copy Markdown
Contributor Author

@sayantan1410 sayantan1410 Mar 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this to precision_recall_curve and was waiting for all the tests to pass for it. Now I will add it to the other files as well.


def update(engine, i):
return (
y_preds[i * size : (i + 1) * size],
y_true[i * size : (i + 1) * size],
y_preds[i * size + rank * offset : (i + 1) * size + rank * offset],
y_true[i * size + rank * offset : (i + 1) * size + rank * offset],
)

engine = Engine(update)
Expand Down
16 changes: 8 additions & 8 deletions tests/ignite/contrib/metrics/regression/test_median_absolute_percentage_error.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,21 +151,21 @@ def _test(metric_device):


def _test_distrib_integration(device):

rank = idist.get_rank()
torch.manual_seed(12)

def _test(n_epochs, metric_device):
metric_device = torch.device(metric_device)
rank = idist.get_rank()
torch.manual_seed(rank)
n_iters = 80
size = 105
y_true = torch.rand(size=(size,)).to(device)
y_preds = torch.rand(size=(size,)).to(device)

offset = n_iters * size
y_true = torch.rand(size=(offset * idist.get_world_size(),)).to(device)
y_preds = torch.rand(size=(offset * idist.get_world_size(),)).to(device)

def update(engine, i):
return (
y_preds[i * size : (i + 1) * size],
y_true[i * size : (i + 1) * size],
y_preds[i * size + rank * offset : (i + 1) * size + rank * offset],
y_true[i * size + rank * offset : (i + 1) * size + rank * offset],
)

engine = Engine(update)
Expand Down
16 changes: 8 additions & 8 deletions tests/ignite/contrib/metrics/regression/test_median_relative_absolute_error.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,21 +151,21 @@ def _test(metric_device):


def _test_distrib_integration(device):

rank = idist.get_rank()
torch.manual_seed(12)

def _test(n_epochs, metric_device):
metric_device = torch.device(metric_device)
rank = idist.get_rank()
torch.manual_seed(rank)
n_iters = 80
size = 151
y_true = torch.rand(size=(size,)).to(device)
y_preds = torch.rand(size=(size,)).to(device)

offset = n_iters * size
y_true = torch.rand(size=(offset * idist.get_world_size(),)).to(device)
y_preds = torch.rand(size=(offset * idist.get_world_size(),)).to(device)

def update(engine, i):
return (
y_preds[i * size : (i + 1) * size],
y_true[i * size : (i + 1) * size],
y_preds[i * size + rank * offset : (i + 1) * size + rank * offset],
y_true[i * size + rank * offset : (i + 1) * size + rank * offset],
)

engine = Engine(update)
Expand Down
22 changes: 11 additions & 11 deletions tests/ignite/contrib/metrics/test_precision_recall_curve.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,21 +194,21 @@ def get_test_cases():


def _test_distrib_integration(device):

rank = idist.get_rank()
torch.manual_seed(12)

def _test(n_epochs, metric_device):
metric_device = torch.device(metric_device)
rank = idist.get_rank()
torch.manual_seed(rank)
n_iters = 80
size = 151
y_true = torch.randint(0, 2, (size,)).to(device)
y_preds = torch.randint(0, 2, (size,)).to(device)

offset = n_iters * size
y_true = torch.randint(0, 2, size=(offset * idist.get_world_size(),)).to(device)
y_preds = torch.randint(0, 2, size=(offset * idist.get_world_size(),)).to(device)

def update(engine, i):
return (
y_preds[i * size : (i + 1) * size],
y_true[i * size : (i + 1) * size],
y_preds[i * size + rank * offset : (i + 1) * size + rank * offset],
y_true[i * size + rank * offset : (i + 1) * size + rank * offset],
)

engine = Engine(update)
Expand All @@ -231,9 +231,9 @@ def update(engine, i):
assert precision.shape == sk_precision.shape
assert recall.shape == sk_recall.shape
assert thresholds.shape == sk_thresholds.shape
assert pytest.approx(precision.cpu().numpy()) == sk_precision
assert pytest.approx(recall.cpu().numpy()) == sk_recall
assert pytest.approx(thresholds.cpu().numpy()) == sk_thresholds
assert pytest.approx(precision.cpu().numpy(), rel=1e-3) == sk_precision
assert pytest.approx(recall.cpu().numpy(), rel=1e-3) == sk_recall
assert pytest.approx(thresholds.cpu().numpy(), rel=1e-3) == sk_thresholds

metric_devices = ["cpu"]
if device.type != "xla":
Expand Down