Add a decorator for marking slow tests. #18231
Conversation
The general strategy: - It's a normal skip decorator, which triggers a skip if PYTORCH_TEST_WITH_SLOW is not set. - It also annotates the method in question that says it's slow. We use this to implement a catch-all skipper in setUp that skips all non-slow tests when PYTORCH_TEST_SKIP_FAST is set. I added a little smoketest to test_torch and showed that I get: ``` Ran 432 tests in 0.017s OK (skipped=431) ``` when running with PYTORCH_TEST_WITH_SLOW=1 and PYTORCH_TEST_SKIP_FAST=1 CI integration coming in later patch, as well as nontrivial uses of this decorator. Signed-off-by: Edward Z. Yang <[email protected]>
|
nit:We can add note that def my_decorator(func):
def wrapper():
func()
return wrapperwill loose |
salexspb
left a comment
salexspb
left a comment
There was a problem hiding this comment.
Any specific tests you would like to apply this to?
What is the plan for internal deployment?
| # This is usually used in conjunction with TEST_WITH_SLOW to | ||
| # run *only* slow tests. (I could have done an enum, but | ||
| # it felt a little awkward. | ||
| TEST_SKIP_FAST = os.getenv('PYTORCH_TEST_SKIP_FAST', '0') == '1' |
There was a problem hiding this comment.
how about having with_slow and with_fast rather than having them an opposite way? IMO its confusing to have only part of the flags as negations.
There was a problem hiding this comment.
I negated the flag in recognition of the defaults in question: we default to running fast tests but skipping slow tests. To me, it makes more sense for FOO=1 to be the non-default case and unset FOO or FOO= to be the default case.
There was a problem hiding this comment.
Don't see an issue with default being non zero, but understand it is a personal choice :)
Summary: Pull Request resolved: #18236 ghimport-source-id: 2bb80d0 Stack from [ghstack](https://github.com/ezyang/ghstack): * **#18236 Enable running of slow tests in CI.** * #18231 Add a decorator for marking slow tests. These tests only run on master, as they are slow. Signed-off-by: Edward Z. Yang <[email protected]> Differential Revision: D14563115 fbshipit-source-id: f54ddef4abedc7e872e58657fc9ac537952773d0
In the most immediate future, test for #17280 and also a stress test for #16854 For now, one off, but maybe we can figure out some global configuration we can apply to globally test that, e.g., 64-bit indexing works. One problem is we don't have any gpu test configuration with > 8GB memory at the moment. |
|
@ezyang can this be applied to tests defined in |
|
In principle yes, but I guess we have to figure out how to put decorators on these invocations. |
|
Yeah, the list supports adding decorators. Example: |
Stack from ghstack:
The general strategy:
PYTORCH_TEST_WITH_SLOW is not set.
slow. We use this to implement a catch-all skipper in
setUp that skips all non-slow tests when
PYTORCH_TEST_SKIP_FAST is set.
I added a little smoketest to test_torch and showed that I get:
when running with PYTORCH_TEST_WITH_SLOW=1 and PYTORCH_TEST_SKIP_FAST=1
CI integration coming in later patch, as well as nontrivial uses of
this decorator.
Signed-off-by: Edward Z. Yang [email protected]
Differential Revision: D14544441