Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

Flaky test_gluon.test_conv causes other tests to fail on Windows #11506

@marcoabreu

Description

@marcoabreu

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1106/pipeline

test_operator_gpu.test_ndarray_crop ... ok

test_operator_gpu.test_cell_fill_shape ... ok

test_operator_gpu.test_conv ... [23:56:08] c:\jenkins_slave\workspace\build-gpu@2\src\operator\nn\cudnn\./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

[23:56:08] C:/jenkins_slave/workspace/build-gpu@2/src/operator/nn/convolution.cu:148: This convolution is not supported by cudnn, MXNET convolution is applied.

[23:56:08] C:/jenkins_slave/workspace/build-gpu@2/src/operator/nn/convolution.cu:227: This convolution is not supported by cudnn, MXNET convolution is applied.

[23:56:08] C:/jenkins_slave/workspace/build-gpu@2/src/operator/nn/convolution.cu:148: This convolution is not supported by cudnn, MXNET convolution is applied.

[23:56:08] C:/jenkins_slave/workspace/build-gpu@2/src/operator/nn/convolution.cu:227: This convolution is not supported by cudnn, MXNET convolution is applied.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=270901180 to reproduce.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1177804468 to reproduce.

ERROR

test_operator_gpu.test_layer_fill_shape ... ERROR

test_operator_gpu.test_ndarray_concatenate ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=820959189 to reproduce.

ERROR

test_operator_gpu.test_normal_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1440948114 to reproduce.

ERROR

test_operator_gpu.test_sparse_nd_transpose ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=479442568 to reproduce.

ERROR

test_operator_gpu.test_slice_channel ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=952262734 to reproduce.

ERROR

test_operator_gpu.test_sparse_nd_storage_fallback ... [23:56:09] c:\jenkins_slave\workspace\build-gpu@2\src\operator\../common/utils.h:447: 

Storage type fallback detected:

operator = broadcast_add

input storage types = [default, default, ]

output storage types = [csr, ]

params = {}

context.dev_mask = gpu

The operator with default storage type will be dispatched for execution. You're seeing this warning message because the operator above is unable to process the given ndarrays with specified storage types, context and parameter. Temporary dense ndarrays are generated in order to execute the operator. This does not affect the correctness of the programme. You can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=2072600123 to reproduce.

ERROR

test_operator_gpu.test_clip ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1670877169 to reproduce.

ERROR

test_operator_gpu.test_convolution_with_type ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1234 to reproduce.

ERROR

test_operator_gpu.test_deconv ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=77155738 to reproduce.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=542068512 to reproduce.

ERROR

test_operator_gpu.test_dot ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1533114250 to reproduce.

ERROR

test_operator_gpu.test_uniform_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1203830416 to reproduce.

ERROR

test sparse random operator on cpu ... ok

test_operator_gpu.test_nag ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=0 to reproduce.

ERROR

test_operator_gpu.test_gamma_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=579518551 to reproduce.

ERROR

test regression operator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1630735113 to reproduce.

ERROR

test_operator_gpu.test_sparse_nd_astype ... ok

test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=493650054 to reproduce.

ERROR

test_operator_gpu.test_sparse_nd_astype_copy ... ok

test_operator_gpu.test_convolution_options ... SKIP: test fails intermittently. temporarily disabled till it gets fixed. tracked at https://github.com/apache/incubator-mxnet/issues/10141

test_operator_gpu.test_reduce ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1367744731 to reproduce.

ERROR

test_operator_gpu.test_pool ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1525342399 to reproduce.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=785623715 to reproduce.

ERROR

test_operator_gpu.test_poisson_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1266754562 to reproduce.

ERROR

test_operator_gpu.test_negative_binomial_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=372547776 to reproduce.

ERROR

test_operator_gpu.test_sparse_nd_pickle ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=0 to reproduce.

ERROR

test_operator_gpu.test_broadcast ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=526949765 to reproduce.

ERROR

test_operator_gpu.test_ftml ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=0 to reproduce.

ERROR

test_operator_gpu.test_batchnorm ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1425658102 to reproduce.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1562292807 to reproduce.

ERROR

test_operator_gpu.test_sparse_nd_save_load ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=0 to reproduce.

ERROR

test_operator_gpu.test_instancenorm ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=832404596 to reproduce.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1764718120 to reproduce.

ERROR

test_operator_gpu.test_multinomial_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1990187257 to reproduce.

ERROR

test_operator_gpu.test_layernorm ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=803289623 to reproduce.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1460168409 to reproduce.

ERROR

test_operator_gpu.test_reflectionpad ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=897856965 to reproduce.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1138894034 to reproduce.

ERROR

test_operator_gpu.test_convolution_versions ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1576481502 to reproduce.

ERROR

test_operator_gpu.test_broadcast_binary ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1660279093 to reproduce.

ERROR

test_operator_gpu.test_reshape ... ok

test_operator_gpu.test_sparse_nd_unsupported ... ok

test_operator_gpu.test_slice ... ok

test_operator_gpu.test_at ... ok

test_operator_gpu.test_create_csr ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=106598815 to reproduce.

ERROR

test_operator_gpu.test_with_random_seed ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1495502410 to reproduce.

ERROR

test_operator_gpu.test_elemwise_csr_same_zeros ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=583265934 to reproduce.

ERROR

test_operator_gpu.test_moveaxis ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=197256063 to reproduce.

ERROR

test_operator_gpu.test_deferred_init ... ok

test_operator_gpu.test_deconvolution_with_type ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=838817885 to reproduce.

ERROR

test_operator_gpu.test_arange ... [23:56:09] c:\jenkins_slave\workspace\build-gpu@2\src\common\../operator/mxnet_op.h:576: Check failed: (err) == (cudaSuccess) Name: mxnet_generic_kernel ErrStr:unspecified launch failure



(py3) C:\jenkins_slave\workspace\ut-python-gpu>IF -1073740791 NEQ 0 exit /b 1

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions