Hi All,

I am running currently distributed learning using the launch.py tool. My plan is to run the distributed learning by using Sync and Async distributed modes. For the Sync PS, the training works fine and I got the number. However, when I switch the kvstore type to "dist_async", the following error shows up:

terminate called after throwing an instance of 'dmlc::Error' what(): [11:58:12] src/kvstore/././kvstore_dist_server.h:294: Check failed: sync_mode_ Updater needs to be set for async mode Stack trace returned 10 entries: [bt] (0) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x30cbe2) [0x2aedf8ebdbe2] [bt] (1) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x30d1b8) [0x2aedf8ebe1b8] [bt] (2) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2ab1f11) [0x2aedfb662f11] [bt] (3) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2ab6f10) [0x2aedfb667f10] [bt] (4) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2ab74d4) [0x2aedfb6684d4] [bt] (5) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2a97b06) [0x2aedfb648b06] [bt] (6) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2f640d5) [0x2aedfbb150d5] [bt] (7) /shared/apps/gcc/4.9.4/lib64/libstdc++.so.6(+0xc9090) [0x2aee34139090] [bt] (8) /lib64/libpthread.so.0(+0x7e25) [0x2aedc9953e25] [bt] (9) /lib64/libc.so.6(clone+0x6d) [0x2aedca36934d]

Any idea why this error happens?

Thanks :)