fix python rpc handler exit crash #27251

zhaojuanmao · 2019-10-02T22:56:40Z

Stack from ghstack:

fix python rpc handler exit crash #27251 fix python rpc handler exit crash

[test all]

Explicitly clean up py::objects to avoid segment faults when py::objects with CPython are cleaned up later at program exit.

See similar issues reported pybind/pybind11#1598
and pybind/pybind11#1493.

Our local tests also caught this segment faults if py::objects are cleaned
up at program exit. The explaination is: CPython cleans up most critical
utitlies before cleaning up PythonRpcHandler singleton, so when
PythonRpcHandler signleton cleans up py::objects and call dec_ref(), it
will crash.

The solution is to clean up py::objects earlier when Rpc agent join().
Be note that py::objects can not be cleaned up when Rpc agent is destroyed
as well, as Rpc agent is global variable and it will have same issue as
PythonRpcHandler.

close #27182

Differential Revision: D17727362

we found python rpc handler exit crash in python 3.5 spwan tests, backtrace showed pythonRpcHandler destructor dec_ref() deallocated python objects, that means python objects in pythonRpcHandler have been cleaned up before destructing pythonRpcHandler while python program exits in python 3.5. This diff is to explicitly clean up python objects of pythonRpcHandler in RpcAgent.join(). so that pythonRpcHandler destructor will not have chances to call dec_ref(). we did not explicitly clean up python objects in RpcAngent destructor, it is also because python objects will be cleaned up before destructing RpcAgent when python program exit in python 3.5. close #27182 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/) [ghstack-poisoned]

we found python rpc handler exit crash in python 3.5 spwan tests, backtrace showed pythonRpcHandler destructor dec_ref() deallocated python objects, that means python objects in pythonRpcHandler have been cleaned up before destructing pythonRpcHandler while python program exits in python 3.5. This diff is to explicitly clean up python objects of pythonRpcHandler in RpcAgent.join(). so that pythonRpcHandler destructor will not have chances to call dec_ref(). we did not explicitly clean up python objects in RpcAngent destructor, it is also because python objects will be cleaned up before destructing RpcAgent when python program exit in python 3.5. close #27182 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/) ghstack-source-id: 91230848 Pull Request resolved: #27251

pritamdamania87 · 2019-10-03T18:13:23Z

torch/csrc/distributed/rpc/process_group_agent.cpp

+  // the python objects in PythonRpcHandler could be cleaned up before
+  // destructing ProcessGroupAgent, then ProcessGroupAgent destructor will
+  // dec_ref deallocated python objects and crash.
+  PythonRpcHandler::getInstance().cleanUp();


What if we just set runUDFFunction_ and loadResultFunction_ to None in the destructor of PythonRPCHandler? Does that work?

if we only set them to None, not free the memory, then we are assuming python objects will be cleaned up before destructing pythonRpcHandler, but this assumption is only true for python 3.5. For python 3.6, it is not the case, we may be at risk of leaking memory in python 3.6 if set them as None only in destructor of PythonRPCHandler

IIUC, PythonRpcHandler is a singleton so it would be destructed only on program exit. As a result, we won't leak any memory since the OS will just clean up all the memory once the process dies. Although, I'm not sure if ASAN would fail due to this. Can you give it a shot?

@pritamdamania87 tests passed in both py3.5 and py3.6, fork asan did not complain as well, not sure whether spawn asan will complain or not because we disabled spawn asan.

I will update diff to see how others think about it as well.

we found python rpc handler exit crash in python 3.5 spwan tests, backtrace showed pythonRpcHandler destructor dec_ref() deallocated python objects, that means python objects in pythonRpcHandler have been cleaned up before destructing pythonRpcHandler while python program exits in python 3.5. This diff is to explicitly clean up python objects of pythonRpcHandler in RpcAgent.join(). so that pythonRpcHandler destructor will not have chances to call dec_ref(). we did not explicitly clean up python objects in RpcAngent destructor, it is also because python objects will be cleaned up before destructing RpcAgent when python program exit in python 3.5. close #27182 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/) [ghstack-poisoned]

Pull Request resolved: #27251 we found python rpc handler exit crash in python 3.5 spwan tests, backtrace showed pythonRpcHandler destructor dec_ref() deallocated python objects, that means python objects in pythonRpcHandler have been cleaned up before destructing pythonRpcHandler while python program exits in python 3.5. This diff is to explicitly clean up python objects of pythonRpcHandler in RpcAgent.join(). so that pythonRpcHandler destructor will not have chances to call dec_ref(). we did not explicitly clean up python objects in RpcAngent destructor, it is also because python objects will be cleaned up before destructing RpcAgent when python program exit in python 3.5. close #27182 ghstack-source-id: 91310248 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/)

mrshenli · 2019-10-04T14:36:37Z

torch/csrc/distributed/rpc/process_group_agent.cpp

+  // ProcessGroupAgent. Otherwise, in Python 3.5, when python program exits,
+  // the python objects in PythonRpcHandler could be cleaned up before
+  // destructing ProcessGroupAgent, then ProcessGroupAgent destructor will
+  // dec_ref deallocated python objects and crash.


Question: do we know why inc_ref was not called properly in the first place? If ProcessGroupAgent is holding a reference as well, should the reference counting be consistent with that?

mrshenli · 2019-10-04T14:37:30Z

torch/csrc/distributed/rpc/process_group_agent.cpp

      SendWork(workerIds_[dst], Message({}, {}, MessageType::SHUTDOWN)));
  threadPool_.waitWorkComplete();
  listenerThread_.join();
+  // explicitly clean up python objects in PythonRpcHandler before destructing


If we expect all RpcAgent implementations would use the same PythonRpcHandler, shall we add this to RpcAgent instead of an implementation of RpcAgent.

mrshenli · 2019-10-04T14:38:51Z

torch/csrc/distributed/rpc/python_rpc_handler.cpp

+void PythonRpcHandler::cleanUp() {
+  AutoGIL ag;
+  if (!runUDFFunction_.is_none()) {
+    runUDFFunction_.dec_ref();


Could you please remind me where is the corresponding inc_ref for this? Is it when we load the function in module.attr? I am a little confused of why we need to do dec_ref on our own but not inc_ref?

zhaojuanmao · 2019-10-04T16:31:05Z

@mrshenli the theory now is: inc_ref works as expected, but python objects are deallocated before destructing pythonRpcHandler when program exits, so when pythonRpcHandler destructor tries to call dec_ref to clean up python objects, it is trying to clean up null objects and crashed.

And this only happens to python 3.5, I do not know why python objects are cleaned up before destructing pythonRpcHandler in python 3.5 only. This is not the case for python 3.6 though.

Does it make sense?

we found python rpc handler exit crash in python 3.5 spwan tests, backtrace showed pythonRpcHandler destructor dec_ref() deallocated python objects, that means python objects in pythonRpcHandler have been cleaned up before destructing pythonRpcHandler while python program exits in python 3.5. This diff is to explicitly clean up python objects of pythonRpcHandler in RpcAgent.join(). so that pythonRpcHandler destructor will not have chances to call dec_ref(). we did not explicitly clean up python objects in RpcAngent destructor, it is also because python objects will be cleaned up before destructing RpcAgent when python program exit in python 3.5. close #27182 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/) [ghstack-poisoned]

Pull Request resolved: #27251 we found python rpc handler exit crash in python 3.5 spwan tests, backtrace showed pythonRpcHandler destructor dec_ref() deallocated python objects, that means python objects in pythonRpcHandler have been cleaned up before destructing pythonRpcHandler while python program exits in python 3.5. This diff is to explicitly clean up python objects of pythonRpcHandler in RpcAgent.join(). so that pythonRpcHandler destructor will not have chances to call dec_ref(). we did not explicitly clean up python objects in RpcAngent destructor, it is also because python objects will be cleaned up before destructing RpcAgent when python program exit in python 3.5. close #27182 ghstack-source-id: 91359967 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/)

mrshenli · 2019-10-04T18:29:39Z

inc_ref works as expected, but python objects are deallocated before destructing pythonRpcHandler when program exits, so when pythonRpcHandler destructor tries to call dec_ref to clean up python objects, it is trying to clean up null objects and crashed.

I am not aware of the root cause for this, but this Python deallocation behavior seems weird to me. If pythonRpcHandler is still holding a reference to the Python object, wouldn't that Python object has a non-zero ref-count? If so, I am confused why they are deallocated before pythonRpcHandler. I would expect Python walk through the reference graph to destruct things in order. Or maybe Python just don't respect the topological order just in case there are circular dependencies.

zhaojuanmao · 2019-10-04T19:07:26Z

@mrshenli right, that is also weird to me as well. But be noted, Python 3.6 does things as we expected. but python 3.5 does not

pritamdamania87 · 2019-10-04T19:34:45Z

torch/csrc/distributed/rpc/python_rpc_handler.cpp

+// PythonRpcHandler singleton when program exits, all memories will be cleaned
+// up by OS.
+PythonRpcHandler::~PythonRpcHandler() {
+  if (!runUDFFunction_.is_none()) {


nit: Do we need these if statements? Can't we just set this to py::none()?

we found python rpc handler exit crash in python 3.5 spwan tests, backtrace showed pythonRpcHandler destructor dec_ref() deallocated python objects, that means python objects in pythonRpcHandler have been cleaned up before destructing pythonRpcHandler while python program exits in python 3.5. This diff is to explicitly clean up python objects of pythonRpcHandler in RpcAgent.join(). so that pythonRpcHandler destructor will not have chances to call dec_ref(). we did not explicitly clean up python objects in RpcAngent destructor, it is also because python objects will be cleaned up before destructing RpcAgent when python program exit in python 3.5. close #27182 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/) [ghstack-poisoned]

Pull Request resolved: #27251 we found python rpc handler exit crash in python 3.5 spwan tests, backtrace showed pythonRpcHandler destructor dec_ref() deallocated python objects, that means python objects in pythonRpcHandler have been cleaned up before destructing pythonRpcHandler while python program exits in python 3.5. This diff is to explicitly clean up python objects of pythonRpcHandler in RpcAgent.join(). so that pythonRpcHandler destructor will not have chances to call dec_ref(). we did not explicitly clean up python objects in RpcAngent destructor, it is also because python objects will be cleaned up before destructing RpcAgent when python program exit in python 3.5. close #27182 ghstack-source-id: 91445621 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/)

mrshenli

There might be conflicts with #27284.

we found python rpc handler exit crash in python 3.5 spwan tests, backtrace showed pythonRpcHandler destructor dec_ref() deallocated python objects, that means python objects in pythonRpcHandler have been cleaned up before destructing pythonRpcHandler while python program exits in python 3.5. This diff is to explicitly clean up python objects of pythonRpcHandler in RpcAgent.join(). so that pythonRpcHandler destructor will not have chances to call dec_ref(). we did not explicitly clean up python objects in RpcAngent destructor, it is also because python objects will be cleaned up before destructing RpcAgent when python program exit in python 3.5. close #27182 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/) [ghstack-poisoned]

Pull Request resolved: #27251 we found python rpc handler exit crash in python 3.5 spwan tests, backtrace showed pythonRpcHandler destructor dec_ref() deallocated python objects, that means python objects in pythonRpcHandler have been cleaned up before destructing pythonRpcHandler while python program exits in python 3.5. This diff is to explicitly clean up python objects of pythonRpcHandler in RpcAgent.join(). so that pythonRpcHandler destructor will not have chances to call dec_ref(). we did not explicitly clean up python objects in RpcAngent destructor, it is also because python objects will be cleaned up before destructing RpcAgent when python program exit in python 3.5. close #27182 ghstack-source-id: 91556219 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/)

pietern · 2019-10-10T14:14:09Z

Typo in title and summary spwan -> spawn

pietern

The two checks were carried over from the previous diff but are no longer needed.

The explanation of why this works on Python 3.5 is counter intuitive to me.

If Python 3.5 is proving to be such a pain, can't we restrict RPC to 3.6?

pietern · 2019-10-10T14:27:40Z

torch/csrc/distributed/rpc/python_rpc_handler.cpp

    std::vector<torch::Tensor>& responseTensorTable) {
  AutoGIL ag;
  auto pargs = py::bytes(pickledPayload.data(), pickledPayload.size());
+  TORCH_CHECK(!pyRunFunction_.is_none(), "pyRunFunction_ is none");


No need for this check. This member is initialized and confirmed to not be none in the constructor.

pietern · 2019-10-10T14:27:47Z

torch/csrc/distributed/rpc/python_rpc_handler.cpp

    const std::vector<torch::Tensor>& tensorTable) {
  AutoGIL ag;
  auto pargs = py::bytes(pickledPayload.data(), pickledPayload.size());
+  TORCH_CHECK(!pyLoadReturnValue_.is_none(), "pyLoadReturnValue_ is none");


Idem as above.

pietern · 2019-10-10T14:41:36Z

test/run_test.py

    'cuda_primary_ctx',
    'dataloader',
    'dist_autograd_fork',
+    'dist_autograd_spawn',


The distributed autograd tests directly import a number of private torch.distributed.rpc APIs. If this is not initialized, the files can't even be sourced, and cause an import error. I have posted #27612 to fix this.

right, I was waiting to rebase on your diff

torch/csrc/distributed/rpc/python_rpc_handler.cpp

[test all] we found python rpc handler exit crash in python 3.5 spwan tests, backtrace showed pythonRpcHandler destructor dec_ref() deallocated python objects, that means python objects in pythonRpcHandler have been cleaned up before destructing pythonRpcHandler while python program exits in python 3.5. This diff is to explicitly clean up python objects of pythonRpcHandler in RpcAgent.join(). so that pythonRpcHandler destructor will not have chances to call dec_ref(). we did not explicitly clean up python objects in RpcAngent destructor, it is also because python objects will be cleaned up before destructing RpcAgent when python program exit in python 3.5. close #27182 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/) [ghstack-poisoned]

[test all] Explicitly clean up py::objects to avoid segment faults when py::objects with CPython are cleaned up later at program exit. See similar issues reported pybind/pybind11#1598 and pybind/pybind11#1493. Our local tests also caught this segment faults if py::objects are cleaned up at program exit. The explaination is: CPython cleans up most critical utitlies before cleaning up PythonRpcHandler singleton, so when PythonRpcHandler signleton cleans up py::objects and call dec_ref(), it will crash. The solution is to clean up py::objects earlier when Rpc agent join(). Be note that py::objects can not be cleaned up when Rpc agent is destroyed as well, as Rpc agent is global variable and it will have same issue as PythonRpcHandler. close #27182 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/) [ghstack-poisoned]

Pull Request resolved: #27251 Explicitly clean up py::objects to avoid segment faults when py::objects with CPython are cleaned up later at program exit. See similar issues reported pybind/pybind11#1598 and pybind/pybind11#1493. Our local tests also caught this segment faults if py::objects are cleaned up at program exit. The explaination is: CPython cleans up most critical utitlies before cleaning up PythonRpcHandler singleton, so when PythonRpcHandler signleton cleans up py::objects and call dec_ref(), it will crash. The solution is to clean up py::objects earlier when Rpc agent join(). Be note that py::objects can not be cleaned up when Rpc agent is destroyed as well, as Rpc agent is global variable and it will have same issue as PythonRpcHandler. close #27182 ghstack-source-id: 91898924 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/)

mrshenli · 2019-10-15T01:58:38Z

torch/csrc/distributed/rpc/python_rpc_handler.cpp


+// if cachedFn is not none, assign it to local function; otherwise, import
+// function from the scratch and assign it to local function;
+py::object getLocalFunction(const py::object& cachedFn, const char* name) {


why do we need this given that the three py functions are already loaded in the constructor?

because the three py functions will be cleaned up in Rpc agent join(), but they could be called after Rpc agent join() in rare cases.

for most cases, we do not need to load them again, just copy cachedFn to localFn, this should be cheap as it is just copying pointer.

Does it make sense?

but they could be called after Rpc agent join() in rare cases.

when could this happen? And if we load them again, do we need to cleanup them again?

e.g. loadPythonUDFResult() is called in future.wait(), which could be called after Rpc agent join().

it does not need to clean up them again, as it is loaded to a local variable intentionally for this rare case

I see. Let's get this PR in to fix master, and let's discuss whether we need to support the above case. I feel that all future.wait() in user code should be resolved before calling join, and if we detect termination in join(), RequestCallbackImpl won't hit this case either (?).

@mrshenli yes, RequestCallbackImpl will not hit this case. but I still add it for runPythonUDF() function for safety reason, because in theory functions in PythonRpcHandler could be called anytime.

mrshenli · 2019-10-15T17:21:13Z

torch/csrc/distributed/rpc/python_rpc_handler.cpp

+  py::object localFn = py::none();
+  if (cachedFn.is_none()) {
+    localFn =
+        getFunction(py::module::import("torch.distributed.rpc.internal"), name);
+  } else {
+    localFn = cachedFn;
+  }
+  return localFn;
+}


How about:

if (cachedFn.is_none()) { return getFunction(py::module::import("torch.distributed.rpc.internal"), name); } else { return cachedFn; }

mrshenli · 2019-10-15T17:22:24Z

torch/csrc/distributed/rpc/python_rpc_handler.h

+  // and https://github.com/pybind/pybind11/issues/1493
+  // Our local tests also caught this segment faults if py::objects are cleaned
+  // up at program exit. The explaination is: CPython cleans up most critical
+  // utitlies before cleaning up PythonRpcHandler singleton, so when


utitlies -> utilities ?

torch/csrc/distributed/rpc/python_rpc_handler.h

mrshenli · 2019-10-15T17:29:57Z

torch/csrc/distributed/rpc/python_rpc_handler.cpp

+    localFn =
+        getFunction(py::module::import("torch.distributed.rpc.internal"), name);
+  } else {
+    localFn = cachedFn;


(Does not need to be fixed in this PR) This will always make a copy of the py::object even if the cachedFn is available. Do you know how expensive it is to make a copy of py::object? It is just similar as creating a shared_ptr? Will it work if we return a const reference here to extend the lifetime of a temporary var?

if I understand correctly, py::object is a pointer, so should be cheap. I would rather not to return a const reference to extend lifetime, because I do not know when the const will be destroyed, may hit the same issue as now

[test all] Explicitly clean up py::objects to avoid segment faults when py::objects with CPython are cleaned up later at program exit. See similar issues reported pybind/pybind11#1598 and pybind/pybind11#1493. Our local tests also caught this segment faults if py::objects are cleaned up at program exit. The explaination is: CPython cleans up most critical utitlies before cleaning up PythonRpcHandler singleton, so when PythonRpcHandler signleton cleans up py::objects and call dec_ref(), it will crash. The solution is to clean up py::objects earlier when Rpc agent join(). Be note that py::objects can not be cleaned up when Rpc agent is destroyed as well, as Rpc agent is global variable and it will have same issue as PythonRpcHandler. close #27182 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/) [ghstack-poisoned]

Pull Request resolved: #27251 Explicitly clean up py::objects to avoid segment faults when py::objects with CPython are cleaned up later at program exit. See similar issues reported pybind/pybind11#1598 and pybind/pybind11#1493. Our local tests also caught this segment faults if py::objects are cleaned up at program exit. The explaination is: CPython cleans up most critical utitlies before cleaning up PythonRpcHandler singleton, so when PythonRpcHandler signleton cleans up py::objects and call dec_ref(), it will crash. The solution is to clean up py::objects earlier when Rpc agent join(). Be note that py::objects can not be cleaned up when Rpc agent is destroyed as well, as Rpc agent is global variable and it will have same issue as PythonRpcHandler. close #27182 ghstack-source-id: 91961049 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/)

mrshenli

The latest version looks good!

zhaojuanmao · 2019-10-16T16:12:56Z

failures are not related

[test all] Explicitly clean up py::objects to avoid segment faults when py::objects with CPython are cleaned up later at program exit. See similar issues reported pybind/pybind11#1598 and pybind/pybind11#1493. Our local tests also caught this segment faults if py::objects are cleaned up at program exit. The explaination is: CPython cleans up most critical utitlies before cleaning up PythonRpcHandler singleton, so when PythonRpcHandler signleton cleans up py::objects and call dec_ref(), it will crash. The solution is to clean up py::objects earlier when Rpc agent join(). Be note that py::objects can not be cleaned up when Rpc agent is destroyed as well, as Rpc agent is global variable and it will have same issue as PythonRpcHandler. close #27182 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/) [ghstack-poisoned]

Pull Request resolved: #27251 Explicitly clean up py::objects to avoid segment faults when py::objects with CPython are cleaned up later at program exit. See similar issues reported pybind/pybind11#1598 and pybind/pybind11#1493. Our local tests also caught this segment faults if py::objects are cleaned up at program exit. The explaination is: CPython cleans up most critical utitlies before cleaning up PythonRpcHandler singleton, so when PythonRpcHandler signleton cleans up py::objects and call dec_ref(), it will crash. The solution is to clean up py::objects earlier when Rpc agent join(). Be note that py::objects can not be cleaned up when Rpc agent is destroyed as well, as Rpc agent is global variable and it will have same issue as PythonRpcHandler. close #27182 ghstack-source-id: 92035069 Differential Revision: [D17727362](https://our.internmc.facebook.com/intern/diff/D17727362/)

facebook-github-bot · 2019-10-17T01:08:18Z

This pull request has been merged in 3214f13.

Summary: Pull Request resolved: pytorch#27251 Explicitly clean up py::objects to avoid segment faults when py::objects with CPython are cleaned up later at program exit. See similar issues reported pybind/pybind11#1598 and pybind/pybind11#1493. Our local tests also caught this segment faults if py::objects are cleaned up at program exit. The explaination is: CPython cleans up most critical utitlies before cleaning up PythonRpcHandler singleton, so when PythonRpcHandler signleton cleans up py::objects and call dec_ref(), it will crash. The solution is to clean up py::objects earlier when Rpc agent join(). Be note that py::objects can not be cleaned up when Rpc agent is destroyed as well, as Rpc agent is global variable and it will have same issue as PythonRpcHandler. close pytorch#27182 ghstack-source-id: 92035069 Test Plan: unit tests on python 3.6 and python 3.5 Differential Revision: D17727362 fbshipit-source-id: c254023f6a85acce35528ba756a4efabba9a519f

zhaojuanmao requested review from mrshenli and pietern as code owners October 2, 2019 22:56

pytorchbot added oncall: distributed Add this issue/PR to distributed oncall triage queue module: pybind Related to our Python bindings / interactions with other Python libraries module: tests Issues related to tests (not the torch.testing module) labels Oct 2, 2019

zhaojuanmao requested a review from pritamdamania87 October 3, 2019 06:00

pritamdamania87 reviewed Oct 3, 2019

View reviewed changes

mrshenli reviewed Oct 4, 2019

View reviewed changes

pritamdamania87 reviewed Oct 4, 2019

View reviewed changes

mrshenli approved these changes Oct 8, 2019

View reviewed changes

pietern suggested changes Oct 10, 2019

View reviewed changes

mrshenli reviewed Oct 14, 2019

View reviewed changes

torch/csrc/distributed/rpc/python_rpc_handler.cpp Show resolved Hide resolved

mrshenli mentioned this pull request Oct 14, 2019

Make RRef::toHere() return a jit::Future #27943

Closed

zhaojuanmao changed the title ~~fix python rpc handler exit crash in python 3.5 spwan tests~~ fix python rpc handler exit crash Oct 14, 2019

mrshenli reviewed Oct 15, 2019

View reviewed changes

mrshenli approved these changes Oct 15, 2019

View reviewed changes

zhaojuanmao requested a review from apaszke as a code owner October 15, 2019 20:49

mrshenli approved these changes Oct 15, 2019

View reviewed changes

facebook-github-bot closed this in 3214f13 Oct 16, 2019

facebook-github-bot added the merged label Oct 17, 2019

facebook-github-bot deleted the gh/zhaojuanmao/9/head branch October 28, 2019 22:23

mruberry added the Merged label Oct 28, 2020

thinking-tower mentioned this pull request Jun 12, 2021

Set kp_debug, kp_info, kp_warning and kp_error to py::none() when the program terminates. KomputeProject/kompute#231

Merged

fix python rpc handler exit crash #27251

fix python rpc handler exit crash #27251

Uh oh!

Conversation

zhaojuanmao commented Oct 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhaojuanmao commented Oct 4, 2019

Uh oh!

mrshenli commented Oct 4, 2019

Uh oh!

zhaojuanmao commented Oct 4, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrshenli left a comment

Choose a reason for hiding this comment

Uh oh!

pietern commented Oct 10, 2019

Uh oh!

pietern left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrshenli left a comment

Choose a reason for hiding this comment

Uh oh!

zhaojuanmao commented Oct 16, 2019

Uh oh!

facebook-github-bot commented Oct 17, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

zhaojuanmao commented Oct 2, 2019 •

edited

Loading