ipython 5 tab completion not compatible with jpype

New version of ipython probably use threads to implement tab completion that crashes the jvm.

import jpype
jpype.startJVM(jpype.get_default_jvm_path())
jpype.java.lang.System.out.<TAB>

should not crash the JVM, however it does with ipython 5.

This did not happen with version 4 and with jupyter-console (that uses IPython 5.1.0 in my case).

Hum, tab completion has not been touch a lot in 5.x , only the interface.

should crash the JVM.

Did you meant should not ?

@jonathanslenders

That's unfortunate. Yes, completion does use threads, and I don't think there's anything we can easily do about it.

Ping @jonathanslenders - I don't particularly expect a fix, but you may want to be aware of this.

Edited the comment to clarify. I think it is related to threads that usually tends to crash jpype. I open an issue in jpype as well. I am not experts in both ipython and jpype internals unfortunately...

Yes, autocompletion runs in a different thread, because it could otherwise block the UI interaction (we want zero-latency typing). Jedi for instance takes some time to generate the completions. It's not a lot, but noticeable if we would do it in the main event loop.

However, auto completion is not supposed to crash anything. Worst case, it should not give any suggestion.

To inspect the package structure (which will also descent into the Java bindings) one would need to attach the current Python thread to the Java virtual machine. Is there some kind of call back interface, which we could use at jpype's side to bind to the current thread? If not, there is not much we (jpype) nor Ipython folks can do about this.

Right now, there is no way to do that. I could add an option to make auto completion synchronous (in the UI thread), but that won't be for right now and I'm afraid that the performance will suffer.

edit: feel free to create an issue for a feature request for synchronous auto completion. That way I won't forget.

Does jpype use the __dir__ method? In that case would it be possible to check inside __dir__ whether the thread, which is calling __dir__, is attached to the JVM and, if not, attach to it?

It seems to work on my class:

   def __dir__(self):
        if jpype.isThreadAttachedToJVM()==0:
            jpype.attachThreadToJVM()
        return sorted(self._dict.keys())

Definitely running into this issue. I would suspect this happens to any non-thread-safe packages as well? Would be good to have a flag to disable this behavior in IPython.

Would be good to have a flag to disable this behavior in IPython.

You really can't do that easily. Most of the code that was handling the froment is gone in 5.x and has been replaced by prompt-toolkit so it's not a question of "just a flag" unfortunately.

I would expect that __dir__ is always thread safe. Normally, doing an attribute listing should not result in any side effects, so there shouldn't be a reason that __dir__ can't be thread safe. It's really weird if dir(my_object) can fail if my_object was created in a different thread. With this bug, even with the normal Python shell, this would crash if the object was created in another thread.

I don't know jpype, but it looks like __dir__ does some kind of reflection through the Java runtime. If we want to have autocompletion while the user types, and we want to switch to Jedi, then even more it is important to run autocompletions asynchronous. I think it's something worth fixing in Jpype.

I just realized that if you enable readline-style autocompletion, that it doesn't use a separate thread.
See this example: https://github.com/jonathanslenders/python-prompt-toolkit/blob/master/examples/autocompletion-like-readline.py (I'm not sure how that can be configured in IPython.)

edit: Probably it's even the case if complete_while_typing=False is set (without readline-style visualisation).

To configure readline-like completions in IPython, set TerminalInteractiveShell.display_completions = 'readlinelike' (see config docs).

complete_while_typing is already disabled by default in IPython, because we use enable_history_search, and the two options are mutually exclusive. It seems like completion still uses a thread, though.

@rdemaria

@rdemaria, I just want to mention that if you would like the old (IPython <= 4.x) readline interface, it is now available separately as rlipython.

pip install rlipython
python -c "import rlipython as r; r.install()"

Right now, it only works in IPython 6.0 (Python 3), but it will also work in Python 2 with the next IPython 5 release (5.4).

Thanks for the information. Thankfully jpype was update to be compatible with the new ipython behaviour, however I find good there is a way back...

we are facing a similar issue in Sagemath, with an extension that embeds a Common Lisp interpreter ECL, which crashes on some versions of Linux:
https://trac.sagemath.org/ticket/23700#comment:11

Namely, tab completion on import of something from a component using this extension crashes with a segfault.

Navigation Menu

Repository navigation

ipython 5 tab completion not compatible with jpype #10315

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ipython 5 tab completion not compatible with jpype #10315

Description

Activity

Carreau commented on Feb 21, 2017

takluyver commented on Feb 21, 2017

rdemaria commented on Feb 21, 2017

jonathanslenders commented on Feb 21, 2017

marscher commented on Feb 21, 2017

jonathanslenders commented on Feb 21, 2017

rdemaria commented on Feb 21, 2017

rdemaria commented on Feb 21, 2017

nimish commented on Feb 22, 2017

Carreau commented on Feb 22, 2017

jonathanslenders commented on Feb 22, 2017

jonathanslenders commented on Mar 4, 2017

takluyver commented on Mar 4, 2017

ivanov commented on May 17, 2017

rdemaria commented on May 18, 2017

dimpase commented on Sep 28, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions