Repository navigation
ipython crashes with the following message. I do not what went wrong. Can you help me identify the problem? #2426
Description
Traceback (most recent call last):
File "/usr/bin/ipython", line 8, in
launch_new_instance()
File "/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/ipapp.py", line 402, in launch_new_instance
app.initialize()
File "", line 2, in initialize
File "/usr/lib/python2.7/dist-packages/IPython/config/application.py", line 84, in catch_config_error
return method(app, _args, *_kwargs)
File "/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/ipapp.py", line 312, in initialize
self.init_shell()
File "/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/ipapp.py", line 332, in init_shell
ipython_dir=self.ipython_dir)
File "/usr/lib/python2.7/dist-packages/IPython/config/configurable.py", line 318, in instance
inst = cls(_args, *_kwargs)
File "/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/interactiveshell.py", line 183, in init
user_module=user_module, custom_exceptions=custom_exceptions
File "/usr/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 431, in init
self.init_history()
File "/usr/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 1439, in init_history
self.history_manager = HistoryManager(shell=self, config=self.config)
File "/usr/lib/python2.7/dist-packages/IPython/core/history.py", line 409, in init
self.new_session()
File "", line 2, in new_session
File "/usr/lib/python2.7/dist-packages/IPython/core/history.py", line 63, in needs_sqlite
return f(a,*kw)
File "/usr/lib/python2.7/dist-packages/IPython/core/history.py", line 428, in new_session
self.session_number = cur.lastrowid
OperationalError: database is locked
Activity
takluyver commentedon Sep 24, 2012
Did you have another copy of IPython available at the time? Is your home directory on a network folder?
ajithpad commentedon Sep 24, 2012
No I did not have any other copy of ipython running at the time. Yes, my home directory is part of the network folder.
takluyver commentedon Sep 24, 2012
OK, that will be the problem - IPython stores history in an sqlite database, and sqlite has problems with locking files on the network. We haven't found any good way to automatically detect whether a folder is on a network drive or not.
To work around it: look for the file
ipython_config.pyin the location given byipython locate profile. If it doesn't exist, runipython profile createto make it. Within that file, search forc.HistoryManager.hist_file, and set it to something on the local filesystem (it can be in '/tmp' if you're not interested in your command history. If nothing on the local system is writeable, set it to':memory:'to use an in-memory database.ajithpad commentedon Sep 24, 2012
Thank you. But two issues
takluyver commentedon Sep 24, 2012
Hmm, what version of IPython are you using? You could try
ipython profile create --resetto make new default config files.Yes, everything in the file is commented out by default, so that we can
change the defaults in later releases. Uncomment anything you want to
change.
ajithpad commentedon Sep 24, 2012
I use version 2.7.3 . ipython profile create --reset also did not also create the file with the c.HistoryManager.hist_file. Is there some work around? Thanks for your help :)
takluyver commentedon Sep 24, 2012
That's your Python version. IPython's version number will probably be 0.12,
0.12.1 or 0.13.
If you search through the file for 'hist', do you see any similar entries?
Another possible workaround is to set the IPYTHONDIR environment variable
to a folder on the local filesystem.
ajithpad commentedon Sep 24, 2012
I used export IPYTHONDIR = /tmp and it worked. I made changes to reflect the same in my .bashrc. But I still don't understand why it worked? Thank you so much for the help and time :)
takluyver commentedon Sep 24, 2012
IPython puts all your files, including the history db, into a directory -
normally .config/ipython or .ipython, but if you specify IPYTHONDIR, it
uses that instead. If it's on a network folder, the history database has
the problem you already saw.
The downside of setting IPYTHONDIR to /tmp is that you can't permanently
set any config, because it looks for config in that folder.
I'll close this issue, because we already know about the problem, but we
don't think there's any good way to fix it.
ajithpad commentedon Sep 24, 2012
Thank you once again..
bfroehle commentedon Sep 24, 2012
We store the history of commands you type into IPython in a SQLite database in the folder @takluyver mentioned earlier. To prevent two processes from accidentally writing to the SQLite database file at the same time, the file is locked --- meaning only one process can edit the file at a time.
Normally this file locking is handled by the operating system in a robust way. However there are some known issues with SQLite on NFS drives. This might be what you are experiencing.
You can read more about this error: SQLite: Database Is Locked.
The
export IPYTHONDIR=/tmpworks by placing a new history database (history.sqlite) in the/tmpdirectory which is presumably local and for which file locking works properly. You might also try to delete the SQLite history database in your home directory, or move it away and copy it back (as the previous link mentions), in an attempt to manually "unlock" the stuck file.bfroehle commentedon Sep 24, 2012
@takluyver Do we have a page on our wiki for this bug? It does seem to come up from time to time. It might be worth catching this error and referencing a wiki page to help users get un-stuck from this bug.
takluyver commentedon Sep 24, 2012
Good idea, that sounds worthwhile. The only issue is that I don't know if the error appears in a consistent place - but we can just catch it in this location, and add catches when other tracebacks are reported. If you've got time to haev a go at this, you're welcome to.
amelio-vazquez-reina commentedon May 17, 2013
Hi all @bfroehle I am running into this problem too. I sometimes have to spawn hundreds of jobs on a cluster all running the same code (on different datasets). Part of this code does the following:
(but it does not start the shell)
WIth so many jobs running in parallel many of them attempt to access a SQLite that is locked at the same time (this is all on a network drive), which seems to cause the Python script running the code above to exit abruptly.
Is there a way to have IPython wait until the SQlite database is unlocked rather than abruptly exiting if is locked? I guess that as you all said, this is handled by the OS and there is no way to circumvent this, correct?
As for the
export IPYTHONDIR = /tmptrick, what would happen if multiple Python processes attempt to access SQlite simultaneously on the same machine? Would the OS handle this situation properly?minrk commentedon May 17, 2013
It should be fine on a normal filesystem. The problem is generally that the way the locking / coordination is implemented is often broken when accessed over NFS.