Also, is there an IPython executable?

Yes, it's called ipython.

Also, is there an IPython executable? I'm wondering if we could get more flexibility by specifying the executable as an argument, modifying e.g. PYTHONPATH, and then launching the executable (vs. importing the relevant interpreter library) - that would allow supporting other interpreter options without modifying Spack.

This would definitely make sense for something that we cannot interact with via Python, but for these interpreters that's not the case so I'm not sure it's warranted yet. Also, we would need to be able to still provide the local environment with spack ready to go, which would be more challenging allowing for the user to specify any old executable. It's a cool idea, but maybe we should consider it when there is evidence that users want to use their own custom interpreter executables?

okay! I've updated the PR to more explicitly exit with an error message for the user when a combination of args is invalid (e.g., a command and anything other than ipython) or if ipython cannot be imported. See a691e9a.

(EDIT: this was already addressed by #20329 (comment))

Also, is there an IPython executable? I'm wondering if we could get more flexibility by specifying the executable as an argument, modifying e.g. PYTHONPATH, and then launching the executable (vs. importing the relevant interpreter library) - that would allow supporting other interpreter options without modifying Spack.

This would definitely make sense for something that we cannot interact with via Python, but for these interpreters that's not the case so I'm not sure it's warranted yet

This makes sense but are there interpreters other than IPython or the default Python which are available as imports? If not, I think -i should be a flag option rather than a user-specified option. -i would be a bit too succinct though so IMO the flag should be called --ipython

Yes, all the ones that I mentioned are available to be called interactively. If it's a Python library that gets called from the command line, logically there is a way to bypass the client and do from within Python. Here are a few examples for the ones I mentioned:

• bpython.embed
• jupyter

I think we should take this approach, the simplest one, until there is strong reason (user demand) to do otherwise.

This seems somewhat overfit to ipython to me personally, I think largely because of the fact the spack python command reuses the existing interpreter rather than executing a new one with the correct PYTHONPATH (which was on my list of things to try and fix actually). This would be ok with me if it's documented that the only supported interpreters for now are something like "current" and "ipython", so we can expand it out to take a generic path later for things like pypy, iron python etc. later.

This would be ok with me if it's documented that the only supported interpreters for now are something like "current" and "ipython", so we can expand it out to take a generic path later for things like pypy, iron python etc. later.

@trws IMO the documentation added here was clear that only ipython was supported as an alternative. Is there a wording tweak you would suggest?

I also felt that this solution wouldn't get in the way of a set-PYTHONPATH-and-invoke-interpreter-executable-based solution, but let me know if you disagree.

I updated the docs to be consistent with the current implementation, and (hopefully) did a better job to address @trws about explicitly stating the two options. 849b4c5. I also used more consistent capitalization for Python and IPython.

Strange! GitHub links the commit to the repository here (with a message that the commit doesn't belong) but it should be linked to my fork, vsoch@849b4c5.

It probably won't get in the way. The docs seem to have some remaining issues though, in that spack python still runs an interactive repl with or without the flag (and thus is most definitely interactive capable), and line 417 seems to imply that spack python -i python results in IPython?

That was a typo, and having the redundancy is probably confusing. Fixed in c4ca219. The very first part of the section does mention that spack python is interactive, but for the script / command use case you wouldn't want it interactive, which is what I was trying to say, maybe I didn't do a good job? @trws is there a better way we could say this?

As a possible solution, I just removed the extra "clarification" (which wasn't very good, my bad!) about why we would only allow python for commands and scripts. It now just directly states the options without trying to explain them, which sounds much better I think.

The doc updates look good. 👍

The thing I'm contemplating now though is that while I occasionally use ipython, I almost never use it interactively. I normally use it to get better errors, or turn on its pdb integration and use it as a debugger, things like this:

❯ ipython ~/scripts/broken-script.py
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/scripts/at-home.py in <module>
      8 host = check_output("hostname", shell=True)
      9
---> 10 if 'hurricane' in host or 'gale' in host:
     11     sys.exit(0)
     12 if "typhoon" in host or "storm" in host or "bolt" in host:

TypeError: a bytes-like object is required, not 'str'

Why prevent this use?

The thing I'm contemplating now though is that while I occasionally use ipython, I almost never use it interactively. I normally use it to get better errors, or turn on its pdb integration and use it as a debugger, things like this:

Oh, interesting, I've never used it like that before! I typically just embed it everywhere to get an interactive session to develop and debug. @trws if I understand correctly, we should allow using ipython for the script use case (and string arguments as well?) and remove the checks that don't allow it?

If possible? I actually use embedding too, but usually by adding an embed() call into the script to break at that point. I may just be strange like that, but if it wouldn't impose a significant amount of extra work it would be really nice, especially for debugging spack itself come to think of it.

@trws I gave it a shot! 8fa2064. One thing I'm wondering about is if it should be allowed to run a command and a script. It seems to functionally work for python (and I tested for ipython but it's not included in the PR here) but I'm not sure it's working in practice. For example, let's say I have a script that prints the current working directory, but doesn't have the os import:

# script.py
print(os.getcwd())

If I run the script with the python interpreter, this is what I'd expect:

$ bin/spack python script.py 
Traceback (most recent call last):
  File "script.py", line 2, in <module>
    print(os.getcwd())
NameError: name 'os' is not defined

Arguably since the console command is run before the script, I should be able to add it, and then see the printing:

$ bin/spack python -c "import os" script.py
# nothing printed

It's not an issue of the import working, because I can change to sys (and I'd expect to see the error message again) but I don't:

$ bin/spack python -c "import sys" script.py
# nothing printed

The only thing that "works" at least to run and still produce the output is to have the -c argument handed as a python arg instead of to the spack argparser:

$ bin/spack python script.py -c "import os"
Traceback (most recent call last):
  File "script.py", line 2, in <module>
    print(os.getcwd())
NameError: name 'os' is not defined

But the import is still not done in context of the script.

So if this is a bug, while it's present we should check that both are defined in the parent function and cut out early if this is the case. If the user wants to run a command alongside a script, arguably it's just easiest to write it into the script. Let me know what you think!

Wow, I had not thought of that at all. So, doing a bit of similar testing with python the executable, it seems as though if you specify -c then the positional arguments are treated as arguments to the command rather than executed as scripts, so I suppose I would do that? Just looked at the documentation, it looks like -c terminates the argument list, so yeah probably simplify that way. Really interesting question, and I like the thought of being able to inject something like that too, but for this probably go for consistency.

So just to clarify, you think that we shouldn’t allow specifying both (the spack -c, not a -c to go to Python) and exit early if this is the case? This is my suggestion because having both doesn’t seem to work! And of course the user is free to test specifying -c after a command so it’s handed off to Python or IPython.

I just pushed a fix 75e5c83 to cut out early if both are specified. So this won't work:

$ spack python -c "import sys" script.py
==> Error: You can only specify a command OR script, but not both.

But this will, and the user is free to do whatever they like! Likely a separate command could be run first, independent of the script, but not something you expect to hand off to the script like an IPython embed.

$ bin/spack python script.py -c "import os"
Traceback (most recent call last):
  File "script.py", line 2, in <module>
    print(os.getcwd())
NameError: name 'os' is not defined

Thanks!

Wooo merged! Thanks y'all! :D

Thank you, good addition, and thanks for working through the feedback.