This should be integrated into spack create. I'm willing to do this after #2718 is complete.

This should be integrated into spack create. I'm willing to do this after #2718 is complete.

Thanks, I'm glad to see you're interested in working on it! But... I'd like to see how far we can push good auto-generation before we worry about how it's integrated into the rest of Spack. A bigger question would be... how would #2709 need to change in order to support this functionality? I'd be happy to have a PyPI class, to be used by auto-generated Python packages.

Another thing to consider is to create a separate repo in Spack for Python packages. It could have its own namespace, we could consider getting rid of the py- prefix stuff.

It would be a replacement because if our packages have long PyPI URLs embedded in them, we no longer need the pypi fetch mechanism.
How does this solve the same problem that #2718 was meant to address? The problem is that we don't want to have these long URLs. There was never any difficulty with adding them manually, it's just annoying and unnecessary.

Maybe we need to be more clear about what problem we're trying to address, and how we feel about the relative benefits of the solution.

I agree, I don't like the long URLs. They are ugly to look at. But now that you point that out, I'm wondering why I've spent so much time on this issue: not having to look at ugly URLs is not worth very much.

A bigger problem was how to find those URLs in the first place. That always seemed to be a challenge. Both #2718 and this proposal address that problem, but at different points in the process --- this proposal at package creation time, #2718 at package execution time. If I had to choose between the two, I'd lean toward this one: it takes complexity out of package execution, which is already complex enough. (I know, I originally proposed #2718; but now I'm wondering if doing it this way would be better).

Beyond the problem of long/ugly/hard-to-find URLs, this PR proposes to address a bigger problem --- getting along with existing packaging systems. So far, we have no good solutions. If this works out, we will we get a bonanza of Python Spack packages. But we will also be able to apply the same ides to R, JavaScript, etc. The possible prize, and win for Spack, is much bigger than getting rid of ugly URLs.

A bigger question would be... how would #2709 need to change in order to support this functionality? I'd be happy to have a PyPI class, to be used by auto-generated Python packages.

I think in #2281 we decided on using a PyPiFetchStrategy over a PyPiPackage, right?

Another thing to consider is to create a separate repo in Spack for Python packages. It could have its own namespace, we could consider getting rid of the py- prefix stuff.

This is unrelated. We can continue this particular debate in #2094.

A bigger problem was how to find those URLs in the first place. That always seemed to be a challenge. Both #2718 and this proposal address that problem, but at different points in the process --- this proposal at package creation time, #2718 at package execution time.

When was it a problem to find the URLs? Anyone can Google pypi six, right click on the proper download link, and use it. #2718 doesn't make it easier to find the URLs, it makes the URLs unnecessary.

Yes, the ugly URL problem isn't that big of a deal. But when the only URL available contains a hash, things like spack versions or spack checksum are useless. My plan is to make them work properly with PyPi packages.

The way I see it, #2718 is important for fetching and finding different versions of packages hosted on PyPi. Your proposal is important for making it easier to create new packages by autofilling things like homepage, description, and dependencies. Both are important, but one does not replace the other. They are orthogonal.

A bigger question would be... how would #2709 need to change in order to support this functionality? I'd be happy to have a PyPI class, to be used by auto-generated Python packages.

I think in #2281 we decided on using a PyPiFetchStrategy over a PyPiPackage, right?

We did, but this is a different question. In #2281, we were looking at how to adjust URLs, and assuming that packages were (essentially) hand-built. Now we're considering auto-generated packages. Auto-generated code frequently looks different from hand-built code, and has different needs. It's entirely reasonable to considering maybe having a base class that supports those needs.

Then again, maybe we don't need a new base class for auto-generated code.

Another thing to consider is to create a separate repo in Spack for Python packages. It could have its own namespace, we could consider getting rid of the py- prefix stuff.
This is unrelated. We can continue this particular debate in #2094.

I disagree. It's reasonable to consider keeping auto-generated packages separate from hand-built packages. It depends on the degree of auto-generation that we achieve.

A bigger problem was how to find those URLs in the first place. That always seemed to be a challenge. Both #2718 and this proposal address that problem, but at different points in the process --- this proposal at package creation time, #2718 at package execution time.

When was it a problem to find the URLs? Anyone can Google pypi six, right click on the proper download link, and use it. #2718 doesn't make it easier to find the URLs, it makes the URLs unnecessary.

Yes, the ugly URL problem isn't that big of a deal. But when the only URL available contains a hash, things like spack versions or spack checksum are useless. My plan is to make them work properly with PyPi packages.

Would moving to a more automated approach would make such things obsolete? What if you could just run spack recreate, and Spack would edit all the auto-generated packages for an entire DAG, adding versions and checksums where appropriate?

The way I see it, #2718 is important for fetching and finding different versions of packages hosted on PyPi. Your proposal is important for making it easier to create new packages by autofilling things like homepage, description, and dependencies.

This PR could also be used to update existing packages, bringing more automation all parts of the process.

It's reasonable to consider keeping auto-generated packages separate from hand-built packages. It depends on the degree of auto-generation that we achieve.

All (or most) packages in Spack were auto-generated by spack create. Your suggestion is just to make auto-generation better. I wouldn't want anything added to Spack that wasn't tested and vetted to make sure the installation works. As you mentioned, the list of dependencies isn't as robust as you would like.

spack recreate sounds an awful lot like spack create --force.

This PR has been stale for almost 3 years and it is in a very early stage of discussion. Since Spack is now extendable with custom commands I would be for closing the PR and perfectioning any attempt of doing what's proposed here in a custom command. Waiting for a while to see if there are objections before closing though.

Since there were no objections I'm closing this. Feel free to reopen if you want to continue working on it.