Skip to content

[Discussion] How to handle os/version/arch-specific downloads#2009

Closed
adamjstewart wants to merge 2 commits intospack:developfrom
adamjstewart:discussion/os-specific-download
Closed

[Discussion] How to handle os/version/arch-specific downloads#2009
adamjstewart wants to merge 2 commits intospack:developfrom
adamjstewart:discussion/os-specific-download

Conversation

@adamjstewart
Copy link
Copy Markdown
Member

This PR is more of a discussion, but could probably be merged anyway if people like it.

The question is, how do we handle software that provides a different download depending on your operating system, version, architecture, etc? I chose allinea-forge as a particularly nasty example. See the downloads page here.

Software like this provides two problems:

  1. A single version might have a different URL depending on the cluster
  2. A single version might have a different checksum depending on the cluster

The second problem could be solved by including the os-version-arch in the version, but I tried to avoid this hack.

My solution was to create a function that determined the appropriate version to download. For example, if you are on RHEL or CentOS, it uses the "Redhat" version. It would be great if os-version-arch stuff like this could be included in the core Spack libraries, but since each package may name things differently, I don't think this is possible. For example, one package could have a "Redhat" version and another could have both a "rhel" and "centos" version.

With this function, I can define the url_for_download() method. And with a case-statement-like if-statement, I can define the checksums for each version, depending on the os-version-arch.

@citibeth This is what I was talking about in #1997.

Note: since I don't have every possible platform available to me, I could only test on CentOS. I couldn't find any good documentation on platform so I don't know the exact string for each possible system, linux distro, etc. I also don't know mappings between Fedora versions and RHEL versions, so I didn't provide support for Fedora. There are probably a lot of distros that could use one of these versions even if they aren't explicitly listed. Mint could probably use the Ubuntu download for example. These would have to be added manually. If you can think of a better solution, let me know.

@davydden
Copy link
Copy Markdown
Member

i think the Version part is what i would also like to see in Spack #1814, but I think it would also make sense to have access to os through classes which are already in Spack. The problem with re-using Spack here is that you need spec, i.e. spec.architecture.platform_os.version.

Even though Spack may not have the same granularity in OS's (Redhat vs rhel /centos), i think it would still save you some lines of code here.

@adamjstewart
Copy link
Copy Markdown
Member Author

Yeah, I wanted to use those. But I don't think it's documented, so I don't know what all of the options are or what possible values they return (if someone wants to do this that would be fantastic). And a lot of those return the same things as platform.* anyway. They're just wrappers, like how join_path is just a copy of os.path.join. So the lines of code are the same and the results are mostly the same. There might be different results for things like BGQ or Cray, but it's not documented and I can't test it. And like you said, you need the spec, which isn't necessarily available outside of install().

@tgamblin
Copy link
Copy Markdown
Member

@adamjstewart: they're wrappers if you assume that the build host is where you'll actually run. The difference is that they're attributes on the spec, which might be a spec for some other architecture (e.g. backend nodes on BG/Q or Cray). So making a Python call isn't always going to give you the same thing as checking an attribute on a spec. But yes, they need documentation otherwise i can't really expect people to use them.

@adamjstewart
Copy link
Copy Markdown
Member Author

That's a good explanation. Yeah, I can use them as soon as they get documented. Aside from that, how do people feel about this strategy? It's kind of a pain in the ass for packages with as many downloads as allinea-forge, but it's really not that bad for other packages that provide a different version for Linux and Mac. Compilers are usually inbetween, and since those are usually downloaded manually, we don't even have to worry about url_for_version().

@davydden
Copy link
Copy Markdown
Member

since you only need to have different url's and hashes, maybe in addition to url_for_version there could be hash_for_version? Both should have access to spec, which would not be the case if we were talking about depends_on.

@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Oct 12, 2016

So, I think we clearly need conditional logic for this, and what you're doing here will work right now. But if it turns out that you need a separate tarball for a backend arch, this won't work.

I think version needs a when clause like the other directives, where you could put when='platform=darwin' or when='platform=centos.*'.

I would probably initially make the when on version only support, e.g., platform and os, as if you make it allow other things, picking a version during concretization can get complicated (it starts depending on more things and you have to iterate). But some future concretizer could handle that.

Thoughts?

@adamjstewart
Copy link
Copy Markdown
Member Author

I think adding when support to version() would make a lot of sense. It would just need to support everything I did in this PR, including os, distro, os-version, and arch. I can't think of a need for it to have full access to the spec aside from that.

@adamjstewart
Copy link
Copy Markdown
Member Author

This currently crashes on macOS because the mirror logic tries to parse the extension but .dmg is invalid. Honestly I'm not sure if it should be valid. I don't think Spack can install a .dmg file so I guess the package just shouldn't support macOS.

@pramodskumbhar
Copy link
Copy Markdown
Contributor

@adamjstewart : is there any reason not to use allinea-forge template provided in this ticket? It's better (more reliable/robust) than current package.py.

@adamjstewart
Copy link
Copy Markdown
Member Author

The problem is that every single unit test fails:

InstallError: No download available for 'Linux', 'debian', 'wheezy/sid', 'x86_64'

This solution is not very robust. We need to add a when= directive to version(). We also need to allow packages to exist that have no available versions (at least for the current arch).

@pramodskumbhar
Copy link
Copy Markdown
Contributor

Oh ok, I saw the tests now.

@adamjstewart
Copy link
Copy Markdown
Member Author

Closing as this discussion ended a long time ago.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants