Now that I think about it, it might be useful to override url_for_version similarly to the way @alalazo did in #422. That way, if the user doesn't have the PGI source code in a mirror or in their current directory, it will let them know that it is licensed software and needs to be downloaded manually.

I'm going to expand upon the original intent of this PR and provide more general support for software that requires a license. Here is what I propose:

Some licensed software needs to be downloaded manually and added to a repository. I will let @xjrc handle this part of the process in #705.

Once the download is in a mirror, things are pretty straightforward. expand=False allows uncompressed downloads to be happy. Spack shouldn't have any trouble installing them with its current capabilities.

The only thing that makes licensed software special is that a license needs to be setup. As part of this PR, I'm adding packages for the PGI and NAG compilers. These have fairly simple licenses that work in similar ways. Once we start dealing with more esoteric/complicated licenses, my design may need to change. But for now, I propose adding the following package attributes:

• license_required: Boolean. If set to True in a package, this software requires a license.
• license_comment: String. Contains symbol used by license manager to denote a comment. Defaults to #
• license_files: List of strings. These are files that the software searches for by default when looking for a license. For example, PGI searches for license.dat in its installation prefix. NAG searches for nag.key, nag.licence, or nagware.licence in the compiler library directory, your home directory, or the current working directory. We would obviously use the compiler library directory for NAG. The files listed in license_files will be relative paths from the installation prefix. Support for absolute paths could be added for software that searches for licenses outside of its installation prefix, but I will wait until we add packages that need this.
• license_vars: List of strings. Environment variables that can be set to tell the software where to look for a license if it is not in the usual license file. PGI uses PGROUPD_LICENSE_FILE or LM_LICENSE_FILE, NAG uses NAG_KUSARI_FILE.
• license_url: A url. This will contain a url to the license setup instructions for the software. For PGI, this would be http://www.pgroup.com/doc/pgiinstall.pdf. For NAG it would be something like http://www.nag.com/doc/inun/np61/lin-mac/klicence.txt.

These would be the only attributes necessary in a package. The rest of the magic will happen in the install logic. If a package requires a license and has license_files set, the first file is used. It will create the file and print the following message template to it:

A license is required to use $package

The recommended solution is to store your license key in this file. By default, nag searches the following file(s) for a license key (relative to the installation prefix):

${license_files[@]}

Alternatively, use one of the following environment variable(s):

${license_vars[@]}

If you choose to store your license in a non-standard location, you may set one of these variable(s) to the full pathname to the license file, or port@host if you store your license keys on a dedicated license server. You will likely want to set this variable in a module file so that it gets loaded every time someone tries to use nag.

For further information on how to acquire a license, please refer to:

$license_url

You may enter your license below.

Obviously, the format of this file will change depending on whether or not license_vars or license_url are set. Once written, this file will be opened, similar to how spack create <url> works.

If license_files is not set, we will print a message telling the user to set license_vars to point to their license. I'm sure there will turn out to be a lot of other ways that licenses work.

If you have any further suggestions or modifications, please let me know. Here is my progress on this task:

• Add PGI package
• Add NAG package
• Add Allinea packages
• Add support for the package attributes listed above
• Add a set_up_license function that gets called by install
• Add support for various comment symbols used by license managers
• Document, document, document

@tgamblin: Let me know what you think of my solution. The only thing I have left to do is add documentation.

Great. PGI (FlexNet) uses # for comments and NAG (Kusari) uses !. I added a license_comment attribute that contains the symbol used by the license manager to denote a comment. It defaults to #.

Alternatively, I could replace that with a license_type attribute that contains "FlexNet" or "Kusari", but then people would have to edit package.py anytime they want to add a new license type. I figured this solution would be simpler and easier to understand.

@lee218llnl: How do you feel about my proposed solution in regards to an Intel compiler package? I took a brief look at the Intel compiler installation instructions and you can use a configuration file to build the package silently. The only stumbling block will be, as you mentioned, the fact that there are various product suites. Do each of these suites have a different license or can we just install the most complete one by default? We certainly need better support for allowing multiple urls and checksums per version, since a lot of packages differ based on OS/arch/suite, but that could be saved for another day. If you think my design looks decent then I'm willing to get started on an Intel compiler package.

I created a new issue #761 to track the packaging of Intel compilers. Feedback would be appreciated.

@adamjstewart I have an initial implementation of the intel compiler package. What's the best way to test integration with your license file additions? Can you push your branch to the repo so I can check it out?

Looking at the current implementation, I'm not sure if this completely handles the way we normally deal with licenses for the Intel compiler. We have a file in /usr/local/etc/license.client.intel.lic who's contents point to our license server. Usually when I install an Intel compiler, I point the installation script to this file (it needs a license to install). After the installation is done, I then create a symlink in the compiler's bin or license directory that points to the .lic file. The nice thing about doing it this way is that only that one file has to be edited if something changes in the license server configuration.

Does this setup sound generally useful enough to implement in Spack for other sites and software?

@lee218llnl:

What's the best way to test integration with your license file additions? Can you push your branch to the repo so I can check it out?

I don't believe I have permission to push it to the Spack repo. But you should be able to clone the branch from my repo and test it locally. Let me know if you don't have permission to do this.

Looking at the current implementation, I'm not sure if this completely handles the way we normally deal with licenses for the Intel compiler.

My attempt with this PR was to provide a flexible method of allowing users to set up licenses. By printing a list of various files and environmental variables that the compiler looks at, I allow the user to decide the best method for them to setup licenses.

Usually when I install an Intel compiler, I point the installation script to this file (it needs a license to install).

Now this could be a serious stumbling block. Is there really no way to install it and point it to the license later? If not, then yes, my current setup won't work. I can imagine that there is probably other licensed software that behaves this way as well.

My license logic always goes after installation. I could change it so that the set_up_license function isn't called unless someone specifies it in a package's install function. That seems reasonable. Then you could call it at the begging of install. The problem will be that the installation directory won't exist until after installation, so you won't be able to use a relative file path unless the set_up_license function is called at the end.

We have a file in /usr/local/etc/license.client.intel.lic who's contents point to our license server.

After the installation is done, I then create a symlink in the compiler's bin or license directory that points to the .lic file. The nice thing about doing it this way is that only that one file has to be edited if something changes in the license server configuration.

I'm not sure if it's possible to automate the process that you are describing. I agree that the way you do things is easy, but I don't think it's compatible with Spack. I could add logic to allow license files to be specified outside of the installation directory, but then when I printed the license file, it would overwrite the original when you install again.

I see several possible solutions:

1. (easy for Spack, sucks for you): call set_up_license at the beginning of install, put the license key or a SERVER line in that file, finish the installation, and overwrite your local license file with a symlink to your global one. How often is your SERVER line going to change anyway?
2. (harder for Spack, easier for you): Maybe write some kind of loop that checks various locations where the license could be found and set the environment variable based on that? I would need to reread the documentation to get a better idea of how this would work, but I think setting ACTIVATION_TYPE to exist_lic might even do this for us.
3. (hardest for Spack, easiest for user): figure out a way to prompt the user, asking them for a path to their license file, or port@host. If they don't provide anything and just hit enter, open up a local license file that they can edit. I would love to be able to do this, as I think it's very intuitive, and would allow Spack to create a symlink for you. But I have no idea how much work it would be to add this.

How do you feel about these? Can you think of any better solutions?

If passed license information via silent.cfg at install time the Intel installer will install a license file in the licenses directory of the target installation directory. I am not sure if that helps here because there would need to be a way to pass the information to silent.cfg via spack.

@adamjstewart perhaps Spack can have a license directory that each user/site can populate with their own licences. Packages that require licenses could then look in there and create the appropriate copies or symlinks. This could also be a configuration file that points to other existing licenses on the system.
@glennpj in my experience, the installer has not created the licenses directory. In any case, passing the license to silent.cfg actually shouldn't be a problem. I already do this in my initial implementation of the Intel compiler package.

@lee218llnl: I love it! To avoid naming confusion, how about this. If license_files is not empty, set_up_license will create a symlink in the installation prefix (if called at the beginning of the function, it may have to create this directory). This symlink will be called $license_files[0] and will point to a $package/$license_files[0] file in the Spack license directory. That should avoid any naming conflict. If this file does not exist, Spack prints my licensing instructional message and opens the file for editing. If it already exists, Spack prints a message saying it found an existing license and is using it. That way all of your licenses are in the same directory and easy to edit if your license server changes. Of course, you are always welcome to later remove the file in the Spack license directory and replace it with a symlink to a file outside of the Spack hierarchy, but I think this will be a better place to store them. The only remaining problem I see is if a package needs a different license for each version, but let's cross that bridge when we get there.

I'll implement these changes on Monday if they sound good to you. Let me know if you have any suggestions for where to put this Spack license directory. I'm thinking $SPACK_ROOT/etc/spack/licenses

@lee218llnl: When I try moving the call to set_up_license from build_process to each package's install method, I run into the same problem I was having before. Opening the file with spack.editor causes Spack to hang without displaying the editor. I think it's because the output of install is being suppressed. Do you have any suggestions for a workaround? I guess I could always call set_up_license before install, but I'll have to create any directories that don't exist before writing the license symlink. Or just write the global license before install and add the symlink after install. Alternatively, I could add a flag that decides whether to run set_up_license before or after install, but I feel like that's a messy solution.

@adamjstewart sorry for the delayed response, I was busy packaging up the Intel compilers in yet another packaging system that we have to deal with here (I'll leave my rant at that and spare you the details). Anyway, I think your naming and symlink ideas sound good to me. Regarding the editor, I think @tgamblin knows the ins and outs of Spack better and can find you a solution. For me, though, I don't imagine hand typing a license when installing things on Spack, so I don't have a strong desire for this.

@lee218llnl: Nice timing. I'm actually pretty much done with the changes. Give me 10 minutes for testing and I'll commit them.

Ok, I think I've implemented everything as mentioned. It now creates a global license file, then installs the package, and then creates a local symlink that points to the global license. @lee218llnl: For your Intel package, when you need to tell it where your license can be found during installation, you should be able to use:

Package.global_license_file()

Let me know if you catch any bugs. It works for me for NAG. It doesn't delete the global license if you uninstall the package, but it allows you to use the same global license for multiple versions of a package without overwriting it if it already exists.

Added a couple of Allinea packages. Works perfectly with my current setup.

@tgamblin: I moved everything except global_license_file() to hooks/licensing.py. I kept global_license_file() in package.py and made it a property to allow users to reference it with self.global_license_file in their packages.

The only consequence that I've noticed so far is that now Spack counts the time it takes for the user to look up and add their license as part of the build time. But that's probably not that important.

@tgamblin: Not sure how to add coverage for this one, but flake8 is happy and everything should be working as I would expect.

@lee218llnl: does this look good to you?

@tgamblin @adamjstewart This looks like it will satisfy my needs for the Intel compilers and tools.

Merged!