Allow a per-system configuration scope in a location that doesn't require root access

Rationale

At our site the third-party-software support staff do not have root access to systems and therefore cannot easily write to /etc.

We prefer to use a single spack instance (on the shared filesystem) for all users and systems, rather than maintaining multiple spack instances. However, different systems have different compilers and external packages.

Spack does sort-of support this, via the "system" configuration scope, however unlike other configuration scopes, this is in a location requiring root access. Asking our sysadmin team to update OS images and ansible scripts for each modification to compilers.yaml and packages.yaml isn't really practical for us.

Description

Alternatives are:

• it is easy to patch architecture.py to check an environment variable and adjust the platform name accordingly, and then use the platform settings in the site scope for system-specific configuration. But this is inelegant (conflates platform with system) and probably unsustainable.
• We can make environments for each system, but then the user needs to activate the relevant environment each time, and while in the environment "spack find" doesn't find upstream installed packages. It is also messy if the user wants to create a new environment, they won't automatically get the system-specific compilers and external packages

An alternative, or configurable, location for system-specific configuration, would solve this

Additional information

General information

• I have run spack --version and reported the version of Spack
  We are using spack/0.16.1, with some local and cherry-picked patches to solve other problems

• I have searched the issues of this repo and believe this is not a duplicate
  More specific configuration scopes #4427 describes a related goal, but is using platform-os-arch as a proxy for system (and is not constrained by a lack of root access)