module_cmd::get_path_from_module_contents broken for lmod modules

Given the following output of `module show cray-mpich`:

```
[...]
===================================================================

]])
whatis("cray-mpich - Cray MPICH Message Passing Interface")
setenv("CRAY_MPICH_VER","8.1.4")
setenv("CRAY_MPICH_VERSION","8.1.4")
setenv("CRAY_MPICH_ROOTDIR","/opt/cray/pe/mpich/8.1.4")
setenv("CRAY_MPICH_BASEDIR","/opt/cray/pe/mpich/8.1.4/ofi")
setenv("PE_MPICH_MODULE_NAME","cray-mpich")
setenv("CRAY_MPICH_DIR","/opt/cray/pe/mpich/8.1.4/ofi/aocc/2.2")
setenv("CRAY_MPICH_PREFIX","/opt/cray/pe/mpich/8.1.4/ofi/aocc/2.2")
setenv("MPICH_DIR","/opt/cray/pe/mpich/8.1.4/ofi/aocc/2.2")
setenv("PE_MPICH_PKGCONFIG_VARIABLES","PE_MPICH_GTL_DIR_@accelerator@:PE_MPICH_GTL_LIBS_@accelerator@")
setenv("PE_MPICH_GTL_DIR_gfx906","-L/opt/cray/pe/mpich/8.1.4/gtl/lib")
setenv("PE_MPICH_GTL_DIR_gfx908","-L/opt/cray/pe/mpich/8.1.4/gtl/lib")
setenv("PE_MPICH_GTL_DIR_nvidia70","-L/opt/cray/pe/mpich/8.1.4/gtl/lib")
setenv("PE_MPICH_GTL_DIR_nvidia80","-L/opt/cray/pe/mpich/8.1.4/gtl/lib")
setenv("PE_MPICH_GTL_LIBS_gfx906","-lmpi_gtl_hsa")
setenv("PE_MPICH_GTL_LIBS_gfx908","-lmpi_gtl_hsa")
setenv("PE_MPICH_GTL_LIBS_nvidia70","-lmpi_gtl_cuda")
setenv("PE_MPICH_GTL_LIBS_nvidia80","-lmpi_gtl_cuda")
prepend_path("PE_MPICH_GENCOMPILERS_AOCC","2.2")
prepend_path("PE_MPICH_FIXED_PRGENV","AOCC")
prepend_path("PE_AOCC_FIXED_PKGCONFIG_PATH","/opt/cray/pe/mpich/8.1.4/ofi/aocc/2.2/lib/pkgconfig")
prepend_path("PE_PKGCONFIG_PRODUCTS","PE_MPICH")
prepend_path("PE_MPICH_PKGCONFIG_LIBS","mpich")
prepend_path("PE_MPICH_FORTRAN_PKGCONFIG_LIBS","mpichf90")
prepend_path("PE_PKGCONFIG_LIBS","mpich")
prepend_path("PE_FORTRAN_PKGCONFIG_LIBS","mpichf90")
setenv("PE_PERFTOOLS_MPICH_LIBDIR","/opt/cray/pe/mpich/8.1.4/ofi/aocc/2.2/lib")
prepend_path("MANPATH","/opt/cray/pe/mpich/8.1.4/man/mpich")
prepend_path("MANPATH","/opt/cray/pe/mpich/8.1.4/ofi/man")
prepend_path("PATH","/opt/cray/pe/mpich/8.1.4/ofi/aocc/2.2/bin")
prepend_path("CRAY_LD_LIBRARY_PATH","/opt/cray/pe/mpich/8.1.4/ofi/aocc/2.2/lib")
setenv("CRAY_LMOD_MPI","cray-mpich/8.0")
prepend_path("MODULEPATH","/opt/cray/pe/lmod/modulefiles/mpi/aocc/2.2/ofi/1.0/cray-mpich/8.0")
prepend_path("MODULEPATH","/opt/cray/pe/lmod/modulefiles/cncm/aocc/2.2/ofi/1.0/x86-rome/1.0/cray-mpich/8.0")
```
and the `packages.yaml`:

```yaml
packages:
  cray-mpich:
    externals:
    - spec: cray-mpich@8.1.4
      modules:
      - cray-mpich/8.1.4
    buildable: False
```

one gets when trying to install a package:

```console
$ spack install cp2k@master%aocc +libvori +spglib ^cray-fftw ^cray-mpich ^cray-libsci+mpi+openmp
[...]
==> Warning: Extracted path from module does not exist [module=cray-mpich/8.1.4, path=/opt/cray/pe/mpich/8.1.4/gtl/lib")]
[...]
==> Error: NoLibrariesError: Unable to recursively locate cray-mpich libraries in /opt/cray/pe/mpich/8.1.4/gtl/lib")

/scratch/e1000/timuel/spack/var/spack/repos/builtin/packages/cp2k/package.py:363, in edit:
        360            ldflags.append(scalapack.search_flags)
        361
        362            libs.extend(scalapack)
  >>    363            libs.extend(spec['mpi:cxx'].libs)
        364            libs.extend(self.compiler.stdcxx_libs)
        365
        366            if 'wannier90' in spec:

See build log for details:
  /tmp/timuel/spack-stage/spack-stage-cp2k-master-wphctyex27pnp4cgstki3n6syqrcormm/spack-build-out.txt
```

The problem here is twofold:
* [ ] the matcher function `match_flag_and_strip` does not strip the trailing `")` of the path found by matching `-L`, hence the invalid path:
https://github.com/spack/spack/blob/4f7bd11d61b67666bf11a827650900666f920672/lib/spack/spack/util/module_cmd.py#L195-L204

* [ ] the module specifies additional library directories only needed under specific circumstances with different env vars, making them come out on top here, so, broken heuristics:
https://github.com/spack/spack/blob/4f7bd11d61b67666bf11a827650900666f920672/lib/spack/spack/util/module_cmd.py#L238-L239

Other possible improvements for this function:

* use regex for `-L` match
* use `re.compile` since the regexes are used quiet a lot
* `continue` after a successful match

	def match_flag_and_strip(line, flag, strip=[]):
	flag_idx = line.find(flag)
	if flag_idx >= 0:
	end = line.find(' ', flag_idx)
	if end >= 0:
	path = line[flag_idx + len(flag):end]
	else:
	path = line[flag_idx + len(flag):]
	path = strip_path(path, strip)
	path_occurrences[path] = path_occurrences.get(path, 0) + 1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

module_cmd::get_path_from_module_contents broken for lmod modules #23445

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if len(path_occurrences) > 0:
	return max(path_occurrences.items(), key=lambda x: x[1])[0]

module_cmd::get_path_from_module_contents broken for lmod modules #23445

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions