`spack setup`: Fix Bugs + Multi-setup (2) by citibeth · Pull Request #5043 · spack/spack

citibeth · 2017-08-09T18:46:14Z

Continued from #2664. The branch for this PR was moved from a fork to the main LLNL Spack repo. I have done my best to merge/rebase in a sensible way, but I have not tried this out.

…ere happening with `spack setup`. Avoid concretizing more than once (slow and was probably also causing bugs).

`

…nothing.

…fault versions of development packages can then all be put into `packages.yaml`, freeing the user the need to type (and remember) them when running `spack setup`.

Also add -j to the common arguments

# Conflicts: # lib/spack/spack/build_systems/cmake.py # lib/spack/spack/cmd/common/arguments.py # lib/spack/spack/cmd/install.py # lib/spack/spack/cmd/setup.py

Add everytrace to Mock repo; will be used for spack setup tests. Added unit test: setup

…FixSetup # Conflicts: # lib/spack/spack/build_systems/cmake.py # lib/spack/spack/cmd/install.py # lib/spack/spack/cmd/setup.py # lib/spack/spack/package.py # lib/spack/spack/test/cmd/install.py

pramodskumbhar · 2017-08-13T17:52:06Z

@citibeth : I am trying this PR and getting below error:

trying to build everytrace as:

git clone https://github.com/citibeth/everytrace.git
cd everytrace
spack -d setup everytrace@develop

Error / Relevant traceback is:

==> '/usr/local/bin/git' 'ls-tree' '--name-only' 'HEAD'
==> Reading config file /Users/kumbhar/workarena/software/sources/spack/etc/spack/defaults/packages.yaml
==> Reading config file /Users/kumbhar/workarena/software/sources/spack/etc/spack/defaults/darwin/packages.yaml
==> Reading config file /Users/kumbhar/.spack/darwin/packages.yaml
==> READ LOCK: /Users/kumbhar/.spack/cache/providers/.builtin-index.yaml.lock[0:0] [Acquiring]
....
==> Error: AttributeError: 'Everytrace' object has no attribute 'package'
/Users/kumbhar/workarena/software/sources/spack/lib/spack/spack/package.py:1353, in build_process:
....
     1341                     with redirection_context as log_redirection:
     1342                         for phase_name, phase in zip(
     1343                                 self.phases, self._InstallPhase_phases):
     1344                             tty.msg(
     1345                                 'Executing phase : \'{0}\''.format(phase_name)
     1346                             )
     1347                             # Redirect stdout and stderr to daemon pipe
     1348                             with log_redirection:
     1349                                 getattr(self, phase)(
     1350                                     self.spec, self.prefix)
     1351                     self.log()
     1352                 # Run post install hooks before build stage is removed.
  >> 1353                 spack.hooks.post_install(self.spec)
     1354
     1355             # Stop timer.
     1356             self._total_time = time.time() - start_time
     1357             build_time = self._total_time - self._fetch_time
     1358


Traceback (most recent call last):
  File "/Users/kumbhar/workarena/software/sources/spack/lib/spack/spack/build_environment.py", line 547, in child_execution
    function(input_stream)
  File "/Users/kumbhar/workarena/software/sources/spack/lib/spack/spack/package.py", line 1311, in build_process
    spack.hooks.pre_install(self)
  File "/Users/kumbhar/workarena/software/sources/spack/lib/spack/spack/hooks/__init__.py", line 73, in __call__
    hook(*args, **kwargs)
  File "/Users/kumbhar/workarena/software/sources/spack/lib/spack/spack/hooks/licensing.py", line 34, in pre_install
    pkg = spec.package
AttributeError: 'Everytrace' object has no attribute 'package'

pramodskumbhar · 2017-08-13T18:44:59Z

@citibeth : temporarily I disabled that pre_install in lib/spack/spack/hooks/licensing.py and then spack setup is working fine.

In some of the (old) packages I have :

            '-DCMAKE_C_COMPILER=%s' % spack_cc,
            '-DCMAKE_CXX_COMPILER=%s' % spack_cxx,

And then spaconfig.py was invoking spack compiler wrappers resulting in obvious error:

  Spack compiler must be run from Spack! Input 'SPACK_PREFIX' is missing.
  make[1]: *** [CMakeFiles/cmTC_f0582.dir/testCCompiler.c.o] Error 1

So if I understood correctly :

CMAKE_C_COMPILER (CXX) should not be set in package
MPI_C_COMPILER (CXX) should be ok because these wrappers are filtered i.e. they use actual compilers (?)

bvanessen · 2017-08-31T15:20:05Z

Something is broken:

spack setup lbann@local build_type=Release dtype=4 %[email protected] cflags="-O3 -g -march=haswell -mtune=haswell" cxxflags="-O3 -g -march=haswell -mtune=haswell" fflags="-O3 -g -march=haswell -mtune=haswell" ^elemental@master blas=openblas ^[email protected]

==> [email protected] : externally installed in /usr/tce/packages/mvapich2/mvapich2-2.2-gcc-7.1.0
==> [email protected] : generating module file
==> [email protected] : registering into DB
==> Installing pkg-config
==> Using cached archive: /usr/workspace/wsa/vanessen/spack.git/var/spack/cache/pkg-config/pkg-config-0.29.2.tar.gz
==> Already staged pkg-config-0.29.2-fgoeyhd6mx5k6knkcujjn2csa4mq2gr5 in /usr/workspace/wsa/vanessen/spack.git/var/spack/stage/pkg-config-0.29.2-fgoeyhd6mx5k6knkcujjn2csa4mq2gr5
==> No patches needed for pkg-config
==> Building pkg-config [AutotoolsPackage]
==> Error: AttributeError: 'PkgConfig' object has no attribute 'package'
/usr/workspace/wsa/vanessen/spack.git/lib/spack/spack/package.py:1353, in build_process:
     1279         def build_process(input_stream):
     1280             """Forked for each build. Has its own process and python
     1281                module space set up by build_environment.fork()."""
     1282
     1283             # We are in the child process. This means that our sys.stdin is
     1284             # equal to open(os.devnull). Python did this to prevent our process
     1285             # and the parent process from possible simultaneous reading from
     1286             # the original standard input. But we assume that the parent
     1287             # process is not going to read from it till we are done here,
     1288             # otherwise it should not have passed us the copy of the stream.
     1289             # Thus, we are free to work with the the copy (input_stream)
     1290             # however we want. For example, we might want to call functions
     1291             # (e.g. input()) that implicitly read from whatever stream is
     1292             # assigned to sys.stdin. Since we want them to work with the
     1293             # original input stream, we are making the following assignment:
     1294             sys.stdin = input_stream
     1295 
     1296             start_time = time.time()
     1297             if not this_fake:
     1298                 if not skip_patch:
     1299                     self.do_patch()
     1300                 else:
     1301                     self.do_stage()
     1302 
     1303             tty.msg(
     1304                 'Building {0} [{1}]'.format(self.name, self.build_system_class)
     1305             )
     1306 
     1307             self.stage.keep = keep_stage
     1308             with self._stage_and_write_lock():
     1309                 # Run the pre-install hook in the child process after
     1310                 # the directory is created.
     1311                 spack.hooks.pre_install(self)
     1312                 if self.name in setup:
     1313                     pass    # Don't write any files in the install...
     1314                 elif fake:
     1315                     self.do_fake_install()
     1316                 else:
     1317                     # Do the real install in the source directory.
     1318                     self.stage.chdir_to_source()
     1319 
     1320                     # Save the build environment in a file before building.
     1321                     env_path = join_path(os.getcwd(), 'spack-build.env')
     1322 
     1323                     # Redirect I/O to a build log (and optionally to
     1324                     # the terminal)
     1325                     log_path = join_path(os.getcwd(), 'spack-build.out')
     1326 
     1327                     # FIXME : refactor this assignment
     1328                     self.log_path = log_path
     1329                     self.env_path = env_path
     1330                     dump_environment(env_path)
     1331 
     1332                     # Spawn a daemon that reads from a pipe and redirects
     1333                     # everything to log_path
     1334                     redirection_context = log_output(
     1335                         log_path,
     1336                         echo=verbose,
     1337                         force_color=sys.stdout.isatty(),
     1338                         debug=True,
     1339                         input_stream=input_stream
     1340                     )
     1341                     with redirection_context as log_redirection:
     1342                         for phase_name, phase in zip(
     1343                                 self.phases, self._InstallPhase_phases):
     1344                             tty.msg(
     1345                                 'Executing phase : \'{0}\''.format(phase_name)
     1346                             )
     1347                             # Redirect stdout and stderr to daemon pipe
     1348                             with log_redirection:
     1349                                 getattr(self, phase)(
     1350                                     self.spec, self.prefix)
     1351                     self.log()
     1352                 # Run post install hooks before build stage is removed.
  >> 1353                 spack.hooks.post_install(self.spec)
     1354 
     1355             # Stop timer.
     1356             self._total_time = time.time() - start_time
     1357             build_time = self._total_time - self._fetch_time
     1358 
     1359             if self.name in setup:
     1360                 tty.msg("Successfully setup %s" % self.name,
     1361                         "Config file is %s" % spconfig_fname)
     1362             else:
     1363                 tty.msg("Successfully installed %s" % self.name,
     1364                         "Fetch: %s.  Build: %s.  Total: %s." %
     1365                         (_hms(self._fetch_time), _hms(build_time),
     1366                          _hms(self._total_time)))
     1367 
     1368             print_pkg(self.prefix)

Spack setup command returned error: 1

EDIT: just formatting as code section for readability

pramodskumbhar · 2017-09-16T13:17:20Z

Just for the record: In mean time there are two additional bugs:

as we removed default value for 'dirty' in Removed default value for the function argument 'dirty' #5109, this results in:

==> WRITE LOCK: /gpfs/bbp.cscs.ch/scratch/gss/bgq/kumbhar-adm/SPACK_HOME/install_home/bbpviz_full/install/.spack-db/lock[0:0] [Released]
Traceback (most recent call last):
  File "/gpfs/bbp.cscs.ch/home/kumbhar-adm/SPACK_HOME/spack/bin/spack", line 81, in <module>
    sys.exit(spack.main.main())
  File "/gpfs/bbp.cscs.ch/home/kumbhar-adm/SPACK_HOME/spack/lib/spack/spack/main.py", line 548, in main
    return _main(command, parser, args, unknown)
  File "/gpfs/bbp.cscs.ch/home/kumbhar-adm/SPACK_HOME/spack/lib/spack/spack/main.py", line 440, in _main
    return _invoke_spack_command(command, parser, args, unknown_args)
  File "/gpfs/bbp.cscs.ch/home/kumbhar-adm/SPACK_HOME/spack/lib/spack/spack/main.py", line 356, in _invoke_spack_command
    return_val = command(parser, args)
  File "/gpfs/bbp.cscs.ch/home/kumbhar-adm/SPACK_HOME/spack/lib/spack/spack/cmd/setup.py", line 176, in setup
    write_spconfig(package)
  File "/gpfs/bbp.cscs.ch/home/kumbhar-adm/SPACK_HOME/spack/lib/spack/spack/cmd/setup.py", line 70, in write_spconfig
    spack.build_environment.setup_package(package)
TypeError: setup_package() takes exactly 2 arguments (1 given)

I quickly tried passing False to setup_package and get:

  File "/gpfs/bbp.cscs.ch/home/kumbhar-adm/SPACK_HOME/spack/bin/spack", line 81, in <module>
    sys.exit(spack.main.main())
  File "/gpfs/bbp.cscs.ch/home/kumbhar-adm/SPACK_HOME/spack/lib/spack/spack/main.py", line 548, in main
    return _main(command, parser, args, unknown)
  File "/gpfs/bbp.cscs.ch/home/kumbhar-adm/SPACK_HOME/spack/lib/spack/spack/main.py", line 440, in _main
    return _invoke_spack_command(command, parser, args, unknown_args)
  File "/gpfs/bbp.cscs.ch/home/kumbhar-adm/SPACK_HOME/spack/lib/spack/spack/main.py", line 356, in _invoke_spack_command
    return_val = command(parser, args)
  File "/gpfs/bbp.cscs.ch/home/kumbhar-adm/SPACK_HOME/spack/lib/spack/spack/cmd/setup.py", line 184, in setup
    install.install(parser, inst_args)
  File "/gpfs/bbp.cscs.ch/home/kumbhar-adm/SPACK_HOME/spack/lib/spack/spack/cmd/install.py", line 373, in install
    package.do_install(**kwargs)
  File "/gpfs/bbp.cscs.ch/home/kumbhar-adm/SPACK_HOME/spack/lib/spack/spack/package.py", line 1247, in do_install
    partial = self.check_for_unfinished_installation(keep_prefix, restage)
  File "/gpfs/bbp.cscs.ch/home/kumbhar-adm/SPACK_HOME/spack/lib/spack/spack/package.py", line 1442, in check_for_unfinished_installation
    stage_is_managed_in_spack = self.stage.path.startswith(
  File "/gpfs/bbp.cscs.ch/home/kumbhar-adm/SPACK_HOME/spack/lib/spack/spack/package.py", line 779, in stage
    self._stage.create()
AttributeError: 'DIYStage' object has no attribute 'create'

With these two bugs setup is not usable at all with latest develop :(

citibeth · 2018-04-19T17:47:51Z

Superceded by #7830

Elizabeth Fischer and others added 18 commits January 19, 2017 12:01

Refactored spack setup and spack install to eliminate bugs that w…

d493954

…ere happening with `spack setup`. Avoid concretizing more than once (slow and was probably also causing bugs).

Turn off verbose Makefile for manual builds

ccc5951

Address error in test.

9e6f847

Add the spack setup command used to generate spconfig.py`

686e652

`

Restored original logic for spack install. Added check for install-…

2ec3e01

…nothing.

Allow user to choose default version, if version is not specified. De…

fe39c96

…fault versions of development packages can then all be put into `packages.yaml`, freeing the user the need to type (and remember) them when running `spack setup`.

spack install --setup works for a multi-setup spec.

e12bf26

Re-do spack setup as just calling spack install

a44f3a5

Add -d and -j options to diy

c47dd00

Also add -j to the common arguments

Merge branch 'develop' into efischer/161221-FixSetup

86286ce

# Conflicts: # lib/spack/spack/build_systems/cmake.py # lib/spack/spack/cmd/common/arguments.py # lib/spack/spack/cmd/install.py # lib/spack/spack/cmd/setup.py

Separate out construction of spconfig.py from the filename it goes to.

c5ce7f8

Add everytrace to Mock repo; will be used for spack setup tests. Added unit test: setup

Unit test works: setup

7218461

Fix existing unit test problem

5620838

flake85

be1da0e

flake8

f43a7e3

flake8

71e1815

Add verbose flag back to install command.

6b588cc

Merge remote-tracking branch 'upstream/develop' into efischer/161221-…

42ec051

…FixSetup # Conflicts: # lib/spack/spack/build_systems/cmake.py # lib/spack/spack/cmd/install.py # lib/spack/spack/cmd/setup.py # lib/spack/spack/package.py # lib/spack/spack/test/cmd/install.py

citibeth mentioned this pull request Aug 9, 2017

spack setup: Fix Bugs + Multi-setup #2664

Closed

2 tasks

scheibelp self-assigned this Aug 9, 2017

This was referenced Aug 31, 2017

spack setup installs package that is being setup #4920

Closed

spack setup with command line compiler flags rebuilds packages #5254

Closed

pramodskumbhar mentioned this pull request Sep 30, 2017

Make setup use all environment changes from build_environment #5546

Closed

krafczyk mentioned this pull request Oct 18, 2017

Add two options diy #2771

Closed

krafczyk mentioned this pull request Oct 25, 2017

Add -d option to diy to specify source path move -j to common args #5963

Merged

tgamblin added this to the v0.12.0 milestone Nov 12, 2017

scheibelp mentioned this pull request Jan 18, 2018

lib/libxml2.so.2: undefined reference to `gzopen64@ZLIB_1.2.3.3' #5758

Closed

davydden mentioned this pull request Jan 26, 2018

blas reports cmake as a dependency but fails to build with cmake dependency spec #7084

Closed

tgamblin modified the milestones: v0.12.0, v0.13.0 Feb 21, 2018

citibeth mentioned this pull request Apr 19, 2018

Spack Setup: Developer Build Tool #7830

Closed

citibeth closed this Apr 19, 2018

citibeth deleted the efischer/170808-FixSetup branch February 12, 2019 22:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`spack setup`: Fix Bugs + Multi-setup (2)#5043

`spack setup`: Fix Bugs + Multi-setup (2)#5043
citibeth wants to merge 18 commits intodevelopfrom
efischer/170808-FixSetup

citibeth commented Aug 9, 2017

Uh oh!

pramodskumbhar commented Aug 13, 2017

Uh oh!

pramodskumbhar commented Aug 13, 2017

Uh oh!

bvanessen commented Aug 31, 2017 •

edited by pramodskumbhar

Loading

Uh oh!

pramodskumbhar commented Sep 16, 2017

Uh oh!

citibeth commented Apr 19, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

citibeth commented Aug 9, 2017

Uh oh!

pramodskumbhar commented Aug 13, 2017

Uh oh!

pramodskumbhar commented Aug 13, 2017

Uh oh!

bvanessen commented Aug 31, 2017 • edited by pramodskumbhar Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pramodskumbhar commented Sep 16, 2017

Uh oh!

citibeth commented Apr 19, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

bvanessen commented Aug 31, 2017 •

edited by pramodskumbhar

Loading