@balay Does the Trilinos update in this PR pose an issue for xSDK builds? It also seems interesting that we never saw that trilinos+tpetra does not allow for two global ordinal types simultaneously. I am not sure why.

It looks like one of the Travis CI checks failed. I just looked at it and it doesn't look like the issue has anything to do with my changes: https://travis-ci.org/spack/spack/jobs/626912608?utm_medium=notification&utm_source=github_status . How do we proceed in this case? I'm guessing it won't be possible to merge the branch w/o the checks passing?

@aprokop I get a couple of build failures in dependent packages - dealii, phist

dealii:

/tmp/balay/spack-stage/spack-stage-dealii-9.1.1-d4cza7opynz7ifjqarmgdtkfpvek2sm7/spack-src/source/lac/trilinos_precondition_muelu.cc:24:14: fatal error: MueLu_CreateEpetraPreconditioner.hpp: No such file or directory
 #    include <MueLu_CreateEpetraPreconditioner.hpp>
              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

phist:

libphist_kernels_tpetra.so: undefined reference to `Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::scale(Teuchos::ArrayView<double const> const&)'

etc..
dealii-spack-build-out.txt.gz
phist-spack-build-out.txt.gz
trilinos-spack-build-out.txt.gz

apt-get install failed

@ikalash for such CI errors - I would make some change in the PR branch - this will trigger a new CI test. [for ex: rebase the PR branch over latest develop branch]

@balay Thanks for checking! The dealii error has nothing to do with this change, it was due to MueLu removing/deprecating that function and header file. The PHIST one, though, is quite relevant, and needs to be checked.

@aprokop dealii builds fine for me without this change [i.e with spack/develop]- so somehow this change is triggering a build error with dealii.

This behaviour occurred with multiple linux builds I had tried [all on a centos7 linux box]

• linux-centos7-haswell / [email protected]
• linux-centos7-skylake_avx512 / [email protected]
• linux-centos7-skylake_avx512 / [email protected]

Thanks for looking at it. If I re-trigger the build, I suspect the dealii issue will persist (right?). How should we proceed in this case?

It looks like this patch disable Epetra support in MueLu (see here) and that's why deal.II crashes.

After looking at it with @Rombur, it seems that we have the following conflict:

• dealii requires MueLu with Epetra, which requires GO = long
• Albany requires GO = long long (or at least just one type)

I think an appropriate workaround for the issue would be to create a multi-valued global ordinal variant for the Trilinos spack package. Default it to long long, but, but allow switching to other types. Then, xsdk can build trilinos go=long and everybody happy.

@aprokop : we might be able to use GO=long. I will try it now and get back to you.

So, it turns out I cannot use GO=long in the Albany Trilinos build. This is because Albany depends on Teko, which requires GO to be long long or int. Is changing it to int going to be a problem for the other packages that rely on Trilinos? One potential issue I suppose is if one is running very big problems where GO=int is not big enough. I'm not sure if there are any customers that use spack+trilinos who would be in this boat.

int does not sound like a good choice for anyone. Exawind project uses Trilinos built with spack to run very large problems, and will likely hit this issue at some point (if not already).

I think the solution is to allow global ordinal to be customizable in the spack package. This should satisfy everybody.

@aprokop : I agree completely. I'm pretty new to spack. What would be the best way to accomplish what you are proposing? Add a +go_long_long (and similarly for other GO types) option to trilinos/packages.py? Since you're more familiar with the trilinos packages.py file and how it should look like, perhaps you could take a stab at adding the option? Then I can modify the albany/packages.py to use the right GO for Albany (long long).

Since you're more familiar with the trilinos packages.py file and how it should look like, perhaps you could take a stab at adding the option?

I'll do it.

Thanks @aprokop ! Let me know when it's ready to I can modify the albany package.py file and test.

I added a multivalued variant gotype to Trilinos. I'm not sure what the right default for it is, but seeming as dealii and PHIST break when it's not long, I set it to that. I would actually prefer that it would be long_long; this would require updating other packages that may be out of scope for this PR.

@balay Can you please test this update again? As the default has not changed, it should pass the tests.

@ikalash Please test the update (I already patched Albany to use long long).

Before this PR is merged, though, we'll need to go through other packages and check that it does not break them:

$   git grep -l "depends_on('trilinos" | cut -f 1 -d / | xargs
albany amp camellia dealii elmerfem fastmath fenics nalu-wind nalu omega-h percept petsc phist xsdk xsdktrilinos

I'm testing b503bb0 and still get errors building dealii

In file included from /tmp/balay/spack-stage/spack-stage-dealii-9.1.1-mogghduvoavojf6j5h7kseuk4trmdwnj/spack-src/include/deal.II/lac/dynamic_sparsity_pattern.h:22:0,
                 from /tmp/balay/spack-stage/spack-stage-dealii-9.1.1-mogghduvoavojf6j5h7kseuk4trmdwnj/spack-src/include/deal.II/lac/block_sparsity_pattern.h:28,
                 from /tmp/balay/spack-stage/spack-stage-dealii-9.1.1-mogghduvoavojf6j5h7kseuk4trmdwnj/spack-src/include/deal.II/lac/sparsity_tools.h:24,
                 from /tmp/balay/spack-stage/spack-stage-dealii-9.1.1-mogghduvoavojf6j5h7kseuk4trmdwnj/spack-src/include/deal.II/base/graph_coloring.h:25,
                 from /tmp/balay/spack-stage/spack-stage-dealii-9.1.1-mogghduvoavojf6j5h7kseuk4trmdwnj/spack-src/include/deal.II/base/work_stream.h:22,
                 from /tmp/balay/spack-stage/spack-stage-dealii-9.1.1-mogghduvoavojf6j5h7kseuk4trmdwnj/spack-src/source/numerics/data_out.cc:16:
/tmp/balay/spack-stage/spack-stage-dealii-9.1.1-mogghduvoavojf6j5h7kseuk4trmdwnj/spack-src/include/deal.II/base/index_set.h:604:5: warning: unused parameter 'other' [-Wunused-parameter]
     IntervalIterator(const IntervalIterator &other) = default;
     ^
/tmp/balay/spack-stage/spack-stage-dealii-9.1.1-mogghduvoavojf6j5h7kseuk4trmdwnj/spack-src/include/deal.II/base/index_set.h: In member function 'dealii::IndexSet::IntervalIterator dealii::IndexSet::IntervalIterator::operator++(int)':
/tmp/balay/spack-stage/spack-stage-dealii-9.1.1-mogghduvoavojf6j5h7kseuk4trmdwnj/spack-src/include/deal.II/base/index_set.h:1156:44: note: synthesized method 'dealii::IndexSet::IntervalIterator::IntervalIterator(const dealii::IndexSet::IntervalIterator&)' first required here
   const IndexSet::IntervalIterator iter = *this;
                                            ^
/tmp/balay/spack-stage/spack-stage-dealii-9.1.1-mogghduvoavojf6j5h7kseuk4trmdwnj/spack-src/source/lac/trilinos_precondition_muelu.cc:24:52: fatal error: MueLu_CreateEpetraPreconditioner.hpp: No such file or directory
 #    include <MueLu_CreateEpetraPreconditioner.hpp>
                                                    ^
compilation terminated.

phist build is clean [with this change]

Albany build is good!

I have trouble figuring out why MueLu_CreateEpetraPreconditioner.hpp is not getting installed. I created an issue trilinos/Trilinos#6490 looking for advice.

The answer from MueLu team:

At present Xpetra/Muelu only enable Epetra support if
  (a) Tpetra is disabled, or
  (b) Tpetra is enabled with GO=int.

So, we have a problem now:

• dealii needs int or disabled Tpetra
• phist needs long
• Albany needs Tpetra

Really not sure what to do next...

What is dealii? Albany does not depends on it unless it’s somehow needed for Trilinos without me knowing it.

Also it’s weird to me that Muelu epetra support cannot be disabled on its own. I did not know you could run muelu with epetra.

deal.II is a finite element library which like phist is part of xSDK. We can disable MueLu support in deal.II but isn't one of the goal of xSDK to be able to have all the codes compiled together...

Since Albany does not depend on dealii, can we just modify the logic to use GO=int for dealii? Or, is the problem that dealii and phist need to co-exist and have the same GO, which is not possible right now per @aprokop 's comment above?

@ikalash xSDK is an ECP project. One of the xSDK's project goals is to be able to install multiple math libraries together using Spack. These include Trilinos, dealii, phist, and many others. Prior to the recent changes in Tpetra it was possible using a single version of Trilinos in spack. Now it is not.

Any updates on this issue? Seems it has reached an impasse due to the xSDK issue described above.

No updates from me.

Would be nice to have this merged to get the global ordinal types in. Otherwise Trilinos is broken here for my project.

I guess the only way to make further progress here is to have multiple Trilinos installations within xsdk, making dealii depend on GO=int version of Trilinos. @balay Is there any requirement on the xsdk side to have a single installation for each xsdk package?

I guess the only way to make further progress here is to have multiple Trilinos installations within xsdk, making dealii depend on GO=int version of Trilinos. @balay Is there any requirement on the xsdk side to have a single installation for each xsdk package?

@aprokop wrt xsdk - I think we should prefer not to have multiple installs of a given xsdk pacakge. within 'spack install xsdk' . Does spack allow that?

And I think its best to not merge in changes that would break [email protected] - the current release.

Any fixes should be tested with @devel version of packages - that are likely to be part of future releases

I ended up just creating my own fork of spack to use with Albany - decided it's easier than trying to get changes into develop. Just wanted to let you know. I'm OK closing the PR given this situation.