Add libflame by s-sajid-ali · Pull Request #12193 · spack/spack

s-sajid-ali · 2019-07-30T05:49:29Z

Add the libflame package that provides LAPACK functionality. To be included as a new interface for Numpy-1.17.0

Note that libflame does not have a blas dependency and one would have to set independently.

adamjstewart · 2019-07-30T17:26:06Z

var/spack/repos/builtin/packages/libflame/package.py

+    url      = "https://github.com/flame/libflame/archive/5.1.0.tar.gz"
+    git      = "https://github.com/flame/libflame.git"
+
+    version('develop', branch='master')


You can just call this version master if you want.

Isn't the convention in spack develop>[numeric versions]>all other string versions ? Since the last release was so long ago, I'd prefer to keep it as develop unless I'm missing sth.

That was the convention, until #1983 was merged. Now develop > master > numeric versions > all other string versions.

Great! , it used to cause a lot confusion outside spack as to why is spack calling a version develop when the package developers call it master.

adamjstewart · 2019-07-30T17:26:24Z

var/spack/repos/builtin/packages/libflame/package.py

+
+    provides('lapack', when='+lapack2flame')
+
+    variant('lapack2flame', default=True)


Needs a description

var/spack/repos/builtin/packages/libflame/package.py

adamjstewart · 2019-07-30T21:15:03Z

var/spack/repos/builtin/packages/libflame/package.py

+    variant('lapack2flame', default=True,
+            description='Map legacy LAPACK routine invocations
+            to their corresponding native C implementations
+            in libflame.')


This will result in extra spaces in spack info. You'll have to either put the entire variant on one line, or wrap it with more quotes.

var/spack/repos/builtin/packages/libflame/package.py

new file: var/spack/repos/builtin/packages/libflame/Makefile_5.1.0.patch new file: var/spack/repos/builtin/packages/libflame/package.py

…flame

adamjstewart · 2019-08-01T22:57:47Z

I tried building this and got the following error during build:

Archiving lib/x86_64-apple-darwin18.6.0/libflame.a
make: ar: Argument list too long
make: *** [lib/x86_64-apple-darwin18.6.0/libflame.a] Error 1
make: *** Waiting for unfinished jobs....
Dynamically linking lib/x86_64-apple-darwin18.6.0/libflame.so
make: /Users/Adam/spack/lib/spack/env/clang/clang: Argument list too long
make: *** [lib/x86_64-apple-darwin18.6.0/libflame.so] Error 1

Any idea how to fix this?

s-sajid-ali · 2019-08-02T00:55:33Z

Have a look at the advice given here : flame/libflame#21

Did you get this error for master or release? I implemented the above for master since release didn’t have any issues.

adamjstewart · 2019-08-02T01:57:56Z

I got the error for the latest release. I'll try adding that flag for all versions.

adamjstewart · 2019-08-02T03:28:54Z

I tried the latest release again with that flag but now I'm seeing a completely different problem:

make: /bin/sh: Bad executable (or shared library)
make: *** [obj/x86_64-apple-darwin18.6.0/lapack/inv/tri/lu/flamec/FLA_Trinv_lu_opt_var2.o] Error 1
make: *** Waiting for unfinished jobs....

Going to try master. If it works, I'll call that good enough.

adamjstewart · 2019-08-02T03:41:17Z

Nope, master failed too, this time with different warnings:

Archiving lib/x86_64-apple-darwin18.6.0/libflame.a
Dynamically linking lib/x86_64-apple-darwin18.6.0/libflame.so
ld: unknown option: -soname
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [lib/x86_64-apple-darwin18.6.0/libflame.so] Error 1
make: *** Waiting for unfinished jobs....
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: lib/x86_64-apple-darwin18.6.0/libflame.a(FLA_bdsqr.o) has no symbols
...

I give up.

s-sajid-ali · 2019-08-02T15:48:59Z

Do you want me to test this with clang on CentOS ? I'm not sure how MacOS/Clang deal with the ARG_MAX limit or if the fix for is different from --enable-max-arg-list-hack .

While I don't have access to a mac build-env, I see no issues on CentOS 7.5 :

[sajid@xrmlite spack]$ spack clean -a                                                                                                                                      
==> Removing all temporary build stages                                                                                                                                    
==> Removing cached downloads                                                                                                                                              
==> Removing cached information on repositories                                                                                                                            
==> Removing python cache files                                                                                                                                            
[sajid@xrmlite spack]$ git checkout add_libflame                                                                                                                           
Already on 'add_libflame'                                                                                                                                                  
Your branch is up to date with 'origin/add_libflame'.                                                                                                                      
[sajid@xrmlite spack]$ spack install libflame%[email protected]                                                                                                                    
==> Installing libflame                                                                                                                                                    
==> Searching for binary cache of libflame                                                                                                                                 
==> Warning: No Spack mirrors are currently configured                                                                                                                     
==> No binary for libflame found: installing from source                                                                                                                   
==> Fetching https://github.com/flame/libflame/archive/5.1.0.tar.gz                                                                                                        
######################################################################## 100.0%                                                                                            
==> Staging archive: /home/sajid/packages/spack/var/spack/stage/libflame-5.1.0-dp3u2o5qxdqpmb33odoi6jcwapoasopk/5.1.0.tar.gz                                               
==> Created stage in /home/sajid/packages/spack/var/spack/stage/libflame-5.1.0-dp3u2o5qxdqpmb33odoi6jcwapoasopk                                                            
==> Applied patch /home/sajid/packages/spack/var/spack/repos/builtin/packages/libflame/Makefile_5.1.0.patch                                                                
==> Building libflame [AutotoolsPackage]                                                                                                                                   
==> Executing phase: 'autoreconf'                                                                                                                                          
==> Executing phase: 'configure'                                                                                                                                           
==> Executing phase: 'build'                                                                                                                                               
==> Executing phase: 'install'                                                                                                                                             
==> Successfully installed libflame                                                                                                                                        
  Fetch: 3.37s.  Build: 2m 4.15s.  Total: 2m 7.52s.                                                                                                                        
[+] /home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/libflame-5.1.0-dp3u2o5qxdqpmb33odoi6jcwapoasopk                                                    
[sajid@xrmlite spack]$ spack install libflame@master%[email protected]                                                                                                             
==> Installing libflame                                                                                                                                                    
==> Searching for binary cache of libflame                                                                                                                                 
==> Warning: No Spack mirrors are currently configured                                                                                                                     
==> No binary for libflame found: installing from source                                                                                                                   
==> Cloning git repository: https://github.com/flame/libflame.git on branch master                                                                                         
==> No checksum needed when fetching with git                                                                                                                              
==> Already staged libflame-master-itdlxoywywuqko47ktvrhvmqmw3eoffc in /home/sajid/packages/spack/var/spack/stage/libflame-master-itdlxoywywuqko47ktvrhvmqmw3eoffc         
==> No patches needed for libflame                                                                                                                                         
==> Building libflame [AutotoolsPackage]                                                                                                                                   
==> Executing phase: 'autoreconf'                                                                                                                                          
==> Executing phase: 'configure'                                                                                                                                           
==> Executing phase: 'build'                                                                                                                                               
==> Executing phase: 'install'                                                                                                                                             
==> Successfully installed libflame                                                                                                                                        
  Fetch: 5.50s.  Build: 1m 25.17s.  Total: 1m 30.66s.                                                                                                                      
[+] /home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-8.3.0/libflame-master-itdlxoywywuqko47ktvrhvmqmw3eoffc

adamjstewart · 2019-08-06T03:08:55Z

With my latest commit in #12170, I've gotten a little closer to building libflame. Now I'm getting the error:

Archiving lib/x86_64-unknown-linux-gnu/libflame.a
Dynamically linking lib/x86_64-unknown-linux-gnu/libflame.so
ar: @ar: No such file or directory
make: *** [lib/x86_64-unknown-linux-gnu/libflame.a] Error 1
make: *** Waiting for unfinished jobs....

EDIT: This is on CentOS 7 with GCC 4.8.5. On macOS 10.14.5, with Clang 10.0.1, I'm getting:

Archiving lib/x86_64-apple-darwin18.6.0/libflame.a
make: ar: Argument list too long
make: *** [lib/x86_64-apple-darwin18.6.0/libflame.a] Error 1
make: *** Waiting for unfinished jobs....
Dynamically linking lib/x86_64-apple-darwin18.6.0/libflame.so
make: /Users/Adam/spack/lib/spack/env/clang/clang: Argument list too long
make: *** [lib/x86_64-apple-darwin18.6.0/libflame.so] Error 1

This is even with the max arg hack flag.

s-sajid-ali · 2019-08-06T14:50:30Z

Was the error for release or master on mac ? Looks like a fix was commited in master as per flame/libflame#3.

With your latest commit, I see :

[sajid@xrmlite spack]$ git fetch adamjstewart
[sajid@xrmlite spack]$ git checkout adamjstewart/packages/py-numpy
Note: checking out 'adamjstewart/packages/py-numpy'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 6faa6a2dc Try to fix libflame package
[sajid@xrmlite spack]$ spack install libflame@master %[email protected]
==> Installing libflame
==> Searching for binary cache of libflame
==> Warning: No Spack mirrors are currently configured
==> No binary for libflame found: installing from source
==> Cloning git repository: https://github.com/flame/libflame.git on branch master
==> No checksum needed when fetching with git
==> Already staged libflame-master-xmo2dhpugpfgb4zetev2lcccvdgybjog in /home/sajid/packages/spack/var/spack/stage/libflame-master-xmo2dhpugpfgb4zetev2lcccvdgybjog
==> No patches needed for libflame
==> Building libflame [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Successfully installed libflame
  Fetch: 4.57s.  Build: 1m 37.47s.  Total: 1m 42.04s.
[+] /home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/libflame-master-xmo2dhpugpfgb4zetev2lcccvdgybjog
[sajid@xrmlite spack]$

Edit : I see an error for the release branch as well with #12170. I've opened an issue upstream : flame/libflame#22

adamjstewart · 2019-08-06T17:47:55Z

I tried the patches listed in flame/libflame#3 but still got the same max arg list error. The master branch got a lot closer on macOS but I still got:

Archiving lib/x86_64-apple-darwin18.7.0/libflame.a
Dynamically linking lib/x86_64-apple-darwin18.7.0/libflame.so
ld: unknown option: -soname
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [lib/x86_64-apple-darwin18.7.0/libflame.so] Error 1
make: *** Waiting for unfinished jobs....

I was able to build libflame on CentOS 7 but ran into the same numpy issue you did.

* Add libflame new file: var/spack/repos/builtin/packages/libflame/Makefile_5.1.0.patch new file: var/spack/repos/builtin/packages/libflame/package.py * modified: var/spack/repos/builtin/packages/libflame/package.py

s-sajid-ali force-pushed the add_libflame branch from 8328be6 to 23c7e0a Compare July 30, 2019 16:03

adamjstewart reviewed Jul 30, 2019

View reviewed changes

adamjstewart added the blas-lapack-scalapack label Jul 30, 2019

adamjstewart reviewed Jul 30, 2019

View reviewed changes

adamjstewart mentioned this pull request Jul 30, 2019

Overhaul numpy package #12170

Merged

26 tasks

s-sajid-ali mentioned this pull request Jul 31, 2019

Build error for master branch flame/libflame#21

Closed

adamjstewart reviewed Jul 31, 2019

View reviewed changes

var/spack/repos/builtin/packages/libflame/package.py Outdated Show resolved Hide resolved

Add libflame

d1c8d8b

new file: var/spack/repos/builtin/packages/libflame/Makefile_5.1.0.patch new file: var/spack/repos/builtin/packages/libflame/package.py

s-sajid-ali force-pushed the add_libflame branch from eebc8c4 to d1c8d8b Compare August 1, 2019 06:16

s-sajid-ali added 2 commits August 1, 2019 01:21

Merge branch 'develop' of https://github.com/spack/spack into add_lib…

dcf6342

…flame

modified: var/spack/repos/builtin/packages/libflame/package.py

156d727

s-sajid-ali changed the title ~~[WIP] Add libflame~~ Add libflame Aug 1, 2019

adamjstewart approved these changes Aug 1, 2019

View reviewed changes

adamjstewart merged commit f30b717 into spack:develop Aug 1, 2019

s-sajid-ali deleted the add_libflame branch August 5, 2019 22:36


		provides('lapack', when='+lapack2flame')

		variant('lapack2flame', default=True)

Conversation

s-sajid-ali commented Jul 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamjstewart Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

s-sajid-ali Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

adamjstewart Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

s-sajid-ali Jul 31, 2019

Choose a reason for hiding this comment

Uh oh!

adamjstewart Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adamjstewart Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adamjstewart commented Aug 1, 2019

Uh oh!

s-sajid-ali commented Aug 2, 2019

Uh oh!

adamjstewart commented Aug 2, 2019

Uh oh!

adamjstewart commented Aug 2, 2019

Uh oh!

adamjstewart commented Aug 2, 2019

Uh oh!

s-sajid-ali commented Aug 2, 2019

Uh oh!

adamjstewart commented Aug 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

s-sajid-ali commented Aug 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamjstewart commented Aug 6, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

s-sajid-ali commented Jul 30, 2019 •

edited

Loading

adamjstewart commented Aug 6, 2019 •

edited

Loading

s-sajid-ali commented Aug 6, 2019 •

edited

Loading