Skip to content

add new charmpp backend#15186

Closed
s-sajid-ali wants to merge 11 commits intospack:developfrom
s-sajid-ali:charmpp_ucx
Closed

add new charmpp backend#15186
s-sajid-ali wants to merge 11 commits intospack:developfrom
s-sajid-ali:charmpp_ucx

Conversation

@s-sajid-ali
Copy link
Copy Markdown
Contributor

@s-sajid-ali s-sajid-ali commented Feb 24, 2020 •
edited
Loading

As per charmplusplus/charm#2477, an extra configuration step is needed when backend=ucx, hence I added a variant ucx_pmi. The default setting of ucx_pmi does not break the current logic but I'm not sure if there is a better way to do this (which would require a variant that is defined only if another variant is set to a particular value! ).

The doc error no longer exists for the latest charmpp release, hence it should only be fixed for older versions.

Fetching the latest version fails on develop, thus I switched to the github url but the checksums don't match. Perhaps someone who knows more about this package can point out what's going on.

[sajid@xrmlite ~]$ cd ~/packages/spack/ && git checkout develop && git log -n 1  && spack fetch [email protected]
Already on 'develop'
Your branch is up to date with 'origin/develop'.
commit 9b3f5f3890025494ffa620d144d22a4734c8fcee (HEAD -> develop, upstream/develop, origin/develop, origin/HEAD)
Author: ktsai7 <[email protected]>
Date:   Mon Feb 24 12:22:25 2020 -0700

    update flecsi and legion package.py (#15159)

    * update flecsi and legion package.py

    * comment out a conflict

    * update to use extend
==> Fetching from /home/sajid/packages/spack/var/spack/cache/_source-cache/archive/7c/7c526a78aa0c202b7f0418b345138e7dc40496f0bb7b9e301e0381980450b25c.tar.gz failed.
==> Fetching file:///tmp/pytest-of-sajid/pytest-7/test_ci_rebuild_basic0/working_dir/local_mirror/charmpp/charmpp-6.10.0.tar.gz
curl: (37) Couldn't open file /tmp/pytest-of-sajid/pytest-7/test_ci_rebuild_basic0/working_dir/local_mirror/charmpp/charmpp-6.10.0.tar.gz
==> Failed to fetch file from URL: file:///tmp/pytest-of-sajid/pytest-7/test_ci_rebuild_basic0/working_dir/local_mirror/charmpp/charmpp-6.10.0.tar.gz
    Curl failed with error 37
==> Fetching from file:///tmp/pytest-of-sajid/pytest-7/test_ci_rebuild_basic0/working_dir/local_mirror/charmpp/charmpp-6.10.0.tar.gz failed.
==> Fetching file:///tmp/pytest-of-sajid/pytest-7/test_ci_rebuild_basic0/working_dir/local_mirror/_source-cache/archive/7c/7c526a78aa0c202b7f0418b345138e7dc40496f0bb7b9e301e0381980450b25c.tar.gz
curl: (37) Couldn't open file /tmp/pytest-of-sajid/pytest-7/test_ci_rebuild_basic0/working_dir/local_mirror/_source-cache/archive/7c/7c526a78aa0c202b7f0418b345138e7dc40496f0bb7b9e301e0381980450b25c.tar.gz
==> Failed to fetch file from URL: file:///tmp/pytest-of-sajid/pytest-7/test_ci_rebuild_basic0/working_dir/local_mirror/_source-cache/archive/7c/7c526a78aa0c202b7f0418b345138e7dc40496f0bb7b9e301e0381980450b25c.tar.gz
    Curl failed with error 37
==> Fetching from file:///tmp/pytest-of-sajid/pytest-7/test_ci_rebuild_basic0/working_dir/local_mirror/_source-cache/archive/7c/7c526a78aa0c202b7f0418b345138e7dc40496f0bb7b9e301e0381980450b25c.tar.gz failed.
==> Fetching https://charm.cs.illinois.edu/distrib/charm-6.10.0.tar.gz

curl: (35) error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol
==> Failed to fetch file from URL: https://charm.cs.illinois.edu/distrib/charm-6.10.0.tar.gz
    Curl failed with error 35
==> Fetching from https://charm.cs.illinois.edu/distrib/charm-6.10.0.tar.gz failed.
==> Error: All fetchers failed for spack-stage-charmpp-6.10.0-tirx7n7uwlvu7qok5etanggopadumynv
[sajid@xrmlite spack]$

Requesting @matthiasdiener for review.

@matthiasdiener
Copy link
Copy Markdown
Contributor

matthiasdiener commented Feb 24, 2020

Thank you for the patch! The ucx/doc changes look fine to me.

Regarding the download error that you see, it seems that there is something wrong in the TLS negotiation. Can you post the output of the following command here?:
$ curl -v -s https://charm.cs.illinois.edu/distrib/charm-6.10.0.tar.gz

@s-sajid-ali
Copy link
Copy Markdown
Contributor Author

s-sajid-ali commented Feb 24, 2020

As requested :

[sajid@xrmlite ~]$ curl -v -s https://charm.cs.illinois.edu/distrib/charm-6.10.0.tar.gz
*   Trying 128.174.239.11...
* TCP_NODELAY set
* Connected to charm.cs.illinois.edu (128.174.239.11) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS alert, protocol version (582):
* error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol
* Closing connection 0
[sajid@xrmlite ~]$

@matthiasdiener
Copy link
Copy Markdown
Contributor

matthiasdiener commented Feb 24, 2020 •
edited
Loading

Thanks, it seems that your curl requires TLS v1.3, while our server only provides TLSv1.2. Could you check whether specifying a TLS version (like here: https://superuser.com/questions/606598/specifying-minor-tls-version-when-using-curl) works around this?

EDIT: Actually, the server only seems to supports TLSv1.0.

@s-sajid-ali
Copy link
Copy Markdown
Contributor Author

s-sajid-ali commented Feb 24, 2020

Doesn't seem to help :

[sajid@xrmlite ~]$ curl -v -s https://charm.cs.illinois.edu/distrib/charm-6.10.0.tar.gz --tls-max 1.2
*   Trying 128.174.239.11...
* TCP_NODELAY set
* Connected to charm.cs.illinois.edu (128.174.239.11) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (OUT), TLS alert, protocol version (582):
* error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol
* Closing connection 0
[sajid@xrmlite ~]$

@matthiasdiener
Copy link
Copy Markdown
Contributor

matthiasdiener commented Feb 24, 2020

Yeah, I think it only supports TLSv1.0.

@s-sajid-ali
Copy link
Copy Markdown
Contributor Author

s-sajid-ali commented Feb 24, 2020 •
edited
Loading

That didn't help either :

[sajid@xrmlite ~]$ curl -v -s https://charm.cs.illinois.edu/distrib/charm-6.10.0.tar.gz --tls-max 1.0
*   Trying 128.174.239.11...
* TCP_NODELAY set
* Connected to charm.cs.illinois.edu (128.174.239.11) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS alert, internal error (592):
* error:141E70BF:SSL routines:tls_construct_client_hello:no protocols available
[sajid@xrmlite ~]$

I'm on centOS-8.1 :

[sajid@xrmlite ~]$ curl --version
curl 7.61.1 (x86_64-redhat-linux-gnu) libcurl/7.61.1 OpenSSL/1.1.1c zlib/1.2.11 brotli/1.0.6 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.0.5) libssh/0.9.0/openssl/zlib nghttp2/1.33.0
Release-Date: 2018-09-05
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz brotli TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL Metalink
[sajid@xrmlite ~]$

@s-sajid-ali
Copy link
Copy Markdown
Contributor Author

s-sajid-ali commented Feb 24, 2020 •
edited
Loading

Is there a reason to not switch the url to github repo ?

@matthiasdiener
Copy link
Copy Markdown
Contributor

matthiasdiener commented Feb 24, 2020

Is there a reason to not switch to url to github repo ?

Currently, the archive on the GitHub repo is autogenerated and not the same archive that we provide on our server. The difference between the archives is that GitHub one requires autoconf/automake, while ours doesn't. I'll discuss with the group regarding what we can do to resolve this.

@s-sajid-ali
Copy link
Copy Markdown
Contributor Author

s-sajid-ali commented Feb 26, 2020 •
edited
Loading

The current spack recipe places the build in the spack defined build directory, but the default charmpp behavior is to place it in the transport_protocol-OS-arch directory. Should this be changed as well ?

I'm asking this because I tried compiling NAMD against a charmpp built using this branch and it failed. The official charmpp docs list spack as a tool to install it but the NAMD docs do not. It would be great if both can be built using spack with for wide variety of configurations.

@s-sajid-ali s-sajid-ali mentioned this pull request Mar 25, 2020
Merged
@matthiasdiener matthiasdiener mentioned this pull request Mar 25, 2020
Merged
@matthiasdiener
Copy link
Copy Markdown
Contributor

matthiasdiener commented Mar 25, 2020

Is there a reason to not switch to url to github repo ?

Currently, the archive on the GitHub repo is autogenerated and not the same archive that we provide on our server. The difference between the archives is that GitHub one requires autoconf/automake, while ours doesn't. I'll discuss with the group regarding what we can do to resolve this.

I switched the package to download via http instead of https until we update our server (#15667).

@s-sajid-ali
Copy link
Copy Markdown
Contributor Author

s-sajid-ali commented Mar 25, 2020

Merge #15666 instead.

@s-sajid-ali s-sajid-ali deleted the charmpp_ucx branch May 12, 2020 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@junghans junghans junghans approved these changes

Assignees

No one assigned

Labels

update-package

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@s-sajid-ali @matthiasdiener @junghans @adamjstewart