Skip to content

ARROW-16752: [R] Rework Linux binary installation#13464

Merged
nealrichardson merged 22 commits intoapache:masterfrom
nealrichardson:r-binary-default
Jul 6, 2022
Merged

ARROW-16752: [R] Rework Linux binary installation#13464
nealrichardson merged 22 commits intoapache:masterfrom
nealrichardson:r-binary-default

Conversation

@nealrichardson
Copy link
Member

@nealrichardson nealrichardson commented Jun 29, 2022 •
edited
Loading

See the jira for the main behavior changes here. Other changes of note:

  • There are more brief messages printed to the installation log, even in the default "quiet" mode, that indicate which branch of the logic in nixlibs.R you've gone through. They're factual and generally connected to the tests that are being run, but they are worded somewhat ambiguously or coded, so as not to run afoul of censors should they appear in the wrong context. This should help us in the triaging of installation failures, even in circumstances where we can't enable greater verbosity.
  • There is a start of a test suite for nixlibs.R, run separately from the package tests. It has been wired up to run in ci/scripts/r_test.sh.

@github-actions
Copy link

github-actions bot commented Jun 29, 2022

https://issues.apache.org/jira/browse/ARROW-16752

jonkeane
jonkeane reviewed Jun 29, 2022
View reviewed changes
r/tools/nixlibs.R Outdated
Comment on lines 76 to 89
Copy link
Member

@jonkeane jonkeane Jun 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙏 For this added clarity + examples

@nealrichardson nealrichardson changed the title WIP ARROW-16752: [R] Rework Linux binary installation ARROW-16752: [R] Rework Linux binary installation Jul 1, 2022
@nealrichardson
Copy link
Member Author

nealrichardson commented Jul 1, 2022

@github-actions crossbow submit -g r

@github-actions
Copy link

github-actions bot commented Jul 1, 2022

Revision: a8fc7d6364cf1a5119048fd47549f1c45934c8dd

Submitted crossbow builds: ursacomputing/crossbow @ actions-b16d0a678e

Task Status
conda-linux-gcc-py37-cpu-r40 Azure
conda-linux-gcc-py37-cpu-r41 Azure
conda-osx-clang-py37-r40 Azure
conda-osx-clang-py37-r41 Azure
conda-win-vs2017-py37-r40 Azure
conda-win-vs2017-py37-r41 Azure
homebrew-r-autobrew Github Actions
homebrew-r-brew Github Actions
r-binary-packages Github Actions
test-fedora-r-clang-sanitizer Azure
test-r-arrow-backwards-compatibility Github Actions
test-r-depsource-bundled Azure
test-r-depsource-system Github Actions
test-r-dev-duckdb Github Actions
test-r-devdocs Github Actions
test-r-gcc-11 Github Actions
test-r-gcc-12 Github Actions
test-r-install-local Github Actions
test-r-linux-as-cran Github Actions
test-r-linux-rchk Github Actions
test-r-linux-valgrind Azure
test-r-minimal-build Azure
test-r-offline-maximal Github Actions
test-r-offline-minimal Azure
test-r-rhub-debian-gcc-devel-lto-latest Azure
test-r-rhub-debian-gcc-release-custom-ccache Azure
test-r-rhub-ubuntu-gcc-release-latest Azure
test-r-rocker-r-base-latest Azure
test-r-rstudio-r-base-4.1-opensuse153 Azure
test-r-rstudio-r-base-4.2-centos7-devtoolset-8 Azure
test-r-rstudio-r-base-4.2-focal Azure
test-r-ubuntu-22.04 Github Actions
test-r-versions Github Actions
test-ubuntu-18.04-r-sanitizer Azure

@nealrichardson
Copy link
Member Author

nealrichardson commented Jul 1, 2022

@github-actions crossbow submit test-ubuntu-18.04-r-sanitizer test-r-offline-minimal r-binary-packages

@github-actions
Copy link

github-actions bot commented Jul 1, 2022 

Failed to render template `r/github.packages.yml` with UndefinedError: 'matrix' is undefined
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/2599663338

@nealrichardson
Copy link
Member Author

nealrichardson commented Jul 1, 2022

@github-actions crossbow submit test-ubuntu-18.04-r-sanitizer test-r-offline-minimal r-binary-packages

@github-actions
Copy link

github-actions bot commented Jul 1, 2022 

Failed to render template `r/github.packages.yml` with UndefinedError: 'matrix' is undefined
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/2599679610

@nealrichardson
Copy link
Member Author

nealrichardson commented Jul 1, 2022

@github-actions crossbow submit test-ubuntu-18.04-r-sanitizer test-r-offline-minimal r-binary-packages

@github-actions
Copy link

github-actions bot commented Jul 1, 2022

Revision: fafe20aaf306103756415b1d430755ef19cff55f

Submitted crossbow builds: ursacomputing/crossbow @ actions-0b61e7d1b0

Task Status
r-binary-packages Github Actions
test-r-offline-minimal Azure
test-ubuntu-18.04-r-sanitizer Azure

@nealrichardson
Copy link
Member Author

nealrichardson commented Jul 2, 2022 •
edited
Loading

Indications that this is working as expected:

Remaining to-dos:

  • I'd like to add a few more docker images to the test-linux-binary job in the packages workflow, just to confirm that the binaries are working across a broader spectrum of system configurations.
    * Installation vignette needs revision

@nealrichardson nealrichardson marked this pull request as ready for review July 2, 2022 12:55
@nealrichardson
Copy link
Member Author

nealrichardson commented Jul 3, 2022

@github-actions crossbow submit r-binary-packages

@github-actions
Copy link

github-actions bot commented Jul 3, 2022

Revision: 94bc2396432ce6e64afc229983e14465cbffe6b1

Submitted crossbow builds: ursacomputing/crossbow @ actions-89157bcd55

Task Status
r-binary-packages Github Actions

@nealrichardson
Copy link
Member Author

nealrichardson commented Jul 3, 2022

@github-actions crossbow submit r-binary-packages

@github-actions
Copy link

github-actions bot commented Jul 3, 2022

Revision: 8b5241cc197093883d6bc959a0a3ccd7319b41c1

Submitted crossbow builds: ursacomputing/crossbow @ actions-63b2a971ab

Task Status
r-binary-packages Github Actions

@nealrichardson
Copy link
Member Author

nealrichardson commented Jul 3, 2022

@github-actions crossbow submit r-binary-packages

@github-actions
Copy link

github-actions bot commented Jul 3, 2022

Revision: 99cb479ab17e3c7990bf18ee797a9d31876ca427

Submitted crossbow builds: ursacomputing/crossbow @ actions-edc3669611

Task Status
r-binary-packages Github Actions

@nealrichardson
Copy link
Member Author

nealrichardson commented Jul 3, 2022

@github-actions crossbow submit r-binary-packages

@github-actions
Copy link

github-actions bot commented Jul 3, 2022

Revision: cc4ae69b9683235801a63844b94d2e14fbc2b9f4

Submitted crossbow builds: ursacomputing/crossbow @ actions-11cf563c62

Task Status
r-binary-packages Github Actions

@nealrichardson
Copy link
Member Author

nealrichardson commented Jul 4, 2022

@github-actions crossbow submit r-binary-packages

@github-actions
Copy link

github-actions bot commented Jul 4, 2022

Revision: cadfbc74b13f67421e841fd67a4d2217e457289b

Submitted crossbow builds: ursacomputing/crossbow @ actions-25457272db

Task Status
r-binary-packages Github Actions

@nealrichardson
Copy link
Member Author

nealrichardson commented Jul 5, 2022

@github-actions crossbow submit r-binary-packages test-r-linux-as-cran

@github-actions
Copy link

github-actions bot commented Jul 5, 2022

Revision: 2e1779e3fd670b5d9f657daeb1398244d3ebf1f1

Submitted crossbow builds: ursacomputing/crossbow @ actions-c61d16cfd9

Task Status
r-binary-packages Github Actions
test-r-linux-as-cran Github Actions

@nealrichardson
Copy link
Member Author

nealrichardson commented Jul 5, 2022

@github-actions crossbow submit r-binary-packages test-r-linux-as-cran

@github-actions
Copy link

github-actions bot commented Jul 5, 2022

Revision: 76eda77ce37b846ec076c4ba92860beffad66220

Submitted crossbow builds: ursacomputing/crossbow @ actions-8c8c741115

Task Status
r-binary-packages Github Actions
test-r-linux-as-cran Github Actions

assignUser
assignUser approved these changes Jul 6, 2022
View reviewed changes
Copy link
Member

@assignUser assignUser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, just a few comments.
Overall an important change to the logic in light of openssl 3 getting more use etc.. Extra 👍 for the tests :D

dev/tasks/r/github.packages.yml Outdated
Copy link
Member

@assignUser assignUser Jul 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was working previously, is this a regression or was it a false positive previously?

Copy link
Member Author

@nealrichardson nealrichardson Jul 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something of a false positive. The rhub image doesn't set libc++ everywhere it should, which is why we edit the Makevars in r_docker_configure.sh. The change in this PR is that I run r_docker_configure.sh in these images.

nealrichardson
nealrichardson commented Jul 6, 2022
View reviewed changes
@nealrichardson @assignUser
Update comments and TODOs
f52e81e 
Co-authored-by: Jacob Wujciak-Jens <[email protected]>
@nealrichardson nealrichardson mentioned this pull request Jul 6, 2022
Merged
jonkeane
jonkeane approved these changes Jul 6, 2022
View reviewed changes
Copy link
Member

@jonkeane jonkeane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, a few comments about comments

ci/scripts/r_test.sh
Comment on lines +29 to +30
# Run the nixlibs.R test suite, which is not included in the installed package
${R_BIN} -e 'setwd("tools"); testthat::test_dir(".")'
Copy link
Member

@jonkeane jonkeane Jul 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😍

r/tools/nixlibs.R
# There was no error in compiling: so we found libcurl and openssl > 1.0.2,
# openssl is < 3.0, glibc is >= 2.27, and we're not using a strict libc++
cat("*** Found libcurl and openssl >= 1.0.2\n")
return("ubuntu-18.04")
Copy link
Member

@jonkeane jonkeane Jul 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I reading this correctly that this doesn't actually mean we're on Ubuntu necessarily, just that we should be able to use the binary for ubuntu-18.04 on this system, yeah?

Copy link
Member Author

@nealrichardson nealrichardson Jul 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, this is just still how the binary zip files are named (based on the OS on which they were built).

@nealrichardson @jonkeane
Update r/tools/nixlibs.R
ee4b6d6 
Co-authored-by: Jonathan Keane <[email protected]>
@nealrichardson nealrichardson merged commit c492ef4 into apache:master Jul 6, 2022
@nealrichardson nealrichardson deleted the r-binary-default branch July 6, 2022 14:16
@asfimport asfimport mentioned this pull request Jul 6, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@paleolimbot paleolimbot paleolimbot left review comments

@jonkeane jonkeane jonkeane approved these changes

@assignUser assignUser assignUser approved these changes

+1 more reviewer

@eitsupi eitsupi eitsupi left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Component: R

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@nealrichardson @jonkeane @paleolimbot @assignUser @eitsupi