Skip to content

arrow-cpp: 9.0.1 -> 11.0.0#213324

Merged
veprbl merged 9 commits intoNixOS:masterfrom
tobim:pkgs/apache-arrow-11
Feb 12, 2023
Merged

arrow-cpp: 9.0.1 -> 11.0.0#213324
veprbl merged 9 commits intoNixOS:masterfrom
tobim:pkgs/apache-arrow-11

Conversation

@tobim
Copy link
Contributor

@tobim tobim commented Jan 29, 2023 •
edited by veprbl
Loading

Description of changes

Based on #198805. Fixed the merge conflicts and did another bump to 11.

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.05 Release Notes (or backporting 22.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

Closes: #198805

@tobim tobim requested review from FRidh and jonringer as code owners January 29, 2023 08:33
@github-actions github-actions bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Jan 29, 2023
@tobim tobim requested a review from veprbl January 29, 2023 08:54
@ofborg ofborg bot requested review from bjornfor and cpcloud January 29, 2023 09:25
@ofborg ofborg bot added 10.rebuild-darwin: 101-500 This PR causes between 101 and 500 packages to rebuild on Darwin. 10.rebuild-linux: 101-500 This PR causes between 101 and 500 packages to rebuild on Linux. labels Jan 29, 2023
@cpcloud
Copy link
Contributor

cpcloud commented Jan 29, 2023

LGTM, but the aarch64-linux builds are failing because of arrow's cxx17 requirement.

@tobim
Copy link
Contributor Author

tobim commented Jan 29, 2023

I propose we mark aarch64-linux as broken until that is sorted out. (#209870 and a follow up bump of the default gcc, I think.)

@tobim
Copy link
Contributor Author

tobim commented Jan 30, 2023

I recommend to rebase onto master (with a few conflicts) before running nixpkgs-review here.
I rebuilt up to pyarrow on staging and had to disable checks for libarchive, e2fsprogs, and python3Packages.ephemeral-port-reserve to get there.

@veprbl
Copy link
Member

veprbl commented Jan 30, 2023

This should be rebased on master.

@veprbl
Copy link
Member

veprbl commented Jan 30, 2023

Result of nixpkgs-review pr 213324 run on x86_64-darwin 1

24 packages marked as broken and skipped:
  • intensity-normalization
  • napari
  • python310Packages.boxx
  • python310Packages.bpycv
  • python310Packages.bsuite
  • python310Packages.caffe
  • python310Packages.caffeWithCuda
  • python310Packages.dalle-mini
  • python310Packages.dask-jobqueue
  • python310Packages.dask-yarn
  • python310Packages.dm-sonnet
  • python310Packages.intensity-normalization
  • python310Packages.mask-rcnn
  • python310Packages.napari
  • python310Packages.pywick
  • python310Packages.qiskit
  • python310Packages.qiskit-machine-learning
  • python310Packages.rl-coach
  • python310Packages.rlax
  • python310Packages.shap
  • python310Packages.streamz
  • python310Packages.tensorflow-datasets
  • python310Packages.vqgan-jax
  • python310Packages.zcs
89 packages failed to build:
  • arrow-cpp
  • arrow-glib
  • databricks-sql-cli
  • easyocr (python310Packages.easyocr)
  • gnuradio
  • gnuradio3_8
  • gnuradio3_8Packages.ais
  • gnuradio3_8Packages.gnuradio
  • gnuradio3_8Packages.grnet
  • gnuradio3_8Packages.osmosdr
  • gnuradio3_8Packages.rds
  • gnuradio3_9
  • gnuradio3_9Packages.gnuradio
  • gnuradio3_9Packages.grnet
  • gnuradioPackages.gnuradio
  • openai-whisper (python310Packages.openai-whisper)
  • parquet-tools
  • python310Packages.apache-beam
  • python310Packages.aplpy
  • python310Packages.autofaiss
  • python310Packages.batchgenerators
  • python310Packages.casa-formats-io
  • python310Packages.clifford
  • python310Packages.dask
  • python310Packages.dask-gateway
  • python310Packages.dask-glm
  • python310Packages.dask-image
  • python310Packages.dask-ml
  • python310Packages.dask-mpi
  • python310Packages.databricks-sql-connector
  • python310Packages.datafusion
  • python310Packages.datasets
  • python310Packages.datashader
  • python310Packages.db-dtypes
  • python310Packages.devito
  • python310Packages.distributed
  • python310Packages.dremel3dpy
  • python310Packages.embedding-reader
  • python310Packages.env-canada
  • python310Packages.glymur
  • python310Packages.google-cloud-bigquery
  • python310Packages.handout
  • python310Packages.ibis-framework
  • python310Packages.image-match
  • python310Packages.imagecorruptions
  • python310Packages.imageio
  • python310Packages.intake
  • python310Packages.intake-parquet
  • python310Packages.moviepy
  • python310Packages.napari-console
  • python310Packages.napari-svg
  • python310Packages.pandas-stubs
  • python310Packages.pims
  • python310Packages.psd-tools
  • python310Packages.pyarrow
  • python310Packages.pyfftw
  • python310Packages.pyvista
  • python310Packages.recordlinkage
  • python310Packages.runway-python
  • python310Packages.scikitimage
  • python310Packages.sfepy
  • python310Packages.slicedimage
  • python310Packages.spacy-transformers
  • python310Packages.sparse
  • python310Packages.spectral-cube
  • python310Packages.stanza
  • python310Packages.stumpy
  • python310Packages.stytra
  • python310Packages.tensorly
  • python310Packages.test-tube
  • python310Packages.tifffile
  • python310Packages.tokenizers
  • python310Packages.transformers
  • python311Packages.apache-beam
  • python311Packages.autofaiss
  • python311Packages.databricks-sql-connector
  • python311Packages.datafusion
  • python311Packages.datasets
  • python311Packages.db-dtypes
  • python311Packages.embedding-reader
  • python311Packages.google-cloud-bigquery
  • python311Packages.ibis-framework
  • python311Packages.pandas-stubs
  • python311Packages.pyarrow
  • python311Packages.recordlinkage
  • python311Packages.tokenizers
  • python311Packages.transformers
  • streamlit
  • thrift

@veprbl
Copy link
Member

veprbl commented Feb 1, 2023

^ failed on python3.10-cffi

@veprbl veprbl changed the base branch from staging to staging-next February 1, 2023 17:01
@tobim tobim force-pushed the pkgs/apache-arrow-11 branch from 4a4d948 to 6c43205 Compare February 1, 2023 23:43
@tobim tobim changed the base branch from staging-next to master February 1, 2023 23:49
@tobim
Copy link
Contributor Author

tobim commented Feb 1, 2023

This should be rebased on master.

Rebased and retargeted now.

^ failed on python3.10-cffi

Can you try again?

@ofborg ofborg bot added 2.status: merge conflict This PR has merge conflicts with the target branch and removed 2.status: merge conflict This PR has merge conflicts with the target branch labels Feb 2, 2023
@veprbl
Copy link
Member

veprbl commented Feb 2, 2023

^ failed on python3.10-cffi

Can you try again?

That won't work until #214010 lands on master.

veprbl
veprbl approved these changes Feb 2, 2023
View reviewed changes
Copy link
Member

@veprbl veprbl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Diff LGTM

@wegank
Copy link
Member

wegank commented Feb 3, 2023

@veprbl Would you please test if the standard workaround of using gcc12Stdenv and passing -lgcc to NIX_LDFLAGS on aarch64-linux works?

@wegank wegank mentioned this pull request Feb 3, 2023
Merged
13 tasks
@mweinelt mweinelt added 12.approvals: 1 This PR was reviewed and approved by one person. 12.approved-by: package-maintainer This PR was reviewed and approved by a maintainer listed in any of the changed packages. labels Feb 5, 2023
@veprbl
Copy link
Member

veprbl commented Feb 9, 2023

@veprbl Would you please test if the standard workaround of using gcc12Stdenv and passing -lgcc to NIX_LDFLAGS on aarch64-linux works?

This looks like quite a rabithole. Are you a user of arrow-cpp on aarch64?

@wegank
Copy link
Member

wegank commented Feb 10, 2023

@veprbl Would you please test if the standard workaround of using gcc12Stdenv and passing -lgcc to NIX_LDFLAGS on aarch64-linux works?

This looks like quite a rabithole. Are you a user of arrow-cpp on aarch64?

I'm maintaining python3Packages.pyvista, which seems to be affected somehow.

@veprbl
Copy link
Member

veprbl commented Feb 10, 2023 

% nix why-depends --derivation nixpkgs#python3Packages.pyvista nixpkgs#arrow-cpp
/nix/store/qnqv1zcbs3vhba4i110l7qpsqxpj9f0m-python3.10-pyvista-0.37.0.drv
└───/nix/store/y258hn1sbvyxd68xvchkiimsiy8hydjs-python3.10-imageio-2.25.0.drv
    └───/nix/store/zx2p0s1ga5911z825s7bv3mr4j5c5r60-python3.10-tifffile-2022.10.10.drv
        └───/nix/store/qydnm6d29sg1zjjn9570yig4jyx0nk3n-python3.10-dask-2023.1.0.drv
            └───/nix/store/pzrwychaqf79l5yf8r1ysdlh00c1fwlh-python3.10-pyarrow-9.0.0.drv
                └───/nix/store/pb4lndv3brkvnb96q1bq98g65h1b2jjz-arrow-cpp-9.0.0.drv

@tobim
Copy link
Contributor Author

tobim commented Feb 10, 2023

tifffile needs dask for tests only. We can remove that input and the test scaffold will automatically skip the affected tests. I just verified that this change is enough to fix the build.

Should we do this in general, only for the affected platform, or is it a bad idea?

@wegank
Copy link
Member

wegank commented Feb 10, 2023

tifffile needs dask for tests only. We can remove that input and the test scaffold will automatically skip the affected tests. I just verified that this change is enough to fix the build.

Should we do this in general, only for the affected platform, or is it a bad idea?

There are over 20 packages that depend directly on python3Packages.dask, I guess the change is still too specific and doesn't help to unbreak other packages (that I don't maintain).

@tobim
Copy link
Contributor Author

tobim commented Feb 10, 2023

There are over 20 packages that depend directly on python3Packages.dask, I guess the change is still too specific and doesn't help to unbreak other packages (that I don't maintain).

Let's check how many of those are not optional first.
That said, I wouldn't want to introduce an abstraction just for those 20, let's rather be explicit and remove the dask dependency where it isn't needed. Until aarch64-linux works again, of course.

@veprbl
Copy link
Member

veprbl commented Feb 11, 2023

@GrahamcOfBorg build python3Packages.pyvista

@ofborg ofborg bot requested a review from veprbl February 11, 2023 02:21
@veprbl veprbl force-pushed the pkgs/apache-arrow-11 branch from b918cc2 to 7f727d8 Compare February 11, 2023 03:52
@veprbl
Copy link
Member

veprbl commented Feb 11, 2023

@GrahamcOfBorg build python3Packages.pyvista

@veprbl veprbl mentioned this pull request Feb 11, 2023
Merged
12 tasks
@veprbl veprbl force-pushed the pkgs/apache-arrow-11 branch from 7f727d8 to 3824c54 Compare February 11, 2023 18:33
@veprbl
Copy link
Member

veprbl commented Feb 11, 2023

@GrahamcOfBorg build python3Packages.pyvista

@tobim
Copy link
Contributor Author

tobim commented Feb 12, 2023

I learned that arrow gained support for system jemalloc in apache/arrow#13373. Should we switch to that in this PR still?

@veprbl veprbl merged commit 44a3569 into NixOS:master Feb 12, 2023
@veprbl
Copy link
Member

veprbl commented Feb 12, 2023

@tobim Let's follow up in another PR.

@dotlambda dotlambda mentioned this pull request Feb 13, 2023
Merged
13 tasks
@mcwitt mcwitt mentioned this pull request Feb 15, 2023
Merged
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@FRidh FRidh Awaiting requested review from FRidh

@jonringer jonringer Awaiting requested review from jonringer

@cpcloud cpcloud Awaiting requested review from cpcloud

@bjornfor bjornfor Awaiting requested review from bjornfor

@veprbl veprbl Awaiting requested review from veprbl

Assignees

No one assigned

Labels

6.topic: python Python is a high-level, general-purpose programming language. 10.rebuild-darwin: 101-500 This PR causes between 101 and 500 packages to rebuild on Darwin. 10.rebuild-linux: 101-500 This PR causes between 101 and 500 packages to rebuild on Linux. 12.approvals: 1 This PR was reviewed and approved by one person. 12.approved-by: package-maintainer This PR was reviewed and approved by a maintainer listed in any of the changed packages.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@tobim @cpcloud @veprbl @wegank @mweinelt