Skip to content

[python] Work around pyarrow>=13 issue on MacOS, main branch#2734

Closed
johnkerl wants to merge 1 commit intomainfrom
kerl/workaround
Closed

[python] Work around pyarrow>=13 issue on MacOS, main branch#2734
johnkerl wants to merge 1 commit intomainfrom
kerl/workaround

Conversation

@johnkerl
Copy link
Copy Markdown
Contributor

@johnkerl johnkerl commented Jun 14, 2024 •
edited
Loading

Issue and/or context: #1849 #1926 #2692 -- please see those for exhaustive details.

TL;DR -- MacOS with pyarrow>=13:

import pyarrow
import tiledb
tiledb.open("s3://anything/at/all")

Fatal error condition occurred in /Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/vcpkg/buildtrees/aws-c-common/src/v0.8.9-fed0b55d0f.clean/source/allocator.c:121: allocator != ((void*)0)
Exiting Application
################################################################################
Stack trace:
################################################################################
1   libarrow.1400.dylib                 0x0000000106f90150 aws_fatal_assert + 80
2   libarrow.1400.dylib                 0x0000000106f8f528 aws_mem_acquire + 64
3   libarrow.1400.dylib                 0x0000000106fa1cc4 aws_string_new_from_cursor + 76
4   libarrow.1400.dylib                 0x0000000106f9bb94 aws_json_value_get_from_object + 44
5   libarrow.1400.dylib                 0x0000000106f8830c aws_endpoints_ruleset_new_from_string + 120
6   libarrow.1400.dylib                 0x0000000106f22430 _ZN3Aws3Crt9Endpoints10RuleEngineC2ERK15aws_byte_cursorS5_P13aws_allocator + 48
7   libarrow.1400.dylib                 0x00000001066cb820 _ZN3Aws8Endpoint23DefaultEndpointProviderINS_2S321S3ClientConfigurationENS2_8Endpoint19S3BuiltInParametersENS4_25S3ClientContextParametersEEC2EPKcm + 120
8   libtiledb.dylib                     0x0000000150982484 _ZN3Aws2S38S3ClientC2ERKNS_6Client19ClientConfigurationENS2_15AWSAuthV4Signer20PayloadSigningPolicyEbNS0_34US_EAST_1_REGIONAL_ENDPOINT_OPTIONE + 900
9   libtiledb.dylib                     0x0000000150151178 _ZN6tiledb6common11make_sharedINS_2sm14TileDBS3ClientELi66EJRKNS2_12S3ParametersERN3Aws6Client19ClientConfigurationENS8_15AWSAuthV4Signer20PayloadSigningPolicyERKbEEENSt3__110shared_ptrIT_EERAT0__KcDpOT1_ + 92
10  libtiledb.dylib                     0x000000015013b7b8 _ZNK6tiledb2sm2S311init_clientEv + 3840
11  libtiledb.dylib                     0x0000000150144d8c _ZNK6tiledb2sm2S313ls_with_sizesERKNS0_3URIERKNSt3__112basic_stringIcNS5_11char_traitsIcEENS5_9allocatorIcEEEEi + 76
12  libtiledb.dylib                     0x0000000150144cf4 _ZNK6tiledb2sm2S313ls_with_sizesERKNS0_3URIE + 44
13  libtiledb.dylib                     0x000000015016dbf4 _ZNK6tiledb2sm3VFS13ls_with_sizesERKNS0_3URIE + 264
14  libtiledb.dylib                     0x000000015002ce18 _ZNK6tiledb2sm14ArrayDirectory2lsERKNS0_3URIE + 60
15  libtiledb.dylib                     0x000000015002ea54 _ZN6tiledb2sm14ArrayDirectory20load_array_meta_urisEv + 144
16  libtiledb.dylib                     0x00000001500312dc _ZNSt3__120__packaged_task_funcIZN6tiledb6common10ThreadPool5asyncIZNS1_2sm14ArrayDirectory4loadEvE3$_3JEEEDaOT_DpOT0_EUlvE_NS_9allocatorISE_EEFNS2_6StatusEvEEclEv + 24
17  libtiledb.dylib                     0x000000015086594c _ZNSt3__113packaged_taskIFN6tiledb6common6StatusEvEEclEv + 80
18  libtiledb.dylib                     0x00000001508655a8 _ZN6tiledb6common10ThreadPool6workerEv + 76
19  libtiledb.dylib                     0x0000000150866adc _ZNSt3__114__thread_proxyB8ue170006INS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEMN6tiledb6common10ThreadPoolEFvvEPS9_EEEEEPvSE_ + 72
20  libsystem_pthread.dylib             0x0000000190c46f94 _pthread_start + 136
21  libsystem_pthread.dylib             0x0000000190c41d34 thread_start + 8
Abort trap: 6

To work around this we must import tiledb before pyarrow.

I tried this before and it was not enough

  • Previously I only had apis/python/src/tiledbsoma/__init__.py -- not enough
  • I had missed apis/python/tests/__init__.py -- this is enough -- on this PR

See #1849 #1926 #2692 for more information.

@johnkerl johnkerl mentioned this pull request Jun 14, 2024
Closed
@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 14, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.91%. Comparing base (eab6557) to head (5e182d6).

Additional details and impacted files 
@@           Coverage Diff           @@
##             main    #2734   +/-   ##
=======================================
  Coverage   89.91%   89.91%           
=======================================
  Files          37       37           
  Lines        4026     4026           
=======================================
  Hits         3620     3620           
  Misses        406      406
Flag Coverage Δ
python 89.91% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
python_api 89.91% <ø> (ø)
libtiledbsoma ∅ <ø> (∅)

jdblischak
jdblischak reviewed Jun 14, 2024
View reviewed changes
Comment thread apis/python/setup.py
@johnkerl johnkerl marked this pull request as ready for review June 14, 2024 16:57
@johnkerl
Copy link
Copy Markdown
Contributor Author

johnkerl commented Jun 14, 2024

Gahhhhhh now it's segfaulting in CI :(

https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9519399319/job/26243930061?pr=2734

Fatal Python error: Segmentation fault

This was referenced Jun 14, 2024
Closed
Closed
Closed
@johnkerl johnkerl changed the title [python] Work around pyarrow>=13 issue on MacOS [python] Work around pyarrow>=13 issue on MacOS, main branch Jun 14, 2024
@johnkerl johnkerl mentioned this pull request Jun 14, 2024
Closed
@johnkerl
Copy link
Copy Markdown
Contributor Author

johnkerl commented Jun 14, 2024

Blocked on apache/arrow#42154 -- see also TileDB-Inc/tiledbsoma-feedstock#171 (comment)

@johnkerl johnkerl closed this Jun 14, 2024
This was referenced Sep 16, 2024
Closed
Merged
@johnkerl johnkerl deleted the kerl/workaround branch November 18, 2024 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jdblischak jdblischak jdblischak approved these changes

@ihnorton ihnorton Awaiting requested review from ihnorton

@ryan-williams ryan-williams Awaiting requested review from ryan-williams

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@johnkerl @jdblischak