Skip to content

Commit 8328237

Browse files
Squashed commit of the following:
commit 777dcfc Author: Bruce Martin <[email protected]> Date: Thu Apr 24 16:10:09 2025 -0700 skip redundant metadata writes (#4014) commit 7448877 Author: XanthosXanthopoulos <[email protected]> Date: Thu Apr 24 08:37:46 2025 -0700 Fix error introduced by renaming lambda (#4012) commit 435a522 Author: XanthosXanthopoulos <[email protected]> Date: Thu Apr 24 08:22:28 2025 -0700 [c++] Optimize `nnz` computation (#3898) * count cell on overlapping region only * Use count aggregate for slow nnz * Fix iteration * Add unit test * Update libtiledbsoma/src/soma/soma_array.cc Co-authored-by: John Kerl <[email protected]> * Add tracking issue * Fix memory leak in unit test * Do not early fall back on `nnz_slow` in partial overlap/consolidated fragment * Extract common logic to lambda function * Apply PRs 3958 and 4001 * Permutate relevant fragment list based on non emprty domain sorting * Add randomized overlap nnz test * Fix spelling --------- Co-authored-by: John Kerl <[email protected]> commit af82ab8 Author: John Kerl <[email protected]> Date: Wed Apr 23 13:25:20 2025 -0400 [python] Improve null-index handling in Arrow dictionaries (#3992) * Unit-test cases * Minimal change for segv-avoidance * iterating * finalize unit-test cases; next: simplify and eludicate * neaten * prep for code-dedupe * more prep for code-deduping * continuing * complete dedupe * add some comment material * add bool-valued tests * space/time tradeoff * revert previous * Apply suggestions from code review Co-authored-by: nguyenv <[email protected]> * code-review feedback, and fix an error from another suggestion --------- Co-authored-by: nguyenv <[email protected]> commit 07e9281 Author: John Kerl <[email protected]> Date: Wed Apr 23 09:02:05 2025 -0400 apis/python/example-scripts/ingest_h5ads.py (#4006) commit 13de2c7 Author: John Kerl <[email protected]> Date: Tue Apr 22 16:16:29 2025 -0400 [c++] Trivial method-order rearrange (#3996) * [c++] Trivial method-order rearrange * one more spelling uniformization commit 58baebb Author: Bruce Martin <[email protected]> Date: Tue Apr 22 08:39:24 2025 -0700 fix bad test (#4002) commit 63d8ae1 Author: John Kerl <[email protected]> Date: Mon Apr 21 17:38:53 2025 -0400 [c++] Fix error in raise-if-slow path (#3999) * [c++] Fix error in raise-if-slow path * Workaround spurious CI failure commit 1326a02 Author: nguyenv <[email protected]> Date: Wed Apr 16 15:48:38 2025 -0500 [python] Have `SOMAFileHandle` encapsulate both `VFS` and `VFSFilebuf` (#3986) * [python] Hold `SOMAContext` smart pointer in `SOMAVFS` * [wip] Refactor SOMAVFS * [wip] Rename to FileHandle and check if open * [wip] Open in constructor * Remove uri member commit 1d3d18d Author: Bruce Martin <[email protected]> Date: Tue Apr 15 15:42:10 2025 -0700 h5py cleanup race fix (#3984) * h5py cleanup race fix * revert bug commit 5268209 Author: Ryan Williams <[email protected]> Date: Tue Apr 15 10:37:25 2025 -0400 [python] `SOMAObject.open`: rm unused `clib_type` kwarg (#3954) * `SOMAObject.open: rm unused `clib_type` kwarg * fix comment typos cf. #2820 * fix error-msg typo cf. #3847 commit 7b9684b Author: John Kerl <[email protected]> Date: Tue Apr 15 09:08:01 2025 -0400 [python] Test `get_experiment_shapes` for underpopulated experiments (#3966) commit 615561a Author: nguyenv <[email protected]> Date: Mon Apr 14 17:21:30 2025 -0500 [python][c++] Correctly manage lifetime of `SOMAVFS` with `SOMAVFSFilebuf` (#3976) * [python] Correctly manage lifetime of `SOMAVFS` with `SOMAVFSFilebuf` * Update apis/python/tests/test_basic_anndata_io.py Co-authored-by: John Kerl <[email protected]> * collect vfs --------- Co-authored-by: John Kerl <[email protected]> commit a8f5e1a Author: Ryan Williams <[email protected]> Date: Mon Apr 14 14:18:17 2025 -0400 [python] Harmonize/GC daily-GHA-failure issue-templates (#3973) * update `python-dockers.yml` failure issue template cf. JasonEtco/create-an-issue#86 (comment) * Harmonize/GC daily-GHA-failure issue-templates commit a996f79 Author: Ryan Williams <[email protected]> Date: Mon Apr 14 14:18:05 2025 -0400 [python] Remove `typing.{Dict,List,Optional,Set,Tuple,Type,Union}` references (#3956) * rm `Union`s * rm `Tuple`s * rm more `Union`s * rm `List`s * rm `Dict`s * rm `Optional`s * rm `Type`s * rm `Set` commit 64e8e5e Author: XanthosXanthopoulos <[email protected]> Date: Mon Apr 14 09:51:14 2025 -0700 [python] Increase unit-test coverage for various metadata value-types (#3972) * Add more unicode sequences * Improve error messaging * Fix macos build issue * Use `bool_` instead of `bool` * Update apis/python/src/tiledbsoma/_tdb_handles.py Co-authored-by: John Kerl <[email protected]> --------- Co-authored-by: John Kerl <[email protected]> commit 8fce365 Author: Bruce Martin <[email protected]> Date: Mon Apr 14 06:57:18 2025 -0700 [python] Fix mishandling of join-ID column (#3970) * fix mishandling of joind column * pr fb commit a6ad758 Author: nguyenv <[email protected]> Date: Fri Apr 11 19:17:03 2025 -0500 [python] Ensure `ndarray` byteorder is little-endian on matrix writes (#3963) commit 78df60e Author: John Kerl <[email protected]> Date: Fri Apr 11 18:09:46 2025 -0400 [python] Fix a typo on #3949 (#3965) commit 0ccd574 Author: John Kerl <[email protected]> Date: Fri Apr 11 16:28:40 2025 -0400 [python] Refactor `io/shaping.py`; `get_experiment_shapes` (#3962) * [python] Refactor `io/shaping.py` * code-review feedback * [python] Implement `tiledbsoma.io.get_experiment_shapes` (#3949) * [python] `get_experiment_shapes` * code-review feedback; rebase coming [skip ci] commit 7d57cc4 Author: nguyenv <[email protected]> Date: Thu Apr 10 20:56:14 2025 -0500 [python] Ensure schema is explicitly passed to `pa.Table.from*` converters (#3957) commit c4198c3 Author: John Kerl <[email protected]> Date: Thu Apr 10 17:33:47 2025 -0400 [python] Fix a spurious exception-class issue (#3958) * fix an issue raised by ryan [wip] * put py::arg in the right place -- thanks google commit b319af6 Author: Ryan Williams <[email protected]> Date: Thu Apr 10 11:06:10 2025 -0400 [python] Remove `pre-commit/mirrors-prettier` hook (#3952) * Use `rbubley/mirrors-prettier` @ 3.5.3 `pre-commit/mirrors-prettier` was archived, this appears to be a reasonably-maintained and -starred fork, which also only updates vs. full (non-alpha) prettier` releases * rm `prettier` pre-commit hook commit 8b8183b Author: Ryan Williams <[email protected]> Date: Wed Apr 9 15:27:24 2025 -0400 python-uv.dockerfile (#3948) commit d96e81c Author: Ryan Williams <[email protected]> Date: Wed Apr 9 15:22:14 2025 -0400 rm numcodecs<0.16 pin (#3946) `zarr==2.18.7` removes the need for it, cf.: - zarr-developers/zarr-python#2963 - #3917 commit f0f1cc5 Author: John Kerl <[email protected]> Date: Wed Apr 9 13:04:32 2025 -0400 [python] Extend previous-version tests for experiment query (#3945) commit e0b594c Author: John Kerl <[email protected]> Date: Wed Apr 9 10:56:13 2025 -0400 [python] Previous-versions testing for enum getter/extender (#3940) * [python] Extend previous-versions testing * extender commit 847d85f Author: John Kerl <[email protected]> Date: Wed Apr 9 10:29:42 2025 -0400 [python] Improve UX for `extend_enumeration_values` with non-Arrow types (#3939) commit fc5d2c1 Author: John Kerl <[email protected]> Date: Wed Apr 9 08:46:12 2025 -0400 [c++] Follow-up to #3873 and #3928 (#3937) commit d65892d Author: Bruce Martin <[email protected]> Date: Tue Apr 8 15:13:55 2025 -0700 fix valgrind errors in unit tests (#3935) commit 4746ae3 Author: XanthosXanthopoulos <[email protected]> Date: Tue Apr 8 13:10:08 2025 -0700 Replace std::unique_ptr with raw pointer for attribute arrow schema (#3928) commit 10b2549 Author: Bruce Martin <[email protected]> Date: Tue Apr 8 12:23:53 2025 -0700 [python] Ingest performance improvements (#3865) * first cut at ingest tuning * mp optional param * fix tests, add load from S3 * reshape fix * lint * add multiprocessing config warning * performance tuning * pass context * lint * add test * interim checkpoint on enum cleanup * add test for subset * add new extend enum code (commented out) * migrate to new enum API * bump typeguard version to work around typeguard#504 * improve doctrings * more cleanup * loosen existance checks when from_anndata/from_h5ad are run in schema_only mode * cleanup and additional error checking * more docstring cleanup * more docstring cleanup * more docstrings * pr fb * pr fb * more pr fb * increase test code coverage * additional code coverage * lint * PR fb * PR fb * more PR fb * lint * remove extraneous line break in docstring commit 92fc2b8 Author: nguyenv <[email protected]> Date: Tue Apr 8 13:21:43 2025 -0500 [python] Return `datetime` `domain`/`maxdomain` as `pa.TimestampScalar` (#3877) - The domain/maxdomain was return as `np.datetime64`. Give that Arrow is used as the default type for all return values, this is now corrected to `pa.TimestampScalar` - Allow the passed-in type of the domain to be `int` and `pa.TimestampScalar` in addition to `np.datetime64` commit 5d7e70b Author: Paul Hoffman <[email protected]> Date: Tue Apr 8 12:51:01 2025 -0400 [r] Add support for reading v5 assays from an axis query (#3008) * [r] Add support for reading v5 assays from an axis query Seurat v5 adds support for ragged arrays, where not every `X` layer has exactly the same cells and features. To handle this, ragged `X` layers need to be re-indexed and re-shaped on ingestion to resize down to only the data present Modified SOMA methods: - `SOMAExperimentAxisQuery$to_seurat()` and `SOMAExperimentAxisQuery$to_seurat_assay()`: now read in as v5 assays New SOMA methods: - `SOMAExperimentAxisQuery$private$.to_seurat_assay_v5()`: helper method to read in ragged and non-ragged arrays into a v5 assay; note this method only handles expression layers, all other assay-level information is handled by parent `$to_seurat_assay()` to share code with v3 assay outgestion Requires #2523 and #3007 [SC-52261](https://app.shortcut.com/tiledb-inc/story/52261/) * Use hint helpers * Fix shape, regex uses * Use hint helper functions * Use updated column names * Apply suggestions from code review Co-authored-by: John Kerl <[email protected]> * Finish incomplete suggestions * more codereview feedback --------- Co-authored-by: John Kerl <[email protected]> commit 0b46462 Author: Bruce Martin <[email protected]> Date: Tue Apr 8 07:10:32 2025 -0700 [python] Fix partition readers' naïve assumption that `nnz` is fast (#3915) * add work-around for single-fragment nnz * add raise_if_slow to nnz, and use in partitioning readers * add raise_if_slow to nnz, and use in partitioning readers * revert likely buggy nnz optimization * improve fallback partitioning for ExperimentAxisQuery * pin numcodecs * pr fb * pr fb * PR fb commit 9d2ab41 Author: John Kerl <[email protected]> Date: Tue Apr 8 09:35:16 2025 -0400 Include 1.16.1 datasets in soma-experiment-versions tests (#3912) commit 5d386d4 Author: Ryan Williams <[email protected]> Date: Mon Apr 7 19:00:44 2025 -0400 work around #3917 in RTD build (#3922) commit 2f4241b Author: John Kerl <[email protected]> Date: Mon Apr 7 17:51:06 2025 -0400 [python] Protect against an AnnData/SciPy incompatibility (#3919) commit 8644785 Author: John Kerl <[email protected]> Date: Mon Apr 7 17:17:20 2025 -0400 [python] Use `typeguard==4.4.2` (#3921) commit ad5fae4 Author: Ryan Williams <[email protected]> Date: Mon Apr 7 13:47:01 2025 -0400 `setup.py`: remove deprecated `License ::` from `classifiers` (#3916) * `setup.py`: remove deprecated `License ::` from `classifiers` [sc-65131] * pin `numcodecs<0.16` cf. zarr-developers/zarr-python#2963 commit cdc0d67 Author: Julia Dark <[email protected]> Date: Mon Apr 7 11:57:44 2025 -0400 [python] Directly read Visium table instead of filtering through AnnData (#3835) * Directly read data from 10x HDF5 instead of opening it with AnnData. * (bug fix) Add `platform_config` to spatial array creation. commit 37480ff Author: XanthosXanthopoulos <[email protected]> Date: Fri Apr 4 17:01:36 2025 -0700 Replace deprecated `dump` function (#3913) commit f9450ba Author: Ryan Williams <[email protected]> Date: Fri Apr 4 12:03:40 2025 -0400 Publish ARM Linux wheels (#3909) * `python-packaging`: rm `continue-on-error: true` from Test PyPI publish That was masking failures to upload to test.pypi.org (due to 10GB space quota being full), e.g. https://github.com/single-cell-data/TileDB-SOMA/actions/runs/13913129586/job/38931629454 * Publish manylinux aarch64 wheels * Audit/Factor `cibuildwheel` env vars * rm unused `cc`/`cxx` smoke-test matrix vars * add 3.9 smoke-test jobs * rm unused `CIBW_BEFORE_BUILD` hook commit 1c6a057 Author: XanthosXanthopoulos <[email protected]> Date: Fri Apr 4 06:10:52 2025 -0700 [c++] Fix Arrow memory leaks in unit tests (#3873) * Accept const reference of unique pointers * Release allocated arrow objects * Fix deallocation of arrow buffers, return raw pointer for all non parent arrow objects * Release allocated memrory in unit tests * Pass raw pointers to `set_array_data` commit d77edea Author: John Kerl <[email protected]> Date: Thu Apr 3 13:28:54 2025 -0400 [c++] Show errant value in not-array-not-group error messages (#3897) * [c++] Show errant value in not-array-not-group error messages * fix dangling WIP bits * code-review feedback commit fb12025 Author: John Kerl <[email protected]> Date: Thu Apr 3 09:41:51 2025 -0400 [ci] Remove ticket-cutter for no-longer-nightly job (#3900) commit 485d4d8 Author: Ryan Williams <[email protected]> Date: Wed Apr 2 23:19:46 2025 -0400 `python-ci-packaging.yml`: only run on `push` events to `main` (#3895) Otherwise it runs on any branch push that touches relevant `paths`, which is not necessary (given the `paths` are already tested for `pull_request` events) commit 6c08b42 Author: John Kerl <[email protected]> Date: Wed Apr 2 19:55:07 2025 -0400 [c++/python] Expose enumeration-extender (#3815) * wip * Constraints testing against old pandas/numpy versions * Improve structuring * Merge in #3845 * fix for older numpy * resolve #3815 (comment) [skip ci] * ordered flag * #3815 (comment) * resolve #3815 (comment) * more on #3815 (comment) * temp/wip [skip ci] * RAII-related code-review feedback * resolve #3815 (comment) * null-checking * iterating on code-review feedback * code-review feedback * add a helpful comment * try to be even more clear * check nan bits * Advice from #3888 thanks to Xanthos * Apply suggestions from code review Co-authored-by: nguyenv <[email protected]> * neatens for code review * Apply suggestions from code review Co-authored-by: Julia Dark <[email protected]> * fix lint induced by code-review suggestions * const-refs for column name -- code-review feedback [skip ci] * some static_cast callsites [skip ci] * docblocks per code-review feedback [skip ci] * docblocks per code-review feedback * try-catch solution; ScopedExecutor coming next * ScopedExecutor --------- Co-authored-by: nguyenv <[email protected]> Co-authored-by: Julia Dark <[email protected]> commit d95ba39 Author: Ryan Williams <[email protected]> Date: Wed Apr 2 11:28:05 2025 -0400 [ci/python] `python-dockers.yml`: improve issue template (#3894) * Improve `python-dockers` issue template * gha wiki link update [skip ci] commit 4a3cfc6 Author: Ryan Williams <[email protected]> Date: Wed Apr 2 11:21:57 2025 -0400 [python] Pin `build-system.requires`, ARM Linux Docker-builds to `cmake<4` (#3891) * pin cmake<4 in setuptools build * pip constraint `cmake<4` in ARM Linux docker builds * `python-dockers.yml`: `fail-fast: false` * `python-dockers.yml`: improve job names commit c89fed9 Author: John Kerl <[email protected]> Date: Tue Apr 1 15:57:59 2025 -0400 [ci] Run remote-storage workflow on all commits (#3885) commit 5e52122 Author: John Kerl <[email protected]> Date: Tue Apr 1 15:12:10 2025 -0400 [c++/python] Revert 3764 for global-order writes (#3886) commit ef96798 Author: XanthosXanthopoulos <[email protected]> Date: Tue Apr 1 09:16:23 2025 -0700 [r] Use exposed logger in C++ bindings (#3868) * Roll back to used `spdl::` [skip ci] * Replace spdl::* with tdbs::* commit 945d195 Author: Ryan Williams <[email protected]> Date: Tue Apr 1 10:23:01 2025 -0400 `python-ci-full.yml`: fix `include` rules (#3881) @bkmartinjr noticed our ubuntu GHAs had `cc`/`cxx` set to `clang`: sc-65188 commit 0aae54d Author: Bruce Martin <[email protected]> Date: Mon Mar 31 18:01:48 2025 -0700 [python] Ingest from S3 performance (#3872) * h5ad read from s3 performance * add simplicity * increase cache size * more macro data refinement * cleanup * darn clang-format * remove dead code * py39 lint * fix pre py 3.12 typeguard error * log python packages installed * pr fb * PR fb * refactor for clarity commit ee8aab4 Author: nguyenv <[email protected]> Date: Mon Mar 31 13:33:16 2025 -0500 [python/c++] Allow GOW multi-submit for single commit (#3764) Co-authored-by: John Kerl <[email protected]> commit 88317fb Author: Julia Dark <[email protected]> Date: Thu Mar 27 14:40:32 2025 -0400 [python] Prepare `io.shaping` for spatial additions (#3856) * Update docstrings to reflect which elements inside the experiment are being accessed. * Refactor `_treewalk` to make adding more elements easier. --------- Co-authored-by: John Kerl <[email protected]> commit 5331558 Author: nguyenv <[email protected]> Date: Thu Mar 27 13:06:52 2025 -0500 [c++] Correctly handle NaN category values (#3845) Add ManagedQuery::_find_enum_match which uses bitwise comparison. Like std::find, the method returns an iterator that points to the match value in the container; if not found, returns an iterator to the end of the container. commit 88c1664 Author: John Kerl <[email protected]> Date: Thu Mar 27 10:54:42 2025 -0400 [python] Fix a venial typo in a unit-test case (#3862)
1 parent 4ab9948 commit 8328237

143 files changed

Lines changed: 8731 additions & 2684 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/daily-remote-tests-issue-template.md renamed to .github/workflows/daily-python-dockers-issue-template.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
---
2-
title: Daily GitHub Actions fail for remote tests on {{ date | date('ddd, MMMM Do YYYY') }}
3-
assignees: nguyenv, johnkerl
2+
title: Daily `python-dockers.yml` build failing
43
labels: bug
54
---
65

.github/workflows/daily-test-build-issue-template.md renamed to .github/workflows/daily-python-packaging-issue-template.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
---
2-
title: Daily GitHub Actions build fail on {{ date | date('ddd, MMMM Do YYYY') }}
3-
assignees: nguyenv, ryan-williams, johnkerl
2+
title: Daily `python-packaging.yml` build failing
43
labels: bug
54
---
65

‎.github/workflows/python-ci-full.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,10 +26,10 @@ jobs:
2626
# https://github.com/single-cell-data/TileDB-SOMA/issues/1849
2727
python-version: ['3.9', '3.10', '3.11', '3.12', '3.13']
2828
include:
29-
- runs-on: ubuntu-24.04
29+
- os: ubuntu-24.04
3030
cc: gcc-13
3131
cxx: g++-13
32-
- runs-on: macos-latest
32+
- os: macos-latest
3333
cc: clang
3434
cxx: clang++
3535
uses: ./.github/workflows/python-ci-single.yml

‎.github/workflows/python-ci-packaging.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ name: TileDB-SOMA Python CI (Packaging)
55

66
on:
77
push:
8+
branches: [ main ]
89
paths:
910
- '.github/workflows/python-ci-packaging.yml'
1011
- 'apis/python/MANIFEST.in'

‎.github/workflows/python-ci-single.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,9 @@ jobs:
118118
- name: Show XCode version
119119
run: clang --version
120120

121+
- name: Log pip dependencies
122+
run: pip list
123+
121124
- name: Run libtiledbsoma unit tests
122125
run: ctest --output-on-failure --test-dir build/libtiledbsoma -C Release --verbose
123126
env:

‎.github/workflows/python-dockers.yml‎

Lines changed: 28 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -11,30 +11,44 @@ on:
1111
jobs:
1212
docker:
1313
strategy:
14+
fail-fast: false
1415
matrix:
1516
os:
1617
- runs-on: ubuntu-24.04
1718
suffix: ''
1819
- runs-on: ubuntu-24.04-arm
19-
suffix: ' (ARM)'
20+
suffix: ' ARM'
2021
# TODO: currently impossible(?) to build Docker images on macOS runners: https://github.com/single-cell-data/TileDB-SOMA/issues/3781
2122
# - macos-15
2223
# TODO: try x86 macOS builds, once https://github.com/single-cell-data/TileDB-SOMA/issues/3787 is resolved
2324
# - macos-13
2425
img:
25-
- name: ubuntu-src
26-
- name: ubuntu
26+
- name: Ubuntu src
27+
file: ubuntu-src
28+
- name: Ubuntu 1.15.7
29+
file: ubuntu
2730
args: v=1.15.7
28-
- name: ubuntu
31+
- name: Ubuntu 1.16.0
32+
file: ubuntu
2933
args: v=1.16.0
30-
- name: bookworm
34+
- name: Debian 1.15.7 gcc12
35+
file: bookworm
3136
args: gcc=12 v=1.15.7
32-
- name: bookworm
37+
- name: Debian 1.15.7 gcc13
38+
file: bookworm
3339
args: gcc=13 v=1.15.7
34-
- name: bookworm
40+
- name: Debian 1.16.0 gcc13
41+
file: bookworm
3542
args: gcc=13 v=1.16.0
36-
- name: langchain
37-
name: ${{ matrix.img.name }} ${{ matrix.img.args }}${{ matrix.os.suffix }}
43+
- name: Langchain
44+
file: langchain
45+
- name: Python 3.12 + uv
46+
file: python-uv
47+
args: python=3.12
48+
- name: Python 3.13 + uv
49+
file: python-uv
50+
args: python=3.13
51+
name: ${{ matrix.img.name }}${{ matrix.os.suffix }}
3852
runs-on: ${{ matrix.os.runs-on }}
3953
steps:
4054
- uses: actions/checkout@v4
@@ -44,11 +58,11 @@ jobs:
4458
for arg in ${{ matrix.img.args }}; do
4559
args+=(--build-arg $arg)
4660
done
47-
cmd=(docker build -f ${{ matrix.img.name }}.dockerfile -t ${{ matrix.img.name }} "${args[@]}" .)
61+
cmd=(docker build -f ${{ matrix.img.file }}.dockerfile -t ${{ matrix.img.file }} "${args[@]}" .)
4862
echo "Running: ${cmd[*]}"
4963
"${cmd[@]}"
5064
working-directory: docker
51-
- run: docker run ${{ matrix.img.name }}
65+
- run: docker run ${{ matrix.img.file }}
5266
working-directory: docker
5367
# File an issue if anything fails, but only for scheduled ones (not manual "workflow_dispatch" runs).
5468
create_issue_on_fail:
@@ -63,4 +77,6 @@ jobs:
6377
env:
6478
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
6579
with:
66-
filename: .github/workflows/daily-test-build-issue-template.md
80+
filename: .github/workflows/daily-python-dockers-issue-template.md
81+
assignees: ryan-williams, johnkerl
82+
update_existing: true

‎.github/workflows/python-packaging.yml‎

Lines changed: 26 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ jobs:
4141

4242
# This step builds wheels and uploads them to GitHub Actions storage.
4343
# See also https://github.com/single-cell-data/TileDB-SOMA/issues/700.
44-
# See also https://github.com/single-cell-data/TileDB-SOMA/wiki/PyPI-packaging-WIP
44+
# See also https://github.com/single-cell-data/TileDB-SOMA/wiki/PyPI-packaging-notes
4545
# for important transitional context.
4646
wheels:
4747
# Note: tries all supported Python versions as specified in apis/python/setup.py
@@ -51,11 +51,17 @@ jobs:
5151
strategy:
5252
matrix:
5353
python-version: [ '39', '310', '311', '312', '313' ]
54-
cibw_build: [ manylinux_x86_64, macosx_x86_64, macosx_arm64 ]
54+
cibw_build: [ manylinux_x86_64, manylinux_aarch64, macosx_x86_64, macosx_arm64 ]
5555
include:
5656
- cibw_build: manylinux_x86_64
5757
os: ubuntu-24.04
58-
wheel-name: manylinux_2_28
58+
wheel-name: manylinux_2_28-x86_64
59+
- cibw_build: manylinux_aarch64
60+
os: ubuntu-24.04-arm
61+
wheel-name: manylinux_2_28-aarch64
62+
CIBW_BEFORE_BUILD_LINUX: dnf install -y ninja-build perl-IPC-Cmd zip
63+
# vcpkg source build requires this on ARM, for some reason; cf. https://github.com/NixOS/nixpkgs/issues/335868
64+
CIBW_ENVIRONMENT_LINUX: VCPKG_FORCE_SYSTEM_BINARIES=1
5965
- cibw_build: macosx_x86_64
6066
os: macos-latest
6167
cibw_archs_macos: x86_64
@@ -90,27 +96,18 @@ jobs:
9096
with:
9197
xcode-version: '15.4'
9298
- name: Build wheels
93-
uses: pypa/cibuildwheel@v2.22.0
99+
uses: pypa/cibuildwheel@v2.23.2
94100
with:
95101
package-dir: tiledbsoma.tar.gz
96102
only: cp${{ matrix.python-version }}-${{ matrix.cibw_build }}
97103
env:
98-
CIBW_BUILD: ${{ matrix.cibw_build }}
99104
CIBW_BUILD_VERBOSITY: 1
100-
CIBW_MANYLINUX_X86_64_IMAGE: manylinux_2_28
101-
CIBW_BEFORE_BUILD_LINUX: yum -y remove gcc-toolset-12\*; yum -y install gcc-toolset-13; bash -x -c 'rm -rf tiledbsoma*/dist_links/dist/lib*'
102-
# ^ Delete lib folder that apis/python/setup.py:find_or_build() looks for in deciding to
103-
# run CMake build or not. Otherwise it'll keep reusing the library file built in the
104-
# first iteration of cibuildwheel's outer loop, resulting in wheels with the library
105-
# built for the wrong python version.
106-
CIBW_BEFORE_BUILD: bash -x -c 'rm -rf tiledbsoma*/dist_links/dist/lib*'
107-
# ^ Delete lib folder that apis/python/setup.py:find_or_build() looks for in deciding to
108-
# run CMake build or not. Otherwise it'll keep reusing the library file built in the
109-
# first iteration of cibuildwheel's outer loop, resulting in wheels with the library
110-
# built for the wrong python version.
105+
CIBW_BEFORE_BUILD_LINUX: ${{ matrix.CIBW_BEFORE_BUILD_LINUX || '' }}
106+
CIBW_ENVIRONMENT_LINUX: ${{ matrix.CIBW_ENVIRONMENT_LINUX || '' }}
107+
# Most recent tags with no listed security issues at https://quay.io/repository/pypa/manylinux_2_28_aarch64?tab=tags, ca. 2025-04-03
108+
CIBW_MANYLINUX_X86_64_IMAGE: quay.io/pypa/manylinux_2_28_x86_64:2025.03.22-2
109+
CIBW_MANYLINUX_AARCH64_IMAGE: quay.io/pypa/manylinux_2_28_aarch64:2025.03.22-2
111110
CIBW_ARCHS_MACOS: ${{ matrix.cibw_archs_macos }}
112-
CIBW_ENVIRONMENT_LINUX : CC=/opt/rh/gcc-toolset-13/root/usr/bin/gcc CXX=/opt/rh/gcc-toolset-13/root/usr/bin/g++
113-
CIBW_TEST_SKIP: "*_arm64"
114111
CMAKE_OSX_ARCHITECTURES: ${{ matrix.cibw_archs_macos }}
115112
MACOSX_DEPLOYMENT_TARGET: "13.3"
116113

@@ -128,6 +125,8 @@ jobs:
128125
strategy:
129126
matrix:
130127
python:
128+
- undotted-version: '39'
129+
dotted-version: '3.9'
131130
- undotted-version: '310'
132131
dotted-version: '3.10'
133132
- undotted-version: '311'
@@ -137,25 +136,23 @@ jobs:
137136
- undotted-version: '313'
138137
dotted-version: '3.13'
139138
wheel-name:
140-
- manylinux_2_28
139+
- manylinux_2_28-x86_64
140+
- manylinux_2_28-aarch64
141141
- macos-x86_64
142142
- macos-arm64
143143
include:
144-
- wheel-name: manylinux_2_28
144+
- wheel-name: manylinux_2_28-x86_64
145145
os: ubuntu-24.04
146146
arch: x86_64
147-
cc: gcc-13
148-
cxx: g++-13
147+
- wheel-name: manylinux_2_28-aarch64
148+
os: ubuntu-24.04-arm
149+
arch: aarch64
149150
- wheel-name: macos-x86_64
150151
os: macos-13
151152
arch: x86_64
152-
cc: clang
153-
cxx: clang++
154153
- wheel-name: macos-arm64
155154
os: macos-14
156155
arch: arm64
157-
cc: clang
158-
cxx: clang++
159156
fail-fast: false
160157
steps:
161158
- name: Set up Python ${{ matrix.python.dotted-version }}
@@ -211,7 +208,6 @@ jobs:
211208
ls -l dist
212209
- name: Publish packages to TestPyPI
213210
uses: pypa/gh-action-pypi-publish@release/v1
214-
continue-on-error: true
215211
with:
216212
repository-url: https://test.pypi.org/legacy/
217213
packages_dir: dist
@@ -257,4 +253,6 @@ jobs:
257253
env:
258254
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
259255
with:
260-
filename: .github/workflows/daily-test-build-issue-template.md
256+
filename: .github/workflows/daily-python-packaging-issue-template.md
257+
assignees: nguyenv, ryan-williams, johnkerl
258+
update_existing: true

‎.github/workflows/python-remote-storage.yml‎

Lines changed: 12 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,19 @@
11
name: TileDB-SOMA Python CI (remote storage)
22

33
on:
4-
# Not for regular use, but you can uncomment this when putting up PRs on this
5-
# file. Just remember to take it back out. There's no need to run
6-
# remote-storage tests on every single PR push to our repo.
7-
#
8-
# pull_request:
9-
#
10-
# Allows for 'as needed' manual trigger:
4+
push:
5+
branches:
6+
- main
7+
- 'release-*'
8+
pull_request:
9+
paths-ignore:
10+
- '**.md'
11+
- 'apis/r/**'
12+
- 'docker/**'
13+
- 'docs/**'
14+
- '.github/**'
15+
- '!.github/workflows/python-remote-storage.yml'
1116
workflow_dispatch:
12-
#
13-
# Use a regular nightly build as well (time is UTC):
14-
schedule:
15-
- cron: "25 5 * * *"
1617

1718
env:
1819
# Don't name this "TILEDB_REST_TOKEN" since that will map into a core
@@ -107,19 +108,3 @@ jobs:
107108
- name: Run pytests for Python
108109
shell: bash
109110
run: python -m pytest apis/python/remote_tests -v --durations=20 --maxfail=50
110-
111-
# File a bug report if anything fails, but don't file tickets for manual runs
112-
# -- only for scheduled ones.
113-
create_issue_on_fail:
114-
runs-on: ubuntu-24.04
115-
needs: [ci]
116-
if: (failure() || cancelled()) && github.event_name != 'workflow_dispatch' && github.run_attempt == 1
117-
steps:
118-
- name: Checkout TileDB-SOMA `main`
119-
uses: actions/checkout@v4
120-
- name: Create Issue if Build Fails
121-
uses: JasonEtco/create-an-issue@v2
122-
env:
123-
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
124-
with:
125-
filename: .github/workflows/daily-remote-tests-issue-template.md

‎.pre-commit-config.yaml‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,3 @@ repos:
2222
- types-setuptools
2323
args: ["--config-file=apis/python/pyproject.toml", "apis/python/src", "apis/python/devtools"]
2424
pass_filenames: false
25-
- repo: https://github.com/pre-commit/mirrors-prettier
26-
rev: v4.0.0-alpha.8
27-
hooks:
28-
- id: prettier
29-
files: "encoding_specification.md"

0 commit comments

Comments
 (0)