Goal of having a single copy of tiledb core via libtiledbsoma, eliminating all use of tiledb-py and tiledb-r:

• Currently present in SOMAReader (as of 2023-02-13):
  • Arrays:
    • read for arrays
      • supporting methods like set_dim_points, submit, results_complete, etc.
    • nnz
  • stats access
• Reads:
  • Arrays
    • Rename SOMAReader to SOMAArrayReader
    • Optimize CSR I/O (for Python) or CSC I/O (for R): see also [c++/python] Optimize CSR I/O #718
    • shape [c++] Add shape-getter to SOMAReader API #927
    • schema reads (as Arrow) [c++] Implement Arrow-schema-getter in C++ #1121
    • metadata reads [c++] Add metadata functions to SOMAArray #926
    • open (for read) and close
  • Groups:
    • Make a SOMAGroupReader [c++] Implement group member-names-to-URIs map in C++ #1122
    • open (for read), and close
    • Get member name-to-URI map [c++] Implement group member-names-to-URIs map in C++ #1122
• Writes:
  • Arrays:
    • Make a SOMAArrayWriter
    • open (for write), and close
    • update metadata
    • write bulk data
  • Groups:
    • Make a SOMAGroupWriter [c++] tiledb.group_create via SOMAGroupWriter #1182
    • create [c++] tiledb.group_create via SOMAGroupWriter #1182
    • open (for write), and close
    • add/delete members
    • update metadata

Query API:

• Port query logic to C++
  • The current query logic (in somacore) uses only the API, with no direct implementation-level accesses.
  • This could almost all be translated into C++.
  • As pre-requisite work, we'll need SOMAArrayReader and SOMAGroupReader feature-complete (as building blocks) before we can implement this in C++
  • SOMAArrayWriter and SOMAGroupWriter development can proceed in parallel with this.

Feature completion:

• Let SOMAReader use all supported types (not just int and string as currently) to index SOMADataFrame
  • https://github.com/single-cell-data/TileDB-SOMA/blob/0.5.0a11/libtiledbsoma/src/pyapi/libtiledbsoma.cc#L328-L350

Read performance, and in particular all of the zero-copy or ecosystem format paths (optmized CSC/CSR, etc)

• in principle, we should consider at a zero-copy path for any format that the user will want to use from R or Python.

Large out-of-core / distributed processing:

• Partitioning support

cc @gspowley @nguyenv @Shelnutt2 @eddelbuettel