Need additional tests. Stuff I can think of:

1. tests that probe the expected error cases the ManagedQuery should catch/check? OTOMH:
   • batches have out-of-order indices
   • out-of-order submits
   • submitted batches with incompatible schema
   • anything else you can think of
2. tests on arrays with more dimensions, and dimensions of other types (both the common cases, e.g., spatial dataframes, and user-specified cases such as a soma_joinid+string dimensions)
3. edge cases, such as empty batches/chunks
4. tests on sparse & dense array
5. (if relevant?) nullability layer and batched? (likely not needed)

thanks for adding additional tests. The one important case missing (IMHO) is a DataFrame with user-specified index columns. PointCloud covers some of this (multiple numeric indices), but not the case where the user specifies a string index.

Something like a DataFrame with index_column_names containing a string and float column in addition to the normal soma_joinid

I'm just trying to cover the edge cases of variable length or non-integral dimensions, e.g.,

df = pd.DataFrame(
    {
        "soma_joinid": pd.Series([4], dtype=np.int64),
        "str_idx": pd.Series(['a'],dtype=str),
        "float_idx": pd.Series([1.1],dtype=np.float32),
        "attr": pd.Series(["hi"], dtype="str"),
    }
)
soma.DataFrame.create(uri, schema=pa.Schema.from_pandas(df), index_column_names=['soma_joinid','str_idx','float_idx'])

which creates:

ArraySchema(
  domain=Domain(*[
    Dim(name='str_col', domain=('', ''), tile=None, dtype='|S0', var=True, filters=FilterList([ZstdFilter(level=3), ])),
    Dim(name='float_idx', domain=(-3.4028235e+38, 3.4028235e+38), tile=1.0, dtype='float32', filters=FilterList([ZstdFilter(level=3), ])),
    Dim(name='soma_joinid', domain=(0, 9223372036854775805), tile=1, dtype='int64', filters=FilterList([ZstdFilter(level=3), ])),
  ]),
  attrs=[
    Attr(name='attr', dtype='<U0', var=True, nullable=True, enum_label=None, filters=FilterList([ZstdFilter(level=-1), ])),
  ],
  cell_order='row-major',
  tile_order='row-major',
  capacity=100000,
  sparse=True,
  allows_duplicates=False,
)

Where/how is this flag (sort_coords) documented for the end user? Or is it only a private interface?

If public, we need docstrings and a HISTORY.md entry at a minimum.

This flag is only used internally. The user sets this option using the platform config.

AFAIK, platform_config is also undocumented.

@aaronwolen - do we need a plan to document this, or are you OK if it lands w/o user-facing help/examples?

I think for now the most logical place to document this for users is in TileDBWriteOptions. @nguyenv could you add those docstrings in this PR? I think your explanation in the CHANGELOG looks good.

is "without changing the user interface" true? It looks like there is a user-facing parameter in platform_config, which has a specific effect on package behavior, so that isn't just an internal optimization.

Suggest something like:

Multiple writes of pre-sorted data may now be written to a single fragment using TileDB global order writes. Enable this performance optimization by setting the platform_config parameter sort_coords to True in the call to write. Will raise an error if data is not written in global sort order.

@aaronwolen - please feel free to suggest better prose for the HISTORY, as we do not have any other docs on this new feature