Skip to content

feat: add unenforced_clustering_key to format spec#6552

Merged
jackye1995 merged 2 commits into
lance-format:mainfrom
beinan:feat/unenforced-clustering-key
May 8, 2026
Merged

feat: add unenforced_clustering_key to format spec#6552
jackye1995 merged 2 commits into
lance-format:mainfrom
beinan:feat/unenforced-clustering-key

Conversation

@beinan

@beinan beinan commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add unenforced_clustering_key metadata to the Lance schema format, mirroring the existing unenforced_primary_key pattern
  • Clustering keys hint at the physical ordering of data within a table, enabling query engine optimizations such as storage-partitioned joins (SPJ)
  • Unlike primary keys, clustering key fields may be nullable

Changes across all layers:

  • Protobuf: unenforced_clustering_key (bool) + unenforced_clustering_key_position (uint32) fields 14-15
  • Rust core: field struct, constants, Arrow metadata parsing, schema method
  • Protobuf serialization: round-trip support with backward compat
  • Java JNI + LanceField: constructor args and getters
  • Python bindings + type stubs: is_unenforced_clustering_key() / unenforced_clustering_key_position()
  • Format docs: clustering key metadata section

Motivation

This was discussed in the lance-spark SPJ PR (lance-format/lance-spark#445). Rather than using custom table properties, embedding clustering key info in the schema metadata follows the established pattern and avoids migration issues.

Test plan

  • cargo check -p lance-core -p lance-file passes
  • cargo test -p lance-core -p lance-file passes (all tests including existing primary key tests)
  • cargo clippy -p lance-core -p lance-file --tests -- -D warnings clean

🤖 Generated with Claude Code

@github-actions github-actions Bot added enhancement New feature or request A-python Python bindings A-java Java bindings + JNI labels Apr 17, 2026
Comment thread docs/src/format/table/schema.md Outdated
### Clustering Key Metadata

Clustering key configuration is handled by two protobuf fields in the Field message:
- **unenforced_clustering_key** (bool): Whether this field is part of the clustering key

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for unenforced primary key, we initially introduced the boolean and later moved to position because position makes the key fields ordered. I think for clustering, we can just go with position directly without the boolean

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Jack! Good call - updated to drop the boolean and use only unenforced_clustering_key_position as the single source of truth.

@codecov

codecov Bot commented Apr 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@beinan
beinan force-pushed the feat/unenforced-clustering-key branch from 415fb4f to 1118bfc Compare April 17, 2026 01:28

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should probably also add getUnenforcedPrimaryKey() and getUnenforcedClusteringKey() in LanceSchema

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and similar comment for python

@beinan
beinan force-pushed the feat/unenforced-clustering-key branch from 1118bfc to ad6157f Compare April 17, 2026 07:40
@beinan

beinan commented Apr 17, 2026

Copy link
Copy Markdown
Contributor Author

Thanks Jack! Added getUnenforcedPrimaryKey() and getUnenforcedClusteringKey() to LanceSchema in both Java and Python, returning fields sorted by position.

@beinan
beinan force-pushed the feat/unenforced-clustering-key branch 2 times, most recently from 3304e87 to 417ddb4 Compare April 18, 2026 06:33
beinan and others added 2 commits April 24, 2026 05:36
Add clustering key metadata to the Lance schema, following the same
pattern as unenforced_primary_key. Clustering keys hint at the physical
ordering of data within a table, enabling query engine optimizations
such as storage-partitioned joins (SPJ).

Changes across all layers:
- Protobuf: two new fields (bool + uint32 position) in Field message
- Rust core: field struct, constants, Arrow metadata parsing
- Rust schema: ordered field collection method
- Protobuf serialization: round-trip support
- Java JNI + LanceField: constructor and getters
- Python bindings + type stubs: is/position methods
- Format docs: clustering key metadata section

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@beinan
beinan force-pushed the feat/unenforced-clustering-key branch from 18e9fe5 to 98088d4 Compare April 24, 2026 06:03
@jackye1995
jackye1995 merged commit b33058a into lance-format:main May 8, 2026
29 checks passed
jackye1995 pushed a commit that referenced this pull request May 17, 2026
…#6812)

## Summary

#6552 added the unenforced clustering key to the schema format, but the
`UpdateConfig` transaction-apply path does not recompute the cached
`Field::unenforced_clustering_key_position` after applying field
metadata
updates — only `unenforced_primary_key_position` is recomputed (added
for the
primary key in #6706). The protobuf encoder reads the cached option, not
the
metadata HashMap, so installing or removing an unenforced clustering key
via
`update_field_metadata` does not round-trip.

This recomputes `unenforced_clustering_key_position` on apply, and
treats the
clustering key as a reserved, immutable schema property — mirroring the
unenforced primary key:

- once a clustering key is set, any commit that changes it, or that
writes its
  reserved metadata key, is rejected;
- writing the reserved key with a value that is not a valid position is
  rejected rather than silently ignored;
- a valid first install (single- or multi-column) and updates to other
field
  metadata are unaffected.

The check runs on every apply, including conflict-rebase, so it also
rejects
the concurrent-writer race.

`LANCE_UNENFORCED_CLUSTERING_KEY_POSITION` is also re-exported from
`lance_core::datatypes`, alongside the primary-key constants.

The metadata-update tests for both the primary and clustering keys are
restructured into individual tests with simpler `lance_datagen`-based
setup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-java Java bindings + JNI A-python Python bindings enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants