[ENH]: Create collection impl in rust sysdb by sanketkedia · Pull Request #6131 · chroma-core/chroma

sanketkedia · 2026-01-08T07:53:04Z

Description of changes

Summarize the changes made by this PR.

Improvements & Bug fixes
- Create collection impl with all the shenanigans
New functionality
- ...

Test plan

How are these changes tested?
Added tests

Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Migration plan

None

Observability plan

None

Documentation Changes

None

github-actions · 2026-01-08T07:53:19Z

sanketkedia · 2026-01-08T07:53:24Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

propel-code-bot · 2026-01-08T07:56:38Z

Implement Rust SysDB collection creation flow

Adds full collection creation support to the Rust SysDB/Spanner backend. The change introduces a transactional create_collection implementation with validation, segment and metadata persistence, get-or-create semantics, and conversion utilities, along with the gRPC surface, request/response plumbing, and extensive emulator-backed tests.

Key Changes

• Implemented SpannerBackend::create_collection to validate input, resolve database IDs, handle get-or-create flows, buffer mutations for collections/segments/metadata/cursors, and materialize Collection results inside the transaction.
• Expanded rust-sysdb types with owned request structs (CreateCollectionRequest plus existing requests), enriched SysDbError variants, added SpannerRows to Collection, and introduced CreateCollectionResponse conversions.
• Updated Runnable/Backend APIs to consume requests by value, wired the gRPC create_collection endpoint in server.rs, and added comprehensive integration tests covering duplicate, get_or_create, metadata/schema, and error scenarios.
• Included supporting adjustments such as chrono dependency, metadata/schema parsing, and helper utilities for building test segments and verifying returned collections.

Affected Areas

• rust/rust-sysdb/src/spanner.rs
• rust/rust-sysdb/src/types.rs
• rust/rust-sysdb/src/backend.rs
• rust/rust-sysdb/src/server.rs
• rust/types/src/collection.rs
• rust/rust-sysdb/src/tests

This summary was automatically generated by @propel-code-bot

propel-code-bot · 2026-01-08T07:58:49Z

rust/rust-sysdb/src/spanner.rs

+                                chroma_types::MetadataValue::Int(i) => (None, Some(*i), None, None),
+                                chroma_types::MetadataValue::Float(f) => (None, None, Some(*f), None),
+                                chroma_types::MetadataValue::Bool(b) => (None, None, None, Some(*b)),
+                                chroma_types::MetadataValue::SparseVector(_) => continue, // Not supported


[Logic] This creates a consistency issue where SparseVector metadata is present in the returned Collection object (cloned from request) but silently dropped from persistence. Subsequent reads will return incomplete metadata.

If SparseVector is not supported, return an error instead of silently ignoring it.

Context for Agents

This creates a consistency issue where `SparseVector` metadata is present in the returned `Collection` object (cloned from request) but silently dropped from persistence. Subsequent reads will return incomplete metadata. If `SparseVector` is not supported, return an error instead of silently ignoring it. File: rust/rust-sysdb/src/spanner.rs Line: 530

rust/types/src/collection.rs

propel-code-bot · 2026-01-08T08:07:50Z

rust/rust-sysdb/src/spanner.rs

+                        }
+                    }
+
+                    let now_timestamp_us = chrono::Utc::now().timestamp_micros();


[Logic] Using chrono::Utc::now() for created_at and updated_at relies on the client's clock and is inconsistent with create_tenant/create_database which use PENDING_COMMIT_TIMESTAMP(). This could lead to clock skew issues or inconsistencies in updated_at ordering.

Consider using google_cloud_spanner::value::CommitTimestamp::new() which serializes to spanner.commit_timestamp() to ensure causal consistency.

Context for Agents

Using `chrono::Utc::now()` for `created_at` and `updated_at` relies on the client's clock and is inconsistent with `create_tenant`/`create_database` which use `PENDING_COMMIT_TIMESTAMP()`. This could lead to clock skew issues or inconsistencies in `updated_at` ordering. Consider using `google_cloud_spanner::value::CommitTimestamp::new()` which serializes to `spanner.commit_timestamp()` to ensure causal consistency. File: rust/rust-sysdb/src/spanner.rs Line: 428

rust/rust-sysdb/src/backend.rs

tanujnay112 · 2026-01-08T23:52:24Z

rust/types/src/collection.rs

+            version: version.map(|v| v as i32).unwrap_or(0),
+            total_records_post_compaction: total_records_post_compaction
+                .map(|v| v as u64)
+                .unwrap_or(0),


shouldn't we expect this field to be Some(x) all the time?

should we silently default this to 0?

rust/rust-sysdb/src/spanner.rs

tanujnay112 · 2026-01-09T00:10:27Z

rust/rust-sysdb/src/spanner.rs

+                    }
+
+                    // Check if collection with same name exists in this database
+                    let mut check_stmt = Statement::new(


i feel like you can merge this check with the one on 418 no?

You mean with the id check?

propel-code-bot · 2026-01-09T22:21:48Z

rust/types/src/sysdb_errors.rs

+            SysDbError::SchemaMissing(_) => ErrorCodes::Internal,
+            SysDbError::InvalidSchemaJson(_) => ErrorCodes::Internal,
+            SysDbError::InvalidSegment(e) => e.code(),
+            SysDbError::InvalidSegmentsCount => ErrorCodes::Internal,
+            SysDbError::InvalidMetadata(e) => e.code(),
+            SysDbError::InvalidDimension(_) => ErrorCodes::Internal,


[Logic] Mapping validation errors (SchemaMissing, InvalidSchemaJson, InvalidDimension, InvalidSegmentsCount) to ErrorCodes::Internal will cause the API to return 500 errors for bad user input.

Map these to ErrorCodes::InvalidArgument to ensure correct 400-level HTTP responses.

Suggested change

SysDbError::SchemaMissing(_) => ErrorCodes::Internal,

SysDbError::InvalidSchemaJson(_) => ErrorCodes::Internal,

SysDbError::InvalidSegment(e) => e.code(),

SysDbError::InvalidSegmentsCount => ErrorCodes::Internal,

SysDbError::InvalidMetadata(e) => e.code(),

SysDbError::InvalidDimension(_) => ErrorCodes::Internal,

SysDbError::SchemaMissing(_) => ErrorCodes::InvalidArgument,

SysDbError::InvalidSchemaJson(_) => ErrorCodes::InvalidArgument,

SysDbError::InvalidSegment(e) => e.code(),

SysDbError::InvalidSegmentsCount => ErrorCodes::InvalidArgument,

SysDbError::InvalidMetadata(e) => e.code(),

SysDbError::InvalidDimension(_) => ErrorCodes::InvalidArgument,

Context for Agents

Mapping validation errors (`SchemaMissing`, `InvalidSchemaJson`, `InvalidDimension`, `InvalidSegmentsCount`) to `ErrorCodes::Internal` will cause the API to return 500 errors for bad user input. Map these to `ErrorCodes::InvalidArgument` to ensure correct 400-level HTTP responses. ```suggestion SysDbError::SchemaMissing(_) => ErrorCodes::InvalidArgument, SysDbError::InvalidSchemaJson(_) => ErrorCodes::InvalidArgument, SysDbError::InvalidSegment(e) => e.code(), SysDbError::InvalidSegmentsCount => ErrorCodes::InvalidArgument, SysDbError::InvalidMetadata(e) => e.code(), SysDbError::InvalidDimension(_) => ErrorCodes::InvalidArgument, ``` File: rust/types/src/sysdb_errors.rs Line: 96

rust/types/src/collection.rs

rust/rust-sysdb/src/types.rs

propel-code-bot · 2026-01-13T17:27:42Z

rust/rust-sysdb/src/types.rs

+            SysDbError::SchemaMissing(_) => ErrorCodes::Internal,
+            SysDbError::InvalidSchemaJson(_) => ErrorCodes::Internal,
+            SysDbError::InvalidSegment(e) => e.code(),
+            SysDbError::InvalidSegmentsCount => ErrorCodes::Internal,
+            SysDbError::InvalidMetadata(e) => e.code(),
+            SysDbError::InvalidDimension(_) => ErrorCodes::Internal,


[Logic] Validation errors caused by invalid user input (SchemaMissing, InvalidSchemaJson, InvalidSegmentsCount, InvalidDimension) should map to ErrorCodes::InvalidArgument (400) rather than ErrorCodes::Internal (500). This ensures the client receives the correct status code for bad requests.

Context for Agents

Validation errors caused by invalid user input (`SchemaMissing`, `InvalidSchemaJson`, `InvalidSegmentsCount`, `InvalidDimension`) should map to `ErrorCodes::InvalidArgument` (400) rather than `ErrorCodes::Internal` (500). This ensures the client receives the correct status code for bad requests. File: rust/rust-sysdb/src/types.rs Line: 688

This was referenced Jan 8, 2026

[ENH]: Database and Tenant Crud enhancements #6119

Merged

[ENH]: Spanner collection and segments Schemas #6123

Merged

sanketkedia requested a review from tanujnay112 January 8, 2026 07:55

sanketkedia marked this pull request as ready for review January 8, 2026 07:55

propel-code-bot bot reviewed Jan 8, 2026

View reviewed changes

rust/types/src/collection.rs Outdated Show resolved Hide resolved

propel-code-bot bot reviewed Jan 8, 2026

View reviewed changes

tanujnay112 reviewed Jan 9, 2026

View reviewed changes

sanketkedia force-pushed the 01-07-_enh_create_collection_impl_in_rust_sysdb branch 2 times, most recently from a5e3758 to 621573e Compare January 9, 2026 21:17

sanketkedia force-pushed the 01-07-_enh_spanner_collection_and_segments_schemas branch from 8943cfb to 118bdf5 Compare January 9, 2026 21:25

sanketkedia force-pushed the 01-07-_enh_create_collection_impl_in_rust_sysdb branch from 621573e to 9e6f199 Compare January 9, 2026 21:25

sanketkedia force-pushed the 01-07-_enh_spanner_collection_and_segments_schemas branch from 118bdf5 to 432297c Compare January 9, 2026 22:19

sanketkedia force-pushed the 01-07-_enh_create_collection_impl_in_rust_sysdb branch from 9e6f199 to 1b859f3 Compare January 9, 2026 22:19

propel-code-bot bot reviewed Jan 9, 2026

View reviewed changes

rust/types/src/collection.rs Outdated Show resolved Hide resolved

sanketkedia force-pushed the 01-07-_enh_spanner_collection_and_segments_schemas branch from 432297c to 50a6033 Compare January 9, 2026 23:02

sanketkedia force-pushed the 01-07-_enh_create_collection_impl_in_rust_sysdb branch 2 times, most recently from 83936fc to 0197b36 Compare January 9, 2026 23:35

This was referenced Jan 11, 2026

[ENH]: Get collections impl #6146

Merged

[ENH]: Get Collection with segments impl #6147

Merged

sanketkedia force-pushed the 01-07-_enh_create_collection_impl_in_rust_sysdb branch from 0197b36 to ecca61f Compare January 11, 2026 21:25

sanketkedia force-pushed the 01-07-_enh_spanner_collection_and_segments_schemas branch from 2398ba0 to 3890bf5 Compare January 12, 2026 15:49

sanketkedia force-pushed the 01-07-_enh_create_collection_impl_in_rust_sysdb branch 3 times, most recently from 272af49 to 545ac09 Compare January 13, 2026 02:20

sanketkedia requested a review from rescrv as a code owner January 13, 2026 02:20

sanketkedia changed the base branch from 01-07-_enh_spanner_collection_and_segments_schemas to main January 13, 2026 02:20

sanketkedia force-pushed the 01-07-_enh_create_collection_impl_in_rust_sysdb branch from 545ac09 to b42abb8 Compare January 13, 2026 17:20

propel-code-bot bot reviewed Jan 13, 2026

View reviewed changes

rust/rust-sysdb/src/types.rs Show resolved Hide resolved

propel-code-bot bot reviewed Jan 13, 2026

View reviewed changes

sanketkedia added 8 commits January 13, 2026 15:40

[ENH]: Create collection impl in rust sysdb

0b9c8e0

Add more tests

be3518e

Fix test

82a379f

Create collection server impl

cf74265

more robust tests

ddd26ac

Fix merge conflicts

4e419c3

remove extraneous residual

852c0bc

review comments

9723f17

sanketkedia force-pushed the 01-07-_enh_create_collection_impl_in_rust_sysdb branch from 448dadd to 9723f17 Compare January 13, 2026 23:40

tanujnay112 approved these changes Jan 14, 2026

View reviewed changes

sanketkedia merged commit ddb54af into main Jan 14, 2026
128 of 130 checks passed

Conversation

sanketkedia commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

Test plan

Migration plan

Observability plan

Documentation Changes

Uh oh!

github-actions bot commented Jan 8, 2026

Reviewer Checklist

Testing, Bugs, Errors, Logs, Documentation

System Compatibility

Quality

Uh oh!

sanketkedia commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

propel-code-bot bot commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

propel-code-bot bot Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

propel-code-bot bot Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tanujnay112 Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tanujnay112 Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

sanketkedia Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

tanujnay112 Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

propel-code-bot bot Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

propel-code-bot bot Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sanketkedia commented Jan 8, 2026 •

edited

Loading

sanketkedia commented Jan 8, 2026 •

edited

Loading

propel-code-bot bot commented Jan 8, 2026 •

edited

Loading