Skip to content

[ENH]: create new vector segment type based on config#6296

Merged
sanketkedia merged 3 commits intomainfrom
01-30-_enh_create_new_vector_segment_type_based_on_config
Feb 3, 2026
Merged

[ENH]: create new vector segment type based on config#6296
sanketkedia merged 3 commits intomainfrom
01-30-_enh_create_new_vector_segment_type_based_on_config

Conversation

@sanketkedia
Copy link
Copy Markdown
Contributor

@sanketkedia sanketkedia commented Jan 31, 2026

Description of changes

Summarize the changes made by this PR.

  • Improvements & Bug fixes
    • Sets a new vector segment type based on the field introduced in PR#6295
    • A subsequent PR will construct the appropriate writer instance based on this
  • New functionality
    • ...

Test plan

How are these changes tested?

  • Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Migration plan

None

Observability plan

None

Documentation Changes

None

Copy link
Copy Markdown
Contributor Author

sanketkedia commented Jan 31, 2026

@github-actions
Copy link
Copy Markdown

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

  • Can you think of any use case in which the code does not behave as intended? Have they been tested?
  • Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
  • If appropriate, are there adequate property based tests?
  • If appropriate, are there adequate unit tests?
  • Should any logging, debugging, tracing information be added or removed?
  • Are error messages user-friendly?
  • Have all documentation changes needed been made?
  • Have all non-obvious changes been commented?

System Compatibility

  • Are there any potential impacts on other parts of the system or backward compatibility?
  • Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

  • Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

@sanketkedia sanketkedia marked this pull request as ready for review January 31, 2026 22:03
@propel-code-bot
Copy link
Copy Markdown
Contributor

propel-code-bot bot commented Jan 31, 2026

Add QuantizedSpann segment plumbing and schema awareness

Introduces the QuantizedSpann segment type throughout the distributed stack so collections whose schema enables SPANN quantization can be reflected as a distinct vector segment. The change wires the new enum value through type conversion, frontend validation, segment reader/writer guards, and schema helpers, laying groundwork for instantiating a dedicated quantized writer in a follow-up PR.

Key Changes

• Extended SegmentType with QuantizedSpann, including From/TryFrom conversions and URN mapping in rust/types/src/segment.rs.
• Updated Segment::prefetch_supported/filepaths_to_prefetch with TODO markers for future quantized support while keeping current behavior unchanged.
• Allowed the SPANN writer/reader constructors in rust/segment/src/distributed_spann.rs to accept either Spann or QuantizedSpann, and documented the forthcoming dedicated writer instantiation.
• Enhanced ServiceBasedFrontend to choose SegmentType::QuantizedSpann when schema-driven SPANN configurations enable quantization, and relaxed capability checks and executor supported-segment lists to include the new type.
• Added Schema::is_quantization_enabled helper so callers can introspect SPANN settings without duplicating traversal logic.

Affected Areas

rust/types/src/segment.rs
rust/types/src/collection_schema.rs
rust/segment/src/distributed_spann.rs
rust/frontend/src/impls/service_based_frontend.rs
rust/frontend/src/executor/distributed.rs

This summary was automatically generated by @propel-code-bot

Comment on lines 539 to +546
if self.enable_schema {
if let Some(schema) = reconciled_schema.as_ref() {
if schema.get_internal_spann_config().is_some() {
vector_segment_type = SegmentType::Spann;
// Use QuantizedSpann if quantization is enabled, otherwise use Spann
if schema.is_quantization_enabled() {
vector_segment_type = SegmentType::QuantizedSpann;
} else {
vector_segment_type = SegmentType::Spann;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

[Logic] The new quantization branch only runs when enable_schema is true, so deployments that still run without schema support will never produce a QuantizedSpann segment even if the user explicitly sets quantize = true in the collection configuration. In that mode we still fall through to the legacy if matches!(config.vector_index, VectorIndexConfiguration::Spann(_)) { vector_segment_type = SegmentType::Spann; }, silently dropping the quantization flag and creating a plain Spann segment. That makes quantization impossible in non-schema environments.

To keep the behavior consistent, mirror this check for the configuration-only path and switch to SegmentType::QuantizedSpann whenever InternalSpannConfiguration.quantize is true:

Suggested change
if self.enable_schema {
if let Some(schema) = reconciled_schema.as_ref() {
if schema.get_internal_spann_config().is_some() {
vector_segment_type = SegmentType::Spann;
// Use QuantizedSpann if quantization is enabled, otherwise use Spann
if schema.is_quantization_enabled() {
vector_segment_type = SegmentType::QuantizedSpann;
} else {
vector_segment_type = SegmentType::Spann;
if schema.get_internal_spann_config().is_some() {
// Use QuantizedSpann if quantization is enabled, otherwise use Spann
if schema.is_quantization_enabled() {
vector_segment_type = SegmentType::QuantizedSpann;
} else {
vector_segment_type = SegmentType::Spann;
}
}
}
}
if let Some(config) = configuration.as_ref() {
if let VectorIndexConfiguration::Spann(ref spann_cfg) = config.vector_index {
vector_segment_type = if spann_cfg.quantize {
SegmentType::QuantizedSpann
} else {
SegmentType::Spann
};
}
}

(Adjust the existing branch below so it inspects spann_cfg.quantize instead of hard-coding SegmentType::Spann.)

Context for Agents
The new quantization branch only runs when `enable_schema` is true, so deployments that still run without schema support will never produce a `QuantizedSpann` segment even if the user explicitly sets `quantize = true` in the collection configuration. In that mode we still fall through to the legacy `if matches!(config.vector_index, VectorIndexConfiguration::Spann(_)) { vector_segment_type = SegmentType::Spann; }`, silently dropping the quantization flag and creating a plain Spann segment. That makes quantization impossible in non-schema environments.

To keep the behavior consistent, mirror this check for the configuration-only path and switch to `SegmentType::QuantizedSpann` whenever `InternalSpannConfiguration.quantize` is true:

```suggestion
                        if schema.get_internal_spann_config().is_some() {
                            // Use QuantizedSpann if quantization is enabled, otherwise use Spann
                            if schema.is_quantization_enabled() {
                                vector_segment_type = SegmentType::QuantizedSpann;
                            } else {
                                vector_segment_type = SegmentType::Spann;
                            }
                        }
                    }
                }
                if let Some(config) = configuration.as_ref() {
                    if let VectorIndexConfiguration::Spann(ref spann_cfg) = config.vector_index {
                        vector_segment_type = if spann_cfg.quantize {
                            SegmentType::QuantizedSpann
                        } else {
                            SegmentType::Spann
                        };
                    }
                }
```

(Adjust the existing branch below so it inspects `spann_cfg.quantize` instead of hard-coding `SegmentType::Spann`.)

File: rust/frontend/src/impls/service_based_frontend.rs
Line: 546

@sanketkedia sanketkedia force-pushed the 01-30-_enh_create_new_vector_segment_type_based_on_config branch from 0dfe648 to afa75d2 Compare January 31, 2026 22:13
impl Segment {
// TODO(Sanket): Add QuantizedSpann to the prefetch supported list when
// we have intelligent prefetching.
pub fn prefetch_supported(&self) -> bool {
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add this

}

// TODO(Sanket): Add file paths for QuantizedSpann when we have intelligent prefetching.
pub fn filepaths_to_prefetch(&self) -> Vec<String> {
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add QuantizedSpann here

@sanketkedia sanketkedia changed the base branch from 01-29-_enh_add_option_to_enable_quantization_in_schema to graphite-base/6296 February 3, 2026 02:15
@sanketkedia sanketkedia force-pushed the 01-30-_enh_create_new_vector_segment_type_based_on_config branch from afa75d2 to cb0819a Compare February 3, 2026 02:16
@graphite-app graphite-app bot changed the base branch from graphite-base/6296 to main February 3, 2026 02:16
@sanketkedia sanketkedia force-pushed the 01-30-_enh_create_new_vector_segment_type_based_on_config branch from cb0819a to 6a1d588 Compare February 3, 2026 02:16
@sanketkedia sanketkedia merged commit 45b01a9 into main Feb 3, 2026
68 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants