[ENH]: create new vector segment type based on config#6296
Conversation
This stack of pull requests is managed by Graphite. Learn more about stacking. |
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
|
Add Introduces the Key Changes• Extended Affected Areas• This summary was automatically generated by @propel-code-bot |
| if self.enable_schema { | ||
| if let Some(schema) = reconciled_schema.as_ref() { | ||
| if schema.get_internal_spann_config().is_some() { | ||
| vector_segment_type = SegmentType::Spann; | ||
| // Use QuantizedSpann if quantization is enabled, otherwise use Spann | ||
| if schema.is_quantization_enabled() { | ||
| vector_segment_type = SegmentType::QuantizedSpann; | ||
| } else { | ||
| vector_segment_type = SegmentType::Spann; |
There was a problem hiding this comment.
[Logic] The new quantization branch only runs when enable_schema is true, so deployments that still run without schema support will never produce a QuantizedSpann segment even if the user explicitly sets quantize = true in the collection configuration. In that mode we still fall through to the legacy if matches!(config.vector_index, VectorIndexConfiguration::Spann(_)) { vector_segment_type = SegmentType::Spann; }, silently dropping the quantization flag and creating a plain Spann segment. That makes quantization impossible in non-schema environments.
To keep the behavior consistent, mirror this check for the configuration-only path and switch to SegmentType::QuantizedSpann whenever InternalSpannConfiguration.quantize is true:
| if self.enable_schema { | |
| if let Some(schema) = reconciled_schema.as_ref() { | |
| if schema.get_internal_spann_config().is_some() { | |
| vector_segment_type = SegmentType::Spann; | |
| // Use QuantizedSpann if quantization is enabled, otherwise use Spann | |
| if schema.is_quantization_enabled() { | |
| vector_segment_type = SegmentType::QuantizedSpann; | |
| } else { | |
| vector_segment_type = SegmentType::Spann; | |
| if schema.get_internal_spann_config().is_some() { | |
| // Use QuantizedSpann if quantization is enabled, otherwise use Spann | |
| if schema.is_quantization_enabled() { | |
| vector_segment_type = SegmentType::QuantizedSpann; | |
| } else { | |
| vector_segment_type = SegmentType::Spann; | |
| } | |
| } | |
| } | |
| } | |
| if let Some(config) = configuration.as_ref() { | |
| if let VectorIndexConfiguration::Spann(ref spann_cfg) = config.vector_index { | |
| vector_segment_type = if spann_cfg.quantize { | |
| SegmentType::QuantizedSpann | |
| } else { | |
| SegmentType::Spann | |
| }; | |
| } | |
| } |
(Adjust the existing branch below so it inspects spann_cfg.quantize instead of hard-coding SegmentType::Spann.)
Context for Agents
The new quantization branch only runs when `enable_schema` is true, so deployments that still run without schema support will never produce a `QuantizedSpann` segment even if the user explicitly sets `quantize = true` in the collection configuration. In that mode we still fall through to the legacy `if matches!(config.vector_index, VectorIndexConfiguration::Spann(_)) { vector_segment_type = SegmentType::Spann; }`, silently dropping the quantization flag and creating a plain Spann segment. That makes quantization impossible in non-schema environments.
To keep the behavior consistent, mirror this check for the configuration-only path and switch to `SegmentType::QuantizedSpann` whenever `InternalSpannConfiguration.quantize` is true:
```suggestion
if schema.get_internal_spann_config().is_some() {
// Use QuantizedSpann if quantization is enabled, otherwise use Spann
if schema.is_quantization_enabled() {
vector_segment_type = SegmentType::QuantizedSpann;
} else {
vector_segment_type = SegmentType::Spann;
}
}
}
}
if let Some(config) = configuration.as_ref() {
if let VectorIndexConfiguration::Spann(ref spann_cfg) = config.vector_index {
vector_segment_type = if spann_cfg.quantize {
SegmentType::QuantizedSpann
} else {
SegmentType::Spann
};
}
}
```
(Adjust the existing branch below so it inspects `spann_cfg.quantize` instead of hard-coding `SegmentType::Spann`.)
File: rust/frontend/src/impls/service_based_frontend.rs
Line: 5460dfe648 to
afa75d2
Compare
| impl Segment { | ||
| // TODO(Sanket): Add QuantizedSpann to the prefetch supported list when | ||
| // we have intelligent prefetching. | ||
| pub fn prefetch_supported(&self) -> bool { |
| } | ||
|
|
||
| // TODO(Sanket): Add file paths for QuantizedSpann when we have intelligent prefetching. | ||
| pub fn filepaths_to_prefetch(&self) -> Vec<String> { |
There was a problem hiding this comment.
add QuantizedSpann here
e41ce4d to
bd295a1
Compare
afa75d2 to
cb0819a
Compare
cb0819a to
6a1d588
Compare

Description of changes
Summarize the changes made by this PR.
Test plan
How are these changes tested?
pytestfor python,yarn testfor js,cargo testfor rustMigration plan
None
Observability plan
None
Documentation Changes
None