Skip to content

Commit ff9c3f4

Browse files
committed
fix(meta): compare partition bounds with the scan comparison coercion
The preflight validator cast the persisted bound directly to the altered column type, which rejected ALTERs the scan itself accepts: the region coerces both the column and the bound to the common comparison type (ExprSimplifier::coerce in PredicateGroup::new), so narrowing a field column with an out-of-range bound still scans. Compare through comparison_coercion(target_type, bound_type) instead and only require the bound to be castable to the common type. Also add a round-trip test in the partition crate that serializes real PartitionExprs and runs them through the validator, so drift between the Persisted* mirror and the partition types fails CI, and fix the error message to match the comparison semantics. Signed-off-by: fengyue-xve <[email protected]>
1 parent a6ed555 commit ff9c3f4

5 files changed

Lines changed: 174 additions & 17 deletions

File tree

‎src/common/meta/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ common-time.workspace = true
4646
common-wal.workspace = true
4747
common-workload.workspace = true
4848
datafusion-common.workspace = true
49+
datafusion-expr.workspace = true
4950
datatypes.workspace = true
5051
deadpool = { workspace = true, optional = true }
5152
deadpool-postgres = { workspace = true, optional = true }

‎src/common/meta/src/ddl/alter_table.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ mod metadata;
1717
mod partition_compat;
1818
mod region_request;
1919

20+
pub use partition_compat::validate_region_partition_bounds;
21+
2022
use std::collections::HashSet;
2123
use std::vec;
2224

‎src/common/meta/src/ddl/alter_table/partition_compat.rs‎

Lines changed: 69 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -15,27 +15,32 @@
1515
//! Preflight validation of persisted partition expressions against an ALTER TABLE schema change.
1616
//!
1717
//! Persisted partition bounds keep the types they were written with. When a column
18-
//! referenced by a partition expression changes its type, the region rewrites the bound
19-
//! literal to the new column type while building the partition predicate on scan. If a
20-
//! bound cannot be represented with the new type (a classic example is a millisecond
21-
//! timestamp bound that overflows when the time index column is widened to nanoseconds),
22-
//! every scan of that region fails at that point. Such an ALTER must be rejected before
23-
//! it is submitted to any region, otherwise already-altered regions keep failing while
24-
//! the rest of the table is untouched.
18+
//! referenced by a partition expression changes its type, the region coerces the bound
19+
//! literal and the new column type to a common comparison type while building the
20+
//! partition predicate on scan. If no common type exists, or the bound cannot be cast
21+
//! to it (a classic example is a millisecond timestamp bound that overflows when the
22+
//! time index column is widened to nanoseconds), every scan of that region fails at
23+
//! that point. Such an ALTER must be rejected before it is submitted to any region,
24+
//! otherwise already-altered regions keep failing while the rest of the table is
25+
//! untouched.
2526
//!
2627
//! The `partition` crate depends on `common-meta`, so this crate cannot depend on
2728
//! `partition` back and cannot reuse [`partition::expr::PartitionExpr`]. The `Persisted*`
2829
//! types below mirror the JSON representation of `partition::partition::PartitionBound` and
2930
//! `partition::expr::PartitionExpr` and must be kept in sync with `partition/src/partition.rs`
30-
//! and `partition/src/expr.rs`. The compatibility check itself deliberately repeats the
31-
//! same conversion (value -> [`ScalarValue`]) and the same checked cast
32-
//! ([`ScalarValue::cast_to`]) that the region performs when it coerces the persisted bound
33-
//! to the new column type, so a bound that would fail on scan is rejected here as well.
31+
//! and `partition/src/expr.rs`; round-trip tests in `partition` run real serialized
32+
//! expressions through this validator so drift fails in CI. The compatibility check itself
33+
//! deliberately repeats the same conversion (value -> [`ScalarValue`]) and the same
34+
//! comparison coercion the region performs when it builds the partition predicate
35+
//! (`ExprSimplifier::coerce` / [`comparison_coercion`]): both sides are cast to the common
36+
//! comparison type, and the persisted bound must be castable to that type with
37+
//! [`ScalarValue::cast_to`]. A bound that would fail on scan is rejected here as well.
3438
3539
use std::collections::HashMap;
3640

3741
use common_telemetry::warn;
3842
use datafusion_common::ScalarValue;
43+
use datafusion_expr::type_coercion::binary::comparison_coercion;
3944
use datatypes::data_type::{ConcreteDataType, DataType};
4045
use datatypes::schema::Schema;
4146
use datatypes::value::{
@@ -111,7 +116,10 @@ pub(crate) fn changed_column_types(
111116
/// Validates that the persisted partition bounds of every region are still representable
112117
/// with the types of `changed_columns`. Returns an error naming the first offending
113118
/// region without modifying anything.
114-
pub(crate) fn validate_region_partition_bounds(
119+
///
120+
/// Public so that round-trip tests in the `partition` crate can run real serialized
121+
/// partition expressions through the same mirror used here.
122+
pub fn validate_region_partition_bounds(
115123
changed_columns: &HashMap<String, ConcreteDataType>,
116124
region_routes: &[RegionRoute],
117125
) -> Result<()> {
@@ -205,11 +213,29 @@ fn validate_comparison(
205213
return Ok(());
206214
};
207215

216+
// Mirror `ExprSimplifier::coerce` in `PredicateGroup::new`: the region casts
217+
// both the column and the persisted bound to the common comparison type. The
218+
// bound is not cast to the column type itself, so narrowing a column (e.g.
219+
// BIGINT to INT with an out-of-range bound) is fine as long as the bound
220+
// survives the common type, while a bound that cannot be cast to it — or a
221+
// pair of types without a common comparison type — fails every scan.
208222
let target_arrow_type = target_type.as_arrow_type();
209-
if scalar.data_type() == target_arrow_type {
210-
return Ok(());
211-
}
212-
if let Err(error) = scalar.cast_to(&target_arrow_type) {
223+
let Some(common_type) = comparison_coercion(&target_arrow_type, &scalar.data_type()) else {
224+
warn!(
225+
"Persisted partition bound has no common comparison type with the altered \
226+
column type, region_id: {}, column: '{}', target_type: {}",
227+
region_id, column, target_type
228+
);
229+
return PartitionExprIncompatibleSnafu {
230+
region_id,
231+
column: column.clone(),
232+
target_type: target_type.to_string(),
233+
}
234+
.fail();
235+
};
236+
if scalar.data_type() != common_type
237+
&& let Err(error) = scalar.cast_to(&common_type)
238+
{
213239
warn!(
214240
error;
215241
"Persisted partition bound is not representable with the altered column type, \
@@ -408,4 +434,31 @@ mod tests {
408434
)
409435
.unwrap();
410436
}
437+
438+
#[test]
439+
fn test_field_narrowing_keeps_bounds_the_scan_coerces() {
440+
// BIGINT bound outside the INT range with the column narrowed to INT: the
441+
// region casts the column up to the common Int64 comparison type, so the
442+
// bound still scans and the ALTER must not be rejected.
443+
let bound =
444+
r#"{"Expr":{"lhs":{"Column":"v"},"op":"Lt","rhs":{"Value":{"Int64":3000000000}}}}"#;
445+
validate_region_partition_bounds(
446+
&changed("v", ConcreteDataType::int32_datatype()),
447+
&routes(&[bound]),
448+
)
449+
.unwrap();
450+
}
451+
452+
#[test]
453+
fn test_bound_without_common_comparison_type_is_rejected() {
454+
// No comparison coercion exists between a boolean bound and an int column,
455+
// so the region's predicate build fails and the ALTER must be rejected.
456+
let bound = r#"{"Expr":{"lhs":{"Column":"v"},"op":"Lt","rhs":{"Value":{"Boolean":true}}}}"#;
457+
let err = validate_region_partition_bounds(
458+
&changed("v", ConcreteDataType::int32_datatype()),
459+
&routes(&[bound]),
460+
)
461+
.unwrap_err();
462+
assert!(err.to_string().contains("'v'"), "unexpected error: {err}");
463+
}
411464
}

‎src/common/meta/src/error.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1143,7 +1143,7 @@ pub enum Error {
11431143
},
11441144

11451145
#[snafu(display(
1146-
"Partition expression of region {} is incompatible with the altered column '{}': the persisted bound cannot be converted to type {}",
1146+
"Partition expression of region {} is incompatible with the altered column '{}': the persisted bound can no longer be compared with the new column type {}",
11471147
region_id,
11481148
column,
11491149
target_type

‎src/partition/src/expr.rs‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -463,6 +463,14 @@ impl Display for PartitionExpr {
463463

464464
#[cfg(test)]
465465
mod tests {
466+
use std::collections::HashMap;
467+
468+
use common_meta::ddl::alter_table::validate_region_partition_bounds;
469+
use common_meta::error::Error;
470+
use common_meta::rpc::router::{Region, RegionRoute};
471+
use datatypes::data_type::ConcreteDataType;
472+
use store_api::storage::RegionId;
473+
466474
use super::*;
467475

468476
#[test]
@@ -863,4 +871,97 @@ mod tests {
863871
assert!(columns.contains("b"));
864872
assert!(columns.contains("c"));
865873
}
874+
875+
/// Real serialized expressions must survive the round trip through
876+
/// `common-meta`'s persisted-expression mirror: whenever that mirror drifts
877+
/// from these types, deserialization fails, validation is skipped and the
878+
/// overflowing cases below stop producing `PartitionExprIncompatible`.
879+
#[test]
880+
fn test_serialized_expressions_validate_against_type_changes() {
881+
let overflow_ms = serde_json::from_str::<Value>(
882+
r#"{"Timestamp":{"value":32503680000000,"unit":"Millisecond"}}"#,
883+
)
884+
.unwrap();
885+
let start_ms =
886+
serde_json::from_str::<Value>(r#"{"Timestamp":{"value":0,"unit":"Millisecond"}}"#)
887+
.unwrap();
888+
let ts_ns = ConcreteDataType::timestamp_nanosecond_datatype();
889+
890+
let routes = |expr: &PartitionExpr| {
891+
vec![RegionRoute {
892+
region: Region {
893+
id: RegionId::new(1024, 1),
894+
partition_expr: expr.as_json_str().unwrap(),
895+
..Default::default()
896+
},
897+
..Default::default()
898+
}]
899+
};
900+
let changed = |column: &str, data_type: ConcreteDataType| {
901+
HashMap::from([(column.to_string(), data_type)])
902+
};
903+
904+
// Compile-time tripwire: exhaustively matching the operand and operator
905+
// variants makes a new variant break this test at compile time, so the
906+
// `partition_compat::Persisted*` mirror and these cases must be extended
907+
// together instead of silently skipping validation.
908+
fn walk(expr: &PartitionExpr) {
909+
walk_operand(&expr.lhs);
910+
walk_operand(&expr.rhs);
911+
match expr.op {
912+
RestrictedOp::Eq
913+
| RestrictedOp::NotEq
914+
| RestrictedOp::Lt
915+
| RestrictedOp::LtEq
916+
| RestrictedOp::Gt
917+
| RestrictedOp::GtEq
918+
| RestrictedOp::And
919+
| RestrictedOp::Or => {}
920+
}
921+
}
922+
fn walk_operand(operand: &Operand) {
923+
match operand {
924+
Operand::Column(_) | Operand::Value(_) => {}
925+
Operand::Expr(expr) => walk(expr),
926+
}
927+
}
928+
929+
// Every expression shape must be rejected while the bound overflows
930+
// nanoseconds, with the error naming the offending column.
931+
let simple = col("ts").lt(overflow_ms.clone());
932+
let nested_and = col("ts")
933+
.gt_eq(start_ms.clone())
934+
.and(col("ts").lt(overflow_ms.clone()));
935+
let nested_or = PartitionExpr::new(
936+
Operand::Expr(col("host").eq(Value::String("a".into()))),
937+
RestrictedOp::Or,
938+
Operand::Expr(col("ts").lt(overflow_ms.clone())),
939+
);
940+
for expr in [&simple, &nested_and, &nested_or] {
941+
walk(expr);
942+
let err =
943+
validate_region_partition_bounds(&changed("ts", ts_ns.clone()), &routes(expr))
944+
.unwrap_err();
945+
assert!(
946+
matches!(
947+
err,
948+
Error::PartitionExprIncompatible { ref column, .. } if column == "ts"
949+
),
950+
"unexpected error: {err}"
951+
);
952+
}
953+
954+
// A representable bound passes.
955+
let compatible = col("ts").lt(start_ms);
956+
validate_region_partition_bounds(&changed("ts", ts_ns), &routes(&compatible)).unwrap();
957+
958+
// Narrowing a field column must not be over-rejected: the scan casts the
959+
// column up to the common Int64 type instead of the bound down to Int32.
960+
let narrowing = col("v").lt(Value::Int64(3_000_000_000));
961+
validate_region_partition_bounds(
962+
&changed("v", ConcreteDataType::int32_datatype()),
963+
&routes(&narrowing),
964+
)
965+
.unwrap();
966+
}
866967
}

0 commit comments

Comments
 (0)