Skip to content

Commit 04e1b76

Browse files
morningmanwyxxxcat
authored andcommitted
[fix](arrow) correct the Flight SQL GetTables schema and the TIMESTAMPTZ arrow reader (apache#66344)
### What problem does this PR solve? Related Issue: apache#65615 Problem Summary: Two places where Doris says one thing about an Arrow value and does another. Both were found while reading Doris over Arrow Flight SQL, and both reproduce on master on their own. **1. `CommandGetTables` describes types the batches do not carry** `FlightSqlSchemaHelper.getArrowType` is documented as mirroring `convert_to_arrow_type` in the backend -- the schema a client reads from `GetTables` is what it types its columns by. Two of the mappings had drifted from what BE actually writes: - A `DATEV2` column was described as `Date(MILLISECOND)` -- Arrow date64 -- while BE writes `arrow::Date32Type`, a day number. A client that trusts the schema renders and compares the column as a datetime and then fails on the first batch with `not support convert to datetimev2 from arrow type: 16`. - A complex column was described with a placeholder child: an array's element type was the Null type, a map's pair was a bare list, a struct had no fields at all. An Arrow ARRAY/MAP/STRUCT carries its element types in its children and nowhere else, so this said "array of nothing" about every array in the catalog while BE emitted `ListType(item)`, `MapType(key, value)` and `StructType(fields)` in the data. `describeTables` already reports the whole tree -- `Column.createChildrenColumn` even names an array's element `item` and a map's pair `key`/`value`, which is what Arrow calls them -- so the children are now built from it, recursively. A descriptor that reports no children keeps the old placeholders: a source that cannot describe its nested types is no worse off than before. **2. A TIMESTAMPTZ column read from Arrow was decoded by copying its bits** `DataTypeTimeStampTzSerDe` never overrode `read_column_from_arrow`, so the one it inherited from `DataTypeNumberSerDe<TYPE_TIMESTAMPTZ>` ran: its fixed-width path `memcpy`s the array's buffer straight into the column. Arrow hands it int64 EPOCH values; the column stores PACKED date/time values. Both are eight bytes wide, so the width check passes, the scan succeeds, and every row is silently wrong -- a Doris datetime read back over Arrow Flight SQL rendered as `+08:05` with no date in it. This is reachable today through the `remote_doris` catalog. Its schema is the remote table's `Column` objects deserialized verbatim (`RemoteDorisRestClient.parseColumns`), so a remote TIMESTAMPTZ column stays TIMESTAMPTZ locally; `RemoteDorisScanNode` plans the scan as `FORMAT_ARROW`; and `remote_doris_reader` materializes it through this serde. The remote side writes the epochs with the serde's own `write_column_to_arrow`, so both halves of that round trip are Doris code and the mismatch is entirely internal. The other Arrow readers -- `arrow_stream_reader` for the `arrow` file format, `python_udtf_function`, `paimon_cpp_reader` -- share the same serde and are covered by the same change; whether each of them can present a TIMESTAMPTZ column today is not something this PR claims either way. The reader converts by unit into the microseconds the column stores. NANO is divided rather than refused, since no Doris datetime type keeps sub-microsecond digits, and the division floors so a pre-1970 instant does not move forward by one microsecond. The value is read as an instant on the UTC line, the inverse of what `write_column_to_arrow` emits, so the round trip is exact. Two cases are deliberate. A null slot is not converted: the bytes under it are whatever the source left there, and running a garbage epoch through the range check would fail a batch whose rows are all well-formed. An Arrow type this serde cannot decode is an error rather than a fallback -- accepting anything eight bytes wide is how the corruption above stayed invisible.
1 parent 029e901 commit 04e1b76

5 files changed

Lines changed: 460 additions & 30 deletions

File tree

‎be/src/core/data_type_serde/data_type_timestamptz_serde.cpp‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,9 @@
2020
#include <arrow/builder.h>
2121
#include <cctz/time_zone.h>
2222

23+
#include "common/config.h"
2324
#include "core/data_type/primitive_type.h"
25+
#include "core/data_type_serde/arrow_validation.h"
2426
#include "core/data_type_serde/decoded_column_view.h"
2527
#include "core/data_type_serde/parquet_decode_source.h"
2628
#include "core/data_type_serde/parquet_timestamp.h"
@@ -357,6 +359,92 @@ Status DataTypeTimeStampTzSerDe::write_column_to_arrow(const IColumn& column,
357359
return Status::OK();
358360
}
359361

362+
/**
363+
* Reads an Arrow timestamp array into a TIMESTAMPTZ column.
364+
*
365+
* <p>Without this the base DataTypeNumberSerDe<TYPE_TIMESTAMPTZ> reader runs instead, and its
366+
* fixed-width path memcpy's the array's int64 epoch values straight into the column -- whose element
367+
* is a PACKED date/time value, not an epoch. Both are 8 bytes wide, so no check catches it: the scan
368+
* succeeds and every row is silently wrong. That is why an unreadable Arrow type below is an error
369+
* rather than a fallback.
370+
*
371+
* <p>The value is read as an instant on the UTC line, which is the inverse of what
372+
* write_column_to_arrow emits (it converts with cctz::utc_time_zone(), not with ctz). ctz is
373+
* therefore unused here: an Arrow timestamp's zone -- whether it names one or not -- describes how
374+
* to DISPLAY the instant, and TIMESTAMPTZ stores the instant itself.
375+
*/
376+
Status DataTypeTimeStampTzSerDe::read_column_from_arrow(IColumn& column,
377+
const arrow::Array* arrow_array,
378+
int64_t start, int64_t end,
379+
const cctz::time_zone& ctz) const {
380+
if (config::enable_arrow_input_validation) {
381+
check_arrow_no_offset(*arrow_array);
382+
}
383+
if (arrow_array->type()->id() != arrow::Type::TIMESTAMP) {
384+
LOG(WARNING) << "not support convert to timestamptz from arrow type:"
385+
<< arrow_array->type()->id();
386+
return Status::InternalError("not support convert to timestamptz from arrow type: {}",
387+
arrow_array->type()->id());
388+
}
389+
const auto* concrete_array = assert_cast<const arrow::TimestampArray*>(arrow_array);
390+
const auto type = std::static_pointer_cast<arrow::TimestampType>(arrow_array->type());
391+
// Scale each unit to the microseconds the column stores. NANO is divided rather than refused:
392+
// sub-microsecond precision is beyond what any Doris datetime type keeps, and rejecting the
393+
// column over a digit would make whole tables unreadable.
394+
int64_t multiplier = 1;
395+
int64_t divisor = 1;
396+
switch (type->unit()) {
397+
case arrow::TimeUnit::type::SECOND:
398+
multiplier = 1000000;
399+
break;
400+
case arrow::TimeUnit::type::MILLI:
401+
multiplier = 1000;
402+
break;
403+
case arrow::TimeUnit::type::MICRO:
404+
break;
405+
case arrow::TimeUnit::type::NANO:
406+
divisor = 1000;
407+
break;
408+
default:
409+
LOG(WARNING) << "not support convert to timestamptz from time_unit:" << type->unit();
410+
return Status::InvalidArgument("not support convert to timestamptz from time_unit: {}",
411+
type->unit());
412+
}
413+
414+
auto& col_data = assert_cast<ColumnTimeStampTz&>(column).get_data();
415+
const auto* base_ptr = reinterpret_cast<const uint8_t*>(concrete_array->raw_values());
416+
const size_t element_size = sizeof(int64_t);
417+
for (auto value_i = start; value_i < end; ++value_i) {
418+
// One value per row including the null ones: the caller (DataTypeNullableSerDe) has already
419+
// taken the validity bitmap and hands the whole range down. The value under a null slot is
420+
// whatever the source left there, so it must not be converted -- a garbage epoch would fail
421+
// the range check below and take a well-formed batch down with it.
422+
if (concrete_array->IsNull(value_i)) {
423+
col_data.push_back(TimestampTzValue());
424+
continue;
425+
}
426+
const uint8_t* raw_byte_ptr = base_ptr + value_i * element_size;
427+
auto value = unaligned_load<int64_t>(raw_byte_ptr);
428+
int64_t timestamp_micros = 0;
429+
if (__builtin_mul_overflow(value, multiplier, &timestamp_micros)) {
430+
return Status::DataQualityError(
431+
"Arrow timestamp {} in unit {} overflows the microsecond range of TIMESTAMPTZ",
432+
value, static_cast<int>(type->unit()));
433+
}
434+
if (divisor != 1) {
435+
// Floor, not truncate: C++ integer division rounds toward zero, which would move a
436+
// pre-1970 instant forward by up to one microsecond.
437+
int64_t remainder = timestamp_micros % divisor;
438+
timestamp_micros /= divisor;
439+
if (remainder < 0) {
440+
--timestamp_micros;
441+
}
442+
}
443+
RETURN_IF_ERROR(append_timestamptz_from_utc_epoch_micros(col_data, timestamp_micros));
444+
}
445+
return Status::OK();
446+
}
447+
360448
Status DataTypeTimeStampTzSerDe::write_column_to_orc(const std::string& timezone,
361449
const IColumn& column, const NullMap* null_map,
362450
orc::ColumnVectorBatch* orc_col_batch,

‎be/src/core/data_type_serde/data_type_timestamptz_serde.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,11 @@ class DataTypeTimeStampTzSerDe : public DataTypeNumberSerDe<PrimitiveType::TYPE_
6868
arrow::ArrayBuilder* array_builder, int64_t start, int64_t end,
6969
const cctz::time_zone& ctz) const override;
7070

71+
// Overridden rather than inherited: DataTypeNumberSerDe's reader would memcpy Arrow's epoch
72+
// integers over this column's packed values, same width and no error. See the definition.
73+
Status read_column_from_arrow(IColumn& column, const arrow::Array* arrow_array, int64_t start,
74+
int64_t end, const cctz::time_zone& ctz) const override;
75+
7176
Status write_column_to_orc(const std::string& timezone, const IColumn& column,
7277
const NullMap* null_map, orc::ColumnVectorBatch* orc_col_batch,
7378
int64_t start, int64_t end, Arena& arena,

‎be/test/core/data_type_serde/data_type_serde_timestamptz_test.cpp‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,9 @@
2424

2525
#include <cstddef>
2626
#include <iostream>
27+
#include <limits>
2728
#include <type_traits>
29+
#include <vector>
2830

2931
#include "core/assert_cast.h"
3032
#include "core/column/column.h"
@@ -274,4 +276,87 @@ TEST_F(DataTypeTimeStampTzSerDeTest, binary_roundtrip) {
274276
test_func(*serde_tz_3, column_tz_3, 3);
275277
test_func(*serde_tz_6, column_tz_6, 6);
276278
}
279+
280+
// Roundtrip through Arrow. Same failure shape as binary_roundtrip above: read_column_from_arrow
281+
// was inherited from DataTypeNumberSerDe, whose fixed-width path memcpy'd Arrow's int64 EPOCH
282+
// values over this column's PACKED values. Both are 8 bytes, so nothing failed -- the read
283+
// succeeded with every row wrong. Comparing the value that comes back against the one that went
284+
// out is what makes deleting the override fail here instead of in a user's result set.
285+
TEST_F(DataTypeTimeStampTzSerDeTest, ArrowRoundTrip) {
286+
auto test_func = [&](const DataTypeTimeStampTzSerDe& serde, arrow::TimeUnit::type unit,
287+
uint16_t year, uint8_t month, uint8_t day, uint8_t hour, uint8_t minute,
288+
uint8_t second, uint32_t microsecond, int64_t expected_arrow_value) {
289+
auto source_column = ColumnTimeStampTz::create();
290+
TimestampTzValue source_value;
291+
source_value.unchecked_set_time(year, month, day, hour, minute, second, microsecond);
292+
source_column->insert_value(source_value);
293+
294+
arrow::TimestampBuilder builder(arrow::timestamp(unit), arrow::default_memory_pool());
295+
ASSERT_TRUE(serde.write_column_to_arrow(*source_column, nullptr, &builder, 0,
296+
source_column->size(), cctz::utc_time_zone())
297+
.ok());
298+
std::shared_ptr<arrow::Array> array;
299+
ASSERT_TRUE(builder.Finish(&array).ok());
300+
// Pinned so the roundtrip cannot pass by both halves sharing one wrong encoding: what
301+
// crosses the wire is an epoch count in the unit, not the column's packed representation.
302+
ASSERT_EQ(expected_arrow_value,
303+
assert_cast<const arrow::TimestampArray*>(array.get())->Value(0));
304+
305+
auto dest_column = ColumnTimeStampTz::create();
306+
ASSERT_TRUE(serde.read_column_from_arrow(*dest_column, array.get(), 0, array->length(),
307+
cctz::utc_time_zone())
308+
.ok());
309+
ASSERT_EQ(1, dest_column->size());
310+
EXPECT_EQ(source_value, dest_column->get_element(0));
311+
};
312+
313+
// Before the epoch, so a division that truncates toward zero instead of flooring lands one
314+
// unit late rather than agreeing by accident.
315+
test_func(*serde_tz_6, arrow::TimeUnit::MICRO, 1969, 12, 31, 23, 59, 59, 123456, -876544);
316+
test_func(*serde_tz_3, arrow::TimeUnit::MILLI, 1969, 12, 31, 23, 59, 59, 123000, -877);
317+
test_func(*serde_tz_0, arrow::TimeUnit::SECOND, 1969, 12, 31, 23, 59, 59, 0, -1);
318+
test_func(*serde_tz_6, arrow::TimeUnit::MICRO, 2023, 1, 2, 3, 4, 5, 123456, 1672628645123456);
319+
}
320+
321+
// An Arrow type this serde cannot decode must fail the scan, not produce values. The inherited
322+
// reader accepted anything 8 bytes wide, which is how the corruption above stayed invisible.
323+
TEST_F(DataTypeTimeStampTzSerDeTest, ReadArrowRejectsNonTimestampType) {
324+
arrow::Int64Builder builder;
325+
ASSERT_TRUE(builder.Append(1672628645123456).ok());
326+
std::shared_ptr<arrow::Array> array;
327+
ASSERT_TRUE(builder.Finish(&array).ok());
328+
329+
auto dest_column = ColumnTimeStampTz::create();
330+
auto st = serde_tz_6->read_column_from_arrow(*dest_column, array.get(), 0, array->length(),
331+
cctz::utc_time_zone());
332+
EXPECT_FALSE(st.ok());
333+
EXPECT_EQ(0, dest_column->size());
334+
}
335+
336+
// A null slot's value buffer holds whatever the source left there. Converting it would range-check
337+
// garbage and fail a batch whose rows are all well-formed, so nulls take a default value instead.
338+
TEST_F(DataTypeTimeStampTzSerDeTest, ReadArrowIgnoresValuesUnderNulls) {
339+
TimestampTzValue source_value;
340+
source_value.unchecked_set_time(2023, 1, 2, 3, 4, 5, 123456);
341+
342+
std::vector<int64_t> values = {1672628645123456, std::numeric_limits<int64_t>::max()};
343+
// Bit 0 set, bit 1 clear: row 0 valid, row 1 null.
344+
std::vector<uint8_t> validity = {0b01};
345+
auto data = arrow::ArrayData::Make(arrow::timestamp(arrow::TimeUnit::MICRO), values.size(),
346+
{arrow::Buffer::Wrap(validity.data(), validity.size()),
347+
arrow::Buffer::Wrap(values.data(), values.size())},
348+
/*null_count=*/1);
349+
std::shared_ptr<arrow::Array> array = arrow::MakeArray(data);
350+
ASSERT_TRUE(array->IsNull(1));
351+
352+
auto dest_column = ColumnTimeStampTz::create();
353+
ASSERT_TRUE(serde_tz_6
354+
->read_column_from_arrow(*dest_column, array.get(), 0, array->length(),
355+
cctz::utc_time_zone())
356+
.ok());
357+
// One value per row, nulls included: the caller took the validity bitmap and expects the
358+
// nested column to have grown by the full range.
359+
ASSERT_EQ(2, dest_column->size());
360+
EXPECT_EQ(source_value, dest_column->get_element(0));
361+
}
277362
} // namespace doris

‎fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/FlightSqlSchemaHelper.java‎

Lines changed: 73 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,11 @@ private static ArrowType getArrowType(PrimitiveType primitiveType, Integer preci
121121
case VARIANT:
122122
return new ArrowType.Utf8();
123123
case DATEV2:
124-
return new ArrowType.Date(DateUnit.MILLISECOND);
124+
// DAY, not MILLISECOND: BE writes a DATEV2 column as arrow::Date32Type (a day number),
125+
// so a MILLISECOND unit here describes the metadata as date64 while the data that
126+
// follows is date32. A client that trusts this schema -- one reading through the ADBC
127+
// Flight SQL driver does -- then types the column as a datetime and fails the read.
128+
return new ArrowType.Date(DateUnit.DAY);
125129
case DATETIMEV2:
126130
if (scale > 3) {
127131
return new ArrowType.Timestamp(TimeUnit.MICROSECOND, timeZone);
@@ -285,41 +289,80 @@ private Map<String, List<Field>> buildTableToFields(String dbName, TDescribeTabl
285289
Integer tableOffset = describeTablesResult.getTablesOffset().get(tableIndex);
286290
for (; columnIndex < tableOffset; columnIndex++) {
287291
TColumnDef columnDef = describeTablesResult.getColumns().get(columnIndex);
288-
TColumnDesc columnDesc = columnDef.getColumnDesc();
289-
final ArrowType columnArrowType = columnDescToArrowType(columnDesc);
290-
291-
List<Field> columnArrowTypeChildren;
292-
// Arrow complex types may require children fields for parsing the schema on C++
293-
switch (columnArrowType.getTypeID()) {
294-
case List:
295-
case LargeList:
296-
case FixedSizeList:
297-
columnArrowTypeChildren = Collections.singletonList(
298-
Field.notNullable(BaseRepeatedValueVector.DATA_VECTOR_NAME,
299-
ZeroVector.INSTANCE.getField().getType()));
300-
break;
301-
case Map:
302-
columnArrowTypeChildren = Collections.singletonList(
303-
Field.notNullable(MapVector.DATA_VECTOR_NAME, new ArrowType.List()));
304-
break;
305-
case Struct:
306-
columnArrowTypeChildren = Collections.emptyList();
307-
break;
308-
default:
309-
columnArrowTypeChildren = null;
310-
break;
311-
}
312-
313-
final Field field = new Field(columnDesc.getColumnName(),
314-
new FieldType(columnDesc.isIsAllowNull(), columnArrowType, null,
315-
createFlightSqlColumnMetadata(dbName, tableName, columnDesc)), columnArrowTypeChildren);
316-
fields.add(field);
292+
fields.add(buildField(dbName, tableName, columnDef.getColumnDesc()));
317293
}
318294
tableToFields.put(tableName, fields);
319295
}
320296
return tableToFields;
321297
}
322298

299+
/** One column, with its nested types described down to the leaves. */
300+
private static Field buildField(String dbName, String tableName, TColumnDesc desc) {
301+
ArrowType arrowType = columnDescToArrowType(desc);
302+
return new Field(desc.getColumnName(),
303+
new FieldType(desc.isIsAllowNull(), arrowType, null,
304+
createFlightSqlColumnMetadata(dbName, tableName, desc)),
305+
arrowChildren(dbName, tableName, desc, arrowType));
306+
}
307+
308+
/**
309+
* The Arrow children of a complex column, built from the descriptor's own children.
310+
*
311+
* <p>These are not decoration. An Arrow ARRAY/MAP/STRUCT type carries its element types in its
312+
* children and nowhere else, so a placeholder child says the column is an array OF NOTHING --
313+
* and BE emits the real element type in the data ({@code convert_to_arrow_type}: ListType(item),
314+
* MapType(key, value), StructType(fields)), which leaves the schema describing one thing and the
315+
* batch carrying another. A client that types its columns from this schema (one reading through
316+
* the ADBC Flight SQL driver does) then rejects the column outright.
317+
*
318+
* <p>{@code describeTables} already reports the tree -- {@code Column.createChildrenColumn} names
319+
* an array's element "item" and a map's pair "key"/"value", which is what Arrow calls them too.
320+
* When it reports none, the old placeholders are kept rather than an empty child list: a source
321+
* that cannot describe its nested types is no worse off than before.
322+
*/
323+
private static List<Field> arrowChildren(String dbName, String tableName, TColumnDesc desc,
324+
ArrowType arrowType) {
325+
List<TColumnDesc> children = desc.isSetChildren() ? desc.getChildren() : Collections.emptyList();
326+
switch (arrowType.getTypeID()) {
327+
case List:
328+
case LargeList:
329+
case FixedSizeList:
330+
if (children.size() != 1) {
331+
return Collections.singletonList(
332+
Field.notNullable(BaseRepeatedValueVector.DATA_VECTOR_NAME,
333+
ZeroVector.INSTANCE.getField().getType()));
334+
}
335+
return Collections.singletonList(buildField(dbName, tableName, children.get(0)));
336+
case Map:
337+
// Arrow spells a map as list<entries: struct<key, value>>, with the entries struct and
338+
// the key both non-nullable -- the descriptor's key nullability is not carried over,
339+
// because an Arrow map with a nullable key is not a valid schema.
340+
if (children.size() != 2) {
341+
return Collections.singletonList(
342+
Field.notNullable(MapVector.DATA_VECTOR_NAME, new ArrowType.List()));
343+
}
344+
Field key = buildField(dbName, tableName, children.get(0));
345+
Field value = buildField(dbName, tableName, children.get(1));
346+
Field entries = new Field(MapVector.DATA_VECTOR_NAME,
347+
new FieldType(false, new ArrowType.Struct(), null),
348+
Arrays.asList(new Field(key.getName(),
349+
new FieldType(false, key.getType(), null), key.getChildren()),
350+
value));
351+
return Collections.singletonList(entries);
352+
case Struct:
353+
if (children.isEmpty()) {
354+
return Collections.emptyList();
355+
}
356+
List<Field> structFields = new ArrayList<>(children.size());
357+
for (TColumnDesc child : children) {
358+
structFields.add(buildField(dbName, tableName, child));
359+
}
360+
return structFields;
361+
default:
362+
return null;
363+
}
364+
}
365+
323366
/**
324367
* for FlightSqlProducer Schemas.GET_CATALOGS_SCHEMA
325368
*/

0 commit comments

Comments
 (0)