Repository navigation
Commit fa687bd
authored
[refactor](arrow-flight) Extract the Doris-to-Arrow type mapping into DorisArrowTypeMapping (#68315)
### What problem does this PR solve?
Issue Number: #67577
Related PR: #66344 (the `GetTables` schema fix whose nested-type tests
move here), #67966 / #68266 (the Flight SQL session work the next
interfaces build on)
Problem Summary:
**Context.** Doris speaks two client protocols: MySQL and Arrow Flight
SQL (the Flight SQL JDBC driver, the ADBC drivers, Python clients). Over
Flight the rows come as Arrow batches produced by BE, and
`convert_to_arrow_type` in `be/src/format/arrow/arrow_row_batch.cpp`
decides which Arrow type each Doris type has on the wire.
Besides data, Flight SQL has metadata commands.
`GetTables(include_schema = true)` is the client asking "which columns
does this table have, and what Arrow type is each one?", and FE answers
with a serialized Arrow schema in the `table_schema` column. Clients
trust that schema: the ADBC driver types its columns from it and then
decodes the batches of later queries as those types. So what FE says has
to match what BE sends; a mismatch is not a degraded answer but a failed
read (DATEV2 described as date64 while BE emitted date32 broke exactly
that way, fixed in #66344).
For that FE keeps a Doris-to-Arrow mapping, which until now was a
private method of `FlightSqlSchemaHelper`, the class that serves
`GetTables`: `getArrowType`, plus the `buildField` / `arrowChildren`
pair that builds a field and its nested children.
**1. The problem, and what it cost**
- A second and a third caller are about to arrive. The next Flight SQL
work all produces Arrow schemas: more metadata commands, the parameter
and result schemas of prepared statements, `ExecuteSchema`. They ask the
same question, "what Arrow type is this Doris type", but the answer was
a private method of another class, so each of them could only copy the
switch. Copies drift: it already happened once between FE and BE (#66761
added TIMESTAMP_NS as one line on each side), and another copy inside FE
would let `GetTables` describe a column as one type and a prepared
statement as another.
- The table was not pinned. The existing tests covered nested types and
DATEV2; the scalar table itself could change without any test noticing.
And some cells are known to disagree with BE and are kept that way on
purpose: TIMESTAMPTZ carries the literal zone `"UTC"` (BE stamps the
session time zone), and TIMEV2 / VARBINARY / AGG_STATE come back as
`Null` (BE emits float64 / binary / binary). They are kept because the
BE Arrow type layer is being reworked by another team and changing the
mapping now would collide with that, so the correction is scheduled as
one step after it (recorded in #67577). Without a test pinning them,
nothing stopped a well-meant one-cell "fix" that would change behaviour
piecemeal and out of step.
**2. What this PR does, and why it helps**
The mapping moves into a class of its own,
`org.apache.doris.arrow.DorisArrowTypeMapping`, the one place in FE that
maps a Doris type to an Arrow type, with no value changed. Three
commits:
1. A table-driven unit test against the old code first: one row per
`PrimitiveType` (41 values; the types whose unit depends on the scale
get a row per band, 47 rows in all), the known-wrong cells pinned as
they are and marked "kept as is (#67577)", plus a check that every
`PrimitiveType` has a row. Green against the old private method.
2. The move. The switch, the nested-type rules and the column metadata
are byte for byte what they were; the test's rows do not change and stay
green, which is the evidence for "no value changed".
3. A regression suite, `arrow_flight_sql_p0/test_get_tables_schema`: a
raw Flight SQL client asks a live cluster for `GetTables` of a table
that declares a column of every type (three-level nesting, BITMAP / HLL
/ AGG_STATE, DECIMAL256 included) and pins, field by field, the type,
the nullability and the column metadata a client sees. It covers what
the unit test cannot: the whole path from `describeTables`' descriptors
through the mapping to the serialization and the client's decoding.
What it buys:
- The coming callers call, they do not copy; every schema FE hands out
agrees with every other by construction.
- When the BE rework lands and the mapping is corrected, the change is
made in this one class (plus the tests' expectations) and every schema
changes together, instead of call site by call site.
- The known-wrong cells are pinned by two layers of tests that say why;
any change to them fails a test, so it can only be made deliberately.
- A new `PrimitiveType` without a mapping fails the test instead of
silently falling into `default -> Null`.
- Nothing changes for users: `GetTables` returns the same bytes. Checked
by dumping every field of the `GetTables` result with pyarrow from an FE
built before this PR and from one built at its head: 52 lines,
identical.
**3. The classes, and how they call each other**
- `DorisFlightSqlProducer` (existing): the Flight SQL server; on
`CommandGetTables`, `getStreamTables` creates a `FlightSqlSchemaHelper`.
- `FlightSqlSchemaHelper` (existing, slimmed): the `GetTables` plumbing
only. It lists databases and tables and calls `describeTables` through
`FrontendServiceImpl` to get each column's `TColumnDesc` (a thrift
descriptor with precision, scale and nested children), hands each column
to the mapping for an Arrow `Field`, and `getSerializedSchema` writes
the fields as Arrow IPC bytes into `table_schema`.
- `DorisArrowTypeMapping` (new):
- `toArrowType(PrimitiveType, precision, scale)`: the table itself (the
switch), a mirror of BE's `convert_to_arrow_type`.
- `toArrowType(TColumnDesc)`: reads precision and scale off the
descriptor and calls the above.
- `toField(db, table, TColumnDesc)`: builds the `Field`: the type, the
nullability, the Flight SQL column metadata a JDBC `ResultSetMetaData`
reads (`TYPE_NAME` / `PRECISION` / `SCALE` / `SCHEMA_NAME` /
`TABLE_NAME` ...), and the children, recursively (an ARRAY's `item`, a
MAP's `entries<key, value>` with the key forced non-nullable, a STRUCT's
fields).
- `FrontendServiceImpl.getColumnDesc` (existing, untouched): turns a
catalog `Column` into a `TColumnDesc`.
- BE `convert_to_arrow_type` (untouched): the sole authority on the
shape of the data on the wire; FE's table mirrors it.
- Tests: `DorisArrowTypeMappingTest` (table + nesting),
`FlightSqlSchemaHelperSerializedSchemaTest` (serialization round trips),
regression `arrow_flight_sql_p0/test_get_tables_schema`.
```
Flight SQL client (JDBC / ADBC)
| GetTables(include_schema = true)
v
DorisFlightSqlProducer.getStreamTables
| new FlightSqlSchemaHelper(ctx).getTables(...)
v
FlightSqlSchemaHelper ----> FrontendServiceImpl.getDbNames / listTableStatus / describeTables
| '- Column --getColumnDesc--> TColumnDesc (precision / scale / children)
| per column: DorisArrowTypeMapping.toField(db, table, desc) <-- the new class, moved out of the helper
| |- toArrowType(desc) --> toArrowType(primitiveType, precision, scale) [the table = mirror of BE convert_to_arrow_type]
| |- flightSqlColumnMetadata(...) [TYPE_NAME / PRECISION / SCALE / SCHEMA_NAME / TABLE_NAME]
| '- children(...) --> toField on each child, recursively [ARRAY item / MAP entries<key, value> / STRUCT fields]
| getSerializedSchema(fields) --> Arrow IPC bytes
v
table_schema column --> the client deserializes it and types its columns from it
--> decodes the batches of later queries with those types (batches produced by BE per convert_to_arrow_type)
Callers after this PR: metadata commands / prepared statement parameter & result schemas / ExecuteSchema --> the same DorisArrowTypeMapping
```
Not touched, so nobody goes looking: BE is unchanged, and FE-side
results (`SHOW ...`, all utf8 in `FlightSqlChannel`) are unchanged too;
that is another recorded item that also waits for the BE rework.1 parent 084144b commit fa687bd
5 files changed
Lines changed: 730 additions & 241 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris
- arrowflight
- arrow
- test/java/org/apache/doris
- arrowflight
- arrow
- regression-test/suites/arrow_flight_sql_p0
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
0 commit comments