Skip to content

Commit 8ebf20d

Browse files
authored
[feature](fe) Support MongoDB ObjectId and Trino JSON types (#69969)
### What problem does this PR solve? Issue Number: N/A MongoDB exposes ObjectId as a plugin-defined Trino type. Doris did not register these types in either plugin loader, rejected ObjectId during schema mapping, and discarded the connector's default hidden `_id` column. Register plugin-defined types in FE and BE, map ObjectId recursively to `VARBINARY(12)`, and expose the default MongoDB `_id` while retaining the hiding of other columns. Existing byte readers preserve all twelve bytes without UTF-8 conversion. Trino JSON columns also lacked a Doris type mapping. Map the built-in Trino JSON type recursively to native Doris JSON, and parse its UTF-8 JNI carrier into JSONB. Preserve SQL NULL independently of the JSON value `null`, including empty and repeated batches. Reuse the existing JSON parser and return malformed-input errors. Correct nested JSON text serialization so arrays, maps and structs contain JSON values with their original quoting; previously an object could become invalid text such as `["{"a":1}"]`. Coordinate typed ARRAY/MAP/STRUCT text parsing with that formatter, retaining JSON string quotes and using uppercase `NULL` for SQL-null JSON children versus lowercase JSON `null`. Both strict and non-strict casts retain the original JSON type and null semantics. Delegate complete child tokens to `from_string_as_nested`: scalar SerDes remove outer quotes, JSONB retains JSON quotes and accepts legacy SQL apostrophe wrappers, and Nullable handles SQL nulls and conversion errors using the child's paired null-token policy. Generic complex parsing no longer queries the concrete JSONB type. Require the fixture's final network-facing MongoDB process before marking it ready, and use bounded tmpfs mounts for both image-declared data directories to avoid orphaned anonymous volumes. Add an opt-in MongoDB 8.0.20 Docker fixture and five regression suites with **138 result checks and ten expected-error checks**. Coverage includes primitive and nested types, BSON documents and Extended JSON, all JSON value shapes, null and missing fields, numeric/date boundaries, Unicode and escapes, schema discovery/permissions, scalar and JSON predicates, joins, aggregates, subqueries, windows, views, internal-table joins, scalar JSON materialization, multi-page reads and repeated early termination. Golden files are generated and compared with the standard regression runner. Source-side MongoDB queries independently confirm critical counts and null semantics. MongoDB remains an operator-installed Trino 435 plugin; no MongoDB runtime dependency is added to Doris. Use the complete matching plugin ZIP on FE and BE. ObjectId retains existing VARBINARY operator/storage restrictions; use `from_binary(_id)` for hexadecimal filtering, joining, grouping or materialization. Nested JSON is readable through catalogs, while internal-table creation retains existing complex-type restrictions, including rejecting `ARRAY<JSON>`. BSON documents inferred automatically remain ROW/STRUCT; use an explicit `json` schema declaration to read a field as a whole JSON value. ### Validation - FE package and native ASAN BE built through `build.sh --fe` / `build.sh --be`. UI, Hive UDF, OBS/COS dependencies and unrelated Java plugins were excluded locally; the native build covers the initial connector and JSON changes. The final SerDe interface refactor was compiled and linked through the ASAN BE unit-test build; the standalone BE package and external integration suites were not rerun for that refactor. Final executable passed the GLIBC_2.17 compatibility gate. - `run-be-ut.sh --run`: **33 related BE unit tests passed**, including existing JSON serde, general complex-type parsing and formatting tests. Typed JSON text covers 66 combinations of container, value shape and strictness, plus 48 legacy apostrophe-wrapper combinations, deeper nesting and malformed children in both modes. Reverting JNI support reproduces three failures; the typed complex JSON test fails before the reader fix and passes after it. - `run-regression-test.sh --run -d external_table_p0/trino_connector/mongodb`: **five suites passed**, with **138 executed result checks and ten expected errors**, real MongoDB and the complete Trino 435 plugin ZIP. - Checkstyle/validate, clang-format 16 for all affected C++ files, header hygiene and clang-tidy passed. - **17 related regression suites passed** across the final comparison and an Elasticsearch rerun after adjusting only local fixture CPU/direct-memory limits. This includes the array-function documentation, JSON/array/aggregate coverage, all four Elasticsearch fixture versions, spill-enabled bucket-shuffle plans, HTTP TVF and FE/BE version metrics. - The generated-version test verifies refreshed numbers without recompiling a cached consumer. Warmed FE builds verify changed and restored release metadata after cache hits; bytecode inspection confirms numeric fields are read at runtime. - Gated MongoDB startup reproduces and fixes the initialization-server readiness race; repeated start/stop cycles leave no anonymous data volumes. - Independent MongoDB queries confirmed source-side null semantics and multi-page counts/sums. Refresh the affected array-function documentation and Elasticsearch golden results through the regression runner. Keep the bucket-shuffle regression focused on physical INTERSECT/EXCEPT nodes when spill is enabled by disabling their two spill rewrite rules. Pin the HTTP TVF INT96 fixture timezone to UTC to preserve its stored wall-clock values and existing golden results. Refresh the FE build-version class after Maven cache restoration so packaged FE/BE version metrics report the current revision. Keep numeric version fields runtime-initialized so cached consumers observe refreshed metadata. A standalone generator test covers replacing Version.class without recompiling its consumer and invokes the actual packaging function with a PATH-only JDK to verify both tool selection and deployed-jar metadata. The fixture README documents setup, JSON schema behavior, typed SQL text null spelling, coverage and current limitations. MongoDB integration suites are disabled by default and require the matching plugin on FE and BE. ### Release note Support MongoDB ObjectId columns, including the default `_id` and nested fields, as `VARBINARY(12)` through Trino connector catalogs. Support Trino JSON columns as native Doris JSON, including nested JSON fields, and preserve JSON values and SQL-null distinctions through typed complex-text round trips. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [x] Manual test: real MongoDB fixture, complete connector ZIP, exact bytes and independent source-side JSON checks - Behavior changed: - [x] Yes. ObjectId and JSON fields become queryable with their native Doris types, the default hidden MongoDB `_id` is exposed, and nested JSON values retain their own quotes and null semantics. - Does this need documentation? - [x] Yes. Setup, JSON schema selection and limitations are documented in `docker/thirdparties/docker-compose/mongodb/README.md`. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
1 parent 3763f09 commit 8ebf20d

44 files changed

Lines changed: 2668 additions & 120 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/core/data_type_serde/complex_type_deserialize_util.h‎

Lines changed: 4 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -78,33 +78,14 @@ struct ComplexTypeDeserializeUtil {
7878
return elements;
7979
}
8080

81-
static bool is_null_string(const StringRef& str) {
82-
if (str.size == 4) {
83-
// null
84-
return str.data[0] == 'n' && str.data[1] == 'u' && str.data[2] == 'l' &&
85-
str.data[3] == 'l';
86-
}
87-
return false;
88-
}
89-
9081
template <bool is_strict_mode>
9182
static Status process_column(const DataTypeSerDeSPtr& serde, IColumn& column, StringRef& str,
9283
const DataTypeSerDe::FormatOptions& options) {
9384
DCHECK(column.is_nullable()) << "Column must be nullable but got " << column.get_name();
94-
if (is_null_string(str)) {
95-
auto& column_nullable = assert_cast<ColumnNullable&>(column);
96-
column_nullable.insert_default();
97-
return Status::OK();
98-
}
99-
auto str_without_quote = str.trim_quote();
100-
if constexpr (is_strict_mode) {
101-
return serde->from_string_strict_mode(str_without_quote, column, options);
102-
} else {
103-
auto st = serde->from_string(str_without_quote, column, options);
104-
DCHECK(st.ok()) << "no strict mode, so should not return error";
105-
return st;
106-
}
85+
auto status = serde->from_string_as_nested(str, column, options, is_strict_mode);
86+
DCHECK(is_strict_mode || status.ok()) << "no strict mode, so should not return error";
87+
return status;
10788
}
10889
};
10990

110-
} // namespace doris
91+
} // namespace doris

‎be/src/core/data_type_serde/data_type_jsonb_serde.cpp‎

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -298,6 +298,20 @@ Status DataTypeJsonbSerDe::from_string(StringRef& str, IColumn& column,
298298
return deserialize_one_cell_from_json(column, slice, options);
299299
}
300300

301+
Status DataTypeJsonbSerDe::from_string_as_nested(StringRef& str, IColumn& column,
302+
const FormatOptions& options,
303+
bool is_strict_mode) const {
304+
// Quotes belong to the JSON value: stripping them turns "123" into a number
305+
// and "null" into JSON null, losing the original type.
306+
auto token = str;
307+
// SQL apostrophes are an optional outer wrapper, e.g. '{"a":1}' or '"123"'.
308+
if (token.size >= 2 && token.front() == '\'' && token.back() == '\'') {
309+
token = token.substring(1, token.size - 2);
310+
}
311+
return is_strict_mode ? from_string_strict_mode(token, column, options)
312+
: from_string(token, column, options);
313+
}
314+
301315
void DataTypeJsonbSerDe::write_one_cell_to_binary(const IColumn& src_column,
302316
ColumnString::Chars& chars,
303317
int64_t row_num) const {
@@ -341,14 +355,10 @@ void DataTypeJsonbSerDe::to_string(const IColumn& column, size_t row_num, Buffer
341355
const auto& col = assert_cast<const ColumnString&, TypeCheckOnRelease::DISABLE>(column);
342356
const auto& data_ref = col.get_data_at(row_num);
343357
if (data_ref.size > 0) {
344-
if (_nesting_level > 1) {
345-
bw.write('"');
346-
}
358+
// JSON text already quotes strings and escapes their contents. Wrapping a nested
359+
// object again would produce invalid text such as ["{"a":1}"].
347360
std::string str = JsonbToJson::jsonb_to_json_string(data_ref.data, data_ref.size);
348361
bw.write(str.c_str(), str.size());
349-
if (_nesting_level > 1) {
350-
bw.write('"');
351-
}
352362
} else {
353363
bw.write("NULL", 4);
354364
}

‎be/src/core/data_type_serde/data_type_jsonb_serde.h‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,12 @@ class DataTypeJsonbSerDe : public DataTypeStringSerDe {
3737

3838
std::string get_name() const override { return "JSONB"; }
3939

40+
// JSON null is a value, so a nested SQL NULL needs a distinct spelling.
41+
StringRef get_nested_null_format() const override { return {"NULL", 4}; }
42+
43+
Status from_string_as_nested(StringRef& str, IColumn& column, const FormatOptions& options,
44+
bool is_strict_mode) const override;
45+
4046
Status from_string(StringRef& str, IColumn& column,
4147
const FormatOptions& options) const override;
4248

‎be/src/core/data_type_serde/data_type_nullable_serde.cpp‎

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -580,8 +580,10 @@ void DataTypeNullableSerDe::to_string(const IColumn& column, size_t row_num, Buf
580580
const FormatOptions& options) const {
581581
const auto& col_null = assert_cast<const ColumnNullable&, TypeCheckOnRelease::DISABLE>(column);
582582
if (col_null.is_null_at(row_num)) {
583+
// A child may reserve lowercase null for a value; use its paired SQL-null marker.
583584
if (_nesting_level > 1) {
584-
bw.write("null", 4);
585+
const auto null_format = nested_serde->get_nested_null_format();
586+
bw.write(null_format.data, null_format.size);
585587
} else {
586588
bw.write("NULL", 4);
587589
}
@@ -594,6 +596,28 @@ std::string DataTypeNullableSerDe::to_olap_string(const Field& field) const {
594596
return nested_serde->to_olap_string(field);
595597
}
596598

599+
Status DataTypeNullableSerDe::from_string_as_nested(StringRef& str, IColumn& column,
600+
const FormatOptions& options,
601+
bool is_strict_mode) const {
602+
auto& null_column = assert_cast<ColumnNullable&>(column);
603+
// Match the unquoted marker before parsing the value: for JSON, null and
604+
// "null" are non-SQL-null values, whereas NULL is the SQL-null marker.
605+
if (str == nested_serde->get_nested_null_format()) {
606+
null_column.insert_default();
607+
return Status::OK();
608+
}
609+
auto status = nested_serde->from_string_as_nested(str, null_column.get_nested_column(), options,
610+
is_strict_mode);
611+
// Malformed children become SQL NULL only in non-strict mode, just as in from_string.
612+
if (!status.ok() && !is_strict_mode) {
613+
null_column.insert_default();
614+
return Status::OK();
615+
}
616+
RETURN_IF_ERROR(status);
617+
null_column.get_null_map_data().push_back(0);
618+
return Status::OK();
619+
}
620+
597621
// In non-strict mode, from string will handle errors by inserting a null value.
598622
Status DataTypeNullableSerDe::from_string(StringRef& str, IColumn& column,
599623
const FormatOptions& options) const {

‎be/src/core/data_type_serde/data_type_nullable_serde.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,9 @@ class DataTypeNullableSerDe : public DataTypeSerDe {
3636

3737
std::string get_name() const override { return "Nullable(" + nested_serde->get_name() + ")"; }
3838

39+
Status from_string_as_nested(StringRef& str, IColumn& column, const FormatOptions& options,
40+
bool is_strict_mode) const override;
41+
3942
Status from_string(StringRef& str, IColumn& column,
4043
const FormatOptions& options) const override;
4144
Status from_zonemap_string(const std::string& str, Field& field) const override;

‎be/src/core/data_type_serde/data_type_serde.cpp‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,14 @@
4040
namespace doris {
4141
DataTypeSerDe::~DataTypeSerDe() = default;
4242

43+
Status DataTypeSerDe::from_string_as_nested(StringRef& str, IColumn& column,
44+
const FormatOptions& options,
45+
bool is_strict_mode) const {
46+
auto unquoted = str.trim_quote();
47+
return is_strict_mode ? from_string_strict_mode(unquoted, column, options)
48+
: from_string(unquoted, column, options);
49+
}
50+
4351
bool decoded_column_view_can_null_on_conversion_failure(const DecodedColumnView& view) {
4452
return !view.enable_strict_mode && view.conversion_failure_null_map != nullptr;
4553
}

‎be/src/core/data_type_serde/data_type_serde.h‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,7 @@ class DataTypeSerDe {
143143
public:
144144
// return type name , such as "BOOL", "BIGINT", "ARRAY<DATE>"
145145
virtual std::string get_name() const = 0;
146+
146147
// Text serialization/deserialization of data types depend on some settings witch we define
147148
// in formatOptions.
148149
struct FormatOptions {
@@ -339,6 +340,14 @@ class DataTypeSerDe {
339340
return from_string(str, column, options);
340341
}
341342

343+
// Parse a complete child token from ARRAY/MAP/STRUCT text. Scalar parsers normally
344+
// receive unquoted text; types whose syntax owns the quotes override this entry point.
345+
virtual Status from_string_as_nested(StringRef& str, IColumn& column,
346+
const FormatOptions& options, bool is_strict_mode) const;
347+
348+
// Nullable uses the same marker when reading and writing a child SQL NULL.
349+
virtual StringRef get_nested_null_format() const { return {"null", 4}; }
350+
342351
// Override functions with _batch suffix only when performance is critical
343352
// Only used for basic data types, such as Ip, Date, Number, etc.
344353
virtual Status from_string_batch(const ColumnString& str, ColumnNullable& column,

‎be/src/format/jni/jni_data_bridge.cpp‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,11 +39,36 @@
3939
#include "core/data_type/primitive_type.h"
4040
#include "core/types.h"
4141
#include "core/value/decimalv2_value.h"
42+
#include "core/value/jsonb_value.h"
4243
#include "util/string_util.h"
4344
#include "util/url_coding.h"
4445

4546
namespace doris {
4647

48+
namespace {
49+
50+
Status fill_jsonb_column(JniDataBridge::TableMetaAddress& address, MutableColumnPtr& column,
51+
const bool* null_map, size_t num_rows) {
52+
const auto* offsets = static_cast<const int*>(address.next_meta_as_ptr());
53+
const auto* chars = static_cast<const char*>(address.next_meta_as_ptr());
54+
JsonBinaryValue value;
55+
size_t start = 0;
56+
for (size_t i = 0; i < num_rows; ++i) {
57+
// JNI carries JSON as UTF-8 text, not JSONB bytes. SQL NULL has no text to parse,
58+
// whereas the non-null text "null" must become a JSON null value.
59+
if (null_map[i]) {
60+
column->insert_default();
61+
} else {
62+
RETURN_IF_ERROR(value.from_json_string(chars + start, offsets[i] - start));
63+
column->insert_data(value.value(), value.size());
64+
}
65+
start = offsets[i];
66+
}
67+
return Status::OK();
68+
}
69+
70+
} // namespace
71+
4772
#define FOR_FIXED_LENGTH_TYPES(M) \
4873
M(PrimitiveType::TYPE_TINYINT, ColumnInt8, Int8) \
4974
M(PrimitiveType::TYPE_BOOLEAN, ColumnUInt8, UInt8) \
@@ -139,6 +164,10 @@ Status JniDataBridge::fill_column(TableMetaAddress& address, ColumnPtr& doris_co
139164
case PrimitiveType::TYPE_VARCHAR:
140165
status = _fill_string_column(address, data_column, num_rows);
141166
break;
167+
case PrimitiveType::TYPE_JSONB:
168+
status = fill_jsonb_column(address, data_column, static_cast<const bool*>(null_map_ptr),
169+
num_rows);
170+
break;
142171
case PrimitiveType::TYPE_ARRAY:
143172
status = _fill_array_column(address, data_column, data_type, num_rows);
144173
break;

0 commit comments

Comments
 (0)