Skip to content

Commit cc55086

Browse files
authored
[feature](external) Support Elasticsearch HTTP scans in FileScannerV2 (apache#69614)
### What problem does this PR solve? Related PR: apache#69601 Problem Summary: Elasticsearch HTTP scans always use FileScanner v1, even when `enable_file_scanner_v2` is enabled. Add an ES TableReader/FileReader pair and route `FORMAT_ES_HTTP` queries through FileScannerV2 when that session option is enabled. The reader exposes a file-local schema, uses MaterializedColumnMapper for projection and predicate localization, and applies residual filters with v2 row accounting. It reuses the existing ES query builder, HTTP scroll transport, authentication, SSL settings, catalog-provided endpoints, and type conversions. Shared host selection and materialized complex-schema helpers keep the two scanner paths consistent. Parse the initial cached response during open so early LIMIT/cancellation can clear its scroll context. Close the stream on EOF, cancellation, open/read errors, and split changes, and report close failures once. The RLE header reach baseline increases by two for the new ES reader and test translation units, which use the existing table/expr header closure. The shared parser retains its existing type dispatch, with a documented complexity suppression on the new projection overload. ### Release note Support Elasticsearch HTTP queries in FileScannerV2 when `enable_file_scanner_v2` is enabled. Disabling the option continues to select FileScanner v1. ### Check List (For Author) - Test - [x] Unit Test: 61 ASAN BE tests covering ES v2 projection, doc values, `_id`, nullable values/arrays, multi-page batching, zero-column row counting, residual filtering, split advancement, cancellation, error cleanup, and a real local HTTP scroll server; existing scanner dispatch, legacy ES, and Remote Doris tests also pass. - [ ] Regression test: added ES 5/6/7/8 v1/v2 differential coverage with source/docvalue scans and small batches. Groovy syntax checked; live external ES regression not run locally. - [x] clang-format 16, clang-tidy on changed C++ lines, and BE build-hygiene checks. - Behavior changed: - [x] Yes. ES queries honor the v2 session option. - Does this need documentation? - [x] No. Uses the existing scanner session option and ES catalog properties. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
1 parent 708531d commit cc55086

19 files changed

Lines changed: 1182 additions & 115 deletions

‎be/src/exec/operator/file_scan_operator.cpp‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -224,7 +224,6 @@ bool FileScanLocalState::should_use_file_scanner_v2(const TQueryOptions& query_o
224224
// after scanner construction. Keep versioned Iceberg scans on V2 without eagerly listing
225225
// every file.
226226
return scanner_v2_requested && !is_load &&
227-
scan_params.format_type != TFileFormatType::FORMAT_ES_HTTP &&
228227
scan_params.format_type != TFileFormatType::FORMAT_LANCE && !is_transactional_hive;
229228
}
230229

‎be/src/exec/scan/file_scanner_v2.cpp‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@
6060
#include "format_v2/jni/max_compute_jni_reader.h"
6161
#include "format_v2/jni/trino_connector_jni_reader.h"
6262
#include "format_v2/table/adbc_reader.h"
63+
#include "format_v2/table/es_reader.h"
6364
#include "format_v2/table/fluss_hybrid_reader.h"
6465
#include "format_v2/table/hive_reader.h"
6566
#include "format_v2/table/hudi_reader.h"
@@ -356,6 +357,8 @@ bool FileScannerV2::is_supported(const TFileScanRangeParams& params, const TFile
356357
if (format_type == TFileFormatType::FORMAT_PARQUET ||
357358
format_type == TFileFormatType::FORMAT_ORC) {
358359
return is_supported_table_format(range);
360+
} else if (format_type == TFileFormatType::FORMAT_ES_HTTP) {
361+
return table_format_name(range) == "es";
359362
} else if (format_type == TFileFormatType::FORMAT_ARROW) {
360363
return is_supported_arrow_table_format(range);
361364
} else if (format_type == TFileFormatType::FORMAT_JNI) {
@@ -801,6 +804,8 @@ Status FileScannerV2::_create_table_reader_for_format(
801804
*reader = std::make_unique<format::max_compute::MaxComputeJniReader>(mc_desc);
802805
} else if (table_format == "trino_connector") {
803806
*reader = std::make_unique<format::trino_connector::TrinoConnectorJniReader>();
807+
} else if (table_format == "es") {
808+
*reader = std::make_unique<format::es::EsReader>();
804809
} else if (table_format == "remote_doris") {
805810
*reader = std::make_unique<format::remote_doris::RemoteDorisReader>();
806811
} else if (table_format == "adbc") {
@@ -1092,6 +1097,9 @@ Status FileScannerV2::_to_file_format(TFileFormatType::type format_type,
10921097
case TFileFormatType::FORMAT_ARROW:
10931098
*file_format = format::FileFormat::ARROW;
10941099
return Status::OK();
1100+
case TFileFormatType::FORMAT_ES_HTTP:
1101+
*file_format = format::FileFormat::ES_HTTP;
1102+
return Status::OK();
10951103
case TFileFormatType::FORMAT_WAL:
10961104
*file_format = format::FileFormat::WAL;
10971105
return Status::OK();

‎be/src/format/table/es/es_http_reader.cpp‎

Lines changed: 3 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -19,17 +19,14 @@
1919

2020
#include <gen_cpp/PlanNodes_types.h>
2121

22-
#include <algorithm>
23-
#include <cctype>
24-
#include <sstream>
22+
#include <utility>
2523

2624
#include "format/table/es/es_scan_reader.h"
2725
#include "format/table/es/es_scroll_parser.h"
2826
#include "format/table/es/es_scroll_query.h"
2927
#include "runtime/descriptors.h"
3028
#include "runtime/runtime_profile.h"
3129
#include "runtime/runtime_state.h"
32-
#include "service/backend_options.h"
3330

3431
namespace doris {
3532

@@ -65,29 +62,7 @@ void EsHttpReader::_init_profile() {
6562
}
6663

6764
std::string EsHttpReader::_extract_hostname(std::string host) {
68-
auto scheme_end = host.find("://");
69-
if (scheme_end != std::string::npos) {
70-
host = host.substr(scheme_end + 3);
71-
}
72-
73-
if (!host.empty() && host.front() == '[') {
74-
auto bracket_end = host.find(']');
75-
if (bracket_end != std::string::npos) {
76-
return host.substr(1, bracket_end - 1);
77-
}
78-
return host;
79-
}
80-
81-
auto colon = host.rfind(':');
82-
if (colon == std::string::npos) {
83-
return host;
84-
}
85-
std::string port = host.substr(colon + 1);
86-
if (!port.empty() && std::all_of(port.begin(), port.end(),
87-
[](unsigned char ch) { return std::isdigit(ch) != 0; })) {
88-
return host.substr(0, colon);
89-
}
90-
return host;
65+
return ESScanReader::extract_hostname(std::move(host));
9166
}
9267

9368
Status EsHttpReader::init_reader() {
@@ -225,32 +200,7 @@ Status EsHttpReader::_get_columns_impl(std::unordered_map<std::string, DataTypeP
225200
}
226201

227202
std::string EsHttpReader::_select_host(const std::map<std::string, std::string>& properties) const {
228-
// If es_hosts contains multiple hosts, prefer localhost for locality
229-
auto it = properties.find(KEY_ES_HOSTS);
230-
if (it != properties.end() && !it->second.empty()) {
231-
std::string localhost = BackendOptions::get_localhost();
232-
std::string best;
233-
std::istringstream stream(it->second);
234-
std::string host;
235-
while (std::getline(stream, host, ',')) {
236-
if (best.empty()) {
237-
best = host;
238-
}
239-
std::string hostname = _extract_hostname(host);
240-
if (hostname == localhost) {
241-
return host;
242-
}
243-
}
244-
if (!best.empty()) {
245-
return best;
246-
}
247-
}
248-
// Fallback to host_port
249-
auto hp = properties.find(ESScanReader::KEY_HOST_PORT);
250-
if (hp != properties.end()) {
251-
return hp->second;
252-
}
253-
return "";
203+
return ESScanReader::select_host(properties);
254204
}
255205

256206
} // namespace doris

‎be/src/format/table/es/es_scan_reader.cpp‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@
1919

2020
#include <stdlib.h>
2121

22+
#include <algorithm>
23+
#include <cctype>
2224
#include <map>
2325
#include <sstream>
2426
#include <string>
@@ -28,6 +30,7 @@
2830
#include "common/status.h"
2931
#include "format/table/es/es_scroll_parser.h"
3032
#include "format/table/es/es_scroll_query.h"
33+
#include "service/backend_options.h"
3134
#include "service/http/http_method.h"
3235

3336
namespace doris {
@@ -266,4 +269,60 @@ Status ESScanReader::close() {
266269
return Status::InternalError("es_scan_reader delete scroll context failure");
267270
}
268271
}
272+
273+
std::string ESScanReader::extract_hostname(std::string host) {
274+
auto scheme_end = host.find("://");
275+
if (scheme_end != std::string::npos) {
276+
host = host.substr(scheme_end + 3);
277+
}
278+
279+
if (!host.empty() && host.front() == '[') {
280+
auto bracket_end = host.find(']');
281+
if (bracket_end != std::string::npos) {
282+
return host.substr(1, bracket_end - 1);
283+
}
284+
return host;
285+
}
286+
287+
auto colon = host.rfind(':');
288+
if (colon == std::string::npos) {
289+
return host;
290+
}
291+
std::string port = host.substr(colon + 1);
292+
if (!port.empty() &&
293+
std::ranges::all_of(port, [](unsigned char ch) { return std::isdigit(ch) != 0; })) {
294+
return host.substr(0, colon);
295+
}
296+
return host;
297+
}
298+
299+
std::string ESScanReader::select_host(const std::map<std::string, std::string>& properties) {
300+
// If es_hosts contains multiple hosts, prefer localhost for locality
301+
auto it = properties.find("es_hosts");
302+
if (it != properties.end() && !it->second.empty()) {
303+
std::string localhost = BackendOptions::get_localhost();
304+
std::string best;
305+
std::istringstream stream(it->second);
306+
std::string host;
307+
while (std::getline(stream, host, ',')) {
308+
if (best.empty()) {
309+
best = host;
310+
}
311+
std::string hostname = extract_hostname(host);
312+
if (hostname == localhost) {
313+
return host;
314+
}
315+
}
316+
if (!best.empty()) {
317+
return best;
318+
}
319+
}
320+
// Fallback to host_port
321+
auto hp = properties.find(ESScanReader::KEY_HOST_PORT);
322+
if (hp != properties.end()) {
323+
return hp->second;
324+
}
325+
return "";
326+
}
327+
269328
} // namespace doris

‎be/src/format/table/es/es_scan_reader.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,9 @@ class ESScanReader {
5858
// clear scroll context from elasticsearch
5959
Status close();
6060

61+
static std::string extract_hostname(std::string host);
62+
static std::string select_host(const std::map<std::string, std::string>& properties);
63+
6164
private:
6265
std::string _target;
6366
std::string _user_name;

‎be/src/format/table/es/es_scroll_parser.cpp‎

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -662,6 +662,17 @@ Status ScrollParser::fill_columns(const TupleDescriptor* tuple_desc,
662662
std::vector<MutableColumnPtr>& columns, bool* line_eof,
663663
const std::map<std::string, std::string>& docvalue_context,
664664
const cctz::time_zone& time_zone) {
665+
return fill_columns(tuple_desc->slots(), columns, line_eof, docvalue_context, time_zone);
666+
}
667+
668+
// Keep identical ES-to-Doris type dispatch for the tuple and file-local projection entry points.
669+
// NOLINTNEXTLINE(readability-function-cognitive-complexity,readability-function-size)
670+
Status ScrollParser::fill_columns(const std::vector<SlotDescriptor*>& slots,
671+
std::vector<MutableColumnPtr>& columns, bool* line_eof,
672+
const std::map<std::string, std::string>& docvalue_context,
673+
const cctz::time_zone& time_zone,
674+
std::span<const size_t> column_positions) {
675+
DORIS_CHECK(column_positions.empty() || column_positions.size() == slots.size());
665676
*line_eof = true;
666677

667678
if (_size <= 0 || _line_index >= _size) {
@@ -681,9 +692,9 @@ Status ScrollParser::fill_columns(const TupleDescriptor* tuple_desc,
681692
line = &obj["fields"];
682693
}
683694

684-
for (int i = 0; i < tuple_desc->slots().size(); ++i) {
685-
const SlotDescriptor* slot_desc = tuple_desc->slots()[i];
686-
auto* col_ptr = columns[i].get();
695+
for (size_t i = 0; i < slots.size(); ++i) {
696+
const SlotDescriptor* slot_desc = slots[i];
697+
auto* col_ptr = columns[column_positions.empty() ? i : column_positions[i]].get();
687698

688699
if (slot_desc->col_name() == FIELD_ID) {
689700
// actually this branch will not be reached, this is guaranteed by Doris FE.
@@ -887,10 +898,10 @@ Status ScrollParser::fill_columns(const TupleDescriptor* tuple_desc,
887898
}
888899
case TYPE_ARRAY: {
889900
Array array;
890-
const auto& sub_type = assert_cast<const DataTypeArray*>(
891-
remove_nullable(tuple_desc->slots()[i]->type()).get())
892-
->get_nested_type()
893-
->get_primitive_type();
901+
const auto& sub_type =
902+
assert_cast<const DataTypeArray*>(remove_nullable(slot_desc->type()).get())
903+
->get_nested_type()
904+
->get_primitive_type();
894905
RETURN_IF_ERROR(parse_column(col, sub_type, pure_doc_value, array, time_zone));
895906
col_ptr->insert(Field::create_field<TYPE_ARRAY>(array));
896907
break;

‎be/src/format/table/es/es_scroll_parser.h‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
#include <rapidjson/rapidjson.h>
2121

2222
#include <map>
23+
#include <span>
2324
#include <string>
2425
#include <vector>
2526

@@ -30,6 +31,7 @@ namespace doris {
3031

3132
class Status;
3233
class TupleDescriptor;
34+
class SlotDescriptor;
3335

3436
class ScrollParser {
3537
public:
@@ -42,6 +44,13 @@ class ScrollParser {
4244
bool* line_eof, const std::map<std::string, std::string>& docvalue_context,
4345
const cctz::time_zone& time_zone);
4446

47+
// Empty positions preserve tuple order; explicit positions address a projected file-local block.
48+
Status fill_columns(const std::vector<SlotDescriptor*>& slots,
49+
std::vector<MutableColumnPtr>& columns, bool* line_eof,
50+
const std::map<std::string, std::string>& docvalue_context,
51+
const cctz::time_zone& time_zone,
52+
std::span<const size_t> column_positions = {});
53+
4554
const std::string& get_scroll_id();
4655
int get_size() const;
4756

‎be/src/format_v2/file_reader.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ enum class FileFormat {
6060
JNI,
6161
ARROW,
6262
WAL,
63+
ES_HTTP,
6364
};
6465

6566
struct FileScanRequest {

‎be/src/format_v2/materialized_reader_util.cpp‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,10 @@
2121

2222
#include "core/block/block.h"
2323
#include "core/column/column_vector.h"
24+
#include "core/data_type/data_type_array.h"
25+
#include "core/data_type/data_type_map.h"
2426
#include "core/data_type/data_type_nullable.h"
27+
#include "core/data_type/data_type_struct.h"
2528
#include "core/types.h"
2629
#include "exprs/vexpr_context.h"
2730
#include "format_v2/file_reader.h"
@@ -36,8 +39,52 @@ void update_counter(RuntimeProfile::Counter* counter, int64_t value) {
3639
}
3740
}
3841

42+
ColumnDefinition materialized_child_definition(const std::string& name, DataTypePtr type,
43+
int32_t local_id) {
44+
ColumnDefinition child;
45+
child.identifier = Field::create_field<TYPE_STRING>(name);
46+
child.local_id = local_id;
47+
child.name = name;
48+
child.type = std::move(type);
49+
child.children = materialized_schema_children(child.type);
50+
return child;
51+
}
52+
3953
} // namespace
4054

55+
std::vector<ColumnDefinition> materialized_schema_children(const DataTypePtr& type) {
56+
std::vector<ColumnDefinition> children;
57+
DORIS_CHECK(type != nullptr);
58+
const auto nested_type = remove_nullable(type);
59+
switch (nested_type->get_primitive_type()) {
60+
case TYPE_ARRAY: {
61+
const auto* array_type = assert_cast<const DataTypeArray*>(nested_type.get());
62+
children.push_back(
63+
materialized_child_definition("element", array_type->get_nested_type(), 0));
64+
break;
65+
}
66+
case TYPE_MAP: {
67+
const auto* map_type = assert_cast<const DataTypeMap*>(nested_type.get());
68+
children.push_back(materialized_child_definition("key", map_type->get_key_type(), 0));
69+
children.push_back(materialized_child_definition("value", map_type->get_value_type(), 1));
70+
break;
71+
}
72+
case TYPE_STRUCT: {
73+
const auto* struct_type = assert_cast<const DataTypeStruct*>(nested_type.get());
74+
children.reserve(struct_type->get_elements().size());
75+
for (size_t idx = 0; idx < struct_type->get_elements().size(); ++idx) {
76+
children.push_back(materialized_child_definition(struct_type->get_element_name(idx),
77+
struct_type->get_element(idx),
78+
cast_set<int32_t>(idx)));
79+
}
80+
break;
81+
}
82+
default:
83+
break;
84+
}
85+
return children;
86+
}
87+
4188
ColumnPtr make_column_nullable_if_needed(ColumnPtr column, const DataTypePtr& target_type) {
4289
if (target_type != nullptr && target_type->is_nullable() && column.get() != nullptr &&
4390
!column->is_nullable()) {

‎be/src/format_v2/materialized_reader_util.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
#pragma once
1919

2020
#include <memory>
21+
#include <vector>
2122

2223
#include "common/status.h"
2324
#include "core/column/column.h"
@@ -33,6 +34,10 @@ struct IOContext;
3334

3435
namespace format {
3536
struct FileScanRequest;
37+
struct ColumnDefinition;
38+
39+
// Describe children of already materialized ARRAY/MAP/STRUCT values in their local schema.
40+
std::vector<ColumnDefinition> materialized_schema_children(const DataTypePtr& type);
3641

3742
// Shared helpers for FileReader implementations that deserialize or build already materialized
3843
// Doris columns and then hand those columns to TableReader for final mapping.

0 commit comments

Comments
 (0)