Skip to content

Commit 3dc7341

Browse files
authored
[fix](external) Preserve ES residual filters and repair scanner regression setup (apache#69650)
### What problem does this PR solve? Related PR: apache#69614 `external_table_p2/es/test_es_file_scanner_v2.groovy` fails on its first legacy-scanner baseline query with `Table [test1] does not exist in database [default_db]`. Its interpolated port key is a Groovy `GString`, while `otherConfigs` is a `Properties` containing `String` keys. `Properties.get(Object)` returns null even when the port is configured, producing `http://<es-host>:null`. Convert the key to `String` and fail explicitly when the configuration is absent. The failure happens before any V2 result comparison or BE scan. The coverage review also found a query correctness issue shared by both scanners: a nested AND can return a partial ES DSL and record an unsupported child, but the single-expression path ignored that residual tracking. For `(a = 1 AND local_predicate) OR a = 2`, the engine could drop the local predicate; `NOT(a = 1 AND local_predicate)` could also become an unsafe `NOT(a = 1)` at the source. Retain the complete expression locally whenever conversion records a residual. Fully pushable expressions and independent top-level AND conjuncts retain their existing pushdown behavior. Expand coverage in the same change: - ES 5/6/7/8 V1/V2 comparisons across source/docvalue scans, parallel/whole-index scrolling, and batch sizes 1/1024. Add scalar/date/array/JSON, IN/NOT IN, null, raw ES query, residual filtering, aggregation and LIMIT cases; restore every changed session variable. - An independent equivalent-query oracle for nested residual OR, since a V1/V2 comparison alone cannot detect a shared FE error. Compare cardinality for unordered LIMIT rather than document identity across BEs. - FE tests for residual tracking, docvalue eligibility and field-count boundaries, alias/context serialization, shard planning and safe early-stop parameters. - BE tests for source-only fallback, missing/null docvalues and empty arrays, real HTTP empty results, early close and scroll 404/500 cleanup. Add an opt-in authenticated live-reader smoke test. ### Release note Preserve local filtering for Elasticsearch compound predicates that cannot be completely pushed down. ### Check List (For Author) - Test - [x] FE: all 127 ES connector tests pass. The two new nested OR/NOT tests fail before the residual-tracking fix. - [x] BE: all 27 selected ES ASAN tests pass against authenticated Elasticsearch 9.0.0 and 9.5.5, including source/docvalue reads with batch size 1. - [x] FE Checkstyle for the ES module and its reactor dependencies; clang-format 16, clang-tidy on changed C++ lines, and BE build-hygiene checks. - [x] Groovy 4.0.19: reproduce the null port before the fix; verify all four port lookups, suite setup/matrix execution, missing configuration, disabled execution and restoration after an injected query failure. The setup probe substitutes SQL execution and does not validate SQL results. - [ ] Full external ES 5/6/7/8 SQL regression run; requires the external Docker fixtures and a Doris cluster. - Behavior changed: - [x] Yes: residual ES predicates remain enforced; regression setup reads configured ports correctly. - Does this need documentation? - [x] No: existing catalog properties and scanner session variables are unchanged. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
1 parent a626232 commit 3dc7341

5 files changed

Lines changed: 428 additions & 13 deletions

File tree

‎be/test/format_v2/table/es_reader_test.cpp‎

Lines changed: 208 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@
2121
#include <rapidjson/document.h>
2222

2323
#include <atomic>
24+
#include <cstdlib>
25+
#include <set>
2426
#include <tuple>
2527
#include <utility>
2628

@@ -111,6 +113,28 @@ EsScrollStreamFactory factory(const std::shared_ptr<StreamState>& state) {
111113
return std::make_unique<ResponseStream>(state);
112114
};
113115
}
116+
117+
class ScrollHttpHandler final : public HttpHandler {
118+
public:
119+
void handle(HttpRequest* req) override {
120+
if (req->method() == DELETE) {
121+
EXPECT_EQ(req->get_request_body(), R"({"scroll_id":"scroll-token"})");
122+
++clears;
123+
HttpChannel::send_reply(req, HttpStatus::OK, R"({"succeeded":true})");
124+
} else if (searches.fetch_add(1) == 0) {
125+
HttpChannel::send_reply(req, HttpStatus::OK,
126+
page(empty ? "" : R"({"_source":{"id":7}})"));
127+
} else {
128+
HttpChannel::send_reply(req, scroll_status, R"({"error":"scroll failed"})");
129+
}
130+
}
131+
132+
bool empty = false;
133+
HttpStatus scroll_status = HttpStatus::NOT_FOUND;
134+
std::atomic<int> searches {0};
135+
std::atomic<int> clears {0};
136+
};
137+
114138
class EsV2ReaderTest : public testing::Test {
115139
protected:
116140
void SetUp() override {
@@ -130,13 +154,30 @@ class EsV2ReaderTest : public testing::Test {
130154
{"_id":"b","_source":{"id":2,"name":null,"values":[]}})"),
131155
page(R"({"_id":"c","_source":{"id":3,"name":"three","values":null}})")};
132156
}
133-
std::unique_ptr<EsFileReader> reader() {
157+
std::unique_ptr<EsFileReader> reader(bool real_http = false) {
134158
auto props = std::make_shared<io::FileSystemProperties>();
135159
auto file = std::make_unique<io::FileDescription>();
136160
file->path = "es://example/0";
161+
auto result = std::make_unique<EsFileReader>(
162+
props, file, _io, &_profile, _range, _params, _slots,
163+
real_http ? EsScrollStreamFactory {} : factory(_responses));
164+
EXPECT_TRUE(result->init(&_state).ok());
165+
return result;
166+
}
167+
std::unique_ptr<EsFileReader> http_reader(EvHttpServer& server, HttpHandler* handler) {
168+
EXPECT_TRUE(server.register_handler(POST, "/example/_search", handler));
169+
EXPECT_TRUE(server.register_handler(POST, "/_search/scroll", handler));
170+
EXPECT_TRUE(server.register_handler(DELETE, "/_search/scroll", handler));
171+
server.start();
172+
EXPECT_GT(server.get_real_port(), 0);
173+
_range.table_format_params.es_params["host_port"] =
174+
"http://127.0.0.1:" + std::to_string(server.get_real_port());
175+
auto props = std::make_shared<io::FileSystemProperties>();
176+
auto file = std::make_unique<io::FileDescription>();
137177
auto result = std::make_unique<EsFileReader>(props, file, _io, &_profile, _range, _params,
138-
_slots, factory(_responses));
178+
_slots);
139179
EXPECT_TRUE(result->init(&_state).ok());
180+
result->set_batch_size(1);
140181
return result;
141182
}
142183
std::shared_ptr<FileScanRequest> request(std::initializer_list<int> ids) {
@@ -537,4 +578,169 @@ TEST_F(EsV2ReaderTest, FullyFilteredBatchesKeepScanningUntilEof) {
537578
EXPECT_EQ(_responses->closes, 1);
538579
}
539580

581+
TEST_F(EsV2ReaderTest, SourceOnlyFieldsDisableDocvalues) {
582+
for (int source_column : {1, 2}) {
583+
_responses = std::make_shared<StreamState>();
584+
_responses->pages = {page(R"({"_id":"a","_source":{"id":1,"name":"one"}})")};
585+
_params.es_properties["doc_values_mode"] = "1";
586+
_params.__set_es_docvalue_context({{"id", "id"}, {"_id", "_id"}});
587+
auto scan = reader();
588+
auto projection = request({0, source_column});
589+
ASSERT_TRUE(scan->open(projection).ok());
590+
EXPECT_FALSE(_responses->docvalues);
591+
rapidjson::Document query;
592+
query.Parse(_responses->properties.at("query").c_str());
593+
ASSERT_FALSE(query.HasParseError());
594+
EXPECT_TRUE(query.HasMember("_source"));
595+
EXPECT_FALSE(query.HasMember("docvalue_fields"));
596+
auto output = block(*scan, *projection);
597+
size_t rows = 0;
598+
bool eof = false;
599+
ASSERT_TRUE(scan->get_block(&output, &rows, &eof).ok());
600+
ASSERT_EQ(rows, 1);
601+
EXPECT_EQ(int_at(output, 0, 0), 1);
602+
EXPECT_EQ(string_at(output, 1, 0), source_column == 1 ? "one" : "a");
603+
ASSERT_TRUE(scan->close().ok());
604+
EXPECT_EQ(_responses->closes, 1);
605+
}
606+
}
607+
608+
TEST_F(EsV2ReaderTest, DocvaluesPreserveMissingFieldsAndEmptyArrays) {
609+
_params.es_properties["doc_values_mode"] = "1";
610+
_params.__set_es_docvalue_context(
611+
{{"id", "id"}, {"name", "name.keyword"}, {"values", "values"}});
612+
_responses->pages = {page(R"({"fields":{"id":[1],"name.keyword":["one"],"values":[1,3]}},
613+
{"fields":{"id":[2],"values":[]}},
614+
{"fields":{"id":[3],"name.keyword":null,"values":null}})")};
615+
auto scan = reader();
616+
auto projection = request({1, 3, 0});
617+
ASSERT_TRUE(scan->open(projection).ok());
618+
EXPECT_TRUE(_responses->docvalues);
619+
auto output = block(*scan, *projection);
620+
size_t rows = 0;
621+
bool eof = false;
622+
ASSERT_TRUE(scan->get_block(&output, &rows, &eof).ok());
623+
ASSERT_EQ(rows, 3);
624+
for (const auto& column : output) {
625+
EXPECT_EQ(column.column->size(), rows);
626+
}
627+
EXPECT_EQ(string_at(output, 0, 0), "one");
628+
EXPECT_TRUE(output.get_by_position(0).column->is_null_at(1));
629+
EXPECT_TRUE(output.get_by_position(0).column->is_null_at(2));
630+
EXPECT_FALSE(output.get_by_position(1).column->is_null_at(1));
631+
EXPECT_TRUE(output.get_by_position(1).column->is_null_at(2));
632+
const auto& values = assert_cast<const ColumnArray&>(
633+
assert_cast<const ColumnNullable&>(*output.get_by_position(1).column)
634+
.get_nested_column());
635+
EXPECT_EQ(values.get_offsets()[0], 2);
636+
EXPECT_EQ(values.get_offsets()[1], 2);
637+
EXPECT_EQ(values.get_offsets()[2], 2);
638+
EXPECT_EQ(int_at(output, 2, 2), 3);
639+
}
640+
641+
TEST_F(EsV2ReaderTest, RealHttpEarlyCloseClearsInitialScroll) {
642+
ScrollHttpHandler handler;
643+
EvHttpServer server {0};
644+
auto scan = http_reader(server, &handler);
645+
ASSERT_TRUE(scan->open(request({0})).ok());
646+
ASSERT_TRUE(scan->close().ok());
647+
scan.reset();
648+
EXPECT_EQ(handler.searches.load(), 1);
649+
EXPECT_EQ(handler.clears.load(), 1);
650+
}
651+
652+
TEST_F(EsV2ReaderTest, RealHttpEmptyResultClearsScroll) {
653+
ScrollHttpHandler handler;
654+
handler.empty = true;
655+
EvHttpServer server {0};
656+
auto scan = http_reader(server, &handler);
657+
auto projection = request({0});
658+
ASSERT_TRUE(scan->open(projection).ok());
659+
auto output = block(*scan, *projection);
660+
size_t rows = 0;
661+
bool eof = false;
662+
ASSERT_TRUE(scan->get_block(&output, &rows, &eof).ok());
663+
EXPECT_EQ(rows, 0);
664+
EXPECT_TRUE(eof);
665+
ASSERT_TRUE(scan->close().ok());
666+
EXPECT_EQ(handler.searches.load(), 1);
667+
EXPECT_EQ(handler.clears.load(), 1);
668+
}
669+
670+
TEST_F(EsV2ReaderTest, RealHttpScrollErrorsFailAndReleaseContext) {
671+
for (auto status : {HttpStatus::NOT_FOUND, HttpStatus::INTERNAL_SERVER_ERROR}) {
672+
ScrollHttpHandler handler;
673+
handler.scroll_status = status;
674+
EvHttpServer server {0};
675+
auto scan = http_reader(server, &handler);
676+
auto projection = request({0});
677+
ASSERT_TRUE(scan->open(projection).ok());
678+
auto output = block(*scan, *projection);
679+
size_t rows = 0;
680+
bool eof = false;
681+
ASSERT_TRUE(scan->get_block(&output, &rows, &eof).ok());
682+
ASSERT_EQ(rows, 1);
683+
output = block(*scan, *projection);
684+
const auto result = scan->get_block(&output, &rows, &eof);
685+
EXPECT_FALSE(result.ok());
686+
EXPECT_NE(result.to_string().find(status == HttpStatus::NOT_FOUND ? "No search context"
687+
: "scroll failed"),
688+
std::string::npos);
689+
ASSERT_TRUE(scan->close().ok());
690+
scan.reset();
691+
EXPECT_EQ(handler.searches.load(), 2);
692+
EXPECT_EQ(handler.clears.load(), 1);
693+
}
694+
}
695+
696+
// The opt-in fixture has integer id / keyword name and rows (1,"alpha"), (2,"beta"), (3,null).
697+
TEST_F(EsV2ReaderTest, ConfiguredHttpSourceSupportsSourceAndDocvalues) {
698+
const char* host = std::getenv("DORIS_ES_TEST_HOST");
699+
if (host == nullptr) {
700+
GTEST_SKIP() << "Set DORIS_ES_TEST_HOST and DORIS_ES_TEST_INDEX for a live ES smoke test";
701+
}
702+
const char* index = std::getenv("DORIS_ES_TEST_INDEX");
703+
ASSERT_NE(index, nullptr);
704+
_range.table_format_params.es_params["host_port"] = host;
705+
_range.table_format_params.es_params["index"] = index;
706+
_range.table_format_params.es_params["shard_id"] = "-1";
707+
for (const auto& [property, variable] : {std::pair {"user", "DORIS_ES_TEST_USER"},
708+
std::pair {"password", "DORIS_ES_TEST_PASSWORD"}}) {
709+
if (const char* value = std::getenv(variable); value != nullptr) {
710+
_params.es_properties[property] = value;
711+
}
712+
}
713+
_params.__set_es_docvalue_context({{"id", "id"}, {"name", "name"}});
714+
for (bool docvalues : {false, true}) {
715+
_params.es_properties["doc_values_mode"] = docvalues ? "1" : "0";
716+
auto scan = reader(true);
717+
scan->set_batch_size(1);
718+
auto projection = request({1, 0});
719+
const auto opened = scan->open(projection);
720+
ASSERT_TRUE(opened.ok()) << opened;
721+
size_t total = 0;
722+
std::set<int32_t> ids;
723+
bool eof = false;
724+
while (!eof) {
725+
auto output = block(*scan, *projection);
726+
size_t rows = 0;
727+
const auto status = scan->get_block(&output, &rows, &eof);
728+
ASSERT_TRUE(status.ok()) << status;
729+
for (size_t row = 0; row < rows; ++row) {
730+
const int32_t id = int_at(output, 1, row);
731+
ids.insert(id);
732+
if (id == 3) {
733+
EXPECT_TRUE(output.get_by_position(0).column->is_null_at(row));
734+
} else {
735+
EXPECT_EQ(string_at(output, 0, row), id == 1 ? "alpha" : "beta");
736+
}
737+
}
738+
total += rows;
739+
}
740+
EXPECT_EQ(total, 3);
741+
EXPECT_EQ(ids, (std::set<int32_t> {1, 2, 3}));
742+
ASSERT_TRUE(scan->close().ok());
743+
}
744+
}
745+
540746
} // namespace doris::format::es

‎fe/fe-connector/fe-connector-es/src/main/java/org/apache/doris/connector/es/EsQueryDslBuilder.java‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -147,7 +147,9 @@ public static EsQueryDslResult buildQueryDslWithResult(ConnectorExpression filte
147147
List<ConnectorExpression> notPushDownList = new ArrayList<>();
148148
String dsl = toEsDsl(filter, notPushDownList, fieldsContext,
149149
likePushDown, needCompatDateFields, column2typeMap);
150-
if (dsl == null) {
150+
// A nested AND can return a partial DSL while recording an unsupported child. OR/NOT must
151+
// remain local in that case: e.g. NOT(a = 1 AND local_predicate) cannot become NOT(a = 1).
152+
if (dsl == null || !notPushDownList.isEmpty()) {
151153
return new EsQueryDslResult(matchAllQuery(), Collections.singletonList(0));
152154
}
153155
return new EsQueryDslResult(dsl, Collections.emptyList());

‎fe/fe-connector/fe-connector-es/src/test/java/org/apache/doris/connector/es/EsQueryDslBuilderTest.java‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,18 +18,73 @@
1818
package org.apache.doris.connector.es;
1919

2020
import org.apache.doris.connector.spi.ConnectorType;
21+
import org.apache.doris.connector.spi.pushdown.ConnectorAnd;
2122
import org.apache.doris.connector.spi.pushdown.ConnectorColumnRef;
2223
import org.apache.doris.connector.spi.pushdown.ConnectorComparison;
24+
import org.apache.doris.connector.spi.pushdown.ConnectorExpression;
25+
import org.apache.doris.connector.spi.pushdown.ConnectorFunctionCall;
2326
import org.apache.doris.connector.spi.pushdown.ConnectorLike;
2427
import org.apache.doris.connector.spi.pushdown.ConnectorLiteral;
28+
import org.apache.doris.connector.spi.pushdown.ConnectorNot;
29+
import org.apache.doris.connector.spi.pushdown.ConnectorOr;
2530

2631
import org.junit.jupiter.api.Assertions;
2732
import org.junit.jupiter.api.Test;
2833

34+
import java.util.Arrays;
2935
import java.util.Collections;
3036

3137
public class EsQueryDslBuilderTest {
3238

39+
private static ConnectorExpression keywordEquals(String value) {
40+
return new ConnectorComparison(ConnectorComparison.Operator.EQ,
41+
new ConnectorColumnRef("name", ConnectorType.of("VARCHAR")), ConnectorLiteral.ofString(value));
42+
}
43+
44+
private static ConnectorExpression partiallyPushableAnd() {
45+
ConnectorExpression local = new ConnectorFunctionCall("unsupported_predicate", ConnectorType.of("BOOLEAN"),
46+
Collections.singletonList(new ConnectorColumnRef("name", ConnectorType.of("VARCHAR"))));
47+
return new ConnectorAnd(Arrays.asList(keywordEquals("one"), local));
48+
}
49+
50+
private static EsQueryDslResult buildWithTracking(ConnectorExpression expression) {
51+
return EsQueryDslBuilder.buildQueryDslWithResult(expression, Collections.emptyMap(),
52+
Collections.singletonMap("name", "keyword"), true, Collections.emptyList());
53+
}
54+
55+
@Test
56+
void partialAndRetainsOnlyItsUnsupportedConjunct() {
57+
EsQueryDslResult result = buildWithTracking(partiallyPushableAnd());
58+
Assertions.assertEquals(Collections.singletonList(1), result.getNotPushedIndices());
59+
Assertions.assertEquals("{\"term\":{\"name\":\"one\"}}", result.getQueryDsl());
60+
}
61+
62+
@Test
63+
void orWithPartiallyPushableAndRemainsLocal() {
64+
EsQueryDslResult result = buildWithTracking(new ConnectorOr(
65+
Arrays.asList(partiallyPushableAnd(), keywordEquals("two"))));
66+
Assertions.assertEquals(Collections.singletonList(0), result.getNotPushedIndices());
67+
Assertions.assertEquals(EsQueryDslBuilder.matchAllQuery(), result.getQueryDsl());
68+
}
69+
70+
@Test
71+
void notWithPartiallyPushableAndRemainsLocal() {
72+
EsQueryDslResult result = buildWithTracking(new ConnectorNot(partiallyPushableAnd()));
73+
Assertions.assertEquals(Collections.singletonList(0), result.getNotPushedIndices());
74+
Assertions.assertEquals(EsQueryDslBuilder.matchAllQuery(), result.getQueryDsl());
75+
}
76+
77+
@Test
78+
void fullyPushableOrAndNotRemainPushed() {
79+
for (ConnectorExpression expression : Arrays.asList(
80+
new ConnectorOr(Arrays.asList(keywordEquals("one"), keywordEquals("two"))),
81+
new ConnectorNot(keywordEquals("one")))) {
82+
EsQueryDslResult result = buildWithTracking(expression);
83+
Assertions.assertTrue(result.getNotPushedIndices().isEmpty());
84+
Assertions.assertNotEquals(EsQueryDslBuilder.matchAllQuery(), result.getQueryDsl());
85+
}
86+
}
87+
3388
@Test
3489
public void testRegexpProducesRegexpQuery() {
3590
// SQL: WHERE name REGEXP '^test.*'

0 commit comments

Comments
 (0)