Skip to content

Commit 31945b1

Browse files
committed
[fix](paimon) Scope schema retention and fence fallback snapshots
### What problem does this PR solve? Related PR: #67904 Problem Summary: Latest-schema retention was implicitly enabled by a schema ID, changing ordinary metadata publication for unrelated connectors. Make retention an explicit Paimon opt-in and remove the collateral Iceberg/Hive changes. Keep schema and physical coordinates in one immutable statement pin, including the fallback branch data fence, so reloads cannot silently select replacement data. Preserve upstream snapshot-aware partition listing when integrating master. ### Release note Preserve Paimon statement schemas and fallback data coordinates. The connector plugin API advances to 8.0 for explicit latest-schema publication. ### Check List (For Author) - Test: Related FE/connector suites, full SPI suite, FE build, standalone FE Checkstyle, and Native/JNI regression queries. - Behavior changed: Yes; only explicitly opted-in latest pins publish retained schemas, and changed fallback generations are rejected. - Does this need documentation: No; existing Paimon query semantics are restored.
2 parents 499a96e + 79fcbea commit 31945b1

1,584 files changed

Lines changed: 63277 additions & 58843 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.asf.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ github:
6161
- Build Third Party Libraries (macOS)
6262
- Build Third Party Libraries (macOS-arm64)
6363
- COMPILE (DORIS_COMPILE)
64+
- code-review
6465
- Cloud UT (Doris Cloud UT)
6566
- performance (Doris Performance)
6667
required_pull_request_reviews:

‎.github/CODEOWNERS‎

Lines changed: 136 additions & 132 deletions
Large diffs are not rendered by default.

‎.github/workflows/build-thirdparty.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -196,6 +196,11 @@ jobs:
196196
#./build-thirdparty.sh -j "$(nproc)"
197197
./build-thirdparty.sh -j 2
198198
199+
# CI has no 64 KiB-page aarch64 host; this drives the libunwind just built
200+
# through an emulated one, so it also runs on this x86_64 4 KiB runner.
201+
cd "${GITHUB_WORKSPACE}"
202+
thirdparty/test/libunwind-page-size-test.sh
203+
199204
build_macos:
200205
name: Build Third Party Libraries (macOS)
201206
needs: changes

‎.github/workflows/code-review-runner.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -60,9 +60,9 @@ jobs:
6060
head_sha: ${{ steps.review_inputs.outputs.head_sha }}
6161
manage_status: ${{ steps.review_inputs.outputs.manage_status }}
6262
pr_number: ${{ steps.review_inputs.outputs.pr_number }}
63-
# Pre-finalization steps can use 188 minutes and auth sync can use 8 more,
63+
# Pre-finalization steps can use 218 minutes and auth sync can use 8 more,
6464
# leaving 12 minutes for runner setup and post-job cleanup.
65-
timeout-minutes: 208
65+
timeout-minutes: 238
6666
if: >-
6767
inputs.pr_number != '' ||
6868
(
@@ -711,7 +711,7 @@ jobs:
711711

712712
- name: Run automated code review
713713
id: review
714-
timeout-minutes: 90
714+
timeout-minutes: 120
715715
continue-on-error: true
716716
env:
717717
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎.github/workflows/third_party_review.yml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -76,8 +76,11 @@ jobs:
7676
# org.junit.jupiter:junit-jupiter is EPL-2.0, a licence allow-licenses above already
7777
# carries, but GitHub's dependency graph reports it as LicenseRef-bad-non-standard, so
7878
# every pull request that adds a module with a JUnit 5 test fails on a licence the
79-
# project has already approved. junit:junit is EPL-1.0, an ASF Category B licence, and
80-
# the JUnit 4 tests that need it run through junit-vintage-engine.
79+
# project has already approved. junit:junit is EPL-1.0, an ASF Category B licence. No
80+
# pom declares it any more - the fe reactor runs no JUnit 4 test and carries no vintage
81+
# engine - but it still arrives transitively via guava-testlib (fe-core, test) and
82+
# hive-exec -> zookeeper -> jline (hive-udf, provided), and the standalone samples/
83+
# projects declare it directly, so the exception stays.
8184
#
8285
# com.sun.jersey:jersey-client (CDDL-1.1 OR GPL-2.0 WITH Classpath-exception-2.0):
8386
# CDDL-1.1 is ASF Category B, usable in binary form as long as the distribution records

‎.licenserc.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,10 +60,16 @@ header:
6060
- "fe/fe-authentication/fe-authentication-spi/src/test/resources/authentication-plugin-surface.txt"
6161
- "fe/fe-authorization/fe-authorization-spi/src/test/resources/authorization-plugin-surface.txt"
6262
- "fe/fe-core/src/test/resources/lineage-plugin-surface.txt"
63+
- "fe/be-java-extensions/jni-spi/src/test/resources/jni-plugin-surface.txt"
6364
# Golden access-control decision matrix. AccessControlBehaviorBaselineTest regenerates
6465
# the whole file and compares it to this one line for line, so a header would read back
6566
# as extra rows; it prints the regenerated path on mismatch instead.
6667
- "fe/fe-core/src/test/resources/access-control-behavior-baseline.txt"
68+
# Golden protocol traffic. MysqlPacketGoldenTest and FlightResultGoldenTest regenerate
69+
# each file in full and compare it to this one byte for byte, so a header would read back
70+
# as unexpected leading content; both tests print the regeneration command on mismatch.
71+
- "fe/fe-core/src/test/resources/protocol-golden/mysql-packets.txt"
72+
- "fe/fe-core/src/test/resources/protocol-golden/flight-results.txt"
6773
# Connector plugin settings templates. build.sh seeds each connector's live
6874
# <name>.conf from its template verbatim (cp -n), so the template's content IS
6975
# the file an administrator edits in plugins/connector/<dir>/. Matched by name

‎be/CMakeLists.txt‎

Lines changed: 18 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -636,7 +636,6 @@ else()
636636
else()
637637
include_directories(${DORIS_JAVA_HOME}/include/darwin)
638638
endif()
639-
add_definitions("-DLIBJVM")
640639
endif()
641640

642641
if (NOT OS_MACOSX)
@@ -686,29 +685,21 @@ set(COMMON_THIRDPARTY
686685
${COMMON_THIRDPARTY}
687686
)
688687

689-
if ((ARCH_AMD64 OR ARCH_AARCH64) AND OS_LINUX)
690-
add_library(hadoop_hdfs STATIC IMPORTED)
691-
set_target_properties(hadoop_hdfs PROPERTIES IMPORTED_LOCATION ${THIRDPARTY_DIR}/lib/hadoop_hdfs_3_4/native/libhdfs.a)
688+
add_library(hadoop_hdfs STATIC IMPORTED)
689+
set_target_properties(hadoop_hdfs PROPERTIES IMPORTED_LOCATION ${THIRDPARTY_DIR}/lib/hadoop_hdfs_3_4/native/libhdfs.a)
692690

693-
set(COMMON_THIRDPARTY
694-
${COMMON_THIRDPARTY}
695-
hadoop_hdfs
696-
)
697-
add_definitions(-DUSE_HADOOP_HDFS)
698-
# USE_DORIS_HADOOP_HDFS means use hadoop deps from doris-thirdparty.
699-
# the hadoop deps from doris-thirdparty contains some modification diff from the standard hadoop, such as log interface
700-
add_definitions(-DUSE_DORIS_HADOOP_HDFS)
701-
else()
702-
add_library(hdfs3 STATIC IMPORTED)
703-
set_target_properties(hdfs3 PROPERTIES IMPORTED_LOCATION ${THIRDPARTY_DIR}/lib/libhdfs3.a)
704-
705-
# TODO: use arm hadoop hdfs to replace this
706-
set(COMMON_THIRDPARTY
707-
${COMMON_THIRDPARTY}
708-
hdfs3
709-
)
710-
add_definitions(-DUSE_LIBHDFS3)
711-
endif()
691+
set(COMMON_THIRDPARTY
692+
${COMMON_THIRDPARTY}
693+
hadoop_hdfs
694+
)
695+
# USE_DORIS_HADOOP_HDFS means use hadoop deps from doris-thirdparty.
696+
# the hadoop deps from doris-thirdparty contains some modification diff from the standard hadoop, such as log interface.
697+
# Unconditional since the libhdfs3 alternative was removed: there is only one libhdfs left, and
698+
# every #ifdef on it went with the alternative - nothing in be/ tests it any more. Kept as a
699+
# definition rather than deleted because it is part of how this build identifies itself: a
700+
# doris-thirdparty libhdfs, whose hdfsSetLogger extension io/hdfs_builder.cpp uses unconditionally
701+
# for that reason.
702+
add_definitions(-DUSE_DORIS_HADOOP_HDFS)
712703

713704
if (absl_FOUND)
714705
set(COMMON_THIRDPARTY
@@ -1164,6 +1155,10 @@ if(NOT BUILD_BENCHMARK)
11641155
${BASE_DIR}/../conf/asan_suppr.conf
11651156
${BASE_DIR}/../conf/lsan_suppr.conf
11661157
${BASE_DIR}/../conf/ubsan_suppr.conf
1158+
# Configures the hadoop on BE's system class path, which is what C++ libhdfs uses; the
1159+
# Java plugins log through JniLogging instead. bin/start_be.sh names it with
1160+
# -Dlog4j2.configurationFile, and only when it is present.
1161+
${BASE_DIR}/../conf/hadoop_log4j2.properties
11671162
DESTINATION ${OUTPUT_DIR}/conf)
11681163
endif()
11691164

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#pragma once
19+
20+
#include <benchmark/benchmark.h>
21+
22+
#include <random>
23+
24+
#include "core/data_type/data_type_number.h"
25+
#include "exprs/vcase_expr.h"
26+
27+
namespace doris {
28+
29+
// Keep the production result assembly visible in disassembly as well as timing it.
30+
template <typename IndexType, PrimitiveType PT>
31+
NO_INLINE ColumnPtr run_case_selection(const VCaseExpr& expr, const IndexType* indices,
32+
std::vector<ColumnPtr>& columns, size_t rows) {
33+
return expr._execute_update_result_impl<IndexType, ColumnVector<PT>>(indices, columns, rows);
34+
}
35+
36+
template <typename IndexType, PrimitiveType PT>
37+
void BM_CaseFloatSelection(benchmark::State& state) {
38+
const size_t rows = state.range(0);
39+
const size_t branches = state.range(1);
40+
const auto distribution = state.range(2);
41+
TExprNode node;
42+
node.__set_node_type(TExprNodeType::CASE_EXPR);
43+
node.__set_type(DataTypeNumber<PT>().to_thrift());
44+
node.__set_is_nullable(false);
45+
node.case_expr.__set_has_else_expr(true);
46+
VCaseExpr expr(node);
47+
std::mt19937 rng(20260912);
48+
std::vector<IndexType> indices(rows);
49+
for (size_t row = 0; row < rows; ++row) {
50+
// Interleaved, random, or 99% ELSE. All inputs are finite for before/after comparison.
51+
indices[row] = static_cast<IndexType>(distribution == 0 ? row % branches
52+
: distribution == 1
53+
? rng() % branches
54+
: (row % 100 == 0 ? rng() % branches : 0));
55+
}
56+
std::vector<ColumnPtr> columns;
57+
for (size_t branch = 0; branch < branches; ++branch) {
58+
auto column = ColumnVector<PT>::create(rows);
59+
for (size_t row = 0; row < rows; ++row) {
60+
column->get_data()[row] =
61+
static_cast<typename ColumnVector<PT>::value_type>((row + branch + 1) * 0.125);
62+
}
63+
columns.push_back(std::move(column));
64+
}
65+
for (auto _ : state) {
66+
auto result = run_case_selection<IndexType, PT>(expr, indices.data(), columns, rows);
67+
benchmark::DoNotOptimize(result);
68+
}
69+
state.SetItemsProcessed(state.iterations() * rows);
70+
}
71+
72+
inline void case_float_arguments(benchmark::internal::Benchmark* benchmark) {
73+
for (int64_t rows : {31, 4096, 65536}) {
74+
for (int64_t branches : {3, 16}) {
75+
for (int64_t distribution : {0, 1, 2}) {
76+
benchmark->Args({rows, branches, distribution});
77+
}
78+
}
79+
}
80+
}
81+
82+
BENCHMARK_TEMPLATE(BM_CaseFloatSelection, uint8_t, TYPE_FLOAT)->Apply(case_float_arguments);
83+
BENCHMARK_TEMPLATE(BM_CaseFloatSelection, uint8_t, TYPE_DOUBLE)->Apply(case_float_arguments);
84+
BENCHMARK_TEMPLATE(BM_CaseFloatSelection, uint16_t, TYPE_FLOAT)
85+
->Args({4096, 257, 0})
86+
->Args({4096, 257, 1})
87+
->Args({4096, 257, 2});
88+
BENCHMARK_TEMPLATE(BM_CaseFloatSelection, uint16_t, TYPE_DOUBLE)
89+
->Args({4096, 257, 0})
90+
->Args({4096, 257, 1})
91+
->Args({4096, 257, 2});
92+
93+
} // namespace doris

‎be/benchmark/benchmark_main.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
#include "benchmark_arrow_validation.hpp"
2727
#include "benchmark_binary_arithmetic.hpp"
2828
#include "benchmark_bit_pack.hpp"
29+
#include "benchmark_case_expr.hpp"
2930
#include "benchmark_column_array_view.hpp"
3031
#include "benchmark_column_array_view_distance.hpp"
3132
#include "benchmark_fastunion.hpp"

‎be/src/agent/task_worker_pool.cpp‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@
9999
#include "util/brpc_client_cache.h"
100100
#include "util/debug_points.h"
101101
#include "util/jni-util.h"
102+
#include "util/jni_plugin_registry.h"
102103
#include "util/mem_info.h"
103104
#include "util/random.h"
104105
#include "util/s3_util.h"
@@ -2592,10 +2593,13 @@ void clean_udf_cache_callback(const TAgentTaskRequest& req) {
25922593
const bool drop_by_function_id = clean_req.__isset.function_id;
25932594

25942595
if (doris::config::enable_java_support) {
2596+
// The id, not just the signature: it is what the java-udf plugin keys its compiled
2597+
// classes by, because the signature carries no database and FE renders a variadic one
2598+
// differently here than on the requests that execute the function.
25952599
WARN_IF_ERROR(
2596-
Jni::Util::clean_udf_class_load_cache(
2597-
clean_req.function_signature,
2598-
drop_by_function_id ? clean_req.function_id : 0),
2600+
Jni::PluginRegistry::clean_udf_cache(
2601+
drop_by_function_id ? clean_req.function_id : 0,
2602+
clean_req.function_signature),
25992603
fmt::format("failed to clean Java UDF cache, function_signature={}, function_id={}",
26002604
clean_req.function_signature, clean_req.function_id));
26012605
}

0 commit comments

Comments
 (0)