Skip to content

Commit b246692

Browse files
committed
[chore](fe) Merge master and preserve cloud transaction tests
### What problem does this PR solve? Issue Number: close apache#67099 Related PR: apache#67813 Problem Summary: Master and the visible-commit cache refresh fix add independent tests at the same location in CloudGlobalTransactionMgrTest. Merge master and retain both the exclusive transaction watermark test and all version-refresh tests. Preserve the original cache refresh patch and upstream production changes. ### Release note None ### Check List (For Author) - Test: Unit Test - 30 tests passed via run-fe-ut.sh (CloudGlobalTransactionMgrTest and CloudPartitionTest). - FE Checkstyle: zero violations; diff checks against merged master passed. - No local product packaging, cluster deployment or SQL regression run. - Behavior changed: No. Conflict resolution preserves both existing changes. - Does this need documentation: No
2 parents f28dc84 + 9f3b084 commit b246692

641 files changed

Lines changed: 38272 additions & 32189 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/third_party_review.yml‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,10 +78,26 @@ jobs:
7878
# every pull request that adds a module with a JUnit 5 test fails on a licence the
7979
# project has already approved. junit:junit is EPL-1.0, an ASF Category B licence, and
8080
# the JUnit 4 tests that need it run through junit-vintage-engine.
81+
#
82+
# com.sun.jersey:jersey-client (CDDL-1.1 OR GPL-2.0 WITH Classpath-exception-2.0):
83+
# CDDL-1.1 is ASF Category B, usable in binary form as long as the distribution records
84+
# it, and dist/LICENSE-dist.txt already lists com.sun.jersey:jersey-client under
85+
# "Common Development and Distribution License (CDDL) v1.1". The FE also already ships
86+
# jersey-core under that same dual license; only the client half is missing, because
87+
# #37575 excluded the jersey-bundle Ranger declares against. Ranger's
88+
# RangerAdminRESTClient is Jersey 1.x, so the Ranger authorization plugins need it to
89+
# reach a Ranger admin at all. The version the build resolves is ${jersey.version},
90+
# 1.19.4, matching the jersey-core the FE already carries.
91+
# javax.ws.rs:jsr311-api (CDDL-1.0): the JAX-RS 1.1 API jersey-client compiles and links
92+
# against, so it comes with it into the distribution. Same Category B reasoning, recorded
93+
# under the "CDDL v1.0" section of dist/LICENSE-dist.txt, which is the version that
94+
# artifact itself declares.
8195
allow-dependencies-licenses: >-
8296
8397
pkg:maven/org.junit.jupiter/junit-jupiter,
84-
pkg:maven/junit/junit
98+
pkg:maven/junit/junit,
99+
pkg:maven/com.sun.jersey/jersey-client,
100+
pkg:maven/javax.ws.rs/jsr311-api
85101
# ([String]). Acknowledged advisories that must not fail the review (optional)
86102
# org.codehaus.jackson:jackson-mapper-asl (GHSA-c27h-mcmw-48hv, GHSA-r6j9-8759-g62w):
87103
# legacy Jackson 1.x is EOL and neither advisory has a fixed version. Hive's metastore

‎.licenserc.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,12 @@ header:
5858
- "fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt"
5959
- "fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt"
6060
- "fe/fe-authentication/fe-authentication-spi/src/test/resources/authentication-plugin-surface.txt"
61+
- "fe/fe-authorization/fe-authorization-spi/src/test/resources/authorization-plugin-surface.txt"
6162
- "fe/fe-core/src/test/resources/lineage-plugin-surface.txt"
63+
# Golden access-control decision matrix. AccessControlBehaviorBaselineTest regenerates
64+
# the whole file and compares it to this one line for line, so a header would read back
65+
# as extra rows; it prints the regenerated path on mismatch instead.
66+
- "fe/fe-core/src/test/resources/access-control-behavior-baseline.txt"
6267
# Connector plugin settings templates. build.sh seeds each connector's live
6368
# <name>.conf from its template verbatim (cp -n), so the template's content IS
6469
# the file an administrator edits in plugins/connector/<dir>/. Matched by name

‎be/AGENTS.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,3 +68,57 @@ past a gate is never to bypass it, but one of:
6868
skip entry present?
6969
- [ ] `git grep` for a deleted/renamed header in skip lists and gate tables:
7070
stale entries fail configure loudly — fix them in the same PR.
71+
72+
## Preserve the glibc 2.17 runtime baseline
73+
74+
Treat glibc 2.17 compatibility as part of the ABI contract of a production Doris BE release.
75+
The final `doris_be` executable and the shared libraries shipped with it must not contain an
76+
undefined symbol-version requirement newer than `GLIBC_2.17` on any supported architecture.
77+
A successful build on a newer Linux distribution is not sufficient evidence of compatibility.
78+
79+
The 2.17 baseline is specifically required for CentOS 7, which ships glibc 2.17 and still has a
80+
large Doris deployment base. Do not dismiss a `GLIBC_2.18` requirement as a minor version increase:
81+
the CentOS 7 dynamic loader cannot satisfy it, so `doris_be` fails during process startup before any
82+
application-level fallback can run. Requiring even one `GLIBC_2.18` symbol therefore drops CentOS 7
83+
compatibility and is a release-blocking regression.
84+
85+
Apply this requirement when reviewing changes to BE source code, CMake or linker options, the LDB
86+
toolchain/sysroot, third-party libraries, Rust crates or static archives, and
87+
`be/src/glibc-compatibility/`. A sysroot helps constrain symbol selection, but it does not by itself
88+
guarantee the final ABI: an archive built against newer headers/libraries or a symbol resolved from
89+
the wrong library or in the wrong link order can still add a newer GLIBC requirement.
90+
91+
For every relevant release build, inspect the completed ELF artifacts rather than only their
92+
individual inputs. At minimum, audit `doris_be` with one of the following and investigate every
93+
version greater than 2.17:
94+
95+
```shell
96+
objdump -T path/to/doris_be \
97+
| grep -oE 'GLIBC_[0-9]+(\.[0-9]+)*' \
98+
| sort -Vu
99+
100+
readelf --version-info --wide path/to/doris_be
101+
```
102+
103+
Use `objdump -T path/to/doris_be | grep GLIBC_...` or `readelf --dyn-syms --wide` to identify the
104+
specific undefined symbols. Repeat the audit for shipped BE shared libraries; a clean executable
105+
does not make a bundled library with a newer requirement safe. When a glibc 2.17 environment is
106+
available, also start and exercise the release artifact there.
107+
108+
Known accidental bindings that deserve explicit attention include `logf`, `log2f`, and `expf` at
109+
`GLIBC_2.27`, `copy_file_range` at `GLIBC_2.27`, `strfromf128` at `GLIBC_2.26`, and
110+
`__cxa_thread_atexit_impl` at `GLIBC_2.18`. These version tags describe what the final binary asks
111+
the dynamic loader to provide; they do not necessarily mean that the underlying API was first
112+
created in that glibc release.
113+
114+
When a dependency needs a post-2.17 libc API, use an existing Doris compatibility implementation
115+
or add a narrowly scoped implementation with matching semantics. If a raw syscall is appropriate,
116+
preserve the caller's expected `ENOSYS` or other fallback behavior on old kernels. If a compatible
117+
definition lives in a static archive or object library, ensure the final link actually extracts and
118+
binds that definition; verify this in the completed ELF symbol table. Do not treat an implementation
119+
present in an archive as proof that it was selected.
120+
121+
Do not solve a compatibility regression by bundling a newer glibc, changing `LD_LIBRARY_PATH` to a
122+
private libc, or allowing host include/library paths to leak into the build. Such changes can mix an
123+
incompatible loader, libc, and NSS/runtime components. Fix the build input, symbol implementation,
124+
or link selection, then repeat the final-artifact audit.

‎be/CMakeLists.txt‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -855,7 +855,17 @@ if (GLIBC_COMPATIBILITY)
855855
# Keep lance_c here instead of COMMON_THIRDPARTY: placing its required libm there
856856
# would resolve -lm symbol before Doris compatibility is scanned, preventing
857857
# the linker from selecting Doris' optimized implementations.
858+
# lance_c can also introduce single-precision math references after the
859+
# compatibility archive has already been scanned. As ClickHouse does for its
860+
# compatibility archive, force the GLIBC_2.27 math symbols to be undefined up
861+
# front so the linker extracts Doris' musl implementations instead of binding
862+
# late references to the toolchain's libm.
858863
set(DORIS_LINK_LIBS ${DORIS_LINK_LIBS}
864+
-Wl,-u,logf
865+
-Wl,-u,powf
866+
-Wl,-u,expf
867+
-Wl,-u,exp2f
868+
-Wl,-u,log2f
859869
glibc-compatibility-explicit
860870
glibc-compatibility
861871
-lm

‎be/benchmark/benchmark_variant_segment.hpp‎

Lines changed: 13 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@
6969
#include "storage/segment/column_meta_accessor.h"
7070
#include "storage/segment/segment.h"
7171
#include "storage/segment/segment_loader.h"
72-
#include "storage/segment/segment_writer.h"
72+
#include "storage/segment/vertical_segment_writer.h"
7373
#include "storage/storage_engine.h"
7474
#include "storage/tablet/tablet.h"
7575
#include "storage/tablet/tablet_column_object_pool.h"
@@ -81,12 +81,12 @@ namespace doris::variant_segment_benchmark {
8181
namespace {
8282

8383
// P0 workload: 200 top-level BIGINT paths, with one hot path and 29 additional paths per row.
84-
// Ingest cases parse the same canonical JSON before SegmentWriter. Read cases use the same
84+
// Ingest cases parse the same canonical JSON before VerticalSegmentWriter. Read cases use the same
8585
// V1-written physical segment and only switch the requested V1/V2 output representation. Input
8686
// generation, buffer destruction, warmup, checksum, physical-layout validation, and route
8787
// validation are paused. Scan-and-rewrite cases query key plus whole Variant from that prevalidated
8888
// source, then time scan/writer initialization, read, append, and destination finalize; destination
89-
// validation is paused. Ingest rotates SegmentWriter at
89+
// validation is paused. Ingest rotates VerticalSegmentWriter at
9090
// DORIS_VARIANT_BENCHMARK_ROWS_PER_SEGMENT; read cases retain their historical single-segment
9191
// semantics.
9292
constexpr uint32_t DEFAULT_ROWS = 1'000'000;
@@ -593,14 +593,14 @@ class VariantSegmentBenchmarkData {
593593
rowset_context.tablet_schema = destination_schema;
594594
rowset_context.tablet_path = _directory;
595595

596-
segment_v2::SegmentWriterOptions writer_options;
596+
segment_v2::VerticalSegmentWriterOptions writer_options;
597597
writer_options.num_rows_per_block = BATCH_ROWS;
598598
writer_options.max_rows_per_segment = _rows_per_segment;
599599
writer_options.compression_type = CompressionTypePB::LZ4;
600600
writer_options.rowset_ctx = &rowset_context;
601601
writer_options.write_type = DataWriteType::TYPE_DIRECT;
602-
segment_v2::SegmentWriter writer(file_writer.get(), 0, destination_schema, nullptr, nullptr,
603-
writer_options, nullptr);
602+
segment_v2::VerticalSegmentWriter writer(file_writer.get(), 0, destination_schema, nullptr,
603+
nullptr, writer_options, nullptr);
604604
RETURN_IF_ERROR(writer.init());
605605
result->writer_init_ns += elapsed_ns(writer_init_start);
606606

@@ -638,8 +638,8 @@ class VariantSegmentBenchmarkData {
638638
}
639639

640640
const auto finalize_start = std::chrono::steady_clock::now();
641-
RETURN_IF_ERROR(writer.finalize(&result->destination_segment_bytes,
642-
&result->destination_index_bytes));
641+
RETURN_IF_ERROR(writer.finalize_columns(&result->destination_index_bytes));
642+
RETURN_IF_ERROR(writer.finalize_footer(&result->destination_segment_bytes));
643643
result->finalize_ns += elapsed_ns(finalize_start);
644644
result->source_segment_bytes = prepared.fixture->segment_bytes;
645645
return Status::OK();
@@ -668,15 +668,15 @@ class VariantSegmentBenchmarkData {
668668
rowset_context.tablet_schema = schema;
669669
rowset_context.tablet_path = _directory;
670670

671-
segment_v2::SegmentWriterOptions options;
671+
segment_v2::VerticalSegmentWriterOptions options;
672672
options.num_rows_per_block = BATCH_ROWS;
673673
options.max_rows_per_segment = _rows_per_segment;
674674
options.compression_type = CompressionTypePB::LZ4;
675675
options.rowset_ctx = &rowset_context;
676676
options.write_type = DataWriteType::TYPE_DIRECT;
677677

678-
segment_v2::SegmentWriter writer(file_writer.get(), segment_id, schema, nullptr,
679-
nullptr, options, nullptr);
678+
segment_v2::VerticalSegmentWriter writer(file_writer.get(), segment_id, schema,
679+
nullptr, nullptr, options, nullptr);
680680
RETURN_IF_ERROR(writer.init());
681681
result->init_ns += elapsed_ns(init_start);
682682

@@ -720,7 +720,8 @@ class VariantSegmentBenchmarkData {
720720
uint64_t bytes = 0;
721721
uint64_t index_bytes = 0;
722722
const auto finalize_start = std::chrono::steady_clock::now();
723-
RETURN_IF_ERROR(writer.finalize(&bytes, &index_bytes));
723+
RETURN_IF_ERROR(writer.finalize_columns(&index_bytes));
724+
RETURN_IF_ERROR(writer.finalize_footer(&bytes));
724725
result->finalize_ns += elapsed_ns(finalize_start);
725726
result->segment_bytes += bytes;
726727
result->index_bytes += index_bytes;
Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
if (NOT DEFINED ARTIFACT OR NOT EXISTS "${ARTIFACT}")
19+
message(FATAL_ERROR "Cannot check glibc compatibility: '${ARTIFACT}' does not exist")
20+
endif()
21+
22+
if (NOT DEFINED BASELINE OR BASELINE STREQUAL "")
23+
message(FATAL_ERROR "Cannot check glibc compatibility: BASELINE is not set")
24+
endif()
25+
26+
if (NOT DEFINED OBJDUMP OR OBJDUMP STREQUAL "" OR NOT EXISTS "${OBJDUMP}")
27+
find_program(OBJDUMP NAMES objdump llvm-objdump REQUIRED)
28+
endif()
29+
30+
execute_process(
31+
COMMAND "${OBJDUMP}" -T "${ARTIFACT}"
32+
RESULT_VARIABLE objdump_result
33+
OUTPUT_VARIABLE dynamic_symbol_table
34+
ERROR_VARIABLE objdump_error)
35+
36+
if (NOT objdump_result EQUAL 0)
37+
message(FATAL_ERROR
38+
"Cannot read the dynamic symbol table from '${ARTIFACT}': ${objdump_error}")
39+
endif()
40+
41+
string(REPLACE "\n" ";" dynamic_symbol_lines "${dynamic_symbol_table}")
42+
set(incompatible_symbols)
43+
44+
foreach (symbol_line IN LISTS dynamic_symbol_lines)
45+
# Only undefined symbols are runtime requirements on the target system.
46+
if (symbol_line MATCHES "\\*UND\\*")
47+
string(REGEX MATCH "GLIBC_[0-9]+(\\.[0-9]+)+" symbol_version "${symbol_line}")
48+
if (NOT "${symbol_version}" STREQUAL "")
49+
string(REGEX REPLACE "^GLIBC_" "" numeric_version "${symbol_version}")
50+
if ("${numeric_version}" VERSION_GREATER "${BASELINE}")
51+
string(REGEX MATCH "[^ \t]+$" symbol_name "${symbol_line}")
52+
list(APPEND incompatible_symbols "${symbol_version} ${symbol_name}")
53+
endif()
54+
endif()
55+
endif()
56+
endforeach()
57+
58+
if (NOT "${incompatible_symbols}" STREQUAL "")
59+
list(REMOVE_DUPLICATES incompatible_symbols)
60+
list(SORT incompatible_symbols)
61+
string(JOIN "\n " formatted_symbols ${incompatible_symbols})
62+
message(FATAL_ERROR
63+
"${ARTIFACT} requires glibc symbols newer than GLIBC_${BASELINE}:\n"
64+
" ${formatted_symbols}\n"
65+
"The production BE must remain runnable on CentOS 7 (glibc ${BASELINE}).")
66+
endif()
67+
68+
message(STATUS
69+
"glibc compatibility check passed: ${ARTIFACT} requires no symbols newer than "
70+
"GLIBC_${BASELINE}")

‎be/src/cloud/cloud_schema_change_job.cpp‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -413,7 +413,6 @@ Status CloudSchemaChangeJob::_convert_historical_rowsets(const SchemaChangeParam
413413
}
414414

415415
context.write_type = DataWriteType::TYPE_SCHEMA_CHANGE;
416-
// TODO if support VerticalSegmentWriter, also need to handle cluster key primary key index
417416
bool vertical = false;
418417
if (sc_sorting && !_new_tablet->tablet_schema()->cluster_key_uids().empty()) {
419418
// see VBaseSchemaChangeWithSorting::_external_sorting

0 commit comments

Comments
 (0)