Skip to content

fix push down - #2

Closed
suxiaogang223 wants to merge 2 commits into
kaka11chen:orc_late_mat_without_sargfrom
suxiaogang223:fix_push_down
Closed

fix push down#2
suxiaogang223 wants to merge 2 commits into
kaka11chen:orc_late_mat_without_sargfrom
suxiaogang223:fix_push_down

Conversation

@suxiaogang223

@suxiaogang223 suxiaogang223 commented Apr 8, 2025

Copy link
Copy Markdown

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@github-actions

github-actions Bot commented Apr 8, 2025

Copy link
Copy Markdown

sh-checker report

To get the full details, please check in the job output.

shellcheck errors 

'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In bin/flight_record_fe.sh line 47:
FE_PID=$(${JAVA_HOME}/bin/jps | grep DorisFE | awk '{print $1}')
         ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
FE_PID=$("${JAVA_HOME}"/bin/jps | grep DorisFE | awk '{print $1}')


In bin/profile_fe.sh line 47:
FE_PID=$(${JAVA_HOME}/bin/jps | grep DorisFE | awk '{print $1}')
         ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
FE_PID=$("${JAVA_HOME}"/bin/jps | grep DorisFE | awk '{print $1}')


In build.sh line 926:
    if [[ "${TARGET_SYSTEM}" == 'Linux' ]] && [[ "$TARGET_ARCH" == 'x86_64' ]]; then
                                                  ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    if [[ "${TARGET_SYSTEM}" == 'Linux' ]] && [[ "${TARGET_ARCH}" == 'x86_64' ]]; then


In build.sh line 930:
    elif [[ "${TARGET_SYSTEM}" == 'Linux' ]] && [[ "$TARGET_ARCH" == 'aarch64' ]]; then
                                                    ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    elif [[ "${TARGET_SYSTEM}" == 'Linux' ]] && [[ "${TARGET_ARCH}" == 'aarch64' ]]; then


In docker/thirdparties/docker-compose/common/hive-configure.sh line 22:
export CORE_CONF_fs_defaultFS=${CORE_CONF_fs_defaultFS:-hdfs://`hostname -f`:8020}
                                                               ^-----------^ SC2006 (style): Use $(...) notation instead of legacy backticks `...`.

Did you mean: 
export CORE_CONF_fs_defaultFS=${CORE_CONF_fs_defaultFS:-hdfs://$(hostname -f):8020}


In docker/thirdparties/docker-compose/common/hive-configure.sh line 29:
  local entry="<property><name>$name</name><value>${value}</value></property>"
                               ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
  local entry="<property><name>${name}</name><value>${value}</value></property>"


In docker/thirdparties/docker-compose/common/hive-configure.sh line 30:
  local escapedEntry=$(echo $entry | sed 's/\//\\\//g')
        ^----------^ SC2155 (warning): Declare and assign separately to avoid masking return values.
                            ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
                            ^----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
  local escapedEntry=$(echo "${entry}" | sed 's/\//\\\//g')


In docker/thirdparties/docker-compose/common/hive-configure.sh line 31:
  sed -i "/<\/configuration>/ s/.*/${escapedEntry}\n&/" $path
                                                        ^---^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                        ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
  sed -i "/<\/configuration>/ s/.*/${escapedEntry}\n&/" "${path}"


In docker/thirdparties/docker-compose/common/hive-configure.sh line 42:
    echo "Configuring $module"
                      ^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    echo "Configuring ${module}"


In docker/thirdparties/docker-compose/common/hive-configure.sh line 43:
    for c in `printenv | perl -sne 'print "$1 " if m/^${envPrefix}_(.+?)=.*/' -- -envPrefix=$envPrefix`; do 
             ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
                                                                                            ^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                                            ^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    for c in $(printenv | perl -sne 'print "$1 " if m/^${envPrefix}_(.+?)=.*/' -- -envPrefix="${envPrefix}"); do 


In docker/thirdparties/docker-compose/common/hive-configure.sh line 44:
        name=`echo ${c} | perl -pe 's/___/-/g; s/__/_/g; s/_/./g'`
             ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
                   ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
        name=$(echo "${c}" | perl -pe 's/___/-/g; s/__/_/g; s/_/./g')


In docker/thirdparties/docker-compose/common/hive-configure.sh line 47:
        echo " - Setting $name=$  "
                         ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        echo " - Setting ${name}=$  "


In docker/thirdparties/docker-compose/common/hive-configure.sh line 48:
        addProperty $path $name "$value"
                    ^---^ SC2086 (info): Double quote to prevent globbing and word splitting.
                    ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                          ^---^ SC2086 (info): Double quote to prevent globbing and word splitting.
                          ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                 ^----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        addProperty "${path}" "${name}" "${value}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 26:
    // clear output file
    ^-- SC1127 (error): Was this intended as a comment? Use # in sh.


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 27:
    echo "" > "$output_file"
               ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    echo "" > "${output_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 31:
        if [ -n "$type_value" ]; then
           ^------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
                 ^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        if [[ -n "${type_value}" ]]; then


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 32:
            echo "{\"index\": {\"_index\": \"$index_name\", \"_type\": \"$type_value\", \"_id\": \"${id_prefix}${id}\"}}"  >> "$output_file"
                                             ^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                         ^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                                                                               ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            echo "{\"index\": {\"_index\": \"${index_name}\", \"_type\": \"${type_value}\", \"_id\": \"${id_prefix}${id}\"}}"  >> "${output_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 34:
            echo "{\"index\": {\"_index\": \"$index_name\", \"_id\": \"${id_prefix}${id}\"}}"  >> "$output_file"
                                             ^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                                                   ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            echo "{\"index\": {\"_index\": \"${index_name}\", \"_id\": \"${id_prefix}${id}\"}}"  >> "${output_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 36:
        echo "$line"  >> "$output_file"
              ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                          ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        echo "${line}"  >> "${output_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 38:
    done < "$data_file"
            ^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    done < "${data_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 79:
generate_bulk_request "composite_type_array" "doc" "item_" "$array_data_file" "$bulk_request_file"
                                                            ^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                               ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
generate_bulk_request "composite_type_array" "doc" "item_" "${array_data_file}" "${bulk_request_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 80:
curl -X POST "http://${ES_5_HOST}:9200/_bulk" --data-binary "@$bulk_request_file" -H "Content-Type: application/json"
                                                              ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
curl -X POST "http://${ES_5_HOST}:9200/_bulk" --data-binary "@${bulk_request_file}" -H "Content-Type: application/json"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 119:
generate_bulk_request "composite_type_array" "doc" "item_" "$array_data_file" "$bulk_request_file"
                                                            ^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                               ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
generate_bulk_request "composite_type_array" "doc" "item_" "${array_data_file}" "${bulk_request_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 120:
curl -X POST "http://${ES_6_HOST}:9200/_bulk" --data-binary "@$bulk_request_file" -H "Content-Type: application/json"
                                                              ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
curl -X POST "http://${ES_6_HOST}:9200/_bulk" --data-binary "@${bulk_request_file}" -H "Content-Type: application/json"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 126:
curl "http://${ES_7_HOST}:9200/test1" -H "Content-Type:application/json" -X PUT -d "@/mnt/scripts/index/es7_test1.json"
             ^----------^ SC2154 (warning): ES_7_HOST is referenced but not assigned.


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 164:
generate_bulk_request "composite_type_array" "_doc" "item_" "$array_data_file" "$bulk_request_file"
                                                             ^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                                ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
generate_bulk_request "composite_type_array" "_doc" "item_" "${array_data_file}" "${bulk_request_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 165:
curl -X POST "http://${ES_7_HOST}:9200/_bulk" --data-binary "@$bulk_request_file" -H "Content-Type: application/json"
                                                              ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
curl -X POST "http://${ES_7_HOST}:9200/_bulk" --data-binary "@${bulk_request_file}" -H "Content-Type: application/json"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 171:
curl "http://${ES_8_HOST}:9200/test1" -H "Content-Type:application/json" -X PUT -d "@/mnt/scripts/index/es7_test1.json"
             ^----------^ SC2154 (warning): ES_8_HOST is referenced but not assigned.


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 207:
generate_bulk_request "composite_type_array" "" "item_" "$array_data_file" "$bulk_request_file"
                                                         ^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                            ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
generate_bulk_request "composite_type_array" "" "item_" "${array_data_file}" "${bulk_request_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 208:
curl -X POST "http://${ES_8_HOST}:9200/_bulk" --data-binary "@$bulk_request_file" -H "Content-Type: application/json"
                                                              ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
curl -X POST "http://${ES_8_HOST}:9200/_bulk" --data-binary "@${bulk_request_file}" -H "Content-Type: application/json"


In docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/logs1_parquet/run.sh line 11:
    cd -
    ^--^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Did you mean: 
    cd - || exit


In docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_complex_types/run.sh line 11:
    cd -
    ^--^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Did you mean: 
    cd - || exit


In docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_compress_partitioned/run.sh line 11:
    cd -
    ^--^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Did you mean: 
    cd - || exit


In docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_wide_table/run.sh line 11:
    cd -
    ^--^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Did you mean: 
    cd - || exit


In docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_hdfs_tvf_compression/run.sh line 9:
    cd ${CUR_DIR}/ && rm -f test_data.tar.gz \
       ^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
    cd "${CUR_DIR}"/ && rm -f test_data.tar.gz \


In docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_hdfs_tvf_compression/run.sh line 12:
    cd -
    ^--^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Did you mean: 
    cd - || exit


In docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_tvf/run.sh line 9:
    cd ${CUR_DIR}/ && rm -f data.tar.gz \
       ^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
    cd "${CUR_DIR}"/ && rm -f data.tar.gz \


In docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_tvf/run.sh line 12:
    cd -
    ^--^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Did you mean: 
    cd - || exit


In docker/thirdparties/docker-compose/hive/scripts/hive-metastore.sh line 33:
while ! $(nc -z localhost "${HMS_PORT:-9083}"); do
        ^-- SC2091 (warning): Remove surrounding $() to avoid executing output (or use eval if intentional).


In docker/thirdparties/docker-compose/hive/scripts/hive-metastore.sh line 42:
find "${DATA_DIR}" -type f -name "run.sh" -print0 | xargs -0 -n 1 -P "${parallel}" -I {} bash -ec '
                                                                                                  ^-- SC2016 (info): Expressions don't expand in single quotes, use double quotes for that.


In docker/thirdparties/docker-compose/hive/scripts/hive-metastore.sh line 148:
ls /mnt/scripts/create_preinstalled_scripts/*.hql | xargs -n 1 -P "${parallel}" -I {} bash -ec '
^-- SC2011 (warning): Use 'find .. -print0 | xargs -0 ..' or 'find .. -exec .. +' to allow non-alphanumeric filenames.
                                                                                               ^-- SC2016 (info): Expressions don't expand in single quotes, use double quotes for that.


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 25:
if [ -z "${HADOOP_HOME}" ]; then
   ^---------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
if [[ -z "${HADOOP_HOME}" ]]; then


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 29:
if [ -z "${HIVE_HOME}" ]; then
   ^-------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
if [[ -z "${HIVE_HOME}" ]]; then


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 35:
HUDI_HIVE_UBER_JAR=`ls -c $DIR/./hudi_docker_compose_attached_file/jar/hoodie-hive-sync-bundle.jar | grep -v source | head -1`
                   ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
                    ^-- SC2010 (warning): Don't use ls | grep. Use a glob or a for loop with a condition to allow non-alphanumeric filenames.
                          ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
                          ^--^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
HUDI_HIVE_UBER_JAR=$(ls -c "${DIR}"/./hudi_docker_compose_attached_file/jar/hoodie-hive-sync-bundle.jar | grep -v source | head -1)


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 37:
if [ -z "$HADOOP_CONF_DIR" ]; then
   ^-----------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
         ^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
if [[ -z "${HADOOP_CONF_DIR}" ]]; then


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 43:
HIVE_EXEC=`ls ${HIVE_HOME}/lib/hive-exec-*.jar | tr '\n' ':'`
          ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
           ^-- SC2012 (info): Use find instead of ls to better handle non-alphanumeric filenames.
              ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
HIVE_EXEC=$(ls "${HIVE_HOME}"/lib/hive-exec-*.jar | tr '\n' ':')


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 44:
HIVE_SERVICE=`ls ${HIVE_HOME}/lib/hive-service-*.jar | grep -v rpc | tr '\n' ':'`
             ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
              ^-- SC2010 (warning): Don't use ls | grep. Use a glob or a for loop with a condition to allow non-alphanumeric filenames.
                 ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
HIVE_SERVICE=$(ls "${HIVE_HOME}"/lib/hive-service-*.jar | grep -v rpc | tr '\n' ':')


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 45:
HIVE_METASTORE=`ls ${HIVE_HOME}/lib/hive-metastore-*.jar | tr '\n' ':'`
               ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
                ^-- SC2012 (info): Use find instead of ls to better handle non-alphanumeric filenames.
                   ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
HIVE_METASTORE=$(ls "${HIVE_HOME}"/lib/hive-metastore-*.jar | tr '\n' ':')


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 46:
HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | tr '\n' ':'`
          ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
           ^-- SC2012 (info): Use find instead of ls to better handle non-alphanumeric filenames.
              ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
HIVE_JDBC=$(ls "${HIVE_HOME}"/lib/hive-jdbc-*.jar | tr '\n' ':')


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 47:
if [ -z "${HIVE_JDBC}" ]; then
   ^-------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
if [[ -z "${HIVE_JDBC}" ]]; then


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 48:
  HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | grep -v handler | tr '\n' ':'`
            ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
             ^-- SC2010 (warning): Don't use ls | grep. Use a glob or a for loop with a condition to allow non-alphanumeric filenames.
                ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
  HIVE_JDBC=$(ls "${HIVE_HOME}"/lib/hive-jdbc-*.jar | grep -v handler | tr '\n' ':')


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 50:
HIVE_JACKSON=`ls ${HIVE_HOME}/lib/jackson-*.jar | tr '\n' ':'`
             ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
              ^-- SC2012 (info): Use find instead of ls to better handle non-alphanumeric filenames.
                 ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
HIVE_JACKSON=$(ls "${HIVE_HOME}"/lib/jackson-*.jar | tr '\n' ':')


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 51:
HIVE_JARS=$HIVE_METASTORE:$HIVE_SERVICE:$HIVE_EXEC:$HIVE_JDBC:$HIVE_JACKSON
          ^-------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                          ^-----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                        ^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                   ^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                              ^-----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
HIVE_JARS=${HIVE_METASTORE}:${HIVE_SERVICE}:${HIVE_EXEC}:${HIVE_JDBC}:${HIVE_JACKSON}


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 53:
HADOOP_HIVE_JARS=${HIVE_JARS}:${HADOOP_HOME}/share/hadoop/common/*:${HADOOP_HOME}/share/hadoop/mapreduce/*:${HADOOP_HOME}/share/hadoop/hdfs/*:${HADOOP_HOME}/share/hadoop/common/lib/*:${HADOOP_HOME}/share/hadoop/hdfs/lib/*
                 ^-- SC2125 (warning): Brace expansions and globs are literal in assignments. Quote it or use an array.


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 55:
echo "Running Command : java -cp ${HADOOP_HIVE_JARS}:${HADOOP_CONF_DIR}:$HUDI_HIVE_UBER_JAR org.apache.hudi.hive.HiveSyncTool $@"
                                                                        ^-----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                                                                              ^-- SC2145 (error): Argument mixes string and array. Use * or separate argument.

Did you mean: 
echo "Running Command : java -cp ${HADOOP_HIVE_JARS}:${HADOOP_CONF_DIR}:${HUDI_HIVE_UBER_JAR} org.apache.hudi.hive.HiveSyncTool $@"


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 56:
java -cp $HUDI_HIVE_UBER_JAR:${HADOOP_HIVE_JARS}:${HADOOP_CONF_DIR} org.apache.hudi.hive.HiveSyncTool "$@"
         ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
         ^-----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                             ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                 ^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
java -cp "${HUDI_HIVE_UBER_JAR}":"${HADOOP_HIVE_JARS}":"${HADOOP_CONF_DIR}" org.apache.hudi.hive.HiveSyncTool "$@"


In docker/thirdparties/docker-compose/hudi/scripts/setup_demo_container_adhoc_1.sh line 20:
cp /var/scripts/config/spark-defaults.conf $SPARK_CONF_DIR/.
                                           ^-------------^ SC2154 (warning): SPARK_CONF_DIR is referenced but not assigned.
                                           ^-------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                           ^-------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
cp /var/scripts/config/spark-defaults.conf "${SPARK_CONF_DIR}"/.


In docker/thirdparties/docker-compose/hudi/scripts/setup_demo_container_adhoc_1.sh line 21:
cp /var/scripts/config/log4j2.properties $SPARK_CONF_DIR/.
                                         ^-------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                         ^-------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
cp /var/scripts/config/log4j2.properties "${SPARK_CONF_DIR}"/.


In docker/thirdparties/docker-compose/hudi/scripts/setup_demo_container_adhoc_2.sh line 20:
cp /var/scripts/config/spark-defaults.conf $SPARK_CONF_DIR/.
                                           ^-------------^ SC2154 (warning): SPARK_CONF_DIR is referenced but not assigned.
                                           ^-------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                           ^-------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
cp /var/scripts/config/spark-defaults.conf "${SPARK_CONF_DIR}"/.


In docker/thirdparties/docker-compose/hudi/scripts/setup_demo_container_adhoc_2.sh line 21:
cp /var/scripts/config/log4j2.properties $SPARK_CONF_DIR/.
                                         ^-------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                         ^-------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
cp /var/scripts/config/log4j2.properties "${SPARK_CONF_DIR}"/.


In docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh line 19:
source /usr/local/common/hive-configure.sh
       ^-- SC1091 (info): Not following: /usr/local/common/hive-configure.sh: openBinaryFile: does not exist (No such file or directory)


In docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh line 20:
source /usr/local/common/event-hook.sh
       ^-----------------------------^ SC1091 (info): Not following: /usr/local/common/event-hook.sh: openBinaryFile: does not exist (No such file or directory)


In docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh line 34:
if [ "$1" == "1" ]; then
   ^-------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
if [[ "$1" == "1" ]]; then


In docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh line 36:
elif [ "$1" == "2" ]; then
     ^-------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
elif [[ "$1" == "2" ]]; then


In docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh line 57:
if [ $i -eq 60 ]; then
   ^-----------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
     ^-- SC2086 (info): Double quote to prevent globbing and word splitting.
     ^-- SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
if [[ "${i}" -eq 60 ]]; then


In docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh line 64:
if [ "$1" == "1" ]; then
   ^-------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
if [[ "$1" == "1" ]]; then


In docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh line 66:
elif [ "$1" == "2" ]; then
     ^-------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
elif [[ "$1" == "2" ]]; then


In docker/thirdparties/docker-compose/kerberos/health-checks/supervisorctl-check.sh line 34:
if [ "$FAILED" == "" ]; then
   ^-----------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
      ^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
if [[ "${FAILED}" == "" ]]; then


In docker/thirdparties/docker-compose/ranger/ranger-admin/ranger-entrypoint.sh line 24:
cd $RANGER_HOME
   ^----------^ SC2154 (warning): RANGER_HOME is referenced but not assigned.
   ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
   ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
cd "${RANGER_HOME}"


In docker/thirdparties/docker-compose/ranger/script/install_doris_ranger_plugins.sh line 16:
#!/bin/bash
^-- SC1128 (error): The shebang must be on the first line. Delete blanks and move comments.


In docker/thirdparties/docker-compose/ranger/script/install_doris_ranger_plugins.sh line 19:
if [ ! -d "${RANGER_HOME}/ews/webapp/WEB-INF/classes/ranger-plugins/doris" ]; then
   ^-- SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
           ^------------^ SC2154 (warning): RANGER_HOME is referenced but not assigned.

Did you mean: 
if [[ ! -d "${RANGER_HOME}/ews/webapp/WEB-INF/classes/ranger-plugins/doris" ]]; then


In docker/thirdparties/docker-compose/ranger/script/install_doris_service_def.sh line 15:
#!/bin/bash
^-- SC1128 (error): The shebang must be on the first line. Delete blanks and move comments.


In docker/thirdparties/run-thirdparties-docker.sh line 117:
    echo ${COMPONENTS}
         ^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
    echo "${COMPONENTS}"


In docker/thirdparties/run-thirdparties-docker.sh line 150:
RUN_OCENABASE=0
^-----------^ SC2034 (warning): RUN_OCENABASE appears unused. Verify use (or export if used externally).


In docker/thirdparties/run-thirdparties-docker.sh line 330:
    IP_HOST=$(ifconfig "${eth_name}" | grep inet | grep -v 127.0.0.1 | grep -v inet6 | awk '{print $2}' | tr -d "addr:" | head -n 1)
                                                                                                                ^-----^ SC2020 (info): tr replaces sets of chars, not words (mentioned due to duplicates).


In docker/thirdparties/run-thirdparties-docker.sh line 339:
        local backup_dir=/home/work/pipline/backup_center
              ^--------^ SC2034 (warning): backup_dir appears unused. Verify use (or export if used externally).


In docker/thirdparties/run-thirdparties-docker.sh line 344:
            echo "docker exec "${container_id}" bash -c echo '/opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server '${ip_host}:19193' --topic '${topic}'"
                               ^-------------^ SC2027 (warning): The surrounding quotes actually unquote this. Remove or escape them.
                               ^-------------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
            echo "docker exec ""${container_id}"" bash -c echo '/opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server '${ip_host}:19193' --topic '${topic}'"


In docker/thirdparties/run-thirdparties-docker.sh line 363:
    IP_HOST=$(ifconfig "${eth_name}" | grep inet | grep -v 127.0.0.1 | grep -v inet6 | awk '{print $2}' | tr -d "addr:" | head -n 1)
                                                                                                                ^-----^ SC2020 (info): tr replaces sets of chars, not words (mentioned due to duplicates).


In docker/thirdparties/run-thirdparties-docker.sh line 365:
    if [ "_${IP_HOST}" == "_" ]; then
       ^----------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
    if [[ "_${IP_HOST}" == "_" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 367:
        exit -1
             ^-- SC2242 (error): Can only exit with status 0-255. Other data should be written to stdout/stderr.


In docker/thirdparties/run-thirdparties-docker.sh line 375:
        if [ -f "$file" ]; then
           ^------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
                 ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        if [[ -f "${file}" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 376:
            echo "Processing $file"
                             ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            echo "Processing ${file}"


In docker/thirdparties/run-thirdparties-docker.sh line 380:
            echo "File not found: $file"
                                  ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            echo "File not found: ${file}"


In docker/thirdparties/run-thirdparties-docker.sh line 391:
    sudo docker compose -p ${CONTAINER_UID}hive2 -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-2x.env down
                           ^--------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
    sudo docker compose -p "${CONTAINER_UID}"hive2 -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-2x.env down


In docker/thirdparties/run-thirdparties-docker.sh line 393:
        sudo docker compose -p ${CONTAINER_UID}hive2 -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-2x.env up --build --remove-orphans -d --wait
                               ^--------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
        sudo docker compose -p "${CONTAINER_UID}"hive2 -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-2x.env up --build --remove-orphans -d --wait


In docker/thirdparties/run-thirdparties-docker.sh line 403:
    IP_HOST=$(ifconfig "${eth_name}" | grep inet | grep -v 127.0.0.1 | grep -v inet6 | awk '{print $2}' | tr -d "addr:" | head -n 1)
                                                                                                                ^-----^ SC2020 (info): tr replaces sets of chars, not words (mentioned due to duplicates).


In docker/thirdparties/run-thirdparties-docker.sh line 404:
    if [ "_${IP_HOST}" == "_" ]; then
       ^----------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
    if [[ "_${IP_HOST}" == "_" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 406:
        exit -1
             ^-- SC2242 (error): Can only exit with status 0-255. Other data should be written to stdout/stderr.


In docker/thirdparties/run-thirdparties-docker.sh line 414:
        if [ -f "$file" ]; then
           ^------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
                 ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        if [[ -f "${file}" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 415:
            echo "Processing $file"
                             ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            echo "Processing ${file}"


In docker/thirdparties/run-thirdparties-docker.sh line 419:
            echo "File not found: $file"
                                  ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            echo "File not found: ${file}"


In docker/thirdparties/run-thirdparties-docker.sh line 430:
    sudo docker compose -p ${CONTAINER_UID}hive3 -f "${ROOT}"/docker-compose/hive/hive-3x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-3x.env down
                           ^--------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
    sudo docker compose -p "${CONTAINER_UID}"hive3 -f "${ROOT}"/docker-compose/hive/hive-3x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-3x.env down


In docker/thirdparties/run-thirdparties-docker.sh line 432:
        sudo docker compose -p ${CONTAINER_UID}hive3 -f "${ROOT}"/docker-compose/hive/hive-3x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-3x.env up --build --remove-orphans -d --wait
                               ^--------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
        sudo docker compose -p "${CONTAINER_UID}"hive3 -f "${ROOT}"/docker-compose/hive/hive-3x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-3x.env up --build --remove-orphans -d --wait


In docker/thirdparties/run-thirdparties-docker.sh line 499:
        cp "${trino_docker}/$file.tpl" "${trino_docker}/$file"
                            ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                        ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        cp "${trino_docker}/${file}.tpl" "${trino_docker}/${file}"


In docker/thirdparties/run-thirdparties-docker.sh line 500:
        if [[ $file != "hive.properties" ]]; then
              ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        if [[ ${file} != "hive.properties" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 501:
            sed -i "s/doris--/${CONTAINER_UID}/g" "${trino_docker}/$file"
                                                                   ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            sed -i "s/doris--/${CONTAINER_UID}/g" "${trino_docker}/${file}"


In docker/thirdparties/run-thirdparties-docker.sh line 510:
        sudo echo "127.0.0.1 ${NAMENODE_CONTAINER_ID}" >>/etc/hosts
                                                       ^-- SC2024 (warning): sudo doesn't affect redirects. Use .. | sudo tee -a file


In docker/thirdparties/run-thirdparties-docker.sh line 512:
        hive_metastore_ip=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ${HIVE_METASTORE_CONTAINER_ID})
                                                                                                               ^----------------------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
        hive_metastore_ip=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "${HIVE_METASTORE_CONTAINER_ID}")


In docker/thirdparties/run-thirdparties-docker.sh line 514:
        if [ -z "$hive_metastore_ip" ]; then
           ^-------------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
                 ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        if [[ -z "${hive_metastore_ip}" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 518:
            echo "Hive Metastore IP address is: $hive_metastore_ip"
                                                ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            echo "Hive Metastore IP address is: ${hive_metastore_ip}"


In docker/thirdparties/run-thirdparties-docker.sh line 533:
            while [ $retries -lt $max_retries ]; do
                  ^---------------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
                    ^------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.
                    ^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                 ^----------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.
                                 ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            while [[ "${retries}" -lt "${max_retries}" ]]; do


In docker/thirdparties/run-thirdparties-docker.sh line 534:
                status=$(docker inspect --format '{{.State.Running}}' ${TRINO_CONTAINER_ID})
                                                                      ^-------------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
                status=$(docker inspect --format '{{.State.Running}}' "${TRINO_CONTAINER_ID}")


In docker/thirdparties/run-thirdparties-docker.sh line 535:
                if [ "${status}" == "${expected_status}" ]; then
                   ^-- SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
                if [[ "${status}" == "${expected_status}" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 546:
            if [ $retries -eq $max_retries ]; then
               ^---------------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
                 ^------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.
                 ^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                              ^----------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.
                              ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            if [[ "${retries}" -eq "${max_retries}" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 547:
                echo "${operation} operation failed to complete after $max_retries attempts."
                                                                      ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
                echo "${operation} operation failed to complete after ${max_retries} attempts."


In docker/thirdparties/run-thirdparties-docker.sh line 552:
        docker stop ${TRINO_CONTAINER_ID}
                    ^-------------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
        docker stop "${TRINO_CONTAINER_ID}"


In docker/thirdparties/run-thirdparties-docker.sh line 557:
        docker start ${TRINO_CONTAINER_ID}
                     ^-------------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
        docker start "${TRINO_CONTAINER_ID}"


In docker/thirdparties/run-thirdparties-docker.sh line 563:
        docker exec -it ${TRINO_CONTAINER_ID} /bin/bash -c 'trino -f /scripts/create_trino_table.sql'
                        ^-------------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
        docker exec -it "${TRINO_CONTAINER_ID}" /bin/bash -c 'trino -f /scripts/create_trino_table.sql'


In docker/thirdparties/run-thirdparties-docker.sh line 601:
        mv *.tbl ../lakesoul/test_files/tpch/data
           ^-- SC2035 (info): Use ./*glob* or -- *glob* so names with dashes won't become options.


In docker/thirdparties/run-thirdparties-docker.sh line 603:
        export TPCH_DATA=$(realpath lakesoul/test_files/tpch/data)
               ^-------^ SC2155 (warning): Declare and assign separately to avoid masking return values.


In docker/thirdparties/run-thirdparties-docker.sh line 615:
    IP_HOST=$(ifconfig "${eth_name}" | grep inet | grep -v 127.0.0.1 | grep -v inet6 | awk '{print $2}' | tr -d "addr:" | head -n 1)
                                                                                                                ^-----^ SC2020 (info): tr replaces sets of chars, not words (mentioned due to duplicates).


In docker/thirdparties/run-thirdparties-docker.sh line 620:
        . "${ROOT}"/docker-compose/kerberos/kerberos${i}_settings.env
          ^-- SC1090 (warning): ShellCheck can't follow non-constant source. Use a directive to specify location.
                                                    ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
        . "${ROOT}"/docker-compose/kerberos/kerberos"${i}"_settings.env


In docker/thirdparties/run-thirdparties-docker.sh line 621:
        envsubst <"${ROOT}"/docker-compose/kerberos/hadoop-hive.env.tpl >"${ROOT}"/docker-compose/kerberos/hadoop-hive-${i}.env
                                                                                                                       ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
        envsubst <"${ROOT}"/docker-compose/kerberos/hadoop-hive.env.tpl >"${ROOT}"/docker-compose/kerberos/hadoop-hive-"${i}".env


In docker/thirdparties/run-thirdparties-docker.sh line 622:
        envsubst <"${ROOT}"/docker-compose/kerberos/conf/my.cnf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/my.cnf
                                                                                                                     ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
        envsubst <"${ROOT}"/docker-compose/kerberos/conf/my.cnf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos"${i}"/my.cnf


In docker/thirdparties/run-thirdparties-docker.sh line 623:
        envsubst <"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/kdc.conf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/kdc.conf
                                                                 ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                                                                                    ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
        envsubst <"${ROOT}"/docker-compose/kerberos/conf/kerberos"${i}"/kdc.conf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos"${i}"/kdc.conf


In docker/thirdparties/run-thirdparties-docker.sh line 624:
        envsubst <"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/krb5.conf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/krb5.conf
                                                                 ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                                                                                     ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
        envsubst <"${ROOT}"/docker-compose/kerberos/conf/kerberos"${i}"/krb5.conf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos"${i}"/krb5.conf


In docker/thirdparties/run-thirdparties-docker.sh line 779:
    wait "${pids[$compose]}" || status=$?
                 ^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    wait "${pids[${compose}]}" || status=$?


In docker/thirdparties/run-thirdparties-docker.sh line 780:
    if [ $status -ne 0 ] && [ $compose != "db2" ]; then
       ^---------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
         ^-----^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.
         ^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                            ^-------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
                              ^------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                              ^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    if [[ "${status}" -ne 0 ]] && [[ "${compose}" != "db2" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 781:
        echo "docker $compose started failed with status $status"
                     ^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                         ^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        echo "docker ${compose} started failed with status ${status}"


In docker/thirdparties/run-thirdparties-docker.sh line 783:
        cat start_${compose}.log || true
                  ^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
        cat start_"${compose}".log || true


In thirdparty/build-thirdparty.sh line 1868:
    cp -r ${TP_SOURCE_DIR}/${JINDOFS_SOURCE}/* "${TP_INSTALL_DIR}/jindofs_libs/"
          ^--------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                           ^---------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
    cp -r "${TP_SOURCE_DIR}"/"${JINDOFS_SOURCE}"/* "${TP_INSTALL_DIR}/jindofs_libs/"


In tools/lzo/build.sh line 1:
# Licensed to the Apache Software Foundation (ASF) under one
^-- SC2148 (error): Tips depend on target shell and yours is unknown. Add a shebang or a 'shell' directive.


In tools/lzo/build.sh line 20:
g++ -o lzo_writer lzo_writer.cpp -I. -Isrc -I${DORIS_THIRDPARTY}/installed/include -L${DORIS_THIRDPARTY}/installed/lib -llzo2 -std=c++17
                                             ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                                     ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
g++ -o lzo_writer lzo_writer.cpp -I. -Isrc -I"${DORIS_THIRDPARTY}"/installed/include -L"${DORIS_THIRDPARTY}"/installed/lib -llzo2 -std=c++17

For more information:
  https://www.shellcheck.net/wiki/SC1127 -- Was this intended as a comment? U...
  https://www.shellcheck.net/wiki/SC1128 -- The shebang must be on the first ...
  https://www.shellcheck.net/wiki/SC2145 -- Argument mixes string and array. ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
shfmt errors 

'shfmt ' returned error 1 finding the following formatting issues:

----------
--- docker/thirdparties/docker-compose/common/event-hook.sh.orig
+++ docker/thirdparties/docker-compose/common/event-hook.sh
--- docker/thirdparties/docker-compose/common/hive-configure.sh.orig
+++ docker/thirdparties/docker-compose/common/hive-configure.sh
@@ -19,16 +19,16 @@
 # Referenced from [docker-hive](https://github.com/big-data-europe/docker-hive)
 
 # Set some sensible defaults
-export CORE_CONF_fs_defaultFS=${CORE_CONF_fs_defaultFS:-hdfs://`hostname -f`:8020}
+export CORE_CONF_fs_defaultFS=${CORE_CONF_fs_defaultFS:-hdfs://$(hostname -f):8020}
 
 function addProperty() {
-  local path=$1
-  local name=$2
-  local value=$3
+    local path=$1
+    local name=$2
+    local value=$3
 
-  local entry="<property><name>$name</name><value>${value}</value></property>"
-  local escapedEntry=$(echo $entry | sed 's/\//\\\//g')
-  sed -i "/<\/configuration>/ s/.*/${escapedEntry}\n&/" $path
+    local entry="<property><name>$name</name><value>${value}</value></property>"
+    local escapedEntry=$(echo $entry | sed 's/\//\\\//g')
+    sed -i "/<\/configuration>/ s/.*/${escapedEntry}\n&/" $path
 }
 
 function configure() {
@@ -38,10 +38,10 @@
 
     local var
     local value
-    
+
     echo "Configuring $module"
-    for c in `printenv | perl -sne 'print "$1 " if m/^${envPrefix}_(.+?)=.*/' -- -envPrefix=$envPrefix`; do 
-        name=`echo ${c} | perl -pe 's/___/-/g; s/__/_/g; s/_/./g'`
+    for c in $(printenv | perl -sne 'print "$1 " if m/^${envPrefix}_(.+?)=.*/' -- -envPrefix=$envPrefix); do
+        name=$(echo ${c} | perl -pe 's/___/-/g; s/__/_/g; s/_/./g')
         var="${envPrefix}_${c}"
         value=${!var}
         echo " - Setting $name=$  "
--- docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh.orig
+++ docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh
@@ -24,18 +24,18 @@
     local output_file=$5
 
     // clear output file
-    echo "" > "$output_file"
+    echo "" >"$output_file"
 
     local id=1
     while IFS= read -r line; do
         if [ -n "$type_value" ]; then
-            echo "{\"index\": {\"_index\": \"$index_name\", \"_type\": \"$type_value\", \"_id\": \"${id_prefix}${id}\"}}"  >> "$output_file"
+            echo "{\"index\": {\"_index\": \"$index_name\", \"_type\": \"$type_value\", \"_id\": \"${id_prefix}${id}\"}}" >>"$output_file"
         else
-            echo "{\"index\": {\"_index\": \"$index_name\", \"_id\": \"${id_prefix}${id}\"}}"  >> "$output_file"
+            echo "{\"index\": {\"_index\": \"$index_name\", \"_id\": \"${id_prefix}${id}\"}}" >>"$output_file"
         fi
-        echo "$line"  >> "$output_file"
+        echo "$line" >>"$output_file"
         id=$((id + 1))
-    done < "$data_file"
+    done <"$data_file"
 }
 
 array_data_file="/mnt/scripts/data/composite_type_array_bulk.json"
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_csv/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_csv/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}/create_table.hql"
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_orc/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_orc/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_parquet/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_parquet/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_config_test/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_config_test/run.sh
@@ -11,4 +11,3 @@
 hive -f "${CUR_DIR}"/create_table.hql
 
 hadoop fs -rm -r /user/doris/suites/default/hive_ignore_absent_partitions_table/country=India
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_array_all_types/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_array_all_types/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_array_delimiter/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_array_delimiter/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_nestedarray/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_nestedarray/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_upper_case_orc/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_upper_case_orc/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_upper_case_parquet/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_upper_case_parquet/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/logs1_parquet/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/logs1_parquet/run.sh
@@ -5,9 +5,9 @@
 
 if [[ ! -d "${CUR_DIR}/data" ]]; then
     echo "${CUR_DIR}/data does not exist"
-    cd "${CUR_DIR}" && rm -f data.tar.gz \
-    && curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/multi_catalog/logs1_parquet/data.tar.gz \
-    && tar xzf data.tar.gz
+    cd "${CUR_DIR}" && rm -f data.tar.gz &&
+        curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/multi_catalog/logs1_parquet/data.tar.gz &&
+        tar xzf data.tar.gz
     cd -
 else
     echo "${CUR_DIR}/data exist, continue !"
@@ -19,4 +19,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/one_partition/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/one_partition/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}/create_table.hql"
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/orc_predicate/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/orc_predicate/run.sh
@@ -5,5 +5,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/orc_predicate_table.hql
-
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/parquet_lz4_compression/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/parquet_lz4_compression/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/parquet_lzo_compression/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/parquet_lzo_compression/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/partition_manual_remove/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/partition_manual_remove/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_complex_types/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_complex_types/run.sh
@@ -5,9 +5,9 @@
 
 if [[ ! -d "${CUR_DIR}/data" ]]; then
     echo "${CUR_DIR}/data does not exist"
-    cd "${CUR_DIR}" && rm -f data.tar.gz \
-    && curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/multi_catalog/test_complex_types/data.tar.gz \
-    && tar xzf data.tar.gz
+    cd "${CUR_DIR}" && rm -f data.tar.gz &&
+        curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/multi_catalog/test_complex_types/data.tar.gz &&
+        tar xzf data.tar.gz
     cd -
 else
     echo "${CUR_DIR}/data exist, continue !"
@@ -19,4 +19,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_compress_partitioned/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_compress_partitioned/run.sh
@@ -5,9 +5,9 @@
 
 if [[ ! -d "${CUR_DIR}/data" ]]; then
     echo "${CUR_DIR}/data does not exist"
-    cd "${CUR_DIR}" && rm -f data.tar.gz \
-    && curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/multi_catalog/test_compress_partitioned/data.tar.gz \
-    && tar xzf data.tar.gz
+    cd "${CUR_DIR}" && rm -f data.tar.gz &&
+        curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/multi_catalog/test_compress_partitioned/data.tar.gz &&
+        tar xzf data.tar.gz
     cd -
 else
     echo "${CUR_DIR}/data exist, continue !"
@@ -19,4 +19,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_csv_format_error/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_csv_format_error/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}/create_table.hql"
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_date_string_partition/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_date_string_partition/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}/create_table.hql"
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_wide_table/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_wide_table/run.sh
@@ -5,9 +5,9 @@
 
 if [[ ! -d "${CUR_DIR}/data" ]]; then
     echo "${CUR_DIR}/data does not exist"
-    cd "${CUR_DIR}" && rm -f data.tar.gz \
-    && curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/multi_catalog/test_wide_table/data.tar.gz \
-    && tar xzf data.tar.gz
+    cd "${CUR_DIR}" && rm -f data.tar.gz &&
+        curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/multi_catalog/test_wide_table/data.tar.gz &&
+        tar xzf data.tar.gz
     cd -
 else
     echo "${CUR_DIR}/data exist, continue !"
@@ -19,4 +19,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/two_partition/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/two_partition/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}/create_table.hql"
-
--- docker/thirdparties/docker-compose/hive/scripts/data/regression/crdmm_data/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/regression/crdmm_data/run.sh
@@ -10,5 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
-
--- docker/thirdparties/docker-compose/hive/scripts/data/regression/serde_prop/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/regression/serde_prop/run.sh
@@ -5,5 +5,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/some_serde_table.hql
-
-
--- docker/thirdparties/docker-compose/hive/scripts/data/statistics/statistics/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/statistics/statistics/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}/create_table.hql"
-
--- docker/thirdparties/docker-compose/hive/scripts/data/tpch_1000_parquet/part/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/tpch_1000_parquet/part/run.sh
@@ -19,4 +19,3 @@
 
 # # create table
 # hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_hdfs_tvf_compression/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_hdfs_tvf_compression/run.sh
@@ -6,9 +6,9 @@
 
 if [[ ! -d "${CUR_DIR}/test_data" ]]; then
     echo "${CUR_DIR}/test_data does not exist"
-    cd ${CUR_DIR}/ && rm -f test_data.tar.gz \
-    && curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/test_hdfs_tvf_compression/test_data.tar.gz \
-    && tar xzf test_data.tar.gz
+    cd ${CUR_DIR}/ && rm -f test_data.tar.gz &&
+        curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/test_hdfs_tvf_compression/test_data.tar.gz &&
+        tar xzf test_data.tar.gz
     cd -
 else
     echo "${CUR_DIR}/test_data exist, continue !"
--- docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_tvf/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_tvf/run.sh
@@ -6,9 +6,9 @@
 
 if [[ ! -d "${CUR_DIR}/tvf" ]]; then
     echo "${CUR_DIR}/tvf does not exist"
-    cd ${CUR_DIR}/ && rm -f data.tar.gz \
-    && curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/test_tvf/data.tar.gz \
-    && tar xzf data.tar.gz
+    cd ${CUR_DIR}/ && rm -f data.tar.gz &&
+        curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/test_tvf/data.tar.gz &&
+        tar xzf data.tar.gz
     cd -
 else
     echo "${CUR_DIR}/tvf exist, continue !"
--- docker/thirdparties/docker-compose/hive/scripts/hive-metastore.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/hive-metastore.sh
@@ -97,7 +97,6 @@
 hadoop_put_pids=()
 hadoop fs -mkdir -p /user/doris/
 
-
 ## put tpch1
 if [[ -z "$(ls /mnt/scripts/tpch1.db)" ]]; then
     echo "tpch1.db does not exist"
@@ -126,7 +125,6 @@
 hadoop fs -copyFromLocal -f /mnt/scripts/preinstalled_data /user/doris/ &
 hadoop_put_pids+=($!)
 
-
 # wait put finish
 set +e
 wait "${hadoop_put_pids[@]}"
--- docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh.orig
+++ docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh
@@ -18,36 +18,36 @@
 # under the License.
 
 function error_exit {
-    echo "$1" >&2   ## Send message to stderr. Exclude >&2 if you don't want it that way.
-    exit "${2:-1}"  ## Return a code specified by $2 or 1 by default.
+    echo "$1" >&2  ## Send message to stderr. Exclude >&2 if you don't want it that way.
+    exit "${2:-1}" ## Return a code specified by $2 or 1 by default.
 }
 
 if [ -z "${HADOOP_HOME}" ]; then
-  error_exit "Please make sure the environment variable HADOOP_HOME is setup"
+    error_exit "Please make sure the environment variable HADOOP_HOME is setup"
 fi
 
 if [ -z "${HIVE_HOME}" ]; then
-  error_exit "Please make sure the environment variable HIVE_HOME is setup"
+    error_exit "Please make sure the environment variable HIVE_HOME is setup"
 fi
 
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 #Ensure we pick the right jar even for hive11 builds
-HUDI_HIVE_UBER_JAR=`ls -c $DIR/./hudi_docker_compose_attached_file/jar/hoodie-hive-sync-bundle.jar | grep -v source | head -1`
+HUDI_HIVE_UBER_JAR=$(ls -c $DIR/./hudi_docker_compose_attached_file/jar/hoodie-hive-sync-bundle.jar | grep -v source | head -1)
 
 if [ -z "$HADOOP_CONF_DIR" ]; then
-  echo "setting hadoop conf dir"
-  HADOOP_CONF_DIR="${HADOOP_HOME}/etc/hadoop"
+    echo "setting hadoop conf dir"
+    HADOOP_CONF_DIR="${HADOOP_HOME}/etc/hadoop"
 fi
 
 ## Include only specific packages from HIVE_HOME/lib to avoid version mismatches
-HIVE_EXEC=`ls ${HIVE_HOME}/lib/hive-exec-*.jar | tr '\n' ':'`
-HIVE_SERVICE=`ls ${HIVE_HOME}/lib/hive-service-*.jar | grep -v rpc | tr '\n' ':'`
-HIVE_METASTORE=`ls ${HIVE_HOME}/lib/hive-metastore-*.jar | tr '\n' ':'`
-HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | tr '\n' ':'`
+HIVE_EXEC=$(ls ${HIVE_HOME}/lib/hive-exec-*.jar | tr '\n' ':')
+HIVE_SERVICE=$(ls ${HIVE_HOME}/lib/hive-service-*.jar | grep -v rpc | tr '\n' ':')
+HIVE_METASTORE=$(ls ${HIVE_HOME}/lib/hive-metastore-*.jar | tr '\n' ':')
+HIVE_JDBC=$(ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | tr '\n' ':')
 if [ -z "${HIVE_JDBC}" ]; then
-  HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | grep -v handler | tr '\n' ':'`
+    HIVE_JDBC=$(ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | grep -v handler | tr '\n' ':')
 fi
-HIVE_JACKSON=`ls ${HIVE_HOME}/lib/jackson-*.jar | tr '\n' ':'`
+HIVE_JACKSON=$(ls ${HIVE_HOME}/lib/jackson-*.jar | tr '\n' ':')
 HIVE_JARS=$HIVE_METASTORE:$HIVE_SERVICE:$HIVE_EXEC:$HIVE_JDBC:$HIVE_JACKSON
 
 HADOOP_HIVE_JARS=${HIVE_JARS}:${HADOOP_HOME}/share/hadoop/common/*:${HADOOP_HOME}/share/hadoop/mapreduce/*:${HADOOP_HOME}/share/hadoop/hdfs/*:${HADOOP_HOME}/share/hadoop/common/lib/*:${HADOOP_HOME}/share/hadoop/hdfs/lib/*
--- docker/thirdparties/docker-compose/hudi/scripts/setup_demo_container_adhoc_2.sh.orig
+++ docker/thirdparties/docker-compose/hudi/scripts/setup_demo_container_adhoc_2.sh
@@ -36,42 +36,42 @@
 
 echo "Start synchronizing the stock_ticks_cow table"
 /var/scripts/run_sync_tool.sh \
-  --jdbc-url jdbc:hive2://hiveserver:10000 \
-  --user hive \
-  --pass hive \
-  --partitioned-by date \
-  --base-path /user/hive/warehouse/stock_ticks_cow \
-  --database default \
-  --table stock_ticks_cow \
-  --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
+    --jdbc-url jdbc:hive2://hiveserver:10000 \
+    --user hive \
+    --pass hive \
+    --partitioned-by date \
+    --base-path /user/hive/warehouse/stock_ticks_cow \
+    --database default \
+    --table stock_ticks_cow \
+    --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
 
 echo "Start synchronizing the stock_ticks_mor table"
 /var/scripts/run_sync_tool.sh \
-  --jdbc-url jdbc:hive2://hiveserver:10000 \
-  --user hive \
-  --pass hive \
-  --partitioned-by date \
-  --base-path /user/hive/warehouse/stock_ticks_mor \
-  --database default \
-  --table stock_ticks_mor \
-  --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
+    --jdbc-url jdbc:hive2://hiveserver:10000 \
+    --user hive \
+    --pass hive \
+    --partitioned-by date \
+    --base-path /user/hive/warehouse/stock_ticks_mor \
+    --database default \
+    --table stock_ticks_mor \
+    --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
 
 echo "Start synchronizing the hudi_cow_pt_tbl table"
 /var/scripts/run_sync_tool.sh \
-  --jdbc-url jdbc:hive2://hiveserver:10000 \
-  --user hive \
-  --pass hive \
-  --partitioned-by dt \
-  --base-path /user/hive/warehouse/hudi_cow_pt_tbl \
-  --database default \
-  --table hudi_cow_pt_tbl \
-  --partition-value-extractor org.apache.hudi.hive.HiveStylePartitionValueExtractor
+    --jdbc-url jdbc:hive2://hiveserver:10000 \
+    --user hive \
+    --pass hive \
+    --partitioned-by dt \
+    --base-path /user/hive/warehouse/hudi_cow_pt_tbl \
+    --database default \
+    --table hudi_cow_pt_tbl \
+    --partition-value-extractor org.apache.hudi.hive.HiveStylePartitionValueExtractor
 
 echo "Start synchronizing the hudi_non_part_cow table"
 /var/scripts/run_sync_tool.sh \
-  --jdbc-url jdbc:hive2://hiveserver:10000 \
-  --user hive \
-  --pass hive \
-  --base-path /user/hive/warehouse/hudi_non_part_cow \
-  --database default \
-  --table hudi_non_part_cow \
+    --jdbc-url jdbc:hive2://hiveserver:10000 \
+    --user hive \
+    --pass hive \
+    --base-path /user/hive/warehouse/hudi_non_part_cow \
+    --database default \
+    --table hudi_non_part_cow
--- docker/thirdparties/docker-compose/iceberg/tools/save_docker.sh.orig
+++ docker/thirdparties/docker-compose/iceberg/tools/save_docker.sh
--- docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh.orig
+++ docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
@@ -69,6 +69,6 @@
     echo "Invalid index parameter. Exiting."
     exit 1
 fi
-hive  -f /usr/local/sql/create_kerberos_hive_table.sql
+hive -f /usr/local/sql/create_kerberos_hive_table.sql
 
 exec_success_hook
--- docker/thirdparties/docker-compose/kerberos/health-checks/health.sh.orig
+++ docker/thirdparties/docker-compose/kerberos/health-checks/health.sh
@@ -29,6 +29,6 @@
 
 if test -d "${HEALTH_D}"; then
     for health_script in "${HEALTH_D}"/*; do
-        "${health_script}" &>> /var/log/container-health.log || exit 1
+        "${health_script}" &>>/var/log/container-health.log || exit 1
     done
 fi
--- docker/thirdparties/docker-compose/kerberos/health-checks/hive-health-check-2.sh.orig
+++ docker/thirdparties/docker-compose/kerberos/health-checks/hive-health-check-2.sh
--- docker/thirdparties/docker-compose/kerberos/health-checks/hive-health-check.sh.orig
+++ docker/thirdparties/docker-compose/kerberos/health-checks/hive-health-check.sh
--- docker/thirdparties/docker-compose/kerberos/health-checks/supervisorctl-check.sh.orig
+++ docker/thirdparties/docker-compose/kerberos/health-checks/supervisorctl-check.sh
@@ -32,9 +32,9 @@
 FAILED=$(supervisorctl status | grep -v RUNNING || true)
 
 if [ "$FAILED" == "" ]; then
-  echo "All services are running"
-  exit 0
+    echo "All services are running"
+    exit 0
 else
-  echo "Some of the services are failing: ${FAILED}"
-  exit 1
+    echo "Some of the services are failing: ${FAILED}"
+    exit 1
 fi
--- docker/thirdparties/docker-compose/ranger/script/install_doris_ranger_plugins.sh.orig
+++ docker/thirdparties/docker-compose/ranger/script/install_doris_ranger_plugins.sh
--- docker/thirdparties/docker-compose/ranger/script/install_doris_service_def.sh.orig
+++ docker/thirdparties/docker-compose/ranger/script/install_doris_service_def.sh
--- docker/thirdparties/run-thirdparties-docker.sh.orig
+++ docker/thirdparties/run-thirdparties-docker.sh
@@ -187,7 +187,7 @@
         RUN_MARIADB=1
     elif [[ "${element}"x == "db2"x ]]; then
         RUN_DB2=1
-    elif [[ "${element}"x == "oceanbase"x ]];then
+    elif [[ "${element}"x == "oceanbase"x ]]; then
         RUN_OCEANBASE=1
     elif [[ "${element}"x == "lakesoul"x ]]; then
         RUN_LAKESOUL=1
@@ -387,7 +387,7 @@
     . "${ROOT}"/docker-compose/hive/hive-2x_settings.env
     envsubst <"${ROOT}"/docker-compose/hive/hive-2x.yaml.tpl >"${ROOT}"/docker-compose/hive/hive-2x.yaml
     envsubst <"${ROOT}"/docker-compose/hive/hadoop-hive.env.tpl >"${ROOT}"/docker-compose/hive/hadoop-hive-2x.env
-    envsubst <"${ROOT}"/docker-compose/hive/hadoop-hive-2x.env.tpl >> "${ROOT}"/docker-compose/hive/hadoop-hive-2x.env
+    envsubst <"${ROOT}"/docker-compose/hive/hadoop-hive-2x.env.tpl >>"${ROOT}"/docker-compose/hive/hadoop-hive-2x.env
     sudo docker compose -p ${CONTAINER_UID}hive2 -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-2x.env down
     if [[ "${STOP}" -ne 1 ]]; then
         sudo docker compose -p ${CONTAINER_UID}hive2 -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-2x.env up --build --remove-orphans -d --wait
@@ -426,7 +426,7 @@
     . "${ROOT}"/docker-compose/hive/hive-3x_settings.env
     envsubst <"${ROOT}"/docker-compose/hive/hive-3x.yaml.tpl >"${ROOT}"/docker-compose/hive/hive-3x.yaml
     envsubst <"${ROOT}"/docker-compose/hive/hadoop-hive.env.tpl >"${ROOT}"/docker-compose/hive/hadoop-hive-3x.env
-    envsubst <"${ROOT}"/docker-compose/hive/hadoop-hive-3x.env.tpl >> "${ROOT}"/docker-compose/hive/hadoop-hive-3x.env
+    envsubst <"${ROOT}"/docker-compose/hive/hadoop-hive-3x.env.tpl >>"${ROOT}"/docker-compose/hive/hadoop-hive-3x.env
     sudo docker compose -p ${CONTAINER_UID}hive3 -f "${ROOT}"/docker-compose/hive/hive-3x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-3x.env down
     if [[ "${STOP}" -ne 1 ]]; then
         sudo docker compose -p ${CONTAINER_UID}hive3 -f "${ROOT}"/docker-compose/hive/hive-3x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-3x.env up --build --remove-orphans -d --wait
@@ -452,12 +452,12 @@
     if [[ "${STOP}" -ne 1 ]]; then
         if [[ ! -d "${ICEBERG_DIR}/data" ]]; then
             echo "${ICEBERG_DIR}/data does not exist"
-            cd "${ICEBERG_DIR}" \
-            && rm -f iceberg_data.zip \
-            && wget -P "${ROOT}"/docker-compose/iceberg https://"${s3BucketName}.${s3Endpoint}"/regression/datalake/pipeline_data/iceberg_data.zip \
-            && sudo unzip iceberg_data.zip \
-            && sudo mv iceberg_data data \
-            && sudo rm -rf iceberg_data.zip
+            cd "${ICEBERG_DIR}" &&
+                rm -f iceberg_data.zip &&
+                wget -P "${ROOT}"/docker-compose/iceberg https://"${s3BucketName}.${s3Endpoint}"/regression/datalake/pipeline_data/iceberg_data.zip &&
+                sudo unzip iceberg_data.zip &&
+                sudo mv iceberg_data data &&
+                sudo rm -rf iceberg_data.zip
             cd -
         else
             echo "${ICEBERG_DIR}/data exist, continue !"
@@ -619,9 +619,9 @@
     for i in {1..2}; do
         . "${ROOT}"/docker-compose/kerberos/kerberos${i}_settings.env
         envsubst <"${ROOT}"/docker-compose/kerberos/hadoop-hive.env.tpl >"${ROOT}"/docker-compose/kerberos/hadoop-hive-${i}.env
-        envsubst <"${ROOT}"/docker-compose/kerberos/conf/my.cnf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/my.cnf
-        envsubst <"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/kdc.conf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/kdc.conf
-        envsubst <"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/krb5.conf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/krb5.conf
+        envsubst <"${ROOT}"/docker-compose/kerberos/conf/my.cnf.tpl >"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/my.cnf
+        envsubst <"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/kdc.conf.tpl >"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/kdc.conf
+        envsubst <"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/krb5.conf.tpl >"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/krb5.conf
     done
     sudo chmod a+w /etc/hosts
     sudo sed -i "1i${IP_HOST} hadoop-master" /etc/hosts
@@ -672,102 +672,102 @@
 declare -A pids
 
 if [[ "${RUN_ES}" -eq 1 ]]; then
-    start_es > start_es.log  2>&1 &
+    start_es >start_es.log 2>&1 &
     pids["es"]=$!
 fi
 
 if [[ "${RUN_MYSQL}" -eq 1 ]]; then
-    start_mysql > start_mysql.log 2>&1 &
+    start_mysql >start_mysql.log 2>&1 &
     pids["mysql"]=$!
 fi
 
 if [[ "${RUN_PG}" -eq 1 ]]; then
-    start_pg > start_pg.log 2>&1 &
+    start_pg >start_pg.log 2>&1 &
     pids["pg"]=$!
 fi
 
 if [[ "${RUN_ORACLE}" -eq 1 ]]; then
-    start_oracle > start_oracle.log 2>&1 &
+    start_oracle >start_oracle.log 2>&1 &
     pids["oracle"]=$!
 fi
 
 if [[ "${RUN_DB2}" -eq 1 ]]; then
-    start_db2 > start_db2.log 2>&1 &
+    start_db2 >start_db2.log 2>&1 &
     pids["db2"]=$!
 fi
 
 if [[ "${RUN_OCEANBASE}" -eq 1 ]]; then
-    start_oceanbase > start_oceanbase.log 2>&1 &
+    start_oceanbase >start_oceanbase.log 2>&1 &
     pids["oceanbase"]=$!
 fi
 
 if [[ "${RUN_SQLSERVER}" -eq 1 ]]; then
-    start_sqlserver > start_sqlserver.log 2>&1 &
+    start_sqlserver >start_sqlserver.log 2>&1 &
     pids["sqlserver"]=$!
 fi
 
 if [[ "${RUN_CLICKHOUSE}" -eq 1 ]]; then
-    start_clickhouse > start_clickhouse.log 2>&1 &
+    start_clickhouse >start_clickhouse.log 2>&1 &
     pids["clickhouse"]=$!
 fi
 
 if [[ "${RUN_KAFKA}" -eq 1 ]]; then
-    start_kafka > start_kafka.log 2>&1 &
+    start_kafka >start_kafka.log 2>&1 &
     pids["kafka"]=$!
 fi
 
 if [[ "${RUN_HIVE2}" -eq 1 ]]; then
-    start_hive2 > start_hive2.log 2>&1 &
+    start_hive2 >start_hive2.log 2>&1 &
     pids["hive2"]=$!
 fi
 
 if [[ "${RUN_HIVE3}" -eq 1 ]]; then
-    start_hive3 > start_hive3.log 2>&1 &
+    start_hive3 >start_hive3.log 2>&1 &
     pids["hive3"]=$!
 fi
 
 if [[ "${RUN_SPARK}" -eq 1 ]]; then
-    start_spark > start_spark.log 2>&1 &
+    start_spark >start_spark.log 2>&1 &
     pids["spark"]=$!
 fi
 
 if [[ "${RUN_ICEBERG}" -eq 1 ]]; then
-    start_iceberg > start_icerberg.log 2>&1 &
+    start_iceberg >start_icerberg.log 2>&1 &
     pids["iceberg"]=$!
 fi
 
 if [[ "${RUN_HUDI}" -eq 1 ]]; then
-    start_hudi > start_hudi.log 2>&1 &
+    start_hudi >start_hudi.log 2>&1 &
     pids["hudi"]=$!
 fi
 
 if [[ "${RUN_TRINO}" -eq 1 ]]; then
-    start_trino > start_trino.log 2>&1 &
+    start_trino >start_trino.log 2>&1 &
     pids["trino"]=$!
 fi
 
 if [[ "${RUN_MARIADB}" -eq 1 ]]; then
-    start_mariadb > start_mariadb.log 2>&1 &
+    start_mariadb >start_mariadb.log 2>&1 &
     pids["mariadb"]=$!
 fi
 
 if [[ "${RUN_LAKESOUL}" -eq 1 ]]; then
-    start_lakesoul > start_lakesoule.log 2>&1 &
+    start_lakesoul >start_lakesoule.log 2>&1 &
     pids["lakesoul"]=$!
 fi
 
 if [[ "${RUN_MINIO}" -eq 1 ]]; then
-    start_minio > start_minio.log 2>&1 &
+    start_minio >start_minio.log 2>&1 &
     pids["minio"]=$!
 fi
 
 if [[ "${RUN_KERBEROS}" -eq 1 ]]; then
-    start_kerberos > start_kerberos.log 2>&1 &
+    start_kerberos >start_kerberos.log 2>&1 &
     pids["kerberos"]=$!
 fi
 
 if [[ "${RUN_RANGER}" -eq 1 ]]; then
-    start_ranger > start_ranger.log 2>&1 &
+    start_ranger >start_ranger.log 2>&1 &
     pids["ranger"]=$!
 fi
 
--- thirdparty/download-thirdparty.sh.orig
+++ thirdparty/download-thirdparty.sh
@@ -603,5 +603,4 @@
     echo "Finished patching ${FAISS_SOURCE}"
 fi
 
-
 # vim: ts=4 sw=4 ts=4 tw=100:
--- thirdparty/vars.sh.orig
+++ thirdparty/vars.sh
@@ -550,7 +550,6 @@
 FAISS_SOURCE="faiss-1.10.0"
 FAISS_MD5SUM="f31edf2492808b27cc963d0ab316a205"
 
-
 # all thirdparties which need to be downloaded is set in array TP_ARCHIVES
 export TP_ARCHIVES=(
     'LIBEVENT'
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

@github-actions

github-actions Bot commented Apr 8, 2025

Copy link
Copy Markdown

sh-checker report

To get the full details, please check in the job output.

shellcheck errors 

'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In bin/flight_record_fe.sh line 47:
FE_PID=$(${JAVA_HOME}/bin/jps | grep DorisFE | awk '{print $1}')
         ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
FE_PID=$("${JAVA_HOME}"/bin/jps | grep DorisFE | awk '{print $1}')


In bin/profile_fe.sh line 47:
FE_PID=$(${JAVA_HOME}/bin/jps | grep DorisFE | awk '{print $1}')
         ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
FE_PID=$("${JAVA_HOME}"/bin/jps | grep DorisFE | awk '{print $1}')


In build.sh line 926:
    if [[ "${TARGET_SYSTEM}" == 'Linux' ]] && [[ "$TARGET_ARCH" == 'x86_64' ]]; then
                                                  ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    if [[ "${TARGET_SYSTEM}" == 'Linux' ]] && [[ "${TARGET_ARCH}" == 'x86_64' ]]; then


In build.sh line 930:
    elif [[ "${TARGET_SYSTEM}" == 'Linux' ]] && [[ "$TARGET_ARCH" == 'aarch64' ]]; then
                                                    ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    elif [[ "${TARGET_SYSTEM}" == 'Linux' ]] && [[ "${TARGET_ARCH}" == 'aarch64' ]]; then


In docker/thirdparties/docker-compose/common/hive-configure.sh line 22:
export CORE_CONF_fs_defaultFS=${CORE_CONF_fs_defaultFS:-hdfs://`hostname -f`:8020}
                                                               ^-----------^ SC2006 (style): Use $(...) notation instead of legacy backticks `...`.

Did you mean: 
export CORE_CONF_fs_defaultFS=${CORE_CONF_fs_defaultFS:-hdfs://$(hostname -f):8020}


In docker/thirdparties/docker-compose/common/hive-configure.sh line 29:
  local entry="<property><name>$name</name><value>${value}</value></property>"
                               ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
  local entry="<property><name>${name}</name><value>${value}</value></property>"


In docker/thirdparties/docker-compose/common/hive-configure.sh line 30:
  local escapedEntry=$(echo $entry | sed 's/\//\\\//g')
        ^----------^ SC2155 (warning): Declare and assign separately to avoid masking return values.
                            ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
                            ^----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
  local escapedEntry=$(echo "${entry}" | sed 's/\//\\\//g')


In docker/thirdparties/docker-compose/common/hive-configure.sh line 31:
  sed -i "/<\/configuration>/ s/.*/${escapedEntry}\n&/" $path
                                                        ^---^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                        ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
  sed -i "/<\/configuration>/ s/.*/${escapedEntry}\n&/" "${path}"


In docker/thirdparties/docker-compose/common/hive-configure.sh line 42:
    echo "Configuring $module"
                      ^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    echo "Configuring ${module}"


In docker/thirdparties/docker-compose/common/hive-configure.sh line 43:
    for c in `printenv | perl -sne 'print "$1 " if m/^${envPrefix}_(.+?)=.*/' -- -envPrefix=$envPrefix`; do 
             ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
                                                                                            ^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                                            ^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    for c in $(printenv | perl -sne 'print "$1 " if m/^${envPrefix}_(.+?)=.*/' -- -envPrefix="${envPrefix}"); do 


In docker/thirdparties/docker-compose/common/hive-configure.sh line 44:
        name=`echo ${c} | perl -pe 's/___/-/g; s/__/_/g; s/_/./g'`
             ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
                   ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
        name=$(echo "${c}" | perl -pe 's/___/-/g; s/__/_/g; s/_/./g')


In docker/thirdparties/docker-compose/common/hive-configure.sh line 47:
        echo " - Setting $name=$  "
                         ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        echo " - Setting ${name}=$  "


In docker/thirdparties/docker-compose/common/hive-configure.sh line 48:
        addProperty $path $name "$value"
                    ^---^ SC2086 (info): Double quote to prevent globbing and word splitting.
                    ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                          ^---^ SC2086 (info): Double quote to prevent globbing and word splitting.
                          ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                 ^----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        addProperty "${path}" "${name}" "${value}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 26:
    // clear output file
    ^-- SC1127 (error): Was this intended as a comment? Use # in sh.


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 27:
    echo "" > "$output_file"
               ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    echo "" > "${output_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 31:
        if [ -n "$type_value" ]; then
           ^------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
                 ^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        if [[ -n "${type_value}" ]]; then


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 32:
            echo "{\"index\": {\"_index\": \"$index_name\", \"_type\": \"$type_value\", \"_id\": \"${id_prefix}${id}\"}}"  >> "$output_file"
                                             ^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                         ^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                                                                               ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            echo "{\"index\": {\"_index\": \"${index_name}\", \"_type\": \"${type_value}\", \"_id\": \"${id_prefix}${id}\"}}"  >> "${output_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 34:
            echo "{\"index\": {\"_index\": \"$index_name\", \"_id\": \"${id_prefix}${id}\"}}"  >> "$output_file"
                                             ^---------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                                                   ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            echo "{\"index\": {\"_index\": \"${index_name}\", \"_id\": \"${id_prefix}${id}\"}}"  >> "${output_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 36:
        echo "$line"  >> "$output_file"
              ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                          ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        echo "${line}"  >> "${output_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 38:
    done < "$data_file"
            ^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    done < "${data_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 79:
generate_bulk_request "composite_type_array" "doc" "item_" "$array_data_file" "$bulk_request_file"
                                                            ^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                               ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
generate_bulk_request "composite_type_array" "doc" "item_" "${array_data_file}" "${bulk_request_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 80:
curl -X POST "http://${ES_5_HOST}:9200/_bulk" --data-binary "@$bulk_request_file" -H "Content-Type: application/json"
                                                              ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
curl -X POST "http://${ES_5_HOST}:9200/_bulk" --data-binary "@${bulk_request_file}" -H "Content-Type: application/json"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 119:
generate_bulk_request "composite_type_array" "doc" "item_" "$array_data_file" "$bulk_request_file"
                                                            ^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                               ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
generate_bulk_request "composite_type_array" "doc" "item_" "${array_data_file}" "${bulk_request_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 120:
curl -X POST "http://${ES_6_HOST}:9200/_bulk" --data-binary "@$bulk_request_file" -H "Content-Type: application/json"
                                                              ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
curl -X POST "http://${ES_6_HOST}:9200/_bulk" --data-binary "@${bulk_request_file}" -H "Content-Type: application/json"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 126:
curl "http://${ES_7_HOST}:9200/test1" -H "Content-Type:application/json" -X PUT -d "@/mnt/scripts/index/es7_test1.json"
             ^----------^ SC2154 (warning): ES_7_HOST is referenced but not assigned.


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 164:
generate_bulk_request "composite_type_array" "_doc" "item_" "$array_data_file" "$bulk_request_file"
                                                             ^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                                ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
generate_bulk_request "composite_type_array" "_doc" "item_" "${array_data_file}" "${bulk_request_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 165:
curl -X POST "http://${ES_7_HOST}:9200/_bulk" --data-binary "@$bulk_request_file" -H "Content-Type: application/json"
                                                              ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
curl -X POST "http://${ES_7_HOST}:9200/_bulk" --data-binary "@${bulk_request_file}" -H "Content-Type: application/json"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 171:
curl "http://${ES_8_HOST}:9200/test1" -H "Content-Type:application/json" -X PUT -d "@/mnt/scripts/index/es7_test1.json"
             ^----------^ SC2154 (warning): ES_8_HOST is referenced but not assigned.


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 207:
generate_bulk_request "composite_type_array" "" "item_" "$array_data_file" "$bulk_request_file"
                                                         ^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                            ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
generate_bulk_request "composite_type_array" "" "item_" "${array_data_file}" "${bulk_request_file}"


In docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh line 208:
curl -X POST "http://${ES_8_HOST}:9200/_bulk" --data-binary "@$bulk_request_file" -H "Content-Type: application/json"
                                                              ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
curl -X POST "http://${ES_8_HOST}:9200/_bulk" --data-binary "@${bulk_request_file}" -H "Content-Type: application/json"


In docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/logs1_parquet/run.sh line 11:
    cd -
    ^--^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Did you mean: 
    cd - || exit


In docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_complex_types/run.sh line 11:
    cd -
    ^--^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Did you mean: 
    cd - || exit


In docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_compress_partitioned/run.sh line 11:
    cd -
    ^--^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Did you mean: 
    cd - || exit


In docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_wide_table/run.sh line 11:
    cd -
    ^--^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Did you mean: 
    cd - || exit


In docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_hdfs_tvf_compression/run.sh line 9:
    cd ${CUR_DIR}/ && rm -f test_data.tar.gz \
       ^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
    cd "${CUR_DIR}"/ && rm -f test_data.tar.gz \


In docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_hdfs_tvf_compression/run.sh line 12:
    cd -
    ^--^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Did you mean: 
    cd - || exit


In docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_tvf/run.sh line 9:
    cd ${CUR_DIR}/ && rm -f data.tar.gz \
       ^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
    cd "${CUR_DIR}"/ && rm -f data.tar.gz \


In docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_tvf/run.sh line 12:
    cd -
    ^--^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Did you mean: 
    cd - || exit


In docker/thirdparties/docker-compose/hive/scripts/hive-metastore.sh line 33:
while ! $(nc -z localhost "${HMS_PORT:-9083}"); do
        ^-- SC2091 (warning): Remove surrounding $() to avoid executing output (or use eval if intentional).


In docker/thirdparties/docker-compose/hive/scripts/hive-metastore.sh line 42:
find "${DATA_DIR}" -type f -name "run.sh" -print0 | xargs -0 -n 1 -P "${parallel}" -I {} bash -ec '
                                                                                                  ^-- SC2016 (info): Expressions don't expand in single quotes, use double quotes for that.


In docker/thirdparties/docker-compose/hive/scripts/hive-metastore.sh line 148:
ls /mnt/scripts/create_preinstalled_scripts/*.hql | xargs -n 1 -P "${parallel}" -I {} bash -ec '
^-- SC2011 (warning): Use 'find .. -print0 | xargs -0 ..' or 'find .. -exec .. +' to allow non-alphanumeric filenames.
                                                                                               ^-- SC2016 (info): Expressions don't expand in single quotes, use double quotes for that.


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 25:
if [ -z "${HADOOP_HOME}" ]; then
   ^---------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
if [[ -z "${HADOOP_HOME}" ]]; then


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 29:
if [ -z "${HIVE_HOME}" ]; then
   ^-------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
if [[ -z "${HIVE_HOME}" ]]; then


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 35:
HUDI_HIVE_UBER_JAR=`ls -c $DIR/./hudi_docker_compose_attached_file/jar/hoodie-hive-sync-bundle.jar | grep -v source | head -1`
                   ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
                    ^-- SC2010 (warning): Don't use ls | grep. Use a glob or a for loop with a condition to allow non-alphanumeric filenames.
                          ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
                          ^--^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
HUDI_HIVE_UBER_JAR=$(ls -c "${DIR}"/./hudi_docker_compose_attached_file/jar/hoodie-hive-sync-bundle.jar | grep -v source | head -1)


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 37:
if [ -z "$HADOOP_CONF_DIR" ]; then
   ^-----------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
         ^--------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
if [[ -z "${HADOOP_CONF_DIR}" ]]; then


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 43:
HIVE_EXEC=`ls ${HIVE_HOME}/lib/hive-exec-*.jar | tr '\n' ':'`
          ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
           ^-- SC2012 (info): Use find instead of ls to better handle non-alphanumeric filenames.
              ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
HIVE_EXEC=$(ls "${HIVE_HOME}"/lib/hive-exec-*.jar | tr '\n' ':')


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 44:
HIVE_SERVICE=`ls ${HIVE_HOME}/lib/hive-service-*.jar | grep -v rpc | tr '\n' ':'`
             ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
              ^-- SC2010 (warning): Don't use ls | grep. Use a glob or a for loop with a condition to allow non-alphanumeric filenames.
                 ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
HIVE_SERVICE=$(ls "${HIVE_HOME}"/lib/hive-service-*.jar | grep -v rpc | tr '\n' ':')


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 45:
HIVE_METASTORE=`ls ${HIVE_HOME}/lib/hive-metastore-*.jar | tr '\n' ':'`
               ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
                ^-- SC2012 (info): Use find instead of ls to better handle non-alphanumeric filenames.
                   ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
HIVE_METASTORE=$(ls "${HIVE_HOME}"/lib/hive-metastore-*.jar | tr '\n' ':')


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 46:
HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | tr '\n' ':'`
          ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
           ^-- SC2012 (info): Use find instead of ls to better handle non-alphanumeric filenames.
              ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
HIVE_JDBC=$(ls "${HIVE_HOME}"/lib/hive-jdbc-*.jar | tr '\n' ':')


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 47:
if [ -z "${HIVE_JDBC}" ]; then
   ^-------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
if [[ -z "${HIVE_JDBC}" ]]; then


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 48:
  HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | grep -v handler | tr '\n' ':'`
            ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
             ^-- SC2010 (warning): Don't use ls | grep. Use a glob or a for loop with a condition to allow non-alphanumeric filenames.
                ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
  HIVE_JDBC=$(ls "${HIVE_HOME}"/lib/hive-jdbc-*.jar | grep -v handler | tr '\n' ':')


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 50:
HIVE_JACKSON=`ls ${HIVE_HOME}/lib/jackson-*.jar | tr '\n' ':'`
             ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.
              ^-- SC2012 (info): Use find instead of ls to better handle non-alphanumeric filenames.
                 ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
HIVE_JACKSON=$(ls "${HIVE_HOME}"/lib/jackson-*.jar | tr '\n' ':')


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 51:
HIVE_JARS=$HIVE_METASTORE:$HIVE_SERVICE:$HIVE_EXEC:$HIVE_JDBC:$HIVE_JACKSON
          ^-------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                          ^-----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                        ^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                   ^--------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                              ^-----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
HIVE_JARS=${HIVE_METASTORE}:${HIVE_SERVICE}:${HIVE_EXEC}:${HIVE_JDBC}:${HIVE_JACKSON}


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 53:
HADOOP_HIVE_JARS=${HIVE_JARS}:${HADOOP_HOME}/share/hadoop/common/*:${HADOOP_HOME}/share/hadoop/mapreduce/*:${HADOOP_HOME}/share/hadoop/hdfs/*:${HADOOP_HOME}/share/hadoop/common/lib/*:${HADOOP_HOME}/share/hadoop/hdfs/lib/*
                 ^-- SC2125 (warning): Brace expansions and globs are literal in assignments. Quote it or use an array.


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 55:
echo "Running Command : java -cp ${HADOOP_HIVE_JARS}:${HADOOP_CONF_DIR}:$HUDI_HIVE_UBER_JAR org.apache.hudi.hive.HiveSyncTool $@"
                                                                        ^-----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                                                                                              ^-- SC2145 (error): Argument mixes string and array. Use * or separate argument.

Did you mean: 
echo "Running Command : java -cp ${HADOOP_HIVE_JARS}:${HADOOP_CONF_DIR}:${HUDI_HIVE_UBER_JAR} org.apache.hudi.hive.HiveSyncTool $@"


In docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh line 56:
java -cp $HUDI_HIVE_UBER_JAR:${HADOOP_HIVE_JARS}:${HADOOP_CONF_DIR} org.apache.hudi.hive.HiveSyncTool "$@"
         ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
         ^-----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                             ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                 ^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
java -cp "${HUDI_HIVE_UBER_JAR}":"${HADOOP_HIVE_JARS}":"${HADOOP_CONF_DIR}" org.apache.hudi.hive.HiveSyncTool "$@"


In docker/thirdparties/docker-compose/hudi/scripts/setup_demo_container_adhoc_1.sh line 20:
cp /var/scripts/config/spark-defaults.conf $SPARK_CONF_DIR/.
                                           ^-------------^ SC2154 (warning): SPARK_CONF_DIR is referenced but not assigned.
                                           ^-------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                           ^-------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
cp /var/scripts/config/spark-defaults.conf "${SPARK_CONF_DIR}"/.


In docker/thirdparties/docker-compose/hudi/scripts/setup_demo_container_adhoc_1.sh line 21:
cp /var/scripts/config/log4j2.properties $SPARK_CONF_DIR/.
                                         ^-------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                         ^-------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
cp /var/scripts/config/log4j2.properties "${SPARK_CONF_DIR}"/.


In docker/thirdparties/docker-compose/hudi/scripts/setup_demo_container_adhoc_2.sh line 20:
cp /var/scripts/config/spark-defaults.conf $SPARK_CONF_DIR/.
                                           ^-------------^ SC2154 (warning): SPARK_CONF_DIR is referenced but not assigned.
                                           ^-------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                           ^-------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
cp /var/scripts/config/spark-defaults.conf "${SPARK_CONF_DIR}"/.


In docker/thirdparties/docker-compose/hudi/scripts/setup_demo_container_adhoc_2.sh line 21:
cp /var/scripts/config/log4j2.properties $SPARK_CONF_DIR/.
                                         ^-------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                         ^-------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
cp /var/scripts/config/log4j2.properties "${SPARK_CONF_DIR}"/.


In docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh line 19:
source /usr/local/common/hive-configure.sh
       ^-- SC1091 (info): Not following: /usr/local/common/hive-configure.sh: openBinaryFile: does not exist (No such file or directory)


In docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh line 20:
source /usr/local/common/event-hook.sh
       ^-----------------------------^ SC1091 (info): Not following: /usr/local/common/event-hook.sh: openBinaryFile: does not exist (No such file or directory)


In docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh line 34:
if [ "$1" == "1" ]; then
   ^-------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
if [[ "$1" == "1" ]]; then


In docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh line 36:
elif [ "$1" == "2" ]; then
     ^-------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
elif [[ "$1" == "2" ]]; then


In docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh line 57:
if [ $i -eq 60 ]; then
   ^-----------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
     ^-- SC2086 (info): Double quote to prevent globbing and word splitting.
     ^-- SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
if [[ "${i}" -eq 60 ]]; then


In docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh line 64:
if [ "$1" == "1" ]; then
   ^-------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
if [[ "$1" == "1" ]]; then


In docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh line 66:
elif [ "$1" == "2" ]; then
     ^-------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
elif [[ "$1" == "2" ]]; then


In docker/thirdparties/docker-compose/kerberos/health-checks/supervisorctl-check.sh line 34:
if [ "$FAILED" == "" ]; then
   ^-----------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
      ^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
if [[ "${FAILED}" == "" ]]; then


In docker/thirdparties/docker-compose/ranger/ranger-admin/ranger-entrypoint.sh line 24:
cd $RANGER_HOME
   ^----------^ SC2154 (warning): RANGER_HOME is referenced but not assigned.
   ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
   ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
cd "${RANGER_HOME}"


In docker/thirdparties/docker-compose/ranger/script/install_doris_ranger_plugins.sh line 16:
#!/bin/bash
^-- SC1128 (error): The shebang must be on the first line. Delete blanks and move comments.


In docker/thirdparties/docker-compose/ranger/script/install_doris_ranger_plugins.sh line 19:
if [ ! -d "${RANGER_HOME}/ews/webapp/WEB-INF/classes/ranger-plugins/doris" ]; then
   ^-- SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
           ^------------^ SC2154 (warning): RANGER_HOME is referenced but not assigned.

Did you mean: 
if [[ ! -d "${RANGER_HOME}/ews/webapp/WEB-INF/classes/ranger-plugins/doris" ]]; then


In docker/thirdparties/docker-compose/ranger/script/install_doris_service_def.sh line 15:
#!/bin/bash
^-- SC1128 (error): The shebang must be on the first line. Delete blanks and move comments.


In docker/thirdparties/run-thirdparties-docker.sh line 117:
    echo ${COMPONENTS}
         ^-----------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
    echo "${COMPONENTS}"


In docker/thirdparties/run-thirdparties-docker.sh line 150:
RUN_OCENABASE=0
^-----------^ SC2034 (warning): RUN_OCENABASE appears unused. Verify use (or export if used externally).


In docker/thirdparties/run-thirdparties-docker.sh line 330:
    IP_HOST=$(ifconfig "${eth_name}" | grep inet | grep -v 127.0.0.1 | grep -v inet6 | awk '{print $2}' | tr -d "addr:" | head -n 1)
                                                                                                                ^-----^ SC2020 (info): tr replaces sets of chars, not words (mentioned due to duplicates).


In docker/thirdparties/run-thirdparties-docker.sh line 339:
        local backup_dir=/home/work/pipline/backup_center
              ^--------^ SC2034 (warning): backup_dir appears unused. Verify use (or export if used externally).


In docker/thirdparties/run-thirdparties-docker.sh line 344:
            echo "docker exec "${container_id}" bash -c echo '/opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server '${ip_host}:19193' --topic '${topic}'"
                               ^-------------^ SC2027 (warning): The surrounding quotes actually unquote this. Remove or escape them.
                               ^-------------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
            echo "docker exec ""${container_id}"" bash -c echo '/opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server '${ip_host}:19193' --topic '${topic}'"


In docker/thirdparties/run-thirdparties-docker.sh line 363:
    IP_HOST=$(ifconfig "${eth_name}" | grep inet | grep -v 127.0.0.1 | grep -v inet6 | awk '{print $2}' | tr -d "addr:" | head -n 1)
                                                                                                                ^-----^ SC2020 (info): tr replaces sets of chars, not words (mentioned due to duplicates).


In docker/thirdparties/run-thirdparties-docker.sh line 365:
    if [ "_${IP_HOST}" == "_" ]; then
       ^----------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
    if [[ "_${IP_HOST}" == "_" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 367:
        exit -1
             ^-- SC2242 (error): Can only exit with status 0-255. Other data should be written to stdout/stderr.


In docker/thirdparties/run-thirdparties-docker.sh line 375:
        if [ -f "$file" ]; then
           ^------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
                 ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        if [[ -f "${file}" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 376:
            echo "Processing $file"
                             ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            echo "Processing ${file}"


In docker/thirdparties/run-thirdparties-docker.sh line 380:
            echo "File not found: $file"
                                  ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            echo "File not found: ${file}"


In docker/thirdparties/run-thirdparties-docker.sh line 391:
    sudo docker compose -p ${CONTAINER_UID}hive2 -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-2x.env down
                           ^--------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
    sudo docker compose -p "${CONTAINER_UID}"hive2 -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-2x.env down


In docker/thirdparties/run-thirdparties-docker.sh line 393:
        sudo docker compose -p ${CONTAINER_UID}hive2 -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-2x.env up --build --remove-orphans -d --wait
                               ^--------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
        sudo docker compose -p "${CONTAINER_UID}"hive2 -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-2x.env up --build --remove-orphans -d --wait


In docker/thirdparties/run-thirdparties-docker.sh line 403:
    IP_HOST=$(ifconfig "${eth_name}" | grep inet | grep -v 127.0.0.1 | grep -v inet6 | awk '{print $2}' | tr -d "addr:" | head -n 1)
                                                                                                                ^-----^ SC2020 (info): tr replaces sets of chars, not words (mentioned due to duplicates).


In docker/thirdparties/run-thirdparties-docker.sh line 404:
    if [ "_${IP_HOST}" == "_" ]; then
       ^----------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
    if [[ "_${IP_HOST}" == "_" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 406:
        exit -1
             ^-- SC2242 (error): Can only exit with status 0-255. Other data should be written to stdout/stderr.


In docker/thirdparties/run-thirdparties-docker.sh line 414:
        if [ -f "$file" ]; then
           ^------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
                 ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        if [[ -f "${file}" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 415:
            echo "Processing $file"
                             ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            echo "Processing ${file}"


In docker/thirdparties/run-thirdparties-docker.sh line 419:
            echo "File not found: $file"
                                  ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            echo "File not found: ${file}"


In docker/thirdparties/run-thirdparties-docker.sh line 430:
    sudo docker compose -p ${CONTAINER_UID}hive3 -f "${ROOT}"/docker-compose/hive/hive-3x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-3x.env down
                           ^--------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
    sudo docker compose -p "${CONTAINER_UID}"hive3 -f "${ROOT}"/docker-compose/hive/hive-3x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-3x.env down


In docker/thirdparties/run-thirdparties-docker.sh line 432:
        sudo docker compose -p ${CONTAINER_UID}hive3 -f "${ROOT}"/docker-compose/hive/hive-3x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-3x.env up --build --remove-orphans -d --wait
                               ^--------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
        sudo docker compose -p "${CONTAINER_UID}"hive3 -f "${ROOT}"/docker-compose/hive/hive-3x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-3x.env up --build --remove-orphans -d --wait


In docker/thirdparties/run-thirdparties-docker.sh line 499:
        cp "${trino_docker}/$file.tpl" "${trino_docker}/$file"
                            ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                        ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        cp "${trino_docker}/${file}.tpl" "${trino_docker}/${file}"


In docker/thirdparties/run-thirdparties-docker.sh line 500:
        if [[ $file != "hive.properties" ]]; then
              ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        if [[ ${file} != "hive.properties" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 501:
            sed -i "s/doris--/${CONTAINER_UID}/g" "${trino_docker}/$file"
                                                                   ^---^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            sed -i "s/doris--/${CONTAINER_UID}/g" "${trino_docker}/${file}"


In docker/thirdparties/run-thirdparties-docker.sh line 510:
        sudo echo "127.0.0.1 ${NAMENODE_CONTAINER_ID}" >>/etc/hosts
                                                       ^-- SC2024 (warning): sudo doesn't affect redirects. Use .. | sudo tee -a file


In docker/thirdparties/run-thirdparties-docker.sh line 512:
        hive_metastore_ip=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' ${HIVE_METASTORE_CONTAINER_ID})
                                                                                                               ^----------------------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
        hive_metastore_ip=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "${HIVE_METASTORE_CONTAINER_ID}")


In docker/thirdparties/run-thirdparties-docker.sh line 514:
        if [ -z "$hive_metastore_ip" ]; then
           ^-------------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
                 ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        if [[ -z "${hive_metastore_ip}" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 518:
            echo "Hive Metastore IP address is: $hive_metastore_ip"
                                                ^----------------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            echo "Hive Metastore IP address is: ${hive_metastore_ip}"


In docker/thirdparties/run-thirdparties-docker.sh line 533:
            while [ $retries -lt $max_retries ]; do
                  ^---------------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
                    ^------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.
                    ^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                 ^----------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.
                                 ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            while [[ "${retries}" -lt "${max_retries}" ]]; do


In docker/thirdparties/run-thirdparties-docker.sh line 534:
                status=$(docker inspect --format '{{.State.Running}}' ${TRINO_CONTAINER_ID})
                                                                      ^-------------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
                status=$(docker inspect --format '{{.State.Running}}' "${TRINO_CONTAINER_ID}")


In docker/thirdparties/run-thirdparties-docker.sh line 535:
                if [ "${status}" == "${expected_status}" ]; then
                   ^-- SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.

Did you mean: 
                if [[ "${status}" == "${expected_status}" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 546:
            if [ $retries -eq $max_retries ]; then
               ^---------------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
                 ^------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.
                 ^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                              ^----------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.
                              ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
            if [[ "${retries}" -eq "${max_retries}" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 547:
                echo "${operation} operation failed to complete after $max_retries attempts."
                                                                      ^----------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
                echo "${operation} operation failed to complete after ${max_retries} attempts."


In docker/thirdparties/run-thirdparties-docker.sh line 552:
        docker stop ${TRINO_CONTAINER_ID}
                    ^-------------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
        docker stop "${TRINO_CONTAINER_ID}"


In docker/thirdparties/run-thirdparties-docker.sh line 557:
        docker start ${TRINO_CONTAINER_ID}
                     ^-------------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
        docker start "${TRINO_CONTAINER_ID}"


In docker/thirdparties/run-thirdparties-docker.sh line 563:
        docker exec -it ${TRINO_CONTAINER_ID} /bin/bash -c 'trino -f /scripts/create_trino_table.sql'
                        ^-------------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
        docker exec -it "${TRINO_CONTAINER_ID}" /bin/bash -c 'trino -f /scripts/create_trino_table.sql'


In docker/thirdparties/run-thirdparties-docker.sh line 601:
        mv *.tbl ../lakesoul/test_files/tpch/data
           ^-- SC2035 (info): Use ./*glob* or -- *glob* so names with dashes won't become options.


In docker/thirdparties/run-thirdparties-docker.sh line 603:
        export TPCH_DATA=$(realpath lakesoul/test_files/tpch/data)
               ^-------^ SC2155 (warning): Declare and assign separately to avoid masking return values.


In docker/thirdparties/run-thirdparties-docker.sh line 615:
    IP_HOST=$(ifconfig "${eth_name}" | grep inet | grep -v 127.0.0.1 | grep -v inet6 | awk '{print $2}' | tr -d "addr:" | head -n 1)
                                                                                                                ^-----^ SC2020 (info): tr replaces sets of chars, not words (mentioned due to duplicates).


In docker/thirdparties/run-thirdparties-docker.sh line 620:
        . "${ROOT}"/docker-compose/kerberos/kerberos${i}_settings.env
          ^-- SC1090 (warning): ShellCheck can't follow non-constant source. Use a directive to specify location.
                                                    ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
        . "${ROOT}"/docker-compose/kerberos/kerberos"${i}"_settings.env


In docker/thirdparties/run-thirdparties-docker.sh line 621:
        envsubst <"${ROOT}"/docker-compose/kerberos/hadoop-hive.env.tpl >"${ROOT}"/docker-compose/kerberos/hadoop-hive-${i}.env
                                                                                                                       ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
        envsubst <"${ROOT}"/docker-compose/kerberos/hadoop-hive.env.tpl >"${ROOT}"/docker-compose/kerberos/hadoop-hive-"${i}".env


In docker/thirdparties/run-thirdparties-docker.sh line 622:
        envsubst <"${ROOT}"/docker-compose/kerberos/conf/my.cnf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/my.cnf
                                                                                                                     ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
        envsubst <"${ROOT}"/docker-compose/kerberos/conf/my.cnf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos"${i}"/my.cnf


In docker/thirdparties/run-thirdparties-docker.sh line 623:
        envsubst <"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/kdc.conf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/kdc.conf
                                                                 ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                                                                                    ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
        envsubst <"${ROOT}"/docker-compose/kerberos/conf/kerberos"${i}"/kdc.conf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos"${i}"/kdc.conf


In docker/thirdparties/run-thirdparties-docker.sh line 624:
        envsubst <"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/krb5.conf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/krb5.conf
                                                                 ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                                                                                     ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
        envsubst <"${ROOT}"/docker-compose/kerberos/conf/kerberos"${i}"/krb5.conf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos"${i}"/krb5.conf


In docker/thirdparties/run-thirdparties-docker.sh line 779:
    wait "${pids[$compose]}" || status=$?
                 ^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    wait "${pids[${compose}]}" || status=$?


In docker/thirdparties/run-thirdparties-docker.sh line 780:
    if [ $status -ne 0 ] && [ $compose != "db2" ]; then
       ^---------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
         ^-----^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.
         ^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                            ^-------------------^ SC2292 (style): Prefer [[ ]] over [ ] for tests in Bash/Ksh.
                              ^------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                              ^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
    if [[ "${status}" -ne 0 ]] && [[ "${compose}" != "db2" ]]; then


In docker/thirdparties/run-thirdparties-docker.sh line 781:
        echo "docker $compose started failed with status $status"
                     ^------^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.
                                                         ^-----^ SC2250 (style): Prefer putting braces around variable references even when not strictly required.

Did you mean: 
        echo "docker ${compose} started failed with status ${status}"


In docker/thirdparties/run-thirdparties-docker.sh line 783:
        cat start_${compose}.log || true
                  ^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
        cat start_"${compose}".log || true


In thirdparty/build-thirdparty.sh line 1868:
    cp -r ${TP_SOURCE_DIR}/${JINDOFS_SOURCE}/* "${TP_INSTALL_DIR}/jindofs_libs/"
          ^--------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                           ^---------------^ SC2248 (style): Prefer double quoting even when variables don't contain special characters.

Did you mean: 
    cp -r "${TP_SOURCE_DIR}"/"${JINDOFS_SOURCE}"/* "${TP_INSTALL_DIR}/jindofs_libs/"


In tools/lzo/build.sh line 1:
# Licensed to the Apache Software Foundation (ASF) under one
^-- SC2148 (error): Tips depend on target shell and yours is unknown. Add a shebang or a 'shell' directive.


In tools/lzo/build.sh line 20:
g++ -o lzo_writer lzo_writer.cpp -I. -Isrc -I${DORIS_THIRDPARTY}/installed/include -L${DORIS_THIRDPARTY}/installed/lib -llzo2 -std=c++17
                                             ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                                                     ^-----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
g++ -o lzo_writer lzo_writer.cpp -I. -Isrc -I"${DORIS_THIRDPARTY}"/installed/include -L"${DORIS_THIRDPARTY}"/installed/lib -llzo2 -std=c++17

For more information:
  https://www.shellcheck.net/wiki/SC1127 -- Was this intended as a comment? U...
  https://www.shellcheck.net/wiki/SC1128 -- The shebang must be on the first ...
  https://www.shellcheck.net/wiki/SC2145 -- Argument mixes string and array. ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
shfmt errors 

'shfmt ' returned error 1 finding the following formatting issues:

----------
--- docker/thirdparties/docker-compose/common/event-hook.sh.orig
+++ docker/thirdparties/docker-compose/common/event-hook.sh
--- docker/thirdparties/docker-compose/common/hive-configure.sh.orig
+++ docker/thirdparties/docker-compose/common/hive-configure.sh
@@ -19,16 +19,16 @@
 # Referenced from [docker-hive](https://github.com/big-data-europe/docker-hive)
 
 # Set some sensible defaults
-export CORE_CONF_fs_defaultFS=${CORE_CONF_fs_defaultFS:-hdfs://`hostname -f`:8020}
+export CORE_CONF_fs_defaultFS=${CORE_CONF_fs_defaultFS:-hdfs://$(hostname -f):8020}
 
 function addProperty() {
-  local path=$1
-  local name=$2
-  local value=$3
+    local path=$1
+    local name=$2
+    local value=$3
 
-  local entry="<property><name>$name</name><value>${value}</value></property>"
-  local escapedEntry=$(echo $entry | sed 's/\//\\\//g')
-  sed -i "/<\/configuration>/ s/.*/${escapedEntry}\n&/" $path
+    local entry="<property><name>$name</name><value>${value}</value></property>"
+    local escapedEntry=$(echo $entry | sed 's/\//\\\//g')
+    sed -i "/<\/configuration>/ s/.*/${escapedEntry}\n&/" $path
 }
 
 function configure() {
@@ -38,10 +38,10 @@
 
     local var
     local value
-    
+
     echo "Configuring $module"
-    for c in `printenv | perl -sne 'print "$1 " if m/^${envPrefix}_(.+?)=.*/' -- -envPrefix=$envPrefix`; do 
-        name=`echo ${c} | perl -pe 's/___/-/g; s/__/_/g; s/_/./g'`
+    for c in $(printenv | perl -sne 'print "$1 " if m/^${envPrefix}_(.+?)=.*/' -- -envPrefix=$envPrefix); do
+        name=$(echo ${c} | perl -pe 's/___/-/g; s/__/_/g; s/_/./g')
         var="${envPrefix}_${c}"
         value=${!var}
         echo " - Setting $name=$  "
--- docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh.orig
+++ docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh
@@ -24,18 +24,18 @@
     local output_file=$5
 
     // clear output file
-    echo "" > "$output_file"
+    echo "" >"$output_file"
 
     local id=1
     while IFS= read -r line; do
         if [ -n "$type_value" ]; then
-            echo "{\"index\": {\"_index\": \"$index_name\", \"_type\": \"$type_value\", \"_id\": \"${id_prefix}${id}\"}}"  >> "$output_file"
+            echo "{\"index\": {\"_index\": \"$index_name\", \"_type\": \"$type_value\", \"_id\": \"${id_prefix}${id}\"}}" >>"$output_file"
         else
-            echo "{\"index\": {\"_index\": \"$index_name\", \"_id\": \"${id_prefix}${id}\"}}"  >> "$output_file"
+            echo "{\"index\": {\"_index\": \"$index_name\", \"_id\": \"${id_prefix}${id}\"}}" >>"$output_file"
         fi
-        echo "$line"  >> "$output_file"
+        echo "$line" >>"$output_file"
         id=$((id + 1))
-    done < "$data_file"
+    done <"$data_file"
 }
 
 array_data_file="/mnt/scripts/data/composite_type_array_bulk.json"
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_csv/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_csv/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}/create_table.hql"
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_orc/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_orc/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_parquet/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/datev2_parquet/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_config_test/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_config_test/run.sh
@@ -11,4 +11,3 @@
 hive -f "${CUR_DIR}"/create_table.hql
 
 hadoop fs -rm -r /user/doris/suites/default/hive_ignore_absent_partitions_table/country=India
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_array_all_types/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_array_all_types/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_array_delimiter/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_array_delimiter/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_nestedarray/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_textfile_nestedarray/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_upper_case_orc/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_upper_case_orc/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_upper_case_parquet/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/hive_upper_case_parquet/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/logs1_parquet/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/logs1_parquet/run.sh
@@ -5,9 +5,9 @@
 
 if [[ ! -d "${CUR_DIR}/data" ]]; then
     echo "${CUR_DIR}/data does not exist"
-    cd "${CUR_DIR}" && rm -f data.tar.gz \
-    && curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/multi_catalog/logs1_parquet/data.tar.gz \
-    && tar xzf data.tar.gz
+    cd "${CUR_DIR}" && rm -f data.tar.gz &&
+        curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/multi_catalog/logs1_parquet/data.tar.gz &&
+        tar xzf data.tar.gz
     cd -
 else
     echo "${CUR_DIR}/data exist, continue !"
@@ -19,4 +19,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/one_partition/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/one_partition/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}/create_table.hql"
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/orc_predicate/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/orc_predicate/run.sh
@@ -5,5 +5,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/orc_predicate_table.hql
-
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/parquet_lz4_compression/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/parquet_lz4_compression/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/parquet_lzo_compression/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/parquet_lzo_compression/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/partition_manual_remove/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/partition_manual_remove/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_complex_types/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_complex_types/run.sh
@@ -5,9 +5,9 @@
 
 if [[ ! -d "${CUR_DIR}/data" ]]; then
     echo "${CUR_DIR}/data does not exist"
-    cd "${CUR_DIR}" && rm -f data.tar.gz \
-    && curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/multi_catalog/test_complex_types/data.tar.gz \
-    && tar xzf data.tar.gz
+    cd "${CUR_DIR}" && rm -f data.tar.gz &&
+        curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/multi_catalog/test_complex_types/data.tar.gz &&
+        tar xzf data.tar.gz
     cd -
 else
     echo "${CUR_DIR}/data exist, continue !"
@@ -19,4 +19,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_compress_partitioned/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_compress_partitioned/run.sh
@@ -5,9 +5,9 @@
 
 if [[ ! -d "${CUR_DIR}/data" ]]; then
     echo "${CUR_DIR}/data does not exist"
-    cd "${CUR_DIR}" && rm -f data.tar.gz \
-    && curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/multi_catalog/test_compress_partitioned/data.tar.gz \
-    && tar xzf data.tar.gz
+    cd "${CUR_DIR}" && rm -f data.tar.gz &&
+        curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/multi_catalog/test_compress_partitioned/data.tar.gz &&
+        tar xzf data.tar.gz
     cd -
 else
     echo "${CUR_DIR}/data exist, continue !"
@@ -19,4 +19,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_csv_format_error/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_csv_format_error/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}/create_table.hql"
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_date_string_partition/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_date_string_partition/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}/create_table.hql"
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_wide_table/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/test_wide_table/run.sh
@@ -5,9 +5,9 @@
 
 if [[ ! -d "${CUR_DIR}/data" ]]; then
     echo "${CUR_DIR}/data does not exist"
-    cd "${CUR_DIR}" && rm -f data.tar.gz \
-    && curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/multi_catalog/test_wide_table/data.tar.gz \
-    && tar xzf data.tar.gz
+    cd "${CUR_DIR}" && rm -f data.tar.gz &&
+        curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/multi_catalog/test_wide_table/data.tar.gz &&
+        tar xzf data.tar.gz
     cd -
 else
     echo "${CUR_DIR}/data exist, continue !"
@@ -19,4 +19,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/two_partition/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/multi_catalog/two_partition/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}/create_table.hql"
-
--- docker/thirdparties/docker-compose/hive/scripts/data/regression/crdmm_data/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/regression/crdmm_data/run.sh
@@ -10,5 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/create_table.hql
-
-
--- docker/thirdparties/docker-compose/hive/scripts/data/regression/serde_prop/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/regression/serde_prop/run.sh
@@ -5,5 +5,3 @@
 
 # create table
 hive -f "${CUR_DIR}"/some_serde_table.hql
-
-
--- docker/thirdparties/docker-compose/hive/scripts/data/statistics/statistics/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/statistics/statistics/run.sh
@@ -10,4 +10,3 @@
 
 # create table
 hive -f "${CUR_DIR}/create_table.hql"
-
--- docker/thirdparties/docker-compose/hive/scripts/data/tpch_1000_parquet/part/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/tpch_1000_parquet/part/run.sh
@@ -19,4 +19,3 @@
 
 # # create table
 # hive -f "${CUR_DIR}"/create_table.hql
-
--- docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_hdfs_tvf_compression/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_hdfs_tvf_compression/run.sh
@@ -6,9 +6,9 @@
 
 if [[ ! -d "${CUR_DIR}/test_data" ]]; then
     echo "${CUR_DIR}/test_data does not exist"
-    cd ${CUR_DIR}/ && rm -f test_data.tar.gz \
-    && curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/test_hdfs_tvf_compression/test_data.tar.gz \
-    && tar xzf test_data.tar.gz
+    cd ${CUR_DIR}/ && rm -f test_data.tar.gz &&
+        curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/test_hdfs_tvf_compression/test_data.tar.gz &&
+        tar xzf test_data.tar.gz
     cd -
 else
     echo "${CUR_DIR}/test_data exist, continue !"
--- docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_tvf/run.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/data/tvf/test_tvf/run.sh
@@ -6,9 +6,9 @@
 
 if [[ ! -d "${CUR_DIR}/tvf" ]]; then
     echo "${CUR_DIR}/tvf does not exist"
-    cd ${CUR_DIR}/ && rm -f data.tar.gz \
-    && curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/test_tvf/data.tar.gz \
-    && tar xzf data.tar.gz
+    cd ${CUR_DIR}/ && rm -f data.tar.gz &&
+        curl -O https://s3BucketName.s3Endpoint/regression/datalake/pipeline_data/test_tvf/data.tar.gz &&
+        tar xzf data.tar.gz
     cd -
 else
     echo "${CUR_DIR}/tvf exist, continue !"
--- docker/thirdparties/docker-compose/hive/scripts/hive-metastore.sh.orig
+++ docker/thirdparties/docker-compose/hive/scripts/hive-metastore.sh
@@ -97,7 +97,6 @@
 hadoop_put_pids=()
 hadoop fs -mkdir -p /user/doris/
 
-
 ## put tpch1
 if [[ -z "$(ls /mnt/scripts/tpch1.db)" ]]; then
     echo "tpch1.db does not exist"
@@ -126,7 +125,6 @@
 hadoop fs -copyFromLocal -f /mnt/scripts/preinstalled_data /user/doris/ &
 hadoop_put_pids+=($!)
 
-
 # wait put finish
 set +e
 wait "${hadoop_put_pids[@]}"
--- docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh.orig
+++ docker/thirdparties/docker-compose/hudi/scripts/run_sync_tool.sh
@@ -18,36 +18,36 @@
 # under the License.
 
 function error_exit {
-    echo "$1" >&2   ## Send message to stderr. Exclude >&2 if you don't want it that way.
-    exit "${2:-1}"  ## Return a code specified by $2 or 1 by default.
+    echo "$1" >&2  ## Send message to stderr. Exclude >&2 if you don't want it that way.
+    exit "${2:-1}" ## Return a code specified by $2 or 1 by default.
 }
 
 if [ -z "${HADOOP_HOME}" ]; then
-  error_exit "Please make sure the environment variable HADOOP_HOME is setup"
+    error_exit "Please make sure the environment variable HADOOP_HOME is setup"
 fi
 
 if [ -z "${HIVE_HOME}" ]; then
-  error_exit "Please make sure the environment variable HIVE_HOME is setup"
+    error_exit "Please make sure the environment variable HIVE_HOME is setup"
 fi
 
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 #Ensure we pick the right jar even for hive11 builds
-HUDI_HIVE_UBER_JAR=`ls -c $DIR/./hudi_docker_compose_attached_file/jar/hoodie-hive-sync-bundle.jar | grep -v source | head -1`
+HUDI_HIVE_UBER_JAR=$(ls -c $DIR/./hudi_docker_compose_attached_file/jar/hoodie-hive-sync-bundle.jar | grep -v source | head -1)
 
 if [ -z "$HADOOP_CONF_DIR" ]; then
-  echo "setting hadoop conf dir"
-  HADOOP_CONF_DIR="${HADOOP_HOME}/etc/hadoop"
+    echo "setting hadoop conf dir"
+    HADOOP_CONF_DIR="${HADOOP_HOME}/etc/hadoop"
 fi
 
 ## Include only specific packages from HIVE_HOME/lib to avoid version mismatches
-HIVE_EXEC=`ls ${HIVE_HOME}/lib/hive-exec-*.jar | tr '\n' ':'`
-HIVE_SERVICE=`ls ${HIVE_HOME}/lib/hive-service-*.jar | grep -v rpc | tr '\n' ':'`
-HIVE_METASTORE=`ls ${HIVE_HOME}/lib/hive-metastore-*.jar | tr '\n' ':'`
-HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | tr '\n' ':'`
+HIVE_EXEC=$(ls ${HIVE_HOME}/lib/hive-exec-*.jar | tr '\n' ':')
+HIVE_SERVICE=$(ls ${HIVE_HOME}/lib/hive-service-*.jar | grep -v rpc | tr '\n' ':')
+HIVE_METASTORE=$(ls ${HIVE_HOME}/lib/hive-metastore-*.jar | tr '\n' ':')
+HIVE_JDBC=$(ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | tr '\n' ':')
 if [ -z "${HIVE_JDBC}" ]; then
-  HIVE_JDBC=`ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | grep -v handler | tr '\n' ':'`
+    HIVE_JDBC=$(ls ${HIVE_HOME}/lib/hive-jdbc-*.jar | grep -v handler | tr '\n' ':')
 fi
-HIVE_JACKSON=`ls ${HIVE_HOME}/lib/jackson-*.jar | tr '\n' ':'`
+HIVE_JACKSON=$(ls ${HIVE_HOME}/lib/jackson-*.jar | tr '\n' ':')
 HIVE_JARS=$HIVE_METASTORE:$HIVE_SERVICE:$HIVE_EXEC:$HIVE_JDBC:$HIVE_JACKSON
 
 HADOOP_HIVE_JARS=${HIVE_JARS}:${HADOOP_HOME}/share/hadoop/common/*:${HADOOP_HOME}/share/hadoop/mapreduce/*:${HADOOP_HOME}/share/hadoop/hdfs/*:${HADOOP_HOME}/share/hadoop/common/lib/*:${HADOOP_HOME}/share/hadoop/hdfs/lib/*
--- docker/thirdparties/docker-compose/hudi/scripts/setup_demo_container_adhoc_2.sh.orig
+++ docker/thirdparties/docker-compose/hudi/scripts/setup_demo_container_adhoc_2.sh
@@ -36,42 +36,42 @@
 
 echo "Start synchronizing the stock_ticks_cow table"
 /var/scripts/run_sync_tool.sh \
-  --jdbc-url jdbc:hive2://hiveserver:10000 \
-  --user hive \
-  --pass hive \
-  --partitioned-by date \
-  --base-path /user/hive/warehouse/stock_ticks_cow \
-  --database default \
-  --table stock_ticks_cow \
-  --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
+    --jdbc-url jdbc:hive2://hiveserver:10000 \
+    --user hive \
+    --pass hive \
+    --partitioned-by date \
+    --base-path /user/hive/warehouse/stock_ticks_cow \
+    --database default \
+    --table stock_ticks_cow \
+    --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
 
 echo "Start synchronizing the stock_ticks_mor table"
 /var/scripts/run_sync_tool.sh \
-  --jdbc-url jdbc:hive2://hiveserver:10000 \
-  --user hive \
-  --pass hive \
-  --partitioned-by date \
-  --base-path /user/hive/warehouse/stock_ticks_mor \
-  --database default \
-  --table stock_ticks_mor \
-  --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
+    --jdbc-url jdbc:hive2://hiveserver:10000 \
+    --user hive \
+    --pass hive \
+    --partitioned-by date \
+    --base-path /user/hive/warehouse/stock_ticks_mor \
+    --database default \
+    --table stock_ticks_mor \
+    --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
 
 echo "Start synchronizing the hudi_cow_pt_tbl table"
 /var/scripts/run_sync_tool.sh \
-  --jdbc-url jdbc:hive2://hiveserver:10000 \
-  --user hive \
-  --pass hive \
-  --partitioned-by dt \
-  --base-path /user/hive/warehouse/hudi_cow_pt_tbl \
-  --database default \
-  --table hudi_cow_pt_tbl \
-  --partition-value-extractor org.apache.hudi.hive.HiveStylePartitionValueExtractor
+    --jdbc-url jdbc:hive2://hiveserver:10000 \
+    --user hive \
+    --pass hive \
+    --partitioned-by dt \
+    --base-path /user/hive/warehouse/hudi_cow_pt_tbl \
+    --database default \
+    --table hudi_cow_pt_tbl \
+    --partition-value-extractor org.apache.hudi.hive.HiveStylePartitionValueExtractor
 
 echo "Start synchronizing the hudi_non_part_cow table"
 /var/scripts/run_sync_tool.sh \
-  --jdbc-url jdbc:hive2://hiveserver:10000 \
-  --user hive \
-  --pass hive \
-  --base-path /user/hive/warehouse/hudi_non_part_cow \
-  --database default \
-  --table hudi_non_part_cow \
+    --jdbc-url jdbc:hive2://hiveserver:10000 \
+    --user hive \
+    --pass hive \
+    --base-path /user/hive/warehouse/hudi_non_part_cow \
+    --database default \
+    --table hudi_non_part_cow
--- docker/thirdparties/docker-compose/iceberg/tools/save_docker.sh.orig
+++ docker/thirdparties/docker-compose/iceberg/tools/save_docker.sh
--- docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh.orig
+++ docker/thirdparties/docker-compose/kerberos/entrypoint-hive-master.sh
@@ -69,6 +69,6 @@
     echo "Invalid index parameter. Exiting."
     exit 1
 fi
-hive  -f /usr/local/sql/create_kerberos_hive_table.sql
+hive -f /usr/local/sql/create_kerberos_hive_table.sql
 
 exec_success_hook
--- docker/thirdparties/docker-compose/kerberos/health-checks/health.sh.orig
+++ docker/thirdparties/docker-compose/kerberos/health-checks/health.sh
@@ -29,6 +29,6 @@
 
 if test -d "${HEALTH_D}"; then
     for health_script in "${HEALTH_D}"/*; do
-        "${health_script}" &>> /var/log/container-health.log || exit 1
+        "${health_script}" &>>/var/log/container-health.log || exit 1
     done
 fi
--- docker/thirdparties/docker-compose/kerberos/health-checks/hive-health-check-2.sh.orig
+++ docker/thirdparties/docker-compose/kerberos/health-checks/hive-health-check-2.sh
--- docker/thirdparties/docker-compose/kerberos/health-checks/hive-health-check.sh.orig
+++ docker/thirdparties/docker-compose/kerberos/health-checks/hive-health-check.sh
--- docker/thirdparties/docker-compose/kerberos/health-checks/supervisorctl-check.sh.orig
+++ docker/thirdparties/docker-compose/kerberos/health-checks/supervisorctl-check.sh
@@ -32,9 +32,9 @@
 FAILED=$(supervisorctl status | grep -v RUNNING || true)
 
 if [ "$FAILED" == "" ]; then
-  echo "All services are running"
-  exit 0
+    echo "All services are running"
+    exit 0
 else
-  echo "Some of the services are failing: ${FAILED}"
-  exit 1
+    echo "Some of the services are failing: ${FAILED}"
+    exit 1
 fi
--- docker/thirdparties/docker-compose/ranger/script/install_doris_ranger_plugins.sh.orig
+++ docker/thirdparties/docker-compose/ranger/script/install_doris_ranger_plugins.sh
--- docker/thirdparties/docker-compose/ranger/script/install_doris_service_def.sh.orig
+++ docker/thirdparties/docker-compose/ranger/script/install_doris_service_def.sh
--- docker/thirdparties/run-thirdparties-docker.sh.orig
+++ docker/thirdparties/run-thirdparties-docker.sh
@@ -187,7 +187,7 @@
         RUN_MARIADB=1
     elif [[ "${element}"x == "db2"x ]]; then
         RUN_DB2=1
-    elif [[ "${element}"x == "oceanbase"x ]];then
+    elif [[ "${element}"x == "oceanbase"x ]]; then
         RUN_OCEANBASE=1
     elif [[ "${element}"x == "lakesoul"x ]]; then
         RUN_LAKESOUL=1
@@ -387,7 +387,7 @@
     . "${ROOT}"/docker-compose/hive/hive-2x_settings.env
     envsubst <"${ROOT}"/docker-compose/hive/hive-2x.yaml.tpl >"${ROOT}"/docker-compose/hive/hive-2x.yaml
     envsubst <"${ROOT}"/docker-compose/hive/hadoop-hive.env.tpl >"${ROOT}"/docker-compose/hive/hadoop-hive-2x.env
-    envsubst <"${ROOT}"/docker-compose/hive/hadoop-hive-2x.env.tpl >> "${ROOT}"/docker-compose/hive/hadoop-hive-2x.env
+    envsubst <"${ROOT}"/docker-compose/hive/hadoop-hive-2x.env.tpl >>"${ROOT}"/docker-compose/hive/hadoop-hive-2x.env
     sudo docker compose -p ${CONTAINER_UID}hive2 -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-2x.env down
     if [[ "${STOP}" -ne 1 ]]; then
         sudo docker compose -p ${CONTAINER_UID}hive2 -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-2x.env up --build --remove-orphans -d --wait
@@ -426,7 +426,7 @@
     . "${ROOT}"/docker-compose/hive/hive-3x_settings.env
     envsubst <"${ROOT}"/docker-compose/hive/hive-3x.yaml.tpl >"${ROOT}"/docker-compose/hive/hive-3x.yaml
     envsubst <"${ROOT}"/docker-compose/hive/hadoop-hive.env.tpl >"${ROOT}"/docker-compose/hive/hadoop-hive-3x.env
-    envsubst <"${ROOT}"/docker-compose/hive/hadoop-hive-3x.env.tpl >> "${ROOT}"/docker-compose/hive/hadoop-hive-3x.env
+    envsubst <"${ROOT}"/docker-compose/hive/hadoop-hive-3x.env.tpl >>"${ROOT}"/docker-compose/hive/hadoop-hive-3x.env
     sudo docker compose -p ${CONTAINER_UID}hive3 -f "${ROOT}"/docker-compose/hive/hive-3x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-3x.env down
     if [[ "${STOP}" -ne 1 ]]; then
         sudo docker compose -p ${CONTAINER_UID}hive3 -f "${ROOT}"/docker-compose/hive/hive-3x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive-3x.env up --build --remove-orphans -d --wait
@@ -452,12 +452,12 @@
     if [[ "${STOP}" -ne 1 ]]; then
         if [[ ! -d "${ICEBERG_DIR}/data" ]]; then
             echo "${ICEBERG_DIR}/data does not exist"
-            cd "${ICEBERG_DIR}" \
-            && rm -f iceberg_data.zip \
-            && wget -P "${ROOT}"/docker-compose/iceberg https://"${s3BucketName}.${s3Endpoint}"/regression/datalake/pipeline_data/iceberg_data.zip \
-            && sudo unzip iceberg_data.zip \
-            && sudo mv iceberg_data data \
-            && sudo rm -rf iceberg_data.zip
+            cd "${ICEBERG_DIR}" &&
+                rm -f iceberg_data.zip &&
+                wget -P "${ROOT}"/docker-compose/iceberg https://"${s3BucketName}.${s3Endpoint}"/regression/datalake/pipeline_data/iceberg_data.zip &&
+                sudo unzip iceberg_data.zip &&
+                sudo mv iceberg_data data &&
+                sudo rm -rf iceberg_data.zip
             cd -
         else
             echo "${ICEBERG_DIR}/data exist, continue !"
@@ -619,9 +619,9 @@
     for i in {1..2}; do
         . "${ROOT}"/docker-compose/kerberos/kerberos${i}_settings.env
         envsubst <"${ROOT}"/docker-compose/kerberos/hadoop-hive.env.tpl >"${ROOT}"/docker-compose/kerberos/hadoop-hive-${i}.env
-        envsubst <"${ROOT}"/docker-compose/kerberos/conf/my.cnf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/my.cnf
-        envsubst <"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/kdc.conf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/kdc.conf
-        envsubst <"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/krb5.conf.tpl > "${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/krb5.conf
+        envsubst <"${ROOT}"/docker-compose/kerberos/conf/my.cnf.tpl >"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/my.cnf
+        envsubst <"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/kdc.conf.tpl >"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/kdc.conf
+        envsubst <"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/krb5.conf.tpl >"${ROOT}"/docker-compose/kerberos/conf/kerberos${i}/krb5.conf
     done
     sudo chmod a+w /etc/hosts
     sudo sed -i "1i${IP_HOST} hadoop-master" /etc/hosts
@@ -672,102 +672,102 @@
 declare -A pids
 
 if [[ "${RUN_ES}" -eq 1 ]]; then
-    start_es > start_es.log  2>&1 &
+    start_es >start_es.log 2>&1 &
     pids["es"]=$!
 fi
 
 if [[ "${RUN_MYSQL}" -eq 1 ]]; then
-    start_mysql > start_mysql.log 2>&1 &
+    start_mysql >start_mysql.log 2>&1 &
     pids["mysql"]=$!
 fi
 
 if [[ "${RUN_PG}" -eq 1 ]]; then
-    start_pg > start_pg.log 2>&1 &
+    start_pg >start_pg.log 2>&1 &
     pids["pg"]=$!
 fi
 
 if [[ "${RUN_ORACLE}" -eq 1 ]]; then
-    start_oracle > start_oracle.log 2>&1 &
+    start_oracle >start_oracle.log 2>&1 &
     pids["oracle"]=$!
 fi
 
 if [[ "${RUN_DB2}" -eq 1 ]]; then
-    start_db2 > start_db2.log 2>&1 &
+    start_db2 >start_db2.log 2>&1 &
     pids["db2"]=$!
 fi
 
 if [[ "${RUN_OCEANBASE}" -eq 1 ]]; then
-    start_oceanbase > start_oceanbase.log 2>&1 &
+    start_oceanbase >start_oceanbase.log 2>&1 &
     pids["oceanbase"]=$!
 fi
 
 if [[ "${RUN_SQLSERVER}" -eq 1 ]]; then
-    start_sqlserver > start_sqlserver.log 2>&1 &
+    start_sqlserver >start_sqlserver.log 2>&1 &
     pids["sqlserver"]=$!
 fi
 
 if [[ "${RUN_CLICKHOUSE}" -eq 1 ]]; then
-    start_clickhouse > start_clickhouse.log 2>&1 &
+    start_clickhouse >start_clickhouse.log 2>&1 &
     pids["clickhouse"]=$!
 fi
 
 if [[ "${RUN_KAFKA}" -eq 1 ]]; then
-    start_kafka > start_kafka.log 2>&1 &
+    start_kafka >start_kafka.log 2>&1 &
     pids["kafka"]=$!
 fi
 
 if [[ "${RUN_HIVE2}" -eq 1 ]]; then
-    start_hive2 > start_hive2.log 2>&1 &
+    start_hive2 >start_hive2.log 2>&1 &
     pids["hive2"]=$!
 fi
 
 if [[ "${RUN_HIVE3}" -eq 1 ]]; then
-    start_hive3 > start_hive3.log 2>&1 &
+    start_hive3 >start_hive3.log 2>&1 &
     pids["hive3"]=$!
 fi
 
 if [[ "${RUN_SPARK}" -eq 1 ]]; then
-    start_spark > start_spark.log 2>&1 &
+    start_spark >start_spark.log 2>&1 &
     pids["spark"]=$!
 fi
 
 if [[ "${RUN_ICEBERG}" -eq 1 ]]; then
-    start_iceberg > start_icerberg.log 2>&1 &
+    start_iceberg >start_icerberg.log 2>&1 &
     pids["iceberg"]=$!
 fi
 
 if [[ "${RUN_HUDI}" -eq 1 ]]; then
-    start_hudi > start_hudi.log 2>&1 &
+    start_hudi >start_hudi.log 2>&1 &
     pids["hudi"]=$!
 fi
 
 if [[ "${RUN_TRINO}" -eq 1 ]]; then
-    start_trino > start_trino.log 2>&1 &
+    start_trino >start_trino.log 2>&1 &
     pids["trino"]=$!
 fi
 
 if [[ "${RUN_MARIADB}" -eq 1 ]]; then
-    start_mariadb > start_mariadb.log 2>&1 &
+    start_mariadb >start_mariadb.log 2>&1 &
     pids["mariadb"]=$!
 fi
 
 if [[ "${RUN_LAKESOUL}" -eq 1 ]]; then
-    start_lakesoul > start_lakesoule.log 2>&1 &
+    start_lakesoul >start_lakesoule.log 2>&1 &
     pids["lakesoul"]=$!
 fi
 
 if [[ "${RUN_MINIO}" -eq 1 ]]; then
-    start_minio > start_minio.log 2>&1 &
+    start_minio >start_minio.log 2>&1 &
     pids["minio"]=$!
 fi
 
 if [[ "${RUN_KERBEROS}" -eq 1 ]]; then
-    start_kerberos > start_kerberos.log 2>&1 &
+    start_kerberos >start_kerberos.log 2>&1 &
     pids["kerberos"]=$!
 fi
 
 if [[ "${RUN_RANGER}" -eq 1 ]]; then
-    start_ranger > start_ranger.log 2>&1 &
+    start_ranger >start_ranger.log 2>&1 &
     pids["ranger"]=$!
 fi
 
--- thirdparty/download-thirdparty.sh.orig
+++ thirdparty/download-thirdparty.sh
@@ -603,5 +603,4 @@
     echo "Finished patching ${FAISS_SOURCE}"
 fi
 
-
 # vim: ts=4 sw=4 ts=4 tw=100:
--- thirdparty/vars.sh.orig
+++ thirdparty/vars.sh
@@ -550,7 +550,6 @@
 FAISS_SOURCE="faiss-1.10.0"
 FAISS_MD5SUM="f31edf2492808b27cc963d0ab316a205"
 
-
 # all thirdparties which need to be downloaded is set in array TP_ARCHIVES
 export TP_ARCHIVES=(
     'LIBEVENT'
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

kaka11chen pushed a commit that referenced this pull request Apr 19, 2025
@jacktengg
[fix](coalesce) fix 'heap-use-after-free' of function coalesce (apach…
12ea20a 
…e#42666) (apache#42668)

## Proposed changes
BP apache#42666

Should not modify state of vexpr when executing the expr, since it may
be executed in parallel.

Issue introduced by apache#42136

```
==75397==ERROR: AddressSanitizer: heap-use-after-free on address 0x60700294b060 at pc 0x56507fb8a370 bp 0x7fc861cd7050 sp 0x7fc861cd7048
19:58:49   READ of size 8 at 0x60700294b060 thread T1463 (Pipe_normal [wo)
19:58:49       #0 0x56507fb8a36f in doris::vectorized::FunctionCoalesce::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function_coalesce.cpp:163:13
19:58:49       #1 0x56507958edfe in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:463:26
19:58:49       #2 0x56507dbcbf5c in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp
19:58:49       #3 0x56507dbc65e3 in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:245:12
19:58:49       apache#4 0x56507dbc683b in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:251:12
19:58:49       apache#5 0x56507958c81c in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:194:19
19:58:49       apache#6 0x565079578bee in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector<unsigned long, std::allocator<unsigned long> >&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:188:5
19:58:49       apache#7 0x56507957a3e4 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:203:12
19:58:49       apache#8 0x56507964994a in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:55:5
19:58:49       apache#9 0x56508f8d7496 in doris::pipeline::OperatorXBase::do_projections(doris::RuntimeState*, doris::vectorized::Block*, doris::vectorized::Block*) const /root/doris/be/src/pipeline/exec/operator.cpp:294:13
19:58:49       apache#10 0x56508f8d96ba in doris::pipeline::OperatorXBase::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/pipeline/exec/operator.cpp:335:18
19:58:49       apache#11 0x565093c2560b in doris::pipeline::PipelineTask::execute(bool*) /root/doris/be/src/pipeline/pipeline_task.cpp:374:13
19:58:49       apache#12 0x565093c6a164 in doris::pipeline::TaskScheduler::_do_work(int) /root/doris/be/src/pipeline/task_scheduler.cpp:138:9
19:58:49       apache#13 0x565062529d1d in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:543:24
19:58:49       apache#14 0x5650625021ce in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
19:58:49       apache#15 0x5650625021ce in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
19:58:49       apache#16 0x7fca1844a608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
19:58:49       apache#17 0x7fca186f7132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
19:58:49   
19:58:49   0x60700294b060 is located 16 bytes inside of 80-byte region [0x60700294b050,0x60700294b0a0)
19:58:49   freed by thread T1462 (Pipe_normal [wo) here:
19:58:49       #0 0x56505dd2f80d in operator delete(void*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x2f79e80d) (BuildId: 60cc2a41212f4e75)
19:58:49       #1 0x565079596a6f in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11
19:58:49       #2 0x565079596a6f in std::__shared_ptr<doris::vectorized::IFunctionBase, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31
19:58:49       #3 0x565079596a6f in std::__shared_ptr<doris::vectorized::IFunctionBase, (__gnu_cxx::_Lock_policy)2>::operator=(std::__shared_ptr<doris::vectorized::IFunctionBase, (__gnu_cxx::_Lock_policy)2>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1245:2
19:58:49       apache#4 0x56507fb89acd in std::shared_ptr<doris::vectorized::IFunctionBase>::operator=(std::shared_ptr<doris::vectorized::IFunctionBase>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:385:27
19:58:49       apache#5 0x56507fb89acd in doris::vectorized::FunctionCoalesce::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function_coalesce.cpp:89:30
19:58:49       apache#6 0x56507958edfe in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const /root/doris/be/src/vec/functions/function.h:463:26
19:58:49       apache#7 0x56507dbcbf5c in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp
19:58:49       apache#8 0x56507dbc65e3 in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:245:12
19:58:49       apache#9 0x56507dbc683b in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:251:12
19:58:49       apache#10 0x56507958c81c in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:194:19
19:58:49       apache#11 0x565079578bee in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector<unsigned long, std::allocator<unsigned long> >&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:188:5
19:58:49       apache#12 0x56507957a3e4 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:203:12
19:58:49       apache#13 0x56507964994a in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:55:5
19:58:49       apache#14 0x56508f8d7496 in doris::pipeline::OperatorXBase::do_projections(doris::RuntimeState*, doris::vectorized::Block*, doris::vectorized::Block*) const /root/doris/be/src/pipeline/exec/operator.cpp:294:13
19:58:49       apache#15 0x56508f8d96ba in doris::pipeline::OperatorXBase::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/pipeline/exec/operator.cpp:335:18
19:58:49       apache#16 0x565093c2560b in doris::pipeline::PipelineTask::execute(bool*) /root/doris/be/src/pipeline/pipeline_task.cpp:374:13
```
kaka11chen pushed a commit that referenced this pull request Apr 19, 2025
@Mryange
[fix](beut) fix core in exchange sink test (apache#47926)
68662f9 
### What problem does this PR solve?

In BE_TEST, a query context is not initialized, resulting in a core
dump.
```
==1732796==ERROR: AddressSanitizer: SEGV on unknown address 0x0000000001e0 (pc 0x55c664a74e74 bp 0x7ffcc76e9d20 sp 0x7ffcc76e9cf0 T0)
==1732796==The signal is caused by a READ memory access.
==1732796==Hint: address points to the zero page.
    #0 0x55c664a74e74 in std::__shared_ptr<doris::ResourceContext, (__gnu_cxx::_Lock_policy)2>::__shared_ptr(std::__shared_ptr<doris::ResourceContext, (__gnu_cxx::_Lock_policy)2> const&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:1522:7
    #1 0x55c664a74e0e in std::shared_ptr<doris::ResourceContext>::shared_ptr(std::shared_ptr<doris::ResourceContext> const&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr.h:204:7
    #2 0x55c66878ff18 in doris::QueryContext::resource_ctx() /mnt/disk12/yanxuecheng/doris/be/src/runtime/query_context.h:292:62
    #3 0x55c69a2f2c19 in doris::vectorized::BlockSerializer::serialize_block(doris::vectorized::Block const*,
```

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [x] Confirm the release note
- [x] Confirm test cases
- [x] Confirm document
- [x] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
kaka11chen pushed a commit that referenced this pull request Apr 19, 2025
@xinyiZzz
[fix](resource) Fix register ResourceContext to RuntimeQueryStatistic…
e7c966c 
…sMgr (apache#48004)

### What problem does this PR solve?

Wait for QueryTaskController to be initialized before register
ResourceContext

fix
```
/root/doris/be/src/runtime/workload_management/task_controller.h:59:44: runtime error: load of value 3200171710, which is not a valid value for type 'TQueryType::type'
    #0 0x555ff9336162 in doris::TaskController::query_type() /root/doris/be/src/runtime/workload_management/task_controller.h:59:44
    #1 0x555ff9336162 in doris::RuntimeQueryStatisticsMgr::report_runtime_query_statistics() /root/doris/be/src/runtime/runtime_query_statistics_mgr.cpp:346:50
    #2 0x555ff50d789d in doris::Daemon::report_runtime_query_statistics_thread() /root/doris/be/src/common/daemon.cpp:487:65
    #3 0x555ff9a3efae in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
    apache#4 0x555ff9a3efae in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
    apache#5 0x7f4935d00608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
    apache#6 0x7f4935fad132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
```
kaka11chen pushed a commit that referenced this pull request Apr 19, 2025
@Gabriel39
[fix](scheduler) Fix invalid access after freed (apache#48168)
b7c973e 
WRITE of size 1 at 0x6160007e86f0 thread T1983 (Pipe_normal [wo)
#0 0x55fc8065b975 in std::__atomic_base<bool>::store(bool,
std::memory_order)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:457:2
#1 0x55fc8065b975 in std::__atomic_base<bool>::operator=(bool)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:349:2
#2 0x55fc8065b975 in std::atomic<bool>::operator=(bool)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/atomic:80:22
#3 0x55fc8065b975 in doris::pipeline::PipelineTask::set_running(bool)
/root/doris/be/src/pipeline/pipeline_task.h:192:47
apache#4 0x55fc8065b975 in
doris::pipeline::TaskScheduler::_do_work(int)::$_0::operator()() const
/root/doris/be/src/pipeline/task_scheduler.cpp:121:23
apache#5 0x55fc8065b975 in
doris::Defer<doris::pipeline::TaskScheduler::_do_work(int)::$_0>::~Defer()
/root/doris/be/src/util/defer_op.h:37:16
apache#6 0x55fc8065b975 in doris::pipeline::TaskScheduler::_do_work(int)
/root/doris/be/src/pipeline/task_scheduler.cpp:162:5
apache#7 0x55fc4c57cd19 in doris::ThreadPool::dispatch_thread()
/root/doris/be/src/util/threadpool.cpp:608:24
apache#8 0x55fc4c55395e in std::function<void ()>::operator()() const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
apache#9 0x55fc4c55395e in doris::Thread::supervise_thread(void*)
/root/doris/be/src/util/thread.cpp:498:5
apache#10 0x7f9ee3d25608 in start_thread
/build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
apache#11 0x7f9ee3fd2132 in __clone
/build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

0x6160007e86f0 is located 624 bytes inside of 632-byte region
[0x6160007e8480,0x6160007e86f8)
freed by thread T1981 (Pipe_normal [wo) here:
#0 0x55fc47aa680d in operator delete(void*)
(/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x3376e80d)
(BuildId: 865149e62959581e)
#1 0x55fc8059db84 in
std::default_delete<doris::pipeline::PipelineTask>::operator()(doris::pipeline::PipelineTask*)
const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
#2 0x55fc8059db84 in std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >::~unique_ptr()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
#3 0x55fc8059db84 in void
std::destroy_at<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >
>(std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
apache#4 0x55fc8059db84 in void
std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >
>(std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7
apache#5 0x55fc8059db84 in void
std::_Destroy_aux<false>::__destroy<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask>
>*>(std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*,
std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6
apache#6 0x55fc8059db84 in void
std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask>
>*>(std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*,
std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7
apache#7 0x55fc8059db84 in void
std::_Destroy<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*,
std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >
>(std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*,
std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >*,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > >&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7
apache#8 0x55fc8059db84 in
std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >::~vector()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2
apache#9 0x55fc8052571c in void
std::destroy_at<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >
>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
apache#10 0x55fc8052571c in void
std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >
>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7
apache#11 0x55fc8052571c in void
std::_Destroy_aux<false>::__destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > >
>*>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*,
std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6
apache#12 0x55fc8052571c in void
std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > >
>*>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*,
std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7
apache#13 0x55fc8052571c in void
std::_Destroy<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*,
std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >
>(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*,
std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*,
std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > > >&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7
apache#14 0x55fc8052571c in
std::vector<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >,
std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > > >
>::_M_erase_at_end(std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >*)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1796:6
apache#15 0x55fc8052571c in
std::vector<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > >,
std::allocator<std::vector<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> >,
std::allocator<std::unique_ptr<doris::pipeline::PipelineTask,
std::default_delete<doris::pipeline::PipelineTask> > > > > >::clear()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1499:9
apache#16 0x55fc8052571c in
doris::pipeline::PipelineFragmentContext::~PipelineFragmentContext()
/root/doris/be/src/pipeline/pipeline_fragment_context.cpp:142:12
apache#17 0x55fc47ad30cc in
std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6
apache#18 0x55fc80658d57 in
std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11
apache#19 0x55fc80658d57 in std::__shared_ptr<doris::TaskExecutionContext,
(__gnu_cxx::_Lock_policy)2>::~__shared_ptr()
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31
apache#20 0x55fc80658d57 in
doris::pipeline::close_task(doris::pipeline::PipelineTask*,
doris::Status) /root/doris/be/src/pipeline/task_scheduler.cpp:100:1
apache#21 0x55fc8065aa17 in doris::pipeline::TaskScheduler::_do_work(int)
/root/doris/be/src/pipeline/task_scheduler.cpp:160:36
apache#22 0x55fc4c57cd19 in doris::ThreadPool::dispatch_thread()
/root/doris/be/src/util/threadpool.cpp:608:24
apache#23 0x55fc4c55395e in std::function<void ()>::operator()() const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
apache#24 0x55fc4c55395e in doris::Thread::supervise_thread(void*)
/root/doris/be/src/util/thread.cpp:498:5
apache#25 0x7f9ee3d25608 in start_thread
/build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
kaka11chen pushed a commit that referenced this pull request Apr 19, 2025
@Gabriel39
[fix](schema scan) Fix invalid pointer access (apache#48313)
961a844 
Schema scanner runs on a separate thread which is executed
asynchronously. We should make sure all context used not be freed once
it is scheduled.

ERROR: AddressSanitizer: heap-buffer-overflow on address 0x613002f33eb2
at pc 0x55e085dccbe3 bp 0x7f345c0e1f10 sp 0x7f345c0e1f08
READ of size 1 at 0x613002f33eb2 thread T2776 (FragmentMgrAsyn)
#0 0x55e085dccbe2 in std::__atomic_base::load(std::memory_order) const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9
#1 0x55e085dccbe2 in std::atomic::operator bool() const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/atomic:87:22
#2 0x55e085dccbe2 in
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0::operator()()
const
/home/zcp/repo_center/doris_master/doris/be/src/exec/schema_scanner.cpp:118:5
#3 0x55e085dccbe2 in void std::__invoke_impl(std::__invoke_other,
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
apache#4 0x55e085dccbe2 in std::enable_if, void>::type
std::__invoke_r(doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
apache#5 0x55e085dccbe2 in std::_Function_handler::_M_invoke(std::_Any_data
const&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
apache#6 0x55e050f081ca in doris::ThreadPool::dispatch_thread()
/home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:608:24
apache#7 0x55e050ede467 in doris::Thread::supervise_thread(void*)
/home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
    apache#8 0x7f376ef5aac2 in start_thread nptl/pthread_create.c:442:8
apache#9 0x7f376efec84f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
kaka11chen pushed a commit that referenced this pull request Apr 19, 2025
@Gabriel39
[fix](schema scan) Fix invalid pointer access (apache#48370)
a3eb715 
==14365==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x613002873c70 at pc 0x5559a290ec05 bp 0x7f6c40385e90 sp 0x7f6c40385e88
READ of size 2 at 0x613002873c70 thread T4019 (FragmentMgrAsyn)
#0 0x5559a290ec04 in std::__atomic_base::load(std::memory_order) const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9
#1 0x5559a290ec04 in doris::AtomicStatus::update(doris::Status const&)
/home/zcp/repo_center/doris_master/doris/be/src/common/status.h:601:44
#2 0x5559d8f289c0 in
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0::operator()()
const
/home/zcp/repo_center/doris_master/doris/be/src/exec/schema_scanner.cpp:118:5
#3 0x5559d8f289c0 in void std::__invoke_impl(std::__invoke_other,
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
apache#4 0x5559d8f289c0 in std::enable_if, void>::type
std::__invoke_r(doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
apache#5 0x5559d8f289c0 in std::_Function_handler::_M_invoke(std::_Any_data
const&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
apache#6 0x5559a4140e4a in doris::ThreadPool::dispatch_thread()
/home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:608:24
apache#7 0x5559a4117877 in doris::Thread::supervise_thread(void*)
/home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
    apache#8 0x7f7912f84ac2 in start_thread nptl/pthread_create.c:442:8
apache#9 0x7f791301684f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
kaka11chen pushed a commit that referenced this pull request Apr 19, 2025
@hust-hhb
[fix](group commit)Fix ut WalManagerTest (apache#49075)
4b931c0 
Fix problem:
01:48:12  ==4739==ERROR: AddressSanitizer: heap-use-after-free on
address 0x6170022e9b90 at pc 0x556df319882f bp 0x7f3c7be04200 sp
0x7f3c7be041f8
01:48:12  READ of size 1 at 0x6170022e9b90 thread T2822
(update_wal_dir_)
01:48:31  #0 0x556df319882e in
std::__atomic_base<bool>::load(std::memory_order) const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9
01:48:31  #1 0x556df319882e in
std::atomic<bool>::load(std::memory_order) const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/atomic:112:22
01:48:31  #2 0x556dfdace32a in
doris::WalManager::_update_wal_dir_info_thread()
/root/doris/be/src/olap/wal/wal_manager.cpp:477:19
01:48:31  #3 0x556dfdad57df in
doris::WalManager::_init_wal_dirs_info()::$_0::operator()() const
/root/doris/be/src/olap/wal/wal_manager.cpp:162:48


http://43.132.222.7:8111/buildConfiguration/Doris_DorisBeUt_BeUt/642336?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandPull+Request+Details=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&showLog=642336_18402_305.18541&logFilter=debug&logView=flowAware
### What problem does this PR solve?
kaka11chen pushed a commit that referenced this pull request Apr 19, 2025
amory
[fix](named_struct) fix named_struct signature which deduce wrong for…
292147c 
… nested decimal precision (apache#48964)

before this pr , we deduce the precision of the function named_struct
containing the decimal type is same with the precision of the decimal
type in select . In this case we can not make right named_struct for
multiple decimal value
like :
```
mysql> create table t01 (a decimal(6,3), d struct<col:bigint, col1:decimal(7,2)>) properties ("replication_num"="1");
Query OK, 0 rows affected (0.14 sec)

mysql> insert into t01 values (123.321, named_struct('col', 1, 'col1', 345.24));
Query OK, 1 row affected (0.26 sec)
{'label':'label_a2dc895ee7834701_841639ad6d93923f', 'status':'VISIBLE', 'txnId':'9010'}

mysql> select named_struct("col_11", a, "col_12", d) from t01;
+--------------------------------------------------------+
| named_struct("col_11", a, "col_12", d)                 |
+--------------------------------------------------------+
| {"col_11":123.321, "col_12":{"col":1, "col1":345.240}} |
+--------------------------------------------------------+
1 row in set (0.17 sec)

mysql> insert into t01 values (123.321, named_struct('col', 1, 'col1', 12345.24));
Query OK, 1 row affected (0.17 sec)
{'label':'label_77ed6047f6a6494d_810c26becfd0535c', 'status':'VISIBLE', 'txnId':'9011'}
mysql> select named_struct("col_11", a, "col_12", d) from t01;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.6)[E-255]Arithmetic overflow when converting value 12345.24 from type Decimal(7, 2) to type Decimal(6, 3)
mysql> desc verbose select named_struct("col_11", a, "col_12", d) from t01;
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                                                                                              |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                                                                                                                                              |
|   OUTPUT EXPRS:                                                                                                                                                                              |
|     named_struct("col_11", a, "col_12", d)[#2]                                                                                                                                               |
|   PARTITION: RANDOM                                                                                                                                                                          |
|                                                                                                                                                                                              |
|   HAS_COLO_PLAN_NODE: false                                                                                                                                                                  |
|                                                                                                                                                                                              |
|   VRESULT SINK                                                                                                                                                                               |
|      MYSQL_PROTOCAL                                                                                                                                                                          |
|                                                                                                                                                                                              |
|   0:VOlapScanNode(50)                                                                                                                                                                        |
|      TABLE: tt.t01(t01), PREAGGREGATION: ON                                                                                                                                                  |
|      partitions=1/1 (t01)                                                                                                                                                                    |
|      tablets=10/10, tabletList=1741751856342,1741751856344,1741751856346 ...                                                                                                                 |
|      cardinality=2, avgRowSize=2955.0, numNodes=1                                                                                                                                            |
|      pushAggOp=NONE                                                                                                                                                                          |
|      final projections: named_struct('col_11', a[#0], 'col_12', CAST(d[#1] AS struct<col:bigint,col1:decimalv3(6,3)>))                                                                       |
|      final project output tuple id: 1                                                                                                                                                        |
|      tuple ids: 0                                                                                                                                                                            |
|                                                                                                                                                                                              |
| Tuples:                                                                                                                                                                                      |
| TupleDescriptor{id=0, tbl=t01}                                                                                                                                                               |
|   SlotDescriptor{id=0, col=a, colUniqueId=0, type=decimalv3(6,3), nullable=true, isAutoIncrement=false, subColPath=null}                                                                     |
|   SlotDescriptor{id=1, col=d, colUniqueId=1, type=struct<col:bigint,col1:decimalv3(7,2)>, nullable=true, isAutoIncrement=false, subColPath=null}                                             |
|                                                                                                                                                                                              |
| TupleDescriptor{id=1, tbl=t01}                                                                                                                                                               |
|   SlotDescriptor{id=2, col=null, colUniqueId=null, type=struct<col_11:decimalv3(6,3),col_12:struct<col:bigint,col1:decimalv3(6,3)>>, nullable=false, isAutoIncrement=false, subColPath=null} |
|                                                                                                                                                                                              |
|                                                                                                                                                                                              |
|
```
kaka11chen pushed a commit that referenced this pull request Apr 19, 2025
@mrhhsg
[fix](function) Undefined behavior in parse_url (apache#49149)
134e86e 
### What problem does this PR solve?

```
/root/doris/be/src/vec/common/pod_array.h:510:29: runtime error: null pointer passed as argument 2, which is declared to never be null
/root/ldb_toolchain/bin/../usr/include/string.h:43:28: note: nonnull attribute specified here
    #0 0x55cb2c2cea1e in void doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul>::insert_assume_reserved<char const*, char const*>(char const*, char const*) /root/doris/be/src/vec/common/pod_array.h:510:9
    #1 0x55cb2c2ce8a7 in void doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul>::insert<char const*, char const*>(char const*, char const*) /root/doris/be/src/vec/common/pod_array.h:472:9
    #2 0x55cb5a0b0d50 in doris::vectorized::StringOP::push_value_string(std::basic_string_view<char, std::char_traits<char>> const&, unsigned long, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul>&, doris::vectorized::PODArray<unsigned int, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul>&) /root/doris/be/src/vec/functions/function_string.h:128:15
    #3 0x55cb5d6843a2 in doris::Status doris::vectorized::FunctionStringParseUrl::vector_parse<false, true>(doris::vectorized::ColumnStr<unsigned int> const*, std::vector<doris::UrlParser::UrlPart, std::allocator<doris::UrlParser::UrlPart>>&, int, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul>&, doris::vectorized::PODArray<unsigned char, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul>&, doris::vectorized::PODArray<unsigned int, 4096ul, Allocator<false, false, false, DefaultMemoryAllocator>, 16ul, 15ul>&) /root/doris/be/src/vec/functions/function_string.h:2806:17
    apache#4 0x55cb5d683e9d in auto doris::vectorized::FunctionStringParseUrl::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const::'lambda'(auto, auto)::operator()<std::integral_constant<bool, false>, std::integral_constant<bool, true>>(auto, auto) const /root/doris/be/src/vec/functions/function_string.h:2783:13
    apache#5 0x55cb5d683c39 in auto std::__invoke_impl<doris::Status, doris::vectorized::FunctionStringParseUrl::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const::'lambda'(auto, auto), std::integral_constant<bool, false>, std::integral_constant<bool, true>>(std::__invoke_other, auto&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, true>&&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    apache#6 0x55cb5d683af5 in std::__invoke_result<auto, std::integral_constant<bool, false>, std::integral_constant<bool, true>>::type std::__invoke<doris::vectorized::FunctionStringParseUrl::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const::'lambda'(auto, auto), std::integral_constant<bool, false>, std::integral_constant<bool, true>>(auto&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, true>&&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14
    apache#7 0x55cb5d682dd3 in std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<doris::Status> (*)(doris::vectorized::FunctionStringParseUrl::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const::'lambda'(auto, auto)&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&)>, std::integer_sequence<unsigned long, 0ul, 1ul>>::__visit_invoke(doris::vectorized::FunctionStringParseUrl::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const::'lambda'(auto, auto)&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/variant:1032:11
    apache#8 0x55cb5d682ad5 in decltype(auto) std::__do_visit<std::__detail::__variant::__deduce_visit_result<doris::Status>, doris::vectorized::FunctionStringParseUrl::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const::'lambda'(auto, auto), std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>(auto&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/variant:1758:15
    apache#9 0x55cb5d67766b in std::invoke_result<auto, std::__conditional<is_lvalue_reference_v<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>>::type<std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&&>, std::__conditional<is_lvalue_reference_v<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>>::type<std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&&>>::type std::visit<doris::vectorized::FunctionStringParseUrl::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const::'lambda'(auto, auto), std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>(auto&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/variant:1859:9
    apache#10 0x55cb5d676604 in doris::vectorized::FunctionStringParseUrl::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const /root/doris/be/src/vec/functions/function_string.h:2783:13
    apache#11 0x55cb51a43fd4 in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const /root/doris/be/src/vec/functions/function.h:434:26
    apache#12 0x55cb583f7dff in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:119:16
    apache#13 0x55cb583e6de9 in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12
    apache#14 0x55cb583e5f53 in doris::vectorized::PreparedFunctionImpl::default_implementation_for_nulls(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool, bool*) const /root/doris/be/src/vec/functions/function.cpp:216:9
    apache#15 0x55cb583f7939 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:110:5
    apache#16 0x55cb583e6de9 in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:244:12
    apache#17 0x55cb583e7069 in doris::vectorized::PreparedFunctionImpl::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /root/doris/be/src/vec/functions/function.cpp:250:12
    apache#18 0x55cb51a3fd95 in doris::vectorized::IFunctionBase::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /root/doris/be/src/vec/functions/function.h:193:19
    apache#19 0x55cb51a2797f in doris::vectorized::VectorizedFnCall::_do_execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*, std::vector<unsigned int, std::allocator<unsigned int>>&) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:187:5
    apache#20 0x55cb51a28a77 in doris::vectorized::VectorizedFnCall::execute(doris::vectorized::VExprContext*, doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vectorized_fn_call.cpp:220:12
    apache#21 0x55cb51b5f3b4 in doris::vectorized::VExprContext::execute(doris::vectorized::Block*, int*) /root/doris/be/src/vec/exprs/vexpr_context.cpp:61:5
    apache#22 0x55cb519a6732 in doris::vectorized::Scanner::_do_projections(doris::vectorized::Block*, doris::vectorized::Block*) /root/doris/be/src/vec/exec/scan/scanner.cpp:200:9
    apache#23 0x55cb519a1b13 in doris::vectorized::Scanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/scanner.cpp:82:16
    apache#24 0x55cb5192a59d in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:241:5
    apache#25 0x55cb51931c38 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:148:21
    apache#26 0x55cb519314ef in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:147:31
    apache#27 0x55cb519312fe in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    apache#28 0x55cb5193123e in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2
    apache#29 0x55cb51930e45 in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9
    apache#30 0x55cb216f8e3f in std::function<void ()>::operator()() const /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
    apache#31 0x55cb51940ec6 in doris::vectorized::SimplifiedScanScheduler::submit_scan_task(doris::vectorized::SimplifiedScanTask)::'lambda'()::operator()() const /root/doris/be/src/vec/exec/scan/scanner_scheduler.h:149:65
    apache#32 0x55cb51940e7e in void std::__invoke_impl<void, doris::vectorized::SimplifiedScanScheduler::submit_scan_task(doris::vectorized::SimplifiedScanTask)::'lambda'()&>(std::__invoke_other, doris::vectorized::SimplifiedScanScheduler::submit_scan_task(doris::vectorized::SimplifiedScanTask)::'lambda'()&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    apache#33 0x55cb51940dbe in std::enable_if<is_invocable_r_v<void, doris::vectorized::SimplifiedScanScheduler::submit_scan_task(doris::vectorized::SimplifiedScanTask)::'lambda'()&>, void>::type std::__invoke_r<void, doris::vectorized::SimplifiedScanScheduler::submit_scan_task(doris::vectorized::SimplifiedScanTask)::'lambda'()&>(doris::vectorized::SimplifiedScanScheduler::submit_scan_task(doris::vectorized::SimplifiedScanTask)::'lambda'()&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2
    apache#34 0x55cb51940745 in std::_Function_handler<void (), doris::vectorized::SimplifiedScanScheduler::submit_scan_task(doris::vectorized::SimplifiedScanTask)::'lambda'()>::_M_invoke(std::_Any_data const&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9
    apache#35 0x55cb216f8e3f in std::function<void ()>::operator()() const /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
    apache#36 0x55cb28bd9844 in doris::FunctionRunnable::run() /root/doris/be/src/util/threadpool.cpp:64:27
    apache#37 0x55cb28bb84b9 in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:616:24
    apache#38 0x55cb28bfd263 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74:14
    apache#39 0x55cb28bfd068 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14
    apache#40 0x55cb28bfcfa0 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:506:11
    apache#41 0x55cb28bfcd95 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:591:17
    apache#42 0x55cb28bfcc8e in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    apache#43 0x55cb28bfcbce in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2
    apache#44 0x55cb28bfc665 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9
    apache#45 0x55cb216f8e3f in std::function<void ()>::operator()() const /root/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
    apache#46 0x55cb28b74241 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:498:5
    apache#47 0x55cb2142be0a in asan_thread_start(void*) crtstuff.c
    apache#48 0x7f17840221c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 7c4add5c7a885e6ff4ce17867d6a2286e4420eec)
    apache#49 0x7f1784a118d2 in clone (/lib64/libc.so.6+0x398d2) (BuildId: 4ee3325955e3b55b6805f33959b7cb77745ad625)
kaka11chen pushed a commit that referenced this pull request Apr 19, 2025
@Mryange
[fix](core) fix subreplace when inputting a large number of empty str…
7af13a4 
…ings (apache#49241)

### What problem does this PR solve?

```
    #0 0x55f5cd170097 in doris::validate_ascii_fast(char const*, unsigned long) /mnt/disk12/yanxuecheng/tmp-doris/be/src/util/simd/vstring_function.h:60:37
    #1 0x55f5cd16e3c0 in doris::simd::VStringFunctions::is_ascii(doris::StringRef const&) /mnt/disk12/yanxuecheng/tmp-doris/be/src/util/simd/vstring_function.h:212:16
    #2 0x55f5cd16dcd3 in auto doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto)::operator()<std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>>(auto, auto, auto, auto) const /mnt/disk12/yanxuecheng/tmp-doris/be/src/vec/functions/function_string.h:3600:25
    #3 0x55f5cd16d926 in auto std::__invoke_impl<void, doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto), std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>>(std::__invoke_other, auto&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    apache#4 0x55f5cd16d8b6 in std::__invoke_result<auto, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>>::type std::__invoke<doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto), std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>>(auto&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14
    apache#5 0x55f5cd16cfed in std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<void> (*)(doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto)&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&)>, std::integer_sequence<unsigned long, 0ul, 0ul, 0ul, 0ul>>::__visit_invoke(doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto)&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/variant:1032:11
    apache#6 0x55f5cd16ce46 in decltype(auto) std::__do_visit<std::__detail::__variant::__deduce_visit_result<void>, doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto), std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>(auto&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/variant:1758:15
    apache#7 0x55f5cd16c7ca in std::invoke_result<auto, std::__conditional<is_lvalue_reference_v<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>>::type<std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&&>, std::__conditional<is_lvalue_reference_v<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>>::type<std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&&>, std::__conditional<is_lvalue_reference_v<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>>::type<std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&&>, std::__conditional<is_lvalue_reference_v<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>>::type<std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&&>>::type std::visit<doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto), std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>(auto&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/variant:1859:9
    apache#8 0x55f5cd16bd75 in doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) /mnt/disk12/yanxuecheng/tmp-doris/be/src/vec/functions/function_string.h:3594:9
```

Related PR:(apache#40929)
kaka11chen pushed a commit that referenced this pull request Apr 19, 2025
amory
[fix](struct)Fixed the issue of inserting into a struct type string l…
edee8c1 
…iteral with one more subfield causing BE coredump (apache#49485)

Fixed the issue of inserting into a struct type string literal with one
more subfield causing BE coredump
some situation like this blow will make BE core
```
create table t(a int, b int, s struct<a:int>) PROPERTIES ("replication_allocation" = "tag.location.default: 1");
insert into t values(1,1,'{1,2}');
```
core info:
```
[WARNING!] /sys/kernel/mm/transparent_hugepage/enabled: [always] madvise never, Doris not recommend turning on THP, which may cause the BE process to use more memory and cannot be freed in time. Turn off THP: `echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled`
start BE in local mode
=================================================================
==2818976==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x603000f17a90 at pc 0x55eba7fdd69c bp 0x7f03243dbd30 sp 0x7f03243dbd28
READ of size 8 at 0x603000f17a90 thread T928 (brpc_light)
    #0 0x55eba7fdd69b in std::__shared_ptr<doris::vectorized::DataTypeSerDe, (__gnu_cxx::_Lock_policy)2>::get() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291:16
    #1 0x55eba7fdd649 in std::__shared_ptr_access<doris::vectorized::DataTypeSerDe, (__gnu_cxx::_Lock_policy)2, false, false>::_M_get() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:990:66
    #2 0x55eba7fda316 in std::__shared_ptr_access<doris::vectorized::DataTypeSerDe, (__gnu_cxx::_Lock_policy)2, false, false>::operator->() const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:984:9
    #3 0x55ebcfc8b19e in doris::vectorized::DataTypeStructSerDe::deserialize_one_cell_from_json(doris::vectorized::IColumn&, doris::Slice&, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_struct_serde.cpp:200:25
    apache#4 0x55ebdac49c8b in doris::vectorized::ConvertImplGenericFromString::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_cast.h:618:36
    apache#5 0x55ebda2234d0 in doris::Status std::__invoke_impl<doris::Status, doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>(std::__invoke_other, doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    apache#6 0x55ebda22323a in std::enable_if<is_invocable_r_v<doris::Status, doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>, doris::Status>::type std::__invoke_r<doris::Status, doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>(doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114:9
    apache#7 0x55ebda222e71 in std::_Function_handler<doris::Status (doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long), doris::Status (*)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)>::_M_invoke(std::_Any_data const&, doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
    apache#8 0x55ebd9b3f1d2 in std::function<doris::Status (doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long)>::operator()(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long) const /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
    apache#9 0x55ebda1ad7c7 in doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::operator()(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_cast.h:2241:17
    apache#10 0x55ebda1acee2 in doris::Status std::__invoke_impl<doris::Status, doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)&, doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>(std::__invoke_other, doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bo@@@
```
kaka11chen pushed a commit that referenced this pull request Apr 19, 2025
@Gabriel39
[fix](shutdown) Fix coredump due to shutdown in the wrong order (apac…
476f734 
…he#49700)

### What problem does this PR solve?

When a BE is down, global object should shut down following a specific
order to ensure BE will not crash. Because all running queries will
finish once fragment mgr finished, stream mgr must be released before it
because it may wake up some blocked tasks.

#0 __GI___pthread_sigmask (how=2, newmask=<optimized out>, oldmask=0x0)
at ./nptl/pthread_sigmask.c:43
#1 0x00007f842b2bb71e in PosixSignals::chained_handler(int, siginfo*,
void*) [clone .part.0] () from
/usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
#2 0x00007f842b2bc206 in JVM_handle_linux_signal () from
/usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
#3  <signal handler called>
apache#4 ___pthread_mutex_lock (mutex=0x1b8) at ./nptl/pthread_mutex_lock.c:80
apache#5 0x000055c5f03edcca in __gthread_mutex_lock (__mutex=0x1b8) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/x86_64-linux-gnu/c++/11/bits/gthr-default.h:749
apache#6 std::mutex::lock (this=0x1b8) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_mutex.h:100
apache#7 std::lock_guard<std::mutex>::lock_guard (__m=..., this=<optimized
out>) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_mutex.h:229
apache#8 doris::RuntimeProfile::add_info_string (this=0x0, key=..., value=...)
at
/home/zcp/repo_center/doris_master/doris/be/src/util/runtime_profile.cpp:354
apache#9 0x000055c5f9cccaa7 in
doris::pipeline::PipelineTask::_state_transition (this=0x7f84015b4000,
new_state=doris::pipeline::PipelineTask::State::RUNNABLE) at
/home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_task.h:339
apache#10 0x000055c5fa53aec5 in doris::pipeline::PipelineTask::wake_up
(this=0x7fffbd28a8e8, dep=<optimized out>) at
/home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_task.cpp:748
apache#11 0x000055c5f9cba3f5 in doris::pipeline::Dependency::set_ready
(this=0x7f832f848310) at
/home/zcp/repo_center/doris_master/doris/be/src/pipeline/dependency.cpp:72
apache#12 0x000055c5f9c53bcd in
doris::vectorized::VDataStreamRecvr::SenderQueue::set_source_ready
(this=0x7f839d485880) at
/home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_recvr.cpp:142
apache#13 doris::vectorized::VDataStreamRecvr::SenderQueue::close
(this=0x7f839d485880) at
/home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_recvr.cpp:301
apache#14 0x000055c5f9c55b39 in doris::vectorized::VDataStreamRecvr::close
(this=0x7f83fbe78940) at
/home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_recvr.cpp:488
apache#15 0x000055c5f9c4dc68 in
doris::vectorized::VDataStreamMgr::~VDataStreamMgr (this=0x7f83f6d99da0)
at
/home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_mgr.cpp:58
apache#16 0x000055c5f00cb651 in doris::ExecEnv::destroy (this=0x55c5fdf7a198
<doris::ExecEnv::GetInstance()::s_exec_env>) at
/home/zcp/repo_center/doris_master/doris/be/src/runtime/exec_env_init.cpp:796
apache#17 0x000055c5ef387909 in main (argc=<optimized out>, argv=<optimized
out>) at
/home/zcp/repo_center/doris_master/doris/be/src/service/doris_main.cpp:635
kaka11chen pushed a commit that referenced this pull request Apr 19, 2025
@deardeng
[fix](microbench) Fix some file cache microbench compile bug (apache#…
c5910ea 
…49269)

### What problem does this PR solve?

1. fix micro BUILD_FILE_CACHE_MICROBENCH_TOOL not work in
file_cache_microbench.cpp
2. fix MircobenchS3FileWriter typo error
3. ./build.sh default not compile file_cache_microbench
4. fix , exit coredump
```
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f54c898a550 in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
[Current thread is 1 (Thread 0x7f54cafad940 (LWP 2810860))]
(gdb) bt
#0  0x00007f54c898a550 in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
#1  0x000055eb6a7ac513 in __gthread_join (__value_ptr=0x0, __threadid=<optimized out>) at /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:669
#2  std::thread::join (this=0x55eb6eda1ce0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:112
#3  0x000055eb5cb548bb in ThreadPool::~ThreadPool (this=this@entry=0x7fff65768480) 
apache#4  0x000055eb5cb4aebe in JobManager::~JobManager (this=this@entry=0x7fff65768420) 
apache#5  0x000055eb5cb49a14 in main (argc=1, argv=<optimized out>) 
```

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [x] No need to test or manual test. Explain why:
- [x] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
kaka11chen pushed a commit that referenced this pull request Apr 19, 2025
@BiteTheDDDDt
[Bug](runtime-filter) add lock for RuntimeFilterConsumer::acquire_exp…
2b90e70 
…r/signal to avoid mult… (apache#49739)

…ithread read and write wrapper

### What problem does this PR solve?
```cpp
=================================================================
==28655==ERROR: AddressSanitizer: heap-use-after-free on address 0x61000331f350 at pc 0x5577863cea6c bp 0x7fac1d2a9ef0 sp 0x7fac1d2a9ee8
READ of size 1 at 0x61000331f350 thread T1933 (Pipe_normal [wo)
    #0 0x5577863cea6b in doris::RuntimeFilterWrapper::debug_string[abi:cxx11]() /root/doris/be/src/runtime_filter/runtime_filter_wrapper.cpp:585:46
    #1 0x55778632f8f5 in doris::RuntimeFilter::_debug_string[abi:cxx11]() const /root/doris/be/src/runtime_filter/runtime_filter.cpp:128:61
    #2 0x55778638d946 in doris::RuntimeFilterConsumer::debug_string[abi:cxx11]() const /root/doris/be/src/runtime_filter/runtime_filter_consumer.h:61:57
    #3 0x55778634ed24 in doris::RuntimeFilterConsumer::_set_state(doris::RuntimeFilterConsumer::State) /root/doris/be/src/runtime_filter/runtime_filter_consumer.h:125:43
    apache#4 0x55778634c6fa in doris::RuntimeFilterConsumer::acquire_expr(std::vector<std::shared_ptr<doris::vectorized::VRuntimeFilterWrapper>, std::allocator<std::shared_ptr<doris::vectorized::VRuntimeFilterWrapper> > >&) /root/doris/be/src/runtime_filter/runtime_filter_consumer.cpp:67:9
    apache#5 0x5577a1fbdd1b in doris::RuntimeFilterConsumerHelper::acquire_runtime_filter(std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext> > >&) /root/doris/be/src/runtime_filter/runtime_filter_consumer_helper.cpp:90:9
    apache#6 0x5577b9aec26c in doris::pipeline::ScanLocalState<doris::pipeline::OlapScanLocalState>::open(doris::RuntimeState*) /root/doris/be/src/pipeline/exec/scan_operator.cpp:100:5
    apache#7 0x5577ba853e05 in doris::pipeline::PipelineTask::_open() /root/doris/be/src/pipeline/pipeline_task.cpp:217:32
    apache#8 0x5577ba8608e0 in doris::pipeline::PipelineTask::execute(bool*) /root/doris/be/src/pipeline/pipeline_task.cpp:407:9
    apache#9 0x5577ba8b4bb5 in doris::pipeline::TaskScheduler::_do_work(int) /root/doris/be/src/pipeline/task_scheduler.cpp:147:9
    apache#10 0x5577869c537c in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:619:24
    apache#11 0x55778699ba1e in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
    apache#12 0x55778699ba1e in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:496:5
    apache#13 0x7fb57bef3608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
    apache#14 0x7fb57c1a0132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

0x61000331f350 is located 16 bytes inside of 192-byte region [0x61000331f340,0x61000331f400)
freed by thread T817 (brpc_light) here:
    #0 0x557781e1c80d in operator delete(void*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x3395280d) (BuildId: 45c9f8cc52f91b28)
    #1 0x557781e778b8 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::operator=(std::__shared_count<(__gnu_cxx::_Lock_policy)2> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:721:15
    #2 0x55778634d1d1 in std::__shared_ptr<doris::RuntimeFilterWrapper, (__gnu_cxx::_Lock_policy)2>::operator=(std::__shared_ptr<doris::RuntimeFilterWrapper, (__gnu_cxx::_Lock_policy)2> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1148:69
    #3 0x55778634d1d1 in std::shared_ptr<doris::RuntimeFilterWrapper>::operator=(std::shared_ptr<doris::RuntimeFilterWrapper> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:359:65
    apache#4 0x55778634d1d1 in doris::RuntimeFilterConsumer::signal(doris::RuntimeFilter*) /root/doris/be/src/runtime_filter/runtime_filter_consumer.cpp:76:14
    apache#5 0x557785dd52c3 in auto doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0::operator()<std::shared_ptr<doris::RuntimeFilterConsumer> >(std::shared_ptr<doris::RuntimeFilterConsumer>&) const /root/doris/be/src/runtime/fragment_mgr.cpp:1295:72
    apache#6 0x557785dd52c3 in void std::__invoke_impl<void, doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0&, std::shared_ptr<doris::RuntimeFilterConsumer>&>(std::__invoke_other, doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0&, std::shared_ptr<doris::RuntimeFilterConsumer>&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    apache#7 0x557785dd52c3 in std::__invoke_result<doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0&, std::shared_ptr<doris::RuntimeFilterConsumer>&>::type std::__invoke<doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0&, std::shared_ptr<doris::RuntimeFilterConsumer>&>(doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0&, std::shared_ptr<doris::RuntimeFilterConsumer>&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
    apache#8 0x557785dd52c3 in std::ranges::in_fun_result<__gnu_cxx::__normal_iterator<std::shared_ptr<doris::RuntimeFilterConsumer>*, std::vector<std::shared_ptr<doris::RuntimeFilterConsumer>, std::allocator<std::shared_ptr<doris::RuntimeFilterConsumer> > > >, doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0> std::ranges::__for_each_fn::operator()<__gnu_cxx::__normal_iterator<std::shared_ptr<doris::RuntimeFilterConsumer>*, std::vector<std::shared_ptr<doris::RuntimeFilterConsumer>, std::allocator<std::shared_ptr<doris::RuntimeFilterConsumer> > > >, __gnu_cxx::__normal_iterator<std::shared_ptr<doris::RuntimeFilterConsumer>*, std::vector<std::shared_ptr<doris::RuntimeFilterConsumer>, std::allocator<std::shared_ptr<doris::RuntimeFilterConsumer> > > >, std::identity, doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0>(__gnu_cxx::__normal_iterator<std::shared_ptr<doris::RuntimeFilterConsumer>*, std::vector<std::shared_ptr<doris::RuntimeFilterConsumer>, std::allocator<std::shared_ptr<doris::RuntimeFilterConsumer> > > >, __gnu_cxx::__normal_iterator<std::shared_ptr<doris::RuntimeFilterConsumer>*, std::vector<std::shared_ptr<doris::RuntimeFilterConsumer>, std::allocator<std::shared_ptr<doris::RuntimeFilterConsumer> > > >, doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*)::$_0, std::identity) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/ranges_algo.h:187:4
    apache#9 0x557785dd52c3 in _ZNKSt6ranges13__for_each_fnclIRSt6vectorISt10shared_ptrIN5doris21RuntimeFilterConsumerEESaIS6_EESt8identityZNS4_11FragmentMgr14apply_filterv2EPKNS4_23PPublishFilterRequestV2EPN5butil26IOBufAsZeroCopyInputStreamEE3$_0EENS_13in_fun_resultINSt11conditionalIL_ZNS_14borrowed_rangeIT_EEEDTclsr8__detailE14__ranges_beginclsr3stdE7declvalIRSM_EEEENS_8danglingEE4typeET1_EEOSM_SS_T0_ /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/ranges_algo.h:197:9
    apache#10 0x557785dd52c3 in doris::FragmentMgr::apply_filterv2(doris::PPublishFilterRequestV2 const*, butil::IOBufAsZeroCopyInputStream*) /root/doris/be/src/runtime/fragment_mgr.cpp:1295:13
    apache#11 0x5577865a517e in doris::PInternalService::apply_filterv2(google::protobuf::RpcController*, doris::PPublishFilterRequestV2 const*, doris::PPublishFilterResponse*, google::protobuf::Closure*)::$_0::operator()() const /root/doris/be/src/service/internal_service.cpp:1444:48
    apache#12 0x5577865a517e in void std::__invoke_impl<void, doris::PInternalService::apply_filterv2(google::protobuf::RpcController*, doris::PPublishFilterRequestV2 const*, doris::PPublishFilterResponse*, google::protobuf::Closure*)::$_0&>(std::__invoke_other, doris::PInternalService::apply_filterv2(google::protobuf::RpcController*, doris::PPublishFilterRequestV2 const*, doris::PPublishFilterResponse*, google::protobuf::Closure*)::$_0&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    apache#13 0x5577865a517e in std::enable_if<is_invocable_r_v<void, doris::PInternalService::apply_filterv2(google::protobuf::RpcController*, doris::PPublishFilterRequestV2 const*, doris::PPublishFilterResponse*, google::protobuf::Closure*)::$_0&>, void>::type std::__invoke_r<void, doris::PInternalService::apply_filterv2(google::protobuf::RpcController*, doris::PPublishFilterRequestV2 const*, doris::PPublishFilterResponse*, google::protobuf::Closure*)::$_0&>(doris::PInternalService::apply_filterv2(google::protobuf::RpcController*, doris::PPublishFilterRequestV2 const*, doris::PPublishFilterResponse*, google::protobuf::Closure*)::$_0&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
    apache#14 0x5577865a517e in std::_Function_handler<void (), doris::PInternalService::apply_filterv2(google::protobuf::RpcController*, doris::PPublishFilterRequestV2 const*, doris::PPublishFilterResponse*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
    apache#15 0x5577865f07c1 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
    apache#16 0x5577865f07c1 in doris::WorkThreadPool<false>::work_thread(int) /root/doris/be/src/util/work_thread_pool.hpp:158:17
    apache#17 0x5577bddb233f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18

previously allocated by thread T781 (brpc_light) here:
    #0 0x557781e1bfad in operator new(unsigned long) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x33951fad) (BuildId: 45c9f8cc52f91b28)
    #1 0x5577863411ae in __gnu_cxx::new_allocator<std::_Sp_counted_ptr_inplace<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, (__gnu_cxx::_Lock_policy)2> >::allocate(unsigned long, void const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:121:27
    #2 0x5577863411ae in std::allocator<std::_Sp_counted_ptr_inplace<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, (__gnu_cxx::_Lock_policy)2> >::allocate(unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:173:32
    #3 0x5577863411ae in std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, (__gnu_cxx::_Lock_policy)2> > >::allocate(std::allocator<std::_Sp_counted_ptr_inplace<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, (__gnu_cxx::_Lock_policy)2> >&, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:460:20
    apache#4 0x5577863411ae in std::__allocated_ptr<std::allocator<std::_Sp_counted_ptr_inplace<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, (__gnu_cxx::_Lock_policy)2> > > std::__allocate_guarded<std::allocator<std::_Sp_counted_ptr_inplace<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, (__gnu_cxx::_Lock_policy)2> > >(std::allocator<std::_Sp_counted_ptr_inplace<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, (__gnu_cxx::_Lock_policy)2> >&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocated_ptr.h:97:21
    apache#5 0x5577863411ae in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, doris::RuntimeFilterParams*>(doris::RuntimeFilterWrapper*&, std::_Sp_alloc_shared_tag<std::allocator<doris::RuntimeFilterWrapper> >, doris::RuntimeFilterParams*&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:648:19
    apache#6 0x55778632e555 in std::__shared_ptr<doris::RuntimeFilterWrapper, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<doris::RuntimeFilterWrapper>, doris::RuntimeFilterParams*>(std::_Sp_alloc_shared_tag<std::allocator<doris::RuntimeFilterWrapper> >, doris::RuntimeFilterParams*&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1337:14
    apache#7 0x55778632e555 in std::shared_ptr<doris::RuntimeFilterWrapper>::shared_ptr<std::allocator<doris::RuntimeFilterWrapper>, doris::RuntimeFilterParams*>(std::_Sp_alloc_shared_tag<std::allocator<doris::RuntimeFilterWrapper> >, doris::RuntimeFilterParams*&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:409:4
    apache#8 0x55778632e555 in std::shared_ptr<doris::RuntimeFilterWrapper> std::allocate_shared<doris::RuntimeFilterWrapper, std::allocator<doris::RuntimeFilterWrapper>, doris::RuntimeFilterParams*>(std::allocator<doris::RuntimeFilterWrapper> const&, doris::RuntimeFilterParams*&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:860:14
    apache#9 0x55778632e555 in std::shared_ptr<doris::RuntimeFilterWrapper> std::make_shared<doris::RuntimeFilterWrapper, doris::RuntimeFilterParams*>(doris::RuntimeFilterParams*&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:876:14
    apache#10 0x55778632e555 in doris::RuntimeFilter::_init_with_desc(doris::TRuntimeFilterDesc const*, doris::TQueryOptions const*) /root/doris/be/src/runtime_filter/runtime_filter.cpp:123:16
    apache#11 0x5577863794de in doris::RuntimeFilterConsumer::create(doris::RuntimeFilterParamsContext*, doris::TRuntimeFilterDesc const*, int, std::shared_ptr<doris::RuntimeFilterConsumer>*, doris::RuntimeProfile*) /root/doris/be/src/runtime_filter/runtime_filter_consumer.h:46:9
    apache#12 0x55778636bb9f in doris::RuntimeFilterMgr::register_consumer_filter(doris::TRuntimeFilterDesc const&, int, std::shared_ptr<doris::RuntimeFilterConsumer>*, doris::RuntimeProfile*) /root/doris/be/src/runtime_filter/runtime_filter_mgr.cpp:80:5
    apache#13 0x557786147c10 in doris::RuntimeState::register_consumer_runtime_filter(doris::TRuntimeFilterDesc const&, bool, int, std::shared_ptr<doris::RuntimeFilterConsumer>*, doris::RuntimeProfile*) /root/doris/be/src/runtime/runtime_state.cpp:514:17
    apache#14 0x5577a1fbc3e1 in doris::RuntimeFilterConsumerHelper::_register_runtime_filter(bool) /root/doris/be/src/runtime_filter/runtime_filter_consumer_helper.cpp:51:9
```

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [x] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [x] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
kaka11chen pushed a commit that referenced this pull request Apr 22, 2025
wangbo
[branch-2.1]fix task queue core (apache#50236)
34cb4f0 
TaskQueue should be released after all  theads stop.
```
=================================================================
==6632==ERROR: AddressSanitizer: heap-use-after-free on address 0x61700518d440 at pc 0x5616fa5c2a87 bp 0x7f7635b709b0 sp 0x7f7635b709a8
READ of size 8 at 0x61700518d440 thread T5188 (Pipe_test_follo)
    #0 0x5616fa5c2a86 in std::__atomic_base<unsigned long>::load(std::memory_order) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9
    #1 0x5616fa5c2a86 in std::__atomic_base<unsigned long>::operator unsigned long() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:341:16
    #2 0x5616fa5c2a86 in doris::pipeline::PriorityTaskQueue::_try_take_unprotected(bool) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_queue.cpp:58:9
    #3 0x5616fa5c2e16 in doris::pipeline::PriorityTaskQueue::take(unsigned int) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_queue.cpp:110:16
    apache#4 0x5616fa5c4b06 in doris::pipeline::MultiCoreTaskQueue::take(int) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_queue.cpp:170:51
    apache#5 0x5616fa5d1919 in doris::pipeline::TaskScheduler::_do_work(unsigned long) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_scheduler.cpp:267:35
    apache#6 0x5616d1e169f8 in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/threadpool.cpp:544:24
    apache#7 0x5616d1df3e38 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
    apache#8 0x5616d1df3e38 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thread.cpp:498:5
    apache#9 0x7f83a4c35ac2 in start_thread nptl/pthread_create.c:442:8
    apache#10 0x7f83a4cc784f  misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

0x61700518d440 is located 704 bytes inside of 728-byte region [0x61700518d180,0x61700518d458)
freed by thread T2166 here:
    #0 0x5616cea38d9d in operator delete(void*) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16dc0d9d) (BuildId: 09b7b6856fee4f7f)
    #1 0x5616fa5cae77 in std::default_delete<doris::pipeline::PriorityTaskQueue>::operator()(doris::pipeline::PriorityTaskQueue*) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
    #2 0x5616fa5cae77 in std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>::~unique_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
    #3 0x5616fa5cae77 in void std::destroy_at<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>(std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
    apache#4 0x5616fa5cae77 in void std::_Destroy<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>(std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7
    apache#5 0x5616fa5cae77 in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*>(std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*, std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6
    apache#6 0x5616fa5cae77 in void std::_Destroy<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*>(std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*, std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7
    apache#7 0x5616fa5cae77 in void std::_Destroy<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*, std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>(std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*, std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>*, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7
    apache#8 0x5616fa5cae77 in std::vector<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>>::~vector() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2
    apache#9 0x5616fa5c466d in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6
    apache#10 0x5616fa5c466d in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11
    apache#11 0x5616fa5c466d in std::__shared_ptr<std::vector<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>>, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31
    apache#12 0x5616fa5c466d in void std::atomic_store<std::vector<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>>>(std::shared_ptr<std::vector<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>>>*, std::shared_ptr<std::vector<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>, std::allocator<std::unique_ptr<doris::pipeline::PriorityTaskQueue, std::default_delete<doris::pipeline::PriorityTaskQueue>>>>>) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_atomic.h:149:7
    apache#13 0x5616fa5c466d in doris::pipeline::MultiCoreTaskQueue::close() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_queue.cpp:151:5
    apache#14 0x5616fa5ce9f4 in doris::pipeline::TaskScheduler::stop() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_scheduler.cpp:437:26
    apache#15 0x5616d19b7d36 in doris::WorkloadGroup::try_stop_schedulers() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group.cpp:654:22
    apache#16 0x5616d19c6042 in doris::WorkloadGroupMgr::delete_workload_group_by_ids(std::set<unsigned long, std::less<unsigned long>, std::allocator<unsigned long>>) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group_manager.cpp:107:13
    apache#17 0x5616d1a7f6a1 in doris::WorkloadGroupListener::handle_topic_info(std::vector<doris::TopicInfo, std::allocator<doris::TopicInfo>> const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/workload_group_listener.cpp:82:38
    apache#18 0x5616d1a7dd7e in doris::TopicSubscriber::handle_topic_info(doris::TPublishTopicRequest const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/topic_subscriber.cpp:45:35
    apache#19 0x5616d2002932 in doris::BackendServiceProcessor::process_publish_topic_info(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:6577:13
    apache#20 0x5616d1fd6f11 in doris::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, int, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:5416:3
    apache#21 0x5616d2039186 in apache::thrift::TDispatchProcessor::process(std::shared_ptr<apache::thrift::protocol::TProtocol>, std::shared_ptr<apache::thrift::protocol::TProtocol>, void*) /home/zcp/repo_center/doris_branch-2.1/doris/thirdparty/installed/include/thrift/TDispatchProcessor.h:121:12
    apache#22 0x5616faee19f7 in apache::thrift::server::TConnectedClient::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432699f7) (BuildId: 09b7b6856fee4f7f)
    apache#23 0x5616faee2cf6 in apache::thrift::server::TThreadedServer::TConnectedClientRunner::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326acf6) (BuildId: 09b7b6856fee4f7f)
    apache#24 0x5616faee830d in apache::thrift::concurrency::Thread::threadMain(std::shared_ptr<apache::thrift::concurrency::Thread>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4327030d) (BuildId: 09b7b6856fee4f7f)
    apache#25 0x5616faee81b3 in void std::__invoke_impl<void, void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(std::__invoke_other, void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432701b3) (BuildId: 09b7b6856fee4f7f)
    apache#26 0x5616faee8106 in std::__invoke_result<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>::type std::__invoke<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270106) (BuildId: 09b7b6856fee4f7f)
    apache#27 0x5616faee8076 in void std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270076) (BuildId: 09b7b6856fee4f7f)
    apache#28 0x5616faee7ffb in std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::operator()() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326fffb) (BuildId: 09b7b6856fee4f7f)
    apache#29 0x5616faee7f9f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>>::_M_run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326ff9f) (BuildId: 09b7b6856fee4f7f)
    apache#30 0x5616fd80568f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18

previously allocated by thread T3392 here:
    #0 0x5616cea3853d in operator new(unsigned long) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16dc053d) (BuildId: 09b7b6856fee4f7f)
    #1 0x5616fa5c3e29 in std::_MakeUniq<doris::pipeline::PriorityTaskQueue>::__single_object std::make_unique<doris::pipeline::PriorityTaskQueue>() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:30
    #2 0x5616fa5c3e29 in doris::pipeline::MultiCoreTaskQueue::MultiCoreTaskQueue(int) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_queue.cpp:139:39
    #3 0x5616d19b276d in decltype(::new((void*)(0)) doris::pipeline::MultiCoreTaskQueue(std::declval<int&>())) std::construct_at<doris::pipeline::MultiCoreTaskQueue, int&>(doris::pipeline::MultiCoreTaskQueue*, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:97:39
    apache#4 0x5616d19b276d in void std::allocator_traits<std::allocator<doris::pipeline::MultiCoreTaskQueue>>::construct<doris::pipeline::MultiCoreTaskQueue, int&>(std::allocator<doris::pipeline::MultiCoreTaskQueue>&, doris::pipeline::MultiCoreTaskQueue*, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:514:4
    apache#5 0x5616d19b276d in std::_Sp_counted_ptr_inplace<doris::pipeline::MultiCoreTaskQueue, std::allocator<doris::pipeline::MultiCoreTaskQueue>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<int&>(std::allocator<doris::pipeline::MultiCoreTaskQueue>, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:519:4
    apache#6 0x5616d19b276d in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<doris::pipeline::MultiCoreTaskQueue, std::allocator<doris::pipeline::MultiCoreTaskQueue>, int&>(doris::pipeline::MultiCoreTaskQueue*&, std::_Sp_alloc_shared_tag<std::allocator<doris::pipeline::MultiCoreTaskQueue>>, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:651:6
    apache#7 0x5616d19b276d in std::__shared_ptr<doris::pipeline::MultiCoreTaskQueue, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<doris::pipeline::MultiCoreTaskQueue>, int&>(std::_Sp_alloc_shared_tag<std::allocator<doris::pipeline::MultiCoreTaskQueue>>, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1337:14
    apache#8 0x5616d19b276d in std::shared_ptr<doris::pipeline::MultiCoreTaskQueue>::shared_ptr<std::allocator<doris::pipeline::MultiCoreTaskQueue>, int&>(std::_Sp_alloc_shared_tag<std::allocator<doris::pipeline::MultiCoreTaskQueue>>, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:409:4
    apache#9 0x5616d19b276d in std::shared_ptr<doris::pipeline::MultiCoreTaskQueue> std::allocate_shared<doris::pipeline::MultiCoreTaskQueue, std::allocator<doris::pipeline::MultiCoreTaskQueue>, int&>(std::allocator<doris::pipeline::MultiCoreTaskQueue> const&, int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:860:14
    apache#10 0x5616d19b276d in std::shared_ptr<doris::pipeline::MultiCoreTaskQueue> std::make_shared<doris::pipeline::MultiCoreTaskQueue, int&>(int&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:876:14
    apache#11 0x5616d19b276d in doris::WorkloadGroup::upsert_thread_pool_no_lock(doris::WorkloadGroupInfo*, std::shared_ptr<doris::CgroupCpuCtl>, doris::ExecEnv*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group.cpp:462:27
    apache#12 0x5616d19b5991 in doris::WorkloadGroup::upsert_task_scheduler(doris::WorkloadGroupInfo*, doris::ExecEnv*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group.cpp:572:9
    apache#13 0x5616d1a7ecd8 in doris::WorkloadGroupListener::handle_topic_info(std::vector<doris::TopicInfo, std::allocator<doris::TopicInfo>> const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/workload_group_listener.cpp:60:13
    apache#14 0x5616d1a7dd7e in doris::TopicSubscriber::handle_topic_info(doris::TPublishTopicRequest const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/topic_subscriber.cpp:45:35
    apache#15 0x5616d2002932 in doris::BackendServiceProcessor::process_publish_topic_info(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:6577:13
    apache#16 0x5616d1fd6f11 in doris::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, int, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:5416:3
    apache#17 0x5616d2039186 in apache::thrift::TDispatchProcessor::process(std::shared_ptr<apache::thrift::protocol::TProtocol>, std::shared_ptr<apache::thrift::protocol::TProtocol>, void*) /home/zcp/repo_center/doris_branch-2.1/doris/thirdparty/installed/include/thrift/TDispatchProcessor.h:121:12
    apache#18 0x5616faee19f7 in apache::thrift::server::TConnectedClient::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432699f7) (BuildId: 09b7b6856fee4f7f)
    apache#19 0x5616faee2cf6 in apache::thrift::server::TThreadedServer::TConnectedClientRunner::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326acf6) (BuildId: 09b7b6856fee4f7f)
    apache#20 0x5616faee830d in apache::thrift::concurrency::Thread::threadMain(std::shared_ptr<apache::thrift::concurrency::Thread>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4327030d) (BuildId: 09b7b6856fee4f7f)
    apache#21 0x5616faee81b3 in void std::__invoke_impl<void, void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(std::__invoke_other, void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432701b3) (BuildId: 09b7b6856fee4f7f)
    apache#22 0x5616faee8106 in std::__invoke_result<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>::type std::__invoke<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270106) (BuildId: 09b7b6856fee4f7f)
    apache#23 0x5616faee8076 in void std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270076) (BuildId: 09b7b6856fee4f7f)
    apache#24 0x5616faee7ffb in std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::operator()() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326fffb) (BuildId: 09b7b6856fee4f7f)
    apache#25 0x5616faee7f9f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>>::_M_run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326ff9f) (BuildId: 09b7b6856fee4f7f)
    apache#26 0x5616fd80568f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18

Thread T5188 (Pipe_test_follo) created by T3392 here:
    #0 0x5616ce9e5caa in pthread_create (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16d6dcaa) (BuildId: 09b7b6856fee4f7f)
    #1 0x5616d1df2cc9 in doris::Thread::start_thread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::function<void ()> const&, unsigned long, scoped_refptr<doris::Thread>*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thread.cpp:449:15
    #2 0x5616d1e1d979 in doris::Status doris::Thread::create<void (doris::ThreadPool::*)(), doris::ThreadPool*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, void (doris::ThreadPool::* const&)(), doris::ThreadPool* const&, scoped_refptr<doris::Thread>*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thread.h:56:16
    #3 0x5616d1e1423d in doris::ThreadPool::create_thread() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/threadpool.cpp:612:12
    apache#4 0x5616d1e13e43 in doris::ThreadPool::init() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/threadpool.cpp:266:25
    apache#5 0x5616cec31535 in doris::Status doris::ThreadPoolBuilder::build<doris::ThreadPool>(std::unique_ptr<doris::ThreadPool, std::default_delete<doris::ThreadPool>>*) const /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/threadpool.h:121:13
    apache#6 0x5616fa5cef22 in doris::pipeline::TaskScheduler::start() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_scheduler.cpp:208:5
    apache#7 0x5616d19b2a9b in doris::WorkloadGroup::upsert_thread_pool_no_lock(doris::WorkloadGroupInfo*, std::shared_ptr<doris::CgroupCpuCtl>, doris::ExecEnv*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group.cpp:467:47
    apache#8 0x5616d19b5991 in doris::WorkloadGroup::upsert_task_scheduler(doris::WorkloadGroupInfo*, doris::ExecEnv*) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/runtime/workload_group/workload_group.cpp:572:9
    apache#9 0x5616d1a7ecd8 in doris::WorkloadGroupListener::handle_topic_info(std::vector<doris::TopicInfo, std::allocator<doris::TopicInfo>> const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/workload_group_listener.cpp:60:13
    apache#10 0x5616d1a7dd7e in doris::TopicSubscriber::handle_topic_info(doris::TPublishTopicRequest const&) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/agent/topic_subscriber.cpp:45:35
    apache#11 0x5616d2002932 in doris::BackendServiceProcessor::process_publish_topic_info(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:6577:13
    apache#12 0x5616d1fd6f11 in doris::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, int, void*) /home/zcp/repo_center/doris_branch-2.1/doris/gensrc/build/gen_cpp/BackendService.cpp:5416:3
    apache#13 0x5616d2039186 in apache::thrift::TDispatchProcessor::process(std::shared_ptr<apache::thrift::protocol::TProtocol>, std::shared_ptr<apache::thrift::protocol::TProtocol>, void*) /home/zcp/repo_center/doris_branch-2.1/doris/thirdparty/installed/include/thrift/TDispatchProcessor.h:121:12
    apache#14 0x5616faee19f7 in apache::thrift::server::TConnectedClient::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432699f7) (BuildId: 09b7b6856fee4f7f)
    apache#15 0x5616faee2cf6 in apache::thrift::server::TThreadedServer::TConnectedClientRunner::run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326acf6) (BuildId: 09b7b6856fee4f7f)
    apache#16 0x5616faee830d in apache::thrift::concurrency::Thread::threadMain(std::shared_ptr<apache::thrift::concurrency::Thread>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4327030d) (BuildId: 09b7b6856fee4f7f)
    apache#17 0x5616faee81b3 in void std::__invoke_impl<void, void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(std::__invoke_other, void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x432701b3) (BuildId: 09b7b6856fee4f7f)
    apache#18 0x5616faee8106 in std::__invoke_result<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>::type std::__invoke<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>(void (*&&)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>&&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270106) (BuildId: 09b7b6856fee4f7f)
    apache#19 0x5616faee8076 in void std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43270076) (BuildId: 09b7b6856fee4f7f)
    apache#20 0x5616faee7ffb in std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>::operator()() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326fffb) (BuildId: 09b7b6856fee4f7f)
    apache#21 0x5616faee7f9f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread>>>>::_M_run() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326ff9f) (BuildId: 09b7b6856fee4f7f)
    apache#22 0x5616fd80568f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18

Thread T3392 created by T933 here:
    #0 0x5616ce9e5caa in pthread_create (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16d6dcaa) (BuildId: 09b7b6856fee4f7f)
    #1 0x5616fd8057b5 in __gthread_create /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663:35
    #2 0x5616fd8057b5 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:147:37
    #3 0x5616faee667c in apache::thrift::concurrency::Thread::start() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326e67c) (BuildId: 09b7b6856fee4f7f)
    apache#4 0x5616faee2a26 in apache::thrift::server::TThreadedServer::onClientConnected(std::shared_ptr<apache::thrift::server::TConnectedClient> const&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326aa26) (BuildId: 09b7b6856fee4f7f)
    apache#5 0x5616faedebc4 in apache::thrift::server::TServerFramework::newlyConnectedClient(std::shared_ptr<apache::thrift::server::TConnectedClient> const&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43266bc4) (BuildId: 09b7b6856fee4f7f)
    apache#6 0x5616faede25f in apache::thrift::server::TServerFramework::serve() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326625f) (BuildId: 09b7b6856fee4f7f)
    apache#7 0x5616faee2791 in apache::thrift::server::TThreadedServer::serve() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326a791) (BuildId: 09b7b6856fee4f7f)
    apache#8 0x5616d1e31419 in doris::ThriftServer::ThriftServerEventProcessor::supervise() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thrift_server.cpp:206:34
    apache#9 0x5616fd80568f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18

Thread T933 created by T0 here:
    #0 0x5616ce9e5caa in pthread_create (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16d6dcaa) (BuildId: 09b7b6856fee4f7f)
    #1 0x5616fd8057b5 in __gthread_create /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663:35
    #2 0x5616fd8057b5 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:147:37
    #3 0x5616d1e37788 in doris::ThriftServer::start() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thrift_server.cpp:421:5
    apache#4 0x5616cea3f174 in main /home/zcp/repo_center/doris_branch-2.1/doris/be/src/service/doris_main.cpp:536:25
    apache#5 0x7f83a4bcad8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

Thread T2166 created by T933 here:
    #0 0x5616ce9e5caa in pthread_create (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x16d6dcaa) (BuildId: 09b7b6856fee4f7f)
    #1 0x5616fd8057b5 in __gthread_create /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663:35
    #2 0x5616fd8057b5 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:147:37
    #3 0x5616faee667c in apache::thrift::concurrency::Thread::start() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326e67c) (BuildId: 09b7b6856fee4f7f)
    apache#4 0x5616faee2a26 in apache::thrift::server::TThreadedServer::onClientConnected(std::shared_ptr<apache::thrift::server::TConnectedClient> const&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326aa26) (BuildId: 09b7b6856fee4f7f)
    apache#5 0x5616faedebc4 in apache::thrift::server::TServerFramework::newlyConnectedClient(std::shared_ptr<apache::thrift::server::TConnectedClient> const&) (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x43266bc4) (BuildId: 09b7b6856fee4f7f)
    apache#6 0x5616faede25f in apache::thrift::server::TServerFramework::serve() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326625f) (BuildId: 09b7b6856fee4f7f)
    apache#7 0x5616faee2791 in apache::thrift::server::TThreadedServer::serve() (/mnt/hdd01/ci/doris-deploy-branch-2.1-local/be/lib/doris_be+0x4326a791) (BuildId: 09b7b6856fee4f7f)
    apache#8 0x5616d1e31419 in doris::ThriftServer::ThriftServerEventProcessor::supervise() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thrift_server.cpp:206:34
    apache#9 0x5616fd80568f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18

SUMMARY: AddressSanitizer: heap-use-after-free /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9 in std::__atomic_base<unsigned long>::load(std::memory_order) const
Shadow bytes around the buggy address:
  0x61700518d180: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61700518d200: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61700518d280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61700518d300: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x61700518d380: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x61700518d400: fd fd fd fd fd fd fd fd[fd]fd fd fa fa fa fa fa
  0x61700518d480: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x61700518d500: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x61700518d580: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x61700518d600: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x61700518d680: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==6632==ABORTING
```
morningman pushed a commit that referenced this pull request May 16, 2025
@mrhhsg
[fix](join)Consider mark join when computing right_col_idx (apache#50720
966865c 
)

### What problem does this PR solve?

If there is a mark join condition, then even in a right semi join, the
columns from the left table involved in the mark join condition will
still appear in the intermediate tuple.

This PR also completes the missing handling logic for right semi joins
with mark join condition.

```
==2126730==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x503004ee02e0 at pc 0x55f4a7d3f354 bp 0x7f6b6b587e70 sp 0x7f6b6b587e68
READ of size 8 at 0x503004ee02e0 thread T1296 (brpc_light)
    #0 0x55f4a7d3f353 in std::__shared_ptr<doris::vectorized::IDataType const, (__gnu_cxx::_Lock_policy)2>::__shared_ptr(std::__shared_ptr<doris::vectorized::IDataType const, (__gnu_cxx::_Lock_policy)2> const&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr_base.h:1522:7
    #1 0x55f4a7d3f01e in std::shared_ptr<doris::vectorized::IDataType const>::shared_ptr(std::shared_ptr<doris::vectorized::IDataType const> const&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/shared_ptr.h:204:7
    #2 0x55f4f8aa231c in doris::pipeline::HashJoinProbeOperatorX::prepare(doris::RuntimeState*) /root/doris/be/src/pipeline/exec/hashjoin_probe_operator.cpp:577:33
    #3 0x55f4fb1e4a28 in doris::pipeline::Pipeline::prepare(doris::RuntimeState*) /root/doris/be/src/pipeline/pipeline.cpp:89:5
    apache#4 0x55f4fb089d3f in doris::pipeline::PipelineFragmentContext::prepare(doris::TPipelineFragmentParams const&, doris::ThreadPool*) /root/doris/be/src/pipeline/pipeline_fragment_context.cpp:352:9
    apache#5 0x55f4ad615a5c in doris::FragmentMgr::exec_plan_fragment(doris::TPipelineFragmentParams const&, doris::QuerySource, std::function<void (doris::RuntimeState*, doris::Status*)> const&, doris::TPipelineFragmentParamsList const&) /root/doris/be/src/runtime/fragment_mgr.cpp:855:9
    apache#6 0x55f4ad613ca6 in doris::FragmentMgr::exec_plan_fragment(doris::TPipelineFragmentParams const&, doris::QuerySource, doris::TPipelineFragmentParamsList const&) /root/doris/be/src/runtime/fragment_mgr.cpp:634:16
    apache#7 0x55f4ae2e867c in doris::PInternalService::_exec_plan_fragment_impl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, doris::PFragmentRequestVersion, bool, std::function<void (doris::RuntimeState*, doris::Status*)> const&) /root/doris/be/src/service/internal_service.cpp:613:17
    apache#8 0x55f4ae2e4b67 in doris::PInternalService::_exec_plan_fragment_in_pthread(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*) /root/doris/be/src/service/internal_service.cpp:343:14
    apache#9 0x55f4ae31d4e1 in doris::PInternalService::exec_plan_fragment_prepare(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*)::$_0::operator()() const /root/doris/be/src/service/internal_service.cpp:367:9
    apache#10 0x55f4ae31d33e in void std::__invoke_impl<void, doris::PInternalService::exec_plan_fragment_prepare(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*)::$_0&>(std::__invoke_other, doris::PInternalService::exec_plan_fragment_prepare(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*)::$_0&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    apache#11 0x55f4ae31d27e in std::enable_if<is_invocable_r_v<void, doris::PInternalService::exec_plan_fragment_prepare(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*)::$_0&>, void>::type std::__invoke_r<void, doris::PInternalService::exec_plan_fragment_prepare(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*)::$_0&>(doris::PInternalService::exec_plan_fragment_prepare(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*)::$_0&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2
    apache#12 0x55f4ae31cf55 in std::_Function_handler<void (), doris::PInternalService::exec_plan_fragment_prepare(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9
    apache#13 0x55f4a74175af in std::function<void ()>::operator()() const /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
    apache#14 0x55f4ae3c21c4 in doris::WorkThreadPool<false>::work_thread(int) /root/doris/be/src/util/work_thread_pool.hpp:158:17
    apache#15 0x55f4ae3c4cc8 in void std::__invoke_impl<void, void (doris::WorkThreadPool<false>::* const&)(int), doris::WorkThreadPool<false>*&, int&>(std::__invoke_memfun_deref, void (doris::WorkThreadPool<false>::* const&)(int), doris::WorkThreadPool<false>*&, int&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:74:14
    apache#16 0x55f4ae3c4a92 in std::__invoke_result<void (doris::WorkThreadPool<false>::* const&)(int), doris::WorkThreadPool<false>*&, int&>::type std::__invoke<void (doris::WorkThreadPool<false>::* const&)(int), doris::WorkThreadPool<false>*&, int&>(void (doris::WorkThreadPool<false>::* const&)(int), doris::WorkThreadPool<false>*&, int&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14
    apache#17 0x55f4ae3c49f8 in decltype(std::__invoke((*this)._M_pmf, std::forward<doris::WorkThreadPool<false>*&>(fp), std::forward<int&>(fp))) std::_Mem_fn_base<void (doris::WorkThreadPool<false>::*)(int), true>::operator()<doris::WorkThreadPool<false>*&, int&>(doris::WorkThreadPool<false>*&, int&) const /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:170:11
    apache#18 0x55f4ae3c4942 in void std::__invoke_impl<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)>&, doris::WorkThreadPool<false>*&, int&>(std::__invoke_other, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)>&, doris::WorkThreadPool<false>*&, int&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    apache#19 0x55f4ae3c4752 in std::enable_if<is_invocable_r_v<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)>&, doris::WorkThreadPool<false>*&, int&>, void>::type std::__invoke_r<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)>&, doris::WorkThreadPool<false>*&, int&>(std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)>&, doris::WorkThreadPool<false>*&, int&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2
    apache#20 0x55f4ae3c4664 in void std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>::__call<void, 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:654:11
    apache#21 0x55f4ae3c4395 in void std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>::operator()<>() /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/functional:713:17
    apache#22 0x55f4ae3c428e in void std::__invoke_impl<void, std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>>(std::__invoke_other, std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>&&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    apache#23 0x55f4ae3c41ce in std::__invoke_result<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>>::type std::__invoke<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>>(std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>&&) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14
    apache#24 0x55f4ae3c417b in void std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:292:13
    apache#25 0x55f4ae3c40f6 in std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>>>::operator()() /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:299:11
    apache#26 0x55f4ae3c3f34 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>>>>::_M_run() /root/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:244:13
    apache#27 0x55f4fea05d2e in execute_native_thread_routine pthread_atfork.c
    apache#28 0x55f4a7162e0a in asan_thread_start(void*) crtstuff.c
    apache#29 0x7f7149c421c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 7c4add5c7a885e6ff4ce17867d6a2286e4420eec)
    apache#30 0x7f714a6318d2 in clone (/lib64/libc.so.6+0x398d2) (BuildId: 4ee3325955e3b55b6805f33959b7cb77745ad625)
```
kaka11chen pushed a commit that referenced this pull request May 21, 2025
@Mryange
[fix](core) Fix the core issue caused by extra fields when casting a …
e8717f2 
…string to a struct. (apache#50851)

```
select cast('{"a":1,"b":"1","c":"1","d":"1"}' as struct<a:int, b:int>);

==3393513==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x506001118e20 at pc 0x55808f91364a bp 0x7f9ec8816af0 sp 0x7f9ec8816ae8
READ of size 8 at 0x506001118e20 thread T1298 (brpc_light)
    #0 0x55808f913649 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_data() const /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:223:28
    #1 0x55808f9386f6 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::data() const /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:2597:16
    #2 0x55808fc4e57a in doris::Slice::Slice(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk12/yanxuecheng/doris/be/src/util/slice.h:72:40
    #3 0x5580bebb7ed9 in doris::vectorized::DataTypeStructSerDe::deserialize_one_cell_from_json(doris::vectorized::IColumn&, doris::Slice&, doris::vectorized::DataTypeSerDe::FormatOptions const&) const /mnt/disk12/yanxuecheng/doris/be/src/vec/data_types/serde/data_type_struct_serde.cpp:141:17
    apache#4 0x5580c90cf235 in doris::vectorized::ConvertImplGenericFromString::execute(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) /mnt/disk12/yanxuecheng/doris/be/src/vec/functions/function_cast.h:513:36
    apache#5 0x5580c89607e6 in doris::Status std::__invoke_impl<doris::Status, doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>(std::__invoke_other, doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    apache#6 0x5580c89605ca in std::enable_if<is_invocable_r_v<doris::Status, doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>, doris::Status>::type std::__invoke_r<doris::Status, doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>(doris::Status (*&)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long), doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:114:9
    apache#7 0x5580c8960201 in std::_Function_handler<doris::Status (doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long), doris::Status (*)(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)>::_M_invoke(std::_Any_data const&, doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9
    apache#8 0x5580c827ff5c in std::function<doris::Status (doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long)>::operator()(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long) const /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
    apache#9 0x5580c88fb7a7 in doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::operator()(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const /mnt/disk12/yanxuecheng/doris/be/src/vec/functions/function_cast.h:2148:17
    apache#10 0x5580c88fae32 in doris::Status std::__invoke_impl<doris::Status, doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)&, doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>(std::__invoke_other, doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)&, doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    apache#11 0x5580c88fac0a in std::enable_if<is_invocable_r_v<doris::Status, doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)&, doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>, doris::Status>::type std::__invoke_r<doris::Status, doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)&, doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long>(doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)&, doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:114:9
    apache#12 0x5580c88fa641 in std::_Function_handler<doris::Status (doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long), doris::vectorized::FunctionCast::prepare_remove_nullable(doris::FunctionContext*, std::shared_ptr<doris::vectorized::IDataType const> const&, std::shared_ptr<doris::vectorized::IDataType const> const&, bool) const::'lambda'(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)>::_M_invoke(std::_Any_data const&, doris::FunctionContext*&&, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long&&, unsigned long&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9
    apache#13 0x5580c827ff5c in std::function<doris::Status (doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long)>::operator()(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned long, unsigned long) const /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
   
```

now
```
mysql> select cast('{"a":1,"b":"1","c":"1","d":"1"}' as struct<a:int, b:int>);
+-----------------------------------------------------------------+
| cast('{"a":1,"b":"1","c":"1","d":"1"}' as struct<a:int, b:int>) |
+-----------------------------------------------------------------+
| NULL                                                            |
+-----------------------------------------------------------------+
kaka11chen pushed a commit that referenced this pull request Jun 11, 2025
@HappenLee
[Bug](fix) try to fix double free materialization status (apache#51518)
63ad04b 
### What problem does this PR solve?
```
=================================================================
==8328==ERROR: AddressSanitizer: attempting double-free on 0x60600333cf80 in thread T1682:
    #0 0x5631ccdaed9d in operator delete(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3caa6d9d) (BuildId: dab4df3df11ecdce)
    #1 0x5631ccdde467 in std::__uniq_ptr_impl>::operator=(std::__uniq_ptr_impl>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:167:2
    #2 0x5631ccdc942d in std::__uniq_ptr_data, true, true>::operator=(std::__uniq_ptr_data, true, true>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:212:61
    #3 0x5631ccdc942d in std::unique_ptr>::operator=(std::unique_ptr>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:371:51
    apache#4 0x5631ccdc942d in doris::Status::operator=(doris::Status&&) /home/zcp/repo_center/doris_master/doris/be/src/common/status.h:400:22
    apache#5 0x56320bfc33ba in doris::pipeline::MaterializationCallback::call() /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/materialization_sink_operator.cpp:88:39
    apache#6 0x56320bfc5cdd in doris::AutoReleaseClosure>::Run() /home/zcp/repo_center/doris_master/doris/be/src/util/brpc_closure.h:98:18
    apache#7 0x5631d03b7b1b in doris::FailureDetectClosure::Run() /home/zcp/repo_center/doris_master/doris/be/src/util/brpc_client_cache.h:83:20
    apache#8 0x56320dce1943 in brpc::Controller::EndRPC(brpc::Controller::CompletionInfo const&) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9d9943) (BuildId: dab4df3df11ecdce)
    apache#9 0x56320dd0cc4e in brpc::policy::ProcessRpcResponse(brpc::InputMessageBase*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7da04c4e) (BuildId: dab4df3df11ecdce)
    apache#10 0x56320dd03f76 in brpc::ProcessInputMessage(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9fbf76) (BuildId: dab4df3df11ecdce)
    apache#11 0x56320dd04920 in brpc::InputMessenger::InputMessageClosure::~InputMessageClosure() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9fc920) (BuildId: dab4df3df11ecdce)
    apache#12 0x56320dd052ab in brpc::InputMessenger::OnNewMessages(brpc::Socket*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9fd2ab) (BuildId: dab4df3df11ecdce)
    apache#13 0x56320de3437d in brpc::Socket::ProcessEvent(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7db2c37d) (BuildId: dab4df3df11ecdce)
    apache#14 0x56320dc970a6 in bthread::TaskGroup::task_runner(long) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d98f0a6) (BuildId: dab4df3df11ecdce)
    apache#15 0x56320dc83660 in bthread_make_fcontext (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d97b660) (BuildId: dab4df3df11ecdce)

0x60600333cf80 is located 0 bytes inside of 64-byte region [0x60600333cf80,0x60600333cfc0)
freed by thread T1791 here:
    #0 0x5631ccdaed9d in operator delete(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3caa6d9d) (BuildId: dab4df3df11ecdce)

previously allocated by thread T1623 here:
    #0 0x5631ccdae53d in operator new(unsigned long) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3caa653d) (BuildId: dab4df3df11ecdce)

Thread T1682 created by T0 here:
    #0 0x5631ccd5bcaa in pthread_create (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3ca53caa) (BuildId: dab4df3df11ecdce)
    #1 0x56320dc907be in bthread::TaskControl::add_workers(int) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9887be) (BuildId: dab4df3df11ecdce)
    #2 0x56320dc7f0ac in bthread_setconcurrency (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9770ac) (BuildId: dab4df3df11ecdce)
    #3 0x56320ddfcc79 in brpc::Server::StartInternal(butil::EndPoint const&, brpc::PortRange const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf4c79) (BuildId: dab4df3df11ecdce)
    apache#4 0x56320ddfecc9 in brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf6cc9) (BuildId: dab4df3df11ecdce)
    apache#5 0x5631d13708df in doris::BRpcService::start(int, int) /home/zcp/repo_center/doris_master/doris/be/src/service/brpc_service.cpp:94:18
    apache#6 0x5631ccdb73e2 in main /home/zcp/repo_center/doris_master/doris/be/src/service/doris_main.cpp:570:28
    apache#7 0x7faec2201d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

Thread T1791 created by T0 here:
    #0 0x5631ccd5bcaa in pthread_create (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3ca53caa) (BuildId: dab4df3df11ecdce)
    #1 0x56320dc907be in bthread::TaskControl::add_workers(int) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9887be) (BuildId: dab4df3df11ecdce)
    #2 0x56320dc7f0ac in bthread_setconcurrency (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9770ac) (BuildId: dab4df3df11ecdce)
    #3 0x56320ddfcc79 in brpc::Server::StartInternal(butil::EndPoint const&, brpc::PortRange const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf4c79) (BuildId: dab4df3df11ecdce)
    apache#4 0x56320ddfecc9 in brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf6cc9) (BuildId: dab4df3df11ecdce)
    apache#5 0x5631d13708df in doris::BRpcService::start(int, int) /home/zcp/repo_center/doris_master/doris/be/src/service/brpc_service.cpp:94:18
    apache#6 0x5631ccdb73e2 in main /home/zcp/repo_center/doris_master/doris/be/src/service/doris_main.cpp:570:28
    apache#7 0x7faec2201d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

Thread T1623 created by T0 here:
    #0 0x5631ccd5bcaa in pthread_create (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3ca53caa) (BuildId: dab4df3df11ecdce)
    #1 0x56320dc907be in bthread::TaskControl::add_workers(int) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9887be) (BuildId: dab4df3df11ecdce)
    #2 0x56320dc7f0ac in bthread_setconcurrency (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9770ac) (BuildId: dab4df3df11ecdce)
    #3 0x56320ddfcc79 in brpc::Server::StartInternal(butil::EndPoint const&, brpc::PortRange const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf4c79) (BuildId: dab4df3df11ecdce)
    apache#4 0x56320ddfecc9 in brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf6cc9) (BuildId: dab4df3df11ecdce)
    apache#5 0x5631d13708df in doris::BRpcService::start(int, int) /home/zcp/repo_center/doris_master/doris/be/src/service/brpc_service.cpp:94:18
    apache#6 0x5631ccdb73e2 in main /home/zcp/repo_center/doris_master/doris/be/src/service/doris_main.cpp:570:28
    apache#7 0x7faec2201d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

SUMMARY: AddressSanitizer: double-free (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3caa6d9d) (BuildId: dab4df3df11ecdce) in operator delete(void*)

```

Add a lock when need change the materialization status


Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [x] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [x] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
kaka11chen pushed a commit that referenced this pull request Jun 16, 2025
@yiguolei
[be](gracefulexit) should stop async result writer when graceful exit (…
1030fe3 
…apache#51743)

During graceful exit, async result writer will not exit and hang because
it will try to pull data block the queue endlessly.
AsyncResultWriter thread should exit when fragment mgr wants to exit.

Thread 652 (Thread 0x7fd20c1b0700 (LWP 434882) "FragmentMgrAsyn"):
#0 futex_abstimed_wait_cancelable (private=<optimized out>,
abstime=0x7fca3442f560, clockid=<optimized out>, expected=0,
futex_word=0x61b000c009f8) at ../sysdeps/nptl/futex-internal.h:320
#1 __pthread_cond_wait_common (abstime=0x7fca3442f560,
clockid=<optimized out>, mutex=0x61b000c009a8, cond=0x61b000c009d0) at
pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x61b000c009d0, mutex=0x61b000c009a8,
abstime=0x7fca3442f560) at pthread_cond_wait.c:665
#3 0x00005566ce61f350 in __gthread_cond_timedwait
(__cond=0x61b000c009d0, __mutex=0x61b000c009a8,
__abs_timeout=0x7fca3442f560) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/x86_64-linux-gnu/c++/11/bits/gthr-default.h:872
apache#4 std::__condvar::wait_until (this=0x61b000c009d0, __m=...,
__abs_time=...) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_mutex.h:162
apache#5
std::condition_variable::__wait_until_impl<std::chrono::duration<long,
std::ratio<1l, 1000000000l> > > (this=<optimized out>, __lock=...,
__atime=...) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/condition_variable:222
apache#6 0x00005566ce61f021 in
std::condition_variable::wait_until<std::chrono::_V2::steady_clock,
std::chrono::duration<long, std::ratio<1l, 1000000000l> > >
(this=<optimized out>, __lock=..., __atime=...) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/condition_variable:135
apache#7 0x000055670b59b0be in std::condition_variable::wait_for<long,
std::ratio<1l, 1l> > (this=0x61b000c009d0, __lock=..., __rtime=...) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/condition_variable:163
apache#8 doris::vectorized::AsyncResultWriter::process_block (this=<optimized
out>, state=0x61e001b3c080, profile=<optimized out>) at
/root/doris/be/src/vec/sink/writer/async_result_writer.cpp:149
apache#9 0x000055670b59eb1f in
doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*,
doris::RuntimeProfile*)::$_0::operator()() const (this=0x60400146b110)
at /root/doris/be/src/vec/sink/writer/async_result_writer.cpp:103
apache#10 std::__invoke_impl<void,
doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*,
doris::RuntimeProfile*)::$_0&>(std::__invoke_other,
doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*,
doris::RuntimeProfile*)::$_0&) (__f=...) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
apache#11 std::__invoke_r<void,
doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*,
doris::RuntimeProfile*)::$_0&>(doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*,
doris::RuntimeProfile*)::$_0&) (__fn=...) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111
apache#12 std::_Function_handler<void (),
doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*,
doris::RuntimeProfile*)::$_0>::_M_invoke(std::_Any_data const&)
(__functor=...) at
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
apache#13 0x00005566d2f2270b in doris::ThreadPool::dispatch_thread
(this=0x616000415580) at /root/doris/be/src/util/threadpool.cpp:615
apache#14 0x00005566d2ef973a in doris::Thread::supervise_thread
(arg=0x611005af0540) at /root/doris/be/src/util/thread.cpp:468
apache#15 0x00007fd5c2034609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
apache#16 0x00007fd5c22e1133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
kaka11chen pushed a commit that referenced this pull request Jul 1, 2025
@swjtu-zhanglei
[fix](be) Fix be aws-sdk-cpp memory leak (apache#52185) (apache#52298)
86ea0f2 
* aws/aws-sdk-cpp#2314
* In the aws-sdk-cpp code, there is a suggestion: "Please call
ShutdownAPI from the same thread from which InitAPI() has been called"
otherwise it will cause memory leak.

```
=================================================================
==3271527==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 216 byte(s) in 1 object(s) allocated from:
    #0 0x555f9ecf55ee in malloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x370815ee) (BuildId: 22b637395dd039b0)
    #1 0x555fd8e19bd7 in CRYPTO_zalloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x711a5bd7) (BuildId: 22b637395dd039b0)
    #2 0x555fda7c9d14 in aws_mqtt_library_init (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72b55d14) (BuildId: 22b637395dd039b0)
    #3 0x555fda74ed1f in Aws::Crt::ApiHandle::ApiHandle(aws_allocator*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72adad1f) (BuildId: 22b637395dd039b0)
    apache#4 0x555fa365f0c9 in doris::S3ClientFactory::instance() /root/doris/be/src/util/s3_util.cpp:188:28
    apache#5 0x555f9f13b87b in doris::io::ObjClientHolder::init() /root/doris/be/src/io/fs/s3_file_system.cpp:78:15
    apache#6 0x555f9f140eee in doris::io::S3FileSystem::init() /root/doris/be/src/io/fs/s3_file_system.cpp:182:21
    ......
```

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
kaka11chen pushed a commit that referenced this pull request Jul 2, 2025
@swjtu-zhanglei
[fix](be) Fix be aws-sdk-cpp memory leak (apache#52185)
4f955c6 
* aws/aws-sdk-cpp#2314
* In the aws-sdk-cpp code, there is a suggestion: "Please call
ShutdownAPI from the same thread from which InitAPI() has been called"
otherwise it will cause memory leak.

```
=================================================================
==3271527==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 216 byte(s) in 1 object(s) allocated from:
    #0 0x555f9ecf55ee in malloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x370815ee) (BuildId: 22b637395dd039b0)
    #1 0x555fd8e19bd7 in CRYPTO_zalloc (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x711a5bd7) (BuildId: 22b637395dd039b0)
    #2 0x555fda7c9d14 in aws_mqtt_library_init (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72b55d14) (BuildId: 22b637395dd039b0)
    #3 0x555fda74ed1f in Aws::Crt::ApiHandle::ApiHandle(aws_allocator*) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x72adad1f) (BuildId: 22b637395dd039b0)
    apache#4 0x555fa365f0c9 in doris::S3ClientFactory::instance() /root/doris/be/src/util/s3_util.cpp:188:28
    apache#5 0x555f9f13b87b in doris::io::ObjClientHolder::init() /root/doris/be/src/io/fs/s3_file_system.cpp:78:15
    apache#6 0x555f9f140eee in doris::io::S3FileSystem::init() /root/doris/be/src/io/fs/s3_file_system.cpp:182:21
    ......
```
kaka11chen pushed a commit that referenced this pull request Jul 2, 2025
@yujun777
[fix](doris compose) be disable java support (apache#52412)
49b8301 
### What problem does this PR solve?

be sometimes hang at libjvm when it start,  we don't known why.

so disable be java support. only after we fix the be hang at libjvm, we
enable java support again.

```
Thread 1 (LWP 867 "doris_be"):
#0  0x00007fc6b09087b2 in __pthread_cond_signal_2_0 (cond=0x0) at old_pthread_cond_signal.c:40
#1  0x00007fffab25a2f0 in ?? ()
#2  0x00000000ab25a2f0 in ?? ()
#3  0x00007fc6b1a9436e in WeakHandle::WeakHandle(OopStorage*, oopDesc*) () from /usr/local/openjdk-17/lib/server/libjvm.so
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
[Inferior 1 (process 867) detached]
```
kaka11chen pushed a commit that referenced this pull request Jul 5, 2025
@Mryange
[fix](ubsan) fix ubsan error in DataTypeHLL::deserialize (apache#52645)
50cdda9 
```
/home/zcp/repo_center/doris_master/doris/be/src/vec/data_types/data_type_hll.cpp:105:16: runtime error: null pointer passed as argument 1, which is declared to never be null
/usr/local/ldb-toolchain-v0.24/bin/../usr/include/string.h:43:28: note: nonnull attribute specified here
    #0 0x564a261d7c01 in doris::vectorized::DataTypeHLL::deserialize(char const*, doris::COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>*, int) const /home/zcp/repo_center/doris_master/doris/be/src/vec/data_types/data_type_hll.cpp:105:9
    #1 0x564a260e8f66 in doris::vectorized::Block::deserialize(doris::PBlock const&) /home/zcp/repo_center/doris_master/doris/be/src/vec/core/block.cpp:156:9
    #2 0x564a37fe02eb in doris::vectorized::VDataStreamRecvr::SenderQueue::BlockItem::get_block(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block>>&) /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_recvr.h:272:17
    #3 0x564a37fccdd3 in doris::vectorized::VDataStreamRecvr::SenderQueue::get_batch(doris::vectorized::Block*, bool*) 
```
kaka11chen pushed a commit that referenced this pull request Jul 5, 2025
@cambyzju
[fix](be java) be hang while enable_java_support (apache#52818)
e98e05d 
### What problem does this PR solve?

### Problem

after enable java support, be can not start correctly, it will hang on
stack:
```
(gdb) bt
#0  0x00007f5fb1e97ce6 in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1  0x00007f5fb1e9a798 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6
#2  0x00007f5fb2c98bf3 in os::PlatformEvent::park() () from /usr/lib/jvm/java-17//lib/server/libjvm.so
#3  0x00007f5fb2c693a5 in ObjectMonitor::wait(long, bool, JavaThread*) () from /usr/lib/jvm/java-17//lib/server/libjvm.so
apache#4  0x00007f5fb2e8b316 in ObjectSynchronizer::wait(Handle, long, JavaThread*) () from /usr/lib/jvm/java-17//lib/server/libjvm.so
apache#5  0x00007f5fb2934a97 in JVM_MonitorWait () from /usr/lib/jvm/java-17//lib/server/libjvm.so
apache#6  0x00007f5f9de245ba in ?? ()
apache#7  0x00007f5f49446158 in ?? ()
apache#8  0x00007f5faeb9fa00 in ?? ()
apache#9  0x00007ffc37f91178 in ?? ()
apache#10 0x00007f5f9de304bd in ?? ()
apache#11 0x00007ffc37f910a0 in ?? ()
apache#12 0x0000000000000000 in ?? ()
```

jstack of be:
```
"main" #1 prio=5 os_prio=0 cpu=931.38ms elapsed=66.08s tid=0x00007fab8e12c400 nid=0x3e68aa in Object.wait()  [0x00007ffdb3d4c000]  
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait([email protected]/Native Method)
    - waiting on <0x00000000bd183b68> (a java.lang.ProcessImpl)
    at java.lang.Object.wait([email protected]/Object.java:338)
    at java.lang.ProcessImpl.waitFor([email protected]/ProcessImpl.java:434)
    - locked <0x00000000bd183b68> (a java.lang.ProcessImpl)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:1061)
    at org.apache.hadoop.util.Shell.run(Shell.java:957)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1282)
    at org.apache.hadoop.util.Shell.isSetsidSupported(Shell.java:853)
    at org.apache.hadoop.util.Shell.<clinit>(Shell.java:838)
    at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79)
    at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1713)
    at org.apache.hadoop.security.SecurityUtil.setConfigurationInternal(SecurityUtil.java:103)
    at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:92)
    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:312)
    - locked <0x00000000bd5fd708> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation)
    at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:300)
    - locked <0x00000000bd5fd708> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:575)
    at org.apache.hadoop.fs.viewfs.ViewFileSystem.<init>(ViewFileSystem.java:279)
    at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0([email protected]/Native Method)
    at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance([email protected]/NativeConstructorAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance([email protected]/DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstanceWithCaller([email protected]/Constructor.java:500)
    at java.lang.reflect.Constructor.newInstance([email protected]/Constructor.java:481)
    at java.util.ServiceLoader$ProviderImpl.newInstance([email protected]/ServiceLoader.java:789)
    at java.util.ServiceLoader$ProviderImpl.get([email protected]/ServiceLoader.java:729)
    at java.util.ServiceLoader$3.next([email protected]/ServiceLoader.java:1403)
    at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3534)
    - locked <0x00000000826529f0> (a java.lang.Class for org.apache.hadoop.fs.FileSystem)
```

This problem comes from: apache#45287

after this fix, we could enable java support:
apache#52412

### Another Fix Method
Add JAVA_OPTS `-Djdk.lang.processReaperUseDefaultStackSize=true` inside
be.conf, also can fix this problem:

https://bugs.openjdk.org/browse/JDK-8153057

From gemini:

![Clipboard_Screenshot_1751683706](https://github.com/user-attachments/assets/a54ffb3a-cb57-4f8a-bc85-5366081c2d9b)
kaka11chen pushed a commit that referenced this pull request Jul 8, 2025
@liaoxin01
[fix](load) Convert RowInBlock* to shared_ptr to fix potential memory…
da9c1cf 
… leaks in MemTable (apache#52902)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Direct leak of 193504 byte(s) in 6047 object(s) allocated from:
#0 0x559f1d2a45fd in operator new(unsigned long)
(/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x2a0015fd)
#1 0x559f1f54c41d in doris::MemTable::insert(doris::vectorized::Block
const*, std::vector<unsigned int, doris::CustomStdAllocator<unsigned
int, doris::Allocator<false, false, false,
doris::DefaultMemoryAllocator> > > const&)
/root/doris/be/src/olap/memtable.cpp:243:38
#2 0x559f1f597c08 in
doris::MemTableWriter::write(doris::vectorized::Block const*,
std::vector<unsigned int, doris::CustomStdAllocator<unsigned int,
doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > >
const&) /root/doris/be/src/olap/memtable_writer.cpp:107:27
#3 0x559f467a48c0 in
doris::DeltaWriterV2::write(doris::vectorized::Block const*,
std::vector<unsigned int, doris::CustomStdAllocator<unsigned int,
doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > >
const&) /root/doris/be/src/olap/delta_writer_v2.cpp:164:30
apache#4 0x559f4711e2f5 in
doris::vectorized::VTabletWriterV2::_write_memtable(std::shared_ptr<doris::vectorized::Block>,
long, doris::vectorized::Rows const&)
/root/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:552:24
apache#5 0x559f4711b4bf in
doris::vectorized::VTabletWriterV2::write(doris::RuntimeState*,
doris::vectorized::Block&)
/root/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:486:9
apache#6 0x559f4702bb4b in
doris::vectorized::AsyncResultWriter::process_block(doris::RuntimeState*,
doris::RuntimeProfile*)
/root/doris/be/src/vec/sink/writer/async_result_writer.cpp:169:23
apache#7 0x559f4702e7e8 in
doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*,
doris::RuntimeProfile*)::$_0::operator()() const
/root/doris/be/src/vec/sink/writer/async_result_writer.cpp:103:5
kaka11chen pushed a commit that referenced this pull request Jul 15, 2025
@freemandealer
[fix](filecache) Fix LRU persist queue thread-safe (apache#53046)
05770cf 
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

1.fix LRU queue crash use after free
2.fix extra LRU queue info when 'need_to_move' flag unset
3.use concurrent queueu to record queueu change info for thread safety

```
ERROR: AddressSanitizer: heap-use-after-free on address 0x603005548c40 at pc 0x55f28e8c4785 bp 0x7f603582e1f0 sp 0x7f603582e1e8
READ of size 8 at 0x603005548c40 thread T201
    #0 0x55f28e8c4784 in std::_Head_base<0ul, doris::io::CacheLRULog*, false>::_Head_base<doris::io::CacheLRULog*>(doris::io::CacheLRULog*&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:190:17
    #1 0x55f28e8c4784 in std::_Tuple_impl<0ul, doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>::_Tuple_impl(std::_Tuple_impl<0ul, doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:292:2
    #2 0x55f28e8c4784 in std::tuple<doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>::tuple(std::tuple<doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:1079:17
    #3 0x55f28e8c4784 in std::_uniq_ptr_impl<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>::uniq_ptr_impl(std::_uniq_ptr_impl<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:162:9
    apache#4 0x55f28e8c4784 in std::_uniq_ptr_data<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>, true, true>::uniq_ptr_data(std::_uniq_ptr_data<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>, true, true>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:211:7
    apache#5 0x55f28e8c4784 in std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>::unique_ptr(std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:327:7
    apache#6 0x55f28e8c4784 in doris::io::LRUQueueRecorder::replay_queue_event(doris::io::FileCacheType) /root/doris/be/src/io/cache/lru_queue_recorder.cpp:40:20
    apache#7 0x55f28e82d620 in doris::io::BlockFileCache::run_background_lru_log_replay() /root/doris/be/src/io/cache/block_file_cache.cpp:2242:24
    apache#8 0x55f2cdc2720f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc+-v3/src/c11/../../../../../libstdc-v3/src/c+11/thread.cc:82:18
    apache#9 0x7f61f1842608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
    apache#10 0x7f61f1aef132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

0x603005548c40 is located 16 bytes inside of 24-byte region [0x603005548c30,0x603005548c48)
freed by thread T201 here:
    #0 0x55f28e51680d in operator delete(void*) (/home/work/unlimit_teamcity/TeamCity/Agents/20250708205944agent_172.16.0.48_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x3975a80d) (BuildId: 8b6ba6101e736655)
    #1 0x55f28e8c3ce0 in std::__cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::pop_front() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:1198:15
    #2 0x55f28e8c3ce0 in doris::io::LRUQueueRecorder::replay_queue_event(doris::io::FileCacheType) /root/doris/be/src/io/cache/lru_queue_recorder.cpp:41:19
    #3 0x55f28e82d620 in doris::io::BlockFileCache::run_background_lru_log_replay() /root/doris/be/src/io/cache/block_file_cache.cpp:2242:24
    apache#4 0x55f2cdc2720f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc+-v3/src/c11/../../../../../libstdc-v3/src/c+11/thread.cc:82:18

previously allocated by thread T607 (CumuCompactionT) here:
    #0 0x55f28e515fad in operator new(unsigned long) (/home/work/unlimit_teamcity/TeamCity/Agents/20250708205944agent_172.16.0.48_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x39759fad) (BuildId: 8b6ba6101e736655)
    #1 0x55f28e8c660d in __gnu_cxx::new_allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::allocate(unsigned long, void const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:121:27
    #2 0x55f28e8c660d in std::allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::allocate(unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:173:32
    #3 0x55f28e8c660d in std::allocator_traits<std::allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>>::allocate(std::allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>&, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:460:20
    apache#4 0x55f28e8c660d in std::__cxx11::_List_base<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::_M_get_node() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:442:16
    apache#5 0x55f28e8c660d in std::List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>* std::_cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::_M_create_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>(std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:634:21
    apache#6 0x55f28e8c660d in void std::__cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::_M_insert<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>(std::_List_iterator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>, std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:1911:18
    apache#7 0x55f28e8c3522 in std::__cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::push_back(std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:1217:15
    apache#8 0x55f28e8c3522 in doris::io::LRUQueueRecorder::record_queue_event(doris::io::FileCacheType, doris::io::CacheLRULogType, doris::io::UInt128Wrapper, unsigned long, unsigned long) /root/doris/be/src/io/cache/lru_queue_recorder.cpp:29:15
    apache#9 0x55f28e82f09b in doris::io::BlockFileCache::use_cell(doris::io::BlockFileCache::FileBlockCell const&, std::__cxx11::list<std::shared_ptr<doris::io::FileBlock>, std::allocator<std::shared_ptr<doris::io::FileBlock>>>*, bool, std::lock_guard<std::mutex>&) /root/doris/be/src/io/cache/block_file_cache.cpp:380:20
    apache#10 0x55f28e833d1b in doris::io::BlockFileCache::get_impl[abi:cxx11](doris::io::UInt128Wrapper const&, doris::io::CacheContext const&, doris::io::FileBlock::Range const&, std::lock_guard<std::mutex>&) /root/doris/be/src/io/cache/block_file_cache.cpp:572:13
    apache#11 0x55f28e83b4ef in doris::io::BlockFileCache::get_or_set(doris::io::UInt128Wrapper const&, unsigned long, unsigned long, doris::io::CacheContext&) /root/doris/be/src/io/cache/block_file_cache.cpp:762:27
    apache#12 0x55f28e7ffcee in doris::io::CachedRemoteFileReader::read_at_impl(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) /root/doris/be/src/io/cache/cached_remote_file_reader.cpp:191:21
    apache#13 0x55f28e7f8017 in doris::io::FileReader::read_at(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) /root/doris/be/src/io/fs/file_reader.cpp:34:17
```

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [x] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

---------

Signed-off-by: zhengyu <[email protected]>
kaka11chen pushed a commit that referenced this pull request Jul 22, 2025
@Gabriel39
[fix](cloud) Fix lifecycle in lambda function (apache#53605)
9a6a412 
==15185==ERROR: AddressSanitizer: heap-use-after-free on address
0x7da44ced96d0 at pc 0x55795dac5f46 bp 0x7bc26b9a3e50 sp 0x7bc26b9a3e48
READ of size 8 at 0x7da44ced96d0 thread T1357
#0 0x55795dac5f45 in std::_Function_base::_M_empty() const
/usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:249:37
#1 0x55795dac5f45 in std::function<doris::Status ()>::operator()() const
/usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:591:6
#2 0x55795dac5f45 in
doris::cloud::bthread_fork_join(std::vector<std::function<doris::Status
()>, std::allocator<std::function<doris::Status ()>>> const&,
int)::$_0::operator()() const
/root/doris/be/src/cloud/cloud_meta_mgr.cpp:106:23
#3 0x55795dac5f45 in void std::_invoke_impl<void,
doris::cloud::bthread_fork_join(std::vector<std::function<doris::Status
()>, std::allocator<std::function<doris::Status ()>>> const&,
int)::$_0&>(std::_invoke_other,
doris::cloud::bthread_fork_join(std::vector<std::function<doris::Status
()>, std::allocator<std::function<doris::Status ()>>> const&,
int)::$_0&)
/usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14
apache#4 0x55795dac5f45 in std::enable_if<is_invocable_r_v<void,
doris::cloud::bthread_fork_join(std::vector<std::function<doris::Status
()>, std::allocator<std::function<doris::Status ()>>> const&,
int)::$0&>, void>::type std::_invoke_r<void,
doris::cloud::bthread_fork_join(std::vector<std::function<doris::Status
()>, std::allocator<std::function<doris::Status ()>>> const&,
int)::$_0&>(doris::cloud::bthread_fork_join(std::vector<std::function<doris::Status
()>, std::allocator<std::function<doris::Status ()>>> const&,
int)::$_0&)
/usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:113:2
apache#5 0x55795dac5f45 in std::_Function_handler<void (),
doris::cloud::bthread_fork_join(std::vector<std::function<doris::Status
()>, std::allocator<std::function<doris::Status ()>>> const&,
int)::$_0>::_M_invoke(std::_Any_data const&)
/usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9
apache#6 0x55795da78d1d in std::function<void ()>::operator()() const
/usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9
apache#7 0x55795da78d1d in doris::cloud::run_bthread_work(void*)
/root/doris/be/src/cloud/cloud_meta_mgr.cpp:74:5
apache#8 0x55795f3da334 in bthread::TaskGroup::task_runner(long)
(/home/work/unlimit_teamcity/TeamCity/Agents/20250718121603agent_172.16.0.202_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x50a4e334)
apache#9 0x55795f3ccc80 in bthread_make_fcontext
(/home/work/unlimit_teamcity/TeamCity/Agents/20250718121603agent_172.16.0.202_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x50a40c80)

0x7da44ced96d0 is located 2640 bytes inside of 2688-byte region
[0x7da44ced8c80,0x7da44ced9700)
freed by thread T1431 here:
#0 0x5579380e1f12 in operator delete(void*, unsigned long)
(/home/work/unlimit_teamcity/TeamCity/Agents/20250718121603agent_172.16.0.202_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x29755f12)

previously allocated by thread T795 (brpc_light) here:
#0 0x5579380e12ad in operator new(unsigned long)
(/home/work/unlimit_teamcity/TeamCity/Agents/20250718121603agent_172.16.0.202_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x297552ad)
#1 0x55795c7b57ab in std::__new_allocator<std::function<doris::Status
()>>::allocate(unsigned long, void const*)
/usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/new_allocator.h:151:27
#2 0x55795c7b57ab in std::allocator<std::function<doris::Status
()>>::allocate(unsigned long)
/usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/allocator.h:203:32
#3 0x55795c7b57ab in
std::allocator_traits<std::allocator<std::function<doris::Status
()>>>::allocate(std::allocator<std::function<doris::Status ()>>&,
unsigned long)
/usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/alloc_traits.h:614:20
apache#4 0x55795c7b57ab in std::_Vector_base<std::function<doris::Status ()>,
std::allocator<std::function<doris::Status ()>>>::_M_allocate(unsigned
long)
/usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_vector.h:387:20
apache#5 0x55795c7b57ab in std::vector<std::function<doris::Status ()>,
std::allocator<std::function<doris::Status ()>>>::reserve(unsigned long)
/usr/local/ldb-toolchain-v0.25/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/vector.tcc:79:22
apache#6 0x55795c7787f6 in
doris::pipeline::OlapScanLocalState::_sync_cloud_tablets(doris::RuntimeState*)
/root/doris/be/src/pipeline/exec/olap_scan_operator.cpp:466:20
apache#7 0x55795c777e7d in
doris::pipeline::OlapScanLocalState::init(doris::RuntimeState*,
doris::pipeline::LocalStateInfo&)
/root/doris/be/src/pipeline/exec/olap_scan_operator.cpp:51:5
apache#8 0x55795b459bf9 in
doris::pipeline::OperatorX<doris::pipeline::OlapScanLocalState>::setup_local_state(doris::RuntimeState*,
doris::pipeline::LocalStateInfo&)
/root/doris/be/src/pipeline/exec/operator.cpp:486:5
apache#9 0x55795d9c9a36 in
doris::pipeline::PipelineTask::prepare(std::vector<doris::TScanRangeParams,
std::allocator<doris::TScanRangeParams>> const&, int, doris::TDataSink
const&) /root/doris/be/src/pipeline/pipeline_task.cpp:118:9
kaka11chen pushed a commit that referenced this pull request Aug 1, 2025
@zhangstar333
[Bug](function) fix to_ipv6 cause stack-buffer-overflow error (apache…
78b0c60 
…#53713)

### What problem does this PR solve?
Problem Summary:

memcpy(dst, &result, sizeof(result));
when use memcpy, it's size if sizeof(result), so use int64 maybe
overflow of dst

```
==3524968==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f18404e1d73 at pc 0x559e2c162b01 bp 0x7f18439e5dc0 sp 0x7f18439e5db8
WRITE of size 8 at 0x7f18404e1d73 thread T1265 (brpc_light)
    #0 0x559e2c162b00 in bool doris::vectorized::parse_ipv4<char const, doris::vectorized::parse_ipv6(char const*, char const*, unsigned char*)::'lambda'()>(char const*&, doris::vectorized::parse_ipv6(char const*, char const*, unsigned char*)::'lambda'(), unsigned char*, long) /mnt/disk8/zhangsida/doris/be/src/vec/common/format_ip.h:165:5
    #1 0x559e2c161eb3 in bool doris::vectorized::parse_ipv6<char const, doris::vectorized::parse_ipv6(char const*, char const*, unsigned char*)::'lambda'()>(char const*&, doris::vectorized::parse_ipv6(char const*, char const*, unsigned char*)::'lambda'(), unsigned char*, int) /mnt/disk8/zhangsida/doris/be/src/vec/common/format_ip.h:416:18
    #2 0x559e2c160c44 in doris::vectorized::parse_ipv6(char const*, char const*, unsigned char*) /mnt/disk8/zhangsida/doris/be/src/vec/common/format_ip.h:467:9
    #3 0x559e2c160c44 in doris::vectorized::parse_ipv6_whole(char const*, char const*, unsigned char*) /mnt/disk8/zhangsida/doris/be/src/vec/common/format_ip.h:475:12
    apache#4 0x559e2c160c44 in doris::IPv6Value::from_string(unsigned __int128&, char const*, unsigned long) /mnt/disk8/zhangsida/doris/be/src/vec/runtime/ipv6_value.h:71:16
    apache#5 0x559e4fdb05f3 in doris::vectorized::FunctionToIP<(doris::vectorized::IPConvertExceptionMode)0, (doris::PrimitiveType)37>::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const /mnt/disk8/zhangsida/doris/be/src/vec/functions/function_ip.h:1180:21
    apache#6 0x559e4c233b1e in doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) const /mnt/disk8/zhangsida/doris/be/src/vec/functions/function.h:447:26
    apache#7 0x559e4eebcef3 in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /mnt/disk8/zhangsida/doris/be/src/vec/functions/function.cpp
    apache#8 0x559e4eeb68c4 in doris::vectorized::PreparedFunctionImpl::default_implementation_for_constant_arguments(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool, bool*) const /mnt/disk8/zhangsida/doris/be/src/vec/functions/function.cpp:168:5
    apache#9 0x559e4eeb8fc4 in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long, bool) const /mnt/disk8/zhangsida/doris/be/src/vec/functions/function.cpp:237:5
```
kaka11chen pushed a commit that referenced this pull request Aug 1, 2025
@BiteTheDDDDt
[Chore](build) set NDEBUG_SANITIZER on asan mode (apache#54028)
e5bddd0 
### What problem does this PR solve?
set NDEBUG_SANITIZER on asan mode

```
  AddressSanitizer:DEADLYSIGNAL
18:05:08   =================================================================
18:05:08   ==17515==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x563cea4a925f bp 0x7ffedd655da0 sp 0x7ffedd655d90 T0)
18:05:08   ==17515==The signal is caused by a READ memory access.
18:05:08   ==17515==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
18:05:17       #0 0x563cea4a925f in absl::lts_20250512::container_internal::btree_node<absl::lts_20250512::container_internal::map_params<S2CellId, S2ShapeIndexCell*, std::less<S2CellId>, std::allocator<std::pair<S2CellId const, S2ShapeIndexCell*>>, 256, false>>::parent() const /var/local/thirdparty/installed/include/absl/container/internal/btree.h:714:39
18:05:17       #1 0x563cea4a9216 in absl::lts_20250512::container_internal::btree_node<absl::lts_20250512::container_internal::map_params<S2CellId, S2ShapeIndexCell*, std::less<S2CellId>, std::allocator<std::pair<S2CellId const, S2ShapeIndexCell*>>, 256, false>>::is_root() const /var/local/thirdparty/installed/include/absl/container/internal/btree.h:718:33
18:05:17       #2 0x563cea4a9fae in absl::lts_20250512::container_internal::btree_node<absl::lts_20250512::container_internal::map_params<S2CellId, S2ShapeIndexCell*, std::less<S2CellId>, std::allocator<std::pair<S2CellId const, S2ShapeIndexCell*>>, 256, false>>::get_root_generation() const /var/local/thirdparty/installed/include/absl/container/internal/btree.h:729:19
18:05:17       #3 0x563cea4a9b86 in absl::lts_20250512::container_internal::btree_node<absl::lts_20250512::container_internal::map_params<S2CellId, S2ShapeIndexCell*, std::less<S2CellId>, std::allocator<std::pair<S2CellId const, S2ShapeIndexCell*>>, 256, false>>::generation() const /var/local/thirdparty/installed/include/absl/container/internal/btree.h:735:41
18:05:17       apache#4 0x563cea4a90bb in void absl::lts_20250512::container_internal::btree_iterator_generation_info_enabled::assert_valid_generation<absl::lts_20250512::container_internal::btree_node<absl::lts_20250512::container_internal::map_params<S2CellId, S2ShapeIndexCell*, std::less<S2CellId>, std::allocator<std::pair<S2CellId const, S2ShapeIndexCell*>>, 256, false>>>(absl::lts_20250512::container_internal::btree_node<absl::lts_20250512::container_internal::map_params<S2CellId, S2ShapeIndexCell*, std::less<S2CellId>, std::allocator<std::pair<S2CellId const, S2ShapeIndexCell*>>, 256, false>> const*) const /var/local/thirdparty/installed/include/absl/container/internal/btree.h:1077:34
18:05:17       apache#5 0x563cea4b9771 in absl::lts_20250512::container_internal::btree_iterator<absl::lts_20250512::container_internal::btree_node<absl::lts_20250512::container_internal::map_params<S2CellId, S2ShapeIndexCell*, std::less<S2CellId>, std::allocator<std::pair<S2CellId const, S2ShapeIndexCell*>>, 256, false>> const, std::pair<S2CellId const, S2ShapeIndexCell*> const&, std::pair<S2CellId const, S2ShapeIndexCell*> const*>::increment() /var/local/thirdparty/installed/include/absl/container/internal/btree.h:1296:5
18:05:17       apache#6 0x563cea4b94fa in absl::lts_20250512::container_internal::btree_iterator<absl::lts_20250512::container_internal::btree_node<absl::lts_20250512::container_internal::map_params<S2CellId, S2ShapeIndexCell*, std::less<S2CellId>, std::allocator<std::pair<S2CellId const, S2ShapeIndexCell*>>, 256, false>> const, std::pair<S2CellId const, S2ShapeIndexCell*> const&, std::pair<S2CellId const, S2ShapeIndexCell*> const*>::operator++() /var/local/thirdparty/installed/include/absl/container/internal/btree.h:1215:5
18:05:17       apache#7 0x563cea4a7dbd in MutableS2ShapeIndex::Iterator::Next() /var/local/thirdparty/installed/include/s2/mutable_s2shape_index.h:720:3
18:05:17       apache#8 0x563d0901dada  (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4b466ada)
18:05:17       apache#9 0x563d0901eeac in s2shapeutil::FindSelfIntersection(S2ShapeIndex const&, S2Error*) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4b467eac)
18:05:17       apache#10 0x563d08fa4fbf in S2Loop::IsValid() const (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4b3edfbf)
18:05:17       apache#11 0x563cea498899 in doris::to_s2loop(doris::GeoCoordinateList const&, std::unique_ptr<S2Loop, std::default_delete<S2Loop>>*) /root/doris/be/src/geo/geo_types.cpp:340:19
18:05:17       apache#12 0x563cea4891cf in doris::to_s2polygon(doris::GeoCoordinateListList const&, std::unique_ptr<S2Polygon, std::default_delete<S2Polygon>>*) /root/doris/be/src/geo/geo_types.cpp:374:20
18:05:17       apache#13 0x563cea488ee4 in doris::GeoPolygon::from_coords(doris::GeoCoordinateListList const&) /root/doris/be/src/geo/geo_types.cpp:786:12
18:05:17       apache#14 0x563cea46d9be in wkt_parse(WktParseContext*) /root/doris/be/src/geo/wkt_yacc.y:130:38
18:05:17       apache#15 0x563cea45fefc in doris::WktParse::parse_wkt(char const*, unsigned long, doris::GeoShape**) /root/doris/be/src/geo/wkt_parse.cpp:37:16
18:05:17       apache#16 0x563cea4814c2 in doris::GeoShape::from_wkt(char const*, unsigned long, doris::GeoParseStatus*) /root/doris/be/src/geo/geo_types.cpp:407:15
18:05:17       apache#17 0x563cda7df4ed in doris::GeoTypesTest_point_intersects_Test::TestBody() /root/doris/be/test/geo/geo_types_test.cpp:108:13
18:05:17       apache#18 0x563d07bf5023 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a03e023)
18:05:17       apache#19 0x563d07be4245 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a02d245)
18:05:17       apache#20 0x563d07bbed66 in testing::Test::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a007d66)
18:05:17       apache#21 0x563d07bbfa4c in testing::TestInfo::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a008a4c)
18:05:17       apache#22 0x563d07bc01df in testing::TestSuite::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a0091df)
18:05:17       apache#23 0x563d07bd199b in testing::internal::UnitTestImpl::RunAllTests() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a01a99b)
18:05:17       apache#24 0x563d07bf6e53 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a03fe53)
18:05:17       apache#25 0x563d07be6465 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a02f465)
18:05:17       apache#26 0x563d07bd152a in testing::UnitTest::Run() (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x4a01a52a)
18:05:17       apache#27 0x563cdd9d3322 in RUN_ALL_TESTS() /var/local/thirdparty/installed/include/gtest/gtest.h:2490:46
18:05:17       apache#28 0x563cdd9c689e in main /root/doris/be/test/testutil/run_all_tests.cpp:111:19
18:05:17       apache#29 0x7f73ae7a4554 in __libc_start_main (/lib64/libc.so.6+0x22554) (BuildId: 1a8fb61bb4614a483833d5334202ab50edda2a25)
18:05:17       apache#30 0x563cd9b0e029 in _start (/root/doris/be/ut_build_ASAN/test/doris_be_test+0x1bf57029)
```
kaka11chen pushed a commit that referenced this pull request Aug 17, 2025
@Mryange
[env](ubsan) Avoid some errors reported by UBSan when compiling with …
4379e8f 
…O0 (apache#54672)

### What problem does this PR solve?

Because on the pipeline we use O1 for compilation, but some UBSan errors
only appear locally when compiling with O0, for example:
```
0x7ba502eb3a01: note: pointer points here
 02 00 00  00 30 52 8c 02 95 7b 00  00 03 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00
              ^ 
    #0 0x55cbe9f9acd2 in doris::Slice::compare(doris::Slice const&) const /mnt/disk8/yanxuecheng/tmp-doris/be/src/util/slice.h
    #1 0x55cbede3405c in doris::FieldTypeTraits<(doris::FieldType)13>::cmp(void const*, void const*) /mnt/disk8/yanxuecheng/tmp-doris/be/src/olap/types.h:1347:25
    #2 0x55cbe9e98dae in doris::ScalarTypeInfo::cmp(void const*, void const*) const /mnt/disk8/yanxuecheng/tmp-doris/be/src/olap/types.h:104:74
    #3 0x55cbeee73cf0 in int doris::Field::compare_cell<doris::RowCursorCell, doris::RowCursorCell>(doris::RowCursorCell const&, doris::RowCursorCell const&) const /mnt/disk8/yanxuecheng/tmp-doris/be/src/olap/field.h:120:41
    apache#4 0x55cbeee6a030 in doris::compare_row_key(doris::RowCursor const&, doris::RowCursor const&) /mnt/disk8/yanxuecheng/tmp-doris/be/src/olap/tablet_reader.h:75:47
    apache#5 0x55cbeee43237 in doris::TabletReader::_capture_rs_readers(doris::TabletReader::ReaderParams const&) /mnt/disk8/yanxuecheng/tmp-doris/be/src/olap/tablet_reader.cpp:188:17
    apache#6 0x55cc1d5f7985 in doris::vectorized::BlockReader::_init_collect_iter(doris::TabletReader::ReaderParams const&) /mnt/disk8/yanxuecheng/tmp-doris/be/src/vec/olap/block_reader.cpp:106:16
    apache#7 0x55cc1d5fd004 in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /mnt/disk8/yanxuecheng/tmp-doris/be/src/vec/olap/block_reader.cpp:226:19
    apache#8 0x55cc15e09f53 in doris::vectorized::OlapScanner::open(doris::RuntimeState*) /mnt/disk8/yanxuecheng/tmp-doris/be/src/vec/exec/scan/olap_scanner.cpp:260:32
    apache#9 0x55cc15d7d89d in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /mnt/disk8/yanxuecheng/tmp-doris/be/src/vec/exec/scan/scanner_scheduler.cpp:182:5
    apache#10 0x55cc15d88ce6 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /mnt/disk8/yanxuecheng/tmp-doris/be/src/vec/exec/scan/scanner_scheduler.cpp:96:17
    apache#11 0x55cc15d884b1 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()::operator()() const /mnt/disk8/yanxuecheng/tmp-doris/be/src/vec/exec/scan/scanner_scheduler.cpp:95:27
    apache#12 0x55cc15d8829c in bool std::__invoke_impl<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&) /mnt/disk6/common/ldb_toolchain_taipan/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14
    apache#13 0x55cc15d8819c in std::enable_if<is_invocable_r_v<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&>, bool>::type std::__invoke_r<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()&) /mnt/disk6/common/ldb_toolchain_taipan/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116:9
    apache#14 0x55cc15d87c94 in std::_Function_handler<bool (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /mnt/disk6/common/ldb_toolchain_taipan/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9
    apache#15 0x55cbe9632efd in std::function<bool ()>::operator()() const /mnt/disk6/common/ldb_toolchain_taipan/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9
    apache#16 0x55cc15d87314 in doris::vectorized::ScannerSplitRunner::process_for(std::chrono::duration<long, std::ratio<1l, 1000000000l>>) /mnt/disk8/yanxuecheng/tmp-doris/be/src/vec/exec/scan/scanner_scheduler.cpp:407:25

```
kaka11chen pushed a commit that referenced this pull request Aug 28, 2025
@RoanHeNaN
[fix](ubsan) Fix ubsan from glibc-compatibility/musl/logf.c (apache#5…
e371014 
…4737)

Related PR: apache#54598

before 54598, logf is using impl from toolchain, after 54598, logf is
using impl from doris glibc-compatibility

Analysis from gpt5:
```
the 0x1ff << 23 part is evaluated in type int.

0x1ff = 511 decimal.

Left-shifting it 23 bits gives 0x3ff800000 in math, which is way bigger than INT_MAX (on 32-bit signed int, any shift producing a bit in the sign position or beyond is undefined behavior in C).

Even if ix and tmp are larger types, that constant expression is first computed as an int before any promotions, triggering the UB warning from UBSan.
```


Fix the problem below
```text
#0 0x5632a618a17 in logf / root/doris/be/src/glibc-compatibility/musl/logf. c: 56:25
#1 0x5632a59fec26 in std:: log(float) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/cmath: 334:12
#2 0x5632a59fcee3 in lucene:: index: :MultiLevelSkipListWriter::MultiLevelSkipListWriter(int, int, int) /root/doris/contrib/clucene/src/core/CLucene/index/SkipListWriter.cpp:70:56
#3 0x5632a59fe497 in lucene:: index: :DefaultSkipListWriter: :DefaultSkipListWriter(int, int, int, lucene::store::IndexOutput*, lucene::store: :IndexOutput*) /root/doris/contrib/clucene/src/core/CLucene/index/SkipListWriter. cpp:175:11
+ echo 'cp -r /mnt/ssd01/pipline/0penSourceDoris/clusterEnv/PO//Cluster0/fe/bin /home/work/pipline/backup_center/54276_0b92341b39685a8528a1d769885cefe95197e7fb_p0/fe/' + cp -r /mnt/ssd01/pipline/0penSourceDoris/clusterEnv/PO//Cluster0/fe/conf /home/work/pipline/backup_center/54276_0b92341b39685a8528a1d769885cefe95197e7fb_p0/fe/
apache#4 0x5632a5affdb in lucene:: index:: SDocumentsWriter<char>: :writeSegment(std: :vector<std::__cxx11::basic_string<char, std: :char_traits<char>, std: :allocator<char> >, std: :allocator<std::_cxx11::basic_string<char, std:: char_traits<chars
apache#5 0x5632a5af009e in lucene:: index::DocumentsWriter<char>:: flush(bool) /root/doris/contrib/clucene/src/core/Lucene/index/DocumentWriter. cpp: 1340:9
apache#6 0x5632a5a41842 in lucene:: index:: IndexWriter:: doFlush(bool) /root/doris/contrib/clucene/src/core/CLucene/index/IndexWriter. cpp:2262:58
apache#7 0x5632a5a1f59b
in lucene:: index:: IndexWriter:: flush(bool, bool) / root/doris/contrib/clucene/src/core/CLucene/index/IndexWriter.cpp:2154:9
apache#8 0x5632a5a1dd09 in lucene:: index:: IndexWriter:: closeInternal(bool) /root/doris/contrib/clucene/src/core/CLucene/index/IndexWriter. cpp: 546:9
apache#9 0x5632a5a1d16e in lucene:: index:: IndexWriter:: close(bool) / root/doris/contrib/clucene/src/core/CLucene/index/IndexWriter.cpp: 519:9
apache#10 0x563282c28c76 in _ZN5doris10segment_v213finally_closeISt10unique_ptrIN6lucene5index11IndexWriterESt14default_deleteIS5_EEQ8HasCloseIT_EEEVRS9_RNS0_12ErrorContextE /root/doris/be/src/olap/rowset/segment_v2/inverted_index_common.h:56
apache#11
0x563282c3b9c2 in doris::segment_v2: :InvertedIndexColumnWriter<(doris::FieldType)17>::finish) /root/doris/be/src/olap/rowset/segment_v2/inverted
index_writer. cpp: 687:9
apache#12 0x563282b9fble in doris:: segment_v2::ArrayColumnWriter::write_inverted_index() /root/doris/be/src/olap/rowset/segment_v2/column_writer.cpp: 940:41
apache#13 0x56328248ec in doris:: segment_v2::VerticalSegmentWriter::_write_inverted_index() /root/doris/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp: 1296:9
+ echo cp -r /mnt/ssd01/pipline/0penSourceDoris/clusterEnv/P0//Cluster0/fe/log /home/work/pipline/backup_center/54276_0b92341b39685a8528a1d769885cefe95197e7fb_p0/fe/'
apache#14 0x5632824d120 in doris::segment_v2::VerticalSegmentWriter::finalize_columns_indexunsigned long*) / root/doris/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp: 1204:5
+ cp
-r /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0//Cluster0/fe/log /home/work/pipline/backup_center/54276_0b92341b39685a8528a1d769885cefe95197e7fb_p0/fe/
apache#15 0x56328251046 in doris: :segment_v2: :VerticalSegmentWriter:: finalize(unsigned long*, unsigned long*) / root/doris/be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp: 1253:5
apache#16
0x563282519a5a in doris: SegmentFlusher: :_flush_segment_writer(std: :unique_ptr<doris::segment_v2::VerticalSegmentWriter, std: :default_delete<doris::segment_v2: :VerticalSegmentWriter> >&, std: :shared_ptr<doris:: TabletSchema>, long*)
apache#17 0x56328251642f in doris::SegmentFlusher::flush_single_block(doris::vectorized::Block const*, int, long*) /root/doris/be/src/olap/rowset/segment_creator.cpp:77:9
apache#18 0x563282521151 in doris::SegmentCreator::flush_single_block(doris::vectorized::Block const*, int,
long*) /root/doris/be/src/olap/rowset/segment_creator.cpp: 389:5
apache#19
0x56329fe59dc4 in doris::BetaRowsetWriterV2:: flush_memtable(doris::vectorized::Block*, int,
long*) /root/doris/be/src/olap/rowset/beta_rowset_writer_v2.cpp: 109:9
apache#20
0x56328215778 in doris:: FlushToken: :_do_flush_memtable(doris::MemTable*, int, long*) / root/doris/be/src/olap/memtable_flush_executor.cpp:210:9
apache#21
apache#22
0x56328217962 in doris:: FlushToken: :_flush_memtable(std: :shared_ptr<doris::MemTable>, int, long) / root/doris/be/src/olap/memtable_flush_executor. cpp:265:16
0x5632821f1f7d in doris::MemtableFlushTask:: run() /root/doris/be/src/olap/memtable_flush_executor.cpp:63:20
apache#23
0x56328437d33 in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:614:24
apache#24 0x56328435аабс in std:: function<void ()›::operator()() const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9
apache#25 0x56328435aac in doris::Thread:: supervise_thread(void*) / root/doris/be/src/util/thread. cpp: 461:5
apache#26
0x56327fc4fd26 in asan_thread_start(void*) (/mnt/ssd01/pipline/0penSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x28da4d26)
apache#27 0x7fdd62967608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c: 477:8
apache#28 0x7fdd6287a132 in _clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S: 95
```
kaka11chen pushed a commit that referenced this pull request Sep 1, 2025
@yujun777
[fix](nereids) fix window expression alias reuse expr id (apache#55286)
8706fc1 
### What problem does this PR solve?

For `alias(slotReference as slotReference)`, can replace with
`slotReference` directly.

For example,  for sql `select first_value(b) over() from t`,

Before this pr,  it will have:
LogicalProject[26] ( distinct=false, projects=[first_value(b) over()#2 AS first_value(b) over()#2] )
    |
LogicalWindow

With this pr, it will have:
LogicalProject[26] ( distinct=false, projects=[first_value(b) over()#2] )
   |
LogicalWindow
kaka11chen added a commit that referenced this pull request Sep 2, 2025
@kaka11chen
[fix](reader) Fix tracing file reader nullptr issue in push handler. (a…
9b5f144 
…pache#55346)

### What problem does this PR solve?

Related PR: apache#53729

Problem Summary:

runtime error: member access within null pointer of type
'FileReaderStats' in `TracingFileReader`
```
/home/zcp/repo_center/doris_branch-3.1/doris/be/src/io/fs/tracing_file_reader.h:34:9: runtime error: member access within null pointer of type 'FileReaderStats'
    #0 0x55c028481ace in doris::io::TracingFileReader::read_at_impl(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/io/fs/tracing_file_reader.h:34:9
    #1 0x55c00922a1f7 in doris::io::FileReader::read_at(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/io/fs/file_reader.cpp:34:17
    #2 0x55c028ae8a18 in doris::vectorized::parse_thrift_footer(std::shared_ptr<doris::io::FileReader>, doris::vectorized::FileMetaData**, unsigned long*, doris::io::IOContext*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/vec/exec/format/parquet/parquet_thrift_util.h:44:5
    #3 0x55c028ae8a18 in doris::vectorized::ParquetReader::_open_file() /home/zcp/repo_center/doris_branch-3.1/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:234:21
    apache#4 0x55c028aeb69c in doris::vectorized::ParquetReader::init_reader(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>> const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::variant<doris::ColumnValueRange<(doris::PrimitiveType)3>, doris::ColumnValueRange<(doris::PrimitiveType)4>, doris::ColumnValueRange<(doris::PrimitiveType)5>, doris::ColumnValueRange<(doris::PrimitiveType)6>, doris::ColumnValueRange<(doris::PrimitiveType)7>, doris::ColumnValueRange<(doris::PrimitiveType)36>, doris::ColumnValueRange<(doris::PrimitiveType)37>, doris::ColumnValueRange<(doris::PrimitiveType)15>, doris::ColumnValueRange<(doris::PrimitiveType)10>, doris::ColumnValueRange<(doris::PrimitiveType)23>, doris::ColumnValueRange<(doris::PrimitiveType)11>, doris::ColumnValueRange<(doris::PrimitiveType)25>, doris::ColumnValueRange<(doris::PrimitiveType)12>, doris::ColumnValueRange<(doris::PrimitiveType)26>, doris::ColumnValueRange<(doris::PrimitiveType)20>, doris::ColumnValueRange<(doris::PrimitiveType)2>, doris::ColumnValueRange<(doris::PrimitiveType)19>, doris::ColumnValueRange<(doris::PrimitiveType)28>, doris::ColumnValueRange<(doris::PrimitiveType)29>, doris::ColumnValueRange<(doris::PrimitiveType)30>, doris::ColumnValueRange<(doris::PrimitiveType)35>>, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const, std::variant<doris::ColumnValueRange<(doris::PrimitiveType)3>, doris::ColumnValueRange<(doris::PrimitiveType)4>, doris::ColumnValueRange<(doris::PrimitiveType)5>, doris::ColumnValueRange<(doris::PrimitiveType)6>, doris::ColumnValueRange<(doris::PrimitiveType)7>, doris::ColumnValueRange<(doris::PrimitiveType)36>, doris::ColumnValueRange<(doris::PrimitiveType)37>, doris::ColumnValueRange<(doris::PrimitiveType)15>, doris::ColumnValueRange<(doris::PrimitiveType)10>, doris::ColumnValueRange<(doris::PrimitiveType)23>, doris::ColumnValueRange<(doris::PrimitiveType)11>, doris::ColumnValueRange<(doris::PrimitiveType)25>, doris::ColumnValueRange<(doris::PrimitiveType)12>, doris::ColumnValueRange<(doris::PrimitiveType)26>, doris::ColumnValueRange<(doris::PrimitiveType)20>, doris::ColumnValueRange<(doris::PrimitiveType)2>, doris::ColumnValueRange<(doris::PrimitiveType)19>, doris::ColumnValueRange<(doris::PrimitiveType)28>, doris::ColumnValueRange<(doris::PrimitiveType)29>, doris::ColumnValueRange<(doris::PrimitiveType)30>, doris::ColumnValueRange<(doris::PrimitiveType)35>>>>> const*, std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext>>> const&, doris::TupleDescriptor const*, doris::RowDescriptor const*, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, int, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const, int>>> const*, std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext>>> const*, std::unordered_map<int, std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext>>>, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext>>>>>> const*, std::shared_ptr<doris::vectorized::TableSchemaChangeHelper::Node>, bool) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:318:5
    apache#5 0x55c00cceda85 in doris::PushBrokerReader::_get_next_reader() /home/zcp/repo_center/doris_branch-3.1/doris/be/src/olap/push_handler.cpp:671:39
    apache#6 0x55c00cce72df in doris::PushBrokerReader::next(doris::vectorized::Block*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/olap/push_handler.cpp:434:9
    apache#7 0x55c00cce27b3 in doris::PushHandler::_convert_v2(std::shared_ptr<doris::Tablet>, std::shared_ptr<doris::Rowset>*, std::shared_ptr<doris::TabletSchema>, doris::PushType) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/olap/push_handler.cpp:293:31
    apache#8 0x55c00ccdd92d in doris::PushHandler::_do_streaming_ingestion(std::shared_ptr<doris::Tablet>, doris::TPushReq const&, doris::PushType, std::vector<doris::TTabletInfo, std::allocator<doris::TTabletInfo>>*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/olap/push_handler.cpp:199:11
    apache#9 0x55c00ccd96f3 in doris::PushHandler::process_streaming_ingestion(std::shared_ptr<doris::Tablet>, doris::TPushReq const&, doris::PushType, std::vector<doris::TTabletInfo, std::allocator<doris::TTabletInfo>>*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/olap/push_handler.cpp:94:11
    apache#10 0x55c00cccff9c in doris::EngineBatchLoadTask::_push(doris::TPushReq const&, std::vector<doris::TTabletInfo, std::allocator<doris::TTabletInfo>>*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/olap/task/engine_batch_load_task.cpp:293:28
    apache#11 0x55c00cccbbc3 in doris::EngineBatchLoadTask::_process() /home/zcp/repo_center/doris_branch-3.1/doris/be/src/olap/task/engine_batch_load_task.cpp:249:18
    apache#12 0x55c00ccc6851 in doris::EngineBatchLoadTask::execute() /home/zcp/repo_center/doris_branch-3.1/doris/be/src/olap/task/engine_batch_load_task.cpp:83:22
    apache#13 0x55c0090293eb in doris::push_callback(doris::StorageEngine&, doris::TAgentTaskRequest const&) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/agent/task_worker_pool.cpp:1880:31
    apache#14 0x55c008ff6554 in doris::PriorTaskWorkerPool::normal_loop() /home/zcp/repo_center/doris_branch-3.1/doris/be/src/agent/task_worker_pool.cpp:687:9
    apache#15 0x55c00db29237 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/util/thread.cpp:498:5
    apache#16 0x7fa0651f2ac2 in start_thread nptl/pthread_create.c:442:8
    apache#17 0x7fa06528484f  misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
```
kaka11chen pushed a commit that referenced this pull request Sep 11, 2025
@hubgeter
branch-3.1: [fix](load)fix ingestion load error case cause be core. (a…
d5577ec 
…pache#55500)

### What problem does this PR solve?
Related PR: apache#45937

Problem Summary:
Fix the error case on ingestion load and the core in parquet reader.

==8898==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62f0020603fc at pc 0x55f634e64ded bp 0x7fba0d03c410 sp 0x7fba0d03bbd8
READ of size 4 at 0x62f0020603fc thread T768 (PUSH-9699)
    #0 0x55f634e64dec in __asan_memcpy (/mnt/hdd01/ci/doris-deploy-branch-3.1-local/be/lib/doris_be+0x39a24dec) (BuildId: 9b04e7f7d3075dac)
    #1 0x55f634eca93f in std::char_traits::copy(char*, char const*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/char_traits.h:409:33
    #2 0x55f634eca93f in std::__cxx11::basic_string, std::allocator>::_S_copy(char*, char const*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:351:4
    #3 0x55f634eca93f in std::__cxx11::basic_string, std::allocator>::_S_copy_chars(char*, char const*, char const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:398:9
    apache#4 0x55f634eca93f in void std::__cxx11::basic_string, std::allocator>::_M_construct(char const*, char const*, std::forward_iterator_tag) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:225:6
    apache#5 0x55f654a4f74d in void std::__cxx11::basic_string, std::allocator>::_M_construct_aux(char const*, char const*, std::__false_type) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:247:11
    apache#6 0x55f654a4f74d in void std::__cxx11::basic_string, std::allocator>::_M_construct(char const*, char const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:266:4
    apache#7 0x55f654a4f74d in std::__cxx11::basic_string, std::allocator>::basic_string(char const*, unsigned long, std::allocator const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:513:9
    apache#8 0x55f654a4f74d in doris::vectorized::parse_thrift_footer(std::shared_ptr, doris::vectorized::FileMetaData**, unsigned long*, doris::io::IOContext*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/vec/exec/format/parquet/parquet_thrift_util.h:55:17
kaka11chen pushed a commit that referenced this pull request Sep 16, 2025
@hubgeter
[fix](load)fix ingestion load error case cause be core. (apache#55906)
bf74dbb 
Related PR: apache#45937
branch-3.1: apache#55500

Problem Summary:
Fix the error case on ingestion load and the core in parquet reader.

```
==8898==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62f0020603fc at pc 0x55f634e64ded bp 0x7fba0d03c410 sp 0x7fba0d03bbd8 READ of size 4 at 0x62f0020603fc thread T768 (PUSH-9699)
    #0 0x55f634e64dec in __asan_memcpy (/mnt/hdd01/ci/doris-deploy-branch-3.1-local/be/lib/doris_be+0x39a24dec) (BuildId: 9b04e7f7d3075dac)
    #1 0x55f634eca93f in std::char_traits::copy(char*, char const*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/char_traits.h:409:33
    #2 0x55f634eca93f in std::__cxx11::basic_string, std::allocator>::_S_copy(char*, char const*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:351:4
    #3 0x55f634eca93f in std::__cxx11::basic_string, std::allocator>::_S_copy_chars(char*, char const*, char const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:398:9
    apache#4 0x55f634eca93f in void std::__cxx11::basic_string, std::allocator>::_M_construct(char const*, char const*, std::forward_iterator_tag) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:225:6
    apache#5 0x55f654a4f74d in void std::__cxx11::basic_string, std::allocator>::_M_construct_aux(char const*, char const*, std::__false_type) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:247:11
    apache#6 0x55f654a4f74d in void std::__cxx11::basic_string, std::allocator>::_M_construct(char const*, char const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:266:4
    apache#7 0x55f654a4f74d in std::__cxx11::basic_string, std::allocator>::basic_string(char const*, unsigned long, std::allocator const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:513:9
    apache#8 0x55f654a4f74d in doris::vectorized::parse_thrift_footer(std::shared_ptr, doris::vectorized::FileMetaData**, unsigned long*, doris::io::IOContext*) /home/zcp/repo_center/doris_branch-3.1/doris/be/src/vec/exec/format/parquet/parquet_thrift_util.h:55:17
```
morningman pushed a commit that referenced this pull request Oct 10, 2025
@bobhan1
[Fix](warmup) Fix coredump in `CloudTablet::complete_rowset_segment_w…
1e738f4 
…armup` due to capture by reference (apache#56395)

### What problem does this PR solve?
introduced in apache#54611

```
(gdb) bt
#0  __GI___pthread_sigmask (how=2, newmask=<optimized out>, oldmask=0x0) at ./nptl/pthread_sigmask.c:43
#1  0x00007f7aa6b5d71e in PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
#2  0x00007f7aa6b5e206 in JVM_handle_linux_signal () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
#3  <signal handler called>
apache#4  0x000055e20050d443 in std::_Hashtable<doris::RowsetId, std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo>, std::allocator<std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo> >, std::__detail::_Select1st, std::equal_to<doris::RowsetId>, std::hash<doris::RowsetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_find_before_node (this=0x7f79e69acde0,
    __bkt=4186920012728959759, __k=..., __code=11200260987994981938) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/hashtable.h:2205
apache#5  std::_Hashtable<doris::RowsetId, std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo>, std::allocator<std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo> >, std::__detail::_Select1st, std::equal_to<doris::RowsetId>, std::hash<doris::RowsetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_locate (this=this@entry=0x7f79e69acde0, __k=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/hashtable.h:2283
apache#6  0x000055e200505a8d in std::_Hashtable<doris::RowsetId, std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo>, std::allocator<std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo> >, std::__detail::_Select1st, std::equal_to<doris::RowsetId>, std::hash<doris::RowsetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::find (this=0x7f79e69acde0, __k=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/hashtable.h:1929
apache#7  std::unordered_map<doris::RowsetId, doris::CloudTablet::RowsetWarmUpInfo, std::hash<doris::RowsetId>, std::equal_to<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo> > >::contains (this=0x7f79e69acde0, __x=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unordered_map.h:999
apache#8  doris::CloudTablet::complete_rowset_segment_warmup (this=<optimized out>, rowset_id=..., status=..., segment_num=1, inverted_idx_num=0) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_tablet.cpp:1692
apache#9  0x000055e2005338ba in doris::CloudWarmUpManager::handle_jobs()::$_0::operator()(doris::Status) const (this=0x7f79e8bebad0, st=...) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_warm_up_manager.cpp:243
apache#10 std::__invoke_impl<void, doris::CloudWarmUpManager::handle_jobs()::$_0&, doris::Status>(std::__invoke_other, doris::CloudWarmUpManager::handle_jobs()::$_0&, doris::Status&&) (__f=..., __args=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63
apache#11 std::__invoke_r<void, doris::CloudWarmUpManager::handle_jobs()::$_0&, doris::Status>(doris::CloudWarmUpManager::handle_jobs()::$_0&, doris::Status&&) (__fn=..., __args=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:113
apache#12 std::_Function_handler<void (doris::Status), doris::CloudWarmUpManager::handle_jobs()::$_0>::_M_invoke(std::_Any_data const&, doris::Status&&) (__functor=..., __args=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292
apache#13 0x000055e200533310 in std::function<void (doris::Status)>::operator()(doris::Status) const (this=0x7f795a3ff1d0, __args=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593
apache#14 doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0::operator()(doris::Status) const (
    this=0x7f79f0710b60, st=...) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_warm_up_manager.cpp:149
apache#15 std::__invoke_impl<void, doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0&, doris::Status>(std::__invoke_other, doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0&, doris::Status&&) (
    __f=..., __args=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63
apache#16 std::__invoke_r<void, doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0&, doris::Status>(doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0&, doris::Status&&) (__fn=..., __args=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:113
apache#17 std::_Function_handler<void (doris::Status), doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0>::_M_invoke(std::_Any_data const&, doris::Status&&) (__functor=..., __args=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292
apache#18 0x000055e200469cb1 in std::function<void (doris::Status)>::operator()(doris::Status) const (this=0x7f796941cee0, __args=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593
apache#19 doris::io::FileCacheBlockDownloader::download_segment_file (this=<optimized out>, meta=...) at /home/zcp/repo_center/doris_master/doris/be/src/io/cache/block_file_cache_downloader.cpp:297
apache#20 0x000055e1fbfcbe85 in doris::ThreadPool::dispatch_thread (this=0x7f7995a1da00) at /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:614
apache#21 0x000055e1fbfc0fac in std::function<void ()>::operator()() const (this=0x7f796941cee0) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593
apache#22 doris::Thread::supervise_thread (arg=0x7f79f13de010) at /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:460
apache#23 0x00007f7aa5bf8ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
apache#24 0x00007f7aa5c8a850 in __closefrom_fallback (from=1685031872, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:45
apache#25 0x0000000000000000 in ?? ()
```

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
morningman pushed a commit that referenced this pull request Oct 10, 2025
@Mryange
[chore](ubsan) fix some ubsan error (apache#55673)
2ce15eb 
```
/home/zcp/repo_center/doris_master/doris/be/src/util/counts.h:127:85: runtime error: signed integer overflow: 147483648 - -2147483648 cannot be represented in type 'int'
    #0 0x558c5a9d58cd in doris::Counts::terminate(double) /home/zcp/repo_center/doris_master/doris/be/src/util/counts.h:127:85
    #1 0x558c5ab0ce99 in doris::vectorized::PercentileState<(doris::PrimitiveType)5>::insert_result_into(doris::vectorized::IColumn&) const /home/zcp/repo_center/doris_master/doris/be/src/vec/aggregate_functions/aggregate_function_percentile.h:430:49
    #2 0x558c5ab0812f in doris::vectorized::AggregateFunctionPercentileArray<(doris::PrimitiveType)5>::insert_result_into(char const*, doris::vectorized::IColumn&) const /home/zcp/repo_center/doris_master/doris/be/src/vec/aggregate_functions/aggregate_function_percentile.h:557:59
    #3 0x558c5ab086a8 in doris::vectorized::IAggregateFunctionHelper>::insert_result_into_vec(std::vector> const&, unsigned long, doris::vectorized::IColumn&, unsigned long) const /home/zcp/repo_center/doris_master/doris/be/src/vec/aggregate_functions/aggregate_function.h:393:22
    apache#4 0x558c68af8445 in _ZZN5doris8pipeline13AggLocalState31_get_with_serialized_key_resultEPNS_12RuntimeStateEPNS_10vectorized5BlockEPbENK3$_1clINS4_15MethodKeysFixedI9PHHashMapIN4wide7integerILm256EjEEPc9HashCRC32ISE_EEEEEEvRT_ /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/aggregation_source_operator.cpp:287:67
    apache#5 0x558c68af8445 in void std::__invoke_impl, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>&>(std::__invoke_other, doris::vectorized::Overload&&, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14
    apache#6 0x558c68af8445 in std::__invoke_result, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>&>::type std::__invoke, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>&>(doris::vectorized::Overload&&, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:98:14
    apache#7 0x558c68af8445 in std::__detail::__variant::__gen_vtable_impl (*)(doris::vectorized::Overload&&, std::variant>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&)>, std::integer_sequence>::__visit_invoke(doris::vectorized::Overload&&, std::variant>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/variant:1055:11
    apache#8 0x558c68a85790 in decltype(auto) std::__do_visit, doris::vectorized::Overload, std::variant>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&>(doris::vectorized::Overload&&, std::variant>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/variant:1858:15
    apache#9 0x558c68a85790 in std::invoke_result, std::__conditional>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&>>::type>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&>()))>::type>::type&&>>::type std::visit, std::variant>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&>(doris::vectorized::Overload&&, std::variant>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodOneNumber>>, doris::vectorized::MethodStringNoCache>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber, PHHashMap, char*, HashCRC32>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodOneNumber>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn, doris::vectorized::DataWithNullKey, char*, HashCRC32>>>>>, doris::vectorized::MethodSingleNullableColumn>>>>, doris::vectorized::MethodKeysFixed>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed, char*, HashCRC32>>>, doris::vectorized::MethodKeysFixed>>>&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/variant:1954:13
    apache#10 0x558c68a85790 in doris::pipeline::AggLocalState::_get_with_serialized_key_result(doris::RuntimeState*, doris::vectorized::Block*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/aggregation_source_operator.cpp:251:5
    apache#11 0x558c68b3e3e7 in doris::pipeline::AggLocalState::init(doris::RuntimeState*, doris::pipeline::LocalStateInfo&)::$_2::operator()(doris::RuntimeState*, doris::vectorized::Block*, bool*) const /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/aggregation_source_operator.cpp:75:24
    apache#12 0x558c68b3e3e7 in doris::Status std::__invoke_impl(std::__invoke_other, doris::pipeline::AggLocalState::init(doris::RuntimeState*, doris::pipeline::LocalStateInfo&)::$_2&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14
    apache#13 0x558c68b3e3e7 in std::enable_if, doris::Status>::type std::__invoke_r(doris::pipeline::AggLocalState::init(doris::RuntimeState*, doris::pipeline::LocalStateInfo&)::$_2&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116:9
    apache#14 0x558c68b3e3e7 in std::_Function_handler::_M_invoke(std::_Any_data const&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9
    apache#15 0x558c68a8b819 in std::function::operator()(doris::RuntimeState*, doris::vectorized::Block*, bool*) const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9
    apache#16 0x558c68a8b819 in doris::pipeline::AggSourceOperatorX::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/aggregation_source_operator.cpp:448:5
    apache#17 0x558c68f652cb in doris::pipeline::PartitionedAggSourceOperatorX::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/partitioned_aggregation_source_operator.cpp:167:36
    apache#18 0x558c676b1ddd in doris::pipeline::OperatorXBase::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/operator.cpp:391:18
    apache#19 0x558c69b0493d in doris::pipeline::PipelineTask::execute(bool*) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_task.cpp:521:13
    apache#20 0x558c69b52916 in doris::pipeline::TaskScheduler::_do_work(int) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/task_scheduler.cpp:147:9
    apache#21 0x558c54fa66f2 in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:614:24
    apache#22 0x558c54f83766 in std::function::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9
    apache#23 0x558c54f83766 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:460:5
    apache#24 0x558c507cbd26 in asan_thread_start(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x1f57fd26)
    apache#25 0x7fb97f744ac2 in start_thread nptl/pthread_create.c:442:8
    apache#26 0x7fb97f7d684f  misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/zcp/repo_center/doris_master/doris/be/src/util/counts.h:127:85 
```
morningman pushed a commit that referenced this pull request Oct 10, 2025
@Gabriel39
[fix](debug) Fix illegal access to runtime states (apache#56439)
33e4c00 
### What problem does this PR solve?

introduced by apache#56405

2 changes here:
1. print local runtime filter mgr info inside each task
2. make a pair for tasks with its corresponding runtime state.

==198869==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x7ba3927ac338 at pc 0x559d81e109b9 bp 0x7b7a5b8ad410 sp 0x7b7a5b8ad408
12:38:06  READ of size 8 at 0x7ba3927ac338 thread T1845 (EvHttpServer
[w)
12:38:06  #0 0x559d81e109b8 in std::__uniq_ptr_impl<doris::RuntimeState,
std::default_delete<doris::RuntimeState> >::_M_ptr() const
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:193:51
12:38:06  #1 0x559d81e109b8 in std::unique_ptr<doris::RuntimeState,
std::default_delete<doris::RuntimeState> >::get() const
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:473:21
12:38:06  #2 0x559d81e109b8 in std::unique_ptr<doris::RuntimeState,
std::default_delete<doris::RuntimeState> >::operator->() const
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:466:9
12:38:06  #3 0x559d81e109b8 in
doris::pipeline::PipelineFragmentContext::debug_string[abi:cxx11]()
/root/doris/be/src/pipeline/pipeline_fragment_context.cpp:1941:28
12:38:06  apache#4 0x559d6d65aa2f in
doris::FragmentMgr::dump_pipeline_tasks[abi:cxx11](long)::$_0::operator()(phmap::flat_hash_map<std::pair<doris::TUniqueId,
int>, std::shared_ptr<doris::pipeline::PipelineFragmentContext>,
phmap::Hash<std::pair<doris::TUniqueId, int> >,
phmap::EqualTo<std::pair<doris::TUniqueId, int> >,
std::allocator<std::pair<std::pair<doris::TUniqueId, int> const,
std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&) const
/root/doris/be/src/runtime/fragment_mgr.cpp:807:36
12:38:06  + echo 'cp -r
/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0//Cluster0/fe/bin
/home/work/pipline/backup_center/56368_04d746711764283b1cbafe40b1f849e10a5bcb6e_p0/fe/'
12:38:06  apache#5 0x559d6d65aa2f in doris::Status
std::__invoke_impl<doris::Status,
doris::FragmentMgr::dump_pipeline_tasks[abi:cxx11](long)::$_0&,
phmap::flat_hash_map<std::pair<doris::TUniqueId, int>,
std::shared_ptr<doris::pipeline::PipelineFragmentContext>,
phmap::Hash<std::pair<doris::TUniqueId, int> >,
phmap::EqualTo<std::pair<doris::TUniqueId, int> >,
std::allocator<std::pair<std::pair<doris::TUniqueId, int> const,
std::shared_ptr<doris::pipeline::PipelineFragmentContext> > >
>&>(std::__invoke_other,
doris::FragmentMgr::dump_pipeline_tasks[abi:cxx11](long)::$_0&,
phmap::flat_hash_map<std::pair<doris::TUniqueId, int>,
std::shared_ptr<doris::pipeline::PipelineFragmentContext>,
phmap::Hash<std::pair<doris::TUniqueId, int> >,
phmap::EqualTo<std::pair<doris::TUniqueId, int> >,
std::allocator<std::pair<std::pair<doris::TUniqueId, int> const,
std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&)
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14
12:38:06  + cp -r
/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0//Cluster0/fe/conf
/home/work/pipline/backup_center/56368_04d746711764283b1cbafe40b1f849e10a5bcb6e_p0/fe/
12:38:06  apache#6 0x559d6d65aa2f in
std::enable_if<is_invocable_r_v<doris::Status,
doris::FragmentMgr::dump_pipeline_tasks[abi:cxx11](long)::$_0&,
phmap::flat_hash_map<std::pair<doris::TUniqueId, int>,
std::shared_ptr<doris::pipeline::PipelineFragmentContext>,
phmap::Hash<std::pair<doris::TUniqueId, int> >,
phmap::EqualTo<std::pair<doris::TUniqueId, int> >,
std::allocator<std::pair<std::pair<doris::TUniqueId, int> const,
std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&>,
doris::Status>::type std::__invoke_r<doris::Status,
doris::FragmentMgr::dump_pipeline_tasks[abi:cxx11](long)::$_0&,
phmap::flat_hash_map<std::pair<doris::TUniqueId, int>,
std::shared_ptr<doris::pipeline::PipelineFragmentContext>,
phmap::Hash<std::pair<doris::TUniqueId, int> >,
phmap::EqualTo<std::pair<doris::TUniqueId, int> >,
std::allocator<std::pair<std::pair<doris::TUniqueId, int> const,
std::shared_ptr<doris::pipeline::PipelineFragmentContext> > >
>&>(doris::FragmentMgr::dump_pipeline_tasks[abi:cxx11](long)::$_0&,
phmap::flat_hash_map<std::pair<doris::TUniqueId, int>,
std::shared_ptr<doris::pipeline::PipelineFragmentContext>,
phmap::Hash<std::pair<doris::TUniqueId, int> >,
phmap::EqualTo<std::pair<doris::TUniqueId, int> >,
std::allocator<std::pair<std::pair<doris::TUniqueId, int> const,
std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&)
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116:9
12:38:06  apache#7 0x559d6d65aa2f in std::_Function_handler<doris::Status
(phmap::flat_hash_map<std::pair<doris::TUniqueId, int>,
std::shared_ptr<doris::pipeline::PipelineFragmentContext>,
phmap::Hash<std::pair<doris::TUniqueId, int> >,
phmap::EqualTo<std::pair<doris::TUniqueId, int> >,
std::allocator<std::pair<std::pair<doris::TUniqueId, int> const,
std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&),
doris::FragmentMgr::dump_pipeline_tasks[abi:cxx11](long)::$_0>::_M_invoke(std::_Any_data
const&, phmap::flat_hash_map<std::pair<doris::TUniqueId, int>,
std::shared_ptr<doris::pipeline::PipelineFragmentContext>,
phmap::Hash<std::pair<doris::TUniqueId, int> >,
phmap::EqualTo<std::pair<doris::TUniqueId, int> >,
std::allocator<std::pair<std::pair<doris::TUniqueId, int> const,
std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&)
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9
12:38:06  apache#8 0x559d6d665d37 in std::function<doris::Status
(phmap::flat_hash_map<std::pair<doris::TUniqueId, int>,
std::shared_ptr<doris::pipeline::PipelineFragmentContext>,
phmap::Hash<std::pair<doris::TUniqueId, int> >,
phmap::EqualTo<std::pair<doris::TUniqueId, int> >,
std::allocator<std::pair<std::pair<doris::TUniqueId, int> const,
std::shared_ptr<doris::pipeline::PipelineFragmentContext> > >
>&)>::operator()(phmap::flat_hash_map<std::pair<doris::TUniqueId, int>,
std::shared_ptr<doris::pipeline::PipelineFragmentContext>,
phmap::Hash<std::pair<doris::TUniqueId, int> >,
phmap::EqualTo<std::pair<doris::TUniqueId, int> >,
std::allocator<std::pair<std::pair<doris::TUniqueId, int> const,
std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&) const
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9
12:38:06  apache#9 0x559d6d665d37 in
doris::ConcurrentContextMap<std::pair<doris::TUniqueId, int>,
std::shared_ptr<doris::pipeline::PipelineFragmentContext>,
doris::pipeline::PipelineFragmentContext>::apply(std::function<doris::Status
(phmap::flat_hash_map<std::pair<doris::TUniqueId, int>,
std::shared_ptr<doris::pipeline::PipelineFragmentContext>,
phmap::Hash<std::pair<doris::TUniqueId, int> >,
phmap::EqualTo<std::pair<doris::TUniqueId, int> >,
std::allocator<std::pair<std::pair<doris::TUniqueId, int> const,
std::shared_ptr<doris::pipeline::PipelineFragmentContext> > > >&)>&&)
/root/doris/be/src/runtime/fragment_mgr.h:94:31
12:38:06  apache#10 0x559d6d647f93 in
doris::FragmentMgr::dump_pipeline_tasks[abi:cxx11](long)
/root/doris/be/src/runtime/fragment_mgr.cpp:792:23
12:38:06  apache#11 0x559d7068942b in
doris::LongPipelineTaskAction::handle(doris::HttpRequest*)
/root/doris/be/src/http/action/pipeline_task_action.cpp:54:69
12:38:06  apache#12 0x559d82c776c6
(/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x385416c6)
12:38:06  apache#13 0x559d82c5775f in bufferevent_run_readcb_
/home/runner/work/doris-thirdparty/doris-thirdparty/thirdparty/src/libevent-release-2.1.12-stable/bufferevent.c:251:3
12:38:06  apache#14 0x559d82c79802 in bufferevent_trigger_nolock_
/home/runner/work/doris-thirdparty/doris-thirdparty/thirdparty/src/libevent-release-2.1.12-stable/bufferevent-internal.h:411:3
12:38:06  apache#15 0x559d82c79802 in bufferevent_readcb
/home/runner/work/doris-thirdparty/doris-thirdparty/thirdparty/src/libevent-release-2.1.12-stable/bufferevent_sock.c:214:2
12:38:06  apache#16 0x559d82c60724 in event_persist_closure
/home/runner/work/doris-thirdparty/doris-thirdparty/thirdparty/src/libevent-release-2.1.12-stable/event.c:1623:9
12:38:06  apache#17 0x559d82c60724 in event_process_active_single_queue
/home/runner/work/doris-thirdparty/doris-thirdparty/thirdparty/src/libevent-release-2.1.12-stable/event.c:1682:4
12:38:06  apache#18 0x559d82c60d76 in event_process_active
/home/runner/work/doris-thirdparty/doris-thirdparty/thirdparty/src/libevent-release-2.1.12-stable/event.c:1783:9
12:38:06  + echo 'cp -r
/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0//Cluster0/fe/log
/home/work/pipline/backup_center/56368_04d746711764283b1cbafe40b1f849e10a5bcb6e_p0/fe/'
12:38:06  apache#19 0x559d82c6355f in event_base_loop.constprop.0
/home/runner/work/doris-thirdparty/doris-thirdparty/thirdparty/src/libevent-release-2.1.12-stable/event.c:2006:12
12:38:06  apache#20 0x559d7072b08c in
doris::EvHttpServer::start()::$_0::operator()() const
/root/doris/be/src/http/ev_http_server.cpp:139:13
12:38:06  + cp -r
/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0//Cluster0/fe/log
/home/work/pipline/backup_center/56368_04d746711764283b1cbafe40b1f849e10a5bcb6e_p0/fe/
12:38:06  apache#21 0x559d7072b08c in void std::__invoke_impl<void,
doris::EvHttpServer::start()::$_0&>(std::__invoke_other,
doris::EvHttpServer::start()::$_0&)
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14
12:38:06  apache#22 0x559d7072b08c in std::enable_if<is_invocable_r_v<void,
doris::EvHttpServer::start()::$_0&>, void>::type std::__invoke_r<void,
doris::EvHttpServer::start()::$_0&>(doris::EvHttpServer::start()::$_0&)
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:113:2
12:38:06  apache#23 0x559d7072b08c in std::_Function_handler<void (),
doris::EvHttpServer::start()::$_0>::_M_invoke(std::_Any_data const&)
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9
12:38:06  apache#24 0x559d6e121772 in doris::ThreadPool::dispatch_thread()
/root/doris/be/src/util/threadpool.cpp:614:24
12:38:06  apache#25 0x559d6e0fe7e6 in std::function<void ()>::operator()()
const
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9
12:38:06  apache#26 0x559d6e0fe7e6 in doris::Thread::supervise_thread(void*)
/root/doris/be/src/util/thread.cpp:460:5
12:38:06  apache#27 0x559d6984ed26 in asan_thread_start(void*)
(/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x1f118d26)
12:38:06  apache#28 0x7f83909ee608 in start_thread
/build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
12:38:06  apache#29 0x7f8390901132 in __clone
/build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
morningman pushed a commit that referenced this pull request Oct 10, 2025
@BiteTheDDDDt
[Chore](thrift) add reopen thrift connection on RuntimeQueryStatistic…
9e41e5f 
…sMgr::report_runtime_query_statistics (apache#56744)

try to avoid asan check fail:
```cpp
OpenJDK 64-Bit Server VM warning: Option CriticalJNINatives was deprecated in version 16.0 and will likely be removed in a future release.
Listening for transport dt_socket at address: 8871
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
start BE in local mode
thrift error, reason=write() send(): Broken pipe=================================================================
AddressSanitizer: CHECK failed: asan_thread.cpp:369 "((ptr[0] == kCurrentStackFrameMagic)) != (0)" (0x0, 0x0) (tid=13525)
    #0 0x557a503468e1 in __asan::CheckUnwind() (/mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be+0x1f2a08e1)
    #1 0x557a50361182 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be+0x1f2bb182)
    #2 0x557a5034a97b in __asan::AsanThread::GetStackFrameAccessByAddr(unsigned long, __asan::AsanThread::StackFrameAccess*) (/mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be+0x1f2a497b)
    #3 0x557a502a4a30 in __asan::AddressDescription::AddressDescription(unsigned long, unsigned long, bool) (/mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be+0x1f1fea30)
    apache#4 0x557a502a67a3 in __asan::ErrorGeneric::ErrorGeneric(unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long) (/mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be+0x1f2007a3)
    apache#5 0x557a50343878 in __asan::ReportGenericError(unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned int, bool) (/mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be+0x1f29d878)
    apache#6 0x557a50344f75 in __asan_report_load8 (/mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be+0x1f29ef75)
    apache#7 0x557a5039791e in std::__cxx11::basic_string, std::allocator>::_M_create(unsigned long&, unsigned long) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/basic_string.tcc:148:11
    apache#8 0x557a503984db in void std::__cxx11::basic_string, std::allocator>::_M_construct(char const*, unsigned long) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/basic_string.tcc:291:12
    apache#9 0x557a503984db in std::__cxx11::basic_string, std::allocator>::basic_string(std::__cxx11::basic_string, std::allocator> const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/basic_string.h:617:2
    apache#10 0x557a55b4d77d in std::pair, std::allocator> const, doris::TQueryStatistics>::pair(std::pair, std::allocator> const, doris::TQueryStatistics> const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_pair.h:312:17
    apache#11 0x557a55b4d77d in _ZSt12construct_atISt4pairIKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEN5doris16TQueryStatisticsEEJRKSA_EQaant20is_unbounded_array_vIT_ErqXgsnwcvPvLi0E_SD_pispclsr3stdE7declvalIT0_EEEEEPSD_SG_DpOSF_ /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_construct.h:110:22
    apache#12 0x557a55b4d77d in void std::allocator_traits, std::allocator> const, doris::TQueryStatistics>>>>::construct, std::allocator> const, doris::TQueryStatistics>, std::pair, std::allocator> const, doris::TQueryStatistics> const&>(std::allocator, std::allocator> const, doris::TQueryStatistics>>>&, std::pair, std::allocator> const, doris::TQueryStatistics>*, std::pair, std::allocator> const, doris::TQueryStatistics> const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/alloc_traits.h:676:4
    apache#13 0x557a55b4d77d in void std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_M_construct_node, std::allocator> const, doris::TQueryStatistics> const&>(std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>*, std::pair, std::allocator> const, doris::TQueryStatistics> const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:1237:8
    apache#14 0x557a55b4cfe3 in std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>* std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node::operator(), std::allocator> const, doris::TQueryStatistics> const&>(std::pair, std::allocator> const, doris::TQueryStatistics> const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h
    apache#15 0x557a55b4cbe6 in std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>* std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_M_clone_node, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node>(std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>*, std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:1287:8
    apache#16 0x557a55b4c6f6 in std::_Rb_tree_node_base* std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_M_copy, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node>(std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>*, std::_Rb_tree_node_base*, std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:2546:20
    apache#17 0x557a55b4c7b6 in std::_Rb_tree_node_base* std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_M_copy, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node>(std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>*, std::_Rb_tree_node_base*, std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:2554:3
    apache#18 0x557a55b4c7b6 in std::_Rb_tree_node_base* std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_M_copy, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node>(std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>*, std::_Rb_tree_node_base*, std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:2554:3
    apache#19 0x557a55b4c7b6 in std::_Rb_tree_node_base* std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_M_copy, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node>(std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>*, std::_Rb_tree_node_base*, std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:2554:3
    apache#20 0x557a55b4c7b6 in std::_Rb_tree_node_base* std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_M_copy, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node>(std::_Rb_tree_node, std::allocator> const, doris::TQueryStatistics>>*, std::_Rb_tree_node_base*, std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:2554:3
    apache#21 0x557a55b4c27a in std::_Rb_tree_node_base* std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_M_copy, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node>(std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>> const&, std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::_Reuse_or_alloc_node&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:1523:6
    apache#22 0x557a55740d4f in std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::operator=(std::_Rb_tree, std::allocator>, std::pair, std::allocator> const, doris::TQueryStatistics>, std::_Select1st, std::allocator> const, doris::TQueryStatistics>>, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>> const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_tree.h:2453:18
    apache#23 0x557a55740d4f in std::map, std::allocator>, doris::TQueryStatistics, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>>::operator=(std::map, std::allocator>, doris::TQueryStatistics, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>> const&) /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_map.h:354:37
    apache#24 0x557a55740d4f in doris::TReportWorkloadRuntimeStatusParams::__set_query_statistics_map(std::map, std::allocator>, doris::TQueryStatistics, std::less, std::allocator>>, std::allocator, std::allocator> const, doris::TQueryStatistics>>> const&) /home/zcp/repo_center/doris_branch-4.0/doris/gensrc/build/gen_cpp/FrontendService_types.cpp:4391:30
    apache#25 0x557a5447e13a in doris::RuntimeQueryStatisticsMgr::report_runtime_query_statistics() /home/zcp/repo_center/doris_branch-4.0/doris/be/src/runtime/runtime_query_statistics_mgr.cpp:403:31
    apache#26 0x557a504f1192 in doris::Daemon::report_runtime_query_statistics_thread() /home/zcp/repo_center/doris_branch-4.0/doris/be/src/common/daemon.cpp:457:65
    apache#27 0x557a54bf2eb6 in std::function::operator()() const /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9
    apache#28 0x557a54bf2eb6 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_branch-4.0/doris/be/src/util/thread.cpp:460:5
    apache#29 0x557a50336d26 in asan_thread_start(void*) (/mnt/hdd01/ci/doris-deploy-branch-4.0-local/be/lib/doris_be+0x1f290d26)
    apache#30 0x7f12a399eac2 in start_thread nptl/pthread_create.c:442:8
    apache#31 0x7f12a3a3084f in __closefrom_fallback misc/../sysdeps/unix/sysv/linux/closefrom_fallback.c:45:19
172.20.59.34 last coredump sql: last SQL query not found
```
kaka11chen pushed a commit that referenced this pull request Oct 16, 2025
@gavinchou
[fix](cloud) Fix unhandled error when get_partition_boundaries (apach…
ed6fb62 
…e#56935)

``` 
(gdb) bt
#0  doris::cloud::FdbTxnKv::get_partition_boundaries (this=0x7f11ac6e3950, boundaries=0x7f1130017f20)
    at /home/zcp/repo_center/doris_branch-4.0/doris/cloud/src/meta-store/txn_kv.cpp:212
#1  0x000055e494de9276 in doris::cloud::export_fdb_kv_ranges_details (kv=0x7f11ac6e3950) at /home/zcp/repo_center/doris_branch-4.0/doris/cloud/src/common/metric.cpp:358
#2  doris::cloud::FdbMetricExporter::export_fdb_metrics (txn_kv=0x7f11ac6e3950) at /home/zcp/repo_center/doris_branch-4.0/doris/cloud/src/common/metric.cpp:393
#3  0x000055e494dec77d in doris::cloud::FdbMetricExporter::start()::$_0::operator()() const (this=<optimized out>)
    at /home/zcp/repo_center/doris_branch-4.0/doris/cloud/src/common/metric.cpp:415
apache#4  std::__invoke_impl<void, doris::cloud::FdbMetricExporter::start()::$_0>(std::__invoke_other, doris::cloud::FdbMetricExporter::start()::$_0&&) (__f=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63
apache#5  std::__invoke<doris::cloud::FdbMetricExporter::start()::$_0>(doris::cloud::FdbMetricExporter::start()::$_0&&) (__fn=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:98
apache#6  std::thread::_Invoker<std::tuple<doris::cloud::FdbMetricExporter::start()::$_0> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=<optimized out>)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_thread.h:303
apache#7  std::thread::_Invoker<std::tuple<doris::cloud::FdbMetricExporter::start()::$_0> >::operator()() (this=<optimized out>)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_thread.h:310
apache#8  std::thread::_State_impl<std::thread::_Invoker<std::tuple<doris::cloud::FdbMetricExporter::start()::$_0> > >::_M_run() (this=<optimized out>)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_thread.h:255
apache#9  0x000055e495e547b0 in execute_native_thread_routine ()
apache#10 0x00007f11b1dfeac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
apache#11 0x00007f11b1e90850 in __closefrom_fallback (from=-1266337808, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:45
apache#12 0x0000000000000000 in ?? ()
```
kaka11chen pushed a commit that referenced this pull request Oct 21, 2025
@mrhhsg
[fix](pipeline) Crashing caused by repeated spill operations (apache#…
46d27a2 
…56755)

### What problem does this PR solve?

```text
#0  __GI___pthread_sigmask (how=2, newmask=<optimized out>, oldmask=0x0) at ./nptl/pthread_sigmask.c:43
#1  0x00007f91fd6c471e in PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
#2  0x00007f91fd6c5206 in JVM_handle_linux_signal () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
#3  <signal handler called>
apache#4  __gnu_cxx::__exchange_and_add (__mem=0xe, __val=-1) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/ext/atomicity.h:68
apache#5  __gnu_cxx::__exchange_and_add_dispatch (__mem=0xe, __val=-1) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/ext/atomicity.h:103
apache#6  std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_weak_release (this=0x2) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:211
apache#7  std::__weak_count<(__gnu_cxx::_Lock_policy)2>::~__weak_count (this=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1168
apache#8  std::__weak_ptr<doris::QueryContext, (__gnu_cxx::_Lock_policy)2>::~__weak_ptr (this=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:2003
apache#9  doris::QueryTaskController::revoke_memory()::$_1::~$_1() (this=<optimized out>) at /root/doris/be/src/runtime/workload_management/query_task_controller.cpp:168
apache#10 std::_Function_base::_Base_manager<doris::QueryTaskController::revoke_memory()::$_1>::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) (__victim=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:177
apache#11 std::_Function_base::_Base_manager<doris::QueryTaskController::revoke_memory()::$_1>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) (__dest=..., __op=std::__destroy_functor, __source=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:205
apache#12 std::_Function_handler<void (doris::pipeline::SpillContext*), doris::QueryTaskController::revoke_memory()::$_1>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) (__dest=..., __source=..., __op=std::__destroy_functor)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:284
apache#13 0x000055f37fba60c5 in std::_Function_base::~_Function_base (this=0x7f8dbb0c9fe0) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:246
apache#14 doris::pipeline::SpillContext::~SpillContext (this=0x7f91536f0150) at /root/doris/be/src/pipeline/exec/spill_utils.h:57
apache#15 0x000055f384259e4b in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7f91536f0140) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:345
apache#16 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::operator= (this=0x7f8d61302518, __r=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1088
apache#17 std::__shared_ptr<doris::pipeline::SpillContext, (__gnu_cxx::_Lock_policy)2>::operator= (this=0x7f8d61302510) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1530
apache#18 std::shared_ptr<doris::pipeline::SpillContext>::operator= (this=0x7f8d61302510) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr.h:413
apache#19 doris::pipeline::PipelineTask::revoke_memory (this=0x7f8d61302490, spill_context=...) at /root/doris/be/src/pipeline/pipeline_task.cpp:812
apache#20 0x000055f37fba3c2d in doris::QueryTaskController::revoke_memory (this=<optimized out>) at /root/doris/be/src/runtime/workload_management/query_task_controller.cpp:185
apache#21 0x000055f37fb99d08 in doris::WorkloadGroupMgr::handle_single_query_ (this=<optimized out>, requestor=..., size_to_reserve=size_to_reserve@entry=1024000, time_in_queue=time_in_queue@entry=27, paused_reason=...)
    at /root/doris/be/src/runtime/workload_group/workload_group_manager.cpp:820
apache#22 0x000055f37fb981d0 in doris::WorkloadGroupMgr::handle_paused_queries (this=0x7f9141d8a800) at /root/doris/be/src/runtime/workload_group/workload_group_manager.cpp:381
apache#23 0x000055f37eb97297 in doris::Daemon::memory_maintenance_thread (this=0x7ffe63cad730) at /root/doris/be/src/common/daemon.cpp:354
apache#24 0x000055f37fd812fc in std::function<void ()>::operator()() const (this=0x7f8dbb0c9fe0) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593
apache#25 doris::Thread::supervise_thread (arg=0x7f914ba25e10) at /root/doris/be/src/util/thread.cpp:460
apache#26 0x00007f91fc75fac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
apache#27 0x00007f91fc7f1850 in __closefrom_fallback (from=1674236160, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:45
apache#28 0x0000000000000000 in ?? ()
```

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
kaka11chen pushed a commit that referenced this pull request Oct 23, 2025
@mrhhsg
[fix](olap) Need to handle floating-point Infinity values in the pred…
f787653 
…icate (apache#57100)

### What problem does this PR solve?

```text
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140649357411904) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140649357411904) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140649357411904, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007feca2a42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
apache#4  0x00007feca2a287f3 in __GI_abort () at ./stdlib/abort.c:79
apache#5  0x000055cd63341b52 in __gnu_cxx::__verbose_terminate_handler() ()
apache#6  0x000055cd6333fe26 in __cxxabiv1::__terminate(void (*)()) ()
apache#7  0x000055cd6333fe79 in std::terminate() ()
apache#8  0x000055cd6333ffc3 in __cxa_throw ()
apache#9  0x000055cd6339556a in std::__throw_out_of_range(char const*) ()
apache#10 0x000055cd5a9a7a68 in __gnu_cxx::__stoa<double, double, char> (__str=0x7fe842726fc0 "9.", '9' <repeats 15 times>, "1e-309", __idx=0x0, __convf=<optimized out>, __name=<optimized out>)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/ext/string_conversions.h:90
apache#11 std::__cxx11::stod (__str=..., __idx=0x0) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/basic_string.h:4490
apache#12 doris::IntegerPredicateCreator<(doris::PrimitiveType)9, (doris::PredicateType)1, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::convert (condition=...)
    at /root/doris_branch-4.0/doris/be/src/olap/predicate_creator.h:68
apache#13 doris::IntegerPredicateCreator<(doris::PrimitiveType)9, (doris::PredicateType)1, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::create (this=<optimized out>, column=..., index=1, conditions=..., opposite=false,
    arena=...) at /root/doris_branch-4.0/doris/be/src/olap/predicate_creator.h:59
apache#14 0x000055cd5a3dd63b in doris::create_predicate<(doris::PredicateType)1, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (column=..., index=1, conditions=..., opposite=false, arena=...)
    at /root/doris_branch-4.0/doris/be/src/olap/predicate_creator.h:270
apache#15 doris::create_comparison_predicate<(doris::PredicateType)1> (column=..., index=1, condition=..., opposite=false, arena=...) at /root/doris_branch-4.0/doris/be/src/olap/predicate_creator.h:278
apache#16 0x000055cd5a3dd435 in doris::parse_to_predicate (column=..., index=1, condition=..., arena=..., opposite=false) at /root/doris_branch-4.0/doris/be/src/olap/predicate_creator.h:326
apache#17 0x000055cd5afc2759 in doris::TabletReader::_init_conditions_param (this=this@entry=0x7fe842e1a800, read_params=...) at /root/doris_branch-4.0/doris/be/src/olap/tablet_reader.cpp:549
apache#18 0x000055cd5afc1022 in doris::TabletReader::_init_params (this=this@entry=0x7fe842e1a800, read_params=...) at /root/doris_branch-4.0/doris/be/src/olap/tablet_reader.cpp:304
apache#19 0x000055cd5afbfe00 in doris::TabletReader::init (this=0x7fe842e1a800, read_params=...) at /root/doris_branch-4.0/doris/be/src/olap/tablet_reader.cpp:127
apache#20 0x000055cd5f019d0f in doris::vectorized::BlockReader::init (this=0x4c1b, read_params=...) at /root/doris_branch-4.0/doris/be/src/vec/olap/block_reader.cpp:206
apache#21 0x000055cd5f79a3f4 in doris::vectorized::OlapScanner::open (this=0x7febe2c16010, state=<optimized out>) at /root/doris_branch-4.0/doris/be/src/vec/exec/scan/olap_scanner.cpp:278
apache#22 0x000055cd5e05303a in doris::vectorized::ScannerScheduler::_scanner_scan (ctx=..., scan_task=...) at /root/doris_branch-4.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:182
apache#23 0x000055cd5e0549e5 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const (this=<optimized out>) at /root/doris_branch-4.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:96
apache#24 doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}::operator()() const (this=0x7febe321dac0)
    at /root/doris_branch-4.0/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:95
apache#25 std::__invoke_impl<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&) (__f=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63
apache#26 std::__invoke_r<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&) (__fn=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116
apache#27 std::_Function_handler<bool (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) (
    __functor=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292
apache#28 0x000055cd5b4ab5d5 in doris::ThreadPool::dispatch_thread (this=0x7fec35e01800) at /root/doris_branch-4.0/doris/be/src/util/threadpool.cpp:614
apache#29 0x000055cd5b4a06fc in std::function<void ()>::operator()() const (this=0x4a17) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593
apache#30 doris::Thread::supervise_thread (arg=0x7fec35872110) at /root/doris_branch-4.0/doris/be/src/util/thread.cpp:460
apache#31 0x00007feca2a94ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
apache#32 0x00007feca2b26850 in __closefrom_fallback (from=-823527648, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:45
apache#33 0x0000000000000000 in ?? ()
```
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
kaka11chen pushed a commit that referenced this pull request Oct 23, 2025
@mrhhsg
[fix](olap) Crash caused by out-of-bounds PODArray access (apache#57239)
8e919ba 
### What problem does this PR solve?

```text
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=136182904088128) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=136182904088128) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=136182904088128, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007fdee8ffb476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
apache#4  0x00007fdee8fe17f3 in __GI_abort () at ./stdlib/abort.c:79
apache#5  0x000055cc31fb9bcd in ?? ()
apache#6  0x000055cc31fadeac in google::LogMessage::SendToLog() ()
apache#7  0x000055cc31fae538 in google::LogMessage::Flush() ()
apache#8  0x000055cc31fb2116 in google::LogMessageFatal::~LogMessageFatal() ()
apache#9  0x000055cc1977f798 in doris::vectorized::PODArray<unsigned int, 4096ul, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator, false>, 16ul, 15ul>::operator[] (this=<optimized out>, n=2) at /root/doris/be/src/vec/common/pod_array.h:466
apache#10 0x000055cc19777580 in doris::vectorized::ColumnStr<unsigned int>::size_at (this=0x7c5eea94dda0, i=2) at /root/doris/be/src/vec/columns/column_string.h:96
apache#11 doris::vectorized::ColumnStr<unsigned int>::compare_at (this=0x7c5eea94dda0, n=2, m=1, rhs_=...) at /root/doris/be/src/vec/columns/column_string.h:482
apache#12 0x000055cc2f2f1138 in doris::vectorized::VCollectIterator::_topn_next (this=<optimized out>, block=<optimized out>) at /root/doris/be/src/vec/olap/vcollect_iterator.cpp:343
apache#13 0x000055cc2f2efa08 in doris::vectorized::VCollectIterator::next (this=0x898f, block=0x6) at /root/doris/be/src/vec/olap/vcollect_iterator.cpp:247
apache#14 0x000055cc2f28e3ea in doris::vectorized::BlockReader::_direct_next_block (this=0x7dbeed736880, block=<optimized out>, eof=<optimized out>) at /root/doris/be/src/vec/olap/block_reader.cpp:267
apache#15 0x000055cc2f28778b in doris::vectorized::BlockReader::next_block_with_aggregation (this=0x7dbeed736880, block=<optimized out>, eof=<optimized out>) at /root/doris/be/src/vec/olap/block_reader.cpp:66
apache#16 0x000055cc2b33300e in doris::vectorized::OlapScanner::_get_block_impl (this=0x7daeee690690, state=<optimized out>, block=0x7daeee6907b8, eof=<optimized out>) at /root/doris/be/src/vec/exec/scan/olap_scanner.cpp:584
apache#17 0x000055cc2b2c990d in doris::vectorized::Scanner::get_block (this=<optimized out>, state=<optimized out>, block=0x7daeee6907b8, eof=<optimized out>) at /root/doris/be/src/vec/exec/scan/scanner.cpp:116
apache#18 0x000055cc2b2c8fa0 in doris::vectorized::Scanner::get_block_after_projects (this=0x7daeee690690, state=0x7dcee9e6ec80, block=0x7caee9b74fe0, eos=0x7bdb809a11f0) at /root/doris/be/src/vec/exec/scan/scanner.cpp:82
apache#19 0x000055cc2b2e51b4 in doris::vectorized::ScannerScheduler::_scanner_scan (ctx=..., scan_task=...) at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:182
apache#20 0x000055cc2b2ed92c in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const (this=<optimized out>) at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:96
apache#21 doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}::operator()() const (this=0x7c0eeec42540)
    at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:95
apache#22 std::__invoke_impl<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&) (__f=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63
apache#23 std::__invoke_r<bool, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}&) (__fn=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116
apache#24 std::_Function_handler<bool (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) (
    __functor=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292
apache#25 0x000055cc1dd09d33 in doris::ThreadPool::dispatch_thread (this=0x7d3ee8680f80) at /root/doris/be/src/util/threadpool.cpp:614
apache#26 0x000055cc1dce6da7 in std::function<void ()>::operator()() const (this=<optimized out>) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593
apache#27 doris::Thread::supervise_thread (arg=<optimized out>) at /root/doris/be/src/util/thread.cpp:460
apache#28 0x000055cc19364d27 in asan_thread_start(void*) ()
apache#29 0x00007fdee904dac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
apache#30 0x00007fdee90df850 in __closefrom_fallback (from=877593024, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:45
apache#31 0x0000000000000000 in ?? ()
```

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
kaka11chen pushed a commit that referenced this pull request Nov 12, 2025
@seawinde
[opt](mtmv) Support window function rewrite when materialized view co…
dfe888e 
…ntains window function (apache#55066)

### What problem does this PR solve?

Support window function rewrite when materialized view contains window
function

Such as mv def is as following:

        CREATE MATERIALIZED VIEW mv1
        BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
        DISTRIBUTED BY RANDOM BUCKETS 2
        PROPERTIES ('replication_num' = '1') 
        AS
        select *
        from (
        select 
        o_orderkey,
        FIRST_VALUE(o_custkey) OVER (
                PARTITION BY o_orderdate 
                ORDER BY o_totalprice NULLS LAST
                RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
            ) AS first_value,
        RANK() OVER (
                PARTITION BY o_orderdate, o_orderstatus 
                ORDER BY o_totalprice NULLS LAST
                RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
            ) AS rank_value,
        LAG(l_extendedprice, 1, 0) over (partition by o_orderdate, l_shipdate order by l_quantity) AS lag_value 
        from 
        lineitem2
        left join orders2 on l_orderkey = o_orderkey and l_shipdate = o_orderdate
        ) t
        where o_orderkey > 1;

if query as following, this can use mv to represent query

select *
            from (
            select 
            o_orderkey,
            FIRST_VALUE(o_custkey) OVER (
                    PARTITION BY o_orderdate 
                    ORDER BY o_totalprice NULLS LAST
                    RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
                ) AS first_value,
            RANK() OVER (
                    PARTITION BY o_orderdate, o_orderstatus 
                    ORDER BY o_totalprice NULLS LAST
                    RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
                ) AS rank_value,
            LAG(l_extendedprice, 1, 0) over (partition by o_orderdate, l_shipdate order by l_quantity) AS lag_value 
            from 
            lineitem2
            left join orders2 on l_orderkey = o_orderkey and l_shipdate = o_orderdate
            ) t
            where o_orderkey > 2;

explain result is as follwing:

+----------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                        |
+----------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                                        |
|   OUTPUT EXPRS:                                                                        |
|     o_orderkey[apache#4]                                                                     |
|     first_value[apache#5]                                                                    |
|     rank_value[apache#6]                                                                     |
|     lag_value[apache#7]                                                                      |
|   PARTITION: RANDOM                                                                    |
|                                                                                        |
|   HAS_COLO_PLAN_NODE: false                                                            |
|                                                                                        |
|   VRESULT SINK                                                                         |
|      MYSQL_PROTOCAL                                                                    |
|                                                                                        |
|   0:VOlapScanNode(445)                                                                 |
|      TABLE: regression_test_nereids_rules_p0_mv_window.mv1(mv1), PREAGGREGATION: ON    |
|      PREDICATES: (o_orderkey[#0] > 2)                                                  |
|      partitions=1/1 (mv1)                                                              |
|      tablets=2/2, tabletList=1755678381149,1755678381151                               |
|      cardinality=1, avgRowSize=0.0, numNodes=1                                         |
|      pushAggOp=NONE                                                                    |
|      final projections: o_orderkey[#0], first_value[#1], rank_value[#2], lag_value[#3] |
|      final project output tuple id: 1                                                  |
|                                                                                        |
|                                                                                        |
| ========== MATERIALIZATIONS ==========                                                 |
|                                                                                        |
| MaterializedView                                                                       |
| MaterializedViewRewriteSuccessAndChose:                                                |
|   internal.regression_test_nereids_rules_p0_mv_window.mv1 chose                        |
|                                                                                        |
| MaterializedViewRewriteSuccessButNotChose:                                             |
|                                                                                        |
| MaterializedViewRewriteFail:                                                           |
|                                                                                        |
|                                                                                        |
| ========== STATISTICS ==========                                                       |
| planed with unknown column statistics                                                  |
+----------------------------------------------------------------------------------------+
kaka11chen pushed a commit that referenced this pull request Nov 22, 2025
@xy720
[fix](mtmx) Fix coredump because common expr push down to scannode wh…
8e0f77a 
…ich belongs to an agg materialized view (apache#58038)

### What problem does this PR solve?

Issue Number: close apache#58037

Problem Summary:

```
#0  0x00007f9aca4a3f8c in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x00007f9aca454a26 in raise () from /lib64/libc.so.6
#2  0x00007f9aca43d87c in abort () from /lib64/libc.so.6
#3  0x0000561dc3d1ea1d in ?? ()
apache#4  0x0000561dc3d1105a in google::LogMessage::Fail() ()
apache#5  0x0000561dc3d14146 in google::LogMessage::SendToLog() ()
apache#6  0x0000561dc3d10b90 in google::LogMessage::Flush() ()
apache#7  0x0000561dc3d14989 in google::LogMessageFatal::~LogMessageFatal() ()
apache#8  0x0000561db854c996 in assert_cast<doris::vectorized::ColumnStr<unsigned int> const&, (TypeCheckOnRelease)1, doris::vectorized::IColumn const&>(doris::vectorized::IColumn const&)::{lambda(auto:1&&)#1}::operator()<doris::vectorized::IColumn const&>(doris::vectorized::IColumn const&) const
    (this=this@entry=0x7f9658ccc1f8, from=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/common/assert_cast.h:58
apache#9  0x0000561db854c7d7 in assert_cast<doris::vectorized::ColumnStr<unsigned int> const&, (TypeCheckOnRelease)1, doris::vectorized::IColumn const&> (from=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/common/assert_cast.h:73
apache#10 0x0000561db854bb0b in doris::vectorized::ColumnStr<unsigned int>::compare_at (this=0x7f957a14e2c0, n=1159288, m=6, rhs_=...)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/columns/column_string.h:526
apache#11 0x0000561dbe108c6b in doris::vectorized::GenericComparisonImpl<doris::vectorized::EqualsOp<int, int> >::vector_constant (a=..., b=..., c=...)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/functions_comparison.h:112
apache#12 doris::vectorized::FunctionComparison<doris::vectorized::EqualsOp, doris::vectorized::NameEquals>::execute_generic_identical_types (
    this=<optimized out>, block=..., result=result@entry=10, c0=0x7f957a14e2c0, c1=<optimized out>)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/functions_comparison.h:506
apache#13 0x0000561dbdf9e97e in doris::vectorized::FunctionComparison<doris::vectorized::EqualsOp, doris::vectorized::NameEquals>::execute_generic (
    this=0x7f96d6fb1b90, block=..., result=10, c0=..., c1=...)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/functions_comparison.h:517
apache#14 doris::vectorized::FunctionComparison<doris::vectorized::EqualsOp, doris::vectorized::NameEquals>::execute_impl (this=0x7f96d6fb1b90, 
    context=<optimized out>, block=..., arguments=..., result=10, input_rows_count=104)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/functions_comparison.h:707
apache#15 0x0000561dbdcf1b8f in doris::vectorized::DefaultExecutable::execute_impl (this=<optimized out>, context=0x6, block=..., arguments=..., 
    result=1, input_rows_count=104) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.h:472
apache#16 0x0000561dbeea76ae in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal (this=this@entry=0x7f99f62a65d0, 
    context=context@entry=0x7f99f6442b00, block=..., args=..., result=result@entry=10, input_rows_count=104, dry_run=<optimized out>)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:121
apache#17 0x0000561dbeea4ce8 in doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns (this=0x7f99f62a65d0, 
    context=0x7f99f6442b00, block=..., args=..., result=10, input_rows_count=104, dry_run=<optimized out>)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:246
apache#18 doris::vectorized::PreparedFunctionImpl::default_implementation_for_nulls (this=this@entry=0x7f99f62a65d0, 
    context=context@entry=0x7f99f6442b00, block=..., args=..., result=result@entry=10, input_rows_count=104, dry_run=<optimized out>, 
    executed=0x7f9658ccc666) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:218
apache#19 0x0000561dbeea4e9c in doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal (this=0x7f99f62a65d0, context=0x7f99f6442b00, 
    block=..., args=..., result=10, input_rows_count=<optimized out>, dry_run=<optimized out>)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:112
apache#20 doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns (this=0x7f99f62a65d0, context=0x7f99f6442b00, block=..., 
    args=..., result=10, input_rows_count=104, dry_run=<optimized out>)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:246
apache#21 0x0000561dbeea4f66 in doris::vectorized::PreparedFunctionImpl::execute (this=0x11b078, context=0x6, block=..., args=..., result=1, 
    input_rows_count=104, dry_run=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.cpp:252
apache#22 0x0000561dbdcf1500 in doris::vectorized::IFunctionBase::execute (this=<optimized out>, context=0x7f99f6442b00, block=..., arguments=..., 
    result=10, input_rows_count=104, dry_run=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/functions/function.h:195
--Type <RET> for more, q to quit, c to continue without paging--c
apache#23 0x0000561dbdceccad in doris::vectorized::VectorizedFnCall::_do_execute (this=0x7f96f0fec510, context=0x7f957f09cdf0, block=0x7f957b02a3b0, 
    result_column_id=0x7f9658ccca14, args=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exprs/vectorized_fn_call.cpp:197
apache#24 0x0000561dbdced2c6 in doris::vectorized::VectorizedFnCall::execute (this=0x11b078, context=0x6, 
    block=0x7f9aca4a3f8c <__pthread_kill_implementation+268>, result_column_id=0x7f9658ccbe10)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exprs/vectorized_fn_call.cpp:212
apache#25 0x0000561dbdd1e51b in doris::vectorized::VExprContext::execute (this=0x7f957f09cdf0, 
    block=0x7f9aca4a3f8c <__pthread_kill_implementation+268>, block@entry=0x7f957b02a3b0, result_column_id=result_column_id@entry=0x7f9658ccca14)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exprs/vexpr_context.cpp:55
apache#26 0x0000561dbdd1fcb5 in doris::vectorized::VExprContext::execute_conjuncts (ctxs=..., filters=filters@entry=0x0, accept_null=false, 
    block=block@entry=0x7f957b02a3b0, result_filter=result_filter@entry=0x7f9658ccccc0, can_filter_all=0x7f9658cccbc7)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exprs/vexpr_context.cpp:174
apache#27 0x0000561dbdd2131f in doris::vectorized::VExprContext::execute_conjuncts_and_filter_block (ctxs=..., block=0x7f957b02a3b0, 
    columns_to_filter=..., column_to_keep=6, filter=...) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exprs/vexpr_context.cpp:354
apache#28 0x0000561db8f49450 in doris::segment_v2::SegmentIterator::_execute_common_expr (this=this@entry=0x7f954e20a000, 
    sel_rowid_idx=0x7f955c0d2000, selected_size=@0x7f9658ccce6e: 104, block=block@entry=0x7f957b02a3b0)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2338
apache#29 0x0000561db8f482f8 in doris::segment_v2::SegmentIterator::_next_batch_internal (this=0x7f954e20a000, block=0x7f957b02a3b0)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2230
apache#30 0x0000561db8f45212 in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const (
    this=<optimized out>) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1953
apache#31 doris::segment_v2::SegmentIterator::next_batch (this=0x7f954e20a000, block=0x6)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:1952
apache#32 0x0000561db8ee49bc in doris::segment_v2::LazyInitSegmentIterator::next_batch (this=0x7f953bc71f80, block=0x7f957b02a3b0)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/segment_v2/lazy_init_segment_iterator.h:44
apache#33 0x0000561db8dab844 in doris::BetaRowsetReader::next_block (this=0x7f9a4e215800, block=0x7f957b02a3b0)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/olap/rowset/beta_rowset_reader.cpp:377
apache#34 0x0000561dc2c9413d in doris::vectorized::VCollectIterator::Level0Iterator::_refresh (this=0x7f953ba137a0)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/vcollect_iterator.h:256
apache#35 doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row (this=this@entry=0x7f953ba137a0)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/vcollect_iterator.cpp:509
apache#36 0x0000561dc2c93bf4 in doris::vectorized::VCollectIterator::Level0Iterator::init (this=0x7f953ba137a0, get_data_by_ref=<optimized out>)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/vcollect_iterator.cpp:461
apache#37 0x0000561dc2c91002 in doris::vectorized::VCollectIterator::build_heap (this=0x7f957a52bb30, rs_readers=...)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/vcollect_iterator.cpp:125
apache#38 0x0000561dc2c7e1f2 in doris::vectorized::BlockReader::_init_collect_iter (this=this@entry=0x7f957a52b400, read_params=...)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/block_reader.cpp:153
apache#39 0x0000561dc2c7f191 in doris::vectorized::BlockReader::init (this=<optimized out>, read_params=...)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/olap/block_reader.cpp:226
apache#40 0x0000561dc3937869 in doris::vectorized::NewOlapScanner::open (this=0x7f9a56270210, state=<optimized out>)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:252
apache#41 0x0000561dbdcc5413 in doris::vectorized::ScannerScheduler::_scanner_scan (ctx=..., scan_task=...)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:221
apache#42 0x0000561dbdcc62bd in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const (this=<optimized out>)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:154
apache#43 doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}::operator()() const (this=0x7f954e25d3c0)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:153
apache#44 std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}&) (__f=...)
    at /data/home/lambxu/installs/ldb_toolchain_bak/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
apache#45 std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}&) (__fn=...)
    at /data/home/lambxu/installs/ldb_toolchain_bak/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111
apache#46 std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...)
    at /data/home/lambxu/installs/ldb_toolchain_bak/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
apache#47 0x0000561db962137a in doris::ThreadPool::dispatch_thread (this=0x7f9a50f9d380)
    at /data/home/lambxu/work/git/doris-3.1/doris/be/src/util/threadpool.cpp:602
apache#48 0x0000561db96159a1 in std::function<void ()>::operator()() const (this=0x11a791)
    at /data/home/lambxu/installs/ldb_toolchain_bak/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560
apache#49 doris::Thread::supervise_thread (arg=0x7f969f569ce0) at /data/home/lambxu/work/git/doris-3.1/doris/be/src/util/thread.cpp:498
apache#50 0x00007f9aca4a2215 in start_thread () from /lib64/libc.so.6
apache#51 0x00007f9aca524bdc in clone3 () from /lib64/libc.so.6
```

Assume that 0,1,2,3,4, is key columns of an AGG mv, because the PreAgg
is OFF at scan node, the block will contain all key columns to merge
data in storage layer.

if we select 0,1 column, with 3,4 column in where clause, then the slot
ids should be 0,1,3,4, and column ids in conjuncts is the index of slot
ids.(which is 2 and 3)

But the plan use the key type of base table which is DUP key, treating
the AGG mv as a DUP mv, so these conjuncts are pushed down to the scan
node which belongs to an AGG mv, these conjuncts will pick the wrong
column 2 and 3 (which shoud be 4 and 5) in block to exucute.

So we should use the key type of mv but not the key type of base table.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [x] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
kaka11chen pushed a commit that referenced this pull request Nov 22, 2025
@Mryange
[fix](core)Provide add_blocks interface for VDataStreamRecvr (apache#…
6deeab8 
…57888)

We previously had a crash. The cause is that we should not access the
request after calling add_block(...) because add_block may enqueue a
closure that runs on another thread and frees the request

```
==730145==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7be1efd803a0 at pc 0x556b38d9d625 bp 0x7b16bf0193f0 sp 0x7b16bf0193e8
READ of size 4 at 0x7be1efd803a0 thread T1559
    #0 0x556b38d9d624 in google::protobuf::internal::RepeatedPtrFieldBase::size() const /home/zcp/repo_center/doris_master/doris/thirdparty/installed/include/google/protobuf/repeated_ptr_field.h:185:29
    #1 0x556b408ab062 in google::protobuf::RepeatedPtrField<doris::PBlock>::size() const /home/zcp/repo_center/doris_master/doris/thirdparty/installed/include/google/protobuf/repeated_ptr_field.h:1248:32
    #2 0x556b408aaff4 in doris::PTransmitDataParams::_internal_blocks_size() const /home/zcp/repo_center/doris_master/doris/be/../gensrc/build/gen_cpp/internal_service.pb.h:32149:25
    #3 0x556b4089731c in doris::PTransmitDataParams::blocks_size() const /home/zcp/repo_center/doris_master/doris/be/../gensrc/build/gen_cpp/internal_service.pb.h:32152:10
    apache#4 0x556b60a83c17 in doris::vectorized::VDataStreamMgr::transmit_block(doris::PTransmitDataParams const*, google::protobuf::Closure**, long) /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_mgr.cpp:150:38
    apache#5 0x556b407f7408 in doris::PInternalService::_transmit_block(google::protobuf::RpcController*, doris::PTransmitDataParams const*, doris::PTransmitDataResult*, google::protobuf::Closure*, doris::Status const&, long) /home/zcp/repo_center/doris_master/doris/be/src/service/internal_service.cpp:1673:40
    apache#6 0x556b407f52bb in doris::PInternalService::transmit_block(google::protobuf::RpcController*, doris::PTransmitDataParams const*, doris::PTransmitDataResult*, google::protobuf::Closure*) /home/zcp/repo_center/doris_master/doris/be/src/service/internal_service.cpp:1610:9
    apache#7 0x556b43fceba2 in doris::PBackendService::CallMethod(google::protobuf::MethodDescriptor const*, google::protobuf::RpcController*, google::protobuf::Message const*, google::protobuf::Message*, google::protobuf::Closure*) /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/internal_service.pb.cc:49452:7
    apache#8 0x556b6736273e in brpc::policy::ProcessRpcRequest(brpc::InputMessageBase*) (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x770bd73e)
    apache#9 0x556b67357426 in brpc::ProcessInputMessage(void*) (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x770b2426)
    apache#10 0x556b67357f20 in brpc::InputMessenger::InputMessageClosure::~InputMessageClosure() (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x770b2f20)
    apache#11 0x556b673588dd in brpc::InputMessenger::OnNewMessages(brpc::Socket*) (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x770b38dd)
    apache#12 0x556b674a0adc in brpc::Socket::ProcessEvent(void*) (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x771fbadc)
    apache#13 0x556b672e0f76 in bthread::TaskGroup::task_runner(long) (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x7703bf76)
    apache#14 0x556b672cbbe0 in bthread_make_fcontext (/mnt/hdd01/selectdb-cloud-chaos/cluster0/be/lib/doris_be+0x77026be0)
```

```
        Note: The done pointer will be saved in add_block and may be called in another thread via done->Run().
        For example, when blocks_size == 1, the process is as follows:
        transmit_block (i=0)
          └─> recvr->add_block(..., done, ...)  // Pass done
               └─> SenderQueue::add_block
                    └─> _pending_closures.push(done)  // done is saved

        get_batch() [another thread]
          └─> closure_pair.first->Run()  // ⚠️ done->Run() is called
               └─> brpc releases request and response

        transmit_block (i=1)  [original thread continues]
          └─> request->blocks_size()  // ⚠️ request has already been released!

        At this point, a use-after-free issue occurs.

        TODO: We should consider refactoring this part because add_block may release the request.
        We should not access the request after calling add_block.
```


apache#50113
kaka11chen pushed a commit that referenced this pull request Dec 3, 2025
@HonestManXin
[Feature] doris cross-cluster query (apache#57898)
70b07cb 
Related issue: apache#57884

```
MySQL [demo]> show frontends;
+-----------------------------------------+--------------+-------------+----------+-----------+---------+--------------------+----------+----------+-----------+------+-------+-------------------+---------------------+---------------------+----------+--------+------------------------+------------------+---------------------+
| Name                                    | Host         | EditLogPort | HttpPort | QueryPort | RpcPort | ArrowFlightSqlPort | Role     | IsMaster | ClusterId | Join | Alive | ReplayedJournalId | LastStartTime       | LastHeartbeat       | IsHelper | ErrMsg | Version                | CurrentConnected | LiveSince           |
+-----------------------------------------+--------------+-------------+----------+-----------+---------+--------------------+----------+----------+-----------+------+-------+-------------------+---------------------+---------------------+----------+--------+------------------------+------------------+---------------------+
| fe_a7c0b6d8_82c2_48f0_8220_fb65dd18be69 | 10.37.75.124 | 9010        | 8030     | 9030      | 9020    | 8070               | FOLLOWER | true     | 742250121 | true | true  | 2409              | 2025-11-11 14:42:16 | 2025-11-11 14:44:06 | true     |        | doris-0.0.0-009c3b552a | Yes              | 2025-11-11 14:42:16 |
+-----------------------------------------+--------------+-------------+----------+-----------+---------+--------------------+----------+----------+-----------+------+-------+-------------------+---------------------+---------------------+----------+--------+------------------------+------------------+---------------------+
1 row in set (0.016 sec)

MySQL [demo]> show catalog edoris;
+-----------------+-------------------------------+
| Key             | Value                         |
+-----------------+-------------------------------+
| create_time     | 2025-11-11 11:25:33.488106853 |
| fe_arrow_hosts  | 10.37.103.28:8070             |
| fe_http_hosts   | 10.37.103.28:8030             |
| fe_thrift_hosts | 10.37.103.28:9020             |
| password        | *XXX                          |
| type            | doris                         |
| use_meta_cache  | true                          |
| user            | test                          |
+-----------------+-------------------------------+
8 rows in set (0.002 sec)

MySQL [demo]> select * from inner_table;
+----------+--------+
| log_type | reason |
+----------+--------+
|        2 | inner2 |
|        3 | inner3 |
|        4 | inner4 |
+----------+--------+
3 rows in set (0.032 sec)

MySQL [demo]> select * from edoris.external.example_tbl_duplicate;
+---------------------+----------+------------+-----------+-------+---------------------+
| log_time            | log_type | error_code | error_msg | op_id | op_time             |
+---------------------+----------+------------+-----------+-------+---------------------+
| 2024-11-01 00:00:00 |        2 |          2 | timeout   |    12 | 2024-11-01 01:00:00 |
+---------------------+----------+------------+-----------+-------+---------------------+
1 row in set (0.059 sec)

MySQL [demo]> select * from inner_table a join edoris.external.example_tbl_duplicate b on (a.log_type = b.log_type);
+----------+--------+---------------------+----------+------------+-----------+-------+---------------------+
| log_type | reason | log_time            | log_type | error_code | error_msg | op_id | op_time             |
+----------+--------+---------------------+----------+------------+-----------+-------+---------------------+
|        2 | inner2 | 2024-11-01 00:00:00 |        2 |          2 | timeout   |    12 | 2024-11-01 01:00:00 |
+----------+--------+---------------------+----------+------------+-----------+-------+---------------------+
1 row in set (0.050 sec)

MySQL [demo]> explain select * from inner_table a join edoris.external.example_tbl_duplicate b on (a.log_type = b.log_type) where error_code=2;
+-------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                                           |
+-------------------------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                                                                                           |
|   OUTPUT EXPRS:                                                                                                                           |
|     log_type[apache#16]                                                                                                                         |
|     reason[apache#17]                                                                                                                           |
|     log_time[apache#18]                                                                                                                         |
|     log_type[apache#19]                                                                                                                         |
|     error_code[apache#20]                                                                                                                       |
|     error_msg[apache#21]                                                                                                                        |
|     op_id[apache#22]                                                                                                                            |
|     op_time[apache#23]                                                                                                                          |
|   PARTITION: HASH_PARTITIONED: log_type[apache#6]                                                                                               |
|                                                                                                                                           |
|   HAS_COLO_PLAN_NODE: false                                                                                                               |
|                                                                                                                                           |
|   VRESULT SINK                                                                                                                            |
|      MYSQL_PROTOCOL                                                                                                                       |
|                                                                                                                                           |
|   3:VHASH JOIN(200)                                                                                                                       |
|   |  join op: INNER JOIN(BROADCAST)[]                                                                                                     |
|   |  equal join conjunct: (log_type[apache#6] = log_type[#1])                                                                                   |
|   |  cardinality=3                                                                                                                        |
|   |  vec output tuple id: 3                                                                                                               |
|   |  output tuple id: 3                                                                                                                   |
|   |  vIntermediate tuple ids: 2                                                                                                           |
|   |  hash output slot ids: 0 1 2 3 4 5 6 7                                                                                                |
|   |  runtime filters: RF000[min_max] <- log_type[#1](1/1/1048576), RF001[in_or_bloom] <- log_type[#1](1/1/1048576)                        |
|   |  final projections: log_type[apache#8], reason[apache#9], log_time[apache#10], log_type[apache#11], error_code[apache#12], error_msg[apache#13], op_id[apache#14], op_time[apache#15] |
|   |  final project output tuple id: 3                                                                                                     |
|   |  distribute expr lists: log_type[apache#6]                                                                                                  |
|   |  distribute expr lists:                                                                                                               |
|   |                                                                                                                                       |
|   |----1:VEXCHANGE                                                                                                                        |
|   |       offset: 0                                                                                                                       |
|   |       distribute expr lists: log_type[#1]                                                                                             |
|   |                                                                                                                                       |
|   2:VOlapScanNode(187)                                                                                                                    |
|      TABLE: demo.inner_table(inner_table), PREAGGREGATION: ON                                                                             |
|      partitions=1/1 (inner_table)                                                                                                         |
|      tablets=1/1, tabletList=1762832514491                                                                                                |
|      cardinality=3, avgRowSize=901.6666, numNodes=1                                                                                       |
|      pushAggOp=NONE                                                                                                                       |
|      runtime filters: RF000[min_max] -> log_type[apache#6], RF001[in_or_bloom] -> log_type[apache#6]                                                  |
|                                                                                                                                           |
| PLAN FRAGMENT 1                                                                                                                           |
|                                                                                                                                           |
|   PARTITION: HASH_PARTITIONED: log_type[#1]                                                                                               |
|                                                                                                                                           |
|   HAS_COLO_PLAN_NODE: false                                                                                                               |
|                                                                                                                                           |
|   STREAM DATA SINK                                                                                                                        |
|     EXCHANGE ID: 01                                                                                                                       |
|     UNPARTITIONED                                                                                                                         |
|                                                                                                                                           |
|   0:VOlapScanNode(188)                                                                                                                    |
|      TABLE: external.example_tbl_duplicate(example_tbl_duplicate), PREAGGREGATION: ON                                                     |
|      PREDICATES: (error_code[#2] = 2)                                                                                                     |
|      partitions=1/1 (example_tbl_duplicate)                                                                                               |
|      tablets=1/1, tabletList=1762481736238                                                                                                |
|      cardinality=1, avgRowSize=7425.0, numNodes=1                                                                                         |
|      pushAggOp=NONE                                                                                                                       |
|                                                                                                                                           |
|                                                                                                                                           |
|                                                                                                                                           |
| ========== STATISTICS ==========                                                                                                          |
| planed with unknown column statistics                                                                                                     |
+-------------------------------------------------------------------------------------------------------------------------------------------+
65 rows in set (0.040 sec)

```
kaka11chen pushed a commit that referenced this pull request Dec 9, 2025
@HonestManXin
[fix](cloud compaction) horizontal compaction coredump due to tablet …
a5fda94 
…shard_ptr not initialized (apache#58751)

```
#0  0x000055f11470ff29 in std::__shared_ptr<doris::TabletMeta, (__gnu_cxx::_Lock_policy)2>::get (this=0x148) at /home/work/env/ldb_toolchain_master/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1673

#1  std::__shared_ptr_access<doris::TabletMeta, (__gnu_cxx::_Lock_policy)2, false, false>::_M_get (this=0x148) at /home/work/env/ldb_toolchain_master/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1370

#2  std::__shared_ptr_access<doris::TabletMeta, (__gnu_cxx::_Lock_policy)2, false, false>::operator-> (this=0x148) at /home/work/env/ldb_toolchain_master/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/shared_ptr_base.h:1364

#3  doris::BaseTablet::tablet_id (this=0x0) at /home/work/doris/be/src/olap/base_tablet.h:73

apache#4  doris::segment_v2::SegmentWriter::finalize (this=0x7fa160bde000, segment_file_size=0x7fa564dcc668, index_size=0x7fa564dcc5b8) at /home/work/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:1044

apache#5  0x000055f11442578c in doris::SegmentFlusher::_flush_segment_writer (this=0x7fa160b8a420, writer=..., flush_size=flush_size@entry=0x0) at /home/work/doris/be/src/olap/rowset/segment_creator.cpp:304

apache#6  0x000055f114426c4e in doris::SegmentFlusher::Writer::flush (this=<optimized out>) at /home/work/doris/be/src/olap/rowset/segment_creator.cpp:376

apache#7  doris::SegmentCreator::flush (this=0x7fa160b8a418) at /home/work/doris/be/src/olap/rowset/segment_creator.cpp:422

apache#8  0x000055f1143d4785 in doris::BaseBetaRowsetWriter::flush (this=<optimized out>) at /home/work/doris/be/src/olap/rowset/beta_rowset_writer.cpp:723

apache#9  0x000055f11436bf47 in doris::Merger::vmerge_rowsets (tablet=..., reader_type=<optimized out>, cur_tablet_schema=..., src_rowset_readers=..., dst_rowset_writer=0x7fa160b8a000, stats_output=0x7fa1df108298) at /home/work/doris/be/src/olap/merger.cpp:159

apache#10 0x000055f114345595 in doris::Compaction::merge_input_rowsets (this=this@entry=0x7fa1df108210) at /home/work/doris/be/src/olap/compaction.cpp:220

apache#11 0x000055f1143550d2 in doris::CloudCompactionMixin::execute_compact_impl (this=this@entry=0x7fa1df108210, permits=permits@entry=6) at /home/work/doris/be/src/olap/compaction.cpp:1491

apache#12 0x000055f114342bc1 in doris::CloudCompactionMixin::execute_compact (this=0x7fa1df108210) at /home/work/doris/be/src/olap/compaction.cpp:1620

apache#13 0x000055f11a22c6d8 in doris::CloudCumulativeCompaction::execute_compact (this=0x7fa1df108210) at /home/work/doris/be/src/cloud/cloud_cumulative_compaction.cpp:203
```
kaka11chen pushed a commit that referenced this pull request Dec 23, 2025
@Gabriel39
[fix](predicate) Fix use-after-free caused by string predicate (apach…
1ec62bd 
…e#59098)

### What problem does this PR solve?

Introduced by apache#58905

==2037076==ERROR: AddressSanitizer: heap-use-after-free on address
0x7baaae908730 at pc 0x561b769a1fd0 bp 0x7b3caf4ebdf0 sp 0x7b3caf4ebde8
22:30:08  READ of size 1 at 0x7baaae908730 thread T12303 (rs_normal
[work)
22:30:08  #0 0x561b769a1fcf in doris::(anonymous
namespace)::string_compare(char const*, long, char const*, long, long)
/root/doris/be/src/vec/common/string_ref.h:170:29
22:30:08  #1 0x561b769a1fcf in
doris::StringRef::compare(doris::StringRef const&) const
/root/doris/be/src/vec/common/string_ref.h:259:30
22:30:08  #2 0x561b76f537cd in doris::StringRef::ge(doris::StringRef
const&) const /root/doris/be/src/vec/common/string_ref.h:282:52
22:30:08  #3 0x561b76f537cd in
doris::StringRef::operator>=(doris::StringRef const&) const
/root/doris/be/src/vec/common/string_ref.h:292:60
22:30:08  apache#4 0x561b76f537cd in bool
doris::Compare::greater_equal<doris::StringRef>(doris::StringRef const&,
doris::StringRef const&) /root/doris/be/src/common/compare.h:42:18
22:30:08  apache#5 0x561b76f537cd in
doris::ComparisonPredicateBase<(doris::PrimitiveType)23,
(doris::PredicateType)6>::camp_field(doris::vectorized::Field const&,
doris::vectorized::Field const&) const
/root/doris/be/src/olap/comparison_predicate.h:192:20
22:30:08  apache#6 0x561b76f4baa4 in
doris::ComparisonPredicateBase<(doris::PrimitiveType)23,
(doris::PredicateType)6>::evaluate_and(doris::vectorized::ParquetPredicate::ColumnStat*)
const /root/doris/be/src/olap/comparison_predicate.h:207:26
22:30:08  apache#7 0x561b76765284 in
doris::AndBlockColumnPredicate::evaluate_and(doris::vectorized::ParquetPredicate::ColumnStat*)
const /root/doris/be/src/olap/block_column_predicate.h:251:42
22:30:08  apache#8 0x561b89acd735 in
doris::vectorized::ParquetReader::_process_column_stat_filter(tparquet::RowGroup
const&, std::vector<std::unique_ptr<doris::MutilColumnBlockPredicate,
std::default_delete<doris::MutilColumnBlockPredicate> >,
std::allocator<std::unique_ptr<doris::MutilColumnBlockPredicate,
std::default_delete<doris::MutilColumnBlockPredicate> > > > const&,
bool*, bool*, bool*)
/root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:1225:25
22:30:08  apache#9 0x561b89ac8dd7 in
doris::vectorized::ParquetReader::_process_min_max_bloom_filter(doris::vectorized::RowGroupReader::RowGroupIndex
const&, tparquet::RowGroup const&,
std::vector<std::unique_ptr<doris::MutilColumnBlockPredicate,
std::default_delete<doris::MutilColumnBlockPredicate> >,
std::allocator<std::unique_ptr<doris::MutilColumnBlockPredicate,
std::default_delete<doris::MutilColumnBlockPredicate> > > > const&,
doris::segment_v2::RowRanges*)
/root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:1108:9
22:30:08  apache#10 0x561b89ac3e73 in
doris::vectorized::ParquetReader::_next_row_group_reader()
/root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:718:9
22:30:08  apache#11 0x561b89ac008f in
doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*,
unsigned long*, bool*)
/root/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:607:21
22:30:08  apache#12 0x561b8a07c6f7 in
doris::vectorized::HiveReader::get_next_block_inner(doris::vectorized::Block*,
unsigned long*, bool*)
/root/doris/be/src/vec/exec/format/table/hive_reader.cpp:32:5
22:30:08  apache#13 0x561b89fee256 in
doris::vectorized::TableFormatReader::get_next_block(doris::vectorized::Block*,
unsigned long*, bool*)
/root/doris/be/src/vec/exec/format/table/table_format_reader.h:81:16
22:30:08  apache#14 0x561b89f71b97 in
doris::vectorized::FileScanner::_get_block_wrapped(doris::RuntimeState*,
doris::vectorized::Block*, bool*)
/root/doris/be/src/vec/exec/scan/file_scanner.cpp:472:13
22:30:08  apache#15 0x561b89f7086f in
doris::vectorized::FileScanner::_get_block_impl(doris::RuntimeState*,
doris::vectorized::Block*, bool*)
/root/doris/be/src/vec/exec/scan/file_scanner.cpp:409:17
22:30:08  apache#16 0x561b8a19f86e in
doris::vectorized::Scanner::get_block(doris::RuntimeState*,
doris::vectorized::Block*, bool*)
/root/doris/be/src/vec/exec/scan/scanner.cpp:109:17
22:30:08  apache#17 0x561b8a19f0a6 in
doris::vectorized::Scanner::get_block_after_projects(doris::RuntimeState*,
doris::vectorized::Block*, bool*)
/root/doris/be/src/vec/exec/scan/scanner.cpp:85:16
22:30:08  apache#18 0x561b8a1ccd0f in
doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>,
std::shared_ptr<doris::vectorized::ScanTask>)
/root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:173:5
22:30:08  apache#19 0x561b8a1d6875 in
doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>,
std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()()
const::'lambda'()::operator()() const::'lambda'()::operator()() const
/root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:76:17
22:30:08  apache#20 0x561b8a1d6875 in
doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>,
std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()()
const::'lambda'()::operator()() const
/root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:75:27
22:30:08  apache#21 0x561b8a1d6875 in bool std::__invoke_impl<bool,
doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>,
std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()()
const::'lambda'()&>(std::__invoke_other,
doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>,
std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()()
const::'lambda'()&)
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63:14
22:30:08  apache#22 0x561b8a1d6875 in std::enable_if<is_invocable_r_v<bool,
doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>,
std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()()
const::'lambda'()&>, bool>::type std::__invoke_r<bool,
doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>,
std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()()
const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>,
std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()()
const::'lambda'()&)
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:116:9
22:30:08  apache#23 0x561b8a1d6875 in std::_Function_handler<bool (),
doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>,
std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()()
const::'lambda'()>::_M_invoke(std::_Any_data const&)
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292:9
22:30:08  apache#24 0x561b8a1d5f07 in std::function<bool ()>::operator()()
const
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9
22:30:08  apache#25 0x561b8a1d5f07 in
doris::vectorized::ScannerSplitRunner::process_for(std::chrono::duration<long,
std::ratio<1l, 1000000000l> >)
/root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:407:25
22:30:08  apache#26 0x561b8a2c56d4 in
doris::vectorized::PrioritizedSplitRunner::process()
/root/doris/be/src/vec/exec/executor/time_sharing/prioritized_split_runner.cpp:103:35
22:30:08  apache#27 0x561b8a29045c in
doris::vectorized::TimeSharingTaskExecutor::_dispatch_thread()
/root/doris/be/src/vec/exec/executor/time_sharing/time_sharing_task_executor.cpp:570:77
22:30:08  apache#28 0x561b7b9fecb6 in std::function<void ()>::operator()()
const
/usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593:9
22:30:08  apache#29 0x561b7b9fecb6 in doris::Thread::supervise_thread(void*)
/root/doris/be/src/util/thread.cpp:460:5
22:30:08  apache#30 0x561b76044d26 in asan_thread_start(void*)
(/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P1/Cluster0/be/lib/doris_be+0x23962d26)
22:30:08  apache#31 0x7f4aaae68608 in start_thread
/build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
22:30:08  apache#32 0x7f4aaad7b132 in __clone
/build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
kaka11chen pushed a commit that referenced this pull request Jan 19, 2026
@bobhan1
[Fix](mow) Fix potential use after free in CalcDeleteBitmapToken (a…
fbcca5e 
…pache#59920)

### What problem does this PR solve?
When `BaseBetaRowsetWriter` is destructed(this may happen when the load
is canceled) before the execution of the task submitted to thread pool,
the task may encounter coredump due to use after free.
```
(gdb) bt
#0  __GI___pthread_sigmask (how=2, newmask=<optimized out>, oldmask=0x0) at ./nptl/pthread_sigmask.c:43
#1  0x00007fa1d0c1171e in PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
#2  0x00007fa1d0c12206 in JVM_handle_linux_signal () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
#3  <signal handler called>
apache#4  doris::TUniqueId::TUniqueId (this=0x7f99955f2208, other51=...) at /home/zcp/repo_center/doris_branch-4.0/doris/gensrc/build/gen_cpp/Types_types.cpp:2571
apache#5  0x00005653d14008ca in doris::AttachTask::init (rc=..., this=<optimized out>) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/runtime/thread_context.cpp:29
apache#6  doris::AttachTask::AttachTask (this=<optimized out>, rc=...) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/runtime/thread_context.cpp:34
apache#7  0x00005653d0d05087 in doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}::operator()() const (this=0x7f9cdf302500)
    at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/olap/calc_delete_bitmap_executor.h:74
apache#8  std::__invoke_impl<void, doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}&>(std::__invoke_other, doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}&) (__f=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63
apache#9  std::__invoke_r<void, doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}&>(doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}&) (__fn=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:113
apache#10 std::_Function_handler<void (), doris::CalcDeleteBitmapToken::submit_func<doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0>(doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int)::$_0&&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (
    __functor=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292
apache#11 0x00005653d16392e5 in doris::ThreadPool::dispatch_thread (this=0x7fa120d9af00) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/util/threadpool.cpp:616
apache#12 0x00005653d162e38c in std::function<void ()>::operator()() const (this=0x7f99955f2208) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593
apache#13 doris::Thread::supervise_thread (arg=0x7fa0c0049110) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/util/thread.cpp:460
apache#14 0x00007fa1cfcacac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
apache#15 0x00007fa1cfd3e850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
```

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
kaka11chen pushed a commit that referenced this pull request Feb 5, 2026
@hubgeter
[Enhancement](explain)Display deleteFileNum for FileScanNode when exp…
cff565f 
…lain verbose (apache#60308)

### What problem does this PR solve?
Problem Summary:
This PR enhances the output of EXPLAIN VERBOSE for File Scan nodes by
adding the following metrics:
`dataFileNum=xxx, deleteFileNum=xxx, deleteSplitNum=xxx`
Especially useful for iceberg/paimon/hive acid

These metrics provide more visibility into the underlying file and split
layout, helping users better tune parameters and control query
performance.
Details:
`dataFileNum` : The number of distinct data files that need to be read.
This is not equivalent to the number of splits, since a single data file
can be divided into multiple splits.

`deleteFileNum` : The number of distinct delete files that need to be
read.

`deleteSplitNum` : Added because the relationship between data files and
delete files is many-to-many:
one data file may be associated with multiple delete files
one delete file may apply to multiple data files
Using deleteSplitNum / dataSplitNum, users can estimate the average
number of delete splits that need to be read per data split.

Example:
```
mysql> explain verbose select * from iceberg.format_v3.dv_test_1w;
+-----------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                                               |
+-----------------------------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                                                                                               |
|   OUTPUT EXPRS:                                                                                                                               |
|     id[#0]                                                                                                                                    |
|     grp[#1]                                                                                                                                   |
|     value[#2]                                                                                                                                 |
|     ts[#3]                                                                                                                                    |
|   PARTITION: RANDOM                                                                                                                           |
|                                                                                                                                               |
|   HAS_COLO_PLAN_NODE: false                                                                                                                   |
|                                                                                                                                               |
|   VRESULT SINK                                                                                                                                |
|      MYSQL_PROTOCOL                                                                                                                           |
|                                                                                                                                               |
|   0:VICEBERG_SCAN_NODE(32)                                                                                                                    |
|      table: iceberg.format_v3.dv_test_1w                                                                                                      |
|      inputSplitNum=220, totalFileSize=720774, scanRanges=220                                                                                  |
|      partition=0/0                                                                                                                            |
|      backends:                                                                                                                                |
|        1769590309070                                                                                                                          |
|          s3://warehouse/wh/format_v3/dv_test_1w/data/00004-51-fc462f9a-d42a-404d-adfc-c8d2781c8d04-0-00001.parquet start: 4 length: 2672      |
|          s3://warehouse/wh/format_v3/dv_test_1w/data/00003-50-fc462f9a-d42a-404d-adfc-c8d2781c8d04-0-00001.parquet start: 4 length: 2852      |
|          s3://warehouse/wh/format_v3/dv_test_1w/data/00000-47-fc462f9a-d42a-404d-adfc-c8d2781c8d04-0-00001.parquet start: 4 length: 2894      |
|          ... other 216 files ...                                                                                                              |
|          s3://warehouse/wh/format_v3/dv_test_1w/data/00001-48-fc462f9a-d42a-404d-adfc-c8d2781c8d04-0-00001.parquet start: 58397 length: 13894 |
|          dataFileNum=10, deleteFileNum=1 deleteSplitNum=220                                                                               |
|      cardinality=33334, numNodes=1                                                                                                            |
|      pushdown agg=NONE                                                                                                                        |
|      tuple ids: 0                                                                                                                             |
|                                                                                                                                               |
| Tuples:                                                                                                                                       |
| TupleDescriptor{id=0, tbl=dv_test_1w}                                                                                                         |
|   SlotDescriptor{id=0, col=id, colUniqueId=1, type=bigint, nullable=true, isAutoIncrement=false, subColPath=null, virtualColumn=null}         |
|   SlotDescriptor{id=1, col=grp, colUniqueId=2, type=int, nullable=true, isAutoIncrement=false, subColPath=null, virtualColumn=null}           |
|   SlotDescriptor{id=2, col=value, colUniqueId=3, type=int, nullable=true, isAutoIncrement=false, subColPath=null, virtualColumn=null}         |
|   SlotDescriptor{id=3, col=ts, colUniqueId=4, type=datetimev2(6), nullable=true, isAutoIncrement=false, subColPath=null, virtualColumn=null}  |
|                                                                                                                                               |
|                                                                                                                                               |
|                                                                                                                                               |
|                                                                                                                                               |
| ========== STATISTICS ==========                                                                                                              |
+-----------------------------------------------------------------------------------------------------------------------------------------------+
```
kaka11chen pushed a commit that referenced this pull request Mar 13, 2026
@linrrzqqq
[Fix](pythonUdf) Fix pythonUdf helper cmd coredump when `enable_pytho…
2a680f5 
…n_udf_support` is not set (apache#61196)

When `enable_python_udf_support` is not set to true, a null pointer
dereference occurs when invoking helper commands (`SHOW PYTHON VERSION/
PACKAGES IN`).

```text
/home/zcp/repo_center/doris_master/doris/be/src/udf/python/python_env.cpp:305:38: runtime error: member call on null pointer of type 'doris::PythonEnvScanner'
    #0 0x559edede162c in doris::PythonVersionManager::env_infos_to_thrift() const /home/zcp/repo_center/doris_master/doris/be/src/udf/python/python_env.cpp:305:38
    #1 0x559ede63acd4 in doris::BaseBackendService::get_python_envs(std::vector>&) /home/zcp/repo_center/doris_master/doris/be/src/service/backend_service.cpp:1316:47
    #2 0x559edf99796a in doris::BackendServiceProcessor::process_get_python_envs(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/BackendService.cpp:7789:13
    #3 0x559edf94a69c in doris::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string, std::allocator> const&, int, void*) /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/BackendService.cpp:6466:3
    apache#4 0x559eac6bad06 in apache::thrift::TDispatchProcessor::process(std::shared_ptr, std::shared_ptr, void*) /home/zcp/repo_center/doris_master/doris/thirdparty/installed/include/thrift/TDispatchProcessor.h:121:12
    apache#5 0x559ee47c071a in apache::thrift::server::TConnectedClient::run() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1c871a)
    apache#6 0x559ee47c1a45 in apache::thrift::server::TThreadedServer::TConnectedClientRunner::run() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1c9a45)
    apache#7 0x559ee47c557f in apache::thrift::concurrency::Thread::threadMain(std::shared_ptr) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd57f)
    apache#8 0x559ee47c52a5 in void std::__invoke_impl), std::shared_ptr>(std::__invoke_other, void (*&&)(std::shared_ptr), std::shared_ptr&&) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd2a5)
    apache#9 0x559ee47c521c in std::__invoke_result), std::shared_ptr>::type std::__invoke), std::shared_ptr>(void (*&&)(std::shared_ptr), std::shared_ptr&&) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd21c)
    apache#10 0x559ee47c51f1 in void std::thread::_Invoker), std::shared_ptr>>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd1f1)
    apache#11 0x559ee47c51b4 in std::thread::_Invoker), std::shared_ptr>>::operator()() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1cd1b4)
    apache#12 0x559ee47c4fd8 in std::thread::_State_impl), std::shared_ptr>>>::_M_run() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x8a1ccfd8)
    apache#13 0x559ef61b5f3f in execute_native_thread_routine archive64.c
    apache#14 0x559eac535d26 in asan_thread_start(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x51f3dd26)
    apache#15 0x7fcba62bfac2 in start_thread nptl/pthread_create.c:442:8
    apache#16 0x7fcba635184f  misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/zcp/repo_center/doris_master/doris/be/src/udf/python/python_env.cpp:305:38 
```

now:
```text
Doris> show python packages in '3.19.18';
ERROR 1105 (HY000): errCode = 2, detailMessage = Failed to get python packages from any backend: [E42] Set 'python_venv_interpreter_paths' in be.conf to enable PythonUDF feature
Doris> show python versions;
ERROR 1105 (HY000): errCode = 2, detailMessage = Failed to get python envs from any backend: [E-236] Set 'python_venv_interpreter_paths' in be.conf to enable PythonUDF feature
```
kaka11chen pushed a commit that referenced this pull request Mar 18, 2026
@wyxxxcat
[fix](sync_point) Fix heap-use-after-free in SyncPoint during program…
c96f348 
… exit (apache#61040)

### What problem does this PR solve?

The SyncPoint singleton was being destroyed during program exit while
background bthread timers were still accessing it, causing
heap-use-after-free.

Changed get_instance() to return a never-destructed heap-allocated
pointer
instead of a stack-allocated static object to prevent destruction order
issues.

This is a standard pattern for singletons accessed by background
threads.

```
11:47:40       #0 0x5582a963153d in operator new(unsigned long) (/root/doris/cloud/ut_build_ASAN/test/recycler_test+0x122b53d) (BuildId: 1191affa24589f52)
11:47:40       #1 0x5582ab0aa52f in doris::SyncPoint::SyncPoint() /root/doris/common/cpp/sync_point.cpp:77:9
11:47:40       #2 0x5582ab0aa46b in doris::SyncPoint::get_instance() /root/doris/common/cpp/sync_point.cpp:73:20
11:47:40       #3 0x5582aae33ee8 in doris::cloud::memkv::Transaction::get(std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::unique_ptr<doris::cloud::RangeGetIterator, std::default_delete<doris::cloud::RangeGetIterator>>*, bool, int) /root/doris/cloud/src/meta-store/mem_txn_kv.cpp:276:5
11:47:40       apache#4 0x5582a968f49f in doris::cloud::txn_get(doris::cloud::TxnKv*, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>, std::unique_ptr<doris::cloud::RangeGetIterator, std::default_delete<doris::cloud::RangeGetIterator>>&) /root/doris/cloud/src/recycler/recycler.cpp:99:18
11:47:40       apache#5 0x5582a9657090 in doris::cloud::InstanceRecycler::scan_and_recycle(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::basic_string_view<char, std::char_traits<char>>, std::function<int (std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>)>, std::function<int ()>) /root/doris/cloud/src/recycler/recycler.cpp:2976:23
11:47:40       apache#6 0x5582a9686153 in doris::cloud::InstanceRecycler::recycle_rowsets() /root/doris/cloud/src/recycler/recycler.cpp:2587:15
11:47:40       apache#7 0x5582a96a8d47 in doris::cloud::RecyclerTest_recycle_empty_Test::TestBody() /root/doris/cloud/test/recycler_test.cpp:1136:5
```
kaka11chen pushed a commit that referenced this pull request Mar 21, 2026
@hubgeter
[fix](orc)fix coredump because rewriteLeaves function heap-use-after-…
f467595 
…free. (apache#61138)

### What problem does this PR solve?
Related PR: apache/doris-thirdparty#383

Problem Summary:
fix coredump:
```
#0 0x5561f1d46f12 in operator delete(void*, unsigned long) (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x50ae4f12)
    #1 0x55622c94b454 in std::__new_allocator<std::shared_ptr<orc::ExpressionTree>>::deallocate(std::shared_ptr<orc::ExpressionTree>*, unsigned long) (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6e9454)
    #2 0x55622c94b41e in std::_Deque_base<std::shared_ptr<orc::ExpressionTree>, std::allocator<std::shared_ptr<orc::ExpressionTree>>>::_M_deallocate_node(std::shared_ptr<orc::ExpressionTree>*) (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6e941e)
    #3 0x55622c950681 in std::deque<std::shared_ptr<orc::ExpressionTree>, std::allocator<std::shared_ptr<orc::ExpressionTree>>>::_M_pop_front_aux() (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6ee681)
    apache#4 0x55622c94868e in std::deque<std::shared_ptr<orc::ExpressionTree>, std::allocator<std::shared_ptr<orc::ExpressionTree>>>::pop_front() (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6e668e)
    apache#5 0x55622c947d79 in orc::rewriteLeaves(std::shared_ptr<orc::ExpressionTree>, unsigned long*) SearchArgument.cc
    apache#6 0x55622c947809 in orc::SearchArgumentBuilderImpl::build() (/mnt/disk2/tengjianping/doris-master/output/be/lib/doris_be+0x8b6e5809)
    apache#7 0x5562195209a6 in doris::vectorized::OrcReader::_init_search_argument(std::vector<std::shared_ptr<doris::vectorized::VExpr>, std::allocator<std::shared_ptr<doris::vectorized::VExpr>>> const&) /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/format/orc/vorc_reader.cpp:1072:27
    apache#8 0x556219527175 in doris::vectorized::OrcReader::set_fill_columns(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::SlotDescriptor const*>, std::hash<std::__cxx11::basic_string<char, 
```
kaka11chen pushed a commit that referenced this pull request Mar 25, 2026
@w41ter
[fix](cloud) Fix txn kv destruction order
2bfca29 
```
==1472432==ERROR: AddressSanitizer: heap-use-after-free on address 0x7bc7535fba50 at pc 0x557ec5178739 bp 0x7b1743746d40 sp 0x7b1743746500
WRITE of size 104 at 0x7bc7535fba50 thread T2
    #0 0x557ec5178738 in memset ~/workspace/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc:87:3
    #1 0x7f17566e64fc in fdb_transaction_set_option (libfdb_c.so+0xc014fc) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
    #2 0x7f17566e2e47 in fdb_transaction_set_option (libfdb_c.so+0xbfde47) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
    #3 0x7f17566e4ec5 in fdb_transaction_set_option (libfdb_c.so+0xbffec5) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
    apache#4 0x7f17564f80ee in fdb_transaction_set_option (libfdb_c.so+0xa130ee) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
    apache#5 0x7f1755e55b71 in fdb_transaction_set_option (libfdb_c.so+0x370b71) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
    apache#6 0x7f1755e2b2e9 in fdb_run_network (libfdb_c.so+0x3462e9) (BuildId: 854ec989d11d3c183aa4963a6d758cfc2b571b07)
    apache#7 0x557ec6595a2b in doris::cloud::fdb::Network::init()::::operator()() const ~/workspace/doris/cloud/ut_build_ASAN/../src/meta-store/txn_kv.cpp:468:28
    apache#8 0x557ec65958e6 in void std::__invoke_impl<void, doris::cloud::fdb::Network::init()::>(std::__invoke_other, doris::cloud::fdb::Network::init()::&&) /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    apache#9 0x557ec6595886 in std::__invoke_result<doris::cloud::fdb::Network::init()::>::type std::__invoke<doris::cloud::fdb::Network::init()::>(doris::cloud::fdb::Network::init()::&&) /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14
    apache#10 0x557ec659584e in void std::thread::_Invoker<std::tuple<doris::cloud::fdb::Network::init()::>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:292:13
    apache#11 0x557ec6595816 in std::thread::_Invoker<std::tuple<doris::cloud::fdb::Network::init()::>>::operator()() /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:299:11
    apache#12 0x557ec659571a in std::thread::_State_impl<std::thread::_Invoker<std::tuple<doris::cloud::fdb::Network::init()::>>>::_M_run() /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_thread.h:244:13
    apache#13 0x557ec7756e43 in execute_native_thread_routine InstrProfilingUtil.c
    apache#14 0x557ec50b1a07 in asan_thread_start(void*) ~/workspace/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:239:43
    apache#15 0x7f17545caaa3 in start_thread nptl/pthread_create.c:447:8
    apache#16 0x7f1754657a63 in clone misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:100
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@suxiaogang223