Skip to content

Commit 74e1260

Browse files
committed
[chore](conf) Remove deprecated/unused FE/BE configs and stale session variables
Issue Number: close #xxx Related PR: #xxx Problem Summary: Doris has accumulated a number of FE configs, BE configs and session variables that are either explicitly marked `@Deprecated`, or simply have no remaining references anywhere in the codebase (production sources, regression tests, docs, or tools). They add noise to the user-facing configuration surface and to the source tree without changing any behavior. This commit removes them in three coordinated parts and adds the minimum amount of backward-compatibility plumbing required to keep a BE-then-FE rolling upgrade safe. 1. FE configs (fe/fe-common/.../Config.java) Remove 13 `@Deprecated` fields with no remaining production references: `meta_publish_timeout_ms`, `backup_plugin_path`, `max_unfinished_load_job`, `use_new_tablet_scheduler`, `enable_concurrent_update`, `cbo_max_statistics_job_num`, `max_cbo_statistics_task_timeout_sec`, `cbo_concurrency_statistics_task_num`, `cbo_default_sample_percentage`, `finish_job_max_saved_second`, `enable_array_type`, `period_analyze_simultaneously_running_task_num`, `maximum_parallelism_of_export_job`. `ConfigBase.java` gains a small `warnDeprecatedKeys()` helper. After `setFields()` runs, if any of the legacy keys above is still listed in `fe.conf`, FE logs a single WARN line so operators notice the stale entry on upgrade. FE startup is not affected: the FE config loader iterates over Java fields, so unknown keys in `fe.conf` were already silently ignored before this change. 2. BE configs (be/src/common/config.{cpp,h}, be/src/cloud/config.{cpp,h}) Remove 24 configs that have zero references in the BE source tree: `doris_scanner_max_run_time_ms`, `multi_get_max_threads`, `num_disks`, `read_size`, `load_max_wg_active_memtable_percent`, `jdbc_connection_pool_cache_clear_time_sec`, `big_column_size_buffer`, `small_column_size_buffer`, `max_fragment_start_wait_time_seconds`, `enable_workload_group_for_scan`, `workload_group_scan_task_wait_timeout_ms`, `variant_use_cloud_schema_dict_cache`, `variant_threshold_rows_to_estimate_sparse_column`, `variant_nested_group_max_depth`, `enable_ttl_cache_evict_using_lru`, `file_cache_background_ttl_gc_batch`, `kerberos_refresh_interval_second`, `enable_debug_log_timeout_secs`, `enable_index_compaction`, `schema_dict_cache_capacity`, `test_s3_resource`, `meta_service_use_load_balancer`, `meta_service_rpc_timeout_ms`, `delete_bitmap_rpc_retry_times`. The BE config loader (`be/src/common/config.cpp` `init()`) also iterates over the registered field map, so unknown keys in `be.conf` were already silently ignored — hard removal is safe on a rolling upgrade. None of the removed options affect on-disk data format or the BE↔BE / BE↔FE wire format, so the standard BE-first rolling upgrade ordering is preserved. 3. Session variables (fe/fe-core/.../qe/SessionVariable.java, .../qe/VariableMgr.java) Remove 15 stale Doris-specific session variables along with their public name constants and any getter/setter methods: `use_v2_rollup`, `rewrite_count_distinct_to_bitmap_hll`, `enable_variant_access_in_original_planner`, `extract_wide_range_expr`, `auto_broadcast_join_threshold`, `runtime_filters_max_num`, `disable_inverted_index_v1_for_variant`, `enable_infer_predicate`, `limit_rows_for_single_instance`, `nereids_star_schema_support`, `enable_cbo_statistics`, `enable_eliminate_sort_node`, `drop_table_if_ctas_failed`, `trace_nereids`, `enable_sync_mv_cost_based_rewrite`. Selection criteria for each removed variable: zero references in `fe/fe-core/src/main/java` and `fe/fe-common/src/main/java` outside `SessionVariable.java` itself (after expanding the search to cover `get`/`set`/`is`-prefixed accessors of the field name, the public name constant, the underlying lowercase string literal, the body of `toThrift()`, and the body of `setForwardedSessionVariables()`), and zero hits in `regression-test/`, `docs/`, `tools/`, and `be/test/`. Upgrade compatibility is the critical concern here. Unlike fe.conf and be.conf, unknown session-variable names cause `VariableMgr` to throw `ERR_UNKNOWN_SYSTEM_VARIABLE` at four sites (`setVar`, `setVarForNonMasterFE`, `fillValue`, `getValue`). A hard delete would therefore break existing user scripts and JDBC client connection init paths that still issue `SET` on the old names during a BE-then-FE rolling upgrade. To stay safe, `VariableMgr` now keeps an explicit `REMOVED_SESSION_VAR_NAMES` set listing the 15 removed lowercase names; each of the four lookup sites short-circuits on this set so that `SET <removed_var> = ...` silently no-ops and `SELECT @@<removed_var>` returns an empty string. The same pattern is already used for `mysql_compat_var_whitelist`. The legacy FE configs and BE configs listed above have been removed. The following session variables have been removed; `SET` and `SELECT @@` against them now silently succeed (with empty/no-op behavior) instead of returning the variable's previous default: `use_v2_rollup`, `rewrite_count_distinct_to_bitmap_hll`, `enable_variant_access_in_original_planner`, `extract_wide_range_expr`, `auto_broadcast_join_threshold`, `runtime_filters_max_num`, `disable_inverted_index_v1_for_variant`, `enable_infer_predicate`, `limit_rows_for_single_instance`, `nereids_star_schema_support`, `enable_cbo_statistics`, `enable_eliminate_sort_node`, `drop_table_if_ctas_failed`, `trace_nereids`, `enable_sync_mv_cost_based_rewrite`. If any of the removed FE keys is still present in `fe.conf` on upgrade, FE logs a WARN line on startup.
1 parent 7fc8e27 commit 74e1260

8 files changed

Lines changed: 131 additions & 346 deletions

File tree

‎be/src/cloud/config.cpp‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,6 @@ DEFINE_String(deploy_mode, "");
2525
DEFINE_mString(cloud_unique_id, "");
2626
DEFINE_mString(meta_service_endpoint, "");
2727
DEFINE_mBool(enable_meta_service_endpoint_consistency_check, "true");
28-
DEFINE_Bool(meta_service_use_load_balancer, "false");
29-
DEFINE_mInt32(meta_service_rpc_timeout_ms, "10000");
3028
DEFINE_Bool(meta_service_connection_pooled, "true");
3129
DEFINE_mInt64(meta_service_connection_pool_size, "20");
3230
DEFINE_mInt32(meta_service_connection_age_base_seconds, "30");
@@ -107,8 +105,6 @@ DEFINE_mBool(enable_use_cloud_unique_id_from_fe, "true");
107105

108106
DEFINE_mBool(enable_cloud_tablet_report, "true");
109107

110-
DEFINE_mInt32(delete_bitmap_rpc_retry_times, "25");
111-
112108
DEFINE_mInt64(meta_service_rpc_reconnect_interval_ms, "100");
113109

114110
DEFINE_mInt32(meta_service_conflict_error_retry_times, "10");

‎be/src/cloud/config.h‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,6 @@ DECLARE_mInt32(meta_service_connection_age_base_seconds);
5858
//
5959
// Only works when meta_service_endpoint is set to a single host.
6060
DECLARE_mInt32(meta_service_idle_connection_timeout_ms);
61-
DECLARE_mInt32(meta_service_rpc_timeout_ms);
6261
DECLARE_mInt32(meta_service_rpc_retry_times);
6362
// default brpc timeout
6463
DECLARE_mInt32(meta_service_brpc_timeout_ms);
@@ -147,8 +146,6 @@ DECLARE_mBool(enable_use_cloud_unique_id_from_fe);
147146

148147
DECLARE_Bool(enable_cloud_tablet_report);
149148

150-
DECLARE_mInt32(delete_bitmap_rpc_retry_times);
151-
152149
DECLARE_mInt64(meta_service_rpc_reconnect_interval_ms);
153150

154151
DECLARE_mInt32(meta_service_conflict_error_retry_times);

‎be/src/common/config.cpp‎

Lines changed: 40 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
#include <memory>
3535
#include <mutex>
3636
#include <random>
37+
#include <set>
3738
#include <string>
3839
#include <string_view>
3940
#include <utility>
@@ -343,8 +344,6 @@ DEFINE_mInt32(doris_scan_range_max_mb, "1024");
343344
DEFINE_mInt32(doris_scanner_row_num, "16384");
344345
// single read execute fragment row bytes
345346
DEFINE_mInt32(doris_scanner_row_bytes, "10485760");
346-
// single read execute fragment max run time millseconds
347-
DEFINE_mInt32(doris_scanner_max_run_time_ms, "1000");
348347
DEFINE_mInt32(doris_scanner_dynamic_interval_ms, "100");
349348
// (Advanced) Maximum size of per-query receive-side buffer
350349
DEFINE_mInt32(exchg_node_buffer_size_bytes, "20485760");
@@ -509,9 +508,6 @@ DEFINE_mInt64(cumulative_compaction_min_deltas, "5");
509508
DEFINE_mInt64(cumulative_compaction_max_deltas, "1000");
510509
DEFINE_mInt32(cumulative_compaction_max_deltas_factor, "10");
511510

512-
// This config can be set to limit thread number in multiget thread pool.
513-
DEFINE_mInt32(multi_get_max_threads, "10");
514-
515511
// The upper limit of "permits" held by all compaction tasks. This config can be set to limit memory consumption for compaction.
516512
DEFINE_mInt64(total_permits_for_compaction_score, "1000000");
517513

@@ -686,13 +682,10 @@ DEFINE_Int32(fragment_mgr_async_work_pool_thread_num_min, "16");
686682
DEFINE_Int32(fragment_mgr_async_work_pool_thread_num_max, "2048");
687683
DEFINE_Int32(fragment_mgr_async_work_pool_queue_size, "4096");
688684

689-
// Control the number of disks on the machine. If 0, this comes from the system settings.
690-
DEFINE_Int32(num_disks, "0");
691685
// The read size is the size of the reads sent to os.
692686
// There is a trade off of latency and throughout, trying to keep disks busy but
693687
// not introduce seeks. The literature seems to agree that with 8 MB reads, random
694688
// io and sequential io perform similarly.
695-
DEFINE_Int32(read_size, "8388608"); // 8 * 1024 * 1024, Read Size (in bytes)
696689
DEFINE_Int32(min_buffer_size, "1024"); // 1024, The minimum read buffer size (in bytes)
697690

698691
// for pprof
@@ -748,9 +741,6 @@ DEFINE_Int32(load_process_soft_mem_limit_percent, "80");
748741
// memtable memory limiter will do nothing.
749742
DEFINE_Int32(load_process_safe_mem_permit_percent, "5");
750743

751-
// If there are a lot of memtable memory, then wait them flush finished.
752-
DEFINE_mDouble(load_max_wg_active_memtable_percent, "0.6");
753-
754744
// result buffer cancelled time (unit: second)
755745
DEFINE_mInt32(result_buffer_cancelled_interval_time, "300");
756746

@@ -978,9 +968,6 @@ DEFINE_Int32(send_batch_thread_pool_queue_size, "102400");
978968
DEFINE_mInt32(max_segment_num_per_rowset, "1000");
979969
DEFINE_mInt32(segment_compression_threshold_kb, "256");
980970

981-
// Time to clean up useless JDBC connection pool cache
982-
DEFINE_mInt32(jdbc_connection_pool_cache_clear_time_sec, "28800");
983-
984971
// Global bitmap cache capacity for aggregation cache, size in bytes
985972
DEFINE_Int64(delete_bitmap_agg_cache_capacity, "104857600");
986973
// The default delete bitmap cache is set to 100MB,
@@ -1054,9 +1041,6 @@ DEFINE_mInt32(merged_hdfs_min_io_size, "8192");
10541041

10551042
// OrcReader
10561043
DEFINE_mInt32(orc_natural_read_size_mb, "8");
1057-
DEFINE_mInt64(big_column_size_buffer, "65535");
1058-
DEFINE_mInt64(small_column_size_buffer, "100");
1059-
10601044
// Perform the always_true check at intervals determined by runtime_filter_sampling_frequency
10611045
DEFINE_mInt32(runtime_filter_sampling_frequency, "32");
10621046
DEFINE_mInt32(execution_max_rpc_timeout_sec, "3600");
@@ -1090,13 +1074,6 @@ DEFINE_mInt64(nodechannel_pending_queue_max_bytes, "67108864");
10901074
// The batch size for sending data by brpc streaming client
10911075
DEFINE_mInt64(brpc_streaming_client_batch_bytes, "262144");
10921076

1093-
// Max waiting time to wait the "plan fragment start" rpc.
1094-
// If timeout, the fragment will be cancelled.
1095-
// This parameter is usually only used when the FE loses connection,
1096-
// and the BE can automatically cancel the relevant fragment after the timeout,
1097-
// so as to avoid occupying the execution thread for a long time.
1098-
DEFINE_mInt32(max_fragment_start_wait_time_seconds, "30");
1099-
11001077
DEFINE_mInt32(fragment_mgr_cancel_worker_interval_seconds, "1");
11011078

11021079
// Node role tag for backend. Mix role is the default role, and computation role have no
@@ -1156,21 +1133,11 @@ DEFINE_Bool(enable_debug_points, "false");
11561133

11571134
DEFINE_Int32(pipeline_executor_size, "0");
11581135
DEFINE_Int32(blocking_pipeline_executor_size, "0");
1159-
DEFINE_Bool(enable_workload_group_for_scan, "false");
1160-
DEFINE_mInt64(workload_group_scan_task_wait_timeout_ms, "10000");
1161-
1162-
// Whether use schema dict in backend side instead of MetaService side(cloud mode)
1163-
DEFINE_mBool(variant_use_cloud_schema_dict_cache, "true");
1164-
DEFINE_mInt64(variant_threshold_rows_to_estimate_sparse_column, "2048");
11651136
DEFINE_mInt32(variant_max_json_key_length, "255");
11661137
DEFINE_mBool(variant_throw_exeception_on_invalid_json, "false");
11671138
DEFINE_mBool(variant_enable_duplicate_json_path_check, "false");
11681139
DEFINE_mBool(enable_vertical_compact_variant_subcolumns, "true");
11691140
DEFINE_mBool(enable_variant_doc_sparse_write_subcolumns, "true");
1170-
// Maximum depth of nested arrays to track with NestedGroup
1171-
// Reserved for future use when NestedGroup expansion moves to storage layer
1172-
// Deeper arrays will be stored as JSONB
1173-
DEFINE_mInt32(variant_nested_group_max_depth, "3");
11741141
DEFINE_mBool(variant_nested_group_discard_scalar_on_conflict, "false");
11751142

11761143
DEFINE_Validator(variant_max_json_key_length,
@@ -1207,9 +1174,6 @@ DEFINE_mInt64(file_cache_evict_in_advance_recycle_keys_num_threshold, "1000");
12071174

12081175
DEFINE_mBool(enable_read_cache_file_directly, "false");
12091176
DEFINE_mBool(file_cache_enable_evict_from_other_queue_by_size, "true");
1210-
// If true, evict the ttl cache using LRU when full.
1211-
// Otherwise, only expiration can evict ttl and new data won't add to cache when full.
1212-
DEFINE_Bool(enable_ttl_cache_evict_using_lru, "true");
12131177
DEFINE_mBool(enbale_dump_error_file, "false");
12141178
// limit the max size of error log on disk
12151179
DEFINE_mInt64(file_cache_error_log_limit_bytes, "209715200"); // 200MB
@@ -1240,7 +1204,6 @@ DEFINE_mInt64(file_cache_background_monitor_interval_ms, "5000");
12401204
DEFINE_mInt64(file_cache_background_ttl_gc_interval_ms, "180000");
12411205
DEFINE_mInt64(file_cache_background_ttl_info_update_interval_ms, "180000");
12421206
DEFINE_mInt64(file_cache_background_tablet_id_flush_interval_ms, "1000");
1243-
DEFINE_mInt64(file_cache_background_ttl_gc_batch, "1000");
12441207
DEFINE_mInt64(file_cache_background_lru_dump_interval_ms, "60000");
12451208
// dump queue only if the queue update specific times through several dump intervals
12461209
DEFINE_mInt64(file_cache_background_lru_dump_update_cnt_threshold, "1000");
@@ -1342,9 +1305,6 @@ DEFINE_mBool(allow_zero_date, "false");
13421305
DEFINE_Bool(allow_invalid_decimalv2_literal, "false");
13431306
DEFINE_mString(kerberos_ccache_path, "/tmp/");
13441307
DEFINE_mString(kerberos_krb5_conf_path, "/etc/krb5.conf");
1345-
// Deprecated
1346-
DEFINE_mInt32(kerberos_refresh_interval_second, "43200");
1347-
13481308
// JDK-8153057: avoid StackOverflowError thrown from the UncaughtExceptionHandler in thread "process reaper"
13491309
DEFINE_mBool(jdk_process_reaper_use_default_stack_size, "true");
13501310

@@ -1460,9 +1420,6 @@ DEFINE_mBool(enable_column_type_check, "true");
14601420
// 128 MB
14611421
DEFINE_mInt64(local_exchange_buffer_mem_limit, "134217728");
14621422

1463-
// Default 300s, if its value <= 0, then log is disabled
1464-
DEFINE_mInt64(enable_debug_log_timeout_secs, "0");
1465-
14661423
// Tolerance for the number of partition id 0 in rowset, default 0
14671424
DEFINE_Int32(ignore_invalid_partition_id_rowset_num, "0");
14681425

@@ -1578,8 +1535,6 @@ DEFINE_Validator(paimon_file_system_scheme_mappings,
15781535

15791536
DEFINE_mInt32(thrift_client_open_num_tries, "1");
15801537

1581-
DEFINE_Bool(enable_index_compaction, "false");
1582-
15831538
// http scheme in S3Client to use. E.g. http or https
15841539
DEFINE_String(s3_client_http_scheme, "http");
15851540
DEFINE_Validator(s3_client_http_scheme, [](const std::string& config) -> bool {
@@ -1693,8 +1648,6 @@ DEFINE_mInt32(max_automatic_compaction_num_per_round, "64");
16931648
DEFINE_mInt32(check_tablet_delete_bitmap_interval_seconds, "300");
16941649
DEFINE_mInt32(check_tablet_delete_bitmap_score_top_n, "10");
16951650
DEFINE_mBool(enable_check_tablet_delete_bitmap_score, "true");
1696-
DEFINE_mInt32(schema_dict_cache_capacity, "4096");
1697-
16981651
// whether to prune rows with delete sign = 1 in base compaction
16991652
// ATTN: this config is only for test
17001653
DEFINE_mBool(enable_prune_delete_sign_when_base_compaction, "true");
@@ -1787,8 +1740,6 @@ DEFINE_mInt32(file_handles_deplenish_frequency_times, "3");
17871740

17881741
// clang-format off
17891742
#ifdef BE_TEST
1790-
// test s3
1791-
DEFINE_String(test_s3_resource, "resource");
17921743
DEFINE_String(test_s3_ak, "ak");
17931744
DEFINE_String(test_s3_sk, "sk");
17941745
DEFINE_String(test_s3_endpoint, "endpoint");
@@ -2095,6 +2046,37 @@ std::ostream& operator<<(std::ostream& out, const std::vector<T>& v) {
20952046
continue; \
20962047
}
20972048

2049+
// Keys that were previously valid BE config entries but have been removed.
2050+
// If a user still has any of these in be.conf / be_custom.conf after upgrade,
2051+
// we emit a warning so they know the value is no longer honored, but BE will
2052+
// continue to start.
2053+
static const std::set<std::string> kDeprecatedConfigKeys = {
2054+
"doris_scanner_max_run_time_ms",
2055+
"multi_get_max_threads",
2056+
"num_disks",
2057+
"read_size",
2058+
"load_max_wg_active_memtable_percent",
2059+
"jdbc_connection_pool_cache_clear_time_sec",
2060+
"big_column_size_buffer",
2061+
"small_column_size_buffer",
2062+
"max_fragment_start_wait_time_seconds",
2063+
"enable_workload_group_for_scan",
2064+
"workload_group_scan_task_wait_timeout_ms",
2065+
"variant_use_cloud_schema_dict_cache",
2066+
"variant_threshold_rows_to_estimate_sparse_column",
2067+
"variant_nested_group_max_depth",
2068+
"enable_ttl_cache_evict_using_lru",
2069+
"file_cache_background_ttl_gc_batch",
2070+
"kerberos_refresh_interval_second",
2071+
"enable_debug_log_timeout_secs",
2072+
"enable_index_compaction",
2073+
"schema_dict_cache_capacity",
2074+
"test_s3_resource",
2075+
"meta_service_use_load_balancer",
2076+
"meta_service_rpc_timeout_ms",
2077+
"delete_bitmap_rpc_retry_times",
2078+
};
2079+
20982080
// init conf fields
20992081
bool init(const char* conf_file, bool fill_conf_map, bool must_exist, bool set_to_default) {
21002082
Properties props;
@@ -2123,6 +2105,14 @@ bool init(const char* conf_file, bool fill_conf_map, bool must_exist, bool set_t
21232105
SET_FIELD(it.second, std::vector<std::string>, fill_conf_map, set_to_default);
21242106
}
21252107

2108+
// Warn about any deprecated keys still present in the conf file.
2109+
for (const auto& kv : props.conf_map()) {
2110+
if (kDeprecatedConfigKeys.count(kv.first) > 0) {
2111+
LOG(WARNING) << "BE config '" << kv.first << "' in " << conf_file
2112+
<< " is deprecated and no longer takes effect; please remove it.";
2113+
}
2114+
}
2115+
21262116
if (config::is_cloud_mode()) {
21272117
auto st = config::set_config("enable_file_cache", "true", true, true);
21282118
LOG(INFO) << "set config enable_file_cache "

0 commit comments

Comments
 (0)