Skip to content

Commit 7d13bb1

Browse files
platonekoluwei16deardeng
committed
Squashed commit of the following(only BE):
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <[email protected]> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <[email protected]> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <[email protected]> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <[email protected]> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <[email protected]> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <[email protected]> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <[email protected]> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <[email protected]> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <[email protected]> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <[email protected]> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <[email protected]> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <[email protected]> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <[email protected]> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <[email protected]> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <[email protected]> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <[email protected]> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <[email protected]> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <[email protected]> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <[email protected]> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <[email protected]> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <[email protected]> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <[email protected]> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <[email protected]> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <[email protected]> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <[email protected]> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <[email protected]> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <[email protected]> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <[email protected]> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <[email protected]> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <[email protected]> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <[email protected]> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <[email protected]> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <[email protected]> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <[email protected]> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <[email protected]> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <[email protected]> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <[email protected]> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <[email protected]> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <[email protected]> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <[email protected]> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <[email protected]> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <[email protected]> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <[email protected]> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <[email protected]> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734) Co-authored-by: Luwei <[email protected]> Co-authored-by: deardeng <[email protected]>
1 parent 006283c commit 7d13bb1

140 files changed

Lines changed: 3694 additions & 1352 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/CMakeLists.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -550,6 +550,7 @@ set(KRB5_LIBS
550550
set(AWS_LIBS
551551
aws-sdk-s3
552552
aws-sdk-core
553+
aws-sdk-transfer
553554
aws-checksums
554555
aws-c-io
555556
aws-c-event-stream
@@ -743,6 +744,7 @@ add_subdirectory(${SRC_DIR}/geo)
743744
add_subdirectory(${SRC_DIR}/io)
744745
add_subdirectory(${SRC_DIR}/gutil)
745746
add_subdirectory(${SRC_DIR}/http)
747+
add_subdirectory(${SRC_DIR}/io)
746748
add_subdirectory(${SRC_DIR}/olap)
747749
add_subdirectory(${SRC_DIR}/runtime)
748750
add_subdirectory(${SRC_DIR}/service)

‎be/src/agent/agent_server.cpp‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ AgentServer::AgentServer(ExecEnv* exec_env, const TMasterInfo& master_info)
3939
: _exec_env(exec_env), _master_info(master_info), _topic_subscriber(new TopicSubscriber()) {
4040
for (auto& path : exec_env->store_paths()) {
4141
try {
42-
string dpp_download_path_str = path.path + DPP_PREFIX;
42+
string dpp_download_path_str = path.path + "/" + DPP_PREFIX;
4343
std::filesystem::path dpp_download_path(dpp_download_path_str);
4444
if (std::filesystem::exists(dpp_download_path)) {
4545
std::filesystem::remove_all(dpp_download_path);
@@ -89,7 +89,8 @@ AgentServer::AgentServer(ExecEnv* exec_env, const TMasterInfo& master_info)
8989
CREATE_AND_START_THREAD(REPORT_DISK_STATE, _report_disk_state_workers);
9090
CREATE_AND_START_THREAD(REPORT_OLAP_TABLE, _report_tablet_workers);
9191
CREATE_AND_START_POOL(SUBMIT_TABLE_COMPACTION, _submit_table_compaction_workers);
92-
CREATE_AND_START_POOL(STORAGE_MEDIUM_MIGRATE_V2, _storage_medium_migrate_v2_workers);
92+
CREATE_AND_START_POOL(REFRESH_STORAGE_POLICY, _storage_refresh_policy_workers);
93+
CREATE_AND_START_POOL(UPDATE_STORAGE_POLICY, _storage_update_policy_workers);
9394
#undef CREATE_AND_START_POOL
9495
#undef CREATE_AND_START_THREAD
9596

@@ -152,8 +153,8 @@ void AgentServer::submit_tasks(TAgentResult& agent_result,
152153
HANDLE_TYPE(TTaskType::UPDATE_TABLET_META_INFO, _update_tablet_meta_info_workers,
153154
update_tablet_meta_info_req);
154155
HANDLE_TYPE(TTaskType::COMPACTION, _submit_table_compaction_workers, compaction_req);
155-
HANDLE_TYPE(TTaskType::STORAGE_MEDIUM_MIGRATE_V2, _storage_medium_migrate_v2_workers,
156-
storage_migration_req_v2);
156+
HANDLE_TYPE(TTaskType::NOTIFY_UPDATE_STORAGE_POLICY, _storage_update_policy_workers,
157+
update_policy);
157158

158159
case TTaskType::REALTIME_PUSH:
159160
case TTaskType::PUSH:

‎be/src/agent/agent_server.h‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -82,8 +82,8 @@ class AgentServer {
8282

8383
std::unique_ptr<TaskWorkerPool> _submit_table_compaction_workers;
8484

85-
std::unique_ptr<TaskWorkerPool> _storage_medium_migrate_v2_workers;
86-
85+
std::unique_ptr<TaskWorkerPool> _storage_refresh_policy_workers;
86+
std::unique_ptr<TaskWorkerPool> _storage_update_policy_workers;
8787
std::unique_ptr<TopicSubscriber> _topic_subscriber;
8888
};
8989

‎be/src/agent/task_worker_pool.cpp‎

Lines changed: 107 additions & 117 deletions
Original file line numberDiff line numberDiff line change
@@ -35,14 +35,14 @@
3535
#include "olap/olap_common.h"
3636
#include "olap/snapshot_manager.h"
3737
#include "olap/storage_engine.h"
38+
#include "olap/storage_policy_mgr.h"
3839
#include "olap/tablet.h"
3940
#include "olap/task/engine_alter_tablet_task.h"
4041
#include "olap/task/engine_batch_load_task.h"
4142
#include "olap/task/engine_checksum_task.h"
4243
#include "olap/task/engine_clone_task.h"
4344
#include "olap/task/engine_publish_version_task.h"
4445
#include "olap/task/engine_storage_migration_task.h"
45-
#include "olap/task/engine_storage_migration_task_v2.h"
4646
#include "olap/utils.h"
4747
#include "runtime/exec_env.h"
4848
#include "runtime/snapshot_loader.h"
@@ -193,9 +193,14 @@ void TaskWorkerPool::start() {
193193
cb = std::bind<void>(&TaskWorkerPool::_submit_table_compaction_worker_thread_callback,
194194
this);
195195
break;
196-
case TaskWorkerType::STORAGE_MEDIUM_MIGRATE_V2:
196+
case TaskWorkerType::REFRESH_STORAGE_POLICY:
197197
_worker_count = 1;
198-
cb = std::bind<void>(&TaskWorkerPool::_storage_medium_migrate_v2_worker_thread_callback,
198+
cb = std::bind<void>(
199+
&TaskWorkerPool::_storage_refresh_storage_policy_worker_thread_callback, this);
200+
break;
201+
case TaskWorkerType::UPDATE_STORAGE_POLICY:
202+
_worker_count = 1;
203+
cb = std::bind<void>(&TaskWorkerPool::_storage_update_storage_policy_worker_thread_callback,
199204
this);
200205
break;
201206
default:
@@ -361,6 +366,7 @@ void TaskWorkerPool::_create_tablet_worker_thread_callback() {
361366
TStatus task_status;
362367

363368
std::vector<TTabletInfo> finish_tablet_infos;
369+
LOG(INFO) << "create tablet: " << create_tablet_req;
364370
Status create_status = _env->storage_engine()->create_tablet(create_tablet_req);
365371
if (!create_status.ok()) {
366372
LOG(WARNING) << "create table failed. status: " << create_status
@@ -437,6 +443,11 @@ void TaskWorkerPool::_drop_tablet_worker_thread_callback() {
437443
StorageEngine::instance()->txn_manager()->force_rollback_tablet_related_txns(
438444
dropped_tablet->data_dir()->get_meta(), drop_tablet_req.tablet_id,
439445
drop_tablet_req.schema_hash, dropped_tablet->tablet_uid());
446+
// We remove remote rowset directly.
447+
// TODO(cyx): do remove in background
448+
if (drop_tablet_req.is_drop_table_or_partition) {
449+
dropped_tablet->remove_all_remote_rowsets();
450+
}
440451
} else {
441452
status_code = TStatusCode::NOT_FOUND;
442453
error_msgs.push_back(err);
@@ -854,8 +865,15 @@ void TaskWorkerPool::_update_tablet_meta_worker_thread_callback() {
854865
tablet->set_partition_id(tablet_meta_info.partition_id);
855866
break;
856867
case TTabletMetaType::INMEMORY:
857-
tablet->tablet_meta()->mutable_tablet_schema()->set_is_in_memory(
858-
tablet_meta_info.is_in_memory);
868+
if (tablet_meta_info.storage_policy.empty()) {
869+
tablet->tablet_meta()->mutable_tablet_schema()->set_is_in_memory(
870+
tablet_meta_info.is_in_memory);
871+
} else {
872+
LOG(INFO) << "set tablet cooldown resource "
873+
<< tablet_meta_info.storage_policy;
874+
tablet->tablet_meta()->set_cooldown_resource(
875+
tablet_meta_info.storage_policy);
876+
}
859877
break;
860878
}
861879
}
@@ -1036,8 +1054,8 @@ Status TaskWorkerPool::_check_migrate_request(const TStorageMediumMigrateReq& re
10361054
return Status::OLAPInternalError(OLAP_REQUEST_FAILED);
10371055
}
10381056

1039-
// check disk capacity
1040-
int64_t tablet_size = tablet->tablet_footprint();
1057+
// check local disk capacity
1058+
int64_t tablet_size = tablet->tablet_local_size();
10411059
if ((*dest_store)->reach_capacity_limit(tablet_size)) {
10421060
LOG(WARNING) << "reach the capacity limit of path: " << (*dest_store)->path()
10431061
<< ", tablet size: " << tablet_size;
@@ -1171,14 +1189,15 @@ void TaskWorkerPool::_report_disk_state_worker_thread_callback() {
11711189
map<string, TDisk> disks;
11721190
for (auto& root_path_info : data_dir_infos) {
11731191
TDisk disk;
1174-
disk.__set_root_path(root_path_info.path_desc.filepath);
1192+
disk.__set_root_path(root_path_info.path);
11751193
disk.__set_path_hash(root_path_info.path_hash);
11761194
disk.__set_storage_medium(root_path_info.storage_medium);
11771195
disk.__set_disk_total_capacity(root_path_info.disk_capacity);
1178-
disk.__set_data_used_capacity(root_path_info.data_used_capacity);
1196+
disk.__set_data_used_capacity(root_path_info.local_used_capacity);
1197+
disk.__set_remote_used_capacity(root_path_info.remote_used_capacity);
11791198
disk.__set_disk_available_capacity(root_path_info.available);
11801199
disk.__set_used(root_path_info.is_used);
1181-
disks[root_path_info.path_desc.filepath] = disk;
1200+
disks[root_path_info.path] = disk;
11821201
}
11831202
request.__set_disks(disks);
11841203
_handle_report(request, ReportType::DISK);
@@ -1565,7 +1584,6 @@ Status TaskWorkerPool::_move_dir(const TTabletId tablet_id, const std::string& s
15651584
return Status::InvalidArgument("Could not find tablet");
15661585
}
15671586

1568-
std::string dest_tablet_dir = tablet->tablet_path_desc().filepath;
15691587
SnapshotLoader loader(_env, job_id, tablet_id);
15701588
Status status = loader.move(src, tablet, overwrite);
15711589

@@ -1683,9 +1701,64 @@ void TaskWorkerPool::_submit_table_compaction_worker_thread_callback() {
16831701
}
16841702
}
16851703

1686-
void TaskWorkerPool::_storage_medium_migrate_v2_worker_thread_callback() {
1704+
void TaskWorkerPool::_storage_refresh_storage_policy_worker_thread_callback() {
1705+
while (_is_work) {
1706+
_is_doing_work = false;
1707+
// wait at most report_task_interval_seconds, or being notified
1708+
std::unique_lock<std::mutex> worker_thread_lock(_worker_thread_lock);
1709+
_worker_thread_condition_variable.wait_for(
1710+
worker_thread_lock,
1711+
std::chrono::seconds(config::storage_refresh_storage_policy_task_interval_seconds));
1712+
if (!_is_work) {
1713+
break;
1714+
}
1715+
1716+
if (_master_info.network_address.port == 0) {
1717+
// port == 0 means not received heartbeat yet
1718+
// sleep a short time and try again
1719+
LOG(INFO)
1720+
<< "waiting to receive first heartbeat from frontend before doing task report";
1721+
continue;
1722+
}
1723+
1724+
_is_doing_work = true;
1725+
1726+
TGetStoragePolicyResult result;
1727+
Status status = _master_client->refresh_storage_policy(&result);
1728+
if (!status.ok()) {
1729+
LOG(WARNING) << "refresh storage policy status not ok. ";
1730+
} else if (result.status.status_code != TStatusCode::OK) {
1731+
LOG(WARNING) << "refresh storage policy result status status_code not ok. ";
1732+
} else {
1733+
// update storage policy mgr.
1734+
StoragePolicyMgr* spm = ExecEnv::GetInstance()->storage_policy_mgr();
1735+
for (const auto& iter : result.result_entrys) {
1736+
shared_ptr<StoragePolicy> policy_ptr = make_shared<StoragePolicy>();
1737+
policy_ptr->storage_policy_name = iter.policy_name;
1738+
policy_ptr->cooldown_datetime = iter.cooldown_datetime;
1739+
policy_ptr->cooldown_ttl = iter.cooldown_ttl;
1740+
policy_ptr->s3_endpoint = iter.s3_storage_param.s3_endpoint;
1741+
policy_ptr->s3_region = iter.s3_storage_param.s3_region;
1742+
policy_ptr->s3_ak = iter.s3_storage_param.s3_ak;
1743+
policy_ptr->s3_sk = iter.s3_storage_param.s3_sk;
1744+
policy_ptr->root_path = iter.s3_storage_param.root_path;
1745+
policy_ptr->bucket = iter.s3_storage_param.bucket;
1746+
policy_ptr->s3_conn_timeout_ms = iter.s3_storage_param.s3_conn_timeout_ms;
1747+
policy_ptr->s3_max_conn = iter.s3_storage_param.s3_max_conn;
1748+
policy_ptr->s3_request_timeout_ms = iter.s3_storage_param.s3_request_timeout_ms;
1749+
policy_ptr->md5_sum = iter.md5_checksum;
1750+
1751+
LOG(INFO) << "refresh storage policy task, policy " << *policy_ptr;
1752+
spm->periodic_put(iter.policy_name, std::move(policy_ptr));
1753+
}
1754+
}
1755+
}
1756+
}
1757+
1758+
void TaskWorkerPool::_storage_update_storage_policy_worker_thread_callback() {
16871759
while (_is_work) {
16881760
TAgentTaskRequest agent_task_req;
1761+
TGetStoragePolicy get_storage_policy_req;
16891762
{
16901763
std::unique_lock<std::mutex> worker_thread_lock(_worker_thread_lock);
16911764
while (_is_work && _tasks.empty()) {
@@ -1696,115 +1769,32 @@ void TaskWorkerPool::_storage_medium_migrate_v2_worker_thread_callback() {
16961769
}
16971770

16981771
agent_task_req = _tasks.front();
1772+
get_storage_policy_req = agent_task_req.update_policy;
16991773
_tasks.pop_front();
17001774
}
1701-
int64_t signature = agent_task_req.signature;
1702-
LOG(INFO) << "get migration table v2 task, signature: " << agent_task_req.signature;
1703-
bool is_task_timeout = false;
1704-
if (agent_task_req.__isset.recv_time) {
1705-
int64_t time_elapsed = time(nullptr) - agent_task_req.recv_time;
1706-
if (time_elapsed > config::report_task_interval_seconds * 20) {
1707-
LOG(INFO) << "task elapsed " << time_elapsed
1708-
<< " seconds since it is inserted to queue, it is timeout";
1709-
is_task_timeout = true;
1710-
}
1711-
}
1712-
if (!is_task_timeout) {
1713-
TFinishTaskRequest finish_task_request;
1714-
TTaskType::type task_type = agent_task_req.task_type;
1715-
switch (task_type) {
1716-
case TTaskType::STORAGE_MEDIUM_MIGRATE_V2:
1717-
_storage_medium_migrate_v2(agent_task_req, signature, task_type,
1718-
&finish_task_request);
1719-
break;
1720-
default:
1721-
// pass
1722-
break;
1723-
}
1724-
_finish_task(finish_task_request);
1725-
}
1726-
_remove_task_info(agent_task_req.task_type, agent_task_req.signature);
1727-
}
1728-
}
1729-
1730-
void TaskWorkerPool::_storage_medium_migrate_v2(const TAgentTaskRequest& agent_task_req,
1731-
int64_t signature, const TTaskType::type task_type,
1732-
TFinishTaskRequest* finish_task_request) {
1733-
Status status = Status::OK();
1734-
TStatus task_status;
1735-
std::vector<string> error_msgs;
1736-
1737-
string process_name;
1738-
switch (task_type) {
1739-
case TTaskType::STORAGE_MEDIUM_MIGRATE_V2:
1740-
process_name = "StorageMediumMigrationV2";
1741-
break;
1742-
default:
1743-
std::string task_name;
1744-
EnumToString(TTaskType, task_type, task_name);
1745-
LOG(WARNING) << "Storage medium migration v2 type invalid. type: " << task_name
1746-
<< ", signature: " << signature;
1747-
status = Status::NotSupported("Storage medium migration v2 type invalid");
1748-
break;
1749-
}
1750-
1751-
// Check last storage medium migration v2 status, if failed delete tablet file
1752-
// Do not need to adjust delete success or not
1753-
// Because if delete failed task will failed
1754-
TTabletId new_tablet_id;
1755-
TSchemaHash new_schema_hash = 0;
1756-
if (status.ok()) {
1757-
new_tablet_id = agent_task_req.storage_migration_req_v2.new_tablet_id;
1758-
new_schema_hash = agent_task_req.storage_migration_req_v2.new_schema_hash;
1759-
EngineStorageMigrationTaskV2 engine_task(agent_task_req.storage_migration_req_v2);
1760-
Status sc_status = _env->storage_engine()->execute_task(&engine_task);
1761-
if (!sc_status.ok()) {
1762-
if (sc_status.precise_code() == OLAP_ERR_DATA_QUALITY_ERR) {
1763-
error_msgs.push_back("The data quality does not satisfy, please check your data. ");
1764-
}
1765-
status = sc_status;
1766-
} else {
1767-
status = Status::OK();
1768-
}
1769-
}
17701775

1771-
if (status.ok()) {
1772-
++_s_report_version;
1773-
LOG(INFO) << process_name << " finished. signature: " << signature;
1774-
}
1775-
1776-
// Return result to fe
1777-
finish_task_request->__set_backend(_backend);
1778-
finish_task_request->__set_report_version(_s_report_version);
1779-
finish_task_request->__set_task_type(task_type);
1780-
finish_task_request->__set_signature(signature);
1781-
1782-
std::vector<TTabletInfo> finish_tablet_infos;
1783-
if (status.ok()) {
1784-
TTabletInfo tablet_info;
1785-
status = _get_tablet_info(new_tablet_id, new_schema_hash, signature, &tablet_info);
1786-
1787-
if (!status.ok()) {
1788-
LOG(WARNING) << process_name << " success, but get new tablet info failed."
1789-
<< "tablet_id: " << new_tablet_id << ", schema_hash: " << new_schema_hash
1790-
<< ", signature: " << signature;
1791-
} else {
1792-
finish_tablet_infos.push_back(tablet_info);
1793-
}
1794-
}
1795-
1796-
if (status.ok()) {
1797-
finish_task_request->__set_finish_tablet_infos(finish_tablet_infos);
1798-
LOG(INFO) << process_name << " success. signature: " << signature;
1799-
error_msgs.push_back(process_name + " success");
1800-
} else {
1801-
LOG(WARNING) << process_name << " failed. signature: " << signature;
1802-
error_msgs.push_back(process_name + " failed");
1803-
error_msgs.push_back("status: " + status.to_string());
1776+
StoragePolicyMgr* spm = ExecEnv::GetInstance()->storage_policy_mgr();
1777+
shared_ptr<StoragePolicy> policy_ptr = make_shared<StoragePolicy>();
1778+
policy_ptr->storage_policy_name = get_storage_policy_req.policy_name;
1779+
policy_ptr->cooldown_datetime = get_storage_policy_req.cooldown_datetime;
1780+
policy_ptr->cooldown_ttl = get_storage_policy_req.cooldown_ttl;
1781+
policy_ptr->s3_endpoint = get_storage_policy_req.s3_storage_param.s3_endpoint;
1782+
policy_ptr->s3_region = get_storage_policy_req.s3_storage_param.s3_region;
1783+
policy_ptr->s3_ak = get_storage_policy_req.s3_storage_param.s3_ak;
1784+
policy_ptr->s3_sk = get_storage_policy_req.s3_storage_param.s3_sk;
1785+
policy_ptr->root_path = get_storage_policy_req.s3_storage_param.root_path;
1786+
policy_ptr->bucket = get_storage_policy_req.s3_storage_param.bucket;
1787+
policy_ptr->s3_conn_timeout_ms = get_storage_policy_req.s3_storage_param.s3_conn_timeout_ms;
1788+
policy_ptr->s3_max_conn = get_storage_policy_req.s3_storage_param.s3_max_conn;
1789+
policy_ptr->s3_request_timeout_ms =
1790+
get_storage_policy_req.s3_storage_param.s3_request_timeout_ms;
1791+
policy_ptr->md5_sum = get_storage_policy_req.md5_checksum;
1792+
1793+
LOG(INFO) << "get storage update policy task, update policy " << *policy_ptr;
1794+
1795+
spm->update(get_storage_policy_req.policy_name, std::move(policy_ptr));
1796+
_remove_task_info(agent_task_req.task_type, agent_task_req.signature);
18041797
}
1805-
task_status.__set_status_code(status.code());
1806-
task_status.__set_error_msgs(error_msgs);
1807-
finish_task_request->__set_task_status(task_status);
18081798
}
18091799

18101800
} // namespace doris

0 commit comments

Comments
 (0)