[Feature](iceberg-writer) Implements iceberg sink basic functionality for inserting into table. by kaka11chen · Pull Request #34929 · apache/doris

kaka11chen · 2024-05-15T13:18:54Z

Proposed changes

[Feature] (iceberg-writer) Implements iceberg sink basic functionality for inserting into table.

Currently only insert into and CTAS are supported. Among them, insert into supports unpartitioned tables and partitioned tables, but partitioned tables only support identity transform partitioned tables.

Future work:

Complete partitioned table transform functionality.
Sorting function.
Insert overwrite function.
More features like update, delete and merge.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-15T13:19:06Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 89. Check the log or trigger a new build to see more.

github-actions · 2024-05-20T05:22:44Z

+namespace doris {
+namespace iceberg {


warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]

Suggested change

namespace doris {

namespace iceberg {

namespace doris::iceberg {

be/src/vec/exec/format/table/iceberg/partition_spec.cpp:69:

- } // namespace iceberg - } // namespace doris + } // namespace doris

github-actions · 2024-05-20T05:22:45Z

+namespace doris {
+namespace iceberg {


warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]

Suggested change

namespace doris {

namespace iceberg {

namespace doris::iceberg {

be/src/vec/exec/format/table/iceberg/partition_spec.h:93:

- } // namespace iceberg - } // namespace doris + } // namespace doris

github-actions · 2024-05-20T05:22:45Z

+    private:
+        int next_field_id() { return ++_last_assigned_field_id; }
+
+    private:


warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]

Suggested change

private:

Additional context

be/src/vec/exec/format/table/iceberg/partition_spec.h:63: previously declared here

private: ^

github-actions · 2024-05-20T05:22:45Z

+
+    private:
+        std::shared_ptr<Schema> _schema;
+        std::vector<PartitionField> _fields;


warning: no template named 'vector' in namespace 'std' [clang-diagnostic-error]

std::vector<PartitionField> _fields; ^

github-actions · 2024-05-20T05:22:45Z

+        std::atomic<int> _last_assigned_field_id;
+    };
+
+    PartitionSpec(std::shared_ptr<Schema> schema, int spec_id, std::vector<PartitionField> fields,


warning: no template named 'vector' in namespace 'std' [clang-diagnostic-error]

PartitionSpec(std::shared_ptr<Schema> schema, int spec_id, std::vector<PartitionField> fields, ^

github-actions · 2024-05-20T05:22:46Z

+
+private:
+    static std::unique_ptr<Type> _type_from_json(const rapidjson::Value& value);
+    static std::unique_ptr<StructType> _struct_from_json(const rapidjson::Value& value);


warning: use of undeclared identifier 'std' [clang-diagnostic-error]

static std::unique_ptr<StructType> _struct_from_json(const rapidjson::Value& value); ^

github-actions · 2024-05-20T05:22:46Z

+
+private:
+    static std::unique_ptr<Type> _type_from_json(const rapidjson::Value& value);
+    static std::unique_ptr<StructType> _struct_from_json(const rapidjson::Value& value);


warning: use of undeclared identifier 'rapidjson' [clang-diagnostic-error]

static std::unique_ptr<StructType> _struct_from_json(const rapidjson::Value& value); ^

github-actions · 2024-05-20T05:22:46Z

+private:
+    static std::unique_ptr<Type> _type_from_json(const rapidjson::Value& value);
+    static std::unique_ptr<StructType> _struct_from_json(const rapidjson::Value& value);
+    static std::unique_ptr<ListType> _list_from_json(const rapidjson::Value& value);


warning: use of undeclared identifier 'std' [clang-diagnostic-error]

static std::unique_ptr<ListType> _list_from_json(const rapidjson::Value& value); ^

github-actions · 2024-05-20T05:22:46Z

+private:
+    static std::unique_ptr<Type> _type_from_json(const rapidjson::Value& value);
+    static std::unique_ptr<StructType> _struct_from_json(const rapidjson::Value& value);
+    static std::unique_ptr<ListType> _list_from_json(const rapidjson::Value& value);


warning: use of undeclared identifier 'rapidjson' [clang-diagnostic-error]

static std::unique_ptr<ListType> _list_from_json(const rapidjson::Value& value); ^

github-actions · 2024-05-20T05:22:46Z

+    static std::unique_ptr<Type> _type_from_json(const rapidjson::Value& value);
+    static std::unique_ptr<StructType> _struct_from_json(const rapidjson::Value& value);
+    static std::unique_ptr<ListType> _list_from_json(const rapidjson::Value& value);
+    static std::unique_ptr<MapType> _map_from_json(const rapidjson::Value& value);


warning: use of undeclared identifier 'std' [clang-diagnostic-error]

static std::unique_ptr<MapType> _map_from_json(const rapidjson::Value& value); ^

kaka11chen · 2024-05-20T05:25:22Z

run buildall

kaka11chen · 2024-05-20T06:25:43Z

run buildall

kaka11chen · 2024-05-20T07:19:48Z

run buildall

kaka11chen · 2024-05-20T10:38:09Z

run buildall

github-actions

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 64. Check the log or trigger a new build to see more.

github-actions · 2024-05-20T10:45:04Z

+namespace doris {
+namespace iceberg {


warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]

Suggested change

namespace doris {

namespace iceberg {

namespace doris::iceberg {

be/src/vec/exec/format/table/iceberg/partition_spec.h:94:

- } // namespace iceberg - } // namespace doris + } // namespace doris

github-actions · 2024-05-20T10:45:04Z

+    private:
+        int next_field_id() { return ++_last_assigned_field_id; }
+
+    private:


warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]

Suggested change

private:

Additional context

be/src/vec/exec/format/table/iceberg/partition_spec.h:64: previously declared here

private: ^

github-actions · 2024-05-20T10:45:04Z

+
+#pragma once
+
+#include <rapidjson/document.h>


warning: 'rapidjson/document.h' file not found [clang-diagnostic-error]

#include <rapidjson/document.h> ^

github-actions · 2024-05-20T10:45:04Z

+namespace doris {
+namespace iceberg {


warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]

Suggested change

namespace doris {

namespace iceberg {

namespace doris::iceberg {

be/src/vec/exec/format/table/iceberg/schema_parser.h:66:

- } // namespace iceberg - } // namespace doris + } // namespace doris

github-actions · 2024-05-20T10:45:04Z

+namespace doris {
+namespace iceberg {


warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]

Suggested change

namespace doris {

namespace iceberg {

namespace doris::iceberg {

be/src/vec/exec/format/table/iceberg/struct_like.h:35:

- } // namespace iceberg - } // namespace doris + } // namespace doris

github-actions · 2024-05-20T10:45:06Z

+
+    StructType(const StructType& other) {}
+
+    virtual TypeID type_id() const override { return TypeID::STRUCT; }


warning: 'virtual' is redundant since the function is already declared 'override' [modernize-use-override]

Suggested change

virtual TypeID type_id() const override { return TypeID::STRUCT; }

TypeID type_id() const override { return TypeID::STRUCT; }

github-actions · 2024-05-20T10:45:06Z

+
+    StructType* as_struct_type() override { return this; }
+
+    virtual std::string to_string() const override;


warning: 'virtual' is redundant since the function is already declared 'override' [modernize-use-override]

Suggested change

virtual std::string to_string() const override;

std::string to_string() const override;

github-actions · 2024-05-20T10:45:06Z

+
+    virtual std::string to_string() const override;
+
+    virtual int field_count() const override { return _fields.size(); }


warning: 'virtual' is redundant since the function is already declared 'override' [modernize-use-override]

Suggested change

virtual int field_count() const override { return _fields.size(); }

int field_count() const override { return _fields.size(); }

github-actions · 2024-05-20T10:45:06Z

+
+    virtual int field_count() const override { return _fields.size(); }
+
+    virtual Type* field_type(const std::string& field_name) override;


warning: 'virtual' is redundant since the function is already declared 'override' [modernize-use-override]

Suggested change

virtual Type* field_type(const std::string& field_name) override;

Type* field_type(const std::string& field_name) override;

github-actions · 2024-05-20T10:45:06Z

+
+    virtual Type* field_type(const std::string& field_name) override;
+
+    virtual const NestedField* field(int field_id) override { return _fields_by_id[field_id]; }


warning: 'virtual' is redundant since the function is already declared 'override' [modernize-use-override]

Suggested change

virtual const NestedField* field(int field_id) override { return _fields_by_id[field_id]; }

const NestedField* field(int field_id) override { return _fields_by_id[field_id]; }

kaka11chen · 2024-05-20T12:00:51Z

run compile

kaka11chen · 2024-05-20T17:26:18Z

run buildall

github-actions

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 39. Check the log or trigger a new build to see more.

github-actions · 2024-05-20T17:32:38Z

+    int precision;
+
+public:
+    ~DecimalType() = default;


warning: annotate this function with 'override' or (rarely) 'final' [modernize-use-override]

Suggested change

~DecimalType() = default;

~DecimalType() override = default;

github-actions · 2024-05-20T17:32:39Z

+
+    DecimalType(int p, int s) : scale(s), precision(p) {}
+
+    virtual TypeID type_id() const override { return TypeID::DECIMAL; }


warning: 'virtual' is redundant since the function is already declared 'override' [modernize-use-override]

Suggested change

virtual TypeID type_id() const override { return TypeID::DECIMAL; }

TypeID type_id() const override { return TypeID::DECIMAL; }

github-actions · 2024-05-20T17:32:39Z

+
+class BinaryType : public PrimitiveType {
+public:
+    ~BinaryType() = default;


warning: annotate this function with 'override' or (rarely) 'final' [modernize-use-override]

Suggested change

~BinaryType() = default;

~BinaryType() override = default;

github-actions · 2024-05-20T17:32:39Z

+
+class FixedType : public PrimitiveType {
+public:
+    ~FixedType() = default;


warning: annotate this function with 'override' or (rarely) 'final' [modernize-use-override]

Suggested change

~FixedType() = default;

~FixedType() override = default;

github-actions · 2024-05-20T17:32:39Z

+    ~FixedType() = default;
+    FixedType(int len) : length(len) {}
+
+    virtual TypeID type_id() const override { return TypeID::FIXED; }


warning: 'virtual' is redundant since the function is already declared 'override' [modernize-use-override]

Suggested change

virtual TypeID type_id() const override { return TypeID::FIXED; }

TypeID type_id() const override { return TypeID::FIXED; }

github-actions · 2024-05-20T17:32:40Z

+};
+
+class IdentityPartitionColumnTransform : public PartitionColumnTransform {
+    virtual const TypeDescriptor get_result_type(const TypeDescriptor& source_type) {


warning: return type 'const doris::TypeDescriptor' is 'const'-qualified at the top level, which may reduce code readability without improving const correctness [readability-const-return-type]

virtual const TypeDescriptor get_result_type(const TypeDescriptor& source_type) { ^

github-actions · 2024-05-20T17:32:41Z

+};
+
+class IdentityPartitionColumnTransform : public PartitionColumnTransform {
+    virtual const TypeDescriptor get_result_type(const TypeDescriptor& source_type) {


warning: prefer using 'override' or (rarely) 'final' instead of 'virtual' [modernize-use-override]

Suggested change

virtual const TypeDescriptor get_result_type(const TypeDescriptor& source_type) {

const TypeDescriptor get_result_type(const TypeDescriptor& source_type) override {

github-actions · 2024-05-20T17:32:41Z

+        return source_type;
+    }
+
+    virtual void apply(IColumn& column) { return; }


warning: prefer using 'override' or (rarely) 'final' instead of 'virtual' [modernize-use-override]

Suggested change

virtual void apply(IColumn& column) { return; }

void apply(IColumn& column) override { return; }

github-actions · 2024-05-20T17:32:41Z

+        return source_type;
+    }
+
+    virtual void apply(IColumn& column) { return; }


warning: redundant return statement at the end of a function with a void return type [readability-redundant-control-flow]

Suggested change

virtual void apply(IColumn& column) { return; }

virtual void apply(IColumn& column) { }

github-actions · 2024-05-20T17:32:41Z

+namespace doris {
+namespace vectorized {


warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]

Suggested change

namespace doris {

namespace vectorized {

namespace doris::vectorized {

be/src/vec/sink/writer/iceberg/viceberg_partition_writer.cpp:245:

- } // namespace vectorized - } // namespace doris + } // namespace doris

doris-robot · 2024-05-20T18:57:07Z

TeamCity be ut coverage result:
Function Coverage: 35.83% (9124/25463)
Line Coverage: 27.35% (74876/273809)
Region Coverage: 26.62% (38754/145595)
Branch Coverage: 23.41% (19732/84280)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b59d26848f5c67e38796fb4f24b75a5fe91a3788_b59d26848f5c67e38796fb4f24b75a5fe91a3788/report/index.html

github-actions

clang-tidy made some suggestions

github-actions · 2024-05-24T14:02:22Z

+namespace doris {
+namespace vectorized {


warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]

Suggested change

namespace doris {

namespace vectorized {

namespace doris::vectorized {

be/src/vec/sink/writer/iceberg/partition_transformers.cpp:147:

- } // namespace vectorized - } // namespace doris + } // namespace doris

github-actions · 2024-05-24T14:02:22Z

+
+    virtual const TypeDescriptor& get_result_type() const { return _source_type; }
+
+    virtual ColumnWithTypeAndName apply(Block& block, int idx);


warning: prefer using 'override' or (rarely) 'final' instead of 'virtual' [modernize-use-override]

Suggested change

virtual ColumnWithTypeAndName apply(Block& block, int idx);

ColumnWithTypeAndName apply(Block& block, int idx) override;

github-actions · 2024-05-24T14:02:22Z

+namespace doris {
+namespace vectorized {


warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]

Suggested change

namespace doris {

namespace vectorized {

namespace doris::vectorized {

be/src/vec/sink/writer/iceberg/viceberg_partition_writer.cpp:248:

- } // namespace vectorized - } // namespace doris + } // namespace doris

github-actions · 2024-05-24T14:02:23Z

+
+#pragma once
+
+#include <gen_cpp/DataSinks_types.h>


warning: 'gen_cpp/DataSinks_types.h' file not found [clang-diagnostic-error]

#include <gen_cpp/DataSinks_types.h> ^

github-actions · 2024-05-24T14:02:24Z

+namespace doris {
+namespace iceberg {


warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]

Suggested change

namespace doris {

namespace iceberg {

namespace doris::iceberg {

be/test/vec/exec/format/table/iceberg/partition_spec_parser_test.cpp:85:

- } // namespace iceberg - } // namespace doris + } // namespace doris

github-actions · 2024-05-24T14:02:24Z

+namespace doris {
+namespace iceberg {


warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]

Suggested change

namespace doris {

namespace iceberg {

namespace doris::iceberg {

be/test/vec/exec/format/table/iceberg/schema_parser_test.cpp:262:

- } // namespace iceberg - } // namespace doris + } // namespace doris

github-actions · 2024-05-24T14:02:24Z

+namespace doris {
+namespace iceberg {


warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]

Suggested change

namespace doris {

namespace iceberg {

namespace doris::iceberg {

be/test/vec/exec/format/table/iceberg/schema_test.cpp:68:

- } // namespace iceberg - } // namespace doris + } // namespace doris

github-actions · 2024-05-24T14:02:24Z

+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>


warning: 'gtest/gtest.h' file not found [clang-diagnostic-error]

#include <gtest/gtest.h> ^

github-actions · 2024-05-24T14:02:24Z

+namespace doris {
+namespace iceberg {


warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]

Suggested change

namespace doris {

namespace iceberg {

namespace doris::iceberg {

be/test/vec/exec/format/table/iceberg/type_test.cpp:174:

- } // namespace iceberg - } // namespace doris + } // namespace doris

kaka11chen · 2024-05-24T14:10:24Z

run buildall

doris-robot · 2024-05-24T15:36:55Z

TPC-H: Total hot run time: 42040 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 63e6b031f9a60e878533ab8a052b64f337185a8d, data reload: false

------ Round 1 ----------------------------------
q1	17604	4422	4257	4257
q2	2035	186	191	186
q3	10459	1285	1219	1219
q4	10192	813	837	813
q5	7479	2735	2775	2735
q6	226	132	130	130
q7	1000	615	605	605
q8	9222	2148	2136	2136
q9	9260	6722	6763	6722
q10	9486	3967	3900	3900
q11	451	237	242	237
q12	450	224	225	224
q13	17214	3149	3297	3149
q14	276	242	231	231
q15	513	464	476	464
q16	480	377	387	377
q17	977	725	722	722
q18	8586	8018	7747	7747
q19	5028	1600	1558	1558
q20	634	322	303	303
q21	5189	4047	4093	4047
q22	355	291	278	278
Total cold run time: 117116 ms
Total hot run time: 42040 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4666	4381	4437	4381
q2	369	266	276	266
q3	3101	2981	2903	2903
q4	1896	1601	1585	1585
q5	5476	5506	5487	5487
q6	216	126	124	124
q7	2185	1832	1824	1824
q8	3235	3402	3395	3395
q9	8702	8691	8681	8681
q10	3964	3742	3771	3742
q11	582	496	497	496
q12	828	621	632	621
q13	15903	3173	3164	3164
q14	303	286	267	267
q15	524	485	486	485
q16	496	431	432	431
q17	1799	1543	1462	1462
q18	7848	7790	7495	7495
q19	1745	1580	1559	1559
q20	2062	1788	1822	1788
q21	15229	4709	4659	4659
q22	555	495	488	488
Total cold run time: 81684 ms
Total hot run time: 55303 ms

doris-robot · 2024-05-24T15:40:05Z

TeamCity be ut coverage result:
Function Coverage: 35.79% (9129/25509)
Line Coverage: 27.32% (74947/274371)
Region Coverage: 26.57% (38831/146129)
Branch Coverage: 23.39% (19763/84488)
Coverage Report: http://coverage.selectdb-in.cc/coverage/63e6b031f9a60e878533ab8a052b64f337185a8d_63e6b031f9a60e878533ab8a052b64f337185a8d/report/index.html

doris-robot · 2024-05-24T15:47:29Z

TPC-DS: Total hot run time: 168709 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 63e6b031f9a60e878533ab8a052b64f337185a8d, data reload: false

query1	903	377	362	362
query2	6431	2345	2264	2264
query3	6634	217	210	210
query4	19163	17064	17458	17064
query5	4105	416	419	416
query6	257	153	156	153
query7	4584	296	290	290
query8	233	188	185	185
query9	8530	2361	2358	2358
query10	470	275	262	262
query11	10597	10100	10011	10011
query12	134	91	84	84
query13	1636	355	355	355
query14	9462	6791	6942	6791
query15	212	165	168	165
query16	7125	261	263	261
query17	1023	524	518	518
query18	1903	266	269	266
query19	203	160	151	151
query20	89	90	85	85
query21	190	139	133	133
query22	4276	3964	3759	3759
query23	33641	33182	32938	32938
query24	6996	2849	2865	2849
query25	532	352	359	352
query26	709	175	155	155
query27	1929	325	323	323
query28	3847	2052	2042	2042
query29	854	600	610	600
query30	244	175	178	175
query31	961	772	726	726
query32	80	50	54	50
query33	536	262	256	256
query34	855	484	463	463
query35	707	621	618	618
query36	1012	910	909	909
query37	108	69	71	69
query38	2891	2790	2761	2761
query39	828	795	784	784
query40	194	127	122	122
query41	45	44	43	43
query42	100	96	93	93
query43	581	557	536	536
query44	1076	710	737	710
query45	176	157	161	157
query46	1061	711	711	711
query47	1847	1768	1774	1768
query48	368	295	290	290
query49	793	380	375	375
query50	767	379	388	379
query51	6960	6658	6854	6658
query52	100	92	90	90
query53	352	281	282	281
query54	578	430	449	430
query55	76	70	71	70
query56	260	231	243	231
query57	1101	1058	1044	1044
query58	219	214	202	202
query59	3369	3337	3174	3174
query60	272	250	255	250
query61	86	87	89	87
query62	564	461	440	440
query63	306	286	297	286
query64	8390	2313	1809	1809
query65	3186	3077	3104	3077
query66	914	332	323	323
query67	14989	14961	15129	14961
query68	4612	535	537	535
query69	445	269	268	268
query70	1125	1089	1094	1089
query71	395	272	271	271
query72	7845	5463	2734	2734
query73	710	320	321	320
query74	5996	5644	5540	5540
query75	3284	2614	2639	2614
query76	2328	914	1028	914
query77	394	263	266	263
query78	10238	9778	9807	9778
query79	2139	512	511	511
query80	1015	433	432	432
query81	525	243	238	238
query82	958	94	93	93
query83	251	170	171	170
query84	266	85	92	85
query85	1025	260	261	260
query86	498	303	301	301
query87	3299	3143	3100	3100
query88	3619	2343	2335	2335
query89	475	373	370	370
query90	2004	196	192	192
query91	128	99	157	99
query92	63	50	48	48
query93	2046	503	497	497
query94	1148	184	187	184
query95	409	313	305	305
query96	582	261	264	261
query97	3160	3022	3024	3022
query98	247	216	213	213
query99	1189	857	857	857
Total cold run time: 256038 ms
Total hot run time: 168709 ms

doris-robot · 2024-05-29T03:53:44Z

TeamCity be ut coverage result:
Function Coverage: 35.90% (9120/25401)
Line Coverage: 27.38% (74922/273653)
Region Coverage: 26.62% (38789/145740)
Branch Coverage: 23.46% (19742/84162)
Coverage Report: http://coverage.selectdb-in.cc/coverage/852b07d64ae9de56da1f75a136b8ebcf56ad4054_852b07d64ae9de56da1f75a136b8ebcf56ad4054/report/index.html

doris-robot · 2024-05-29T03:53:45Z

TPC-H: Total hot run time: 40922 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 852b07d64ae9de56da1f75a136b8ebcf56ad4054, data reload: false

------ Round 1 ----------------------------------
q1	17821	4489	4452	4452
q2	2733	194	203	194
q3	11249	1177	1227	1177
q4	10272	863	850	850
q5	8160	2725	2726	2725
q6	231	132	135	132
q7	964	616	602	602
q8	9656	2088	2108	2088
q9	8943	6531	6505	6505
q10	8969	3760	3747	3747
q11	457	253	237	237
q12	414	225	224	224
q13	17761	2999	3040	2999
q14	252	217	214	214
q15	533	480	488	480
q16	521	395	378	378
q17	977	666	656	656
q18	8149	7479	7315	7315
q19	3901	1546	1512	1512
q20	646	308	312	308
q21	5021	3910	3849	3849
q22	341	278	287	278
Total cold run time: 117971 ms
Total hot run time: 40922 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4348	4212	4226	4212
q2	377	273	260	260
q3	3013	2825	2776	2776
q4	1906	1577	1576	1576
q5	5296	5328	5299	5299
q6	210	124	124	124
q7	2151	1734	1752	1734
q8	3192	3348	3309	3309
q9	8326	8399	8356	8356
q10	3929	3702	3637	3637
q11	576	498	497	497
q12	729	611	580	580
q13	16405	2993	3027	2993
q14	281	259	263	259
q15	511	469	480	469
q16	490	406	407	406
q17	1770	1484	1469	1469
q18	7710	7523	7477	7477
q19	1709	1563	1589	1563
q20	1960	1782	1757	1757
q21	4950	4844	4766	4766
q22	574	500	483	483
Total cold run time: 70413 ms
Total hot run time: 54002 ms

doris-robot · 2024-05-29T04:04:44Z

TPC-DS: Total hot run time: 172405 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 852b07d64ae9de56da1f75a136b8ebcf56ad4054, data reload: false

query1	926	393	377	377
query2	6457	2430	2371	2371
query3	6660	215	217	215
query4	19418	17285	17359	17285
query5	4159	436	421	421
query6	247	164	151	151
query7	4589	298	290	290
query8	257	185	178	178
query9	8713	2414	2385	2385
query10	449	274	260	260
query11	10576	10015	10131	10015
query12	140	88	87	87
query13	1633	367	353	353
query14	8575	7816	7024	7024
query15	219	171	167	167
query16	7800	268	264	264
query17	1841	524	509	509
query18	1923	269	269	269
query19	203	176	150	150
query20	95	86	91	86
query21	203	130	136	130
query22	4495	3908	4077	3908
query23	33708	33015	32844	32844
query24	11755	2865	2815	2815
query25	654	349	359	349
query26	1833	156	164	156
query27	3017	319	323	319
query28	7720	2030	2022	2022
query29	1119	611	618	611
query30	292	152	151	151
query31	964	745	735	735
query32	95	54	54	54
query33	771	272	273	272
query34	1063	489	486	486
query35	748	609	586	586
query36	1094	938	917	917
query37	274	69	66	66
query38	2914	2771	2745	2745
query39	850	824	771	771
query40	279	130	125	125
query41	49	46	47	46
query42	104	97	99	97
query43	581	578	548	548
query44	1230	727	746	727
query45	181	159	166	159
query46	1073	694	733	694
query47	1837	1786	1757	1757
query48	373	305	296	296
query49	1235	392	405	392
query50	778	393	408	393
query51	6915	6920	6737	6737
query52	106	125	100	100
query53	344	294	294	294
query54	1006	430	428	428
query55	74	72	72	72
query56	267	239	249	239
query57	1143	1047	1006	1006
query58	240	210	215	210
query59	3321	3313	3451	3313
query60	286	262	257	257
query61	92	89	88	88
query62	642	431	459	431
query63	312	299	284	284
query64	9864	2236	1736	1736
query65	3178	3090	3156	3090
query66	1361	329	327	327
query67	15422	14996	14739	14739
query68	4592	539	526	526
query69	437	270	272	270
query70	1158	1156	1158	1156
query71	399	273	274	273
query72	7909	5771	5741	5741
query73	713	325	324	324
query74	6009	5602	5626	5602
query75	3407	2612	2613	2612
query76	2839	1006	961	961
query77	414	274	275	274
query78	10217	10032	9815	9815
query79	2336	513	509	509
query80	950	451	434	434
query81	530	218	221	218
query82	654	91	87	87
query83	250	173	173	173
query84	238	86	84	84
query85	1914	301	274	274
query86	491	321	274	274
query87	3291	3077	3114	3077
query88	4309	2315	2332	2315
query89	491	393	388	388
query90	1992	186	189	186
query91	125	99	97	97
query92	63	47	51	47
query93	2367	501	492	492
query94	1272	184	182	182
query95	397	305	309	305
query96	592	265	271	265
query97	3247	3026	2996	2996
query98	241	226	215	215
query99	1085	854	877	854
Total cold run time: 275671 ms
Total hot run time: 172405 ms

doris-robot · 2024-05-29T04:10:22Z

ClickBench: Total hot run time: 30.42 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 852b07d64ae9de56da1f75a136b8ebcf56ad4054, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.04	0.04
query3	0.24	0.05	0.04
query4	1.68	0.08	0.07
query5	0.51	0.48	0.49
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.49	0.48
query10	0.55	0.55	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.60
query14	0.78	0.79	0.77
query15	0.83	0.81	0.80
query16	0.38	0.37	0.37
query17	0.97	0.99	0.96
query18	0.23	0.24	0.26
query19	1.79	1.69	1.70
query20	0.01	0.01	0.01
query21	15.54	0.70	0.66
query22	3.98	7.94	1.66
query23	18.29	1.42	1.38
query24	1.68	0.26	0.28
query25	0.15	0.09	0.08
query26	0.26	0.16	0.16
query27	0.08	0.08	0.07
query28	13.26	1.04	1.00
query29	13.16	3.30	3.31
query30	0.25	0.05	0.05
query31	2.85	0.39	0.38
query32	3.25	0.47	0.46
query33	2.93	2.90	2.93
query34	17.17	4.45	4.45
query35	4.47	4.51	4.46
query36	0.67	0.46	0.47
query37	0.17	0.16	0.15
query38	0.15	0.14	0.15
query39	0.04	0.03	0.04
query40	0.16	0.14	0.14
query41	0.09	0.05	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.42 s
Total hot run time: 30.42 s

kaka11chen · 2024-05-29T04:44:09Z

run buildall

apache#31442 test in apache#34929 When null value is used as the partition value, BE will return the "null" string, so this string needs to be processed specially. (cherry picked from commit d86cd1b)

doris-robot · 2024-05-29T05:02:06Z

TPC-H: Total hot run time: 40243 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a4758261471ca115f5914e932e17d501b1b3bcbd, data reload: false

------ Round 1 ----------------------------------
q1	17751	4454	4251	4251
q2	2034	204	188	188
q3	10517	1315	1153	1153
q4	10495	834	789	789
q5	7839	2731	2707	2707
q6	225	136	135	135
q7	975	604	600	600
q8	9360	2103	2128	2103
q9	9555	6624	6628	6624
q10	9207	3861	3711	3711
q11	451	246	246	246
q12	448	228	217	217
q13	17760	2965	3012	2965
q14	270	228	222	222
q15	518	475	464	464
q16	503	375	391	375
q17	968	697	742	697
q18	8120	7472	7358	7358
q19	1670	1546	1546	1546
q20	659	308	300	300
q21	5031	3337	3316	3316
q22	350	276	277	276
Total cold run time: 114706 ms
Total hot run time: 40243 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4289	4375	4256	4256
q2	370	264	258	258
q3	2959	2758	2778	2758
q4	1849	1619	1609	1609
q5	5224	5277	5328	5277
q6	218	127	130	127
q7	2138	1744	1726	1726
q8	3166	3288	3311	3288
q9	8355	8366	8330	8330
q10	3862	3661	3701	3661
q11	599	497	484	484
q12	741	574	617	574
q13	17392	2966	2984	2966
q14	290	261	263	261
q15	513	470	471	470
q16	465	416	428	416
q17	1787	1501	1456	1456
q18	7787	7673	7336	7336
q19	3422	1588	1540	1540
q20	1974	1775	1791	1775
q21	4937	4789	4863	4789
q22	555	471	487	471
Total cold run time: 72892 ms
Total hot run time: 53828 ms

doris-robot · 2024-05-29T05:02:21Z

TeamCity be ut coverage result:
Function Coverage: 35.91% (9122/25401)
Line Coverage: 27.39% (74948/273665)
Region Coverage: 26.63% (38813/145740)
Branch Coverage: 23.47% (19749/84162)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a4758261471ca115f5914e932e17d501b1b3bcbd_a4758261471ca115f5914e932e17d501b1b3bcbd/report/index.html

doris-robot · 2024-05-29T05:13:06Z

TPC-DS: Total hot run time: 171692 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a4758261471ca115f5914e932e17d501b1b3bcbd, data reload: false

query1	931	373	372	372
query2	6473	2452	2413	2413
query3	6651	216	213	213
query4	18848	17350	17092	17092
query5	4102	433	423	423
query6	251	163	153	153
query7	4592	291	287	287
query8	241	191	202	191
query9	8415	2396	2379	2379
query10	447	272	259	259
query11	10387	10246	9903	9903
query12	130	92	87	87
query13	1633	358	356	356
query14	9422	7698	7433	7433
query15	232	168	167	167
query16	8076	266	258	258
query17	1840	509	503	503
query18	2031	271	269	269
query19	197	155	153	153
query20	93	85	81	81
query21	214	143	135	135
query22	4031	4115	3988	3988
query23	33693	32673	33045	32673
query24	11915	3000	2867	2867
query25	649	350	355	350
query26	1799	153	160	153
query27	3024	317	329	317
query28	7580	2031	2039	2031
query29	1047	619	613	613
query30	298	150	155	150
query31	993	768	749	749
query32	90	51	81	51
query33	768	268	252	252
query34	1077	485	481	481
query35	727	617	601	601
query36	1043	928	910	910
query37	281	68	69	68
query38	2915	2773	2775	2773
query39	853	809	781	781
query40	268	124	123	123
query41	49	45	44	44
query42	108	96	98	96
query43	599	555	552	552
query44	1247	722	750	722
query45	183	160	161	160
query46	1065	732	703	703
query47	1835	1760	1756	1756
query48	388	308	290	290
query49	1222	376	382	376
query50	782	376	389	376
query51	6900	6670	6874	6670
query52	108	90	89	89
query53	352	296	291	291
query54	948	422	419	419
query55	73	69	72	69
query56	265	244	240	240
query57	1145	1041	1030	1030
query58	231	207	208	207
query59	3471	3264	3277	3264
query60	295	260	254	254
query61	91	84	89	84
query62	652	451	441	441
query63	312	298	284	284
query64	9833	2209	1702	1702
query65	3198	3103	3105	3103
query66	1383	332	337	332
query67	15272	15033	15023	15023
query68	4595	535	528	528
query69	500	270	265	265
query70	1159	1155	1029	1029
query71	437	266	264	264
query72	7380	5065	5299	5065
query73	716	321	324	321
query74	6071	5655	5592	5592
query75	3732	2652	2638	2638
query76	3374	1014	976	976
query77	613	263	269	263
query78	10340	9803	9640	9640
query79	2443	513	520	513
query80	1349	440	434	434
query81	503	216	221	216
query82	827	89	92	89
query83	194	180	173	173
query84	262	84	91	84
query85	1446	269	268	268
query86	464	295	288	288
query87	3351	3082	3070	3070
query88	4550	2352	2354	2352
query89	486	388	395	388
query90	2067	182	188	182
query91	124	97	94	94
query92	56	47	48	47
query93	3271	503	490	490
query94	1166	187	184	184
query95	406	313	308	308
query96	598	268	262	262
query97	3180	3026	2965	2965
query98	240	218	228	218
query99	1234	874	851	851
Total cold run time: 277398 ms
Total hot run time: 171692 ms

doris-robot · 2024-05-29T05:18:37Z

ClickBench: Total hot run time: 29.71 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a4758261471ca115f5914e932e17d501b1b3bcbd, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.65	0.08	0.07
query5	0.51	0.50	0.49
query6	1.12	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.53	0.48	0.49
query10	0.55	0.55	0.55
query11	0.15	0.11	0.12
query12	0.15	0.13	0.12
query13	0.60	0.60	0.60
query14	0.76	0.78	0.77
query15	0.82	0.81	0.80
query16	0.37	0.36	0.36
query17	0.99	1.02	0.94
query18	0.22	0.24	0.26
query19	1.79	1.67	1.71
query20	0.02	0.01	0.01
query21	15.74	0.69	0.67
query22	4.70	7.89	1.20
query23	18.29	1.35	1.28
query24	1.92	0.23	0.20
query25	0.14	0.08	0.08
query26	0.26	0.16	0.17
query27	0.07	0.08	0.09
query28	13.32	1.00	1.03
query29	13.21	3.27	3.23
query30	0.25	0.06	0.06
query31	2.86	0.37	0.37
query32	3.29	0.47	0.46
query33	2.88	2.88	2.88
query34	16.99	4.41	4.47
query35	4.51	4.47	4.47
query36	0.66	0.49	0.47
query37	0.17	0.16	0.15
query38	0.16	0.15	0.14
query39	0.05	0.03	0.03
query40	0.16	0.15	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 110.47 s
Total hot run time: 29.71 s

github-actions · 2024-05-29T06:44:53Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-29T06:44:55Z

PR approved by anyone and no changes requested.

#31442 test in #34929 When null value is used as the partition value, BE will return the "null" string, so this string needs to be processed specially.

… for inserting into table. (#35587) backport #34929

#31442 test in #34929 When null value is used as the partition value, BE will return the "null" string, so this string needs to be processed specially.

… for inserting into table. (#34929) Currently only insert into and CTAS are supported. Among them, insert into supports unpartitioned tables and partitioned tables, but partitioned tables only support identity transform partitioned tables. ### Future work: - Complete partitioned table transform functionality. - Sorting function. - Insert overwrite function. - More features like update, delete and merge.

apache#31442 test in apache#34929 When null value is used as the partition value, BE will return the "null" string, so this string needs to be processed specially.

… for inserting into table. (apache#34929) Currently only insert into and CTAS are supported. Among them, insert into supports unpartitioned tables and partitioned tables, but partitioned tables only support identity transform partitioned tables. ### Future work: - Complete partitioned table transform functionality. - Sorting function. - Insert overwrite function. - More features like update, delete and merge.

kaka11chen marked this pull request as draft May 15, 2024 13:19

kaka11chen force-pushed the iceberg_sink branch 5 times, most recently from e62faf3 to b7b865d Compare May 20, 2024 05:15

kaka11chen marked this pull request as ready for review May 20, 2024 05:15

github-actions Bot reviewed May 20, 2024

View reviewed changes

kaka11chen force-pushed the iceberg_sink branch from b7b865d to 1235534 Compare May 20, 2024 06:25

kaka11chen force-pushed the iceberg_sink branch 2 times, most recently from c47d5da to 80a0136 Compare May 20, 2024 07:19

kaka11chen force-pushed the iceberg_sink branch from 80a0136 to 48d661f Compare May 20, 2024 10:37

github-actions Bot reviewed May 20, 2024

View reviewed changes

kaka11chen force-pushed the iceberg_sink branch from 48d661f to b59d268 Compare May 20, 2024 17:25

github-actions Bot reviewed May 20, 2024

View reviewed changes

kaka11chen changed the title ~~[Feature] (iceberg-writer) Implements iceberg sink.~~ [Feature] (iceberg-writer) Implements iceberg sink basic functionality for inserting into table. May 21, 2024

kaka11chen changed the title ~~[Feature] (iceberg-writer) Implements iceberg sink basic functionality for inserting into table.~~ [Feature](iceberg-writer) Implements iceberg sink basic functionality for inserting into table. May 21, 2024

github-actions Bot reviewed May 24, 2024

View reviewed changes

Comment out regression tests and fix compiling on mac os.

a475826

kaka11chen force-pushed the iceberg_sink branch from 852b07d to a475826 Compare May 29, 2024 04:43

morningman approved these changes May 29, 2024

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label May 29, 2024

github-actions Bot added the reviewed label May 29, 2024

kaka11chen mentioned this pull request May 29, 2024

[Feature](iceberg-writer) Implements iceberg sink basic functionality for inserting into table. #35587

Merged

wuwenchi approved these changes May 29, 2024

View reviewed changes

kaka11chen merged commit d7d5235 into apache:master May 29, 2024

yiguolei pushed a commit that referenced this pull request May 29, 2024

[bugfix](iceberg)support null values as partition (#35503)

aaa89ec

#31442 test in #34929 When null value is used as the partition value, BE will return the "null" string, so this string needs to be processed specially.

morningman pushed a commit that referenced this pull request May 29, 2024

[Feature](iceberg-writer) Implements iceberg sink basic functionality…

b91d2ca

… for inserting into table. (#35587) backport #34929

morningman added dev/3.0.x dev/2.1.4-merged labels May 29, 2024

dataroaring pushed a commit that referenced this pull request May 31, 2024

[bugfix](iceberg)support null values as partition (#35503)

d16a26d

#31442 test in #34929 When null value is used as the partition value, BE will return the "null" string, so this string needs to be processed specially.

gavinchou added dev/3.0.0-merged and removed dev/3.0.x labels Jun 11, 2024

	namespace doris {
	namespace iceberg {
	namespace doris::iceberg {


		StructType(const StructType& other) {}

		virtual TypeID type_id() const override { return TypeID::STRUCT; }

	virtual TypeID type_id() const override { return TypeID::STRUCT; }
	TypeID type_id() const override { return TypeID::STRUCT; }


		StructType* as_struct_type() override { return this; }

		virtual std::string to_string() const override;

	virtual std::string to_string() const override;
	std::string to_string() const override;


		virtual std::string to_string() const override;

		virtual int field_count() const override { return _fields.size(); }

	virtual int field_count() const override { return _fields.size(); }
	int field_count() const override { return _fields.size(); }


		virtual int field_count() const override { return _fields.size(); }

		virtual Type* field_type(const std::string& field_name) override;

	virtual Type* field_type(const std::string& field_name) override;
	Type* field_type(const std::string& field_name) override;


		virtual Type* field_type(const std::string& field_name) override;

		virtual const NestedField* field(int field_id) override { return _fields_by_id[field_id]; }

	virtual const NestedField* field(int field_id) override { return _fields_by_id[field_id]; }
	const NestedField* field(int field_id) override { return _fields_by_id[field_id]; }


		DecimalType(int p, int s) : scale(s), precision(p) {}

		virtual TypeID type_id() const override { return TypeID::DECIMAL; }

	virtual const TypeDescriptor get_result_type(const TypeDescriptor& source_type) {
	const TypeDescriptor get_result_type(const TypeDescriptor& source_type) override {

	virtual void apply(IColumn& column) { return; }
	void apply(IColumn& column) override { return; }

	virtual void apply(IColumn& column) { return; }
	virtual void apply(IColumn& column) { }

	namespace doris {
	namespace vectorized {
	namespace doris::vectorized {


		virtual const TypeDescriptor& get_result_type() const { return _source_type; }

		virtual ColumnWithTypeAndName apply(Block& block, int idx);

	virtual ColumnWithTypeAndName apply(Block& block, int idx);
	ColumnWithTypeAndName apply(Block& block, int idx) override;

Conversation

kaka11chen commented May 15, 2024 • edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Future work:

Further comments

Uh oh!

doris-robot commented May 15, 2024

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

kaka11chen commented May 20, 2024

Uh oh!

kaka11chen commented May 20, 2024

Uh oh!

kaka11chen commented May 20, 2024

Uh oh!

kaka11chen commented May 20, 2024

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot May 20, 2024

Choose a reason for hiding this comment

Uh oh!

kaka11chen commented May 15, 2024 •

edited by morningman

Loading