Skip to content

[Feature](iceberg-writer) Implements iceberg sink basic functionality for inserting into table. - #35587

Merged
morningman merged 1 commit into
apache:branch-2.1from
kaka11chen:cherry-pick-iceberg_sink_2.1
May 29, 2024
Merged

morningman merged 1 commit into
apache:branch-2.1from
kaka11chen:cherry-pick-iceberg_sink_2.1

Conversation

@kaka11chen

Copy link
Copy Markdown
Contributor

Proposed changes

backport #34929

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@kaka11chen
kaka11chen marked this pull request as ready for review May 29, 2024 06:51

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 69. Check the log or trigger a new build to see more.

Comment on lines +24 to +26
namespace doris {

namespace pipeline {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]

Suggested change
namespace doris {
namespace pipeline {
namespace doris::pipeline {

be/src/pipeline/exec/iceberg_table_sink_operator.h:113:

- } // namespace pipeline
- } // namespace doris
+ } // namespace doris

Comment on lines +27 to +28
namespace doris {
namespace iceberg {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]

Suggested change
namespace doris {
namespace iceberg {
namespace doris::iceberg {

be/src/vec/exec/format/table/iceberg/partition_spec.cpp:69:

- } // namespace iceberg
- } // namespace doris
+ } // namespace doris

Comment on lines +24 to +25
namespace doris {
namespace iceberg {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]

Suggested change
namespace doris {
namespace iceberg {
namespace doris::iceberg {

be/src/vec/exec/format/table/iceberg/partition_spec.h:94:

- } // namespace iceberg
- } // namespace doris
+ } // namespace doris

private:
int next_field_id() { return ++_last_assigned_field_id; }

private:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]

Suggested change
private:
Additional context

be/src/vec/exec/format/table/iceberg/partition_spec.h:64: previously declared here

    private:
    ^


const Schema& schema() const { return *_schema; }

int spec_id() { return _spec_id; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: method 'spec_id' can be made const [readability-make-member-function-const]

Suggested change
int spec_id() { return _spec_id; }
int spec_id() const { return _spec_id; }


virtual int field_count() const override { return 2; }

virtual Type* field_type(const std::string& field_name) override;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'virtual' is redundant since the function is already declared 'override' [modernize-use-override]

Suggested change
virtual Type* field_type(const std::string& field_name) override;
Type* field_type(const std::string& field_name) override;


virtual Type* field_type(const std::string& field_name) override;

virtual const NestedField* field(int field_id) override;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'virtual' is redundant since the function is already declared 'override' [modernize-use-override]

Suggested change
virtual const NestedField* field(int field_id) override;
const NestedField* field(int field_id) override;


class ListType : public NestedType {
public:
~ListType() = default;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: annotate this function with 'override' or (rarely) 'final' [modernize-use-override]

Suggested change
~ListType() = default;
~ListType() override = default;

static std::unique_ptr<ListType> of_required(int element_id,
std::unique_ptr<Type> element_type);

virtual TypeID type_id() const override { return TypeID::LIST; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'virtual' is redundant since the function is already declared 'override' [modernize-use-override]

Suggested change
virtual TypeID type_id() const override { return TypeID::LIST; }
TypeID type_id() const override { return TypeID::LIST; }


const NestedField& element_field() const { return _element_field; }

virtual std::string to_string() const override;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'virtual' is redundant since the function is already declared 'override' [modernize-use-override]

Suggested change
virtual std::string to_string() const override;
std::string to_string() const override;

@morningman
morningman merged commit b91d2ca into apache:branch-2.1 May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants