Skip to content

[Fix][Connector-V2] Fix maxcompute write with multi parallelism #9089

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
wuchunfu merged 1 commit into from
Apr 8, 2025
Merged

[Fix][Connector-V2] Fix maxcompute write with multi parallelism #9089

wuchunfu merged 1 commit into from
Apr 8, 2025

Conversation

@Hisoka-X
Copy link
Member

@Hisoka-X Hisoka-X commented Mar 31, 2025 •
edited
Loading

Purpose of this pull request

Fix maxcompute write with multi parallelism. Before this patch, it will throw

Caused by: RequestId=202503311431065ca7301a00fe5681, ErrorCode=MetaTransactionAborted, ErrorMessage=Operation aborted due to cannot satisfy ODPS ACID constraints. This is normally caused by concurrent operations.

The change let maxcompute automatically manage blocks internally to prevent multiple threads from using the same blockid.

Does this PR introduce any user-facing change?

no

How was this patch tested?

tested in local

Check list

@wuchunfu wuchunfu merged commit 9426b7b into apache:dev Apr 8, 2025
5 checks passed
@dybyte
Copy link
Contributor

dybyte commented Jun 6, 2025

Hi @Hisoka-X
I noticed the change from session.commit(new Long[] {BLOCK_0}) to session.commit(),
and I saw that the commit message mentions “Fix maxcompute write with multi parallelism”.
I’m curious — could you please explain how this change helps with parallelism?
Thanks again!

@Hisoka-X
Copy link
Member Author

Hisoka-X commented Jun 6, 2025

Hi @Hisoka-X I noticed the change from session.commit(new Long[] {BLOCK_0}) to session.commit(), and I saw that the commit message mentions “Fix maxcompute write with multi parallelism”. I’m curious — could you please explain how this change helps with parallelism? Thanks again!

Hi @dybyte , the change let maxcompute automatically manage blocks internally to prevent multiple threads from using the same blockid.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hailin0 hailin0 hailin0 approved these changes

@wuchunfu wuchunfu wuchunfu approved these changes

Assignees

No one assigned

Labels

approved connectors-v2 maxcompute reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Hisoka-X @dybyte @hailin0 @wuchunfu