Skip to content

[Hotfix][FileSink Base] Fix WriteStrategy parallel writing thread unsafe issue #5546

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Hisoka-X merged 1 commit into from
Sep 26, 2023
Merged

[Hotfix][FileSink Base] Fix WriteStrategy parallel writing thread unsafe issue #5546

Hisoka-X merged 1 commit into from
Sep 26, 2023

Conversation

@hailin0
Copy link
Member

@hailin0 hailin0 commented Sep 22, 2023 •
edited
Loading

Purpose of this pull request

Fix WriteStrategy parallel writing thread unsafe issue

Thread unsafe exception occurs when parallel writes are run on a single node, should not shares thread unsafe object to writers.

Does this PR introduce any user-facing change?

No

How was this patch tested?

e2e already exists

Check list

@hailin0
[Hotfix][File-Connector] Fix WriteStrategy parallel writing thread un…
7386c37 
…safe issue

Thread unsafe exception occurs when parallel writes are run on a single node,
should not shares thread unsafe object to writers.
@hailin0 hailin0 self-assigned this Sep 22, 2023
@hailin0
Copy link
Member Author

hailin0 commented Sep 22, 2023

image

@lightzhao
Copy link
Contributor

lightzhao commented Sep 26, 2023

seatunnel/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/OrcWriteStrategy.java

Line 71 in b889618

public void write(@NonNull SeaTunnelRow seaTunnelRow) {

Which variable in this method is thread-unsafe?

@Hisoka-X Hisoka-X merged commit 1177d02 into apache:dev Sep 26, 2023
@Hisoka-X
Copy link
Member

Hisoka-X commented Sep 26, 2023

Which variable in this method is thread-unsafe?

@lightzhao WriteStrategy implement class, like LinkedHashMap<String, Writer> beingWrittenWriter in OrcWriteStrategy

@lightzhao
Copy link
Contributor

lightzhao commented Sep 26, 2023

Which variable in this method is thread-unsafe?

@lightzhao WriteStrategy implement class, like LinkedHashMap<String, Writer> beingWrittenWriter in OrcWriteStrategy

ok, The modification is indeed more thread-safe. but judging from the exception screenshot, it has nothing to do with this variable.

@hailin0
Copy link
Member Author

hailin0 commented Sep 26, 2023 •
edited
Loading

Which variable in this method is thread-unsafe?

@lightzhao WriteStrategy implement class, like LinkedHashMap<String, Writer> beingWrittenWriter in OrcWriteStrategy

ok, The modification is indeed more thread-safe. but judging from the exception screenshot, it has nothing to do with this variable.

@lightzhao

thread-unsafe class：org.apache.orc.impl.Writerimpl

reference
yuokada/embulk-output-orc#7
yuokada/embulk-output-orc#10

gnehil pushed a commit to gnehil/seatunnel that referenced this pull request Oct 12, 2023
@hailin0
[Hotfix][File-Connector] Fix WriteStrategy parallel writing thread un…
512c47d 
…safe issue (apache#5546)

Thread unsafe exception occurs when parallel writes are run on a single node,
should not shares thread unsafe object to writers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@EricJoy2048 EricJoy2048 EricJoy2048 approved these changes

@Hisoka-X Hisoka-X Hisoka-X approved these changes

@TyrantLucifer TyrantLucifer Awaiting requested review from TyrantLucifer

+1 more reviewer

@FlechazoW FlechazoW FlechazoW approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@hailin0 hailin0

Labels

approved bug connectors-v2 file reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hailin0 @lightzhao @Hisoka-X @EricJoy2048 @FlechazoW