Skip to content

Conversation

@loserwang1024
Copy link
Contributor

@loserwang1024 loserwang1024 commented Aug 23, 2023

What is the purpose of the change

In PostgresSourceEnumerator#createSlotForGlobalStreamSplit, PostgresReplicationConnection will be closed after create slot.
However, not any PostgresReplicationConnection.close will close the PostgresConnection in messageDecoder, which is generated by Hikari pool and is passed to PostgresReplicationConnection.

See AbstractMessageDecoder.close:

  • NonStreamingWal2JsonMessageDecoder.close: do nothing
  • PgProtoMessageDecoder.close: do nothing
  • StreamingWal2JsonMessageDecoder.close: do nothing
  • PgOutputMessageDecoder.close: the only one to close jdbc connection.

Thus, the Hikari pool will be full if there is too much pg source.

Verified this problem

Debug PostgresReplicationConnection.close in any test.

loserwang1024 added a commit to loserwang1024/flink-cdc-connectors that referenced this pull request Aug 23, 2023
loserwang1024 added a commit to loserwang1024/flink-cdc-connectors that referenced this pull request Aug 23, 2023
loserwang1024 added a commit to loserwang1024/flink-cdc-connectors that referenced this pull request Aug 24, 2023
@loserwang1024
Copy link
Contributor Author

@TyrantLucifer CC

Copy link
Contributor

@ruanhang1993 ruanhang1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Copy link
Contributor

@ruanhang1993 ruanhang1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@ruanhang1993 ruanhang1993 merged commit 82e3053 into apache:master Sep 26, 2023
Copy link
Member

@TyrantLucifer TyrantLucifer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

e-mhui pushed a commit to e-mhui/flink-cdc-connectors that referenced this pull request Oct 18, 2023
GOODBOY008 pushed a commit to GOODBOY008/flink-cdc that referenced this pull request Oct 30, 2023
GOODBOY008 pushed a commit to GOODBOY008/flink-cdc that referenced this pull request Oct 30, 2023
GOODBOY008 pushed a commit to GOODBOY008/flink-cdc that referenced this pull request Oct 30, 2023
leonardBang pushed a commit that referenced this pull request Oct 31, 2023
zhongqishang pushed a commit to zhongqishang/flink-cdc that referenced this pull request Dec 7, 2023
ChaomingZhangCN pushed a commit to ChaomingZhangCN/flink-cdc that referenced this pull request Jan 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants