Skip to content

Remove redundant fsync on coordination logs rotation#22707

Merged
alesapin merged 2 commits intomasterfrom
don_t_fsync_on_log_rotate
Apr 6, 2021
Merged

Remove redundant fsync on coordination logs rotation#22707
alesapin merged 2 commits intomasterfrom
don_t_fsync_on_log_rotate

Conversation

@alesapin
Copy link
Copy Markdown
Member

@alesapin alesapin commented Apr 6, 2021

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

@robot-clickhouse robot-clickhouse added the pr-not-for-changelog This PR should not be mentioned in the changelog label Apr 6, 2021
@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Apr 6, 2021 •
edited
Loading

From logs:

2021.04.04 14:46:23.148835 [ 210 ] RaftInstance: Receive a client_request message from 0 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0 and Term=0
2021.04.04 14:46:29.851160 [ 210 ] KeeperLogStore: Starting new changelog /var/lib/clickhouse/coordination/logs/changelog_100001_110000.bin
2021.04.04 14:46:29.958778 [ 210 ] RaftInstance: append at log_idx 100001

Seems like fsync is disrupting operation for our CI.

@alesapin alesapin added the jepsen-test Need to test this PR with jepsen tests label Apr 6, 2021
@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Apr 6, 2021

A failed test is not related to changes, all sessions were stable since creation:

$ zgrep 'KeeperTCPHandler' clickhouse-server.log.gz 
2021.04.06 21:17:37.608463 [ 193 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: [::1]:54028
2021.04.06 21:17:37.608649 [ 193 ] {} <Information> KeeperTCPHandler: Requesting session ID for the new client
2021.04.06 21:17:37.610083 [ 193 ] {} <Information> KeeperTCPHandler: Received session ID 2
2021.04.06 21:17:47.610416 [ 193 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #2
2021.04.06 21:17:57.610450 [ 193 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #2
2021.04.06 21:18:17.610511 [ 193 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #2
...
2021.04.06 22:56:57.628621 [ 193 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #2
2021.04.06 22:57:07.628626 [ 193 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #2
2021.04.06 22:57:17.628704 [ 193 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #2

$ zgrep -a 'KeeperTCPHandler' clickhouse-server1.log.gz 
2021.04.06 21:17:37.606679 [ 245 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: [::1]:59600
2021.04.06 21:17:37.606852 [ 245 ] {} <Information> KeeperTCPHandler: Requesting session ID for the new client
2021.04.06 21:17:37.608963 [ 245 ] {} <Information> KeeperTCPHandler: Received session ID 1
2021.04.06 21:17:47.609392 [ 245 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #1
2021.04.06 21:17:57.609383 [ 245 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #1
2021.04.06 21:18:07.609417 [ 245 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #1
...
2021.04.06 22:57:07.626615 [ 245 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #1
2021.04.06 22:57:17.626641 [ 245 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #1
2021.04.06 22:57:27.626663 [ 245 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #1

$ zgrep -a 'KeeperTCPHandler' clickhouse-server2.log.gz 
2021.04.06 21:17:37.624698 [ 285 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: [::1]:38424
2021.04.06 21:17:37.624868 [ 285 ] {} <Information> KeeperTCPHandler: Requesting session ID for the new client
2021.04.06 21:17:37.626554 [ 285 ] {} <Information> KeeperTCPHandler: Received session ID 3
2021.04.06 21:17:47.626853 [ 285 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #3
2021.04.06 21:17:57.626928 [ 285 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #3
2021.04.06 21:18:07.626929 [ 285 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #3
...
2021.04.06 22:57:07.644708 [ 285 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #3
2021.04.06 22:57:17.644752 [ 285 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #3
2021.04.06 22:57:27.644829 [ 285 ] {} <Trace> KeeperTCPHandler: Received heartbeat for session #3

So I'll merge.

@alesapin
Copy link
Copy Markdown
Member Author

alesapin commented Apr 6, 2021

Jepsen also looks ok https://clickhouse-test-reports.s3.yandex.net/22707/80d441ec9536dd93b3db3d03434dfb684f3d45a2/clickhouse_keeper_jepsen.html

@alesapin alesapin merged commit c77a006 into master Apr 6, 2021
@alesapin alesapin deleted the don_t_fsync_on_log_rotate branch April 6, 2021 20:51
@azat azat mentioned this pull request Apr 20, 2021
Closed
@alexey-milovidov alexey-milovidov mentioned this pull request Nov 24, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

jepsen-test Need to test this PR with jepsen tests pr-not-for-changelog This PR should not be mentioned in the changelog

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@alesapin @robot-clickhouse