Skip to content

Destroy stale sessions without a lock#81310

Merged
tavplubix merged 1 commit intomasterfrom
better_sessions_cleanup
Jun 5, 2025
Merged

Destroy stale sessions without a lock#81310
tavplubix merged 1 commit intomasterfrom
better_sessions_cleanup

Conversation

@tavplubix
Copy link
Copy Markdown
Member

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Don't hold NamedSessionsStorage::mutex while destroying stale sessions. Fixes #71307

Logs: https://pastila.nl/?000c4a4b/384edaa6b888a3d8fcfbf275b1e1fac2#YGtn/olp6K1dM4cz+3D8/w==

Thread 123121 got stuck somewhere for 6 seconds while getting a session for tx_async 19 "alter table test update value = 18 where id = 2". In the meantime, tx_async 19 "commit" was waiting for the previous query in tx_wait, but timeout 5 tail --pid=$query_pid -f /dev/null timed out in 5 seconds. Then we also look for the previous query in system.processes, but it wasn't there yet because thread 123121 was still stuck on getting session and didn't start processing the query. So tx_async 19 "commit" started executing and got "Session is locked by a concurrent client"

Looks like thread 123121 was waiting on the mutex in acquireSession. The lock was held by thread 10168 while cleaning old sessions and rolling back unfinished transactions: https://pastila.nl/?00f522cd/99e913c3fd765a0354d6279d59d086bc#iKrL5stckRaL46ceM1WEGg==

Let's destroy session objects without holding the lock

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

@clickhouse-gh
Copy link
Copy Markdown
Contributor

clickhouse-gh bot commented Jun 4, 2025

Workflow [PR], commit [4556e74]

@clickhouse-gh clickhouse-gh bot added the pr-not-for-changelog This PR should not be mentioned in the changelog label Jun 4, 2025
@tavplubix tavplubix added this pull request to the merge queue Jun 5, 2025
Merged via the queue into master with commit 98aa058 Jun 5, 2025
118 of 121 checks passed
@tavplubix tavplubix deleted the better_sessions_cleanup branch June 5, 2025 01:26
@robot-clickhouse-ci-1 robot-clickhouse-ci-1 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jun 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-not-for-changelog This PR should not be mentioned in the changelog pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test 01167_isolation_hermitage is flaky

2 participants