Skip to content

Resolve correctly when dictionary depends on dictionary#4962

Merged
vitlibar merged 3 commits intoClickHouse:masterfrom
vitlibar:resolve-dictionary-depends-on-dictionary
Apr 16, 2019
Merged

Resolve correctly when dictionary depends on dictionary#4962
vitlibar merged 3 commits intoClickHouse:masterfrom
vitlibar:resolve-dictionary-depends-on-dictionary

Conversation

@vitlibar
Copy link
Copy Markdown
Member

@vitlibar vitlibar commented Apr 10, 2019

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Fix hanging on start of the server when a dictionary depends on another dictionary via a database with engine=Dictionary.

#4695

Category:

  • Bug Fix

@vitlibar vitlibar force-pushed the resolve-dictionary-depends-on-dictionary branch 2 times, most recently from 85703f1 to 8d7e228 Compare April 10, 2019 19:52
@alexey-milovidov alexey-milovidov added the pr-bugfix Pull request with bugfix, not backported by default label Apr 10, 2019
@vitlibar vitlibar force-pushed the resolve-dictionary-depends-on-dictionary branch 2 times, most recently from e931ab4 to 060e453 Compare April 11, 2019 23:38
@vitlibar vitlibar marked this pull request as ready for review April 11, 2019 23:38
@vitlibar vitlibar changed the title [WIP] Resolve correctly when dictionary depends on dictionary Resolve correctly when dictionary depends on dictionary Apr 11, 2019
@vitlibar vitlibar changed the title Resolve correctly when dictionary depends on dictionary [WIP] Resolve correctly when dictionary depends on dictionary Apr 12, 2019
@vitlibar vitlibar force-pushed the resolve-dictionary-depends-on-dictionary branch 2 times, most recently from 406d28b to 6faa4ec Compare April 13, 2019 08:26
@vitlibar vitlibar changed the title [WIP] Resolve correctly when dictionary depends on dictionary Resolve correctly when dictionary depends on dictionary Apr 13, 2019
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed that here we will wait until all dictionaries will be reloaded (in case ExternalLoader::reload() was called).

Copy link
Copy Markdown
Member Author

@vitlibar vitlibar Apr 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I replaced all_mutex.lock() with all_mutex.try_lock() here, seems it's enough to fix this problem.

@vitlibar vitlibar force-pushed the resolve-dictionary-depends-on-dictionary branch 3 times, most recently from 49e8ebb to 4645086 Compare April 15, 2019 22:22
@vitlibar vitlibar force-pushed the resolve-dictionary-depends-on-dictionary branch from 4645086 to 39c7107 Compare April 16, 2019 11:41
/// the current version of this loadable object. That's why we use try_lock() instead of lock() here.
std::unique_lock all_lock{all_mutex, std::defer_lock};
if (all_lock.try_lock())
finishReload(name, throw_on_error);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like if no other thread which is reloading dictionaries, we will wait. (It's not true).
May be just check that dictionaries have loaded once?
Or first check that dictionary is not in loadable_objects, and then try lock.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try_lock() never waits, it never blocks the current thread. If all_mutex is locked by another thread try_lock() just returns false without blocking.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. It just looks like in case no other thread locks all_mutex, and we successfully locked, all dictionaries will be reloading.

Copy link
Copy Markdown
Member

@KochetovNicolai KochetovNicolai Apr 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, successful lock is only possible in updating thread. Probably, your solution is better. (system reload dictionary will also resolve dependences fine)

@vitlibar vitlibar merged commit f0c7e56 into ClickHouse:master Apr 16, 2019
@vitlibar vitlibar deleted the resolve-dictionary-depends-on-dictionary branch April 16, 2019 14:37
abyss7 pushed a commit that referenced this pull request Apr 26, 2019
…dictionary

Resolve correctly when dictionary depends on dictionary

(cherry picked from commit f0c7e56)
@abyss7 abyss7 added the v19.5 label Apr 26, 2019
@filimonov filimonov added the comp-dictionary Dictionaries (in-memory key-value, periodically refreshed from sources). label May 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp-dictionary Dictionaries (in-memory key-value, periodically refreshed from sources). pr-bugfix Pull request with bugfix, not backported by default

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants