fix(attachment): Avoid recursive lock of `Sync` object on worker detach by TreeHunter9 · Pull Request #8872 · FirebirdSQL/firebird

TreeHunter9 · 2026-01-27T13:48:26Z

Due to double locking of mainSync inside WorkerStableAttachment::fini() (AttSyncLockGuard + BackgroundContextHolder) EngineCheckout will become useless (it uses in LCK_fini() -> LockManager::shutdownOwner()). If EngineCheckout will not do its job, we can get a deadlock, for example inside LockManager::shutdownOwner():

	while (owner->own_ast_count)
	{
		{ // checkout scope
			LockTableCheckout checkout(this, FB_FUNCTION);
			callbacks.checkoutRun([] {
				Thread::sleep(10);
			});
		}

		owner = (own*) SRQ_ABS_PTR(owner_offset);
	}

If the AST will be called for our worker attachment we will be stuck in this loop, AST cannot continue its execution due to lock on mainSync and attachment cannot finish its own routine due to recursive lock.

This case can be reproduced with next steps.

1. Set `ParallelWorkers = 2` in firebird.conf;
2. Create index on some field;
3. Next steps need to be done instantly in one time:
3.1 Wait 1 minute until timeout will detach worker attachment;
3.2 Alter table for which the index was created to trigger the `rescan_ast_relation()` AST;

It's very hard to reproduce due to race condition nature of the bug, but it can still happen accidentally in production.

v5 is also affected by this bug.

- Do not lock `mainSync` second time, so that `EngineCheckout` can do its job;

hvlad · 2026-01-27T21:22:46Z

Could you provide call stacks for two deadlocked threads ?

TreeHunter9 · 2026-01-28T07:44:04Z

Could you provide call stacks for two deadlocked threads ?

Sure.

Thread 1

libEngine14.so!Thread::sleep(unsigned int milliseconds) (src/common/ThreadStart.cpp:240)
libEngine14.so!operator()(const struct {...} * const __closure) (src/lock/lock.cpp:416)
libEngine14.so!std::__invoke_impl<void, Jrd::LockManager::shutdownOwner(const Callbacks&, SLONG*)::<lambda()>&>(std::__invoke_other, struct {...} &)(struct {...} & __f) (/usr/include/c++/13/bits/invoke.h:61)
libEngine14.so!std::__invoke_r<void, Jrd::LockManager::shutdownOwner(const Callbacks&, SLONG*)::<lambda()>&>(struct {...} &)(struct {...} & __fn) (/usr/include/c++/13/bits/invoke.h:111)
libEngine14.so!std::_Function_handler<void(), Jrd::LockManager::shutdownOwner(const Callbacks&, SLONG*)::<lambda()> >::_M_invoke(const std::_Any_data &)(const union std::_Any_data & __functor) (/usr/include/c++/13/bits/std_function.h:290)
libEngine14.so!std::function<void ()>::operator()() const(const class std::function<void()> * const this) (/usr/include/c++/13/bits/std_function.h:591)
libEngine14.so!Jrd::LockManagerEngineCallbacks::checkoutRun(std::function<void ()>) const(const class Jrd::LockManagerEngineCallbacks * const this, class std::function<void()> func) (src/jrd/lck.cpp:112)
libEngine14.so!Jrd::LockManager::shutdownOwner(class Jrd::LockManager * const this, const class Jrd::LockManager::Callbacks & callbacks, SLONG * owner_handle) (src/lock/lock.cpp:415)
libEngine14.so!LCK_fini(class Jrd::thread_db * tdbb, enum Jrd::lck_owner_t owner_type) (src/jrd/lck.cpp:520)
libEngine14.so!Jrd::WorkerStableAttachment::fini(class Jrd::WorkerStableAttachment * const this) (src/jrd/WorkerAttachment.cpp:130)
libEngine14.so!Jrd::WorkerAttachment::doDetach(Jrd::FbStatusVector * status, class Jrd::StableAttachmentPart * sAtt) (src/jrd/WorkerAttachment.cpp:516)
libEngine14.so!Jrd::WorkerAttachment::detachIdle(class Jrd::StableAttachmentPart * sAtt) (src/jrd/WorkerAttachment.cpp:462)
libEngine14.so!Jrd::WorkerStableAttachment::doOnIdleTimer(class Jrd::WorkerStableAttachment * const this, class Firebird::TimerImpl * timer) (src/jrd/WorkerAttachment.cpp:108)
libEngine14.so!Jrd::StableAttachmentPart::onIdleTimer(class Jrd::StableAttachmentPart * const this, class Firebird::TimerImpl * timer) (src/jrd/Attachment.h:383)
libEngine14.so!std::__invoke_impl<void, void (Jrd::StableAttachmentPart::*&)(Firebird::TimerImpl*), Jrd::StableAttachmentPart*&, Firebird::TimerImpl*>(void (Jrd::StableAttachmentPart::*&)(class Jrd::StableAttachmentPart * const, class Firebird::TimerImpl *) __f, class Jrd::StableAttachmentPart *& __t) (/usr/include/c++/13/bits/invoke.h:74)
libEngine14.so!std::__invoke<void (Jrd::StableAttachmentPart::*&)(Firebird::TimerImpl*), Jrd::StableAttachmentPart*&, Firebird::TimerImpl*>(void (Jrd::StableAttachmentPart::*&)(class Jrd::StableAttachmentPart * const, class Firebird::TimerImpl *) __fn) (/usr/include/c++/13/bits/invoke.h:96)
libEngine14.so!std::_Bind<void (Jrd::StableAttachmentPart::*(Jrd::StableAttachmentPart*, std::_Placeholder<1>))(Firebird::TimerImpl*)>::__call<void, Firebird::TimerImpl*&&, 0ul, 1ul>(std::tuple<Firebird::TimerImpl*&&>&&, std::_Index_tuple<0ul, 1ul>)(class std::_Bind<void (Jrd::StableAttachmentPart::*(Jrd::StableAttachmentPart*, std::_Placeholder<1>))(Firebird::TimerImpl*)> * const this, class std::tuple<Firebird::TimerImpl*&&> && __args) (/usr/include/c++/13/functional:506)
libEngine14.so!std::_Bind<void (Jrd::StableAttachmentPart::*(Jrd::StableAttachmentPart*, std::_Placeholder<1>))(Firebird::TimerImpl*)>::operator()<Firebird::TimerImpl*, void>(Firebird::TimerImpl*&&)(class std::_Bind<void (Jrd::StableAttachmentPart::*(Jrd::StableAttachmentPart*, std::_Placeholder<1>))(Firebird::TimerImpl*)> * const this) (/usr/include/c++/13/functional:591)
libEngine14.so!std::__invoke_impl<void, std::_Bind<void (Jrd::StableAttachmentPart::*(Jrd::StableAttachmentPart*, std::_Placeholder<1>))(Firebird::TimerImpl*)>&, Firebird::TimerImpl*>(std::__invoke_other, std::_Bind<void (Jrd::StableAttachmentPart::*(Jrd::StableAttachmentPart*, std::_Placeholder<1>))(Firebird::TimerImpl*)>&, Firebird::TimerImpl*&&)(std::_Bind<void (Jrd::StableAttachmentPart::*(Jrd::StableAttachmentPart*, std::_Placeholder<1>))(Firebird::TimerImpl*)> & __f) (/usr/include/c++/13/bits/invoke.h:61)
libEngine14.so!std::__invoke_r<void, std::_Bind<void (Jrd::StableAttachmentPart::*(Jrd::StableAttachmentPart*, std::_Placeholder<1>))(Firebird::TimerImpl*)>&, Firebird::TimerImpl*>(std::_Bind<void (Jrd::StableAttachmentPart::*(Jrd::StableAttachmentPart*, std::_Placeholder<1>))(Firebird::TimerImpl*)>&, Firebird::TimerImpl*&&)(std::_Bind<void (Jrd::StableAttachmentPart::*(Jrd::StableAttachmentPart*, std::_Placeholder<1>))(Firebird::TimerImpl*)> & __fn) (/usr/include/c++/13/bits/invoke.h:111)
libEngine14.so!std::_Function_handler<void (Firebird::TimerImpl*), std::_Bind<void (Jrd::StableAttachmentPart::*(Jrd::StableAttachmentPart*, std::_Placeholder<1>))(Firebird::TimerImpl*)> >::_M_invoke(std::_Any_data const&, Firebird::TimerImpl*&&)(const std::_Any_data & __functor, Firebird::TimerImpl *&& __args#0) (/usr/include/c++/13/bits/std_function.h:290)
libEngine14.so!std::function<void (Firebird::TimerImpl*)>::operator()(Firebird::TimerImpl*) const(const std::function<void(Firebird::TimerImpl*)> * const this, Firebird::TimerImpl * __args#0) (/usr/include/c++/13/bits/std_function.h:591)
libEngine14.so!Firebird::TimerImpl::handler(Firebird::TimerImpl * const this) (src/common/classes/TimerImpl.cpp:62)
libEngine14.so!Firebird::ITimerBaseImpl<Firebird::TimerImpl, Firebird::CheckStatusWrapper, Firebird::IReferenceCountedImpl<Firebird::TimerImpl, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IVersionedImpl<Firebird::TimerImpl, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::ITimer> > > > >::cloophandlerDispatcher(Firebird::ITimer * self) (src/include/firebird/IdlFbInterfaces.h:16436)
libfbclient.so.2!Firebird::ITimer::handler(Firebird::ITimer * const this) (src/include/firebird/IdlFbInterfaces.h:4746)
libfbclient.so.2!Why::(anonymous namespace)::TimerEntry::timeThread() (src/yvalve/MasterImplementation.cpp:285)
libfbclient.so.2!(anonymous namespace)::ThreadArgs::run((anonymous namespace)::ThreadArgs * const this) (src/common/ThreadStart.cpp:80)
libfbclient.so.2!(anonymous namespace)::threadStart(void * arg) (src/common/ThreadStart.cpp:94)
libc.so.6!start_thread(void * arg) (pthread_create.c:447)
libc.so.6!clone3() (clone3.S:78)

Thread 2

libc.so.6!futex_wait(int private, unsigned int expected, unsigned int * futex_word) (futex-internal.h:146)
libc.so.6!__GI___lll_lock_wait(int * futex, int * futex@entry, int private) (lowlevellock.c:49)
libc.so.6!lll_mutex_lock_optimized(pthread_mutex_t * mutex) (pthread_mutex_lock.c:48)
libc.so.6!___pthread_mutex_lock(pthread_mutex_t * mutex) (pthread_mutex_lock.c:128)
libEngine14.so!Firebird::Mutex::enter(class Firebird::Mutex * const this, const char * aReason) (src/common/classes/locks.h:191)
libEngine14.so!Jrd::StableAttachmentPart::Sync::enter(class Jrd::StableAttachmentPart::Sync * const this, const char * aReason) (src/jrd/Attachment.h:240)
libEngine14.so!Jrd::Attachment::SyncGuard::init(class Jrd::Attachment::SyncGuard * const this, const char * f, bool optional) (src/jrd/Attachment.cpp:1015)
libEngine14.so!Jrd::Attachment::SyncGuard::SyncGuard(class Jrd::Attachment::SyncGuard * const this, class Jrd::StableAttachmentPart * js, const char * f, bool optional) (src/jrd/Attachment.h:416)
libEngine14.so!Jrd::AsyncContextHolder::AsyncContextHolder(class Jrd::AsyncContextHolder * const this, class Jrd::Database * dbb, const char * f, class Jrd::Lock * lck) (src/jrd/jrd.h:1086)
libEngine14.so!rescan_ast_relation(void * ast_object) (temp/Debug/jrd/met.cpp:11171)
libEngine14.so!operator()(const struct {...} * const __closure) (src/lock/lock.cpp:1405)
libEngine14.so!std::__invoke_impl<void, Jrd::LockManager::blocking_action(const Callbacks&, SLONG)::<lambda()>&>(std::__invoke_other, struct {...} &)(struct {...} & __f) (/usr/include/c++/13/bits/invoke.h:61)
libEngine14.so!std::__invoke_r<void, Jrd::LockManager::blocking_action(const Callbacks&, SLONG)::<lambda()>&>(struct {...} &)(struct {...} & __fn) (/usr/include/c++/13/bits/invoke.h:111)
libEngine14.so!std::_Function_handler<void(), Jrd::LockManager::blocking_action(const Callbacks&, SLONG)::<lambda()> >::_M_invoke(const std::_Any_data &)(const union std::_Any_data & __functor) (/usr/include/c++/13/bits/std_function.h:290)
libEngine14.so!std::function<void ()>::operator()() const(const class std::function<void()> * const this) (/usr/include/c++/13/bits/std_function.h:591)
libEngine14.so!Jrd::LockManagerEngineCallbacks::checkoutRun(std::function<void ()>) const(const class Jrd::LockManagerEngineCallbacks * const this, class std::function<void()> func) (src/jrd/lck.cpp:112)
libEngine14.so!Jrd::LockManager::blocking_action(class Jrd::LockManager * const this, const class Jrd::LockManager::Callbacks & callbacks, SLONG blocking_owner_offset) (src/lock/lock.cpp:1402)
libEngine14.so!Jrd::LockManager::signal_owner(class Jrd::LockManager * const this, const class Jrd::LockManager::Callbacks & callbacks, struct own * blocking_owner) (src/lock/lock.cpp:3229)
libEngine14.so!Jrd::LockManager::post_blockage(class Jrd::LockManager * const this, const class Jrd::LockManager::Callbacks & callbacks, struct lrq * request, struct lbl * lock) (src/lock/lock.cpp:2700)
libEngine14.so!Jrd::LockManager::wait_for_request(class Jrd::LockManager * const this, const class Jrd::LockManager::Callbacks & callbacks, struct lrq * request, SSHORT lck_wait) (src/lock/lock.cpp:3790)
libEngine14.so!Jrd::LockManager::grant_or_que(class Jrd::LockManager * const this, const class Jrd::LockManager::Callbacks & callbacks, struct lrq * request, struct lbl * lock, SSHORT lck_wait) (src/lock/lock.cpp:2227)
libEngine14.so!Jrd::LockManager::enqueue(class Jrd::LockManager * const this, const class Jrd::LockManager::Callbacks & callbacks, class Firebird::CheckStatusWrapper * statusVector, SLONG prior_request, const USHORT series, const UCHAR * value, const USHORT length, UCHAR type, lock_ast_t ast_routine, void * ast_argument, LOCK_DATA_T data, SSHORT lck_wait, SLONG owner_offset) (src/lock/lock.cpp:531)
libEngine14.so!enqueue(class Jrd::thread_db * tdbb, class Firebird::CheckStatusWrapper * statusVector, class Jrd::Lock * lock, USHORT level, SSHORT wait) (src/jrd/lck.cpp:967)
libEngine14.so!ENQUEUE(class Jrd::thread_db * tdbb, class Firebird::CheckStatusWrapper * statusVector, class Jrd::Lock * lock, USHORT level, SSHORT wait) (src/jrd/lck.cpp:167)
libEngine14.so!LCK_lock(class Jrd::thread_db * tdbb, class Jrd::Lock * lock, USHORT level, SSHORT wait) (src/jrd/lck.cpp:693)
libEngine14.so!make_version(class Jrd::thread_db * tdbb, SSHORT phase, class Jrd::DeferredWork * work, class Jrd::jrd_tra * transaction) (temp/Debug/jrd/dfw.cpp:11488)
libEngine14.so!DFW_perform_work(class Jrd::thread_db * tdbb, class Jrd::jrd_tra * transaction) (temp/Debug/jrd/dfw.cpp:5343)
libEngine14.so!TRA_commit(class Jrd::thread_db * tdbb, class Jrd::jrd_tra * transaction, const bool retaining_flag) (src/jrd/tra.cpp:486)
libEngine14.so!commit(class Jrd::thread_db * tdbb, class Jrd::jrd_tra * transaction, const bool retaining_flag) (src/jrd/jrd.cpp:6942)
libEngine14.so!JRD_commit_transaction(class Jrd::thread_db * tdbb, class Jrd::jrd_tra * transaction) (src/jrd/jrd.cpp:9651)
libEngine14.so!Jrd::JTransaction::internalCommit(class Jrd::JTransaction * const this, class Firebird::CheckStatusWrapper * user_status) (src/jrd/jrd.cpp:2661)
libEngine14.so!Jrd::JTransaction::commit(class Jrd::JTransaction * const this, class Firebird::CheckStatusWrapper * user_status) (src/jrd/jrd.cpp:2630)
libEngine14.so!Firebird::ITransactionBaseImpl<Jrd::JTransaction, Firebird::CheckStatusWrapper, Firebird::IReferenceCountedImpl<Jrd::JTransaction, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JTransaction, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::ITransaction> > > > >::cloopcommitDispatcher(class Firebird::ITransaction * self, class Firebird::IStatus * status) (src/include/firebird/IdlFbInterfaces.h:9957)
libfbclient.so.2!Firebird::ITransaction::commit<Firebird::CheckStatusWrapper>(class Firebird::ITransaction * const this, class Firebird::CheckStatusWrapper * status) (src/include/firebird/IdlFbInterfaces.h:1331)
libfbclient.so.2!operator()(const struct {...} * const __closure) (src/yvalve/why.cpp:5379)
libfbclient.so.2!std::__invoke_impl<void, Why::YTransaction::commit(Firebird::CheckStatusWrapper*)::<lambda()>&>(std::__invoke_other, struct {...} &)(struct {...} & __f) (/usr/include/c++/13/bits/invoke.h:61)
libfbclient.so.2!std::__invoke_r<void, Why::YTransaction::commit(Firebird::CheckStatusWrapper*)::<lambda()>&>(struct {...} &)(struct {...} & __fn) (/usr/include/c++/13/bits/invoke.h:111)
libfbclient.so.2!std::_Function_handler<void(), Why::YTransaction::commit(Firebird::CheckStatusWrapper*)::<lambda()> >::_M_invoke(const std::_Any_data &)(const union std::_Any_data & __functor) (/usr/include/c++/13/bits/std_function.h:290)
libfbclient.so.2!std::function<void ()>::operator()() const(const class std::function<void()> * const this) (/usr/include/c++/13/bits/std_function.h:591)
libfbclient.so.2!Why::done<Why::YTransaction>(Firebird::CheckStatusWrapper*, Why::YEntry<Why::YTransaction>&, Why::YTransaction*, std::function<void ()>, std::function<void ()>)(class Firebird::CheckStatusWrapper * status, class Why::YEntry<Why::YTransaction> & entry, class Why::YTransaction * y, class std::function<void()> newClose, class std::function<void()> oldClose) (src/yvalve/why.cpp:1359)
libfbclient.so.2!Why::YTransaction::commit(class Why::YTransaction * const this, class Firebird::CheckStatusWrapper * status) (src/yvalve/why.cpp:5379)
libfbclient.so.2!Firebird::ITransactionBaseImpl<Why::YTransaction, Firebird::CheckStatusWrapper, Firebird::IReferenceCountedImpl<Why::YTransaction, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::IVersionedImpl<Why::YTransaction, Firebird::CheckStatusWrapper, Firebird::Inherit<Firebird::ITransaction> > > > >::cloopcommitDispatcher(class Firebird::ITransaction * self, class Firebird::IStatus * status) (src/include/firebird/IdlFbInterfaces.h:9957)
Firebird::ITransaction::commit<Firebird::CheckStatusWrapper>(class Firebird::ITransaction * const this, class Firebird::CheckStatusWrapper * status) (src/include/firebird/IdlFbInterfaces.h:1331)
rem_port::end_transaction(struct rem_port * const this, enum P_OP operation, P_RLSE * release, PACKET * sendL) (src/remote/server/server.cpp:3436)
process_packet(struct rem_port * port, PACKET * sendL, PACKET * receive, struct rem_port ** result) (src/remote/server/server.cpp:5181)
loopThread() (src/remote/server/server.cpp:6987)
(anonymous namespace)::ThreadArgs::run(class (anonymous namespace)::ThreadArgs * const this) (src/common/ThreadStart.cpp:80)
(anonymous namespace)::threadStart(void * arg) (src/common/ThreadStart.cpp:94)
libc.so.6!start_thread(void * arg) (pthread_create.c:447)
libc.so.6!clone3() (clone3.S:78)

hvlad · 2026-01-28T13:03:55Z

Thanks, it helps

TreeHunter9 · 2026-01-29T06:48:43Z

@hvlad Should I create another PR for v5 or you will cherry-pick this commit?

dyemanov · 2026-01-29T07:01:17Z

I will cherry-pick it.

hvlad · 2026-01-29T12:25:25Z

I will cherry-pick it.

Thanks ;)

fix(attachment): Avoid recursive lock of Sync object on worker detach

8dce818

- Do not lock `mainSync` second time, so that `EngineCheckout` can do its job;

hvlad self-assigned this Jan 27, 2026

hvlad merged commit b4faa28 into FirebirdSQL:master Jan 28, 2026
23 checks passed

hvlad added type: bug affect-version: 6.0 Initial fix-version: 6.0 Alpha 1 affect-version: 5.0.0 labels Jan 28, 2026

pavel-zotov added the qa: cannot be tested label Jan 28, 2026

dyemanov added the fix-version: 5.0.4 label Jan 29, 2026

TreeHunter9 deleted the master_worker_recursive_lock branch January 29, 2026 07:10

mrotteveel added the rlsnotes60: no Intentionally not added to the Firebird 6.0 release notes. label Mar 3, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(attachment): Avoid recursive lock of `Sync` object on worker detach#8872

fix(attachment): Avoid recursive lock of `Sync` object on worker detach#8872
hvlad merged 1 commit intoFirebirdSQL:masterfrom
TreeHunter9:master_worker_recursive_lock

TreeHunter9 commented Jan 27, 2026

Uh oh!

hvlad commented Jan 27, 2026

Uh oh!

TreeHunter9 commented Jan 28, 2026

Uh oh!

hvlad commented Jan 28, 2026

Uh oh!

Uh oh!

TreeHunter9 commented Jan 29, 2026

Uh oh!

dyemanov commented Jan 29, 2026

Uh oh!

hvlad commented Jan 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

TreeHunter9 commented Jan 27, 2026

Uh oh!

hvlad commented Jan 27, 2026

Uh oh!

TreeHunter9 commented Jan 28, 2026

Uh oh!

hvlad commented Jan 28, 2026

Uh oh!

Uh oh!

TreeHunter9 commented Jan 29, 2026

Uh oh!

dyemanov commented Jan 29, 2026

Uh oh!

hvlad commented Jan 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants