What happened?
Description
We were seeing Could not acquire a mutex lock for the queue errors on a project occurring randomly, sometimes between jobs executing (when trying to get the next job to run) and sometimes in the middle of jobs executing (updating job progress etc). This led to various problems with jobs getting stuck etc.
A single queue processing daemon for each environment, production & staging, were running at the time.
Whilst trying to identify what was grabbing the queue locks I went down the rabbit hole of trying to determine if different environments could be impacting one another, and I believe they can.
MySQL's get_lock() sets a global lock, irrespective of database selected or connected user (confirmed this with a bit of manual testing).
Yii/Craft does not include the environment name in the mutex lock name for the queue (and likely other stuff but I haven't checked).
This can therefore lead to mutex locks in different logical environments colliding with one another if they are using a shared database instance, as is common outside of larger scale infrastructure setups.
I'm wondering whether including the current environment name in mutex names might be a useful addition to avoid this, or whether changing the APP_ID between environments is a preferred approach (although that feels a bit weird to me).
Steps to reproduce
- Run two copies of Craft, sharing a single MySQL DB instance but using different databases to store data.
- Mutex collisions will occur between environments, especially on the queue if both environments are busy doing things.
Craft CMS version
4.9.7
PHP version
8.1
Operating system and version
Alpine 3.18
Database type and version
MariaDB 11 but I believe MySQL 8 acts in the same way
Image driver and version
Installed plugins and versions
What happened?
Description
We were seeing
Could not acquire a mutex lock for the queueerrors on a project occurring randomly, sometimes between jobs executing (when trying to get the next job to run) and sometimes in the middle of jobs executing (updating job progress etc). This led to various problems with jobs getting stuck etc.A single queue processing daemon for each environment, production & staging, were running at the time.
Whilst trying to identify what was grabbing the queue locks I went down the rabbit hole of trying to determine if different environments could be impacting one another, and I believe they can.
MySQL's
get_lock()sets a global lock, irrespective of database selected or connected user (confirmed this with a bit of manual testing).Yii/Craft does not include the environment name in the mutex lock name for the queue (and likely other stuff but I haven't checked).
This can therefore lead to mutex locks in different logical environments colliding with one another if they are using a shared database instance, as is common outside of larger scale infrastructure setups.
I'm wondering whether including the current environment name in mutex names might be a useful addition to avoid this, or whether changing the APP_ID between environments is a preferred approach (although that feels a bit weird to me).
Steps to reproduce
Craft CMS version
4.9.7
PHP version
8.1
Operating system and version
Alpine 3.18
Database type and version
MariaDB 11 but I believe MySQL 8 acts in the same way
Image driver and version
Installed plugins and versions