What do you see as an issue?
Similar to how sql_alchemy_pool_recycle defaults to 1800 seconds for the Airflow metastore: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#config-database-sql-alchemy-pool-recycle
If users are using celery as their backend it provides extra stability to set pool_recycle. This problem is particularly acute for users who are using MySQL as backend for tasks because MySQL disconnects connections after 8 hours of being idle. While Airflow can usually force celery to retry connecting it does not always work and tasks can fail.
This is specifically reccomended by the SqlAlchemy docs:
Solving the problem
We currently have a file which looks like this:
from airflow.config_templates.default_celery import DEFAULT_CELERY_CONFIG
database_engine_options = DEFAULT_CELERY_CONFIG.get(
"database_engine_options", {}
)
# Use pool_pre_ping to detect stale db connections
# https://github.com/apache/airflow/discussions/22113
database_engine_options["pool_pre_ping"] = True
# Use pool recyle due to MySQL disconnecting sessions after 8 hours
# https://docs.sqlalchemy.org/en/14/core/pooling.html#setting-pool-recycle
# https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine.params.pool_recycle
# https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_wait_timeout
database_engine_options["pool_recycle"] = 1800
DEFAULT_CELERY_CONFIG["database_engine_options"] = database_engine_optionsAnd we point the env var AIRFLOW__CELERY__CELERY_CONFIG_OPTIONS to this object, not sure if this is best practise?
Anything else
Maybe just change the default options to include this?
Are you willing to submit PR?
Code of Conduct
What do you see as an issue?
Similar to how
sql_alchemy_pool_recycledefaults to 1800 seconds for the Airflow metastore: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#config-database-sql-alchemy-pool-recycleIf users are using celery as their backend it provides extra stability to set
pool_recycle. This problem is particularly acute for users who are using MySQL as backend for tasks because MySQL disconnects connections after 8 hours of being idle. While Airflow can usually force celery to retry connecting it does not always work and tasks can fail.This is specifically reccomended by the SqlAlchemy docs:
Solving the problem
We currently have a file which looks like this:
And we point the env var
AIRFLOW__CELERY__CELERY_CONFIG_OPTIONSto this object, not sure if this is best practise?Anything else
Maybe just change the default options to include this?
Are you willing to submit PR?
Code of Conduct