-
Notifications
You must be signed in to change notification settings - Fork 16.5k
Closed
Labels
Description
Apache Airflow Provider(s)
amazon
Versions of Apache Airflow Providers
Providers included in apache/airflow:2.4.0 docker image:
apache-airflow==2.4.0
apache-airflow-providers-amazon==5.1.0
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-cncf-kubernetes==4.3.0
apache-airflow-providers-common-sql==1.2.0
apache-airflow-providers-docker==3.1.0
apache-airflow-providers-elasticsearch==4.2.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-google==8.3.0
apache-airflow-providers-grpc==3.0.0
apache-airflow-providers-hashicorp==3.1.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-microsoft-azure==4.2.0
apache-airflow-providers-mysql==3.2.0
apache-airflow-providers-odbc==3.1.1
apache-airflow-providers-postgres==5.2.1
apache-airflow-providers-redis==3.0.0
apache-airflow-providers-sendgrid==3.0.0
apache-airflow-providers-sftp==4.0.0
apache-airflow-providers-slack==5.1.0
apache-airflow-providers-sqlite==3.2.1
apache-airflow-providers-ssh==3.1.0
Apache Airflow version
2.4
Operating System
AWS Fargate
Deployment
Docker-Compose
Deployment details
We have configure the alternative secrets backend to use AWS SMP:
[secrets]
backend = airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
backend_kwargs = {"config_prefix": "/airflow2/config", "connections_prefix": "/airflow2/connections", "variables_prefix": "/airflow2/variables"}
What happened
All processes fail with:
`Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py", line 178, in _get_secret
response = self.client.get_parameter(Name=ssm_path, WithDecryption=True)
File "/home/airflow/.local/lib/python3.7/site-packages/cached_property.py", line 36, in __get__
value = obj.__dict__[self.func.__name__] = self.func(obj)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py", line 98, in client
from airflow.providers.amazon.aws.hooks.base_aws import SessionFactory
ImportError: cannot import name 'SessionFactory' from 'airflow.providers.amazon.aws.hooks.base_aws' (/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 122, in _get_config_value_from_secret_backend
return secrets_client.get_config(config_key)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py", line 167, in get_config
return self._get_secret(self.config_prefix, key)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py", line 180, in _get_secret
except self.client.exceptions.ParameterNotFound:
File "/home/airflow/.local/lib/python3.7/site-packages/cached_property.py", line 36, in __get__
value = obj.__dict__[self.func.__name__] = self.func(obj)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py", line 98, in client
from airflow.providers.amazon.aws.hooks.base_aws import SessionFactory
ImportError: cannot import name 'SessionFactory' from 'airflow.providers.amazon.aws.hooks.base_aws' (/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py", line 178, in _get_secret
response = self.client.get_parameter(Name=ssm_path, WithDecryption=True)
File "/home/airflow/.local/lib/python3.7/site-packages/cached_property.py", line 36, in __get__
value = obj.__dict__[self.func.__name__] = self.func(obj)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/systems_manager.py", line 98, in client
from airflow.providers.amazon.aws.hooks.base_aws import SessionFactory
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py", line 49, in <module>
from airflow.models.connection import Connection
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/connection.py", line 32, in <module>
from airflow.models.base import ID_LEN, Base
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py", line 76, in <module>
COLLATION_ARGS = get_id_collation_args()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py", line 70, in get_id_collation_args
conn = conf.get('database', 'sql_alchemy_conn', fallback='')
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 557, in get
option = self._get_option_from_secrets(deprecated_key, deprecated_section, key, section)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 577, in _get_option_from_secrets
option = self._get_secret_option(section, key)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 502, in _get_secret_option
return _get_config_value_from_secret_backend(secrets_path)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", line 125, in _get_config_value_from_secret_backend
'Cannot retrieve config from alternative secrets backend. '
airflow.exceptions.AirflowConfigException: Cannot retrieve config from alternative secrets backend. Make sure it is configured properly and that the Backend is accessible.
cannot import name 'SessionFactory' from 'airflow.providers.amazon.aws.hooks.base_aws' (/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py)
`
What you think should happen instead
Airflow 2.3.4 was using amazon provider 5.0.0 and everything was working fine. Looking at the SystemsManagerParameterStoreBackend class, the client method changed in amazon 5.1.0 (coming with AF 2.4). There use to be a simple boto3.session call. The new code calls for an import of SessionFactory. I do not understand why this import fails though.
How to reproduce
I assume anyone that sets up the parameter store as backend and try to use the docker image (FROM apache/airflow:2.4.0) will run into this issue.
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Reactions are currently unavailable