-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Closed as not planned
airflow-helm/charts
#772Labels
area:coreinvalidkind:bugThis is a clearly a bugThis is a clearly a bugneeds-triagelabel for new issues that we didn't triage yetlabel for new issues that we didn't triage yet
Description
Apache Airflow version
2.7.0
What happened
After deploying Airflow Helm Chart (User Community) with image.tag set to 2.7.0-python3.8 and creating a user, sync-airflow-users pod exits with following log:
sync-airflow-users /home/airflow/.local/lib/python3.8/site-packages/flask_limiter/extension.py:293 UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
sync-airflow-users Traceback (most recent call last):
sync-airflow-users File "/mnt/scripts/sync_users.py", line 146, in <module>
sync-airflow-users from airflow.www.fab_security.sqla.models import User, Role
sync-airflow-users ModuleNotFoundError: No module named 'airflow.www.fab_security.sqla.models'
sync-airflow-users
sync-airflow-users During handling of the above exception, another exception occurred:
sync-airflow-users
sync-airflow-users Traceback (most recent call last):
sync-airflow-users File "/mnt/scripts/sync_users.py", line 148, in <module>
sync-airflow-users from flask_appbuilder.security.sqla.models import User, Role
sync-airflow-users File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/security/sqla/models.py", line 48, in <module>
sync-airflow-users assoc_permissionview_role = Table(
sync-airflow-users File "<string>", line 2, in __new__
sync-airflow-users File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py", line 375, in warned
sync-airflow-users return fn(*args, **kwargs)
sync-airflow-users File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 596, in __new__
sync-airflow-users raise exc.InvalidRequestError(
sync-airflow-users sqlalchemy.exc.InvalidRequestError: Table 'ab_permission_view_role' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object.
Stream closed EOF for airflow/airflow-sync-users-b989f959c-zz2dp (wait-for-db-migrations)
Stream closed EOF for airflow/airflow-sync-users-b989f959c-zz2dp (check-db)
Stream closed EOF for airflow/airflow-sync-users-b989f959c-zz2dp (sync-airflow-users)
I've seen these two issues which mention ab_permission_view_role, but the solutions provided there aren't a fix for my issue:
- Airflow 2.4.1 ab_permission_view_role database error #27138
- Airflow sqlalchemy.exc.InvalidRequestError: Table 'ab_permission_view_role' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object airflow-helm/charts#747
So setting:
web:
webserverConfig:
stringOverride: |
from airflow.www.fab_security.manager import AUTH_DB
AUTH_TYPE = AUTH_DB
doesn't fix the issue.
What you think should happen instead
sqlalchemy.exc.InvalidRequestError shouldn't be thrown.
sync-airflow-users pod shouldn't exit and keep restarting.
How to reproduce
- Deploy EKS cluster v.1.2.5.
- Deploy RDS cluster (PostgreSQL db).
- Deploy Airflow community helm chart using Terraform code provided below.
- Shell into
airflow-weband create a user usingairflow users create -u ... -f ... -l ... -p ... -r Admin -e .... sync-airflow-userspod will be restarted because of the error provided above.
Terraform code used to deploy Airflow:
locals {
# workaround: https://github.com/hashicorp/terraform-provider-helm/issues/669#issuecomment-765171927
values = {
airflow = {
image = {
tag = "2.7.0-python3.8"
}
connections = [{
"id" = "aws_default",
"type" = "aws",
"description" = "AWS connection to send emails using SES",
"extra" = <<EOF
{
"region_name": "${data.aws_region.current.name}"
}
EOF
}]
}
}
}
resource "helm_release" "airflow" {
name = "airflow"
repository = "https://airflow-helm.github.io/charts"
version = "8.7.1"
chart = "airflow"
namespace = local.namespace
atomic = true
wait = true
timeout = 60 * 20 # 20 minutes
dynamic "set" {
for_each = {
"airflow.executor" = "KubernetesExecutor"
"workers.persistence.enabled" = false
"postgresql.persistence.enabled" = false
"scheduler.logCleanup.enabled" = false
"flower.enabled" = false
"workers.enabled" = false
"redis.enabled" = false
"ingress.enabled" = true
"postgresql.enabled": false,
"pgbouncer.enabled": true,
"dags.persistence.enabled" = true
"dags.persistence.existingClaim" = local.pvc_name_dags
"dags.persistence.accessMode" = "ReadOnlyMany"
"logs.persistence.enabled": true
"logs.persistence.existingClaim": local.pvc_name_logs
"logs.persistence.accessMode": "ReadWriteMany"
"rbac.create": false,
"serviceAccount.create": false,
"serviceAccount.name": kubernetes_service_account.airflow_services.metadata.0.name,
"externalDatabase.type": "postgres",
"externalDatabase.host": var.postgres_endpoint,
"externalDatabase.port": var.postgres_port,
"externalDatabase.database": "...",
"externalDatabase.userSecret": var.postgres_secret_names[0],
"externalDatabase.userSecretKey": "username",
"externalDatabase.passwordSecret": var.postgres_secret_names[0],
"externalDatabase.passwordSecretKey": "password"
}
content {
name = set.key
value = set.value
}
}
# workaround: https://github.com/hashicorp/terraform-provider-helm/issues/669#issuecomment-765171927
values = [yamlencode(local.values)]
depends_on = [...]
}
Operating System
Debian GNU/Linux 11 (bullseye)
Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==7.4.2
Deployment
Other 3rd-party Helm chart
Deployment details
EKS Fargate deployment type in AWS.
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Metadata
Metadata
Assignees
Labels
area:coreinvalidkind:bugThis is a clearly a bugThis is a clearly a bugneeds-triagelabel for new issues that we didn't triage yetlabel for new issues that we didn't triage yet