Skip to content

Error deploying Airflow Helm Chart 2.7.0-python3.8: sync-airflow-users pod issue with 'ab_permission_view_role' is already defined #33635

@VictorAtPL

Description

@VictorAtPL

Apache Airflow version

2.7.0

What happened

After deploying Airflow Helm Chart (User Community) with image.tag set to 2.7.0-python3.8 and creating a user, sync-airflow-users pod exits with following log:

sync-airflow-users /home/airflow/.local/lib/python3.8/site-packages/flask_limiter/extension.py:293 UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
sync-airflow-users Traceback (most recent call last):
sync-airflow-users   File "/mnt/scripts/sync_users.py", line 146, in <module>
sync-airflow-users     from airflow.www.fab_security.sqla.models import User, Role
sync-airflow-users ModuleNotFoundError: No module named 'airflow.www.fab_security.sqla.models'
sync-airflow-users 
sync-airflow-users During handling of the above exception, another exception occurred:
sync-airflow-users 
sync-airflow-users Traceback (most recent call last):
sync-airflow-users   File "/mnt/scripts/sync_users.py", line 148, in <module>
sync-airflow-users     from flask_appbuilder.security.sqla.models import User, Role
sync-airflow-users   File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/security/sqla/models.py", line 48, in <module>
sync-airflow-users     assoc_permissionview_role = Table(
sync-airflow-users   File "<string>", line 2, in __new__
sync-airflow-users   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py", line 375, in warned
sync-airflow-users     return fn(*args, **kwargs)
sync-airflow-users   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 596, in __new__
sync-airflow-users     raise exc.InvalidRequestError(
sync-airflow-users sqlalchemy.exc.InvalidRequestError: Table 'ab_permission_view_role' is already defined for this MetaData instance.  Specify 'extend_existing=True' to redefine options and columns on an existing Table object.
Stream closed EOF for airflow/airflow-sync-users-b989f959c-zz2dp (wait-for-db-migrations)
Stream closed EOF for airflow/airflow-sync-users-b989f959c-zz2dp (check-db)
Stream closed EOF for airflow/airflow-sync-users-b989f959c-zz2dp (sync-airflow-users)

I've seen these two issues which mention ab_permission_view_role, but the solutions provided there aren't a fix for my issue:

  1. Airflow 2.4.1 ab_permission_view_role database error #27138
  2. Airflow sqlalchemy.exc.InvalidRequestError: Table 'ab_permission_view_role' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object airflow-helm/charts#747

So setting:

web:
webserverConfig:
stringOverride: |
from airflow.www.fab_security.manager import AUTH_DB
AUTH_TYPE = AUTH_DB

doesn't fix the issue.

What you think should happen instead

sqlalchemy.exc.InvalidRequestError shouldn't be thrown.

sync-airflow-users pod shouldn't exit and keep restarting.

How to reproduce

  1. Deploy EKS cluster v.1.2.5.
  2. Deploy RDS cluster (PostgreSQL db).
  3. Deploy Airflow community helm chart using Terraform code provided below.
  4. Shell into airflow-web and create a user using airflow users create -u ... -f ... -l ... -p ... -r Admin -e ....
  5. sync-airflow-users pod will be restarted because of the error provided above.

Terraform code used to deploy Airflow:

locals {
  # workaround: https://github.com/hashicorp/terraform-provider-helm/issues/669#issuecomment-765171927
  values = {
    airflow = {
      image = {
        tag = "2.7.0-python3.8"
      }
      connections = [{
        "id" = "aws_default",
        "type" = "aws",
        "description" = "AWS connection to send emails using SES",
        "extra" = <<EOF
{
"region_name": "${data.aws_region.current.name}"
}
EOF
      }]
    }
  }
}

resource "helm_release" "airflow" {
  name       = "airflow"
  repository = "https://airflow-helm.github.io/charts"
  version = "8.7.1"
  chart      = "airflow"
  namespace = local.namespace
  atomic = true
  wait = true
  timeout = 60 * 20 # 20 minutes

  dynamic "set" {
    for_each = {
      "airflow.executor" = "KubernetesExecutor"
      "workers.persistence.enabled" = false
      "postgresql.persistence.enabled" = false
      "scheduler.logCleanup.enabled" = false
      "flower.enabled" = false
      "workers.enabled" = false
      "redis.enabled" = false
      "ingress.enabled" = true
      "postgresql.enabled": false,
      "pgbouncer.enabled": true,
      "dags.persistence.enabled" = true
      "dags.persistence.existingClaim" = local.pvc_name_dags
      "dags.persistence.accessMode" = "ReadOnlyMany"
      "logs.persistence.enabled": true
      "logs.persistence.existingClaim": local.pvc_name_logs
      "logs.persistence.accessMode": "ReadWriteMany"
      "rbac.create": false,
      "serviceAccount.create": false,
      "serviceAccount.name": kubernetes_service_account.airflow_services.metadata.0.name,
      "externalDatabase.type": "postgres",
      "externalDatabase.host": var.postgres_endpoint,
      "externalDatabase.port": var.postgres_port,
      "externalDatabase.database": "...",
      "externalDatabase.userSecret": var.postgres_secret_names[0],
      "externalDatabase.userSecretKey": "username",
      "externalDatabase.passwordSecret": var.postgres_secret_names[0],
      "externalDatabase.passwordSecretKey": "password"
    }

    content {
      name  = set.key
      value = set.value
    }
  }

  # workaround: https://github.com/hashicorp/terraform-provider-helm/issues/669#issuecomment-765171927
  values = [yamlencode(local.values)]

  depends_on = [...]
}

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes==7.4.2

Deployment

Other 3rd-party Helm chart

Deployment details

EKS Fargate deployment type in AWS.

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions