Skip to content

Make BigQueryHook compatible with sqlalchemy #19507

@david30907d

Description

@david30907d

Description

hook = BigQueryHook('google_cloud_default', use_legacy_sql=False)
hook.get_sqlalchemy_engine()

this one would raise this exception 👇

[2021-11-10 08:26:06,927] {taskinstance.py:1501} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1361, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 150, in execute
    return_value = self.execute_callable()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 161, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/opt/airflow/dags/dags/utils/others/subscription_related.py", line 98, in wrapper
    return func(*args, **kwargs)
  File "/opt/airflow/dags/dags/ods/ingest_emails/udfs/ingest_emails.py", line 59, in extract_and_load_campaign_emails
    esp.extract_and_load_campaign_emails(context)
  File "/opt/airflow/dags/dags/utils/factories/esp_factory/klaviyo_esp.py", line 230, in extract_and_load_campaign_emails
    olap_session = engine2session(self.olap_hook.get_sqlalchemy_engine())
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/hooks/dbapi.py", line 100, in get_sqlalchemy_engine
    return create_engine(self.get_uri(), **engine_kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/__init__.py", line 525, in create_engine
    return strategy.create(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 61, in create
    entrypoint = u._get_entrypoint()
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/url.py", line 172, in _get_entrypoint
    cls = registry.load(name)
  File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 277, in load
    raise exc.NoSuchModuleError(
sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:google_cloud_platform

Use case/motivation

I hope this sqlalchemy toy example works

from sqlalchemy.orm import sessionmaker, scoped_session
engine = hook.get_sqlalchemy_engine()
scoped_session(sessionmaker(bind=engine, autocommit=True))

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions