Skip to content

BigQueryTableExistenceSensor needs to specify keyword arguments for hook.table_exists #9830

@nathadfield

Description

@nathadfield

Apache Airflow version: 1.10.*
Environment:

  • Cloud provider or hardware configuration: GCP

  • OS (e.g. from /etc/os-release):
    PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
    NAME="Debian GNU/Linux"
    VERSION_ID="9"
    VERSION="9 (stretch)"
    VERSION_CODENAME=stretch
    ID=debian

  • Kernel (e.g. uname -a): Linux ad0a7f00d44a 4.19.76-linuxkit Improving the search functionality in the graph view #1 SMP Tue May 26 11:42:35 UTC 2020 x86_64 GNU/Linux

  • Install tools: python3.7, pip install apache-airflow[all]==1.10.11

  • Others:

What happened:

When using the BigQueryTableExistenceSensor to check on the presence of a table in BigQuery the following error is returned.

Traceback (most recent call last):
File "/usr/local/bin/airflow", line 37, in
args.func(args)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 76, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 724, in test
ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1107, in run
session=session)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 70, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.7/site-packages/airflow/sensors/base_sensor_operator.py", line 107, in execute
while not self.poke(context):
File "/usr/local/lib/python3.7/site-packages/airflow/providers/google/cloud/sensors/bigquery.py", line 74, in poke
return hook.table_exists(self.project_id, self.dataset_id, self.table_id)
File "/usr/local/lib/python3.7/site-packages/airflow/providers/google/common/hooks/base_google.py", line 346, in inner_wrapper
"You must use keyword arguments in this methods rather than"
airflow.exceptions.AirflowException: You must use keyword arguments in this methods rather than positional

What you expected to happen:

I think this issue is because, when the operator calls the hook.table_exists method, it does not specifically state each argument.

https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/sensors/bigquery.py#L74

The test for this method does call it correctly.

https://github.com/apache/airflow/blob/master/tests/providers/google/cloud/hooks/test_bigquery.py#L105

How to reproduce it: Simply running a task based on this operator should reproduce this error.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind:bugThis is a clearly a bug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions