Skip to content

Comments

Typecast bigquery job response col value#27236

Merged
potiuk merged 5 commits intoapache:mainfrom
astronomer:fix_bq_check_op
Oct 31, 2022
Merged

Typecast bigquery job response col value#27236
potiuk merged 5 commits intoapache:mainfrom
astronomer:fix_bq_check_op

Conversation

@pankajastro
Copy link
Member

@pankajastro pankajastro commented Oct 24, 2022

closes: #27237

The Bigquery hook get_records always return a list of string irrespective of the Bigquery table column type. So if even my table column has value 0 the task succeeds since bool("0") returns True.

This PR fixes it by typecasting the column value using _bq_cast


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Oct 24, 2022
@pankajastro pankajastro marked this pull request as ready for review October 24, 2022 20:12
@eladkal
Copy link
Contributor

eladkal commented Oct 24, 2022

Can you add unit test to avoid regression?

BTW you can open PRs directly. No need to submit issue first

@pankajastro
Copy link
Member Author

Can you add unit test to avoid regression?

BTW you can open PRs directly. No need to submit issue first

Added tests, please let me know how it's looking now.
While creating the issue by mistake I selected the area core can you please update the correct label remove core and add provider and google?

@potiuk potiuk merged commit 1447158 into apache:main Oct 31, 2022
@pankajastro pankajastro changed the title Typecast biquery job response col value Typecast bigquery job response col value Nov 1, 2022
@pankajastro pankajastro deleted the fix_bq_check_op branch November 5, 2022 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BigQueryCheckOperator fail in deferrable mode even if col val 0

3 participants