fix: `to_gbq` allows strings for DATE and floats for NUMERIC, require pandas 0.24+ and db-dtypes #423

tswast · 2021-11-11T21:51:36Z

deps: require pandas 0.24+ and db-dtypes for TIME/DATE extension dtypes

~~Review #420 first! This PR is based on changes to the system tests introduced there.~~

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Fixes #421 🦕

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…igquery-pandas into issue362-to_gbq-date

…MERIC This improves `api_method="load_parquet"` compatibility with the previous `api_method="load_csv"` behavior.

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…1-numeric

tswast · 2021-11-11T22:51:09Z

Need to package db-dtypes package in order to pass conda session.

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

plamut

Found a few things worth double-checking, but the general picture looks good.

pandas_gbq/load.py

tests/system/test_to_gbq.py

tests/unit/test_load.py

owlbot.py

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

tswast · 2021-11-17T20:26:35Z

nox > Running session system-3.7
nox > Creating virtual environment (virtualenv) using python3.7 in .nox/system-3-7
nox > python -m pip install --pre grpcio
nox > python -m pip install mock pytest google-cloud-testutils -c /tmpfs/src/github/python-bigquery-pandas/testing/constraints-3.7.txt
nox > python -m pip install -e .[tqdm] -c /tmpfs/src/github/python-bigquery-pandas/testing/constraints-3.7.txt
nox > py.test --quiet --junitxml=system_3.7_sponge_log.xml tests/system
.sss......F.FF.........................................................s [ 73%]
..........................                                               [100%]
=================================== FAILURES ===================================
_____ TestReadGBQIntegration.test_should_properly_handle_nullable_integers _____

self = <system.test_gbq.TestReadGBQIntegration object at 0x7f550a5fe910>
project_id = 'precise-truck-742'

    def test_should_properly_handle_nullable_integers(self, project_id):
        if PANDAS_VERSION < NULLABLE_INT_PANDAS_VERSION:
            pytest.skip(msg=NULLABLE_INT_MESSAGE)
    
        query = """SELECT * FROM
                    UNNEST([1, NULL]) AS nullable_integer
                """
        df = gbq.read_gbq(
            query,
            project_id=project_id,
            credentials=self.credentials,
            dialect="standard",
            dtypes={"nullable_integer": "Int64"},
        )
        tm.assert_frame_equal(
            df,
            DataFrame(
>               {"nullable_integer": pandas.Series([1, pandas.NA], dtype="Int64")}
            ),
        )
E       AttributeError: module 'pandas' has no attribute 'NA'

tests/system/test_gbq.py:192: AttributeError
----------------------------- Captured stderr call -----------------------------

Downloading: 0rows [00:00, ?rows/s]
Downloading: 100%|██████████| 2/2 [00:00<00:00,  5.88rows/s]
______ TestReadGBQIntegration.test_should_properly_handle_nullable_longs _______

self = <system.test_gbq.TestReadGBQIntegration object at 0x7f550a580bd0>
project_id = 'precise-truck-742'

    def test_should_properly_handle_nullable_longs(self, project_id):
        if PANDAS_VERSION < NULLABLE_INT_PANDAS_VERSION:
            pytest.skip(msg=NULLABLE_INT_MESSAGE)
    
        query = """SELECT * FROM
                    UNNEST([1 << 62, NULL]) AS nullable_long
                """
        df = gbq.read_gbq(
            query,
            project_id=project_id,
            credentials=self.credentials,
            dialect="standard",
            dtypes={"nullable_long": "Int64"},
        )
        tm.assert_frame_equal(
            df,
            DataFrame(
>               {"nullable_long": pandas.Series([1 << 62, pandas.NA], dtype="Int64")}
            ),
        )
E       AttributeError: module 'pandas' has no attribute 'NA'

tests/system/test_gbq.py:223: AttributeError
----------------------------- Captured stderr call -----------------------------

Downloading: 0rows [00:00, ?rows/s]
Downloading: 100%|██████████| 2/2 [00:00<00:00, 10.43rows/s]
_______ TestReadGBQIntegration.test_should_properly_handle_null_integers _______

self = <system.test_gbq.TestReadGBQIntegration object at 0x7f550a5c5850>
project_id = 'precise-truck-742'

    def test_should_properly_handle_null_integers(self, project_id):
        if PANDAS_VERSION < NULLABLE_INT_PANDAS_VERSION:
            pytest.skip(msg=NULLABLE_INT_MESSAGE)
    
        query = "SELECT CAST(NULL AS INT64) AS null_integer"
        df = gbq.read_gbq(
            query,
            project_id=project_id,
            credentials=self.credentials,
            dialect="standard",
            dtypes={"null_integer": "Int64"},
        )
        tm.assert_frame_equal(
>           df, DataFrame({"null_integer": pandas.Series([pandas.NA], dtype="Int64")}),
        )
E       AttributeError: module 'pandas' has no attribute 'NA'

tests/system/test_gbq.py:240: AttributeError
----------------------------- Captured stderr call -----------------------------

Possible we need to adjust NULLABLE_INT_PANDAS_VERSION

tswast · 2021-11-17T20:42:58Z

Marking as DO NOT MERGE as a reminder to make sure

deps: require pandas 0.24+ and db-dtypes for TIME/DATE extension dtypes
  Committer: @tswast
  Source-Link: googleapis/python-bigquery-pandas@19df618d9728eef07a9d70bca6d9600dc440ac63

is included as a footer to the PR, as I'd like to see if we can use this feature: googleapis/release-please#686

tswast · 2021-11-17T20:54:54Z

Per googleapis/release-please#821, the extra metadata isn't important. It's that the extra commits are listed as a footer to the commit message without anything in-between.

tswast · 2021-11-18T22:47:30Z

Ready for (re)review.

plamut

LGTM now, thanks for the cleanup!

plamut · 2021-11-20T13:38:44Z

The nightly test failure is probably flakiness? The logs say "killed"?

tswast · 2021-11-22T15:26:34Z

Nightly failure isn't even flakey. It always fails because the conda package installation times out. #424

Switching to mamba for installs might help.

🤖 I have created a release *beep* *boop* --- ## [0.17.0](v0.16.0...v0.17.0) (2022-01-19) ### ⚠ BREAKING CHANGES * use nullable Int64 and boolean dtypes if available (#445) ### Features * accepts a table ID, which downloads the table without a query ([#443](#443)) ([bf0e863](bf0e863)) * use nullable Int64 and boolean dtypes if available ([#445](#445)) ([89078f8](89078f8)) ### Bug Fixes * `read_gbq` supports extreme DATETIME values such as `0001-01-01 00:00:00` ([#444](#444)) ([d120f8f](d120f8f)) * `to_gbq` allows strings for DATE and floats for NUMERIC with `api_method="load_parquet"` ([#423](#423)) ([2180836](2180836)) * allow extreme DATE values such as `datetime.date(1, 1, 1)` in `load_gbq` ([#442](#442)) ([e13abaf](e13abaf)) * avoid iteritems deprecation in pandas prerelease ([#469](#469)) ([7379cdc](7379cdc)) * use data project for destination in `to_gbq` ([#455](#455)) ([891a00c](891a00c)) ### Miscellaneous Chores * release 0.17.0 ([#470](#470)) ([29ac8c3](29ac8c3)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [0.17.0](googleapis/python-bigquery-pandas@v0.16.0...v0.17.0) (2022-01-19) ### ⚠ BREAKING CHANGES * use nullable Int64 and boolean dtypes if available (#445) ### Features * accepts a table ID, which downloads the table without a query ([#443](googleapis/python-bigquery-pandas#443)) ([bf0e863](googleapis/python-bigquery-pandas@bf0e863)) * use nullable Int64 and boolean dtypes if available ([#445](googleapis/python-bigquery-pandas#445)) ([89078f8](googleapis/python-bigquery-pandas@89078f8)) ### Bug Fixes * `read_gbq` supports extreme DATETIME values such as `0001-01-01 00:00:00` ([#444](googleapis/python-bigquery-pandas#444)) ([d120f8f](googleapis/python-bigquery-pandas@d120f8f)) * `to_gbq` allows strings for DATE and floats for NUMERIC with `api_method="load_parquet"` ([#423](googleapis/python-bigquery-pandas#423)) ([2180836](googleapis/python-bigquery-pandas@2180836)) * allow extreme DATE values such as `datetime.date(1, 1, 1)` in `load_gbq` ([#442](googleapis/python-bigquery-pandas#442)) ([e13abaf](googleapis/python-bigquery-pandas@e13abaf)) * avoid iteritems deprecation in pandas prerelease ([#469](googleapis/python-bigquery-pandas#469)) ([7379cdc](googleapis/python-bigquery-pandas@7379cdc)) * use data project for destination in `to_gbq` ([#455](googleapis/python-bigquery-pandas#455)) ([891a00c](googleapis/python-bigquery-pandas@891a00c)) ### Miscellaneous Chores * release 0.17.0 ([#470](googleapis/python-bigquery-pandas#470)) ([29ac8c3](googleapis/python-bigquery-pandas@29ac8c3)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

tswast and others added 8 commits November 10, 2021 17:05

test: upload DATE column with various dtypes

9b57878

add dbdate tests

907a1d5

test with db-dtypes only with newer pandas

cf2a5bf

🦉 Updates from OwlBot

1ea8fda

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

sort by row number

b869a9f

Merge branch 'issue362-to_gbq-date' of github.com:googleapis/python-b…

4d304f5

…igquery-pandas into issue362-to_gbq-date

Merge branch 'main' into issue362-to_gbq-date

b876cc7

fix: allow strings when writing to DATE and floats when writing to NU…

625df5a

…MERIC This improves `api_method="load_parquet"` compatibility with the previous `api_method="load_csv"` behavior.

google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Nov 11, 2021

product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-pandas API. label Nov 11, 2021

gcf-owl-bot bot and others added 4 commits November 11, 2021 21:53

🦉 Updates from OwlBot

bd3604e

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

require db-dtypes

19df618

Merge remote-tracking branch 'upstream/issue421-numeric' into issue42…

e0c0a1f

…1-numeric

add unit tests for dataframe conversion

ab85d8a

tswast marked this pull request as ready for review November 11, 2021 22:33

tswast requested a review from a team as a code owner November 11, 2021 22:33

tswast and others added 2 commits November 11, 2021 16:51

fix unit tests

a08f90d

🦉 Updates from OwlBot

4426957

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

tswast mentioned this pull request Nov 11, 2021

[Package request] db-dtypes conda-forge/staged-recipes#16903

Closed

2 tasks

remove 'default' from system tests. redundant with load_parquet

15d7b51

plamut mentioned this pull request Nov 12, 2021

test: upload DATE column with various dtypes #420

Merged

4 tasks

plamut reviewed Nov 12, 2021

View reviewed changes

pandas_gbq/load.py Show resolved Hide resolved

pandas_gbq/load.py Outdated Show resolved Hide resolved

tests/system/test_to_gbq.py Outdated Show resolved Hide resolved

tests/unit/test_load.py Outdated Show resolved Hide resolved

owlbot.py Show resolved Hide resolved

tswast and others added 5 commits November 16, 2021 16:59

Merge branch 'main' into issue421-numeric

5ded1e8

🦉 Updates from OwlBot

4ddcf9d

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

correct repeated_col name

95f051f

used namedtuple in tests

f8dddb2

protect against explicit None

7563017

update tests since pandas 0.24+ is required

92837e6

tswast changed the title ~~fix: allow strings when writing to DATE and floats when writing to NUMERIC~~ fix: to_gbq allows strings for DATE and floats for NUMERIC, require pandas 0.24+ and db-dtypes Nov 17, 2021

tswast requested a review from plamut November 17, 2021 20:40

tswast added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Nov 17, 2021

update conda test pandas version

f981705

tswast requested a review from loferris November 18, 2021 22:47

plamut approved these changes Nov 20, 2021

View reviewed changes

tswast merged commit 2180836 into main Nov 22, 2021

tswast deleted the issue421-numeric branch November 22, 2021 15:28

tswast mentioned this pull request Dec 1, 2021

chore: release 0.16.1 #427

Closed

release-please bot mentioned this pull request Jan 11, 2022

chore(main): release 1.0.0 #461

Closed

release-please bot mentioned this pull request Jan 19, 2022

chore(main): release 0.17.0 #472

Merged

tswast mentioned this pull request Feb 5, 2025

ImportError: pandas-gbq requires db-dtypes #873

Closed

blunderbuss-gcf bot assigned yirutang Sep 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: `to_gbq` allows strings for DATE and floats for NUMERIC, require pandas 0.24+ and db-dtypes #423

fix: `to_gbq` allows strings for DATE and floats for NUMERIC, require pandas 0.24+ and db-dtypes #423

Uh oh!

tswast commented Nov 11, 2021 •

edited

Loading

Uh oh!

tswast commented Nov 11, 2021

Uh oh!

plamut left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tswast commented Nov 17, 2021

Uh oh!

tswast commented Nov 17, 2021 •

edited

Loading

Uh oh!

tswast commented Nov 17, 2021

Uh oh!

tswast commented Nov 18, 2021

Uh oh!

plamut left a comment

Uh oh!

plamut commented Nov 20, 2021

Uh oh!

tswast commented Nov 22, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fix: to_gbq allows strings for DATE and floats for NUMERIC, require pandas 0.24+ and db-dtypes #423

fix: to_gbq allows strings for DATE and floats for NUMERIC, require pandas 0.24+ and db-dtypes #423

Uh oh!

Conversation

tswast commented Nov 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tswast commented Nov 11, 2021

Uh oh!

plamut left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tswast commented Nov 17, 2021

Uh oh!

tswast commented Nov 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tswast commented Nov 17, 2021

Uh oh!

tswast commented Nov 18, 2021

Uh oh!

plamut left a comment

Choose a reason for hiding this comment

Uh oh!

plamut commented Nov 20, 2021

Uh oh!

tswast commented Nov 22, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fix: `to_gbq` allows strings for DATE and floats for NUMERIC, require pandas 0.24+ and db-dtypes #423

fix: `to_gbq` allows strings for DATE and floats for NUMERIC, require pandas 0.24+ and db-dtypes #423

tswast commented Nov 11, 2021 •

edited

Loading

tswast commented Nov 17, 2021 •

edited

Loading