Skip to content

feat(deps): Update BigQuery plugin-sdk to v1.39.0 #8230

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
kodiakhq merged 9 commits into from
Feb 22, 2023
Merged

feat(deps): Update BigQuery plugin-sdk to v1.39.0 #8230

kodiakhq merged 9 commits into from
Feb 22, 2023

Conversation

@erezrokah
Copy link
Member

@erezrokah erezrokah commented Feb 19, 2023

Summary

Extracted from #8156.
The new SDK adds a new test:

  1. Create initial scheme and write stuff
  2. Add a new column
  3. Migrate to new schema
  4. Write more stuff
  5. Validate new stuff was created

BiqQuery can have a few minutes delay until the schema is updated, so the new test fails when validating the new data (the data is missing the new column).
This PR adds a test configuration to wait additional time for schema propagation. Another option is to skip the test

@erezrokah erezrokah requested review from a team, amanenk and hermanschaaf and removed request for a team and amanenk February 19, 2023 15:43
candiduslynx
candiduslynx suggested changes Feb 19, 2023
View reviewed changes
@yevgenypats
Copy link
Contributor

yevgenypats commented Feb 20, 2023

Looks good but seems tests still fail, is it because of the timeout? I would prefer the delay instead of failing the tests

@erezrokah
Copy link
Member Author

erezrokah commented Feb 20, 2023

Looks good but seems tests still fail, is it because of the timeout? I would prefer the delay instead of failing the tests

I'm taking another look, there might be an additional issue regarding comparison order similar to cloudquery/plugin-sdk#565 and cloudquery/plugin-sdk#561

@erezrokah erezrokah changed the title feat(deps): Update BigQuery plugin-sdk to v1.38.1 feat(deps): Update BigQuery plugin-sdk to v1.38.2 Feb 20, 2023
@erezrokah erezrokah force-pushed the feat/update_bigquery_sdk branch from 9b0c257 to 997f04e Compare February 20, 2023 13:19
@erezrokah erezrokah force-pushed the feat/update_bigquery_sdk branch from 997f04e to a876057 Compare February 20, 2023 13:21
@erezrokah erezrokah mentioned this pull request Feb 20, 2023
Closed
erezrokah
erezrokah commented Feb 20, 2023
View reviewed changes
@erezrokah erezrokah self-assigned this Feb 20, 2023
@erezrokah erezrokah added the automerge Automatically merge once required checks pass label Feb 21, 2023
@hermanschaaf
Copy link
Member

hermanschaaf commented Feb 21, 2023 •
edited
Loading

Looks like the tests are now failing @erezrokah:

--- FAIL: TestPlugin (368.79s)
    --- FAIL: TestPlugin/TestMigrateAppend (246.12s)
        --- FAIL: TestPlugin/TestMigrateAppend/add_column (123.29s)
            plugin_testing_migrate.go:67: 
                	Error Trace:	/home/runner/work/cloudquery/cloudquery/plugins/destination/bigquery/client/plugin_testing_migrate.go:67
                	            				/home/runner/work/cloudquery/cloudquery/plugins/destination/bigquery/client/plugin_testing_migrate.go:119
                	Error:      	[]schema.CQTypes{schema.CQTypes{(*schema.Text)(0xc0004c07b0), (*schema.Timestamptz)(0xc0000886e0), (*schema.UUID)(0xc000044780), (*schema.UUID)(0xc000044798), (*schema.Bool)(0xc00040661e)}, schema.CQTypes{(*schema.Text)(0xc0004c07c8), (*schema.Timestamptz)(0xc000088700), (*schema.UUID)(0xc0000447b0), (*schema.UUID)(0xc0000447c8), (*schema.Bool)(0xc00040662e)}} does not contain schema.CQTypes{(*schema.Text)(0xc00029c0d8), (*schema.Timestamptz)(0xc0001fe1a0), (*schema.UUID)(0xc0002d60a8), (*schema.UUID)(0xc0002d61e0), (*schema.Bool)(0xc00003e98e)}
                	Test:       	TestPlugin/TestMigrateAppend/add_column

@erezrokah
Copy link
Member Author

erezrokah commented Feb 21, 2023

Looks like the tests are now failing @erezrokah:

Yeah, I think I'll either need to increase the wait time or use an emulator. It's already quite slow to test, so I'll try the emulator

@erezrokah erezrokah removed the automerge Automatically merge once required checks pass label Feb 21, 2023
@erezrokah erezrokah marked this pull request as draft February 21, 2023 10:41
@erezrokah
Copy link
Member Author

erezrokah commented Feb 22, 2023

Tried the emulator but ran into goccy/bigquery-emulator#150

@erezrokah erezrokah marked this pull request as ready for review February 22, 2023 17:20
@erezrokah
Copy link
Member Author

erezrokah commented Feb 22, 2023 •
edited
Loading

I'll give the test delay one more go. If it doesn't work we'll have to disable some of the tests otherwise they will take too long

@erezrokah erezrokah force-pushed the feat/update_bigquery_sdk branch from 021984d to 0cff5ed Compare February 22, 2023 17:28
@erezrokah erezrokah added the automerge Automatically merge once required checks pass label Feb 22, 2023
@erezrokah
Copy link
Member Author

erezrokah commented Feb 22, 2023

Decided to skip the migrate test for now

@erezrokah erezrokah force-pushed the feat/update_bigquery_sdk branch from 0f5042f to 425ccad Compare February 22, 2023 17:30
@erezrokah erezrokah force-pushed the feat/update_bigquery_sdk branch from a7561d6 to de2abb2 Compare February 22, 2023 17:31
@erezrokah erezrokah changed the title feat(deps): Update BigQuery plugin-sdk to v1.38.2 feat(deps): Update BigQuery plugin-sdk to v1.39.0 Feb 22, 2023
@kodiakhq kodiakhq bot merged commit 295bc90 into main Feb 22, 2023
@kodiakhq kodiakhq bot deleted the feat/update_bigquery_sdk branch February 22, 2023 17:34
@cq-bot cq-bot mentioned this pull request Feb 22, 2023
Merged
kodiakhq bot pushed a commit that referenced this pull request Feb 22, 2023
@cq-bot
chore(main): Release plugins-destination-bigquery v2.2.0 (#8345)
8d410a3 
🤖 I have created a release *beep* *boop*
---


## [2.2.0](plugins-destination-bigquery-v2.1.7...plugins-destination-bigquery-v2.2.0) (2023-02-22)


### Features

* **deps:** Update BigQuery plugin-sdk to v1.39.0 ([#8230](#8230)) ([295bc90](295bc90))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@disq disq disq approved these changes

@hermanschaaf hermanschaaf Awaiting requested review from hermanschaaf

+1 more reviewer

@candiduslynx candiduslynx candiduslynx approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@erezrokah erezrokah

Labels

automerge Automatically merge once required checks pass

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@erezrokah @yevgenypats @hermanschaaf @disq @candiduslynx @cq-bot