Skip to content

feat!: Upgrade Hacker News plugin to SDK v4 #11855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
kodiakhq merged 16 commits into from
Jul 3, 2023
Merged

feat!: Upgrade Hacker News plugin to SDK v4 #11855

kodiakhq merged 16 commits into from
Jul 3, 2023

Conversation

@hermanschaaf
Copy link
Member

@hermanschaaf hermanschaaf commented Jun 30, 2023 •
edited
Loading

Upgrades Hacker News plugin to SDK v4, which allows any destination plugin to be used as a backend.

This will only work with recent CLI versions and destinations that are also upgraded to V4.

The backend should now be configured like this:

kind: source
spec:
  name: "hackernews"
  version: "vX.Y.Z"
  path: "cloudquery/hackernews"
  tables: 
    ["*"]
  destinations: 
    - "postgresql"
  backend_options:
    table_name: "cq_state_hackernews"
    connection: "@@plugins.postgresql.connection"
  spec:
    item_concurrency: 100
    start_time: "2023-07-01T00:00:00Z"
---
kind: destination
spec:
  name: "postgresql"
  version: "vX.Y.Z"
  registry: "github"
  path: "cloudquery/postgresql"
  write_mode: "overwrite-delete-stale"
  spec:
    connection_string: "postgresql://postgres:pass@localhost:5432/cloudquery?sslmode=disable"

yevgenypats
yevgenypats reviewed Jul 1, 2023
View reviewed changes
@yevgenypats yevgenypats mentioned this pull request Jul 1, 2023
Merged
yevgenypats added a commit to cloudquery/plugin-sdk that referenced this pull request Jul 1, 2023
@yevgenypats
fix: State add flush and fix migration bug (#1039)
8c10291 
This fixes a bug in state backend where table weren't created.  

I also tested it with
cloudquery/cloudquery#11855 which seems to work
with minor modifications (will open a PR shortly to the CLI PR)
@cq-bot cq-bot removed the cli label Jul 3, 2023
@hermanschaaf hermanschaaf changed the title feat: Update CLI v4 for state management, migrate Hacker News plugin as POC feat: Upgrade Hacker News plugin to SDK v4 Jul 3, 2023
@hermanschaaf hermanschaaf changed the title feat: Upgrade Hacker News plugin to SDK v4 feat!: Upgrade Hacker News plugin to SDK v4 Jul 3, 2023
@cq-bot cq-bot added the website label Jul 3, 2023
@cloudquery cloudquery deleted a comment from github-actions bot Jul 3, 2023
@hermanschaaf hermanschaaf marked this pull request as ready for review July 3, 2023 16:04
@hermanschaaf hermanschaaf requested review from a team and removed request for a team July 3, 2023 16:04
@hermanschaaf hermanschaaf requested a review from candiduslynx July 3, 2023 16:04
@hermanschaaf
Copy link
Member Author

hermanschaaf commented Jul 3, 2023

This is now ready and working, however we need to release all V4-compatible destinations first, before releasing this one.

@yevgenypats yevgenypats added automerge Automatically merge once required checks pass and removed no automerge labels Jul 3, 2023
@yevgenypats
Copy link
Contributor

yevgenypats commented Jul 3, 2023

Looks great! Merging (will hold-off the release)

@kodiakhq kodiakhq bot merged commit 3c686be into main Jul 3, 2023
@kodiakhq kodiakhq bot deleted the v4-hackernews-source branch July 3, 2023 17:23
@cq-bot cq-bot mentioned this pull request Jul 3, 2023
Merged
@candiduslynx candiduslynx linked an issue Jul 4, 2023 that may be closed by this pull request
Migrate plugins/source/hackernews to github.com/cloudquery/plugin-sdk/v4 #11751
Closed
kodiakhq bot pushed a commit that referenced this pull request Jul 7, 2023
@cq-bot
chore(main): Release plugins-source-hackernews v3.0.0 (#11586)
44c7334 
🤖 I have created a release *beep* *boop*
---


## [3.0.0](plugins-source-hackernews-v2.0.2...plugins-source-hackernews-v3.0.0) (2023-07-07)


### ⚠ BREAKING CHANGES

* Upgrade Hacker News plugin to SDK v4 ([#11855](#11855))

### Features

* Upgrade Hacker News plugin to SDK v4 ([#11855](#11855)) ([3c686be](3c686be))


### Bug Fixes

* **deps:** Update github.com/apache/arrow/go/v13 digest to 5a06b2e ([#11857](#11857)) ([43c2f5f](43c2f5f))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to 0656028 ([#11739](#11739)) ([7a6ad49](7a6ad49))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to 1e68c51 ([#11637](#11637)) ([46043bc](46043bc))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to 43638cb ([#11672](#11672)) ([3c60bbb](3c60bbb))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to 4d76231 ([#11532](#11532)) ([6f04233](6f04233))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to 8366a22 ([#11717](#11717)) ([8eeff5b](8eeff5b))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to 95d3199 ([#11708](#11708)) ([03f214f](03f214f))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to b0832be ([#11651](#11651)) ([71e8c29](71e8c29))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to d864719 ([#11611](#11611)) ([557a290](557a290))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to df3b664 ([#11882](#11882)) ([9635b22](9635b22))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to f060192 ([#11730](#11730)) ([c7019c2](c7019c2))
* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to f0dffc6 ([#11689](#11689)) ([18ac0e9](18ac0e9))
* **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.1.0 ([#11665](#11665)) ([d8947c9](d8947c9))
* **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.2.0 ([#11720](#11720)) ([7ef521d](7ef521d))
* **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.2.1 ([#11722](#11722)) ([309be72](309be72))
* **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.3.3 ([#11726](#11726)) ([f0ca611](f0ca611))
* **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.3.4 ([#11753](#11753)) ([cd4fe1c](cd4fe1c))
* **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.5.0 ([#11850](#11850)) ([3255857](3255857))
* **deps:** Update module github.com/cloudquery/plugin-sdk/v3 to v3.10.6 ([#11473](#11473)) ([7272133](7272133))
* **postgresql:** Rerun release please ([#12002](#12002)) ([9d12843](9d12843))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@candiduslynx candiduslynx Awaiting requested review from candiduslynx

1 more reviewer

@yevgenypats yevgenypats yevgenypats approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

automerge Automatically merge once required checks pass

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Migrate plugins/source/hackernews to github.com/cloudquery/plugin-sdk/v4

4 participants

@hermanschaaf @yevgenypats @cq-bot