feat!: SDK V4 (One plugin and protocol for sources and destinations) #915
Conversation
yevgenypats
changed the title
writers/batch.go
Outdated
| resources = make([]arrow.Record, 0) | ||
| if upsertBatch != r.Upsert { | ||
| w.flush(ctx, tableName, upsertBatch, resources) | ||
| resources = make([]*plugin.MessageInsert, 0) |
There was a problem hiding this comment.
Would consider resources = resources[:0] type syntax in here and below
…ugin-sdk into feat/migrate_to_one_plugin
| return items | ||
| } | ||
|
|
||
| func (messages Messages) InsertMessage() Inserts { |
There was a problem hiding this comment.
The naming isn't very clear here, sounds like you're inserting a message. Call it GetInserts() (or just Inserts()) maybe? (also, unused I think)
| @@ -0,0 +1,199 @@ | |||
| mode: set | |||
There was a problem hiding this comment.
This file should be removed I think
| return s.testListenerConn.Dial() | ||
| } | ||
|
|
||
| func (s *PluginServe) Serve(ctx context.Context) error { |
There was a problem hiding this comment.
The returned error here is not used (or is a hassle to use) from a main.go POV. Maybe make a ServeConsole() helper func that calls Serve with context.Background(), doesn't return error, but prints error to stderr (and exits with proper error code) instead? Could even be a generic helper fn, just needs to conform to Serve's signature of ctx-in-error-out.
| SkipTimes bool // time of day types | ||
| SkipLargeTypes bool // e.g. large binary, large string | ||
| TimePrecision time.Duration | ||
| SkipDecimals bool |
There was a problem hiding this comment.
Doesn't look like this is used anywhere.
Update: it was used but in this PR we don't add the decimal type column anymore
There was a problem hiding this comment.
Also, it's not shown on the diff for some reason (probably since the PR needs a rebase), but SkipMaps is not used either and we don't add the maps at all
…ugin-sdk into feat/migrate_to_one_plugin
|
This branch is also missing a bunch of Scalar values, like |
| MigrateForce: s.migrateMode == plugin.MigrateModeForce, | ||
| }, writeCh) | ||
| }) | ||
| for _, table := range tables { |
There was a problem hiding this comment.
| for _, table := range tables { | |
| for _, table := range tables.FlattenTables() { |
Maybe?
| }, msgs) | ||
| }) | ||
|
|
||
| for _, table := range tables { |
There was a problem hiding this comment.
| for _, table := range tables { | |
| for _, table := range tables.FlattenTables() { |
Not sure if tables are unflattened here (or in Migrate) but we seem to be calling MigrateTable a lot?
Instead of this #915 (fix rebase conflicts) This introduce a single API for both destination and sources. Now plugins should be able to both read and write. This also goes together with a proto upgrade https://github.com/cloudquery/plugin-pb-go/tree/main/pb/plugin/v3 --------- Co-authored-by: Herman Schaaf <[email protected]> Co-authored-by: Kemal Hadimli <[email protected]> Co-authored-by: Erez Rokah <[email protected]>
|
Closing in favour of #984 |
#### Summary Similar to #11696. ~~I had to comment change some of the tests due to cloudquery/plugin-sdk#982 and cloudquery/plugin-sdk#915 (comment), so not ready to merge~~ BEGIN_COMMIT_OVERRIDE feat!: Upgrades the postgresql source plugin to use plugin-sdk v4. This version does not contain any user-facing breaking changes, but because it is now using CloudQuery gRPC protocol v3, it does require use of a destination plugin that also supports protocol v3. All recent destination plugin versions support this. feat!: To enable CDC in this version you'll need to use the `cdc_id` configuration string property, instead of the `cdc` boolean one. Please refer to the [docs](https://www.cloudquery.io/docs/plugins/sources/postgresql/overview) for more information END_COMMIT_OVERRIDE <!--
#### Summary This is based on the MemDB plugin from cloudquery/plugin-sdk#915 Fixes #11755 BEGIN_COMMIT_OVERRIDE feat!: Upgrades the mysql source plugin to use plugin-sdk v4. This version does not contain any user-facing breaking changes, but because it is now using CloudQuery gRPC protocol v3, it does require use of a destination plugin that also supports protocol v3. All recent destination plugin versions support this. END_COMMIT_OVERRIDE <!--
Playing with the idea of having only one plugin and protocol.