Skip to content

Conversation

@hermanschaaf
Copy link
Member

Closes #10112

BEGIN_COMMIT_OVERRIDE
feat: Update to use Apache Arrow type system

BREAKING-CHANGE: This release introduces an internal change to our type system to use Apache Arrow. This should not have any visible breaking changes for CSV or JSON output formats, however the Parquet output changes for UUID columns, which now have dashes, and timestamps, which now uses the default Arrow time format (e.g. 2023-01-02 12:23:45). If you encounter an issue during the upgrade, please submit a bug report.

END_COMMIT_OVERRIDE

@cq-bot cq-bot added the s3 label Apr 18, 2023
@hermanschaaf hermanschaaf marked this pull request as ready for review April 20, 2023 08:48
@hermanschaaf hermanschaaf requested review from a team and yevgenypats and removed request for a team April 20, 2023 08:48
tableName := schema.TableName(arrowSchema)
if !c.pluginSpec.NoRotate {
return fmt.Errorf("reading is not supported when no_rotate is false. Table: %q; Source: %q", table.Name, sourceName)
return fmt.Errorf("reading is not supported when no_rotate is false. Table: %q; Source: %q", tableName, sourceName)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More backticks? Although I just noticed these errors are in Read which is only used by the tests

@hermanschaaf hermanschaaf changed the title fix(s3)!: Migrate to plugin-sdk v2 and filetypes v2 fix!: Upgrade S3 plugin-sdk and filetypes to v2 Apr 20, 2023
Copy link
Contributor

@yevgenypats yevgenypats left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. few comments.

Copy link
Contributor

@candiduslynx candiduslynx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a small refactor for loop (nit)

@hermanschaaf hermanschaaf added the automerge Automatically merge once required checks pass label Apr 21, 2023
@kodiakhq kodiakhq bot merged commit 33b5382 into main Apr 21, 2023
@kodiakhq kodiakhq bot deleted the s3-arrow branch April 21, 2023 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge Automatically merge once required checks pass

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Migrate plugin/destination/s3 to github.com/cloudquery/plugin-sdk/v2

6 participants