Skip to content

Conversation

@yevgenypats
Copy link
Contributor

@yevgenypats yevgenypats commented May 15, 2023

Closes #10733

BEGIN_COMMIT_OVERRIDE
feat: Update to use Apache Arrow type system (#10797)

BREAKING-CHANGE: This release introduces an internal change to our type system to use Apache Arrow. This should not have any visible breaking changes, however due to the size of the change we are introducing it under a major version bump to communicate that it might have some bugs that we weren't able to catch during our internal tests. If you encounter an issue during the upgrade, please submit a bug report. You will also need to update destinations depending on which one you use:

  • Azure Blob Storage >= v3.2.0
  • BigQuery >= v3.0.0
  • ClickHouse >= v3.1.1
  • DuckDB >= v1.1.6
  • Elasticsearch >= v2.0.0
  • File >= v3.2.0
  • Firehose >= v2.0.2
  • GCS >= v3.2.0
  • Gremlin >= v2.1.10
  • Kafka >= v3.0.1
  • Meilisearch >= v2.0.1
  • Microsoft SQL Server >= v4.2.0
  • MongoDB >= v2.0.1
  • MySQL >= v2.0.2
  • Neo4j >= v3.0.0
  • PostgreSQL >= v4.2.0
  • S3 >= v4.4.0
  • Snowflake >= v2.1.1
  • SQLite >= v2.2.0

END_COMMIT_OVERRIDE

@yevgenypats yevgenypats requested a review from bbernays May 15, 2023 16:06
@yevgenypats yevgenypats requested a review from hermanschaaf as a code owner May 15, 2023 16:06
@cq-bot cq-bot added the aws label May 15, 2023
no breaking changes to schema should be visible but marking as
breaking for safety.
@yevgenypats yevgenypats force-pushed the feat/aws_arrow_native branch from 25e8147 to aab4202 Compare May 15, 2023 22:10
@yevgenypats yevgenypats requested a review from erezrokah as a code owner May 25, 2023 11:08
@cq-bot cq-bot added the ci label May 25, 2023
@yevgenypats yevgenypats requested review from bbernays and disq and removed request for erezrokah May 25, 2023 11:09
@cq-bot cq-bot added the website label May 25, 2023
@github-actions
Copy link

This PR has the following changes to source plugin(s) tables:

  • ⚠️ BREAKING CHANGE: Table aws_wafv2_ipsets: column type changed from InetArray to list<item: inet, nullable> for addresses

@yevgenypats yevgenypats requested a review from candiduslynx May 25, 2023 14:09
@yevgenypats yevgenypats added the automerge Automatically merge once required checks pass label May 25, 2023
Copy link
Member

@hermanschaaf hermanschaaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@hermanschaaf hermanschaaf removed the automerge Automatically merge once required checks pass label May 25, 2023
@yevgenypats yevgenypats added the automerge Automatically merge once required checks pass label May 25, 2023
kodiakhq bot pushed a commit that referenced this pull request May 25, 2023
@hermanschaaf hermanschaaf removed the automerge Automatically merge once required checks pass label May 25, 2023
@hermanschaaf hermanschaaf merged commit e355d14 into main May 25, 2023
@hermanschaaf hermanschaaf deleted the feat/aws_arrow_native branch May 25, 2023 14:47
kodiakhq bot pushed a commit that referenced this pull request May 29, 2023
🤖 I have created a release *beep* *boop*
---


## [18.0.0](plugins-source-aws-v17.4.0...plugins-source-aws-v18.0.0) (2023-05-29)


### ⚠ BREAKING CHANGES

* **aws:** Change names of columns which had `_ar_ns` instead of `_arns` ([#10802](#10802))
* This release introduces an internal change to our type system to use [Apache Arrow](https://arrow.apache.org/). This should not have any visible breaking changes, however due to the size of the change we are introducing it under a major version bump to communicate that it might have some bugs that we weren't able to catch during our internal tests. If you encounter an issue during the upgrade, please submit a [bug report](https://github.com/cloudquery/cloudquery/issues/new/choose). You will also need to update destinations depending on which one you use:
    - Azure Blob Storage >= v3.2.0
    - BigQuery >= v3.0.0
    - ClickHouse >= v3.1.1
    - DuckDB >= v1.1.6
    - Elasticsearch >= v2.0.0
    - File >= v3.2.0
    - Firehose >= v2.0.2
    - GCS >= v3.2.0
    - Gremlin >= v2.1.10
    - Kafka >= v3.0.1
    - Meilisearch >= v2.0.1
    - Microsoft SQL Server >= v4.2.0
    - MongoDB >= v2.0.1
    - MySQL >= v2.0.2
    - Neo4j >= v3.0.0
    - PostgreSQL >= v4.2.0
    - S3 >= v4.4.0
    - Snowflake >= v2.1.1
    - SQLite >= v2.2.0

### This Release has the Following Changes to Tables
- Table `aws_apigateway_rest_api_authorizers`: column `provider_ar_ns` removed from table (:warning: breaking)
- Table `aws_apigateway_rest_api_authorizers`: column added with name `provider_arns` and type `list<item: utf8, nullable>`
- Table `aws_autoscaling_groups`: column `target_group_ar_ns` removed from table (:warning: breaking)
- Table `aws_autoscaling_groups`: column added with name `target_group_arns` and type `list<item: utf8, nullable>`
- Table `aws_cloudformation_stacks`: column `notification_ar_ns` removed from table (:warning: breaking)
- Table `aws_cloudformation_stacks`: column added with name `notification_arns` and type `list<item: utf8, nullable>`
- Table `aws_cognito_identity_pools`: column `open_id_connect_provider_ar_ns` removed from table (:warning: breaking)
- Table `aws_cognito_identity_pools`: column `saml_provider_ar_ns` removed from table (:warning: breaking)
- Table `aws_cognito_identity_pools`: column added with name `open_id_connect_provider_arns` and type `list<item: utf8, nullable>`
- Table `aws_cognito_identity_pools`: column added with name `saml_provider_arns` and type `list<item: utf8, nullable>`
- Table `aws_ssoadmin_permission_sets`: column added with name `request_account_id` and type `utf8`
- Table `aws_ssoadmin_permission_sets`: column added with name `request_region` and type `utf8`

### Features

* Update to use [Apache Arrow](https://arrow.apache.org/) type system ([#10797](#10797)) ([e355d14](e355d14))


### Bug Fixes

* **aws:** Change names of columns which had `_ar_ns` instead of `_arns` ([#10802](#10802)) ([e00ac44](e00ac44))
* **aws:** Remove Hardcoded fix for AWS issue ([#10972](#10972)) ([ede53a7](ede53a7))
* **aws:** Support syncing AWS SSO Account Assignments for non management accounts ([#10881](#10881)) ([a715e4f](a715e4f))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs to v1.20.11 ([#11066](#11066)) ([be8e23b](be8e23b))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/codebuild to v1.20.13 ([#11067](#11067)) ([c3c831a](c3c831a))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/cognitoidentity to v1.15.11 ([#11068](#11068)) ([6708fec](6708fec))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/cognitoidentityprovider to v1.22.10 ([#11069](#11069)) ([28698dc](28698dc))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/costexplorer to v1.25.10 ([#11070](#11070)) ([8da3107](8da3107))
* **deps:** Update module github.com/cloudquery/plugin-sdk to v1.45.0 ([#11041](#11041)) ([035e461](035e461))
* **deps:** Update module github.com/cloudquery/plugin-sdk/v3 to v3.6.7 ([#11043](#11043)) ([3c6d885](3c6d885))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Migrate plugin/source/aws to github.com/cloudquery/plugin-sdk/v3

5 participants