Skip to content

Conversation

@erezrokah
Copy link
Member

@erezrokah erezrokah commented Jul 20, 2023

Summary

Fixes #12423. Not marking as breaking as it never worked

I remember we had something in the SDK to check for duplicate tables. Maybe it got lost in the v4 migration

@github-actions
Copy link

This PR has the following changes to source plugin(s) tables:

  • ⚠️ BREAKING CHANGE: Table aws_appconfig_applications: column deployment_duration_in_minutes removed from table
  • ⚠️ BREAKING CHANGE: Table aws_appconfig_applications: column final_bake_time_in_minutes removed from table
  • ⚠️ BREAKING CHANGE: Table aws_appconfig_applications: column growth_factor removed from table
  • ⚠️ BREAKING CHANGE: Table aws_appconfig_applications: column growth_type removed from table
  • ⚠️ BREAKING CHANGE: Table aws_appconfig_applications: column replicate_to removed from table
  • Table aws_appconfig_deployment_strategies was added

@kodiakhq kodiakhq bot merged commit dda4205 into cloudquery:main Jul 20, 2023
@erezrokah erezrokah deleted the fix/aws_appconfig_deployment_strategies branch July 20, 2023 12:25
kodiakhq bot pushed a commit that referenced this pull request Jul 20, 2023
🤖 I have created a release *beep* *boop*
---


## [22.0.0](plugins-source-aws-v21.1.0...plugins-source-aws-v22.0.0) (2023-07-20)


### ⚠ BREAKING CHANGES

* **aws:** Define composite primary key for regions ([#12415](#12415))
* **aws:** Define primary key for s3_bucket_encryption_rules ([#12408](#12408))
* **aws:** Define primary key for dynamodb_table_continuous_backups ([#12409](#12409))
* **aws:** Define composite primary key for lambda_function_versions ([#12402](#12402))

### This Release has the Following Changes to Tables
- Table `aws_appconfig_deployment_strategies` was added
- Table `aws_dynamodb_table_continuous_backups`: primary key constraint added to column `table_arn` (:warning: breaking)
- Table `aws_dynamodb_table_continuous_backups`: primary key constraint removed from column `_cq_id` (:warning: breaking)
- Table `aws_lambda_function_versions`: primary key constraint added to column `function_arn` (:warning: breaking)
- Table `aws_lambda_function_versions`: primary key constraint added to column `version` (:warning: breaking)
- Table `aws_lambda_function_versions`: primary key constraint removed from column `_cq_id` (:warning: breaking)
- Table `aws_regions`: primary key constraint added to column `account_id` (:warning: breaking)
- Table `aws_regions`: primary key constraint added to column `region` (:warning: breaking)
- Table `aws_regions`: primary key constraint removed from column `_cq_id` (:warning: breaking)
- Table `aws_s3_bucket_encryption_rules`: primary key constraint added to column `bucket_arn` (:warning: breaking)
- Table `aws_s3_bucket_encryption_rules`: primary key constraint removed from column `_cq_id` (:warning: breaking)

### Features

* **aws:** Define composite primary key for lambda_function_versions ([#12402](#12402)) ([d1add18](d1add18))
* **aws:** Define composite primary key for regions ([#12415](#12415)) ([681ea97](681ea97))
* **aws:** Define primary key for dynamodb_table_continuous_backups ([#12409](#12409)) ([7d4a657](7d4a657))
* **aws:** Define primary key for s3_bucket_encryption_rules ([#12408](#12408)) ([a026b98](a026b98))


### Bug Fixes

* **sync:** Pass `DeterministicCQID` option to scheduler ([#12424](#12424)) ([eaac2e6](eaac2e6))
* Use correct name for `DeploymentStrategies` table ([#12428](#12428)) ([dda4205](dda4205))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
kodiakhq bot pushed a commit to cloudquery/plugin-sdk that referenced this pull request Aug 18, 2023

Fixes #1106

Plugin validation was removed in https://github.com/cloudquery/plugin-sdk/pull/984/files#diff-95f04007d983a3bfdb080b338b9de5a8bbd73a3d65910fbe664f97788a9021b3.

As a result we don't validate tables and columns anymore, see resulted bug in cloudquery/cloudquery#12428.

This PR brings it back and per a better option puts the validation in the `Init` method. Other options:
1. Have each plugin call `.validate` on the plugin side, probably after transformation [here](https://github.com/cloudquery/cloudquery/blob/03fae6d803a4e2ac00ff83d9e90a4169b247c346/plugins/source/aws/resources/plugin/tables.go#L565). Downside that each plugin needs to implement the call
2. Have the validation done as a part of `schema.TransformTables` [here](https://github.com/cloudquery/plugin-sdk/blob/8a6a2cc87e303d66c810e629a497f9acd3c4982c/transformers/tables.go#L30). Downside it will happen before any custom transformation that are executed on tables like [`titleTransformer`](https://github.com/cloudquery/cloudquery/blob/03fae6d803a4e2ac00ff83d9e90a4169b247c346/plugins/source/aws/resources/plugin/tables.go#L567)

> Please note that before it was removed `validate` was called during `NewPlugin` which means it errored right after calling the `serve` command, meaning the plugin would not start. With this fix the plugin will error out during `Init`, which means you need to run `sync/migrate` to see the error

---
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge Automatically merge once required checks pass

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: table aws_appconfig_applications - column "deployment_duration_in_minutes" of relation "aws_appconfig_applications" does not exist

4 participants