feat(aws): Step Function Resources #7911

bbernays · 2023-02-10T18:37:06Z

Summary

github-actions · 2023-02-10T18:37:24Z

This PR has the following changes to source plugin(s) tables:

Table aws_stepfunctions_activities was added
Table aws_stepfunctions_executions was added
Table aws_stepfunctions_executions_map_runs was added

🤖 I have created a release *beep* *boop* --- ## [14.0.0](plugins-source-aws-v13.3.0...plugins-source-aws-v14.0.0) (2023-02-14) ### ⚠ BREAKING CHANGES * **aws:** Use `log_group_arn` + `filter_name` for `aws_cloudwatchlogs_metric_filters` PK ([#8061](#8061)) * **aws:** Add `account_id` and `region` to `aws_ec2_transit_gateways` PK ([#8060](#8060)) * **aws:** Rename `user_arn` field to `arn` in `aws_quicksight_group_members` * **aws:** Resource Access Manager (RAM) PK issues ([#8010](#8010)) * **aws:** Add `resource_share_arn` and `region` to `aws_ram_principals` PK ([#7985](#7985)) * **aws:** Add `account_id` and `region` to PK in `aws_directconnect_gateways` ([#7984](#7984)) * **aws:** Use `physical_resource_identifier` instead of `physical_resource_id` in `aws_resiliencehub_app_version_resources` and `aws_resiliencehub_app_version_resource_mappings` PKs * **aws:** PK for `aws_s3_bucket_grants` ([#7822](#7822)) * **aws:** PK for `aws_docdb_certificates` ([#7820](#7820)) * **aws:** PK for `aws_ec2_images` ([#7821](#7821)) * **aws:** Fix PK for `aws_rds_certificates` ([#7817](#7817)) ### Features * Athena resource view creation ([#7908](#7908)) ([a8769ea](a8769ea)) * **aws-resources:** Add EMR Cluster Instances ([#7814](#7814)) ([f2883d0](f2883d0)) * **aws-resources:** Add EMR Instance Fleets and Groups ([#7818](#7818)) ([df8cfea](df8cfea)) * **aws-resources:** Add EMR Security Configuration ([#7812](#7812)) ([3d70111](3d70111)) * **aws:** Add Method and Integration Resources for API Gateway ([#7923](#7923)) ([f3d669a](f3d669a)) * **aws:** Add New Elasticache Resources and Attributes ([#7925](#7925)) ([e18ca20](e18ca20)) * **aws:** Add New Validation for AWS Tags ([#7651](#7651)) ([2440e89](2440e89)) * **aws:** Add Support for CloudWatch Filter Subscriptions ([#8073](#8073)) ([cbc2fb1](cbc2fb1)) * **aws:** Add Support for Directconnect Locations ([#7906](#7906)) ([0187098](0187098)) * **aws:** Add Support for EC2 Managed Prefix List ([#7942](#7942)) ([743fdd2](743fdd2)) * **aws:** Add Support for Route53 Operations ([#7944](#7944)) ([306f0bd](306f0bd)) * **aws:** Step Function Resources ([#7911](#7911)) ([ffb3275](ffb3275)) * **aws:** Support Availability Zones ([#7914](#7914)) ([f082057](f082057)) ### Bug Fixes * **aws:** Add `account_id` and `region` to `aws_appstream_images` PK ([#7972](#7972)) ([a7cfd33](a7cfd33)) * **aws:** Add `account_id` and `region` to `aws_ec2_transit_gateways` PK ([#8060](#8060)) ([37fbf63](37fbf63)) * **aws:** Add `account_id` and `region` to PK in `aws_directconnect_gateways` ([#7984](#7984)) ([588bc88](588bc88)) * **aws:** Add `account_id` and region to `aws_quicksight_*` resources ([4d0fdb4](4d0fdb4)) * **aws:** Add `request_account_id` & `request_region` to `aws_securityhub_findings` PK ([#7971](#7971)) ([aa74371](aa74371)) * **aws:** Add `resource_share_arn` and `region` to `aws_ram_principals` PK ([#7985](#7985)) ([fd24a5b](fd24a5b)) * **aws:** Add parent table ARNs as PKs to child tables for `aws_resiliencehub_*` resources ([2f70712](2f70712)) * **aws:** Fix `aws_iam_accounts` unmarshaling ([#7899](#7899)) ([abf28ed](abf28ed)) * **aws:** Fix PK for `aws_rds_certificates` ([#7817](#7817)) ([8082b2a](8082b2a)) * **aws:** PK for `aws_docdb_certificates` ([#7820](#7820)) ([b2b06ea](b2b06ea)) * **aws:** PK for `aws_ec2_images` ([#7821](#7821)) ([d568cde](d568cde)) * **aws:** PK for `aws_s3_bucket_grants` ([#7822](#7822)) ([4efbf32](4efbf32)) * **aws:** Remove `account_id` and `region` columns from `aws_resiliencehub_*` PKs ([2f70712](2f70712)) * **aws:** Rename `user_arn` field to `arn` in `aws_quicksight_group_members` ([4d0fdb4](4d0fdb4)) * **aws:** Resource Access Manager (RAM) PK issues ([#8010](#8010)) ([901dcf4](901dcf4)) * **aws:** Use `log_group_arn` + `filter_name` for `aws_cloudwatchlogs_metric_filters` PK ([#8061](#8061)) ([d29f19a](d29f19a)) * **aws:** Use `physical_resource_identifier` instead of `physical_resource_id` in `aws_resiliencehub_app_version_resources` and `aws_resiliencehub_app_version_resource_mappings` PKs ([2f70712](2f70712)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/codepipeline to v1.14.1 ([#7945](#7945)) ([bd1eee8](bd1eee8)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/cognitoidentity to v1.15.1 ([#7946](#7946)) ([7747eab](7747eab)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/cognitoidentityprovider to v1.22.1 ([#7947](#7947)) ([f3307f2](f3307f2)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/configservice to v1.29.2 ([#7948](#7948)) ([db7b7af](db7b7af)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/databasemigrationservice to v1.23.2 ([#7949](#7949)) ([ff3e2d1](ff3e2d1)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/dax to v1.12.1 ([#7950](#7950)) ([3127200](3127200)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/directconnect to v1.18.2 ([#7951](#7951)) ([c220d49](c220d49)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/docdb to v1.20.2 ([#7952](#7952)) ([f021f80](f021f80)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/dynamodb to v1.18.2 ([#7953](#7953)) ([b50cb36](b50cb36)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/ecr to v1.18.2 ([#7954](#7954)) ([610165a](610165a)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/ecrpublic to v1.15.1 ([#7955](#7955)) ([70a2b8b](70a2b8b)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/ecs to v1.23.2 ([#7956](#7956)) ([66b4cee](66b4cee)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/efs to v1.19.3 ([#7957](#7957)) ([202d467](202d467)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/eks to v1.27.2 ([#7958](#7958)) ([8f08e65](8f08e65)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elasticache to v1.26.2 ([#7959](#7959)) ([4ca09b8](4ca09b8)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elasticbeanstalk to v1.15.1 ([#7960](#7960)) ([e859fd0](e859fd0)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elasticloadbalancing to v1.15.2 ([#7961](#7961)) ([d7b1163](d7b1163)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 to v1.19.3 ([#7962](#7962)) ([4afa209](4afa209)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elasticsearchservice to v1.18.2 ([#7963](#7963)) ([69752f3](69752f3)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elastictranscoder to v1.14.1 ([#7964](#7964)) ([d8ca305](d8ca305)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/emr to v1.22.2 ([#7965](#7965)) ([1ee800c](1ee800c)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/firehose to v1.16.2 ([#7966](#7966)) ([48a4faf](48a4faf)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/fsx to v1.28.2 ([#7967](#7967)) ([2ff2fce](2ff2fce)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/glacier to v1.14.2 ([#7968](#7968)) ([0a5fe22](0a5fe22)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/guardduty to v1.17.2 ([#7969](#7969)) ([7fcb80a](7fcb80a)) * **deps:** Update module github.com/cloudquery/codegen to v0.2.1 ([#7875](#7875)) ([9d10ac1](9d10ac1)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.36.0 ([#7809](#7809)) ([c85a9cb](c85a9cb)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.36.1 ([#7930](#7930)) ([39dccc1](39dccc1)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.37.0 ([#7933](#7933)) ([dc9cffb](dc9cffb)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.37.1 ([#8008](#8008)) ([c47aac0](c47aac0)) * Fix ECR image ARN to use repository name ([#7839](#7839)) ([e585d61](e585d61)) * Update endpoints ([#7795](#7795)) ([7f5260c](7f5260c)) * Update endpoints ([#7797](#7797)) ([3e72d3e](3e72d3e)) * Update endpoints ([#7798](#7798)) ([0a21b16](0a21b16)) * Update endpoints ([#7799](#7799)) ([ea62bbc](ea62bbc)) * Update endpoints ([#7810](#7810)) ([c797331](c797331)) * Update endpoints ([#7811](#7811)) ([5b1566c](5b1566c)) * Update endpoints ([#7816](#7816)) ([e1cca0c](e1cca0c)) * Update endpoints ([#7991](#7991)) ([3e81234](3e81234)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

koltyakov · 2023-02-14T22:35:37Z

@bbernays I'm trying to get step functions related data and experiencing an error:

2023-02-14T22:27:15Z ERR table resolver finished with error error="operation error SFN: ListExecutions, https response error StatusCode: 400, RequestID: 41e6678b-10d9-4924-8a4b-927966b86d51, ValidationException: Must provide a StateMachine ARN or MapRun ARN" client=***:us-east-1 module=aws-src table=aws_stepfunctions_executions

The config is:

kind: source
spec:
  name: "aws"
  path: "cloudquery/aws"
  version: "v14.0.0"
  tables: ["aws_stepfunctions_state_machines", "aws_stepfunctions_activities", "aws_stepfunctions_executions", "aws_stepfunctions_executions_map_runs"]
  destinations: ["postgresql"]
  spec:
    regions: 
      - us-east-1

Sync completed successfully. Resources: 3, Errors: 1, Panics: 0, Time: 3s

Could you please suggest am I missing something?

bbernays · 2023-02-14T22:37:27Z

@koltyakov - Looking into this now

bbernays · 2023-02-14T22:49:24Z

@koltyakov - Can confirm there is an issue with the code. Looking into this some more. Will ping you when we release a new (fixed) version

koltyakov · 2023-02-14T22:49:59Z

Thank you @bbernays!

bbernays · 2023-02-15T19:13:32Z

@koltyakov - We have released a new version of the AWS plugin (v15.0.0) that contains the appropriate fixes for the step_function issues you were facing yesterday. If you have any issues with it please open a new issue and we would be happy to look into it!

koltyakov · 2023-02-15T20:04:12Z

You are fantastic! Thank you! Confirming it works (happy path) now with v15.0.0 and amended tables list.

P.S. Sorry it's taking time to check it's actually complete with all step functions execution I have Syncing resources... (26529/-, 0 resources/hr) [42m48s]. Will UPD here when it's completed to make sure no errors somewhere in an entity which is in a pipeline.

UPD:

Sync completed successfully. Resources: 40657, Errors: 21343, Panics: 0, Time: 1h12m32s.

Yet, the secondary run:

- Syncing resources... (1205/-, 202 resources/s) [12s] runtime: failed to create new OS thread
failed to terminate source client:  destination plugin process failed with exit status 1
Error: failed to sync v1 source aws: failed to sync source aws (v15.0.0): failed to fetch resources from stream: rpc error: code = Unavailable desc = error reading from server: unexpected EOF

Yet another run is in progress so maybe something intermittent or on my end.

bbernays added 2 commits February 10, 2023 12:27

initial

95926cc

docs

3f5aaf9

bbernays requested review from hermanschaaf and yevgenypats as code owners February 10, 2023 18:37

bbernays linked an issue Feb 10, 2023 that may be closed by this pull request

AWS Step Functions: Activities, MapRuns, Executions #7703

Closed

cq-bot added aws labels Feb 10, 2023

vercel bot deployed to Preview February 10, 2023 18:39 View deployment

update

5f657a3

bbernays marked this pull request as draft February 10, 2023 23:56

bbernays marked this pull request as ready for review February 11, 2023 19:54

erezrokah approved these changes Feb 12, 2023

View reviewed changes

erezrokah added the automerge Automatically merge once required checks pass label Feb 12, 2023

Merge branch 'main' into step-fn-resources

2988b0f

vercel bot deployed to Preview February 12, 2023 11:19 View deployment

kodiakhq bot merged commit ffb3275 into cloudquery:main Feb 12, 2023

cq-bot mentioned this pull request Feb 12, 2023

chore(main): Release plugins-source-aws v14.0.0 #7796

Merged

bbernays deleted the step-fn-resources branch February 14, 2023 22:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(aws): Step Function Resources #7911

feat(aws): Step Function Resources #7911

Uh oh!

bbernays commented Feb 10, 2023 •

edited

Loading

Uh oh!

github-actions bot commented Feb 10, 2023

Uh oh!

koltyakov commented Feb 14, 2023

Uh oh!

bbernays commented Feb 14, 2023

Uh oh!

bbernays commented Feb 14, 2023

Uh oh!

koltyakov commented Feb 14, 2023

Uh oh!

bbernays commented Feb 15, 2023

Uh oh!

koltyakov commented Feb 15, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat(aws): Step Function Resources #7911

feat(aws): Step Function Resources #7911

Uh oh!

Conversation

bbernays commented Feb 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

github-actions bot commented Feb 10, 2023

This PR has the following changes to source plugin(s) tables:

Uh oh!

koltyakov commented Feb 14, 2023

Uh oh!

bbernays commented Feb 14, 2023

Uh oh!

bbernays commented Feb 14, 2023

Uh oh!

koltyakov commented Feb 14, 2023

Uh oh!

bbernays commented Feb 15, 2023

Uh oh!

koltyakov commented Feb 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bbernays commented Feb 10, 2023 •

edited

Loading

koltyakov commented Feb 15, 2023 •

edited

Loading