Skip to content

feat(aws): Add Support for CloudTrail Events With Incremental Table support #8333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
kodiakhq merged 11 commits into from
Feb 22, 2023
Merged

feat(aws): Add Support for CloudTrail Events With Incremental Table support #8333

kodiakhq merged 11 commits into from
Feb 22, 2023

Conversation

@bbernays
Copy link
Collaborator

@bbernays bbernays commented Feb 22, 2023

Summary

@cq-bot cq-bot added the aws label Feb 22, 2023
@cq-bot cq-bot added the website label Feb 22, 2023
@github-actions
Copy link

github-actions bot commented Feb 22, 2023

This PR has the following changes to source plugin(s) tables:

  • Table aws_cloudtrail_events was added

@bbernays bbernays marked this pull request as ready for review February 22, 2023 13:10
@bbernays bbernays linked an issue Feb 22, 2023 that may be closed by this pull request
CloudTrail Events #6784
Closed
@bbernays bbernays changed the title feat(aws): Add Support for CloudWatch Events With Incremental Table support feat(aws): Add Support for CloudTrail Events With Incremental Table support Feb 22, 2023
hermanschaaf
hermanschaaf reviewed Feb 22, 2023
View reviewed changes
hermanschaaf
hermanschaaf requested changes Feb 22, 2023
View reviewed changes
Copy link
Member

@hermanschaaf hermanschaaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, I just think the key name might be incorrect

@bbernays bbernays added the automerge Automatically merge once required checks pass label Feb 22, 2023
@kodiakhq kodiakhq bot merged commit a90b95c into cloudquery:main Feb 22, 2023
@cq-bot cq-bot mentioned this pull request Feb 22, 2023
Merged
kodiakhq bot pushed a commit that referenced this pull request Mar 8, 2023
@cq-bot
chore(main): Release plugins-source-aws v15.3.0 (#8334)
51d8ea8 
🤖 I have created a release *beep* *boop*
---


## [15.3.0](plugins-source-aws-v15.2.0...plugins-source-aws-v15.3.0) (2023-03-08)


### Features

* Add AWS plugin support for EC2 Launch Templates ([#8675](#8675)) ([9ea75a2](9ea75a2))
* Add RDS BackTracks, Events and Option Groups ([#8671](#8671)) ([b8edd0f](b8edd0f))
* **aws-resources:** Add IAM Access Advisor tables: `aws_iam_group_last_accessed_details`, `aws_iam_policy_last_accessed_details`, `aws_iam_role_last_accessed_details` and `aws_iam_user_last_accessed_details`. These might be slow to sync on some accounts. You can skip them if needed via `skip_tables: ["aws_iam_*_last_accessed_details"]` ([bc53529](bc53529))
* **aws:** Add EC2 DHCP Options (aws_ec2_dhcp_options) ([#8678](#8678)) ([ec99007](ec99007))
* **aws:** Add Support for CloudTrail Events With Incremental Table support ([#8333](#8333)) ([a90b95c](a90b95c))
* **aws:** Add Support for S3 Bucket Static Website ([#8497](#8497)) ([255e9b5](255e9b5))
* **aws:** Add Support for syncing Delegated Administrators ([#8342](#8342)) ([7c1dc40](7c1dc40))
* **aws:** EC2 Spot resources ([#8679](#8679)) ([ef464b5](ef464b5))
* **aws:** Support for Org Policies, Organizational Units and Roots ([#8134](#8134)) ([7cf277e](7cf277e))
* **docs:** Render tables as a part of the Website and add a [tables search box](https://www.cloudquery.io/tables). The equivalent of the GitHub README.md file is now under each plugin's docs section, for example https://www.cloudquery.io/docs/plugins/sources/aws/tables. The Website HTML page is built from the GitHub markdown file located under each plugin's path in our Website code, for example https://github.com/cloudquery/cloudquery/blob/main/website/pages/docs/plugins/sources/aws/tables.md. For the list of all plugins table files as they are stored on GitHub see https://github.com/cloudquery/cloudquery/tree/main/website/tables ([342b0c5](342b0c5))


### Bug Fixes

* **aws-resources:** Use QueryUnescape instead of PathUnescape when decoding iam policy version list document ([#8522](#8522)) ([3f28658](3f28658))
* **aws:** EC2 Image Duplicate Records ([#8434](#8434)) ([e4e1599](e4e1599))
* **deps:** Update github.com/gocarina/gocsv digest to 70c27cb ([#8559](#8559)) ([edae209](edae209))
* **deps:** Update golang.org/x/exp digest to c95f2b4 ([#8560](#8560)) ([9c3bd5b](9c3bd5b))
* **deps:** Update module github.com/aws/aws-sdk-go-v2 to v1.17.5 ([#8446](#8446)) ([e86922b](e86922b))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/config to v1.18.15 ([#8447](#8447)) ([98cb352](98cb352))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/feature/ec2/imds to v1.12.23 ([#8449](#8449)) ([c59f43e](c59f43e))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/feature/s3/manager to v1.11.55 ([#8450](#8450)) ([416a435](416a435))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/internal/ini to v1.3.30 ([#8453](#8453)) ([912401b](912401b))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/internal/v4a to v1.0.21 ([#8454](#8454)) ([7820d00](7820d00))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/accessanalyzer to v1.19.5 ([#8455](#8455)) ([4e55283](4e55283))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/acm to v1.17.5 ([#8456](#8456)) ([670bd96](670bd96))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/amp to v1.16.4 ([#8457](#8457)) ([b605e4e](b605e4e))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/amplify to v1.13.4 ([#8458](#8458)) ([b203f30](b203f30))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/apigateway to v1.16.5 ([#8459](#8459)) ([729099c](729099c))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/apigatewayv2 to v1.13.5 ([#8460](#8460)) ([01a9d06](01a9d06))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/applicationautoscaling to v1.17.5 ([#8461](#8461)) ([38603e6](38603e6))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/athena to v1.22.4 ([#8462](#8462)) ([03cd7cc](03cd7cc))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/cloudhsmv2 to v1.14.4 ([#8463](#8463)) ([670b86c](670b86c))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/cloudwatch to v1.25.4 ([#8464](#8464)) ([8e70932](8e70932))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs to v1.20.5 ([#8465](#8465)) ([677d41f](677d41f))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/codebuild to v1.20.5 ([#8466](#8466)) ([4fdfde6](4fdfde6))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/codepipeline to v1.14.4 ([#8467](#8467)) ([564d1cd](564d1cd))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/cognitoidentity to v1.15.4 ([#8468](#8468)) ([c4b0d80](c4b0d80))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/cognitoidentityprovider to v1.22.4 ([#8469](#8469)) ([a3e1580](a3e1580))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/configservice to v1.29.5 ([#8470](#8470)) ([3c479c6](3c479c6))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/databasemigrationservice to v1.23.5 ([#8697](#8697)) ([31e3a71](31e3a71))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/dax to v1.12.4 ([#8698](#8698)) ([aefccae](aefccae))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/directconnect to v1.18.5 ([#8699](#8699)) ([b10bcf3](b10bcf3))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/docdb to v1.20.4 ([#8700](#8700)) ([c2ab59c](c2ab59c))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/dynamodb to v1.18.6 ([#8701](#8701)) ([ad00fa7](ad00fa7))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/ecr to v1.18.5 ([#8702](#8702)) ([a6bbc91](a6bbc91))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/ecrpublic to v1.15.4 ([#8703](#8703)) ([7902150](7902150))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/efs to v1.19.7 ([#8704](#8704)) ([9805d2b](9805d2b))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/eks to v1.27.5 ([#8705](#8705)) ([843d015](843d015))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elasticache to v1.26.4 ([#8706](#8706)) ([09a47d5](09a47d5))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elasticbeanstalk to v1.15.4 ([#8707](#8707)) ([642a303](642a303))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elasticloadbalancing to v1.15.4 ([#8708](#8708)) ([b9628b7](b9628b7))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 to v1.19.5 ([#8709](#8709)) ([c15a071](c15a071))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elasticsearchservice to v1.18.5 ([#8710](#8710)) ([ac19d06](ac19d06))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elastictranscoder to v1.14.4 ([#8711](#8711)) ([7d5b530](7d5b530))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/firehose to v1.16.5 ([#8712](#8712)) ([50eef2e](50eef2e))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/fsx to v1.28.5 ([#8713](#8713)) ([babee7c](babee7c))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/glacier to v1.14.5 ([#8714](#8714)) ([4cefef8](4cefef8))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/guardduty to v1.17.6 ([#8715](#8715)) ([c0858fe](c0858fe))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/iam to v1.19.4 ([#8716](#8716)) ([968a740](968a740))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/identitystore to v1.16.4 ([#8717](#8717)) ([05088bc](05088bc))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/inspector to v1.13.4 ([#8718](#8718)) ([7730a9e](7730a9e))
* **deps:** Update module github.com/aws/aws-sdk-go-v2/service/inspector2 to v1.11.5 ([#8719](#8719)) ([89a11f9](89a11f9))
* **deps:** Update module github.com/cloudquery/plugin-sdk to v1.39.0 ([#8344](#8344)) ([9c57544](9c57544))
* **deps:** Update module github.com/cloudquery/plugin-sdk to v1.39.1 ([#8371](#8371)) ([e3274c1](e3274c1))
* **deps:** Update module github.com/cloudquery/plugin-sdk to v1.40.0 ([#8401](#8401)) ([4cf36d6](4cf36d6))
* **deps:** Update module github.com/cloudquery/plugin-sdk to v1.41.0 ([#8682](#8682)) ([ea9d065](ea9d065))
* **deps:** Update module github.com/cloudquery/plugin-sdk to v1.42.0 ([#8725](#8725)) ([b83b277](b83b277))
* **deps:** Update module github.com/stretchr/testify to v1.8.2 ([#8599](#8599)) ([2ec8086](2ec8086))
* **docs:** Fix documentation link for AWS S3 bucket websites ([#8502](#8502)) ([dcccd40](dcccd40))
* Fix typo in aws_organizations_delegated_administrators table name ([#8477](#8477)) ([bb081c2](bb081c2))
* Update endpoints ([#8499](#8499)) ([22fce4e](22fce4e))
* Update endpoints ([#8737](#8737)) ([3582b8f](3582b8f))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
@kabachook
Copy link

kabachook commented Apr 10, 2023 •
edited
Loading

Sync time goes brrrrrr

@bbernays @hermanschaaf @yevgenypats
Hi, is that supposed to work only in incremental mode?
I run sync every hour, every sync produces about 20k events in trail. Now I upgraded the plugin version, It started syncing the whole trail which has millions of events =)
Maybe there is a setting to manually set the last sync time for table, so it won't fetch historical data?

@hermanschaaf
Copy link
Member

hermanschaaf commented Apr 10, 2023

@kabachook The default behavior is to do a full sync of all the data every time. If you need the data in the aws_cloudtrail_events table but want it to be fetched incrementally, you will need to enable the local state backend as described here: https://www.cloudquery.io/docs/advanced-topics/managing-incremental-tables. Alternatively, if you don't need the data in the table, you can add it to the skip_tables section (or remove it from the tables section) in the source config.

If you have any follow-up questions, an issue would be a more appropriate place to discuss this, would you mind opening a new one or reaching out in Discord?

@bbernays bbernays deleted the cwe-support branch April 10, 2023 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hermanschaaf hermanschaaf hermanschaaf approved these changes

@yevgenypats yevgenypats Awaiting requested review from yevgenypats

Assignees

No one assigned

Labels

automerge Automatically merge once required checks pass

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

CloudTrail Events

4 participants

@bbernays @kabachook @hermanschaaf @cq-bot