feat(aws): Parallelize Initialization of Accounts after discovery #10177

bbernays · 2023-04-18T16:07:07Z

Summary

closes #10151

In our small org this change brought the initialization time down from ~5 seconds to ~1 second

#### Summary This is a preliminary PR to make it easier to follow and understand the changes coming for #10177

#### Summary More naming refactor to reduce complexity in #10177

plugins/source/aws/client/client.go

plugins/source/aws/client/spec.go

#### Summary refactor to make #10177 easier to read

Co-authored-by: Kemal <[email protected]>

into parallelization

yevgenypats · 2023-04-18T20:22:43Z

website/pages/docs/plugins/sources/aws/configuration.md


  In AWS organization mode, CloudQuery will source all accounts underneath automatically

+- `initialization_concurrency` (int) (default: 4)


why such a low default if we have 50000 as the default for the syncing goroutines?

50,000 is a number that is designed to be spread out over every single table being resolved... This is only for setting up the SDK (assuming a role and getting the enabled/disabled regions)... If a user needs a higher value they can always increase it

I don't think we get anything by setting this to a super high number that we cannot validate as being appropriate. If we get feedback from users that using it at much high values increases the performance without any errors I am fully onboard raising this value

Non blocking for this PR, but we could test by modifying the code to run on more accounts than we have in our playground.
In Azure I tested by creating 1800 duplicated subscriptions, see https://github.com/cloudquery/cloudquery/compare/main...erezrokah:test/azure_memory?expand=1#diff-c5caf1d71566c18f3cf9dfd107693cc1fca4ea980176d33db01d94b013685afeR226

The only catch is the need to keep client IDs unique

So tested it with 7000 faked accounts... Only ran into rate limit issues when an account was duplicated... which should ever happen

Nice! Does this mean we can use a higher default concurrency? If we're not sure we can keep it lower and. change the default later based on users reports

I would rather keep it low until we hear from users

plugins/source/aws/client/client.go

hermanschaaf

Nice!

Co-authored-by: Herman Schaaf <[email protected]>

🤖 I have created a release *beep* *boop* --- ## [16.3.0](plugins-source-aws-v16.2.0...plugins-source-aws-v16.3.0) (2023-04-20) ### Features * **aws-services:** Support newly added regions ([#10152](#10152)) ([d395a94](d395a94)) * **aws:** Add Support for Config Delivery Channels ([#10150](#10150)) ([361df3f](361df3f)) * **aws:** Parallelize Initialization of Accounts after discovery ([#10177](#10177)) ([3838e80](3838e80)) * **aws:** Upgrade to `github.com/cloudquery/plugin-sdk/v2` ([#9938](#9938)) ([a3fb436](a3fb436)), closes [#9937](#9937) ### Bug Fixes * **aws:** Update EBS Snapshot Permissions Check Query ([#10149](#10149)) ([f65d9da](f65d9da)), closes [#10140](#10140) * **deps:** Update module github.com/aws/aws-sdk-go-v2/config to v1.18.21 ([#10127](#10127)) ([3bcde69](3bcde69)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/feature/s3/manager to v1.11.62 ([#10129](#10129)) ([13f8670](13f8670)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/accessanalyzer to v1.19.10 ([#10131](#10131)) ([eefbad5](eefbad5)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/acm to v1.17.9 ([#10132](#10132)) ([7f6d235](7f6d235)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v2 to v2.2.0 ([#10135](#10135)) ([cf33b89](cf33b89)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v2 to v2.2.2 ([#10143](#10143)) ([8f887e0](8f887e0)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v2 to v2.3.0 ([#10163](#10163)) ([9a7f214](9a7f214)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v2 to v2.3.1 ([#10175](#10175)) ([5b53423](5b53423)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v2 to v2.3.3 ([#10187](#10187)) ([b185248](b185248)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v2 to v2.3.4 ([#10196](#10196)) ([c6d2f59](c6d2f59)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v2 to v2.3.5 ([#10200](#10200)) ([5a33693](5a33693)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v2 to v2.3.6 ([#10208](#10208)) ([91c80a7](91c80a7)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

Parallelize Initialization of accounts after discovery

bbce26f

bbernays requested review from hermanschaaf and yevgenypats as code owners April 18, 2023 16:07

cq-bot added the aws label Apr 18, 2023

bbernays mentioned this pull request Apr 18, 2023

refactor(aws): Renamed Variable In Client #10178

Merged

kodiakhq bot pushed a commit that referenced this pull request Apr 18, 2023

refactor(aws): Renamed Variable In Client (#10178)

b5699be

#### Summary This is a preliminary PR to make it easier to follow and understand the changes coming for #10177

bbernays added 4 commits April 18, 2023 11:20

Merge remote-tracking branch 'upstream/main' into parallelization

2f6d6bf

Merge remote-tracking branch 'upstream/main' into parallelization

c4f076b

Update organizations.go

4bad13d

Update organizations_test.go

dc54783

bbernays mentioned this pull request Apr 18, 2023

refactor(aws): Naming Refactor #10180

Merged

remove

c1d122b

kodiakhq bot pushed a commit that referenced this pull request Apr 18, 2023

refactor(aws): Naming Refactor (#10180)

a8989ab

#### Summary More naming refactor to reduce complexity in #10177

bbernays added 8 commits April 18, 2023 12:12

Update client.go

e9240f6

Merge remote-tracking branch 'upstream/main' into parallelization

264fef3

Merge remote-tracking branch 'upstream/main' into parallelization

ffc0e96

Update client.go

8923154

Update organizations.go

5634a7b

remove changes

f90ea83

Update spec.go

4686329

reorg

dc80079

bbernays commented Apr 18, 2023

View reviewed changes

plugins/source/aws/client/client.go Outdated Show resolved Hide resolved

bbernays commented Apr 18, 2023

View reviewed changes

plugins/source/aws/client/client.go Outdated Show resolved Hide resolved

bbernays commented Apr 18, 2023

View reviewed changes

plugins/source/aws/client/client.go Outdated Show resolved Hide resolved

bbernays commented Apr 18, 2023

View reviewed changes

plugins/source/aws/client/client.go Outdated Show resolved Hide resolved

bbernays commented Apr 18, 2023

View reviewed changes

plugins/source/aws/client/client.go Outdated Show resolved Hide resolved

disq reviewed Apr 18, 2023

View reviewed changes

plugins/source/aws/client/client.go Outdated Show resolved Hide resolved

disq reviewed Apr 18, 2023

View reviewed changes

plugins/source/aws/client/spec.go Outdated Show resolved Hide resolved

Update client.go

5a05b63

disq approved these changes Apr 18, 2023

View reviewed changes

bbernays mentioned this pull request Apr 18, 2023

refactor(aws): Move To Separate files #10181

Merged

kodiakhq bot pushed a commit that referenced this pull request Apr 18, 2023

refactor(aws): Move To Separate files (#10181)

0334d8a

#### Summary refactor to make #10177 easier to read

bbernays and others added 4 commits April 18, 2023 13:04

Merge remote-tracking branch 'upstream/main' into parallelization

9d6da63

Update plugins/source/aws/client/spec.go

f3badc2

Co-authored-by: Kemal <[email protected]>

Update configuration.md

98c4b58

Merge branch 'parallelization' of https://github.com/bbernays/cloudquery

b3911d5

into parallelization

cq-bot added the website label Apr 18, 2023

yevgenypats reviewed Apr 18, 2023

View reviewed changes

hermanschaaf reviewed Apr 19, 2023

View reviewed changes

plugins/source/aws/client/client.go Outdated Show resolved Hide resolved

hermanschaaf approved these changes Apr 19, 2023

View reviewed changes

Update plugins/source/aws/client/client.go

eda8a96

Co-authored-by: Herman Schaaf <[email protected]>

bbernays added the automerge Automatically merge once required checks pass label Apr 19, 2023

Merge branch 'main' into parallelization

82585cf

vercel bot deployed to Preview April 19, 2023 19:56 View deployment

kodiakhq bot merged commit 3838e80 into cloudquery:main Apr 19, 2023

cq-bot mentioned this pull request Apr 19, 2023

chore(main): Release plugins-source-aws v16.3.0 #10115

Merged

bbernays deleted the parallelization branch April 19, 2023 20:50


		In AWS organization mode, CloudQuery will source all accounts underneath automatically

		- `initialization_concurrency` (int) (default: 4)

feat(aws): Parallelize Initialization of Accounts after discovery #10177

feat(aws): Parallelize Initialization of Accounts after discovery #10177

Uh oh!

Conversation

bbernays commented Apr 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yevgenypats Apr 18, 2023

Choose a reason for hiding this comment

Uh oh!

bbernays Apr 18, 2023

Choose a reason for hiding this comment

Uh oh!

bbernays Apr 18, 2023

Choose a reason for hiding this comment

Uh oh!

erezrokah Apr 19, 2023

Choose a reason for hiding this comment

Uh oh!

bbernays Apr 19, 2023

Choose a reason for hiding this comment

Uh oh!

erezrokah Apr 20, 2023

Choose a reason for hiding this comment

Uh oh!

bbernays Apr 20, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hermanschaaf left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

bbernays commented Apr 18, 2023 •

edited

Loading