Skip to content

Rate limit github clone calls#142

Merged
james00012 merged 2 commits into
gruntwork-io:masterfrom
cbrand:master
Nov 18, 2024
Merged

Rate limit github clone calls#142
james00012 merged 2 commits into
gruntwork-io:masterfrom
cbrand:master

Conversation

@cbrand

@cbrand cbrand commented Jan 8, 2024

Copy link
Copy Markdown
Contributor

Description

Fixes #139.

Add rate limiting to github clone calls, avoiding issues with github "permission denied" and EOF errors when cloning too many repositories in parallel.

TODOs

Read the Gruntwork contribution guidelines.

  • Update the docs.
  • Run the relevant tests successfully, including pre-commit checks.
  • Ensure any 3rd party code adheres with our license policy or delete this line if its not applicable.
  • Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes

Added

  • Rate limiting Github Clone calls to 4 per default, allows being parametrized by the --max-concurrent-clones flag

Migration Guide

None needed

Add rate limiting to github clone calls, avoiding issues
with github "permission denied" and EOF errors when cloning too many repositories in parallel.

Fixes gruntwork-io#139
@xavierchanth

xavierchanth commented Feb 23, 2024

Copy link
Copy Markdown

For the reviewers, I was able to build git-xargs and used this build to clone ~80 repos mostly successfully, while the current release was only cloning 1 repo.

@flinn

flinn commented Mar 19, 2024

Copy link
Copy Markdown

@brikis98 @hongil0316 What all is necessary in order to help get these changes merged/published? Asking as we've started to use git-xargs recently and without this feature we'll be very unlikely to proceed in using this tool as we're attempting to use it across ~300+ repositories.

@yogur

yogur commented Aug 23, 2024

Copy link
Copy Markdown

We have 2000+ repos to update and I had to use a script to execute git-xargs in batches of 50 repos to go around repo clone rate-limiting, which somewhat defeats the purpose of batching offered by this tool!

I compiled the changes in this PR and used it. It solved our issue! Would be great if it gets merged soon.

@james00012

Copy link
Copy Markdown
Contributor

Sorry for the delay. It seems like some of the tests for repository is failing. Can you make sure those tests pass?

FAIL github.com/gruntwork-io/git-xargs/repository 300.028s

@james00012

Copy link
Copy Markdown
Contributor

Friendly ping on this PR.

@cbrand

cbrand commented Nov 14, 2024

Copy link
Copy Markdown
Contributor Author

@james03160927 Hey I had a chance to take a look and seem to have fixed the unit tests. Please take another look:

?   	github.com/gruntwork-io/git-xargs/common	[no test files]
?   	github.com/gruntwork-io/git-xargs/config	[no test files]
?   	github.com/gruntwork-io/git-xargs/local	[no test files]
?   	github.com/gruntwork-io/git-xargs/mocks	[no test files]
?   	github.com/gruntwork-io/git-xargs/printer	[no test files]
?   	github.com/gruntwork-io/git-xargs/stats	[no test files]
?   	github.com/gruntwork-io/git-xargs/util	[no test files]
ok  	github.com/gruntwork-io/git-xargs	0.488s
ok  	github.com/gruntwork-io/git-xargs/auth	0.223s
ok  	github.com/gruntwork-io/git-xargs/cmd	1.893s
ok  	github.com/gruntwork-io/git-xargs/io	0.680s
ok  	github.com/gruntwork-io/git-xargs/repository	2.154s
ok  	github.com/gruntwork-io/git-xargs/types	1.158s

@james00012 james00012 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Passed all the tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rate limiting for repository cloning?

5 participants