Rate limit github clone calls#142
Conversation
Add rate limiting to github clone calls, avoiding issues with github "permission denied" and EOF errors when cloning too many repositories in parallel. Fixes gruntwork-io#139
|
For the reviewers, I was able to build git-xargs and used this build to clone ~80 repos mostly successfully, while the current release was only cloning 1 repo. |
|
@brikis98 @hongil0316 What all is necessary in order to help get these changes merged/published? Asking as we've started to use git-xargs recently and without this feature we'll be very unlikely to proceed in using this tool as we're attempting to use it across ~300+ repositories. |
|
We have 2000+ repos to update and I had to use a script to execute I compiled the changes in this PR and used it. It solved our issue! Would be great if it gets merged soon. |
|
Sorry for the delay. It seems like some of the tests for repository is failing. Can you make sure those tests pass?
|
|
Friendly ping on this PR. |
|
@james03160927 Hey I had a chance to take a look and seem to have fixed the unit tests. Please take another look: |
james00012
left a comment
There was a problem hiding this comment.
LGTM. Passed all the tests.
Description
Fixes #139.
Add rate limiting to github clone calls, avoiding issues with github "permission denied" and EOF errors when cloning too many repositories in parallel.
TODOs
Read the Gruntwork contribution guidelines.
Release Notes
Added
--max-concurrent-clonesflagMigration Guide
None needed