Skip to content

Hitting quota limit when starting a large number of pub/sub subscribers simultaneously #2111

@rdmoore

Description

@rdmoore

I have been spinning up a lot of pods within large node-pools in ContainerEngine. When the pods start subscribing to pub/sub I often get
RESOURCE_EXHAUSTED: Insufficient tokens for quota 'administrator' and limit 'CLIENT_PROJECT-100s'
I think this is due to the subscriber starting multiple channels per core. First, isn't core-count improperly biased in kubernetes - I have a machine that has 16 cores but might be sharing them with other pods. My use case results in 10-20 minute of processing per pub/sub message when using 16 cores. Isn't 160 channels overkill? I am assuming that starting 10,000 to 20,000 channels simultaneously the reason I'm hitting the quota. Is the large number of channels for use cases that need to consume large number of messages per second?

I'd like to add a configuration parameter to limit the number of channels. Are the other considerations for having large numbers of channels? Perhaps a minimum and/or maximum number of channels?

I can create a pull request but I'm relatively new to the pub/sub code base and wanted to make sure I was understanding things appropriately.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions