-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Hitting quota limit when starting a large number of pub/sub subscribers simultaneously #2111
Description
I have been spinning up a lot of pods within large node-pools in ContainerEngine. When the pods start subscribing to pub/sub I often get
RESOURCE_EXHAUSTED: Insufficient tokens for quota 'administrator' and limit 'CLIENT_PROJECT-100s'
I think this is due to the subscriber starting multiple channels per core. First, isn't core-count improperly biased in kubernetes - I have a machine that has 16 cores but might be sharing them with other pods. My use case results in 10-20 minute of processing per pub/sub message when using 16 cores. Isn't 160 channels overkill? I am assuming that starting 10,000 to 20,000 channels simultaneously the reason I'm hitting the quota. Is the large number of channels for use cases that need to consume large number of messages per second?
I'd like to add a configuration parameter to limit the number of channels. Are the other considerations for having large numbers of channels? Perhaps a minimum and/or maximum number of channels?
I can create a pull request but I'm relatively new to the pub/sub code base and wanted to make sure I was understanding things appropriately.