Hi, I ran into an issue these last few days with pubsub processing more messages than I expected. After digging into this a bit, it looks like I may have misinterpreted what this means. Is this a bug or intended behavior?
On the surface, after reading this description, I expected that if I set NumGoroutines to 1, my code will only receive 1 message at a time.
// NumGoroutines is the number of goroutines Receive will spawn to pull
// messages concurrently. If NumGoroutines is less than 1, it will be treated
// as if it were DefaultReceiveSettings.NumGoroutines.
NumGoroutines int
However, while it's true that pubsub spawns only 1 Goroutine to receive messages, it also calls the receiving function in another goroutine. This means all the messages pubsub receives will be processed in parallel.
The task my code works on is both cpu and memory intensive. By pubsub spawning goroutines for every message, the system quickly runs out of memory.
group.Go(func() error {
// TODO(jba): call release when the message is available for GC.
// This considers the message to be released when
// f is finished, but f may ack early or not at all.
defer fc.release(len(msg.Data))
f(ctx2, msg)
return nil
})
What's the right way to control the flow of messages? Is there anyway to control the flow so only 1 message is processed at a time? (btw, I realize I can do my own flow control using a channel, I was curious the point of NumGoroutine if every message spawns a new goroutine.)
Hi, I ran into an issue these last few days with pubsub processing more messages than I expected. After digging into this a bit, it looks like I may have misinterpreted what this means. Is this a bug or intended behavior?
On the surface, after reading this description, I expected that if I set NumGoroutines to 1, my code will only receive 1 message at a time.
However, while it's true that pubsub spawns only 1 Goroutine to receive messages, it also calls the receiving function in another goroutine. This means all the messages pubsub receives will be processed in parallel.
The task my code works on is both cpu and memory intensive. By pubsub spawning goroutines for every message, the system quickly runs out of memory.
What's the right way to control the flow of messages? Is there anyway to control the flow so only 1 message is processed at a time? (btw, I realize I can do my own flow control using a channel, I was curious the point of NumGoroutine if every message spawns a new goroutine.)