Skip to content

[DISCUSS] Avoid broker OOM #5751

@codelipenghui

Description

@codelipenghui

Is your feature request related to a problem? Please describe.
Currently, we have do some performance tests, if broker with a smaller direct memory(e.g. 2G) and run pulsar-perf to write messages, the broker becomes unstable, because broker read messages from the channel automatically and the ByteBuf released until the entry write to Bookie successfully or timeout.

#3986 and #5710 introduce the topic level and broker level publish rate limiter which can allow users to limit the publish rate, this can effectively avoid broker OOM.

If only think about the broker OOM(the publish rate limiter is more useful in terms of resource constraints), use publish rate limiter to avoid broker OOM will limit the performance of the broker.

For example, if the bookie write timeout is 10sec, and the direct memory is 2GB, if we use publish rate limiter to avoid broker OOM, we need to limit the publish rate in bytes < 200MB/s (2GB/10s).

It's better to add a publish buffer size limit(default is 1/4 total direct memory) , when the message incoming, the publish buffer size increase and when the message is write to bookie successful or timeout the publish buffer size decrease. If the buffer size is over the limitation, stop auto read from the producer channel.

Metadata

Metadata

Assignees

No one assigned

    Labels

    lifecycle/staletype/featureThe PR added a new feature or issue requested a new feature

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions