Skip to content

Single SMLC/DMLC with multiple consumers: set consumer priority as its index to even load distribution accross servers #3092

@jonenst

Description

@jonenst

** Problem **
When having multiple servers that each have multiple consumers all on the same queue, I would like to get even message distribution on the servers (not on each individual consumer). That is, no server should consume more than one message if at least another server is idle. This allows to relax ram requirements under small load, and allows to spread the cpu load on the different servers.

Expected Behavior

Currently i'm using one SMLC with setConcurrentConsumers() (would be the same with DMLC setConsumersPerQueue()) with a value >1. I think my problem would be solved if I was able to set each consumer to have a consumer priority equal to its index. This way, accros all servers, there would be one priority N thread, one N-1, ... down to priority 1.

Current Behavior

Currently, the same consumer parameters of the container are set on all the consumers

** alternatives **
are there better solutions than having different consumer priorities in the same SMLC / DMLC ? I can think of the following but maybe they are not even feaseable:

Context

I'm using spring cloud stream which is using SMLC or DMLC directly.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions