Skip to content

[multistage] Assigning Workers for Intermediate Stages is not tenant aware #10605

@vvivekiyer

Description

@vvivekiyer

Currently, when we assign servers (workers) for processing intermediate stages, we use the BrokerRoutingManager.getEnabledServerInstanceMap() function to pick potential servers for processing the intermediate stage.

However, BrokerRoutingManager.getEnabledServerInstanceMap() returns the list of all (enabled) server instances in the cluster. Considering a large cluster deployment like Linkedin where the cluster is divided based into multiple tenants, this potentially means that servers from a different (not related) tenant could be processing intermediate stages for query.

If the tables involved in a query are across two different tenants, it is possible that there is no common server tenant between them. So, I'm thinking of assigning a server from any one of the tenants belonging to any one of the tables.

I am working on a potential fix for this issue.

cc: @walterddr @somandal @siddharthteotia

Metadata

Metadata

Assignees

Labels

bugmulti-stageRelated to the multi-stage query engine

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions