Skip to content

[Feature] Improving arrow flight sql with Kubernetes setup #44599

Closed

Description

@jonasbrami

Search before asking

  • I had searched in the issues and found no similar issues.

Description

Related to this merge request #43281.

I am afraid of the performance degradation of using Arrow Flight SQL from outside the Kubernetes cluster because from my understanding, it would be like setting enable_parallel_result_sink = False . Would it be theoretically possible, instead of having random BE resolution for the reverse proxy, have a smarter reverse proxy that could be, for example, based on path prefix (doris-url/be1, doris-url/be2) ? Then those doris-url/be{i} would become the "advertised" urls for the grpc client and would be reachable from outside the kubernetes cluster. Do you think it would be a good idea?

Use case

Faster queries when using arrow flight sql from outside Kubernetes cluster

Related issues

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. changed the title [-][Feature] [/-] [+][Feature] Improving arrow flight sql in Kubernetes[/+] on Nov 26, 2024
  2. changed the title [-][Feature] Improving arrow flight sql in Kubernetes[/-] [+][Feature] Improving arrow flight sql with Kubernetes setup[/+] on Nov 26, 2024
  3. xinyiZzz commented on Dec 2, 2024

    @xinyiZzz
    Contributor

    Hi @jonasbrami , thanks for your suggestion.

    Are you talking about implementing a custom reverse proxy in Kubernetes?

    That's a great idea, when you're ready, you can describe how to do it, which hopefully will be a common feature.

  4. jonasbrami commented on Dec 3, 2024

    @jonasbrami
    Author

    I was thinking about the following:

    1. Doris Operator exposes configuration of a base public hostname for the Doris backends. For example public-hostname-doris.dev.
    2. The operator would set in the be conf the public host to be something like public-hostname-doris.dev/be{I}.
    3. Then it is possible to leverage something like Traefik to route all HTTP calls to public-hostname-doris.dev/be{I} to the right internal service.

    Maybe it is more an issue to open for Doris Kubernetes operator than for Doris itself. The only requirement for Doris is to expose the public hostname config but it is already the case

  5. xinyiZzz commented on Dec 5, 2024

    @xinyiZzz
    Contributor

    I was thinking about the following:

    1. Doris Operator exposes configuration of a base public hostname for the Doris backends. For example public-hostname-doris.dev.
    2. The operator would set in the be conf the public host to be something like public-hostname-doris.dev/be{I}.
    3. Then it is possible to leverage something like Traefik to route all HTTP calls to public-hostname-doris.dev/be{I} to the right internal service.

    Maybe it is more an issue to open for Doris Kubernetes operator than for Doris itself. The only requirement for Doris is to expose the public hostname config but it is already the case

    Sounds like a good idea, I will contact the developers of Doris Kubernetes Operator to confirm this.

  6. jonasbrami commented on Dec 5, 2024

    @jonasbrami
    Author

    Coool! Thank you! I would be happy to try implementing this

  7. hechao-ustc commented on Dec 10, 2024

    @hechao-ustc
    Contributor

    I was thinking about the following:

    1. Doris Operator exposes configuration of a base public hostname for the Doris backends. For example public-hostname-doris.dev.
    2. The operator would set in the be conf the public host to be something like public-hostname-doris.dev/be{I}.
    3. Then it is possible to leverage something like Traefik to route all HTTP calls to public-hostname-doris.dev/be{I} to the right internal service.

    Maybe it is more an issue to open for Doris Kubernetes operator than for Doris itself. The only requirement for Doris is to expose the public hostname config but it is already the case

    Hi, @jonasbrami, #42881 can this PR meet your 1~2 requirements. If yes, you can wait for this PR to be merged before doing the third requirement.

  8. github-actions commented on Sep 12, 2025

    @github-actions
    Contributor

    We're closing this issue because it hasn't been updated in a while.
    If you'd like to revive this issue, please reopen it and feel free a maintainer to remove the Stale tag!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Stalekind/featureCategorizes issue or PR as related to a new feature.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions