Skip to content

[Enhancement] Define and standardize some common configurations for all Sources #5101

@cnzakii

Description

@cnzakii

Search before asking

  • I had searched in the issues and found no similar issues.

Enhancement Request

In the various implementations of Source, there are often some common configurations scattered across different places, such as the capacity of the storage queue, the number of elements to poll each time, etc. Additionally, in some source implementations, the poll method can block for up to several seconds in the worst-case scenario, reaching maxBatchSize * pollTimeout. Therefore, it is necessary to define common parameters that support configuration and also limit the maximum waiting time for poll.

Describe the solution you'd like

  1. Added a PollConfig configuration class that includes parameters for capacity, maxBatchSize, and maxWaitTime, and incorporated it as a variable in SourceConfig to support user configuration with default values.
  2. Replaced the fixed constants for capacity, maxBatchSize, and maxWaitTime in the original source implementations with the corresponding variables from PollConfig.
  3. Optimized some poll methods to ensure the maximum waiting time does not exceed maxWaitTime, preventing blocking for several seconds.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions