Skip to content

[Multi-stage] OpChain stats handling improvement #12426

New issue
New issue
Open
Open
[Multi-stage] OpChain stats handling improvement#12426
Labels
multi-stageRelated to the multi-stage query engineobservabilityperformance
@Jackie-Jiang

Description

@Jackie-Jiang
Jackie-Jiang
opened on Feb 15, 2024
  • OpChain stats is a mapping from worker id to serialized map, then serialized as JSON bytes. It is very inefficient. Since we know the metadata key upfront, we can encode the metadata much more efficient. See DataTable for example of how v1 handles query stats
  • When tracing is not enabled, no need to send per worker stats. We should aggregate the stats for each stage when finishing the stage, instead of sending stats for each worker
  • EOS block containing the stats are duplicated to all workers. Sending the stats to only one worker should be enough.

Metadata

Metadata

Assignees

No one assigned

    Labels

    multi-stageRelated to the multi-stage query engineobservabilityperformance

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions