Skip to content

Ingester creates new bigchunk subchunk immediately after handover #1220

@bboreham

Description

@bboreham

Bigchunks are held as a slice of sub-chunks, and a new sub-chunk is added every N samples by reference to remainingSamples which counts down from the maximum (currently 120).

remainingSamples is not initialized in Unmarshal(), so after an ingester hand-over a new sub-chunk is created immediately.

Leaving the smaller sub-chunk reduces overall compression ratios and slows everything down a bit.

While it would be expensive to count remainingSamples accurately, it seems to me we could estimate it based on the byte size of the last sub-chunk. We can count samples at the same time as we determine first and last sample for each sub-chunk.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions