Calculate Ingest Queue Compression Ratio Over Time

Analyze the ratio between uncompressed and compressed ingest queue bytes

Query

flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2[/Filter/] 3{Conditional} 4{{Aggregate}} 5[\Add Field/] 6[/Drop Field\] result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> 4 4 --> 5 5 --> 6 6 --> result style 6 fill:#2ac76d; click 6 #examples-ingest-queue-6
logscale
#type=humio #kind=metrics
| name=/^ingest-writer-(?<un>un)?compressed-bytes$/
| case {
un=*
| un := m1;
comp := m1 }
| timeChart(function=[sum(un,as=un),sum(comp,as=comp)], minSpan=1m)
| ratio:=un/comp
| drop([un,comp])

Introduction

The timeChart() function can be used to visualize time-based metrics data in a chart format, allowing for multiple aggregation functions to be applied to different fields.

This query is used to calculate ingest queue average compression. A compression ratio is used to express the amount of data that has been saved by compressing. A 10x ratio would mean that 100 GB of data is compressed down to 10 GB of data. This value is discovered by dividing the initial data size by the compressed data size, so for example 100/10.

In this example, the timeChart() function is used to compare uncompressed and compressed ingest bytes over time to calculate a compression ratio. The ratio helps monitor the efficiency of LogScale's data compression over time.

Example incoming data might look like this:

@timestamp#type#kindnamem1
2025-11-05T10:00:00Zhumiometricsingest-writer-uncompressed-bytes1500000
2025-11-05T10:00:00Zhumiometricsingest-writer-compressed-bytes300000
2025-11-05T10:01:00Zhumiometricsingest-writer-uncompressed-bytes1800000
2025-11-05T10:01:00Zhumiometricsingest-writer-compressed-bytes360000
2025-11-05T10:02:00Zhumiometricsingest-writer-uncompressed-bytes2000000
2025-11-05T10:02:00Zhumiometricsingest-writer-compressed-bytes400000

Step-by-Step

  1. Starting with the source repository events.

  2. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2[/Filter/] 3{Conditional} 4{{Aggregate}} 5[\Add Field/] 6[/Drop Field\] result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> 4 4 --> 5 5 --> 6 6 --> result style 6 fill:#2ac76d; click 6 #examples-ingest-queue-6 style 1 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    #type=humio #kind=metrics

    Filters for LogScale internal metrics by matching events where #type equals humio and #kind equals metrics.

  3. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2[/Filter/] 3{Conditional} 4{{Aggregate}} 5[\Add Field/] 6[/Drop Field\] result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> 4 4 --> 5 5 --> 6 6 --> result style 6 fill:#2ac76d; click 6 #examples-ingest-queue-6 style 2 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    | name=/^ingest-writer-(?<un>un)?compressed-bytes$/

    Filters for the field name where the string starts with ingest-writer and calculates the ingest queue average compression. Creates a new field named un if the data is uncompressed by using a regular expression match looking for uncompressed-bytes.

    The regular expression:

    • Matches both ingest-writer-compressed-bytes and ingest-writer-uncompressed-bytes.

    • Uses (?<un>un)? to capture the presence of 'un' in an optional capture group named un.

    • The capture group will only exist for uncompressed metrics, helping differentiate between the two types.

  4. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2[/Filter/] 3{Conditional} 4{{Aggregate}} 5[\Add Field/] 6[/Drop Field\] result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> 4 4 --> 5 5 --> 6 6 --> result style 6 fill:#2ac76d; click 6 #examples-ingest-queue-6 style 3 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    | case {
un=*
| un := m1;
comp := m1 }

    Uses a case statement to create two new fields based on the presence of the un field:

    • If un exists (uncompressed bytes), assigns the metric value to field un.

    • For all events (compressed bytes), assigns the metric value to field comp.

  5. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2[/Filter/] 3{Conditional} 4{{Aggregate}} 5[\Add Field/] 6[/Drop Field\] result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> 4 4 --> 5 5 --> 6 6 --> result style 6 fill:#2ac76d; click 6 #examples-ingest-queue-6 style 4 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    | timeChart(function=[sum(un,as=un),sum(comp,as=comp)], minSpan=1m)

    Creates a time chart with one-minute intervals that sums both the uncompressed bytes (un) for each interval and the compressed bytes (comp) for each interval.

    Uses minSpan=1m to ensure one-minute granularity.

  6. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2[/Filter/] 3{Conditional} 4{{Aggregate}} 5[\Add Field/] 6[/Drop Field\] result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> 4 4 --> 5 5 --> 6 6 --> result style 6 fill:#2ac76d; click 6 #examples-ingest-queue-6 style 5 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    | ratio:=un/comp

    Creates a new field named ratio by dividing the uncompressed bytes by the compressed bytes, showing the compression ratio achieved.

  7. flowchart LR; %%{init: {"flowchart": {"defaultRenderer": "elk"}} }%% repo{{Events}} 1[/Filter/] 2[/Filter/] 3{Conditional} 4{{Aggregate}} 5[\Add Field/] 6[/Drop Field\] result{{Result Set}} repo --> 1 1 --> 2 2 --> 3 3 --> 4 4 --> 5 5 --> 6 6 --> result style 6 fill:#2ac76d; click 6 #examples-ingest-queue-6 style 6 fill:#ff0000,stroke-width:4px,stroke:#000;
    logscale
    | drop([un,comp])

    Removes the intermediate fields un and comp, leaving only the timestamp and ratio in the final output.

  8. Event Result set.

Summary and Results

The query is used to calculate and visualize the compression ratio of ingested data over time by comparing uncompressed and compressed byte metrics. A higher ratio indicates better compression efficiency, where for example, a ratio of 5 means the original data was compressed to one-fifth of its original size.

This query is useful, for example, to monitor the effectiveness of data compression in LogScale, identify any changes in compression ratios that might indicate issues with data ingestion, track storage efficiency and capacity planning, and detect anomalies in data compression patterns.

Sample output from the incoming example data:

_bucketratio
2025-11-05T10:00:00Z5.0
2025-11-05T10:01:00Z5.0
2025-11-05T10:02:00Z5.0

Note that the output shows a consistent 5:1 compression ratio, meaning that the uncompressed data is five times larger than the compressed data. In this case, for every 5 bytes of original data, it is compressed down to 1 byte of stored data.