Component(s)
exporter/exporterhelper
What happened?
Describe the bug
When sending_queue::sizer is explicitly set to bytes and sending_queue::batch::sizer is not set, the batch silently uses the default items sizer instead of inheriting bytes from the parent. This causes sending_queue::batch::max_size to be interpreted as an item count rather than a byte limit, effectively disabling batch splitting for byte-sized payloads.
For example, the batches then exceed the downstream gRPC server's 4 MiB MaxRecvMsgSize, causing permanent ResourceExhausted errors and data loss.
Steps to reproduce
- Configure an OTLP gRPC exporter with
sending_queue::sizer: bytes and a batch section without sizer:
exporters:
otlp_grpc/downstream:
endpoint: "downstream:4317"
sending_queue:
enabled: true
sizer: bytes
queue_size: 500000000
num_consumers: 10
batch:
flush_timeout: 300s
min_size: 1000000
max_size: 3500000 # intended as 3.5 MB
-
Send enough metrics that a single batch exceeds 4 MiB of proto bytes but stays well under 3,500,000 data points.
-
Observe:
error internal/queue_sender.go:50 Exporting failed. Dropping data.
"error": "not retryable error: Permanent error: rpc error: code = ResourceExhausted
desc = grpc: received message after decompression larger than max 4194304",
"dropped_items": 59777
The 59,777 items in a single batch confirms max_size: 3500000 is being interpreted as 3.5M items (never reached), not 3.5 MB.
What did you expect to see?
With sending_queue::sizer: bytes, the batch::sizer should inherit bytes, making max_size: 3500000 mean 3.5 MB. Batches should be split to stay under that limit.
What did you see instead?
The batch sizer silently defaults to items. Batches are never split because the item count never reaches 3.5M. The resulting proto payloads exceed the 4 MiB gRPC limit, causing permanent ResourceExhausted errors and data loss with no prior warning.
Workaround
Explicitly set batch::sizer: bytes:
batch:
sizer: bytes # must be explicit
flush_timeout: 200ms
min_size: 1000000
max_size: 3500000
Collector version
v0.144.0
Environment information
OpenTelemetry Collector configuration
Log output
Additional context
No response
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.
Component(s)
exporter/exporterhelper
What happened?
Describe the bug
When
sending_queue::sizeris explicitly set tobytesandsending_queue::batch::sizeris not set, the batch silently uses the defaultitemssizer instead of inheritingbytesfrom the parent. This causessending_queue::batch::max_sizeto be interpreted as an item count rather than a byte limit, effectively disabling batch splitting for byte-sized payloads.For example, the batches then exceed the downstream gRPC server's 4 MiB
MaxRecvMsgSize, causing permanentResourceExhaustederrors and data loss.Steps to reproduce
sending_queue::sizer: bytesand abatchsection withoutsizer:Send enough metrics that a single batch exceeds 4 MiB of proto bytes but stays well under 3,500,000 data points.
Observe:
The 59,777 items in a single batch confirms
max_size: 3500000is being interpreted as 3.5M items (never reached), not 3.5 MB.What did you expect to see?
With
sending_queue::sizer: bytes, thebatch::sizershould inheritbytes, makingmax_size: 3500000mean 3.5 MB. Batches should be split to stay under that limit.What did you see instead?
The batch sizer silently defaults to
items. Batches are never split because the item count never reaches 3.5M. The resulting proto payloads exceed the 4 MiB gRPC limit, causing permanentResourceExhaustederrors and data loss with no prior warning.Workaround
Explicitly set
batch::sizer: bytes:Collector version
v0.144.0
Environment information
OpenTelemetry Collector configuration
Log output
Additional context
No response
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding
+1orme too, to help us triage it. Learn more here.