Skip to content

[otap-dataflow] Arrow IPC compression misbehaving #1129

@clhain

Description

@clhain

Related to #1085. cc @albertlockett

I think the additional arrow compression stuff may not be behaving quite as expected. I was running the demo earlier this week on an image prior to this change. Testing today from main I noticed the cpu use went through the roof on any arrow exporter, and I think it's due to the default enabled zstd compression introduced in 1085. See screenshot below.

The lower cpu use is with what was previously the default behavior (payload_compression: none). On the right, everything is the same except removing that statement.

Image

Metadata

Metadata

Assignees

Labels

apache-arrowLow level Apache Arrow tasksengine-capabilityInternal engine featuresperf-optPerformance and optimization

Type

Projects

Status

Done

Relationships

None yet

Development

No branches or pull requests

Issue actions