Fix memory corruption and fatal error in Snappy by bogdandrutu · Pull Request #15321 · open-telemetry/opentelemetry-collector

bogdandrutu · 2026-05-19T17:26:04Z

Description

The problem happens because:

T1: Executes writer.Close() and gets descheduled after z.pool.Put(z) but before leaving the Do function.
T2: Executes compressor.Decompress, get the latest re-added writer from T1, and overwrites the once variable.
T1: Resumes and tries to unlock the overwritten once which is not locked.

This is an UB and everything can happen when we corrupt the memory like this.

fatal error: sync: unlock of unlocked mutex
goroutine 48499 [running]:
internal/sync.fatal({0xa4032ec?, 0x373e910417c0?})
	GOROOT/src/runtime/panic.go:1191 +0x18
internal/sync.(*Mutex).unlockSlow(0x373e910417d4, 0xffffffff)
	GOROOT/src/internal/sync/mutex.go:204 +0x35
internal/sync.(*Mutex).Unlock(...)
	GOROOT/src/internal/sync/mutex.go:198
sync.(*Mutex).Unlock(...)
	GOROOT/src/sync/mutex.go:65
sync.(*Once).doSlow(0x373e91c44f50?, 0x373eadc29fed?)
	GOROOT/src/sync/once.go:80 +0xe4
sync.(*Once).Do(...)
	GOROOT/src/sync/once.go:69
go.opentelemetry.io/collector/internal/grpccompression/snappy.(*reader).Read(0x373e910417c0, {0x373eadc29fed?, 0x373e4d674300?, 0x8000?})
	external/io_opentelemetry_go_collector/internal/grpccompression/snappy/snappy.go:96 +0x65
io.(*LimitedReader).Read(0x373eaa8bc6c0, {0x373eadc29fed?, 0x373e4d6526c0?, 0x373eaa8bc6c0?})
	GOROOT/src/io/io.go:479 +0x43
google.golang.org/grpc/mem.ReadAll({0xad75ec0, 0x373eaa8bc6c0}, {0xadfedd8, 0x373e4d6526c0})
	external/org_golang_google_grpc/mem/buffer_slice.go:268 +0x188
google.golang.org/grpc.decompress({0xae3eea0, 0x373e4d66a320}, {0x373e4d836c60?, 0x373e68b4c8b0?, 0x425745?}, {0x0?, 0x0?}, 0x40000000, {0xadfedd8, 0x373e4d6526c0})
	external/org_golang_google_grpc/rpc_util.go:999 +0x15c
google.golang.org/grpc.recvAndDecompress(0x373e9e41f170, {0xad7dfa0, 0x373e96992820}, {0x0, 0x0}, 0x40000000, 0x373e91041d00, {0xae3eea0, 0x373e4d66a320}, 0x1)
	external/org_golang_google_grpc/rpc_util.go:947 +0x2ab
google.golang.org/grpc.(*Server).processUnaryRPC(0x373e4f182008, {0xae516e0, 0x373e9e41f0b0}, 0x373e96992820, 0x373e4e86a060, 0x10da6290, 0x0)
	external/org_golang_google_grpc/server.go:1386 +0xf25
google.golang.org/grpc.(*Server).handleStream(0x373e4f182008, {0xae53b28, 0x373e4d859d40}, 0x373e96992820)
	external/org_golang_google_grpc/server.go:1856 +0xc2e
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	external/org_golang_google_grpc/server.go:1065 +0x7f
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 302
	external/org_golang_google_grpc/server.go:1076 +0x11d

Link to tracking issue

Fixes #15237
Fixes #15320

Testing

Documentation

Signed-off-by: Bogdan Drutu <[email protected]>

codspeed-hq · 2026-05-19T18:04:49Z

Merging this PR will degrade performance by 30.25%

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 2 regressed benchmarks
✅ 5 untouched benchmarks
⏩ 76 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
❌	`BenchmarkMemoryQueueWaitForResult`	49.9 µs	74.3 µs	-32.82%
❌	`BenchmarkPersistentQueue`	145.1 µs	200.3 µs	-27.58%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing bogdandrutu:fix-snappy (0315a7b) with main (78f2e2a)}

76 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

codecov · 2026-05-19T18:14:14Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.23%. Comparing base (6fe6256) to head (0315a7b).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #15321      +/-   ##
==========================================
- Coverage   91.24%   91.23%   -0.01%     
==========================================
  Files         706      706              
  Lines       46173    46165       -8     
==========================================
- Hits        42129    42118      -11     
- Misses       2834     2836       +2     
- Partials     1210     1211       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

#### Description The problem happens because: T1: Executes `writer.Close()` and gets descheduled after `z.pool.Put(z)` but before leaving the `Do` function. T2: Executes `compressor.Decompress`, get the latest re-added writer from T1, and overwrites the `once` variable. T1: Resumes and tries to unlock the overwritten `once` which is not locked. This is an UB and everything can happen when we corrupt the memory like this. ``` fatal error: sync: unlock of unlocked mutex goroutine 48499 [running]: internal/sync.fatal({0xa4032ec?, 0x373e910417c0?}) GOROOT/src/runtime/panic.go:1191 +0x18 internal/sync.(*Mutex).unlockSlow(0x373e910417d4, 0xffffffff) GOROOT/src/internal/sync/mutex.go:204 +0x35 internal/sync.(*Mutex).Unlock(...) GOROOT/src/internal/sync/mutex.go:198 sync.(*Mutex).Unlock(...) GOROOT/src/sync/mutex.go:65 sync.(*Once).doSlow(0x373e91c44f50?, 0x373eadc29fed?) GOROOT/src/sync/once.go:80 +0xe4 sync.(*Once).Do(...) GOROOT/src/sync/once.go:69 go.opentelemetry.io/collector/internal/grpccompression/snappy.(*reader).Read(0x373e910417c0, {0x373eadc29fed?, 0x373e4d674300?, 0x8000?}) external/io_opentelemetry_go_collector/internal/grpccompression/snappy/snappy.go:96 +0x65 io.(*LimitedReader).Read(0x373eaa8bc6c0, {0x373eadc29fed?, 0x373e4d6526c0?, 0x373eaa8bc6c0?}) GOROOT/src/io/io.go:479 +0x43 google.golang.org/grpc/mem.ReadAll({0xad75ec0, 0x373eaa8bc6c0}, {0xadfedd8, 0x373e4d6526c0}) external/org_golang_google_grpc/mem/buffer_slice.go:268 +0x188 google.golang.org/grpc.decompress({0xae3eea0, 0x373e4d66a320}, {0x373e4d836c60?, 0x373e68b4c8b0?, 0x425745?}, {0x0?, 0x0?}, 0x40000000, {0xadfedd8, 0x373e4d6526c0}) external/org_golang_google_grpc/rpc_util.go:999 +0x15c google.golang.org/grpc.recvAndDecompress(0x373e9e41f170, {0xad7dfa0, 0x373e96992820}, {0x0, 0x0}, 0x40000000, 0x373e91041d00, {0xae3eea0, 0x373e4d66a320}, 0x1) external/org_golang_google_grpc/rpc_util.go:947 +0x2ab google.golang.org/grpc.(*Server).processUnaryRPC(0x373e4f182008, {0xae516e0, 0x373e9e41f0b0}, 0x373e96992820, 0x373e4e86a060, 0x10da6290, 0x0) external/org_golang_google_grpc/server.go:1386 +0xf25 google.golang.org/grpc.(*Server).handleStream(0x373e4f182008, {0xae53b28, 0x373e4d859d40}, 0x373e96992820) external/org_golang_google_grpc/server.go:1856 +0xc2e google.golang.org/grpc.(*Server).serveStreams.func2.1() external/org_golang_google_grpc/server.go:1065 +0x7f created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 302 external/org_golang_google_grpc/server.go:1076 +0x11d ```  #### Link to tracking issue Fixes #15237 Fixes #15320  #### Testing  #### Documentation   #### Description  #### Link to tracking issue Fixes #  #### Testing  #### Documentation  Signed-off-by: Bogdan Drutu <[email protected]>

#### Description The problem happens because: T1: Executes `writer.Close()` and gets descheduled after `z.pool.Put(z)` but before leaving the `Do` function. T2: Executes `compressor.Decompress`, get the latest re-added writer from T1, and overwrites the `once` variable. T1: Resumes and tries to unlock the overwritten `once` which is not locked. This is an UB and everything can happen when we corrupt the memory like this. ``` fatal error: sync: unlock of unlocked mutex goroutine 48499 [running]: internal/sync.fatal({0xa4032ec?, 0x373e910417c0?}) GOROOT/src/runtime/panic.go:1191 +0x18 internal/sync.(*Mutex).unlockSlow(0x373e910417d4, 0xffffffff) GOROOT/src/internal/sync/mutex.go:204 +0x35 internal/sync.(*Mutex).Unlock(...) GOROOT/src/internal/sync/mutex.go:198 sync.(*Mutex).Unlock(...) GOROOT/src/sync/mutex.go:65 sync.(*Once).doSlow(0x373e91c44f50?, 0x373eadc29fed?) GOROOT/src/sync/once.go:80 +0xe4 sync.(*Once).Do(...) GOROOT/src/sync/once.go:69 go.opentelemetry.io/collector/internal/grpccompression/snappy.(*reader).Read(0x373e910417c0, {0x373eadc29fed?, 0x373e4d674300?, 0x8000?}) external/io_opentelemetry_go_collector/internal/grpccompression/snappy/snappy.go:96 +0x65 io.(*LimitedReader).Read(0x373eaa8bc6c0, {0x373eadc29fed?, 0x373e4d6526c0?, 0x373eaa8bc6c0?}) GOROOT/src/io/io.go:479 +0x43 google.golang.org/grpc/mem.ReadAll({0xad75ec0, 0x373eaa8bc6c0}, {0xadfedd8, 0x373e4d6526c0}) external/org_golang_google_grpc/mem/buffer_slice.go:268 +0x188 google.golang.org/grpc.decompress({0xae3eea0, 0x373e4d66a320}, {0x373e4d836c60?, 0x373e68b4c8b0?, 0x425745?}, {0x0?, 0x0?}, 0x40000000, {0xadfedd8, 0x373e4d6526c0}) external/org_golang_google_grpc/rpc_util.go:999 +0x15c google.golang.org/grpc.recvAndDecompress(0x373e9e41f170, {0xad7dfa0, 0x373e96992820}, {0x0, 0x0}, 0x40000000, 0x373e91041d00, {0xae3eea0, 0x373e4d66a320}, 0x1) external/org_golang_google_grpc/rpc_util.go:947 +0x2ab google.golang.org/grpc.(*Server).processUnaryRPC(0x373e4f182008, {0xae516e0, 0x373e9e41f0b0}, 0x373e96992820, 0x373e4e86a060, 0x10da6290, 0x0) external/org_golang_google_grpc/server.go:1386 +0xf25 google.golang.org/grpc.(*Server).handleStream(0x373e4f182008, {0xae53b28, 0x373e4d859d40}, 0x373e96992820) external/org_golang_google_grpc/server.go:1856 +0xc2e google.golang.org/grpc.(*Server).serveStreams.func2.1() external/org_golang_google_grpc/server.go:1065 +0x7f created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 302 external/org_golang_google_grpc/server.go:1076 +0x11d ```  #### Link to tracking issue Fixes open-telemetry#15237 Fixes open-telemetry#15320  #### Testing  #### Documentation  Signed-off-by: Bogdan Drutu <[email protected]>

bogdandrutu requested a review from a team as a code owner May 19, 2026 17:26

bogdandrutu requested a review from evan-bradley May 19, 2026 17:26

bogdandrutu enabled auto-merge May 19, 2026 17:28

codeboten approved these changes May 19, 2026

View reviewed changes

bogdandrutu force-pushed the fix-snappy branch 3 times, most recently from 60333b3 to 0e28d50 Compare May 19, 2026 17:45

Fix memory corruption and fatal error in Snappy

0315a7b

Signed-off-by: Bogdan Drutu <[email protected]>

bogdandrutu force-pushed the fix-snappy branch from 0e28d50 to 0315a7b Compare May 19, 2026 17:56

dmitryax approved these changes May 19, 2026

View reviewed changes

bogdandrutu added this pull request to the merge queue May 19, 2026

Merged via the queue into open-telemetry:main with commit 40a0e3d May 19, 2026
65 of 66 checks passed

bogdandrutu deleted the fix-snappy branch May 19, 2026 18:48

droidnxs mentioned this pull request May 25, 2026

[grpccompression/snappy]: avoid sync.Once reuse on pooled snappy compressors #15238

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix memory corruption and fatal error in Snappy#15321

Fix memory corruption and fatal error in Snappy#15321
bogdandrutu merged 1 commit into
open-telemetry:mainfrom
bogdandrutu:fix-snappy

bogdandrutu commented May 19, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented May 19, 2026

Uh oh!

codecov Bot commented May 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

bogdandrutu commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Link to tracking issue

Testing

Documentation

Uh oh!

codspeed-hq Bot commented May 19, 2026

Merging this PR will degrade performance by 30.25%

Performance Changes

Footnotes

Uh oh!

codecov Bot commented May 19, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bogdandrutu commented May 19, 2026 •

edited

Loading