Tracer Version(s)
1.73.1
Go Version(s)
go version go1.24.2 darwin/arm64
Bug Report
We are getting a deadlock on the main.main goroutine when calling the method tracer.Stop(), consequently the app is hanging and not finishing properly.
Looking at the implementation, the internal/remoteconfig/remoteconfig.go#Start:191 client is being initialized before evaluating if the value of the env DD_REMOTE_CONFIGURATION_ENABLED is true|false, which causes the internal channel consumer to not be created (but the client was initialized).
Then, when the tracer.Stop method is called, it internally calls the internal/remoteconfig/remoteconfig.go#Stop:226 function, which checks if the client was initialized (which indeed was) and then sends a message to the internal stop channel (that has no consumers), leading the main goroutine to be stuck forever.
This issue occurs when instrumenting an application with serverless-init, which seems to force the value of the env to be false by default.
The suggested fix would be to move the check on line remoteconfig.go:191 before client, err := newClient(config) is invoked.
Goroutine dump
goroutine 1 gp=0xc0000061c0 m=nil [chan send]:
runtime.gopark(0x7fffb8cf4598?, 0xc000a25c78?, 0xa5?, 0x6a?, 0x7fffb8cf4568?)
/usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000a25c48 sp=0xc000a25c28 pc=0x47162e
runtime.chansend(0xc0000b6070, 0xc000a25d18, 0x1, 0xc0000b6000?)
/usr/local/go/src/runtime/chan.go:270 +0x38d fp=0xc000a25cb8 sp=0xc000a25c48 pc=0x40916d
runtime.chansend1(0x129d0df?, 0x2c?)
/usr/local/go/src/runtime/chan.go:156 +0x17 fp=0xc000a25ce8 sp=0xc000a25cb8 pc=0x408dd7
gopkg.in/DataDog/dd-trace-go.v1/internal/remoteconfig.Stop.func1()
/root/go/pkg/mod/gopkg.in/!data!dog/[email protected]/internal/remoteconfig/remoteconfig.go:226 +0x45 fp=0xc000a25d58 sp=0xc000a25ce8 pc=0xa354c5
sync.(*Once).doSlow(0x1d404a0?, 0xc000a25db0?)
/usr/local/go/src/sync/once.go:76 +0xb4 fp=0xc000a25db8 sp=0xc000a25d58 pc=0x486cf4
sync.(*Once).Do(...)
/usr/local/go/src/sync/once.go:67
gopkg.in/DataDog/dd-trace-go.v1/internal/remoteconfig.Stop()
/root/go/pkg/mod/gopkg.in/!data!dog/[email protected]/internal/remoteconfig/remoteconfig.go:224 +0x37 fp=0xc000a25dd8 sp=0xc000a25db8 pc=0xa313f7
gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.(*tracer).Stop(0xc0001be270)
/root/go/pkg/mod/gopkg.in/!data!dog/[email protected]/ddtrace/tracer/tracer.go:767 +0xf7 fp=0xc000a25e10 sp=0xc000a25dd8 pc=0xd78f17
gopkg.in/DataDog/dd-trace-go.v1/ddtrace/internal.SetGlobalTracer({0x146f1b8, 0x1d3f8a0})
/root/go/pkg/mod/gopkg.in/!data!dog/[email protected]/ddtrace/internal/globaltracer.go:30 +0x8f fp=0xc000a25e40 sp=0xc000a25e10 pc=0x8c4e8f
gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.Stop()
/root/go/pkg/mod/gopkg.in/!data!dog/[email protected]/ddtrace/tracer/tracer.go:246 +0x25 fp=0xc000a25e60 sp=0xc000a25e40 pc=0xd755a5
main.main.func1()
/go/src/REDACTED/main.go:23 +0x18 fp=0xc000a25e70 sp=0xc000a25e60 pc=0xfa75b8
main.main()
/go/src/REDACTED/main.go:45 +0x22d fp=0xc000a25f50 sp=0xc000a25e70 pc=0xfa756d
runtime.main()
/usr/local/go/src/runtime/proc.go:272 +0x28b fp=0xc000a25fe0 sp=0xc000a25f50 pc=0x43d4eb
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000a25fe8 sp=0xc000a25fe0 pc=0x4798a1
Code evidences
Stop() sending to the stop channel:
Client initialization skipping the consumer initialization:
Reproduction Code
package main
import (
"context"
ddtracer "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
"os"
)
func main() {
os.Setenv("DD_REMOTE_CONFIGURATION_ENABLED","false")
// default context
ctx, cancel := context.WithCancel(context.Background())
// tracer
ddtracer.Start(ddtracer.WithRuntimeMetrics())
defer func() {
cancel()
ddtracer.Stop()
}()
// do something with ctx
// myAwesomeFn(ctx)
// main goroutine leaks after the deferred function is invoked,
// if we set manually the env `DD_REMOTE_CONFIGURATION_ENABLED=true`, the routine finishes as expected
}Error Logs
No response
Go Env Output
No response
Tracer Version(s)
1.73.1
Go Version(s)
go version go1.24.2 darwin/arm64
Bug Report
We are getting a deadlock on the
main.maingoroutine when calling the methodtracer.Stop(), consequently the app is hanging and not finishing properly.Looking at the implementation, the
internal/remoteconfig/remoteconfig.go#Start:191client is being initialized before evaluating if the value of the envDD_REMOTE_CONFIGURATION_ENABLEDis true|false, which causes the internal channel consumer to not be created (but the client was initialized).Then, when the
tracer.Stopmethod is called, it internally calls theinternal/remoteconfig/remoteconfig.go#Stop:226function, which checks if the client was initialized (which indeed was) and then sends a message to the internalstopchannel (that has no consumers), leading the main goroutine to be stuck forever.This issue occurs when instrumenting an application with serverless-init, which seems to force the value of the env to be false by default.
The suggested fix would be to move the check on line
remoteconfig.go:191beforeclient, err := newClient(config)is invoked.Goroutine dump
Code evidences
Stop() sending to the
stopchannel:Client initialization skipping the consumer initialization:
Reproduction Code
Error Logs
No response
Go Env Output
No response