Skip to content

fix(ddtrace/tracer): add finished check on NoDebugStack and WithError code branches#3441

Merged
darccio merged 5 commits into
mainfrom
dario.castane/circuit-breaker-span-finish
Apr 16, 2025
Merged

fix(ddtrace/tracer): add finished check on NoDebugStack and WithError code branches#3441
darccio merged 5 commits into
mainfrom
dario.castane/circuit-breaker-span-finish

Conversation

@darccio

@darccio darccio commented Apr 16, 2025

Copy link
Copy Markdown
Member

What does this PR do?

Follow-up from #3435 & #3427. Adds a check on span.finished when applying NoDebugStack.

Motivation

Adds an extra layer of resilience on high concurrency scenarios.

Reviewer's Checklist

  • Changed code has unit tests for its functionality at or near 100% coverage.
  • System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
  • There is a benchmark for any new code, or changes to existing code.
  • If this interacts with the agent in a new way, a system test has been added.
  • New code is free of linting errors. You can check this by running golangci-lint run locally.
  • Add an appropriate team label so this PR gets put in the right place for the release notes.
  • Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.

Unsure? Have a question? Request a review!

@felixge

felixge commented Apr 16, 2025

Copy link
Copy Markdown
Member

Adds an extra layer of resilience on high concurrency scenarios.

Can you give an example? By releasing the lock after checking for finished, the value of finished might be modified by another goroutine calling Finish at the same time. So I don't see how this could provide additional resilience.

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Apr 16, 2025

Copy link
Copy Markdown

Datadog Report

Branch report: dario.castane/circuit-breaker-span-finish
Commit report: c94b1cc
Test service: dd-trace-go

✅ 0 Failed, 4481 Passed, 64 Skipped, 3m 48.36s Total Time

@pr-commenter

pr-commenter Bot commented Apr 16, 2025

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2025-04-16 07:06:28

Comparing candidate commit af71857 in PR branch dario.castane/circuit-breaker-span-finish with baseline commit 23a4bfe in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 55 metrics, 1 unstable metrics.

@darccio
darccio marked this pull request as ready for review April 16, 2025 09:51
@darccio
darccio requested a review from a team as a code owner April 16, 2025 09:51
@darccio darccio changed the title fix(ddtrace/tracer): use a read lock to circuit break on span.finished fix(ddtrace/tracer): add span.finished check in NoDebugStack code branch Apr 16, 2025
Comment thread ddtrace/tracer/span.go Outdated
@darccio
darccio force-pushed the dario.castane/circuit-breaker-span-finish branch from e9e858e to 590a89a Compare April 16, 2025 10:04
@darccio darccio changed the title fix(ddtrace/tracer): add span.finished check in NoDebugStack code branch fix(ddtrace/tracer): add finished check on NoDebugStack and WithError code branches Apr 16, 2025
@darccio
darccio enabled auto-merge (squash) April 16, 2025 10:12
@darccio
darccio merged commit f3d284b into main Apr 16, 2025
@darccio
darccio deleted the dario.castane/circuit-breaker-span-finish branch April 16, 2025 10:30
Comment thread ddtrace/tracer/span.go
s.RUnlock()
return
}
s.RUnlock()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems incorrect. Consider two goroutines calling finish concurrency:

G1: Checks `s.finished` and finds it to be false and continues
G2: Checks `s.finished` and finds it to be false and continues
G1: Manages to call `s.finish` setting `finished` to true
Worker: Starts encoding the span
G2: Calls `delete` on `s.meta`
💣

There is no weak version of Invariant I that will do here.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@darccio We should reduce the number of calls to acquiring lock here to one. It will make things to easier to reason about. This is my current approach with the refactoring work.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right. Fixed in #3446.

darccio added a commit that referenced this pull request Apr 16, 2025
@darccio darccio added the v2.0 label Jun 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants