Skip to content

fix(ci-visibility): harden review edge cases#4735

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
mainfrom
feat/fix-civisibility-review-findings
May 7, 2026
Merged

fix(ci-visibility): harden review edge cases#4735
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
mainfrom
feat/fix-civisibility-review-findings

Conversation

@tonyredondo

@tonyredondo tonyredondo commented May 6, 2026

Copy link
Copy Markdown
Member

What does this PR do?

Fixes three CI Visibility review findings around logs, settings bootstrap, and shallow Git recovery.

This PR makes the CI Visibility logs writer safe when log writes, flushes, payload rotation, and shutdown overlap. The package-level logs state is now synchronized, writer payload rotation is guarded by a writer mutex, uploads are reserved before goroutines start, and log writes accepted before shutdown are flushed exactly once. It also updates the stale msgpack comments for the JSON logs payload path.

It also hardens settings initialization so a nil settings response no longer panics. The bootstrap now logs nil settings responses separately from request errors, leaves settings at their zero value, and preserves the existing upload wait behavior. A test seam for client creation lets the nil-response paths be covered directly.

Finally, UnshallowGitRepository now treats a successful but quiet git fetch as success. The HEAD and upstream fetch fallbacks now run only after real command errors, avoiding the previous nil-error panic and unnecessary fallback behavior.

Motivation

Follow-up review found concurrency and nil-handling gaps that could make CI Visibility unstable under parallel log writes, unusual client responses, or quiet Git commands.

The highest-risk issue was in logs: the payload object had its own locking, but the writer swapped the payload pointer without a writer-level lock. Concurrent writes, flushes, and stop could race with rotation and lose accepted log entries.

The settings and Git fixes address defensive edge cases: nil settings responses should disable features safely, and Git commands that exit successfully do not need stdout to be considered successful.

Testing

  • go test ./internal/civisibility/integrations/logs
  • go test -race ./internal/civisibility/integrations/logs
  • go test ./internal/civisibility/integrations -run 'TestEnsureSettingsInitialization|TestUploadRepositoryChanges|TestSearchCommitsResponse'
  • go test ./internal/civisibility/utils -run 'TestUnshallowGitRepository'
  • go test ./internal/civisibility/...
  • go test -race ./internal/civisibility/...
  • go vet ./internal/civisibility/...

@tonyredondo
tonyredondo force-pushed the feat/fix-civisibility-review-findings branch from 9a3ddc0 to a3a2821 Compare May 6, 2026 18:31
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented May 6, 2026 •
edited by datadog-prod-us1-4 Bot
Loading

Copy link
Copy Markdown

Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 82.35%
• Overall Coverage: 61.64% (+0.21%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d92e058 | Docs | Datadog PR Page | Give us feedback!

@codecov

codecov Bot commented May 6, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.63636% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.34%. Comparing base (32680c8) to head (d92e058).

Files with missing lines Patch % Lines
...rnal/civisibility/integrations/logs/logs_writer.go 83.33% 5 Missing and 1 partial ⚠️
...civisibility/integrations/gotesting/reflections.go 80.00% 4 Missing and 1 partial ⚠️
...sibility/integrations/gotesting/instrumentation.go 80.00% 2 Missing and 1 partial ⚠️
internal/civisibility/integrations/logs/logs.go 81.25% 1 Missing and 2 partials ⚠️
internal/civisibility/utils/git.go 85.71% 0 Missing and 1 partial ⚠️
Additional details and impacted files
Files with missing lines Coverage Δ
...civisibility/integrations/civisibility_features.go 58.87% <100.00%> (ø)
...nal/civisibility/integrations/logs/logs_payload.go 88.23% <ø> (ø)
internal/civisibility/utils/git.go 61.80% <85.71%> (ø)
...sibility/integrations/gotesting/instrumentation.go 76.81% <80.00%> (ø)
internal/civisibility/integrations/logs/logs.go 89.36% <81.25%> (ø)
...civisibility/integrations/gotesting/reflections.go 59.31% <80.00%> (ø)
...rnal/civisibility/integrations/logs/logs_writer.go 87.09% <83.33%> (ø)

... and 444 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tonyredondo
tonyredondo force-pushed the feat/fix-civisibility-review-findings branch 2 times, most recently from 2450bfb to 0054f0b Compare May 6, 2026 19:01
@tonyredondo
tonyredondo force-pushed the feat/fix-civisibility-review-findings branch from 0054f0b to 1645ac3 Compare May 6, 2026 19:20
@pr-commenter

pr-commenter Bot commented May 6, 2026 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-05-06 22:15:11

Comparing candidate commit d92e058 in PR branch feat/fix-civisibility-review-findings with baseline commit 32680c8 in branch main.

Found 0 performance improvements and 4 performance regressions! Performance is the same for 238 metrics, 9 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:BenchmarkOTLPProtoSize/1000spans-25

  • 🟥 execution_time [+15.804µs; +16.992µs] or [+5.258%; +5.653%]

scenario:BenchmarkOTLPProtoSize/100spans-25

  • 🟥 execution_time [+1.749µs; +1.855µs] or [+5.851%; +6.204%]

scenario:BenchmarkOTLPProtoSize/10spans-25

  • 🟥 execution_time [+194.199ns; +208.401ns] or [+6.349%; +6.814%]

scenario:BenchmarkOTLPProtoSize/1span-25

  • 🟥 execution_time [+24.601ns; +26.939ns] or [+6.601%; +7.229%]

@tonyredondo
tonyredondo marked this pull request as ready for review May 7, 2026 09:48
@tonyredondo
tonyredondo requested a review from a team as a code owner May 7, 2026 09:48
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 42d74ee into main May 7, 2026
194 of 195 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the feat/fix-civisibility-review-findings branch May 7, 2026 13:11
@tonyredondo tonyredondo mentioned this pull request May 26, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@anmarchenko anmarchenko anmarchenko approved these changes

Assignees

No one assigned

Labels

mergequeue-status: done

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tonyredondo @anmarchenko