Skip to content

Alternative approach to fixing flaky DSM Kafka tests#7211

Merged
andrewlock merged 3 commits into
masterfrom
andrew/ci/retry-kafka-flake
Jul 14, 2025
Merged

Alternative approach to fixing flaky DSM Kafka tests#7211
andrewlock merged 3 commits into
masterfrom
andrew/ci/retry-kafka-flake

Conversation

@andrewlock

@andrewlock andrewlock commented Jul 11, 2025

Copy link
Copy Markdown
Member

Summary of changes

Fixes flake in DSM tests by using unique but stable topic names

Supersedes #7209

Reason for change

Saw some flake in a test recently: 10:19:53 [DBG] Unhandled exception. Confluent.Kafka.ProduceException2[System.String,System.String]: Local: Unknown topic`

What's more, the creation logs are kind of strange.

10:19:53 [DBG]  Creating topics...
10:19:53 [DBG]  Trying to delete topic data-streams-1...
10:19:53 [DBG]  Trying to create topic data-streams-1...
10:19:53 [DBG]  Topic already exists
10:19:53 [DBG]  Trying to delete topic data-streams-2...
10:19:53 [DBG]  Trying to delete topic data-streams-2...
10:19:53 [DBG]  Topic did not exist, skipping
10:19:53 [DBG]  Trying to create topic data-streams-2...
10:19:53 [DBG]  Topic created
10:19:53 [DBG]  Trying to delete topic data-streams-3...
10:19:53 [DBG]  Trying to create topic data-streams-3...
10:19:53 [DBG]  Topic created
10:19:53 [DBG]  Finished creating topics: 0:00:02.1158037

I can't really understand that sequence of logs, but deletes don't happen atomically, so not entirely surprised by weirdness.

#7209 tried to work around it by using random topic names, but the checkpoints are dependent on the topic names. This approach uses stable, but unique, values

Implementation details

Pass the topic name suffix to use to the sample from the test. This is the approach we use in the existing non-DSM KafkaTests to avoid flake.

Test coverage

The existing coverage is sufficient, but requires updating all the snapshots.

Other details

Supersedes #7209

@andrewlock
andrewlock requested review from a team as code owners July 11, 2025 14:37
@andrewlock andrewlock added area:tests unit tests, integration tests type:flake-fix labels Jul 11, 2025
@andrewlock andrewlock changed the title Add flaky attribute to DSM kafka tests Alternative approach to fixing flaky DSM Kafka tests Jul 11, 2025
@dd-trace-dotnet-ci-bot

dd-trace-dotnet-ci-bot Bot commented Jul 11, 2025

Copy link
Copy Markdown

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7211) - mean (72ms)  : 71, 73
     .   : milestone, 72,
    master - mean (72ms)  : 71, 73
     .   : milestone, 72,

    section Baseline
    This PR (7211) - mean (68ms)  : 66, 71
     .   : milestone, 68,
    master - mean (69ms)  : 66, 71
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (7211) - mean (1,019ms)  : 992, 1046
     .   : milestone, 1019,
    master - mean (1,022ms)  : 999, 1046
     .   : milestone, 1022,

Loading
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7211) - mean (105ms)  : 104, 106
     .   : milestone, 105,
    master - mean (105ms)  : 104, 106
     .   : milestone, 105,

    section Baseline
    This PR (7211) - mean (104ms)  : 102, 107
     .   : milestone, 104,
    master - mean (104ms)  : 101, 106
     .   : milestone, 104,

    section CallTarget+Inlining+NGEN
    This PR (7211) - mean (699ms)  : 682, 716
     .   : milestone, 699,
    master - mean (703ms)  : 689, 718
     .   : milestone, 703,

Loading
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7211) - mean (92ms)  : 91, 93
     .   : milestone, 92,
    master - mean (93ms)  : 91, 94
     .   : milestone, 93,

    section Baseline
    This PR (7211) - mean (92ms)  : 90, 94
     .   : milestone, 92,
    master - mean (92ms)  : 90, 94
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (7211) - mean (661ms)  : 634, 688
     .   : milestone, 661,
    master - mean (667ms)  : 644, 689
     .   : milestone, 667,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7211) - mean (196ms)  : 190, 202
     .   : milestone, 196,
    master - mean (198ms)  : 192, 203
     .   : milestone, 198,

    section Baseline
    This PR (7211) - mean (193ms)  : 186, 200
     .   : milestone, 193,
    master - mean (195ms)  : 188, 203
     .   : milestone, 195,

    section CallTarget+Inlining+NGEN
    This PR (7211) - mean (1,140ms)  : 1113, 1167
     .   : milestone, 1140,
    master - mean (1,146ms)  : 1110, 1182
     .   : milestone, 1146,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7211) - mean (275ms)  : 268, 282
     .   : milestone, 275,
    master - mean (275ms)  : 267, 283
     .   : milestone, 275,

    section Baseline
    This PR (7211) - mean (275ms)  : 267, 283
     .   : milestone, 275,
    master - mean (278ms)  : 271, 285
     .   : milestone, 278,

    section CallTarget+Inlining+NGEN
    This PR (7211) - mean (904ms)  : 878, 930
     .   : milestone, 904,
    master - mean (908ms)  : 872, 945
     .   : milestone, 908,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7211) - mean (268ms)  : 261, 275
     .   : milestone, 268,
    master - mean (268ms)  : 263, 273
     .   : milestone, 268,

    section Baseline
    This PR (7211) - mean (268ms)  : 263, 274
     .   : milestone, 268,
    master - mean (270ms)  : 263, 277
     .   : milestone, 270,

    section CallTarget+Inlining+NGEN
    This PR (7211) - mean (889ms)  : 859, 920
     .   : milestone, 889,
    master - mean (900ms)  : 872, 927
     .   : milestone, 900,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7211) - mean (271ms)  : 265, 277
     .   : milestone, 271,
    master - mean (272ms)  : 262, 282
     .   : milestone, 272,

    section Baseline
    This PR (7211) - mean (269ms)  : 263, 275
     .   : milestone, 269,
    master - mean (268ms)  : 260, 276
     .   : milestone, 268,

    section CallTarget+Inlining+NGEN
    This PR (7211) - mean (797ms)  : 771, 822
     .   : milestone, 797,
    master - mean (809ms)  : 784, 833
     .   : milestone, 809,

Loading

@pr-commenter

pr-commenter Bot commented Jul 11, 2025

Copy link
Copy Markdown

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmarks for #7211 compared to master:

  • 3 benchmarks are slower, with geometric mean 1.194
  • 40 benchmarks have fewer allocations
  • 7 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7211

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑netcoreapp3.1 5.75 KB 5.7 KB -51 B -0.89%
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑net6.0 5.58 KB 5.48 KB -99 B -1.78%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 11.2μs 59.7ns 322ns 0 0 0 5.58 KB
master StartStopWithChild netcoreapp3.1 14.1μs 67.5ns 286ns 0 0 0 5.75 KB
master StartStopWithChild net472 22.1μs 119ns 651ns 1.03 0.411 0.103 6.09 KB
#7211 StartStopWithChild net6.0 10.2μs 56ns 341ns 0 0 0 5.48 KB
#7211 StartStopWithChild netcoreapp3.1 13.8μs 68.2ns 289ns 0 0 0 5.7 KB
#7211 StartStopWithChild net472 22μs 115ns 585ns 0.949 0.211 0 6.06 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7211

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces‑net472 3.33 KB 3.31 KB -23 B -0.69%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 927μs 26.6ns 103ns 0 0 0 2.71 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 1.04ms 404ns 1.56μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 1.25ms 357ns 1.38μs 0 0 0 3.33 KB
#7211 WriteAndFlushEnrichedTraces net6.0 941μs 76.2ns 285ns 0 0 0 2.71 KB
#7211 WriteAndFlushEnrichedTraces netcoreapp3.1 1.04ms 452ns 1.75μs 0 0 0 2.7 KB
#7211 WriteAndFlushEnrichedTraces net472 1.2ms 158ns 610ns 0 0 0 3.31 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7211

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net472 236.35 KB 239.66 KB 3.3 KB 1.40%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net472 239.87 KB 243.17 KB 3.3 KB 1.38%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 331μs 1.76μs 8.42μs 0 0 0 197.06 KB
master AllCycleSimpleBody netcoreapp3.1 510μs 1.45μs 5.61μs 0 0 0 204.77 KB
master AllCycleSimpleBody net472 436μs 119ns 460ns 36.6 2.16 0 236.35 KB
master AllCycleMoreComplexBody net6.0 338μs 1.76μs 8.8μs 0 0 0 200.56 KB
master AllCycleMoreComplexBody netcoreapp3.1 495μs 987ns 3.56μs 0 0 0 208.18 KB
master AllCycleMoreComplexBody net472 446μs 106ns 412ns 36.6 2.16 0 239.87 KB
master ObjectExtractorSimpleBody net6.0 311ns 1.77ns 12.3ns 0 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 409ns 1.93ns 8.2ns 0 0 0 272 B
master ObjectExtractorSimpleBody net472 303ns 0.175ns 0.676ns 0.0442 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 6.52μs 29.5ns 110ns 0 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 7.76μs 36.2ns 140ns 0 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 6.66μs 0.89ns 3.33ns 0.599 0 0 3.8 KB
#7211 AllCycleSimpleBody net6.0 329μs 1.43μs 5.53μs 0 0 0 197.61 KB
#7211 AllCycleSimpleBody netcoreapp3.1 493μs 927ns 3.59μs 0 0 0 205.37 KB
#7211 AllCycleSimpleBody net472 443μs 194ns 751ns 37.9 2.23 0 239.66 KB
#7211 AllCycleMoreComplexBody net6.0 342μs 1.33μs 5.14μs 0 0 0 201.12 KB
#7211 AllCycleMoreComplexBody netcoreapp3.1 519μs 2.23μs 8.65μs 0 0 0 208.78 KB
#7211 AllCycleMoreComplexBody net472 451μs 68.6ns 266ns 37.9 2.23 0 243.17 KB
#7211 ObjectExtractorSimpleBody net6.0 314ns 1.54ns 6.34ns 0 0 0 280 B
#7211 ObjectExtractorSimpleBody netcoreapp3.1 411ns 1.19ns 4.62ns 0 0 0 272 B
#7211 ObjectExtractorSimpleBody net472 309ns 0.0908ns 0.352ns 0.0436 0 0 281 B
#7211 ObjectExtractorMoreComplexBody net6.0 6.26μs 30.9ns 131ns 0 0 0 3.78 KB
#7211 ObjectExtractorMoreComplexBody netcoreapp3.1 7.66μs 38.8ns 174ns 0 0 0 3.69 KB
#7211 ObjectExtractorMoreComplexBody net472 6.78μs 1.74ns 6.29ns 0.577 0 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7211

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs‑net6.0 2.16 KB 2.15 KB -11 B -0.51%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 73.3μs 263ns 984ns 0 0 0 32.41 KB
master EncodeArgs netcoreapp3.1 95.7μs 36.2ns 130ns 0 0 0 32.4 KB
master EncodeArgs net472 107μs 17.6ns 65.8ns 4.82 0 0 32.51 KB
master EncodeLegacyArgs net6.0 143μs 122ns 472ns 0 0 0 2.16 KB
master EncodeLegacyArgs netcoreapp3.1 197μs 42.5ns 147ns 0 0 0 2.14 KB
master EncodeLegacyArgs net472 261μs 66.8ns 250ns 0 0 0 2.16 KB
#7211 EncodeArgs net6.0 76.6μs 275ns 1.03μs 0 0 0 32.4 KB
#7211 EncodeArgs netcoreapp3.1 95.2μs 331ns 1.28μs 0 0 0 32.4 KB
#7211 EncodeArgs net472 110μs 36.7ns 142ns 4.96 0 0 32.51 KB
#7211 EncodeLegacyArgs net6.0 141μs 44.5ns 172ns 0 0 0 2.15 KB
#7211 EncodeLegacyArgs netcoreapp3.1 196μs 211ns 819ns 0 0 0 2.14 KB
#7211 EncodeLegacyArgs net472 263μs 83.8ns 313ns 0 0 0 2.16 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 271μs 133ns 499ns 0 0 0 4.55 KB
master RunWafRealisticBenchmark netcoreapp3.1 294μs 264ns 989ns 0 0 0 4.48 KB
master RunWafRealisticBenchmark net472 307μs 38.1ns 147ns 0 0 0 4.66 KB
master RunWafRealisticBenchmarkWithAttack net6.0 181μs 80.1ns 300ns 0 0 0 2.24 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 198μs 104ns 404ns 0 0 0 2.22 KB
master RunWafRealisticBenchmarkWithAttack net472 207μs 45.3ns 169ns 0 0 0 2.28 KB
#7211 RunWafRealisticBenchmark net6.0 271μs 53.5ns 207ns 0 0 0 4.55 KB
#7211 RunWafRealisticBenchmark netcoreapp3.1 292μs 101ns 391ns 0 0 0 4.48 KB
#7211 RunWafRealisticBenchmark net472 310μs 55.5ns 208ns 0 0 0 4.65 KB
#7211 RunWafRealisticBenchmarkWithAttack net6.0 181μs 58.5ns 227ns 0 0 0 2.24 KB
#7211 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 193μs 40.6ns 141ns 0 0 0 2.22 KB
#7211 RunWafRealisticBenchmarkWithAttack net472 209μs 42.8ns 166ns 0 0 0 2.29 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 61.1μs 29ns 104ns 0 0 0 14.53 KB
master SendRequest netcoreapp3.1 69.9μs 118ns 440ns 0 0 0 17.42 KB
master SendRequest net472 0.0171ns 0.00147ns 0.00551ns 0 0 0 0 b
#7211 SendRequest net6.0 61.1μs 271ns 1.05μs 0 0 0 14.52 KB
#7211 SendRequest netcoreapp3.1 72.1μs 355ns 1.5μs 0 0 0 17.42 KB
#7211 SendRequest net472 0.0214ns 0.0021ns 0.00815ns 0 0 0 0 b
Benchmarks.Trace.CharSliceBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
#7211 OriginalCharSlice net6.0 1.93ms 4.37μs 16.9μs 0 0 0 640.01 KB
#7211 OriginalCharSlice netcoreapp3.1 2.16ms 2.01μs 7.53μs 0 0 0 640 KB
#7211 OriginalCharSlice net472 2.67ms 1.57μs 5.67μs 100 0 0 641.95 KB
#7211 OptimizedCharSlice net6.0 1.47ms 109ns 420ns 0 0 0 6 B
#7211 OptimizedCharSlice netcoreapp3.1 1.7ms 156ns 605ns 0 0 0 1 B
#7211 OptimizedCharSlice net472 2ms 157ns 588ns 0 0 0 73 B
#7211 OptimizedCharSliceWithPool net6.0 863μs 27.8ns 108ns 0 0 0 2 B
#7211 OptimizedCharSliceWithPool netcoreapp3.1 839μs 32.7ns 127ns 0 0 0 0 b
#7211 OptimizedCharSliceWithPool net472 1.13ms 55.5ns 207ns 0 0 0 47 B
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Slower ⚠️ More allocations ⚠️

Slower ⚠️ in #7211

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 1.237 631,437.50 780,787.05
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 1.140 645,405.00 735,830.47

More allocations ⚠️ in #7211

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net472 55.75 KB 56.22 KB 465 B 0.83%
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 41.73 KB 42.03 KB 300 B 0.72%
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 41.89 KB 42.11 KB 226 B 0.54%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 645μs 512ns 1.98μs 0 0 0 41.73 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 631μs 1.67μs 6.67μs 0 0 0 41.89 KB
master WriteAndFlushEnrichedTraces net472 923μs 1.93μs 7.2μs 4.46 0 0 55.75 KB
#7211 WriteAndFlushEnrichedTraces net6.0 736μs 2.06μs 7.98μs 0 0 0 42.03 KB
#7211 WriteAndFlushEnrichedTraces netcoreapp3.1 781μs 1.73μs 6.69μs 0 0 0 42.11 KB
#7211 WriteAndFlushEnrichedTraces net472 923μs 2.95μs 11.4μs 8.33 0 0 56.22 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7211

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery‑net6.0 1.03 KB 1.02 KB -8 B -0.78%
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery‑netcoreapp3.1 1.02 KB 1.02 KB -8 B -0.78%
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery‑net472 995 B 987 B -8 B -0.80%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.99μs 10.2ns 46.8ns 0 0 0 1.03 KB
master ExecuteNonQuery netcoreapp3.1 2.53μs 3.43ns 13.3ns 0 0 0 1.02 KB
master ExecuteNonQuery net472 2.7μs 2.41ns 9.34ns 0.147 0.0134 0 995 B
#7211 ExecuteNonQuery net6.0 1.88μs 1ns 3.89ns 0 0 0 1.02 KB
#7211 ExecuteNonQuery netcoreapp3.1 2.82μs 10.1ns 39.2ns 0 0 0 1.02 KB
#7211 ExecuteNonQuery net472 2.76μs 2.65ns 10.3ns 0.153 0.0139 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7211

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync‑net472 1.11 KB 1.1 KB -8 B -0.72%
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync‑netcoreapp3.1 1.09 KB 1.08 KB -8 B -0.74%
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net472 1.05 KB 1.04 KB -8 B -0.76%
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net6.0 1.04 KB 1.03 KB -8 B -0.77%
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑netcoreapp3.1 1.04 KB 1.03 KB -8 B -0.77%
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync‑net6.0 1.02 KB 1.01 KB -8 B -0.79%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.83μs 5.85ns 22.7ns 0 0 0 1.04 KB
master CallElasticsearch netcoreapp3.1 2.3μs 11.8ns 56.6ns 0 0 0 1.04 KB
master CallElasticsearch net472 3.55μs 3.05ns 11.8ns 0.159 0 0 1.05 KB
master CallElasticsearchAsync net6.0 1.81μs 3.89ns 14.5ns 0 0 0 1.02 KB
master CallElasticsearchAsync netcoreapp3.1 2.35μs 7.26ns 28.1ns 0 0 0 1.09 KB
master CallElasticsearchAsync net472 3.84μs 3.19ns 12.4ns 0.169 0 0 1.11 KB
#7211 CallElasticsearch net6.0 1.87μs 1.28ns 4.97ns 0 0 0 1.03 KB
#7211 CallElasticsearch netcoreapp3.1 2.37μs 4.69ns 18.1ns 0 0 0 1.03 KB
#7211 CallElasticsearch net472 3.66μs 1.44ns 5.59ns 0.165 0 0 1.04 KB
#7211 CallElasticsearchAsync net6.0 1.82μs 9.47ns 46.4ns 0 0 0 1.01 KB
#7211 CallElasticsearchAsync netcoreapp3.1 2.52μs 12.6ns 53.6ns 0 0 0 1.08 KB
#7211 CallElasticsearchAsync net472 3.8μs 2.41ns 9.32ns 0.171 0 0 1.1 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7211

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net6.0 960 B 952 B -8 B -0.83%
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑netcoreapp3.1 960 B 952 B -8 B -0.83%
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net472 923 B 915 B -8 B -0.87%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.78μs 9.27ns 43.5ns 0 0 0 960 B
master ExecuteAsync netcoreapp3.1 2.29μs 7.45ns 25.8ns 0 0 0 960 B
master ExecuteAsync net472 2.58μs 1.78ns 6.88ns 0.143 0 0 923 B
#7211 ExecuteAsync net6.0 1.87μs 2.68ns 10.4ns 0 0 0 952 B
#7211 ExecuteAsync netcoreapp3.1 2.32μs 8.55ns 33.1ns 0 0 0 952 B
#7211 ExecuteAsync net472 2.54μs 0.956ns 3.7ns 0.139 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 7.1μs 10.4ns 39ns 0 0 0 2.37 KB
master SendAsync netcoreapp3.1 8.69μs 14.1ns 54.6ns 0 0 0 2.9 KB
master SendAsync net472 12.5μs 10.3ns 38.5ns 0.498 0 0 3.19 KB
#7211 SendAsync net6.0 6.78μs 7.25ns 26.1ns 0 0 0 2.36 KB
#7211 SendAsync netcoreapp3.1 8.75μs 14.2ns 55.1ns 0 0 0 2.9 KB
#7211 SendAsync net472 12μs 4.64ns 16.7ns 0.479 0 0 3.18 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Slower ⚠️ More allocations ⚠️

Slower ⚠️ in #7211

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 1.208 413,700.00 499,800.00

More allocations ⚠️ in #7211

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0 259.96 KB 277.3 KB 17.34 KB 6.67%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net6.0 43.83 KB 44.07 KB 240 B 0.55%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 44.6μs 212ns 1.6μs 0 0 0 43.83 KB
master StringConcatBenchmark netcoreapp3.1 47.2μs 224ns 838ns 0 0 0 42.64 KB
master StringConcatBenchmark net472 56.8μs 259ns 968ns 0 0 0 57.34 KB
master StringConcatAspectBenchmark net6.0 458μs 1.08μs 3.89μs 0 0 0 259.96 KB
master StringConcatAspectBenchmark netcoreapp3.1 447μs 6.44μs 63.7μs 0 0 0 274.93 KB
master StringConcatAspectBenchmark net472 410μs 2.07μs 9.27μs 0 0 0 286.72 KB
#7211 StringConcatBenchmark net6.0 41μs 156ns 541ns 0 0 0 44.07 KB
#7211 StringConcatBenchmark netcoreapp3.1 49.4μs 252ns 1.53μs 0 0 0 42.7 KB
#7211 StringConcatBenchmark net472 59.5μs 254ns 949ns 0 0 0 57.34 KB
#7211 StringConcatAspectBenchmark net6.0 474μs 2.2μs 8.79μs 0 0 0 277.3 KB
#7211 StringConcatAspectBenchmark netcoreapp3.1 501μs 2.12μs 7.63μs 0 0 0 274.71 KB
#7211 StringConcatAspectBenchmark net472 403μs 1.93μs 8.2μs 0 0 0 286.72 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7211

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog‑net6.0 1.76 KB 1.7 KB -56 B -3.18%
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog‑netcoreapp3.1 1.76 KB 1.7 KB -56 B -3.18%
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog‑net472 1.69 KB 1.64 KB -56 B -3.31%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.63μs 2.6ns 10.1ns 0 0 0 1.76 KB
master EnrichedLog netcoreapp3.1 3.44μs 4.38ns 17ns 0 0 0 1.76 KB
master EnrichedLog net472 4.05μs 4.83ns 18.7ns 0.265 0 0 1.69 KB
#7211 EnrichedLog net6.0 2.56μs 5.02ns 18.8ns 0 0 0 1.7 KB
#7211 EnrichedLog netcoreapp3.1 3.49μs 17.4ns 71.7ns 0 0 0 1.7 KB
#7211 EnrichedLog net472 3.94μs 3.77ns 14.6ns 0.255 0 0 1.64 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7211

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Log4netBenchmark.EnrichedLog‑net6.0 4.37 KB 4.31 KB -56 B -1.28%
Benchmarks.Trace.Log4netBenchmark.EnrichedLog‑netcoreapp3.1 4.37 KB 4.31 KB -56 B -1.28%
Benchmarks.Trace.Log4netBenchmark.EnrichedLog‑net472 4.57 KB 4.51 KB -60 B -1.31%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 123μs 145ns 563ns 0 0 0 4.37 KB
master EnrichedLog netcoreapp3.1 126μs 330ns 1.24μs 0 0 0 4.37 KB
master EnrichedLog net472 167μs 161ns 603ns 0 0 0 4.57 KB
#7211 EnrichedLog net6.0 123μs 45.3ns 175ns 0 0 0 4.31 KB
#7211 EnrichedLog netcoreapp3.1 128μs 284ns 1.1μs 0 0 0 4.31 KB
#7211 EnrichedLog net472 165μs 115ns 431ns 0 0 0 4.51 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7211

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.NLogBenchmark.EnrichedLog‑net6.0 2.32 KB 2.26 KB -56 B -2.41%
Benchmarks.Trace.NLogBenchmark.EnrichedLog‑netcoreapp3.1 2.32 KB 2.26 KB -56 B -2.41%
Benchmarks.Trace.NLogBenchmark.EnrichedLog‑net472 2.14 KB 2.08 KB -56 B -2.62%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.94μs 19.3ns 74.6ns 0 0 0 2.32 KB
master EnrichedLog netcoreapp3.1 6.77μs 22.2ns 79.9ns 0 0 0 2.32 KB
master EnrichedLog net472 7.45μs 8.04ns 31.1ns 0.335 0 0 2.14 KB
#7211 EnrichedLog net6.0 4.92μs 15.8ns 61.4ns 0 0 0 2.26 KB
#7211 EnrichedLog netcoreapp3.1 6.83μs 5.51ns 21.4ns 0 0 0 2.26 KB
#7211 EnrichedLog net472 7.46μs 6.83ns 26.5ns 0.297 0 0 2.08 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7211

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.RedisBenchmark.SendReceive‑net472 1.21 KB 1.2 KB -8 B -0.66%
Benchmarks.Trace.RedisBenchmark.SendReceive‑net6.0 1.21 KB 1.2 KB -8 B -0.66%
Benchmarks.Trace.RedisBenchmark.SendReceive‑netcoreapp3.1 1.21 KB 1.2 KB -8 B -0.66%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 2.04μs 10.5ns 49.4ns 0 0 0 1.21 KB
master SendReceive netcoreapp3.1 2.53μs 11.8ns 47.4ns 0 0 0 1.21 KB
master SendReceive net472 3.28μs 2.71ns 10.5ns 0.178 0 0 1.21 KB
#7211 SendReceive net6.0 2.07μs 5.9ns 21.3ns 0 0 0 1.2 KB
#7211 SendReceive netcoreapp3.1 2.75μs 11.2ns 43.3ns 0 0 0 1.2 KB
#7211 SendReceive net472 3.2μs 7.23ns 28ns 0.191 0 0 1.2 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7211

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.SerilogBenchmark.EnrichedLog‑net472 2.08 KB 2.03 KB -56 B -2.69%
Benchmarks.Trace.SerilogBenchmark.EnrichedLog‑netcoreapp3.1 1.69 KB 1.63 KB -56 B -3.32%
Benchmarks.Trace.SerilogBenchmark.EnrichedLog‑net6.0 1.64 KB 1.58 KB -56 B -3.41%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.18μs 0.92ns 3.44ns 0 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 5.64μs 23ns 89.3ns 0 0 0 1.69 KB
master EnrichedLog net472 6.67μs 7.28ns 27.3ns 0.298 0 0 2.08 KB
#7211 EnrichedLog net6.0 4.08μs 4.31ns 15.5ns 0 0 0 1.58 KB
#7211 EnrichedLog netcoreapp3.1 5.62μs 18.6ns 71.9ns 0 0 0 1.63 KB
#7211 EnrichedLog net472 6.61μs 6ns 23.2ns 0.299 0 0 2.03 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7211

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 704 B 696 B -8 B -1.14%
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1 704 B 696 B -8 B -1.14%
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472 666 B 658 B -8 B -1.20%
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472 586 B 578 B -8 B -1.37%
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 584 B 576 B -8 B -1.37%
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1 584 B 576 B -8 B -1.37%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 751ns 3.97ns 22.4ns 0 0 0 584 B
master StartFinishSpan netcoreapp3.1 953ns 4.42ns 17.7ns 0 0 0 584 B
master StartFinishSpan net472 914ns 0.809ns 3.13ns 0.0912 0 0 586 B
master StartFinishScope net6.0 918ns 0.479ns 1.73ns 0 0 0 704 B
master StartFinishScope netcoreapp3.1 1.15μs 6.24ns 34.2ns 0 0 0 704 B
master StartFinishScope net472 1.09μs 0.174ns 0.652ns 0.104 0 0 666 B
#7211 StartFinishSpan net6.0 752ns 3.54ns 13.2ns 0 0 0 576 B
#7211 StartFinishSpan netcoreapp3.1 938ns 4.84ns 22.2ns 0 0 0 576 B
#7211 StartFinishSpan net472 898ns 0.154ns 0.577ns 0.0902 0 0 578 B
#7211 StartFinishScope net6.0 899ns 4.42ns 18.8ns 0 0 0 696 B
#7211 StartFinishScope netcoreapp3.1 1.19μs 5.87ns 24.9ns 0 0 0 696 B
#7211 StartFinishScope net472 1.08μs 0.718ns 2.78ns 0.103 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7211

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 704 B 696 B -8 B -1.14%
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑netcoreapp3.1 704 B 696 B -8 B -1.14%
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net472 666 B 658 B -8 B -1.20%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 1.02μs 5.36ns 26.8ns 0 0 0 704 B
master RunOnMethodBegin netcoreapp3.1 1.38μs 2.28ns 8.82ns 0 0 0 704 B
master RunOnMethodBegin net472 1.36μs 0.141ns 0.544ns 0.102 0 0 666 B
#7211 RunOnMethodBegin net6.0 1.08μs 5.25ns 21.6ns 0 0 0 696 B
#7211 RunOnMethodBegin netcoreapp3.1 1.49μs 7.13ns 29.4ns 0 0 0 696 B
#7211 RunOnMethodBegin net472 1.39μs 1.46ns 5.47ns 0.0983 0 0 658 B

@github-actions

Copy link
Copy Markdown
Contributor

Snapshots difference summary

The following differences have been observed in committed snapshots. It is meant to help the reviewer.
The diff is simplistic, so please check some files anyway while we improve it.

2 occurrences of :

-            topic:data-streams-batch-processing-1,
+            topic:data-streams-batch-processing-1-HandlesBatchProcessing,
[...]
-          Hash: 4913488602916801083,
+          Hash: 9751442692034380744,

2 occurrences of :

-            topic:data-streams-batch-processing-1,
+            topic:data-streams-batch-processing-1-HandlesBatchProcessing,
[...]
-          Hash: 9315422718960432191,
-          ParentHash: 4913488602916801083,
+          Hash: 11254684039468539015,
+          ParentHash: 9751442692034380744,

2 occurrences of :

-            topic:data-streams-batch-processing-2,
+            topic:data-streams-batch-processing-2-HandlesBatchProcessing,
[...]
-          Hash: 9937680799879052874,
-          ParentHash: 9315422718960432191,
+          Hash: 13798769173313021740,
+          ParentHash: 11254684039468539015,

2 occurrences of :

-            topic:data-streams-batch-processing-2,
+            topic:data-streams-batch-processing-2-HandlesBatchProcessing,
[...]
-          Hash: 8667739576086808638,
-          ParentHash: 9937680799879052874,
+          Hash: 13395884887236079874,
+          ParentHash: 13798769173313021740,

3 occurrences of :

-            topic:data-streams-batch-processing-1,
+            topic:data-streams-batch-processing-1-HandlesBatchProcessing,

2 occurrences of :

-            topic:data-streams-batch-processing-2,
+            topic:data-streams-batch-processing-2-HandlesBatchProcessing,

1 occurrences of :

-            topic:data-streams-batch-processing-2,
+            topic:data-streams-batch-processing-2-HandlesBatchProcessing,
[...]
-            topic:data-streams-batch-processing-1,
+            topic:data-streams-batch-processing-1-HandlesBatchProcessing,
[...]
-            topic:data-streams-batch-processing-2,
+            topic:data-streams-batch-processing-2-HandlesBatchProcessing,
[...]
-            topic:data-streams-batch-processing-1,
+            topic:data-streams-batch-processing-1-HandlesBatchProcessing,
[...]
-            topic:data-streams-batch-processing-2,
+            topic:data-streams-batch-processing-2-HandlesBatchProcessing,
[...]
-            topic:data-streams-batch-processing-1,
+            topic:data-streams-batch-processing-1-HandlesBatchProcessing,
[...]
-            topic:data-streams-batch-processing-2,
+            topic:data-streams-batch-processing-2-HandlesBatchProcessing,

@andrewlock
andrewlock merged commit 3feefdb into master Jul 14, 2025
150 checks passed
@andrewlock
andrewlock deleted the andrew/ci/retry-kafka-flake branch July 14, 2025 11:17
@github-actions github-actions Bot added this to the vNext-v3 milestone Jul 14, 2025
chojomok pushed a commit that referenced this pull request Jul 15, 2025
## Summary of changes

Fixes flake in DSM tests by using unique but stable topic names

> Supersedes #7209

## Reason for change

[Saw some flake in a test
recently](https://dev.azure.com/datadoghq/dd-trace-dotnet/_build/results?buildId=181163&view=logs&j=b7fc299e-b4c1-5ec0-1e5a-2167c3cc6270&t=2406fbf1-c048-53d0-bb5f-56f6b7adbd43):
`10:19:53 [DBG] Unhandled exception.
Confluent.Kafka.ProduceException`2[System.String,System.String]: Local:
Unknown topic`

What's more, the creation logs are kind of strange.

```
10:19:53 [DBG]  Creating topics...
10:19:53 [DBG]  Trying to delete topic data-streams-1...
10:19:53 [DBG]  Trying to create topic data-streams-1...
10:19:53 [DBG]  Topic already exists
10:19:53 [DBG]  Trying to delete topic data-streams-2...
10:19:53 [DBG]  Trying to delete topic data-streams-2...
10:19:53 [DBG]  Topic did not exist, skipping
10:19:53 [DBG]  Trying to create topic data-streams-2...
10:19:53 [DBG]  Topic created
10:19:53 [DBG]  Trying to delete topic data-streams-3...
10:19:53 [DBG]  Trying to create topic data-streams-3...
10:19:53 [DBG]  Topic created
10:19:53 [DBG]  Finished creating topics: 0:00:02.1158037
```

I can't really understand that sequence of logs, but deletes don't
happen atomically, so not entirely surprised by weirdness.

#7209 tried to work around it by using random topic names, but the
checkpoints are dependent on the topic names. This approach uses stable,
but unique, values

## Implementation details

Pass the topic name suffix to use to the sample from the test. This is
the approach we use in the existing non-DSM `KafkaTests` to avoid flake.

## Test coverage

The existing coverage is sufficient, but requires updating all the
snapshots.

## Other details

Supersedes #7209
NachoEchevarria added a commit that referenced this pull request May 20, 2026
## Summary of changes

Fix the `DataStreamsMonitoringKafkaTests.HandlesBatchProcessing` (and
related) flake by making the test sample wait for Kafka metadata to
propagate after topic creation, and by hardening the partition-picker in
the producer helper against a transient `0` partition count.

## Reason for change

The `HandlesBatchProcessing` test failed in CI with a Verify mismatch
where every `kafka_produce` backlog entry collapsed onto `partition:0`:

```
- partition:1, topic:data-streams-batch-processing-1-..., type:kafka_produce
- partition:1, topic:data-streams-batch-processing-2-..., type:kafka_produce
- partition:2, topic:data-streams-batch-processing-1-..., type:kafka_produce
- partition:2, topic:data-streams-batch-processing-2-..., type:kafka_produce
```

The four missing entries are all producer-side backlogs for partitions 1
and 2. Consumer-side `kafka_commit` backlogs are present for all
partitions, and the per-topic in/out DSM edges are present too — only
the producer collapsed onto partition 0.

**Direct confirmation in the sample's stdout** — every produce went to
partition `[[0]]`:

```
Produced message to: data-streams-batch-processing-1-HandlesBatchProcessing [[0]] @0
Produced message to: data-streams-batch-processing-1-HandlesBatchProcessing [[0]] @1
Produced message to: data-streams-batch-processing-1-HandlesBatchProcessing [[0]] @2
...
Produced message to: data-streams-batch-processing-2-HandlesBatchProcessing [[0]] @0
Produced message to: data-streams-batch-processing-2-HandlesBatchProcessing [[0]] @1
Produced message to: data-streams-batch-processing-2-HandlesBatchProcessing [[0]] @2
```

…instead of the expected `[[0]]`, `[[1]]`, `[[2]]` distribution. Topic
creation took 4.2s (`Finished creating topics: 0:00:04.2190968`) and
producing started ~1s later — well within the metadata-propagation
window where `AdminClient.GetMetadata` can return the topic with zero
partitions.

**Root cause** is a race in the test sample:

1. `TopicHelpers.TryCreateTopic` returns as soon as Kafka acks the
`CreateTopics` request, but broker metadata propagation can lag.
2. `Producer.GetPartition` calls `AdminClient.GetMetadata(topic, 5s)` to
learn the partition count. If metadata hasn't propagated yet, the topic
comes back with zero partitions and `GetTopicPartitionCount` returns
`0`.
3. That `0` gets cached in a static `ConcurrentDictionary<string,int>`
for the lifetime of the sample process via `GetOrAdd`.
4. With `numPartitions == 0`, the `partition >= numPartitions` guard is
always true, so every subsequent produce is pinned to partition 0 — for
the entire run.

This is timing-dependent (flaky, not always failing), and once it loses
the race the whole run is doomed.

This is the same family of flake addressed in #7211 ("Alternative
approach to fixing flaky DSM Kafka tests"), but the partition-count race
in the sample wasn't covered there.

## Implementation details

Two changes, both in the test sample only — no tracer code touched.

**`Samples.Kafka/TopicHelpers.cs` — fix the root cause.** After
`CreateTopicsAsync` succeeds (or returns `TopicAlreadyExists`), poll
`GetMetadata` until the topic is visible with the expected partition
count, with a 30s bounded timeout. If metadata never propagates, throw a
descriptive exception instead of silently returning. The original
throw-on-exhausted-retries behavior is preserved.

**`Samples.Kafka/Producer.cs` — defense in depth.** `GetPartition` no
longer caches a `0` partition count. If `GetTopicPartitionCount` returns
0 (shouldn't happen anymore after the TopicHelpers fix, but in case
anyone calls this without going through `TryCreateTopic`), we fall back
to partition 0 for that single call and re-query on the next produce
instead of pinning the whole run.

## Test coverage

Existing `DataStreamsMonitoringKafkaTests` cover the scenario; the
failure mode this PR fixes is exactly the one observed in the failing
build. No new tests added — this is a flake fix in a sample app, not a
behavior change in the tracer.

## Other details

- Test sample code only; no tracer / production code changes.
- `Samples.Kafka/TopicHelpers.cs` is used by every Kafka-producing
sample test, so it broadly hardens that family of tests against the
metadata-propagation race.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:tests unit tests, integration tests type:flake-fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants