feat(data-pipeline)!: include reason for chunks dropped telemetry#1449
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1449 +/- ##
==========================================
+ Coverage 71.11% 71.19% +0.07%
==========================================
Files 419 419
Lines 67067 67165 +98
==========================================
+ Hits 47694 47817 +123
+ Misses 19373 19348 -25
🚀 New features to boost your workflow:
|
BenchmarksComparisonBenchmark execution time: 2026-01-22 20:03:31 Comparing candidate commit c188e8b in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 57 metrics, 2 unstable metrics. CandidateCandidate benchmark detailsGroup 1
Group 2
Group 3
Group 4
Group 5
Group 6
Group 7
Group 8
Group 9
Group 10
Group 11
Group 12
Group 13
Group 14
Group 15
Group 16
Group 17
Group 18
Group 19
BaselineOmitted due to size. |
bantonsson
left a comment
There was a problem hiding this comment.
The change looks good, but there are new lines missing test coverage. Also, how do we deal with semver? This is a breaking change API wise.
| if data.chunks_dropped > 0 { | ||
| let key = self.metrics.get(metrics::MetricKind::ChunksDropped); | ||
| if data.chunks_dropped_p0 > 0 { | ||
| let key = self.metrics.get(metrics::MetricKind::ChunksDroppedP0); |
There was a problem hiding this comment.
Good catch, added some more tests.
25be86c to
071c7c3
Compare
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-apple-darwin
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-apple-darwin
x86_64-unknown-linux-gnu
|
|
/merge |
|
View all feedbacks in Devflow UI.
This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
[email protected] unqueued this merge request |
|
/remove |
|
View all feedbacks in Devflow UI.
|
071c7c3 to
40724be
Compare
@bantonsson - The semver check was correct. It's technically a breaking change because we made a function less public. In reality, no one should have been using that function but the rules are the rules. |
40724be to
f3bbc8a
Compare
# Conflicts: # libdd-data-pipeline/src/trace_exporter/stats.rs
f3bbc8a to
c188e8b
Compare
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
99be5d7
into
main
# Release proposal for libdd-data-pipeline and its dependencies This PR contains version bumps based on public API changes and commits since last release. ## libdd-common **Next version:** `2.0.0` **Semver bump:** `major` **Tag:** `libdd-common-v2.0.0` ### Commits - refactor(ddcommon)!: remove direct dependency on hyper client everywhere in common (#1604) - feat!: enable non-blocking DNS for reqwest (#1558) - feat: unify Azure tags (#1553) - feat(common): add current thread id API (#1569) - refactor!: switch from multipart to multer to resolve deprecation warnings and dependabot alerts (#1540) - feat: single source of truth for headers (fixes issue in profiling with missing headers) (#1493) - refactor!: make reqwest available in common (#1504) ## libdd-ddsketch **Next version:** `1.0.1` **Semver bump:** `patch` **Tag:** `libdd-ddsketch-v1.0.1` ### Commits - chore: remove manual changelog modifications (#1472) - build: update `prost` crates (#1426) - chore: add changelog for every published crate (#1396) ## libdd-trace-protobuf **Next version:** `1.1.0` **Semver bump:** `minor` **Tag:** `libdd-trace-protobuf-v1.1.0` ### Commits - chore: remove manual changelog modifications (#1472) - build: update `prost` crates (#1426) - chore: add changelog for every published crate (#1396) - Handle null span tag values (#1394) ## libdd-dogstatsd-client **Next version:** `1.0.1` **Semver bump:** `patch` **Tag:** `libdd-dogstatsd-client-v1.0.1` ### Commits - refactor(ddcommon)!: remove direct dependency on hyper client everywhere in common (#1604) - chore: release libddcommon-v1.1.0 (#1456) - chore: add changelog for every published crate (#1396) - style: fix recent clippy warnings (#1346) ## libdd-telemetry **Next version:** `3.0.0` **Semver bump:** `major` **Tag:** `libdd-telemetry-v3.0.0` ### Commits - refactor(ddcommon)!: remove direct dependency on hyper client everywhere in common (#1604) - chore(deps): bump tracing-subscriber to remove regex dep duplicate (#1608) - feat(telemetry)!: add process_tags to Application in telemetry (#1459) - fix(telemetry)!: fix logs payload format [APMSP-2590] (#1498) - feat(appsec): add endpoints collection (#1182) ## libdd-trace-normalization **Next version:** `1.0.1` **Semver bump:** `patch` **Tag:** `libdd-trace-normalization-v1.0.1` ### Commits - Prevent span start overflow panic (#1373) - [CHAOSPLT-932] Add support for internal fuzzing infra (#1372) - chore: add changelog for every published crate (#1396) ## libdd-trace-utils **Next version:** `2.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-utils-v2.0.0` ### Commits - refactor(ddcommon)!: remove direct dependency on hyper client everywhere in common (#1604) - feat: unify Azure tags (#1553) - fix(serverless): set hostname on stats from tracer to empty string (#1530) - chore: remove manual changelog modifications (#1472) - feat(sidecar)!: introduce TraceData to unify text and binary data (#1247) - chore: release libddcommon-v1.1.0 (#1456) - fix(test-agent): undo commenting arg in docker cmd (#1439) - [SLES-2652] Log error details when trace request fails (2) (#1441) - build: update `prost` crates (#1426) - chore(trace-utils): bump the test agent version used for integration tests (#1417) - [Serverless] Skip AAS metadata tagging when span is from API Management (#1409) - chore: add changelog for every published crate (#1396) - Handle null span tag values (#1394) - [SVLS-7934] Log error details when trace request fails (#1392) - Fix trace utils clippy warning (#1397) - feat(trace_utils): Allow sending trace stats using custom HTTP client (#1345) ## libdd-trace-stats **Next version:** `1.0.1` **Semver bump:** `patch` **Tag:** `libdd-trace-stats-v1.0.1` ### Commits - feat(sidecar)!: introduce TraceData to unify text and binary data (#1247) - chore: add changelog for every published crate (#1396) ## libdd-data-pipeline **Next version:** `2.0.0` **Semver bump:** `major` **Tag:** `libdd-data-pipeline-v2.0.0` ### Commits - refactor(ddcommon)!: remove direct dependency on hyper client everywhere in common (#1604) - refactor(trace_exporter)!: remove Proxy TraceExporter input mode (#1583) - refactor(libdd-data-pipeline): health metrics (#1433) - feat(data-pipeline)!: include reason for chunks dropped telemetry (#1449) - feat(sidecar)!: introduce TraceData to unify text and binary data (#1247) - chore: release libddcommon-v1.1.0 (#1456) - chore: prepare libdd-telemetry-v2.0.0 (#1457) - Allow submitting Vec<Vec<Span>> asynchronously (#1302) - test(data-pipeline): handle EINTR in test_health_metrics_disabled (#1430) - chore: add changelog for every published crate (#1396) [APMSP-2590]: https://datadoghq.atlassian.net/browse/APMSP-2590?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: dd-octo-sts[bot] <200755185+dd-octo-sts[bot]@users.noreply.github.com>
# Release proposal for libdd-data-pipeline and its dependencies This PR contains version bumps based on public API changes and commits since last release. ## libdd-common **Next version:** `2.0.0` **Semver bump:** `major` **Tag:** `libdd-common-v2.0.0` ### Commits - refactor(ddcommon)!: remove direct dependency on hyper client everywhere in common (#1604) - feat!: enable non-blocking DNS for reqwest (#1558) - feat: unify Azure tags (#1553) - feat(common): add current thread id API (#1569) - refactor!: switch from multipart to multer to resolve deprecation warnings and dependabot alerts (#1540) - feat: single source of truth for headers (fixes issue in profiling with missing headers) (#1493) - refactor!: make reqwest available in common (#1504) ## libdd-ddsketch **Next version:** `1.0.1` **Semver bump:** `patch` **Tag:** `libdd-ddsketch-v1.0.1` ### Commits - chore: remove manual changelog modifications (#1472) - build: update `prost` crates (#1426) - chore: add changelog for every published crate (#1396) ## libdd-trace-protobuf **Next version:** `1.1.0` **Semver bump:** `minor` **Tag:** `libdd-trace-protobuf-v1.1.0` ### Commits - chore: remove manual changelog modifications (#1472) - build: update `prost` crates (#1426) - chore: add changelog for every published crate (#1396) - Handle null span tag values (#1394) ## libdd-dogstatsd-client **Next version:** `1.0.1` **Semver bump:** `patch` **Tag:** `libdd-dogstatsd-client-v1.0.1` ### Commits - refactor(ddcommon)!: remove direct dependency on hyper client everywhere in common (#1604) - chore: release libddcommon-v1.1.0 (#1456) - chore: add changelog for every published crate (#1396) - style: fix recent clippy warnings (#1346) ## libdd-telemetry **Next version:** `3.0.0` **Semver bump:** `major` **Tag:** `libdd-telemetry-v3.0.0` ### Commits - refactor(ddcommon)!: remove direct dependency on hyper client everywhere in common (#1604) - chore(deps): bump tracing-subscriber to remove regex dep duplicate (#1608) - feat(telemetry)!: add process_tags to Application in telemetry (#1459) - fix(telemetry)!: fix logs payload format [APMSP-2590] (#1498) - feat(appsec): add endpoints collection (#1182) ## libdd-trace-normalization **Next version:** `1.0.1` **Semver bump:** `patch` **Tag:** `libdd-trace-normalization-v1.0.1` ### Commits - Prevent span start overflow panic (#1373) - [CHAOSPLT-932] Add support for internal fuzzing infra (#1372) - chore: add changelog for every published crate (#1396) ## libdd-trace-utils **Next version:** `2.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-utils-v2.0.0` ### Commits - refactor(ddcommon)!: remove direct dependency on hyper client everywhere in common (#1604) - feat: unify Azure tags (#1553) - fix(serverless): set hostname on stats from tracer to empty string (#1530) - chore: remove manual changelog modifications (#1472) - feat(sidecar)!: introduce TraceData to unify text and binary data (#1247) - chore: release libddcommon-v1.1.0 (#1456) - fix(test-agent): undo commenting arg in docker cmd (#1439) - [SLES-2652] Log error details when trace request fails (2) (#1441) - build: update `prost` crates (#1426) - chore(trace-utils): bump the test agent version used for integration tests (#1417) - [Serverless] Skip AAS metadata tagging when span is from API Management (#1409) - chore: add changelog for every published crate (#1396) - Handle null span tag values (#1394) - [SVLS-7934] Log error details when trace request fails (#1392) - Fix trace utils clippy warning (#1397) - feat(trace_utils): Allow sending trace stats using custom HTTP client (#1345) ## libdd-trace-stats **Next version:** `1.0.1` **Semver bump:** `patch` **Tag:** `libdd-trace-stats-v1.0.1` ### Commits - feat(sidecar)!: introduce TraceData to unify text and binary data (#1247) - chore: add changelog for every published crate (#1396) ## libdd-data-pipeline **Next version:** `2.0.0` **Semver bump:** `major` **Tag:** `libdd-data-pipeline-v2.0.0` ### Commits - refactor(ddcommon)!: remove direct dependency on hyper client everywhere in common (#1604) - refactor(trace_exporter)!: remove Proxy TraceExporter input mode (#1583) - refactor(libdd-data-pipeline): health metrics (#1433) - feat(data-pipeline)!: include reason for chunks dropped telemetry (#1449) - feat(sidecar)!: introduce TraceData to unify text and binary data (#1247) - chore: release libddcommon-v1.1.0 (#1456) - chore: prepare libdd-telemetry-v2.0.0 (#1457) - Allow submitting Vec<Vec<Span>> asynchronously (#1302) - test(data-pipeline): handle EINTR in test_health_metrics_disabled (#1430) - chore: add changelog for every published crate (#1396) [APMSP-2590]: https://datadoghq.atlassian.net/browse/APMSP-2590?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: dd-octo-sts[bot] <200755185+dd-octo-sts[bot]@users.noreply.github.com>
What does this PR do?
When we drop trace chunks due to a problem we emit telemetry, but we weren't including the reason tags. We also were not emitting chunks dropped telemetry for chunks dropped for valid sampling reasons. This PR addresses both issues. Now telemetry should have one of the following reasons tagged for
trace_chunks_dropped:p0_dropserialization_errorsend_failureMotivation
More accurate telemetry is good
Additional Notes
breaking change: The telemetry module in
libdd-data-pipelinewaspub. It should bepub(crate).How to test the change?
Existing tests have been updated.