[build] Vendor Google.Protobuf and opentelemetry-proto files#7653
Closed
zacharycmontoya wants to merge 3 commits into
Closed
[build] Vendor Google.Protobuf and opentelemetry-proto files#7653zacharycmontoya wants to merge 3 commits into
zacharycmontoya wants to merge 3 commits into
Conversation
…o files Adds feature to UpdateVendors to customize the destination path of the vendored source code with a PathToDestination property
Note: This does not replace the base namespace of the vendored Google.Protobuf code so that we can import the Grpc.Tools NuGet package at build time to generate the C# Protobuf classes. In a separate commit we can try to rename the namespace and generate the C# Protobuf files ahead-of-time.
…dateVendors call. The files will now be persisted in the source tree
zacharycmontoya
added a commit
that referenced
this pull request
May 22, 2026
…rt (#8645) ## Summary of changes Adds `http/protobuf` support to the OTLP traces exporter introduced in #8211, which introduced the exporter and `http/json` support. This feature is enabled by setting `OTEL_TRACES_EXPORTER=otlp` and `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf` (or the parent `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`). ## Reason for change When #8211 landed, `http/json` was the only functional OTLP traces protocol — setting any other protocol would fall back (with a log message) to the Datadog MessagePack encoding. `http/protobuf` is the OTLP default protocol per the OpenTelemetry specification and is what most OTel collectors, Datadog Agent included, expect to receive, so we need to support it to be useful out-of-the-box for users adopting the OTLP traces export. ## Implementation details ### Serializer Adds `Datadog.Trace.OpenTelemetry.Traces.OtlpTracesProtobufSerializer`, a new `ISpanBufferSerializer` that emits the [OTLP `ExportTraceServiceRequest`](https://github.com/open-telemetry/opentelemetry-proto/blob/v1.2.0/opentelemetry/proto/trace/v1/trace.proto) protobuf payload directly into `SpanBuffer`'s output buffer: - The `resource_spans` and `scope_spans` envelopes are opened on the first `SerializeSpans` call, span entries are appended per chunk, and `FinishBody` patches the two length-prefix placeholders using absolute positions into the destination buffer. - The serializer respects the same per-span / per-event / per-link attribute count limits (`128`) as `OtlpTracesJsonSerializer`, and re-uses the `OtlpMapper` logic to map resource attributes and tags onto the serialized spans. - `OtlpTracesProtobufSerializer` is **stateful** (it tracks the two length-placeholder positions), so `AgentWriter` now creates a fresh instance for the front and back buffers rather than sharing one — the previous shared `OtlpTracesJsonSerializer` was stateless and didn't need this. ### Vendored OpenTelemetry SDK Builds on the protobuf primitives vendored from the OpenTelemetry .NET SDK in #7653 (`ProtobufSerializer`, `ProtobufWireType`): - Adds new field-number constant files: `ProtobufOtlpTraceFieldNumberConstants` and `ProtobufOtlpResourceFieldNumberConstants` (split out so the OTLP trace serializer only depends on what it needs). - Removes the `#if NETCOREAPP3_1_OR_GREATER` guard from the vendored `ProtobufSerializer` — now that we vendor a `Span<>` polyfill the serializer compiles and runs on all supported runtimes (.NET Framework 4.6.1+). - Drops the matching `#if NET6_0_OR_GREATER` guard around the `SubmitsOtlpTraces` integration test (`http/protobuf` and `http/json` both run on .NET Framework now too). ### Wiring - `ExporterSettings`: `OtlpProtocol.HttpProtobuf` now resolves to `TracesEncoding.OtlpProtobuf` instead of falling back to `DatadogV0_4`. - `AgentWriter.GetSpanSerializer`: selects `OtlpTracesProtobufSerializer` when `TracesEncoding == OtlpProtobuf`. - `ApiOtlp.SendTracesAsync`: chooses `application/x-protobuf` vs `application/json` for the request `Content-Type` based on `_tracesEncoding`. Adds the `application/x-protobuf` MIME type constant to `Datadog.Trace.Agent.Transports.MimeTypes`. - `supported-configurations.yaml`: documents `http/protobuf` as a valid value for `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` and regenerates `ConfigurationKeys.OpenTelemetry.g.cs` for all TFMs. ## Test coverage - **Unit tests**: New `OtlpTracesProtobufSerializerTests` exercises the serializer end-to-end using `Google.Protobuf` and a little custom parsing. Test cases include: - Empty buffer (`FinishBody` returns 0) - Multiple traces reported in a single-chunk - Serializing span events/links/status - Exceeding buffer `maxSize` honoring behavior - Exceeding span limits - **Integration tests**: `OpenTelemetrySdkTests.SubmitsOtlpTraces` now runs both `http/json` and `http/protobuf` cases for the Datadog SDK on all TFMs. - **Snapshot unification**: To keep one Datadog-SDK snapshot regardless of OTLP protocol, snapshots for `http/protobuf` are normalized to the `http/json` shape via scrubbers that translate the test agent's protobuf rendering (snake_case field names, string-form enum values like `SPAN_KIND_SERVER`) into the JSON serializer's shape (camelCase field names, integer enum values). - **Snapshot rename**: `SubmitsOtlpTraces_DD_http_json.verified.txt` renamed to `SubmitsOtlpTraces_DD.verified.txt` (one snapshot now covers both protocols). ## Other details With this PR, `http/json` and `http/protobuf` are both functional; `grpc` still falls back to Datadog MessagePack encoding with a startup warning. ### Follow-up work - Implement the `grpc` OTLP traces protocol - Add OTLP traces telemetry counts (the `TODO: Telemetry - Record OTLP Traces API submissions` in `ApiOtlp.SendTracesAsync` still applies) - Honor `OTEL_EXPORTER_OTLP_TRACES_TIMEOUT` in `ApiOtlp` (still unused) --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
Vendors the following libraries into
Datadog.Trace.dll:Google.Protobuf=> vendored to the namespaceDatadog.Trace.Vendors.Google.Protobufopentelemetry-proto=> The.protofiles that define the OTLP protocol are stored intracer/src/Datadog.Trace/Vendors/protos/**/*.protoThis also updates the
UpdateVendoredCodeNuke build target to download theGrpc.ToolsNuGet package and compile the.protofiles into C# code, and the resulting files are stored astracer/src/Datadog.Trace/OpenTelemetry/Proto/**/*.g.csReason for change
We currently must produce protobuf for the OTel Metris API feature (
http/protobuf), so vendoringGoogle.Protobufand theopentelemetry-protorepo makes our protobuf generation maintainable. Also, as we look to add agrpcOTLP exporter, we may need to useGoogle.Protobufand some of theGrpclibraries. This at least starts the process of adding the requisite libraries.Implementation details
Google.Protobufand theopentelemetry-protorepositoriesGoogle.Protobufto build correctlyUpdateVendoredCodeNuke target to also run the protobuf compiler to convert the.protofiles to corresponding.g.csfilesTest coverage
Locally, I've measured the following
Datadog.Trace.dllsize differences:I haven't done any tests to identify startup costs.
Other details