Skip to content

[build] Vendor Google.Protobuf and opentelemetry-proto files#7653

Closed
zacharycmontoya wants to merge 3 commits into
masterfrom
zach.montoya/vendor-protobuf
Closed

[build] Vendor Google.Protobuf and opentelemetry-proto files#7653
zacharycmontoya wants to merge 3 commits into
masterfrom
zach.montoya/vendor-protobuf

Conversation

@zacharycmontoya

Copy link
Copy Markdown
Contributor

Summary of changes

Vendors the following libraries into Datadog.Trace.dll:

  • Google.Protobuf => vendored to the namespace Datadog.Trace.Vendors.Google.Protobuf
  • opentelemetry-proto => The .proto files that define the OTLP protocol are stored in tracer/src/Datadog.Trace/Vendors/protos/**/*.proto

This also updates the UpdateVendoredCode Nuke build target to download the Grpc.Tools NuGet package and compile the .proto files into C# code, and the resulting files are stored as tracer/src/Datadog.Trace/OpenTelemetry/Proto/**/*.g.cs

Reason for change

We currently must produce protobuf for the OTel Metris API feature (http/protobuf), so vendoring Google.Protobuf and the opentelemetry-proto repo makes our protobuf generation maintainable. Also, as we look to add a grpc OTLP exporter, we may need to use Google.Protobuf and some of the Grpc libraries. This at least starts the process of adding the requisite libraries.

Implementation details

  • Vendors Google.Protobuf and the opentelemetry-proto repositories
  • Adds the required file replacements for Google.Protobuf to build correctly
  • Updates the UpdateVendoredCode Nuke target to also run the protobuf compiler to convert the .proto files to corresponding .g.cs files

Test coverage

Locally, I've measured the following Datadog.Trace.dll size differences:

Framework Size Before (KB) Size After (KB) Absolute Diff (KB) Percent Diff
net6.0 8037 8775 738 9.18%
net461 8181 8917 736 9.00%
netcoreapp3.1 7961 8698 737 9.26%
netstandard2.0 7991 8728 737 9.22%

I haven't done any tests to identify startup costs.

Other details

…o files

Adds feature to UpdateVendors to customize the destination path of the vendored source code with a PathToDestination property
Note: This does not replace the base namespace of the vendored Google.Protobuf code so that we can import the Grpc.Tools NuGet package at build time to generate the C# Protobuf classes. In a separate commit we can try to rename the namespace and generate the C# Protobuf files ahead-of-time.
…dateVendors call. The files will now be persisted in the source tree
@github-actions github-actions Bot added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Oct 14, 2025
@zacharycmontoya
zacharycmontoya deleted the zach.montoya/vendor-protobuf branch April 1, 2026 15:48
zacharycmontoya added a commit that referenced this pull request May 22, 2026
…rt (#8645)

## Summary of changes
Adds `http/protobuf` support to the OTLP traces exporter introduced in
#8211, which introduced the exporter and `http/json` support.

This feature is enabled by setting `OTEL_TRACES_EXPORTER=otlp` and
`OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf` (or the parent
`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`).

## Reason for change
When #8211 landed, `http/json` was the only functional OTLP traces
protocol — setting any other protocol would fall back (with a log
message) to the Datadog MessagePack encoding.

`http/protobuf` is the OTLP default protocol per the OpenTelemetry
specification and is what most OTel collectors, Datadog Agent included,
expect to receive, so we need to support it to be useful out-of-the-box
for users adopting the OTLP traces export.

## Implementation details

### Serializer
Adds `Datadog.Trace.OpenTelemetry.Traces.OtlpTracesProtobufSerializer`,
a new `ISpanBufferSerializer` that emits the [OTLP
`ExportTraceServiceRequest`](https://github.com/open-telemetry/opentelemetry-proto/blob/v1.2.0/opentelemetry/proto/trace/v1/trace.proto)
protobuf payload directly into `SpanBuffer`'s output buffer:
- The `resource_spans` and `scope_spans` envelopes are opened on the
first `SerializeSpans` call, span entries are appended per chunk, and
`FinishBody` patches the two length-prefix placeholders using absolute
positions into the destination buffer.
- The serializer respects the same per-span / per-event / per-link
attribute count limits (`128`) as `OtlpTracesJsonSerializer`, and
re-uses the `OtlpMapper` logic to map resource attributes and tags onto
the serialized spans.
- `OtlpTracesProtobufSerializer` is **stateful** (it tracks the two
length-placeholder positions), so `AgentWriter` now creates a fresh
instance for the front and back buffers rather than sharing one — the
previous shared `OtlpTracesJsonSerializer` was stateless and didn't need
this.

### Vendored OpenTelemetry SDK
Builds on the protobuf primitives vendored from the OpenTelemetry .NET
SDK in #7653 (`ProtobufSerializer`, `ProtobufWireType`):
- Adds new field-number constant files:
`ProtobufOtlpTraceFieldNumberConstants` and
`ProtobufOtlpResourceFieldNumberConstants` (split out so the OTLP trace
serializer only depends on what it needs).
- Removes the `#if NETCOREAPP3_1_OR_GREATER` guard from the vendored
`ProtobufSerializer` — now that we vendor a `Span<>` polyfill the
serializer compiles and runs on all supported runtimes (.NET Framework
4.6.1+).
- Drops the matching `#if NET6_0_OR_GREATER` guard around the
`SubmitsOtlpTraces` integration test (`http/protobuf` and `http/json`
both run on .NET Framework now too).

### Wiring
- `ExporterSettings`: `OtlpProtocol.HttpProtobuf` now resolves to
`TracesEncoding.OtlpProtobuf` instead of falling back to `DatadogV0_4`.
- `AgentWriter.GetSpanSerializer`: selects
`OtlpTracesProtobufSerializer` when `TracesEncoding == OtlpProtobuf`.
- `ApiOtlp.SendTracesAsync`: chooses `application/x-protobuf` vs
`application/json` for the request `Content-Type` based on
`_tracesEncoding`. Adds the `application/x-protobuf` MIME type constant
to `Datadog.Trace.Agent.Transports.MimeTypes`.
- `supported-configurations.yaml`: documents `http/protobuf` as a valid
value for `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` and regenerates
`ConfigurationKeys.OpenTelemetry.g.cs` for all TFMs.

## Test coverage
- **Unit tests**: New `OtlpTracesProtobufSerializerTests` exercises the
serializer end-to-end using `Google.Protobuf` and a little custom
parsing. Test cases include:
  - Empty buffer (`FinishBody` returns 0)
  - Multiple traces reported in a single-chunk
  - Serializing span events/links/status
  - Exceeding buffer `maxSize` honoring behavior
  - Exceeding span limits
- **Integration tests**: `OpenTelemetrySdkTests.SubmitsOtlpTraces` now
runs both `http/json` and `http/protobuf` cases for the Datadog SDK on
all TFMs.
- **Snapshot unification**: To keep one Datadog-SDK snapshot regardless
of OTLP protocol, snapshots for `http/protobuf` are normalized to the
`http/json` shape via scrubbers that translate the test agent's protobuf
rendering (snake_case field names, string-form enum values like
`SPAN_KIND_SERVER`) into the JSON serializer's shape (camelCase field
names, integer enum values).
- **Snapshot rename**: `SubmitsOtlpTraces_DD_http_json.verified.txt`
renamed to `SubmitsOtlpTraces_DD.verified.txt` (one snapshot now covers
both protocols).

## Other details
With this PR, `http/json` and `http/protobuf` are both functional;
`grpc` still falls back to Datadog MessagePack encoding with a startup
warning.

### Follow-up work
- Implement the `grpc` OTLP traces protocol
- Add OTLP traces telemetry counts (the `TODO: Telemetry - Record OTLP
Traces API submissions` in `ApiOtlp.SendTracesAsync` still applies)
- Honor `OTEL_EXPORTER_OTLP_TRACES_TIMEOUT` in `ApiOtlp` (still unused)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:builds project files, build scripts, pipelines, versioning, releases, packages

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant