Skip to content

fix(tracing/otlp): apply tls_config to gRPC exporter#8714

Merged
GiedriusS merged 2 commits into
thanos-io:mainfrom
roth-wine:fix/otlp-grpc-tls-config
Mar 11, 2026
Merged

fix(tracing/otlp): apply tls_config to gRPC exporter#8714
GiedriusS merged 2 commits into
thanos-io:mainfrom
roth-wine:fix/otlp-grpc-tls-config

Conversation

@roth-wine

@roth-wine roth-wine commented Mar 11, 2026

Copy link
Copy Markdown
Contributor
  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

Changes

The traceGRPCOptions function did not apply TLSConfig to the gRPC
exporter, while traceHTTPOptions already did so correctly via
exthttp.NewTLSConfig. This caused tls_config.ca_file, cert_file
and key_file to have no effect when using client_type: grpc.

When no TLS credentials were supplied, the OTel SDK fell back to
credentials.NewTLS(nil) which only trusts system root CAs. Deployments
using a private or internal CA therefore failed at the TLS handshake.

Fix by mirroring the HTTP path: when insecure: false, build
credentials.TransportCredentials from the parsed TLSConfig via
exthttp.NewTLSConfig and pass it to the gRPC exporter via
WithTLSCredentials.

Verification

Added an integration test (TestNewTracerProvider_GRPCTLSConfigApplied)
that spins up an in-process gRPC server secured with a self-signed
certificate issued by a private CA (not present in the system trust
store). The test configures NewTracerProvider with client_type: grpc
and tls_config.ca_file pointing to that CA certificate, creates a
span, and asserts that tp.Shutdown succeeds.

Without the fix, the TLS handshake would fail with
x509: certificate signed by unknown authority and the test would
error. With the fix applied, the handshake succeeds and the span is
exported successfully.

traceGRPCOptions silently ignored the TLSConfig struct that is parsed
from the YAML configuration, while traceHTTPOptions already applied it
correctly via exthttp.NewTLSConfig. As a result, the ca_file, cert_file
and key_file fields had no effect for gRPC clients, even though the
configuration was valid and the values were present.

Without TLS credentials being supplied to the gRPC dialer, the exporter
fell through to the default credentials.NewTLS(nil) path in the OTel SDK
which only trusts system root CAs. Any deployment using a private or
internal CA therefore failed with:

  transport: authentication handshake failed: tls: failed to verify
  certificate: x509: certificate signed by unknown authority

The only available workaround was to supply the OTEL_EXPORTER_OTLP_CERTIFICATE,
OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE and OTEL_EXPORTER_OTLP_CLIENT_KEY
environment variables directly, bypassing the Thanos configuration entirely.

Fix this by mirroring the HTTP behaviour: when insecure is false, build a
*tls.Config from the parsed TLSConfig via exthttp.NewTLSConfig and pass it
to the gRPC exporter as credentials.TransportCredentials via
WithTLSCredentials. This enables ca_file (custom CA verification), cert_file
and key_file (mTLS client authentication) to work correctly for gRPC.

Signed-off-by: roth-wine <[email protected]>
Before the fix in the preceding commit, the tls_config fields ca_file,
cert_file and key_file were silently ignored by traceGRPCOptions, making
it impossible to verify through existing tests that TLS configuration
was actually applied to the gRPC exporter. This commit adds a test that
directly validates the fix by exercising the full export path against a
real TLS-secured gRPC server.

The test spins up an in-process gRPC server secured with a self-signed
ECDSA certificate issued by a private CA (not present in the system
trust store). It then configures NewTracerProvider with client_type: grpc
and tls_config.ca_file pointing to that CA certificate. A span is created
and tp.Shutdown is called to force a synchronous flush, which triggers the
actual gRPC connection and TLS handshake. If tls_config.ca_file were not
applied to the gRPC dialer, the TLS handshake would fail with:

  x509: certificate signed by unknown authority

and tp.Shutdown would return an error, failing the test.

The test infrastructure consists of:
- generateTestCert: generates a self-signed ECDSA P-256 certificate in
  memory, valid for 127.0.0.1, usable as both CA and server certificate.
- startTLSGRPCServer: starts a grpc.Server with TLS credentials and
  registers a minimal mock OTLP TraceService that accepts any export
  request, ensuring the full export RPC succeeds once TLS is established.

Signed-off-by: roth-wine <[email protected]>
@roth-wine
roth-wine force-pushed the fix/otlp-grpc-tls-config branch from 7fde34a to 41dc169 Compare March 11, 2026 08:25
@GiedriusS
GiedriusS enabled auto-merge March 11, 2026 08:29
@roth-wine

Copy link
Copy Markdown
Contributor Author

@GiedriusS The e2e tests seem flacky. Would you restart them? 😄

@GiedriusS
GiedriusS disabled auto-merge March 11, 2026 09:28
@GiedriusS
GiedriusS merged commit ae995d3 into thanos-io:main Mar 11, 2026
19 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants