fix(tracing/otlp): apply tls_config to gRPC exporter#8714
Merged
Conversation
traceGRPCOptions silently ignored the TLSConfig struct that is parsed from the YAML configuration, while traceHTTPOptions already applied it correctly via exthttp.NewTLSConfig. As a result, the ca_file, cert_file and key_file fields had no effect for gRPC clients, even though the configuration was valid and the values were present. Without TLS credentials being supplied to the gRPC dialer, the exporter fell through to the default credentials.NewTLS(nil) path in the OTel SDK which only trusts system root CAs. Any deployment using a private or internal CA therefore failed with: transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority The only available workaround was to supply the OTEL_EXPORTER_OTLP_CERTIFICATE, OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE and OTEL_EXPORTER_OTLP_CLIENT_KEY environment variables directly, bypassing the Thanos configuration entirely. Fix this by mirroring the HTTP behaviour: when insecure is false, build a *tls.Config from the parsed TLSConfig via exthttp.NewTLSConfig and pass it to the gRPC exporter as credentials.TransportCredentials via WithTLSCredentials. This enables ca_file (custom CA verification), cert_file and key_file (mTLS client authentication) to work correctly for gRPC. Signed-off-by: roth-wine <[email protected]>
roth-wine
force-pushed
the
fix/otlp-grpc-tls-config
branch
from
March 11, 2026 08:22
d6ea1e7 to
7fde34a
Compare
Before the fix in the preceding commit, the tls_config fields ca_file, cert_file and key_file were silently ignored by traceGRPCOptions, making it impossible to verify through existing tests that TLS configuration was actually applied to the gRPC exporter. This commit adds a test that directly validates the fix by exercising the full export path against a real TLS-secured gRPC server. The test spins up an in-process gRPC server secured with a self-signed ECDSA certificate issued by a private CA (not present in the system trust store). It then configures NewTracerProvider with client_type: grpc and tls_config.ca_file pointing to that CA certificate. A span is created and tp.Shutdown is called to force a synchronous flush, which triggers the actual gRPC connection and TLS handshake. If tls_config.ca_file were not applied to the gRPC dialer, the TLS handshake would fail with: x509: certificate signed by unknown authority and tp.Shutdown would return an error, failing the test. The test infrastructure consists of: - generateTestCert: generates a self-signed ECDSA P-256 certificate in memory, valid for 127.0.0.1, usable as both CA and server certificate. - startTLSGRPCServer: starts a grpc.Server with TLS credentials and registers a minimal mock OTLP TraceService that accepts any export request, ensuring the full export RPC succeeds once TLS is established. Signed-off-by: roth-wine <[email protected]>
roth-wine
force-pushed
the
fix/otlp-grpc-tls-config
branch
from
March 11, 2026 08:25
7fde34a to
41dc169
Compare
GiedriusS
approved these changes
Mar 11, 2026
GiedriusS
enabled auto-merge
March 11, 2026 08:29
Contributor
Author
|
@GiedriusS The e2e tests seem flacky. Would you restart them? 😄 |
GiedriusS
disabled auto-merge
March 11, 2026 09:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Changes
The
traceGRPCOptionsfunction did not applyTLSConfigto the gRPCexporter, while
traceHTTPOptionsalready did so correctly viaexthttp.NewTLSConfig. This causedtls_config.ca_file,cert_fileand
key_fileto have no effect when usingclient_type: grpc.When no TLS credentials were supplied, the OTel SDK fell back to
credentials.NewTLS(nil)which only trusts system root CAs. Deploymentsusing a private or internal CA therefore failed at the TLS handshake.
Fix by mirroring the HTTP path: when
insecure: false, buildcredentials.TransportCredentialsfrom the parsedTLSConfigviaexthttp.NewTLSConfigand pass it to the gRPC exporter viaWithTLSCredentials.Verification
Added an integration test (TestNewTracerProvider_GRPCTLSConfigApplied)
that spins up an in-process gRPC server secured with a self-signed
certificate issued by a private CA (not present in the system trust
store). The test configures NewTracerProvider with client_type: grpc
and tls_config.ca_file pointing to that CA certificate, creates a
span, and asserts that tp.Shutdown succeeds.
Without the fix, the TLS handshake would fail with
x509: certificate signed by unknown authority and the test would
error. With the fix applied, the handshake succeeds and the span is
exported successfully.