Skip to content

[TT-17356] Join the MCP agent's trace from params._meta and propagate upstream#8276

Merged
lghiur merged 9 commits into
masterfrom
TT-17356-mcp-trace-context
Jun 9, 2026
Merged

[TT-17356] Join the MCP agent's trace from params._meta and propagate upstream#8276
lghiur merged 9 commits into
masterfrom
TT-17356-mcp-trace-context

Conversation

@lghiur

@lghiur lghiur commented Jun 3, 2026

Copy link
Copy Markdown
Collaborator

MCP-native agents carry the W3C trace context inside the JSON-RPC body (params._meta, SEP-414), not the traceparent header. Tyk only read the header, so for those requests it started a fresh, disconnected trace — its spans, access logs, and analytics couldn't be correlated to the agent's journey.

This makes Tyk a first-class participant in an MCP request's trace:

Read/join the trace context from configurable sources; when the header carried none, install it so every span/log/audit later in the request inherits the agent's trace_id. Reconcile (never fork) when both channels are present.
Write the current trace context into the outbound body so the MCP server joins too — the body-channel parallel of the traceparent header Tyk already injects for ordinary upstreams. Runs automatically when tracing is on (no per-API flag).
Span labels: mcp.method.name, mcp.{tool,resource,prompt}.name (keyed by primitive type), mcp.trace_source.
Config: opentelemetry.mcp_trace_context.read_sources (ordered, first-match-wins; default [{header}, {body, params._meta}]). Applies to all MCP primitive types; no-op when tracing is off or the body is non-MCP/malformed. Covered by unit tests (internal/mcp, internal/otel) and gateway integration tests.
Screenshot 2026-06-03 at 20 03 31

Description

Related Issue

Motivation and Context

How This Has Been Tested

Screenshots (if appropriate)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Refactoring or add test (improvements in base code or adds test coverage to functionality)

Checklist

  • I ensured that the documentation is up to date
  • I explained why this PR updates go.mod in detail with reasoning why it's required
  • I would like a code coverage CI quality gate exception and have explained why

Ticket Details

TT-17356
Status In Code Review
Summary MCP trace-context bridge, join the agent's trace via params._meta (read/write)

Generated at: 2026-06-04 12:35:36

@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

🎯 Recommended Merge Targets

Based on JIRA ticket TT-17356: MCP trace-context bridge, join the agent's trace via params._meta (read/write)

Fix Version: Tyk 5.13.1

Required:

  • release-5.13 - Minor version branch for 5.13.x patches - required for creating Tyk 5.13.1
  • master - Main development branch - ensures fix is in all future releases

Fix Version: Tyk 5.14.0

⚠️ Warning: Expected release branches not found in repository

Required:

  • master - No matching release branches found. Fix will be included in future releases.

📋 Workflow

  1. Merge this PR to master first

  2. Cherry-pick to release branches by commenting on the merged PR:

    • /release to release-5.13
  3. Automated backport - The bot will automatically create backport PRs to the specified release branches

@probelabs

probelabs Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

This PR introduces end-to-end distributed tracing for MCP (Model-as-a-Coprocessor) APIs. It resolves an issue where Tyk would start a new, disconnected trace for MCP requests because MCP-native agents send the W3C trace context in the JSON-RPC body (params._meta) rather than as an HTTP header.

This change allows Tyk to read the trace context from configurable sources (defaulting to the HTTP header, then the request body), join the agent's existing trace, and propagate the context upstream to the MCP server. This ensures that gateway spans, access logs, and analytics are correctly correlated with the entire request journey.

Files Changed Analysis

The changes are well-structured across 14 files, with a significant portion of the ~1100 new lines dedicated to comprehensive testing.

  • gateway/mw_jsonrpc.go: The core JSON-RPC middleware is updated to integrate the new tracing logic. It now reads the request body, invokes the new OpenTelemetry functions to join and propagate the trace context, and rewrites the body for upstream requests.
  • internal/otel/mcp/tracecontext.go: A new module is introduced to handle the low-level mechanics of reading from and writing the W3C trace context to the JSON body, isolating the SEP-414 specification details.
  • internal/otel/mcp_trace_context.go: This new file contains the high-level logic. It implements the configurable trace resolution strategy (header-wins), context joining, and defines new MCP-specific span attributes (mcp.method.name, mcp.tool.name, etc.).
  • internal/otel/config.go: This is extended with the new opentelemetry.traces.mcp configuration.
  • New Test Files: Seven new test files (*_test.go) provide extensive coverage for the new functionality, including unit tests for JSON manipulation, trace context resolution, context joining, and integration tests within the JSON-RPC middleware.

Architecture & Impact Assessment

  • What this PR accomplishes:

    • Enables unified, end-to-end distributed tracing for MCP APIs by bridging the gap between standard header-based tracing and MCP's body-based context propagation.
    • Ensures Tyk's telemetry for MCP requests can be correctly correlated with the trace initiated by the agent.
  • Key technical changes introduced:

    1. Configurable Trace Sources: A new setting, opentelemetry.traces.mcp.read_sources, allows defining an ordered, first-match-wins list of sources for the trace context. The default [{channel: "header"}, {channel: "body", path: "params._meta"}] ensures backward compatibility while adding MCP support.
    2. Trace Context Joining: The middleware joins the trace from the request body only if a trace context is not already present in the HTTP header, preventing the creation of a forked trace.
    3. Upstream Propagation: The active trace context is injected back into the params._meta of the request body before it's sent upstream, allowing the downstream MCP server to participate in the same trace.
    4. Enriched Spans: MCP request spans are now enriched with semantic attributes like mcp.method.name, mcp.tool.name, and mcp.trace_source for improved observability.
  • Affected system components:

    • JSON-RPC Middleware: The primary integration point for APIs with the mcp application protocol.
    • OpenTelemetry Instrumentation: Extended to handle MCP-specific trace context propagation.
    • Gateway Configuration: A new global configuration is introduced under opentelemetry settings.

Request Flow Visualization

sequenceDiagram
    participant Agent
    participant Tyk Gateway
    participant Upstream MCP Server

    Agent->>+Tyk Gateway: POST /mcp (traceparent in JSON body)
    Tyk Gateway->>Tyk Gateway: JSONRPCMiddleware: Reads body, finds traceparent in params._meta
    Tyk Gateway->>Tyk Gateway: otel.JoinMCPTraceContext(): Installs agent's trace context into request
    Tyk Gateway->>Tyk Gateway: otelmcp.WriteMetaTraceContext(): Injects current trace context into outbound body
    Tyk Gateway->>+Upstream MCP Server: POST / (traceparent in JSON body)
    Upstream MCP Server-->>-Tyk Gateway: Response
    Tyk Gateway-->>-Agent: Response
Loading

Scope Discovery & Context Expansion

  • The impact is strictly limited to APIs configured with application_protocol: "mcp" and has no effect on standard HTTP proxying.
  • The feature is only active when OpenTelemetry tracing is globally enabled. When disabled, the new logic is a no-op, and the request body is passed through unchanged.
  • The configuration is global, applying to all MCP APIs on the gateway. The defaults are non-disruptive.
  • The implementation gracefully handles malformed or non-JSON bodies by skipping the trace logic, ensuring gateway stability.
Metadata
  • Review Effort: 4 / 5
  • Primary Label: feature

Powered by Visor from Probelabs

Last updated: 2026-06-09T09:51:18.550Z | Triggered by: pr_updated | Commit: 8184f6f

💡 TIP: You can chat with Visor using /visor ask <your question>

@probelabs

probelabs Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

✅ Security Check Passed

No security issues found – changes LGTM.

✅ Architecture Check Passed

No architecture issues found – changes LGTM.

✅ Security Check Passed

No security issues found – changes LGTM.

\n\n

✅ Architecture Check Passed

No architecture issues found – changes LGTM.

\n\n

Performance Issues (1)

Severity Location Issue
🟡 Warning gateway/mw_jsonrpc.go:211-226
The JSON request body is parsed multiple times for each MCP request where tracing is enabled, leading to unnecessary CPU overhead. The body is unmarshaled once in `readAndParseJSONRPC`, a second time to extract the trace context within `otel.JoinMCPTraceContext`, and a third time to inject the trace context within `m.writeMCPTraceContext`.
💡 SuggestionRefactor the logic to parse the JSON body only once. The initial parsing in `readAndParseJSONRPC` could produce a `map[string]json.RawMessage` which is then passed to the trace handling functions (`JoinMCPTraceContext` and `writeMCPTraceContext`). These functions can then read from and modify this map directly, avoiding repeated unmarshaling. The final map would be marshaled back to bytes only once at the end of the process if modifications were made.

✅ Quality Check Passed

No quality issues found – changes LGTM.


Powered by Visor from Probelabs

Last updated: 2026-06-09T09:51:09.497Z | Triggered by: pr_updated | Commit: 8184f6f

💡 TIP: You can chat with Visor using /visor ask <your question>

@tbuchaillot tbuchaillot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please have a look at the config comment and add some tracing test case for this

Comment thread internal/otel/config.go Outdated
// MCPTraceContext configures where Tyk reads an MCP request's W3C trace
// context from (ordered, first-match-wins). Omitted ⇒ the canonical
// [{header}, {body, path: params._meta}] default.
MCPTraceContext MCPTraceContextConfig `json:"mcp_trace_context"`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be part of traceConfig structure. In 5.13 we introduced this division in configuration for metrics and traces so new tracing related config configs should live inside opentelemetry.traces instead of root level opentelemetry

@@ -0,0 +1,183 @@
package mcp

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick - shouldn't this live in otel package or otel/mcp since it's purely related to opentelemetry?

Comment thread internal/otel/mcp_trace_context.go Outdated
// extractTraceContext returns ctx with tc installed as the active (remote) span
// context via the W3C propagator — the same propagator Tyk uses for inbound
// header extraction, so the joined trace is byte-identical to a header-carried one.
func extractTraceContext(ctx context.Context, tc mcp.TraceContext) context.Context {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if this functions shouldn't be part of https://github.com/TykTechnologies/opentelemetry/tree/main/trace package so we avoid importing/using go.opentelemetry.io/otel directly here

@lghiur lghiur added the deps-reviewed Dependency changes reviewed and approved for CI execution label Jun 8, 2026
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

@edsonmichaque edsonmichaque left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

🚨 Jira Linter Failed

Commit: 8184f6f
Failed at: 2026-06-09 09:49:16 UTC

The Jira linter failed to validate your PR. Please check the error details below:

🔍 Click to view error details
failed to get Jira issue: failed to fetch Jira issue TT-17356: Issue does not exist or you do not have permission to see it.: request failed. Please analyze the request body for more details. Status code: 404

Next Steps

  • Ensure your branch name contains a valid Jira ticket ID (e.g., ABC-123)
  • Verify your PR title matches the branch's Jira ticket ID
  • Check that the Jira ticket exists and is accessible

This comment will be automatically deleted once the linter passes.

@sonarqubecloud

sonarqubecloud Bot commented Jun 9, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
90.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@lghiur
lghiur merged commit d72a70e into master Jun 9, 2026
119 of 125 checks passed
@lghiur
lghiur deleted the TT-17356-mcp-trace-context branch June 9, 2026 12:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deps-reviewed Dependency changes reviewed and approved for CI execution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants