feat(graphql): trace Apollo Federation entity resolvers and skip the health-check#9134
Conversation
Overall package sizeSelf size: 6.71 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.1 | 122.62 kB | 438.86 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: bdc17a5 | Docs | Datadog PR Page | Give us feedback! |
BenchmarksBenchmark execution time: 2026-07-13 12:13:23 Comparing candidate commit bdc17a5 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2326 metrics, 32 unstable metrics.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9134 +/- ##
========================================
Coverage 96.58% 96.58%
========================================
Files 918 918
Lines 121584 121705 +121
Branches 21169 20726 -443
========================================
+ Hits 117436 117554 +118
- Misses 4148 4151 +3 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
02c0b3c to
6e15f5a
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6e15f5a698
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
9e9429c to
9f56422
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9f5642226c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
crysmags
left a comment
There was a problem hiding this comment.
no blockers from me. The shape-gated health-check skip and per-schema abstract traversal look right.
I left two small non-blocking comments: one to make the cached-document health-check matcher fully exact, and one to make the "no graphql spans" test assert execute/resolve too.
The resolver-wrapping traversal only descended through lists and non-nulls, so it never reached object types referenced solely as union members or interface implementations. graphql resolves the concrete type at runtime, so an Apollo Federation subgraph (whose entity types are reachable only through the synthetic `_Entity` union returned by `_entities`) produced no graphql.resolve spans for those fields. Descend into union members via getTypes() and interface implementations via schema.getPossibleTypes(), keyed off the realm-agnostic `Symbol.toStringTag` discriminator so an interface without an explicit resolveType is still covered. patchedTypes keeps the walk one-time per type, so cyclic abstract types terminate and there is no per-resolve cost. Fixes: #1057
Apollo Gateway's polling subgraph health check sends a fixed
`query __ApolloServiceHealthCheck__ { __typename }` on every poll interval,
burying real traffic under health-check noise. Skip that named operation by
exact name before span creation. The skip is unconditional because the name is
spec-reserved (the `__` prefix) and the operation carries no user input, so
there is nothing for AppSec or IAST to inspect and no legitimate operation to
collide with. The gateway's other probe, the anonymous startup
`{ __typename }` query, is left alone on purpose: it fires only on startup and
schema update, so it is not a source of recurring noise.
Fixes: #1057
1. The health-check skip matched the reserved operation name alone. Since
operation names are client-controlled, a request naming itself
`__ApolloServiceHealthCheck__` while selecting a real field (or using a
mutation, or adding selections beside `__typename`) was silently dropped
from tracing and from the AppSec/IAST resolver channels. Confirm the
gateway's exact spec-fixed shape (`query __ApolloServiceHealthCheck__
{ __typename }`) before skipping so the reserved name can't become a
tracing/security bypass.
2. `getPossibleTypes` is schema-specific, but the interface-implementation
descent shared the global `patchedTypes` guard. Two schemas reusing one
interface instance meant the first schema's walk marked the interface and
the second schema's implementations were never wrapped, so their resolvers
went untraced. Key the interface descent per (schema, interface) instead.
The resolver-wrapping walk terminated on a global per-type guard, but the walk is schema-dependent: union members and interface implementations are read through the schema (getTypes / getPossibleTypes) and differ between schemas. Two schemas sharing any type on the path to an abstract type — a reused root Query, a shared interface — meant the first schema's walk marked that type, and the second schema's walk returned at it before reaching its own members or implementations, leaving those resolvers untraced. Key the walk's visited set per (schema, type) so each schema runs its own full traversal. Resolver wrapping is already idempotent via patchedResolvers, so re-walking a type shared across schemas never double-wraps a resolver. Fixes: #1057
The health-check skip only suppressed the execute and resolver spans, so a gateway poll still emitted graphql.parse and graphql.validate. graphql runs both before execute, so every cold poll (and every poll on servers without a document cache, e.g. graphql-yoga) left two heartbeat spans a customer's operation filter would flag. Detect the poll from the raw query string at parse — the only input parse has — and mark the produced document so validate and execute skip it too. Apollo Server caches parsed+validated documents in its documentStore and reuses the same document object across polls, so a warm poll reaches execute with the already-marked document; when a health-check document reaches execute without ever passing through the instrumented parse (pre-populated cache, hand-built document), the operation-shape check remains the backstop. Fixes: #1057
Health-check operation names and parsed documents are caller-controlled. Requiring Apollo's complete constant shape keeps transformed or multi-operation documents, and operations with variables or directives, on the traced AppSec and IAST path.
dda97b2 to
2c8dab4
Compare
crysmags
left a comment
There was a problem hiding this comment.
Non-blocking question: should the Apollo health-check skip also cover the Mercurius graphql.request span? GraphQLRequestPlugin starts that span before parse/validate/execute run, so the new skips still leave one top-level request span per gateway poll for Mercurius subgraphs. If the goal is only graphql-js/Apollo spans then this is fine, but if we want the health-check to disappear across GraphQL drivers, request.js probably needs the same early guard too.
…health-check (#9134) * fix(graphql): trace resolvers reached through unions and interfaces The resolver-wrapping traversal only descended through lists and non-nulls, so it never reached object types referenced solely as union members or interface implementations. graphql resolves the concrete type at runtime, so an Apollo Federation subgraph (whose entity types are reachable only through the synthetic `_Entity` union returned by `_entities`) produced no graphql.resolve spans for those fields. Descend into union members via getTypes() and interface implementations via schema.getPossibleTypes(), keyed off the realm-agnostic `Symbol.toStringTag` discriminator so an interface without an explicit resolveType is still covered. patchedTypes keeps the walk one-time per type, so cyclic abstract types terminate and there is no per-resolve cost. Fixes: #1057 * fix(graphql): skip the Apollo Federation health-check operation Apollo Gateway's polling subgraph health check sends a fixed `query __ApolloServiceHealthCheck__ { __typename }` on every poll interval, burying real traffic under health-check noise. Skip that named operation by exact name before span creation. The skip is unconditional because the name is spec-reserved (the `__` prefix) and the operation carries no user input, so there is nothing for AppSec or IAST to inspect and no legitimate operation to collide with. The gateway's other probe, the anonymous startup `{ __typename }` query, is left alone on purpose: it fires only on startup and schema update, so it is not a source of recurring noise. Fixes: #1057 * fix(graphql): harden federation health-check skip and interface walk 1. The health-check skip matched the reserved operation name alone. Since operation names are client-controlled, a request naming itself `__ApolloServiceHealthCheck__` while selecting a real field (or using a mutation, or adding selections beside `__typename`) was silently dropped from tracing and from the AppSec/IAST resolver channels. Confirm the gateway's exact spec-fixed shape (`query __ApolloServiceHealthCheck__ { __typename }`) before skipping so the reserved name can't become a tracing/security bypass. 2. `getPossibleTypes` is schema-specific, but the interface-implementation descent shared the global `patchedTypes` guard. Two schemas reusing one interface instance meant the first schema's walk marked the interface and the second schema's implementations were never wrapped, so their resolvers went untraced. Key the interface descent per (schema, interface) instead. * fix(graphql): wrap resolvers per schema when types are shared The resolver-wrapping walk terminated on a global per-type guard, but the walk is schema-dependent: union members and interface implementations are read through the schema (getTypes / getPossibleTypes) and differ between schemas. Two schemas sharing any type on the path to an abstract type — a reused root Query, a shared interface — meant the first schema's walk marked that type, and the second schema's walk returned at it before reaching its own members or implementations, leaving those resolvers untraced. Key the walk's visited set per (schema, type) so each schema runs its own full traversal. Resolver wrapping is already idempotent via patchedResolvers, so re-walking a type shared across schemas never double-wraps a resolver. Fixes: #1057 * fix(graphql): skip parse and validate spans for the health-check The health-check skip only suppressed the execute and resolver spans, so a gateway poll still emitted graphql.parse and graphql.validate. graphql runs both before execute, so every cold poll (and every poll on servers without a document cache, e.g. graphql-yoga) left two heartbeat spans a customer's operation filter would flag. Detect the poll from the raw query string at parse — the only input parse has — and mark the produced document so validate and execute skip it too. Apollo Server caches parsed+validated documents in its documentStore and reuses the same document object across polls, so a warm poll reaches execute with the already-marked document; when a health-check document reaches execute without ever passing through the instrumented parse (pre-populated cache, hand-built document), the operation-shape check remains the backstop. Fixes: #1057 * fix(graphql): require exact Apollo health-check documents Health-check operation names and parsed documents are caller-controlled. Requiring Apollo's complete constant shape keeps transformed or multi-operation documents, and operations with variables or directives, on the traced AppSec and IAST path.
…health-check (#9134) * fix(graphql): trace resolvers reached through unions and interfaces The resolver-wrapping traversal only descended through lists and non-nulls, so it never reached object types referenced solely as union members or interface implementations. graphql resolves the concrete type at runtime, so an Apollo Federation subgraph (whose entity types are reachable only through the synthetic `_Entity` union returned by `_entities`) produced no graphql.resolve spans for those fields. Descend into union members via getTypes() and interface implementations via schema.getPossibleTypes(), keyed off the realm-agnostic `Symbol.toStringTag` discriminator so an interface without an explicit resolveType is still covered. patchedTypes keeps the walk one-time per type, so cyclic abstract types terminate and there is no per-resolve cost. Fixes: #1057 * fix(graphql): skip the Apollo Federation health-check operation Apollo Gateway's polling subgraph health check sends a fixed `query __ApolloServiceHealthCheck__ { __typename }` on every poll interval, burying real traffic under health-check noise. Skip that named operation by exact name before span creation. The skip is unconditional because the name is spec-reserved (the `__` prefix) and the operation carries no user input, so there is nothing for AppSec or IAST to inspect and no legitimate operation to collide with. The gateway's other probe, the anonymous startup `{ __typename }` query, is left alone on purpose: it fires only on startup and schema update, so it is not a source of recurring noise. Fixes: #1057 * fix(graphql): harden federation health-check skip and interface walk 1. The health-check skip matched the reserved operation name alone. Since operation names are client-controlled, a request naming itself `__ApolloServiceHealthCheck__` while selecting a real field (or using a mutation, or adding selections beside `__typename`) was silently dropped from tracing and from the AppSec/IAST resolver channels. Confirm the gateway's exact spec-fixed shape (`query __ApolloServiceHealthCheck__ { __typename }`) before skipping so the reserved name can't become a tracing/security bypass. 2. `getPossibleTypes` is schema-specific, but the interface-implementation descent shared the global `patchedTypes` guard. Two schemas reusing one interface instance meant the first schema's walk marked the interface and the second schema's implementations were never wrapped, so their resolvers went untraced. Key the interface descent per (schema, interface) instead. * fix(graphql): wrap resolvers per schema when types are shared The resolver-wrapping walk terminated on a global per-type guard, but the walk is schema-dependent: union members and interface implementations are read through the schema (getTypes / getPossibleTypes) and differ between schemas. Two schemas sharing any type on the path to an abstract type — a reused root Query, a shared interface — meant the first schema's walk marked that type, and the second schema's walk returned at it before reaching its own members or implementations, leaving those resolvers untraced. Key the walk's visited set per (schema, type) so each schema runs its own full traversal. Resolver wrapping is already idempotent via patchedResolvers, so re-walking a type shared across schemas never double-wraps a resolver. Fixes: #1057 * fix(graphql): skip parse and validate spans for the health-check The health-check skip only suppressed the execute and resolver spans, so a gateway poll still emitted graphql.parse and graphql.validate. graphql runs both before execute, so every cold poll (and every poll on servers without a document cache, e.g. graphql-yoga) left two heartbeat spans a customer's operation filter would flag. Detect the poll from the raw query string at parse — the only input parse has — and mark the produced document so validate and execute skip it too. Apollo Server caches parsed+validated documents in its documentStore and reuses the same document object across polls, so a warm poll reaches execute with the already-marked document; when a health-check document reaches execute without ever passing through the instrumented parse (pre-populated cache, hand-built document), the operation-shape check remains the backstop. Fixes: #1057 * fix(graphql): require exact Apollo health-check documents Health-check operation names and parsed documents are caller-controlled. Requiring Apollo's complete constant shape keeps transformed or multi-operation documents, and operations with variables or directives, on the traced AppSec and IAST path.
Summary
Apollo Federation subgraphs lost resolver tracing in two ways:
The resolver-wrapping traversal only descended through lists and non-nulls, so object types referenced solely as union members or interface implementations were never wrapped. A subgraph's entity types are reachable only through the synthetic
_Entityunion returned by_entities, so their fields produced nographql.resolvespans. The walk now descends into union members (getTypes()) and interface implementations (schema.getPossibleTypes()), keyed off the realm-agnosticSymbol.toStringTagdiscriminator;patchedTypeskeeps it one-time per type, so cyclic abstract types terminate and there is no per-resolve cost.Apollo Gateway polls every subgraph with a fixed
__ApolloServiceHealthCheck__operation, burying real traffic under health-check noise. That operation is now skipped by exact name before span creation, the same built-in treatment mongodb gives its heartbeat command.Test plan
PLUGINS=graphql— federation/abstract-type specs run on graphql 16 (built with the real@apollo/subgraph), the rest unchanged.PLUGINS=apollo— gateway tracing unaffected.Fixes: #1057