fix(express): restore loopback middleware phase ordering#9062
Conversation
This makes sure loopback's phase-based middleware sorting keeps working under
tracing. loopback's `_findLayerByHandler` maps an express layer back to the
user handler by scanning the layer handle's enumerable properties, then tags
the layer with its phase. Replacing `layer.handle` with the tracing wrapper hid
the original handler, so the lookup failed, the layer kept no phase, and an
`app.middleware('final', ...)` handler ran in insertion order ahead of the
route handlers instead of last. Restoring the `_datadog_orig` back-reference
that the router instrumentation rewrite dropped lets the lookup succeed again.
Refs: #643
1. The block passed the plugin export to `withVersions` instead of the `'express'` integration name, so no installed version matched and the suite never ran. Switching to the name runs it. 2. Legacy loopback emits the deprecated `util._extend` warning at module load, which the harness deprecation guard turns into a thrown error; allowlist that one message so the modules load.
Overall package sizeSelf size: 6.34 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.2.0 | 104.26 kB | 843.44 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
🎉 All green!🧪 All tests passed 🔗 Commit SHA: f12fcdf | Docs | Datadog PR Page | Give us feedback! |
BenchmarksBenchmark execution time: 2026-06-25 18:16:24 Comparing candidate commit f12fcdf in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 1951 metrics, 14 unstable metrics.
|
| }) | ||
|
|
||
| withVersions(plugin, 'loopback', loopbackVersion => { | ||
| withVersions('express', 'loopback', loopbackVersion => { |
There was a problem hiding this comment.
maybe withVersions should throw if no matches are returned? That change may uncover additional similar situations
There was a problem hiding this comment.
I like that suggestion! Yes!
A withVersions call whose module argument matches no loaded instrumentation produced zero test cases and registered no suites, so the spec passed while running nothing. Passing the plugin export instead of its integration name was the silent trigger. Throw instead, so a misconfigured call fails loudly rather than skipping its whole suite. Refs: #9062 (comment)
Middleware tracing replaced `layer.handle` with a wrapper and re-exposed the user handler through an enumerable `_datadog_orig` back-reference, the only property loopback's `_findLayerByHandler` could follow to tag a layer with its phase. That ties the tracer to loopback's private reflection heuristic, where the property name and its enumerability are both load-bearing. Wrap the host `Layer` prototype dispatch (`handle_request`/`handle_error`, and `handleRequest`/`handleError` on router >=2) and read per-layer metadata from a WeakMap, leaving `layer.handle` the user's function. loopback's first lookup (`layer.handle === handler`) then matches with no tracer-specific contract, so the back-reference, the `express-async-errors` `__handle` branch, and the `_name` write-back onto the user handler are no longer needed. A synchronous throw or a rejected promise becomes `next(error)` in the host dispatch, so the wrapped `next` captures both without a tracer-side try/catch. Refs: #9062
…9069) * test(harness): fail withVersions when no installed module matches A withVersions call whose module argument matches no loaded instrumentation produced zero test cases and registered no suites, so the spec passed while running nothing. Passing the plugin export instead of its integration name was the silent trigger. Throw instead, so a misconfigured call fails loudly rather than skipping its whole suite. Refs: #9062 (comment) * test(graphql): run the silently-skipped apollo-server-core suite The suite passed the plugin export to withVersions instead of the 'graphql' integration name, so externals.js never resolved apollo-server-core and it ran zero tests. Switch to the integration name; the suite stays describe.skip as before. * test(opensearch): drop the never-instrumented bare opensearch module The instrumentation only hooks @opensearch-project/opensearch; the bare opensearch package has never been hooked, so withVersions resolved no instrumentation for it and the suite ran zero tests for that entry. The new guard turns that silent skip into a throw, so remove the dead module.
This makes sure loopback's phase-based middleware sorting keeps working under
tracing. loopback's `_findLayerByHandler` maps an express layer back to the
user handler by scanning the layer handle's enumerable properties, then tags
the layer with its phase. Replacing `layer.handle` with the tracing wrapper hid
the original handler, so the lookup failed, the layer kept no phase, and an
`app.middleware('final', ...)` handler ran in insertion order ahead of the
route handlers instead of last. Restoring the `_datadog_orig` back-reference
that the router instrumentation rewrite dropped lets the lookup succeed again.
Refs: #643
* test(express): run the silently-skipped loopback suite
1. The block passed the plugin export to `withVersions` instead of the
`'express'` integration name, so no installed version matched and the suite
never ran. Switching to the name runs it.
2. Legacy loopback emits the deprecated `util._extend` warning at module load,
which the harness deprecation guard turns into a thrown error; allowlist that
one message so the modules load.
…9069) * test(harness): fail withVersions when no installed module matches A withVersions call whose module argument matches no loaded instrumentation produced zero test cases and registered no suites, so the spec passed while running nothing. Passing the plugin export instead of its integration name was the silent trigger. Throw instead, so a misconfigured call fails loudly rather than skipping its whole suite. Refs: #9062 (comment) * test(graphql): run the silently-skipped apollo-server-core suite The suite passed the plugin export to withVersions instead of the 'graphql' integration name, so externals.js never resolved apollo-server-core and it ran zero tests. Switch to the integration name; the suite stays describe.skip as before. * test(opensearch): drop the never-instrumented bare opensearch module The instrumentation only hooks @opensearch-project/opensearch; the bare opensearch package has never been hooked, so withVersions resolved no instrumentation for it and the suite ran zero tests for that entry. The new guard turns that silent skip into a throw, so remove the dead module.
This makes sure loopback's phase-based middleware sorting keeps working under
tracing. loopback's `_findLayerByHandler` maps an express layer back to the
user handler by scanning the layer handle's enumerable properties, then tags
the layer with its phase. Replacing `layer.handle` with the tracing wrapper hid
the original handler, so the lookup failed, the layer kept no phase, and an
`app.middleware('final', ...)` handler ran in insertion order ahead of the
route handlers instead of last. Restoring the `_datadog_orig` back-reference
that the router instrumentation rewrite dropped lets the lookup succeed again.
Refs: #643
* test(express): run the silently-skipped loopback suite
1. The block passed the plugin export to `withVersions` instead of the
`'express'` integration name, so no installed version matched and the suite
never ran. Switching to the name runs it.
2. Legacy loopback emits the deprecated `util._extend` warning at module load,
which the harness deprecation guard turns into a thrown error; allowlist that
one message so the modules load.
…9069) * test(harness): fail withVersions when no installed module matches A withVersions call whose module argument matches no loaded instrumentation produced zero test cases and registered no suites, so the spec passed while running nothing. Passing the plugin export instead of its integration name was the silent trigger. Throw instead, so a misconfigured call fails loudly rather than skipping its whole suite. Refs: #9062 (comment) * test(graphql): run the silently-skipped apollo-server-core suite The suite passed the plugin export to withVersions instead of the 'graphql' integration name, so externals.js never resolved apollo-server-core and it ran zero tests. Switch to the integration name; the suite stays describe.skip as before. * test(opensearch): drop the never-instrumented bare opensearch module The instrumentation only hooks @opensearch-project/opensearch; the bare opensearch package has never been hooked, so withVersions resolved no instrumentation for it and the suite ran zero tests for that entry. The new guard turns that silent skip into a throw, so remove the dead module.
This makes sure loopback's phase-based middleware sorting keeps working under
tracing. loopback's `_findLayerByHandler` maps an express layer back to the
user handler by scanning the layer handle's enumerable properties, then tags
the layer with its phase. Replacing `layer.handle` with the tracing wrapper hid
the original handler, so the lookup failed, the layer kept no phase, and an
`app.middleware('final', ...)` handler ran in insertion order ahead of the
route handlers instead of last. Restoring the `_datadog_orig` back-reference
that the router instrumentation rewrite dropped lets the lookup succeed again.
Refs: #643
* test(express): run the silently-skipped loopback suite
1. The block passed the plugin export to `withVersions` instead of the
`'express'` integration name, so no installed version matched and the suite
never ran. Switching to the name runs it.
2. Legacy loopback emits the deprecated `util._extend` warning at module load,
which the harness deprecation guard turns into a thrown error; allowlist that
one message so the modules load.
…9069) * test(harness): fail withVersions when no installed module matches A withVersions call whose module argument matches no loaded instrumentation produced zero test cases and registered no suites, so the spec passed while running nothing. Passing the plugin export instead of its integration name was the silent trigger. Throw instead, so a misconfigured call fails loudly rather than skipping its whole suite. Refs: #9062 (comment) * test(graphql): run the silently-skipped apollo-server-core suite The suite passed the plugin export to withVersions instead of the 'graphql' integration name, so externals.js never resolved apollo-server-core and it ran zero tests. Switch to the integration name; the suite stays describe.skip as before. * test(opensearch): drop the never-instrumented bare opensearch module The instrumentation only hooks @opensearch-project/opensearch; the bare opensearch package has never been hooked, so withVersions resolved no instrumentation for it and the suite ran zero tests for that entry. The new guard turns that silent skip into a throw, so remove the dead module.
* refactor(express): trace middleware via layer prototype dispatch Middleware tracing replaced `layer.handle` with a wrapper and re-exposed the user handler through an enumerable `_datadog_orig` back-reference, the only property loopback's `_findLayerByHandler` could follow to tag a layer with its phase. That ties the tracer to loopback's private reflection heuristic, where the property name and its enumerability are both load-bearing. Wrap the host `Layer` prototype dispatch (`handle_request`/`handle_error`, and `handleRequest`/`handleError` on router >=2) and read per-layer metadata from a WeakMap, leaving `layer.handle` the user's function. loopback's first lookup (`layer.handle === handler`) then matches with no tracer-specific contract, so the back-reference, the `express-async-errors` `__handle` branch, and the `_name` write-back onto the user handler are no longer needed. A synchronous throw or a rejected promise becomes `next(error)` in the host dispatch, so the wrapped `next` captures both without a tracer-side try/catch. Refs: #9062 * fix(express): trace middleware on express <4.3.0 (no prototype dispatch) express <4.3.0 has no `Layer.prototype.handle_request`/`handle_error`; the router invokes `layer.handle` directly, so wrapping the prototype traced nothing and dropped middleware, route-handler, and code-origin spans there. Replace `layer.handle` in place (arity preserved so the host still routes error handlers) when the layer exposes no dispatch method. express 4.3.0+, express 5, the router package, and loopback keep `handle` pristine via the prototype wraps. Also point the router-helper spec at the renamed `setLayerMeta` helper. * test(router): cover express-async-errors on prototype-dispatch hosts express-async-errors redefines `handle` as a getter/setter that stores the wrapped handler in `__handle` and patches `handle` only, never `handle_request`. On express 4.3.0+ the tracer's prototype-dispatch wrap therefore has to survive that patch, the arity gate has to read the real 3-arg handler, and a rejected async handler has to reach the wrapped `next` as `next(error)`. Pin that path so a regression surfaces here instead of in a downstream tracer test. * fix(express): restore error publishing for throwing middleware on express <4.6.0 On express <4.6.0 the router has no `Layer` prototype dispatch: it runs `layer.handle` directly and, on a synchronous throw, catches outside the layer and calls its own `next(error)` — never the wrapped `next` the tracer installed. The prototype-dispatch rewrite dropped the legacy handle wrap's catch, so a throwing middleware or route handler no longer published `middleware:error` / `middleware:next` / `middleware:finish`: the span lost its error tag and lingered on the stack until request finish. Restore the catch in `wrapLegacyHandle` to publish error/next/finish and rethrow, mirroring `wrapNext`. The prototype hosts (express >=4.6.0, express 5, router) already route the throw through the wrapped next, so only the legacy path needs it. Also correct the version boundary in the surrounding comments: `handle_request` landed in express 4.6.0, not 4.3.0, so the legacy path covers `>=4.0.0 <4.6.0`. Refs: #9067 (comment) * fix(express): publish middleware next/finish once per continuation A handler that calls `next()` and then rejects (`async (req, res, next) => { next(); await bg() }` where `bg()` rejects) makes the host invoke the same wrapped `next` twice: the clean `next()` finishes and pops the middleware span, then the host's rejection pass calls `next(error)` again. Since the middleware stack is already empty, that second pass tagged the error on the parent span, inventing a request-level error the middleware never surfaced. Guard the wrapped continuation so its error/next/finish publishes fire at most once per dispatch; the second call still forwards to the host chain untouched. Add the missing JSDoc on `hasLayerDispatch`.
* refactor(express): trace middleware via layer prototype dispatch Middleware tracing replaced `layer.handle` with a wrapper and re-exposed the user handler through an enumerable `_datadog_orig` back-reference, the only property loopback's `_findLayerByHandler` could follow to tag a layer with its phase. That ties the tracer to loopback's private reflection heuristic, where the property name and its enumerability are both load-bearing. Wrap the host `Layer` prototype dispatch (`handle_request`/`handle_error`, and `handleRequest`/`handleError` on router >=2) and read per-layer metadata from a WeakMap, leaving `layer.handle` the user's function. loopback's first lookup (`layer.handle === handler`) then matches with no tracer-specific contract, so the back-reference, the `express-async-errors` `__handle` branch, and the `_name` write-back onto the user handler are no longer needed. A synchronous throw or a rejected promise becomes `next(error)` in the host dispatch, so the wrapped `next` captures both without a tracer-side try/catch. Refs: #9062 * fix(express): trace middleware on express <4.3.0 (no prototype dispatch) express <4.3.0 has no `Layer.prototype.handle_request`/`handle_error`; the router invokes `layer.handle` directly, so wrapping the prototype traced nothing and dropped middleware, route-handler, and code-origin spans there. Replace `layer.handle` in place (arity preserved so the host still routes error handlers) when the layer exposes no dispatch method. express 4.3.0+, express 5, the router package, and loopback keep `handle` pristine via the prototype wraps. Also point the router-helper spec at the renamed `setLayerMeta` helper. * test(router): cover express-async-errors on prototype-dispatch hosts express-async-errors redefines `handle` as a getter/setter that stores the wrapped handler in `__handle` and patches `handle` only, never `handle_request`. On express 4.3.0+ the tracer's prototype-dispatch wrap therefore has to survive that patch, the arity gate has to read the real 3-arg handler, and a rejected async handler has to reach the wrapped `next` as `next(error)`. Pin that path so a regression surfaces here instead of in a downstream tracer test. * fix(express): restore error publishing for throwing middleware on express <4.6.0 On express <4.6.0 the router has no `Layer` prototype dispatch: it runs `layer.handle` directly and, on a synchronous throw, catches outside the layer and calls its own `next(error)` — never the wrapped `next` the tracer installed. The prototype-dispatch rewrite dropped the legacy handle wrap's catch, so a throwing middleware or route handler no longer published `middleware:error` / `middleware:next` / `middleware:finish`: the span lost its error tag and lingered on the stack until request finish. Restore the catch in `wrapLegacyHandle` to publish error/next/finish and rethrow, mirroring `wrapNext`. The prototype hosts (express >=4.6.0, express 5, router) already route the throw through the wrapped next, so only the legacy path needs it. Also correct the version boundary in the surrounding comments: `handle_request` landed in express 4.6.0, not 4.3.0, so the legacy path covers `>=4.0.0 <4.6.0`. Refs: #9067 (comment) * fix(express): publish middleware next/finish once per continuation A handler that calls `next()` and then rejects (`async (req, res, next) => { next(); await bg() }` where `bg()` rejects) makes the host invoke the same wrapped `next` twice: the clean `next()` finishes and pops the middleware span, then the host's rejection pass calls `next(error)` again. Since the middleware stack is already empty, that second pass tagged the error on the parent span, inventing a request-level error the middleware never surfaced. Guard the wrapped continuation so its error/next/finish publishes fire at most once per dispatch; the second call still forwards to the host chain untouched. Add the missing JSDoc on `hasLayerDispatch`.
* refactor(express): trace middleware via layer prototype dispatch Middleware tracing replaced `layer.handle` with a wrapper and re-exposed the user handler through an enumerable `_datadog_orig` back-reference, the only property loopback's `_findLayerByHandler` could follow to tag a layer with its phase. That ties the tracer to loopback's private reflection heuristic, where the property name and its enumerability are both load-bearing. Wrap the host `Layer` prototype dispatch (`handle_request`/`handle_error`, and `handleRequest`/`handleError` on router >=2) and read per-layer metadata from a WeakMap, leaving `layer.handle` the user's function. loopback's first lookup (`layer.handle === handler`) then matches with no tracer-specific contract, so the back-reference, the `express-async-errors` `__handle` branch, and the `_name` write-back onto the user handler are no longer needed. A synchronous throw or a rejected promise becomes `next(error)` in the host dispatch, so the wrapped `next` captures both without a tracer-side try/catch. Refs: #9062 * fix(express): trace middleware on express <4.3.0 (no prototype dispatch) express <4.3.0 has no `Layer.prototype.handle_request`/`handle_error`; the router invokes `layer.handle` directly, so wrapping the prototype traced nothing and dropped middleware, route-handler, and code-origin spans there. Replace `layer.handle` in place (arity preserved so the host still routes error handlers) when the layer exposes no dispatch method. express 4.3.0+, express 5, the router package, and loopback keep `handle` pristine via the prototype wraps. Also point the router-helper spec at the renamed `setLayerMeta` helper. * test(router): cover express-async-errors on prototype-dispatch hosts express-async-errors redefines `handle` as a getter/setter that stores the wrapped handler in `__handle` and patches `handle` only, never `handle_request`. On express 4.3.0+ the tracer's prototype-dispatch wrap therefore has to survive that patch, the arity gate has to read the real 3-arg handler, and a rejected async handler has to reach the wrapped `next` as `next(error)`. Pin that path so a regression surfaces here instead of in a downstream tracer test. * fix(express): restore error publishing for throwing middleware on express <4.6.0 On express <4.6.0 the router has no `Layer` prototype dispatch: it runs `layer.handle` directly and, on a synchronous throw, catches outside the layer and calls its own `next(error)` — never the wrapped `next` the tracer installed. The prototype-dispatch rewrite dropped the legacy handle wrap's catch, so a throwing middleware or route handler no longer published `middleware:error` / `middleware:next` / `middleware:finish`: the span lost its error tag and lingered on the stack until request finish. Restore the catch in `wrapLegacyHandle` to publish error/next/finish and rethrow, mirroring `wrapNext`. The prototype hosts (express >=4.6.0, express 5, router) already route the throw through the wrapped next, so only the legacy path needs it. Also correct the version boundary in the surrounding comments: `handle_request` landed in express 4.6.0, not 4.3.0, so the legacy path covers `>=4.0.0 <4.6.0`. Refs: #9067 (comment) * fix(express): publish middleware next/finish once per continuation A handler that calls `next()` and then rejects (`async (req, res, next) => { next(); await bg() }` where `bg()` rejects) makes the host invoke the same wrapped `next` twice: the clean `next()` finishes and pops the middleware span, then the host's rejection pass calls `next(error)` again. Since the middleware stack is already empty, that second pass tagged the error on the parent span, inventing a request-level error the middleware never surfaced. Guard the wrapped continuation so its error/next/finish publishes fire at most once per dispatch; the second call still forwards to the host chain untouched. Add the missing JSDoc on `hasLayerDispatch`.
Summary
The loopback integration test suite was silently skipped because the
withVersionscall passed the plugin export instead of the'express'integration name, so no installed version ever matched. Enabling it surfaced a
real regression: dd-trace replaces each express
layer.handlewith a tracingwrapper, which hides the original handler from loopback's
_findLayerByHandler.That lookup is how loopback tags a layer with its phase, so the tag was never
set and
app.middleware('final', ...)handlers ran in insertion order ahead ofthe route handlers instead of last.
fix(express): restore the_datadog_origback-reference on the wrappedlayer handle (dropped in the router instrumentation rewrite) so loopback can
map the layer back to the user handler and sort by phase again.
test(express): switch the suite to the'express'name so it runs, andallowlist legacy loopback's deprecated
util._extendload-time warning thatthe harness deprecation guard would otherwise throw on.
Note
I am working on a better fix, while that turned out to be quite a massive amount
of work, so I am suggesting we just land this as stop gap.
Refs: #643