fix(kafkajs): read clusterId from existing metadata, stop mutating user input#8253
fix(kafkajs): read clusterId from existing metadata, stop mutating user input#8253BridgeAR wants to merge 6 commits into
Conversation
Overall package sizeSelf size: 5.81 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.0.1 | 82.56 kB | 817.39 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
🎉 All green!❄️ No new flaky tests detected 🎯 Code Coverage (details) 🔗 Commit SHA: d06fc2e | Docs | Datadog PR Page | Give us feedback! |
bf29a34 to
67fa7ed
Compare
BenchmarksBenchmark execution time: 2026-05-09 15:16:05 Comparing candidate commit d06fc2e in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 1735 metrics, 109 unstable metrics. |
67fa7ed to
4fae935
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4fae935d57
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…st disable Two coordinated changes addressing review feedback on the producer wrap: 1. Replace the Symbol-keyed cluster reference (`producer[kCluster] = ...`) with a `WeakMap` exported from `helpers/kafka`. Symbols still mutate the kafkajs producer/consumer: they leak through `Reflect.ownKeys` / `Object.getOwnPropertySymbols`, force a hidden-class transition on the kafkajs-returned object, and trip on frozen producers. The side-table keeps the kafkajs object untouched. The integration test that downgrades the broker version reads the cluster through the same map instead of `Object.getOwnPropertySymbols`. 2. `refreshHeaderSupport` becomes a `let` that swaps to a module-level `noop` at both disable sites — proactive Produce <v3 detection and the reactive `KafkaJSProtocolError UNKNOWN` handler — so subsequent sends skip the helper call entirely. Drive-by fix: * Blank line between adjacent `it` blocks in `index.spec.js`. Refs: #8253
Match the wrapper shape already used in the kafkajs index spec for the same assertion. Refs: #8253
Brokers that fail when any header field is present — kafkajs's
`KafkaJSProtocolError UNKNOWN`, `@confluentinc/kafka-javascript`'s
`KafkaJSError ERR_UNKNOWN`, kafkajs negotiating Produce <v3 — also reject
empty `headers: {}`. The disable path was meant to recover for those brokers,
but `cloneMessagesForInjection` still seeded `{}` on every cloned message so
the client kept seeing a header field and the broker kept rejecting it.
Refs: #8253
…nsureHeaders)
The two clone helpers in `helpers/kafka.js` differed only in what they did
with messages that arrived without a `headers` field: one preserved the
absence, the other seeded `headers: {}` for the producer plugin to inject
into. Pass that as a second argument to a single `cloneMessages` and let
the call site decide based on `disableHeaderInjection`. Same hot-path cost,
one fewer code path to keep in sync.
Refs: #8253 (comment)
…st disable Two coordinated changes addressing review feedback on the producer wrap: 1. Replace the Symbol-keyed cluster reference (`producer[kCluster] = ...`) with a `WeakMap` exported from `helpers/kafka`. Symbols still mutate the kafkajs producer/consumer: they leak through `Reflect.ownKeys` / `Object.getOwnPropertySymbols`, force a hidden-class transition on the kafkajs-returned object, and trip on frozen producers. The side-table keeps the kafkajs object untouched. The integration test that downgrades the broker version reads the cluster through the same map instead of `Object.getOwnPropertySymbols`. 2. `refreshHeaderSupport` becomes a `let` that swaps to a module-level `noop` at both disable sites — proactive Produce <v3 detection and the reactive `KafkaJSProtocolError UNKNOWN` handler — so subsequent sends skip the helper call entirely. Drive-by fix: * Blank line between adjacent `it` blocks in `index.spec.js`. Refs: #8253
Match the wrapper shape already used in the kafkajs index spec for the same assertion. Refs: #8253
Brokers that fail when any header field is present — kafkajs's
`KafkaJSProtocolError UNKNOWN`, `@confluentinc/kafka-javascript`'s
`KafkaJSError ERR_UNKNOWN`, kafkajs negotiating Produce <v3 — also reject
empty `headers: {}`. The disable path was meant to recover for those brokers,
but `cloneMessagesForInjection` still seeded `{}` on every cloned message so
the client kept seeing a header field and the broker kept rejecting it.
Refs: #8253
a43f28d to
6f2463e
Compare
…nsureHeaders)
The two clone helpers in `helpers/kafka.js` differed only in what they did
with messages that arrived without a `headers` field: one preserved the
absence, the other seeded `headers: {}` for the producer plugin to inject
into. Pass that as a second argument to a single `cloneMessages` and let
the call site decide based on `disableHeaderInjection`. Same hot-path cost,
one fewer code path to keep in sync.
Refs: #8253 (comment)
…er input Three coordinated changes to the kafka producer wrap: 1. The cluster id discovery used a separate admin connection whose `describeCluster()` rejection wasn't awaited, surfacing as `process.unhandledRejection`. Read `cluster.brokerPool.metadata.clusterId` instead, priming it on first send via `cluster.refreshMetadataIfNecessary()`; `sharedPromiseTo` collapses our call with kafkajs's internal one, so total latency is unchanged. 2. Header injection mutated caller-owned `message.headers` in place. The boundary now hands kafkajs and the plugin a shallow clone instead. 3. The Produce request didn't carry headers until v3 (Kafka 0.11), so injecting on older brokers either wastes work or trips `UNKNOWN_SERVER_ERROR` on a per-leader version mismatch. Read `cluster.brokerPool.versions[0].maxVersion` and skip injection when the broker negotiated <v3; the existing UNKNOWN handler stays as a safety net for mixed-version clusters. Refs: #5837 Refs: #5270
…JS-compat path The KafkaJS-compat producer's `send` wrapper handed the caller's `payload.messages` array straight to the underlying client, so both trace-header injection and the client's own `headers: null` post-publish fixup mutated those objects in place. Shallow-clone the array on every send so neither path touches caller-owned objects; the native producer path already builds its own wrapper objects and is unaffected. The disabled-header tracking moves from a module-level WeakSet to a per-producer closure so the state's lifetime matches the producer it describes. Refs: #5837
…st disable Two coordinated changes addressing review feedback on the producer wrap: 1. Replace the Symbol-keyed cluster reference (`producer[kCluster] = ...`) with a `WeakMap` exported from `helpers/kafka`. Symbols still mutate the kafkajs producer/consumer: they leak through `Reflect.ownKeys` / `Object.getOwnPropertySymbols`, force a hidden-class transition on the kafkajs-returned object, and trip on frozen producers. The side-table keeps the kafkajs object untouched. The integration test that downgrades the broker version reads the cluster through the same map instead of `Object.getOwnPropertySymbols`. 2. `refreshHeaderSupport` becomes a `let` that swaps to a module-level `noop` at both disable sites — proactive Produce <v3 detection and the reactive `KafkaJSProtocolError UNKNOWN` handler — so subsequent sends skip the helper call entirely. Drive-by fix: * Blank line between adjacent `it` blocks in `index.spec.js`. Refs: #8253
Match the wrapper shape already used in the kafkajs index spec for the same assertion. Refs: #8253
Brokers that fail when any header field is present — kafkajs's
`KafkaJSProtocolError UNKNOWN`, `@confluentinc/kafka-javascript`'s
`KafkaJSError ERR_UNKNOWN`, kafkajs negotiating Produce <v3 — also reject
empty `headers: {}`. The disable path was meant to recover for those brokers,
but `cloneMessagesForInjection` still seeded `{}` on every cloned message so
the client kept seeing a header field and the broker kept rejecting it.
Refs: #8253
…nsureHeaders)
The two clone helpers in `helpers/kafka.js` differed only in what they did
with messages that arrived without a `headers` field: one preserved the
absence, the other seeded `headers: {}` for the producer plugin to inject
into. Pass that as a second argument to a single `cloneMessages` and let
the call site decide based on `disableHeaderInjection`. Same hot-path cost,
one fewer code path to keep in sync.
Refs: #8253 (comment)
6f2463e to
d06fc2e
Compare
Both kafkajs's producer wrap and the `@confluentinc/kafka-javascript`
KafkaJS-compat producer wrap mutated the caller's `message.headers` in
place. The kafkajs boundary seeded `headers: {}` on every message before
the plugin injected trace context; the confluent KafkaJS-compat path
handed `payload.messages` straight to the underlying client, which then
wrote `headers: null` on messages without headers. Either path leaves
the caller holding the side effects of someone else's instrumentation.
The boundary now hands kafkajs and the underlying confluent client a
shallow clone of the messages array via a single `cloneMessages`
helper. With injection enabled the helper seeds `headers: {}` on the
clone for the plugin to inject into; with injection disabled (after the
broker reported `KafkaJSProtocolError UNKNOWN` /
`KafkaJSError ERR_UNKNOWN`) the helper preserves the absence of
`headers` so brokers that reject any header field can recover.
Refs: #5270
Refs: #8253
The cluster id discovery used a separate `kafka.admin()` connection whose `describeCluster()` rejection wasn't awaited, surfacing as `process.unhandledRejection` whenever the admin handshake failed (auth, broker restart, network blip). Read `cluster.brokerPool.metadata.clusterId` instead, priming it on first send via `cluster.refreshMetadataIfNecessary()`; kafkajs's `sharedPromiseTo` collapses our call with its internal one, so total latency is unchanged. The cluster reference for each producer/consumer is captured at creation time via two new `addHook` entries on `src/producer/index.js` / `src/consumer/index.js` and stored in a `WeakMap` exported from `helpers/kafka`. The kafkajs object stays untouched — no Symbol-keyed property to leak through `Reflect.ownKeys`, no string-keyed underscore for user serializers to pick up, and the entry drops as soon as the producer is GC'd. Refs: #5270 Refs: #8253
|
Split into smaller PRs |
The cluster id discovery used a separate `kafka.admin()` connection whose `describeCluster()` rejection wasn't awaited, surfacing as `process.unhandledRejection` whenever the admin handshake failed (auth, broker restart, network blip). Read `cluster.brokerPool.metadata.clusterId` instead, priming it on first send via `cluster.refreshMetadataIfNecessary()`; kafkajs's `sharedPromiseTo` collapses our call with its internal one, so total latency is unchanged. The cluster reference for each producer/consumer is captured at creation time via two new `addHook` entries on `src/producer/index.js` / `src/consumer/index.js` and stored in a `WeakMap` exported from `helpers/kafka`. The kafkajs object stays untouched — no Symbol-keyed property to leak through `Reflect.ownKeys`, no string-keyed underscore for user serializers to pick up, and the entry drops as soon as the producer is GC'd. Refs: #5270 Refs: #8253
The Produce request didn't carry headers until v3 (Kafka 0.11), so injecting on older brokers either wastes work or trips `UNKNOWN_SERVER_ERROR` on a per-leader version mismatch in mixed-version clusters. Read `cluster.brokerPool.versions[0].maxVersion` once on first send and disable injection proactively when the broker negotiated <v3; the existing `KafkaJSProtocolError UNKNOWN` handler stays as the safety net for mixed-version clusters where the seed broker advertised v3+ but the leader couldn't parse the headers. `refreshHeaderSupport` is a `let` that swaps to module-level `noop` at both disable sites, so subsequent sends skip the helper call entirely. The reactive `disableHeaderInjection` moves from the module-level `disabledHeaderWeakSet` to a per-producer closure so the lifetime matches the producer it describes. Refs: #8253
The cluster id discovery used a separate `kafka.admin()` connection whose `describeCluster()` rejection wasn't awaited, surfacing as `process.unhandledRejection` whenever the admin handshake failed (auth, broker restart, network blip). Read `cluster.brokerPool.metadata.clusterId` instead, priming it on first send via `cluster.refreshMetadataIfNecessary()`; kafkajs's `sharedPromiseTo` collapses our call with its internal one, so total latency is unchanged. The cluster reference for each producer/consumer is captured at creation time via two new `addHook` entries on `src/producer/index.js` / `src/consumer/index.js` and stored in a `WeakMap` exported from `helpers/kafka`. The kafkajs object stays untouched — no Symbol-keyed property to leak through `Reflect.ownKeys`, no string-keyed underscore for user serializers to pick up, and the entry drops as soon as the producer is GC'd. Refs: #5270 Refs: #8253
Both kafkajs's producer wrap and the `@confluentinc/kafka-javascript`
KafkaJS-compat producer wrap mutated the caller's `message.headers` in
place. The kafkajs boundary seeded `headers: {}` on every message before
the plugin injected trace context; the confluent KafkaJS-compat path
handed `payload.messages` straight to the underlying client, which then
wrote `headers: null` on messages without headers. Either path leaves
the caller holding the side effects of someone else's instrumentation.
The boundary now hands kafkajs and the underlying confluent client a
shallow clone of the messages array via a single `cloneMessages`
helper. With injection enabled the helper seeds `headers: {}` on the
clone for the plugin to inject into; with injection disabled (after the
broker reported `KafkaJSProtocolError UNKNOWN` /
`KafkaJSError ERR_UNKNOWN`) the helper preserves the absence of
`headers` so brokers that reject any header field can recover.
Refs: #5270
Refs: #8253
The cluster id discovery used a separate `kafka.admin()` connection whose `describeCluster()` rejection wasn't awaited, surfacing as `process.unhandledRejection` whenever the admin handshake failed (auth, broker restart, network blip). Read `cluster.brokerPool.metadata.clusterId` instead, priming it on first send via `cluster.refreshMetadataIfNecessary()`; kafkajs's `sharedPromiseTo` collapses our call with its internal one, so total latency is unchanged. The cluster reference for each producer/consumer is captured at creation time via two new `addHook` entries on `src/producer/index.js` / `src/consumer/index.js` and stored in a `WeakMap` exported from `helpers/kafka`. The kafkajs object stays untouched — no Symbol-keyed property to leak through `Reflect.ownKeys`, no string-keyed underscore for user serializers to pick up, and the entry drops as soon as the producer is GC'd. Refs: #5270 Refs: #8253
Three coordinated changes to the kafka producer wrap:
The cluster id discovery used a separate admin connection whose
describeCluster()rejection wasn't awaited, surfacing asprocess.unhandledRejection. Readcluster.brokerPool.metadata.clusterIdinstead, priming it on firstsend via
cluster.refreshMetadataIfNecessary();sharedPromiseTocollapses our call with kafkajs's internal one, so total latency is
unchanged.
Header injection mutated caller-owned
message.headersin place. Theboundary now hands kafkajs and the plugin a shallow clone instead.
The Produce request didn't carry headers until v3 (Kafka 0.11), so
injecting on older brokers either wastes work or trips
UNKNOWN_SERVER_ERRORon a per-leader version mismatch. Readcluster.brokerPool.versions[0].maxVersionand skip injection whenthe broker negotiated <v3; the existing UNKNOWN handler stays as a
safety net for mixed-version clusters.
Refs: #5837
Refs: #5270