Skip to content

fix(kafkajs): read clusterId from existing metadata, stop mutating user input#8253

Closed
BridgeAR wants to merge 6 commits into
masterfrom
BridgeAR/2026-05-04-fix-kafka-clusterId
Closed

fix(kafkajs): read clusterId from existing metadata, stop mutating user input#8253
BridgeAR wants to merge 6 commits into
masterfrom
BridgeAR/2026-05-04-fix-kafka-clusterId

Conversation

@BridgeAR

@BridgeAR BridgeAR commented May 4, 2026

Copy link
Copy Markdown
Member

Three coordinated changes to the kafka producer wrap:

  1. The cluster id discovery used a separate admin connection whose
    describeCluster() rejection wasn't awaited, surfacing as
    process.unhandledRejection. Read
    cluster.brokerPool.metadata.clusterId instead, priming it on first
    send via cluster.refreshMetadataIfNecessary(); sharedPromiseTo
    collapses our call with kafkajs's internal one, so total latency is
    unchanged.

  2. Header injection mutated caller-owned message.headers in place. The
    boundary now hands kafkajs and the plugin a shallow clone instead.

  3. The Produce request didn't carry headers until v3 (Kafka 0.11), so
    injecting on older brokers either wastes work or trips
    UNKNOWN_SERVER_ERROR on a per-leader version mismatch. Read
    cluster.brokerPool.versions[0].maxVersion and skip injection when
    the broker negotiated <v3; the existing UNKNOWN handler stays as a
    safety net for mixed-version clusters.

Refs: #5837
Refs: #5270

@codecov

codecov Bot commented May 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.20301% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.97%. Comparing base (807fceb) to head (d06fc2e).

Files with missing lines Patch % Lines
packages/datadog-instrumentations/src/kafkajs.js 82.17% 18 Missing ⚠️
...trumentations/src/confluentinc-kafka-javascript.js 71.42% 4 Missing ⚠️
...ages/datadog-instrumentations/src/helpers/kafka.js 83.33% 3 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           master    #8253   +/-   ##
=======================================
  Coverage   89.96%   89.97%           
=======================================
  Files         836      837    +1     
  Lines       44867    44894   +27     
  Branches     8320     8325    +5     
=======================================
+ Hits        40365    40393   +28     
+ Misses       4502     4501    -1     
Flag Coverage Δ
aiguard-integration-active 40.55% <ø> (ø)
aiguard-integration-latest 40.50% <ø> (ø)
aiguard-integration-maintenance 40.55% <ø> (ø)
aiguard-macos 35.23% <ø> (-0.09%) ⬇️
aiguard-ubuntu 35.33% <ø> (-0.09%) ⬇️
aiguard-windows 35.14% <ø> (-0.09%) ⬇️
apm-capabilities-tracing-macos 49.04% <ø> (+0.04%) ⬆️
apm-capabilities-tracing-ubuntu-active 49.06% <ø> (ø)
apm-capabilities-tracing-ubuntu-latest 49.03% <ø> (ø)
apm-capabilities-tracing-ubuntu-maintenance 49.24% <ø> (ø)
apm-capabilities-tracing-ubuntu-oldest 49.05% <ø> (ø)
apm-capabilities-tracing-windows 48.86% <ø> (-0.01%) ⬇️
apm-integrations-aerospike-18-gte.5.2.0 34.27% <ø> (-0.09%) ⬇️
apm-integrations-aerospike-20-gte.5.5.0 34.28% <ø> (-0.09%) ⬇️
apm-integrations-aerospike-22-gte.5.12.1 34.28% <ø> (-0.09%) ⬇️
apm-integrations-aerospike-22-gte.6.0.0 34.28% <ø> (-0.09%) ⬇️
apm-integrations-aerospike-eol- 34.19% <ø> (-0.09%) ⬇️
apm-integrations-child-process 35.25% <ø> (-0.09%) ⬇️
apm-integrations-confluentinc-kafka-javascript-18 41.15% <77.77%> (-0.03%) ⬇️
apm-integrations-confluentinc-kafka-javascript-20 41.17% <77.77%> (-0.03%) ⬇️
apm-integrations-confluentinc-kafka-javascript-22 41.17% <77.77%> (-0.03%) ⬇️
apm-integrations-confluentinc-kafka-javascript-24 41.10% <77.77%> (-0.03%) ⬇️
apm-integrations-couchbase-18 34.59% <ø> (-0.17%) ⬇️
apm-integrations-couchbase-eol 34.69% <ø> (-0.09%) ⬇️
apm-integrations-dns 34.13% <ø> (-0.09%) ⬇️
apm-integrations-elasticsearch 34.93% <ø> (-0.08%) ⬇️
apm-integrations-http-latest 42.08% <ø> (-0.08%) ⬇️
apm-integrations-http-maintenance 42.14% <ø> (-0.08%) ⬇️
apm-integrations-http-oldest 42.14% <ø> (-0.08%) ⬇️
apm-integrations-http2 39.45% <ø> (-0.09%) ⬇️
apm-integrations-kafkajs-latest 41.08% <88.99%> (+0.02%) ⬆️
apm-integrations-kafkajs-oldest 41.13% <88.99%> (+0.01%) ⬆️
apm-integrations-net 34.82% <ø> (-0.09%) ⬇️
apm-integrations-next-11.1.4 29.06% <ø> (-0.06%) ⬇️
apm-integrations-next-13.2.0 30.72% <ø> (-0.08%) ⬇️
apm-integrations-next-gte.10.2.0.and.lt.11 22.99% <ø> (ø)
apm-integrations-next-gte.11.0.0.and.lt.13 30.73% <ø> (-0.08%) ⬇️
apm-integrations-next-gte.13.0.0.and.lt.14 30.97% <ø> (-0.08%) ⬇️
apm-integrations-next-gte.14.0.0.and.lte.14.2.6 30.79% <ø> (-0.08%) ⬇️
apm-integrations-next-gte.14.2.7.and.lt.15 30.79% <ø> (-0.08%) ⬇️
apm-integrations-next-gte.15.0.0 30.86% <ø> (-0.08%) ⬇️
apm-integrations-oracledb 34.57% <ø> (-0.09%) ⬇️
apm-integrations-prisma-18-gte.6.16.0.and.lt.7.0.0 34.95% <ø> (-0.08%) ⬇️
apm-integrations-prisma-latest-all 35.36% <ø> (-0.08%) ⬇️
apm-integrations-restify 36.32% <ø> (-0.09%) ⬇️
apm-integrations-sharedb 33.81% <ø> (-0.09%) ⬇️
apm-integrations-tedious 34.37% <ø> (-0.09%) ⬇️
appsec-express 51.91% <ø> (-0.07%) ⬇️
appsec-fastify 48.62% <ø> (-0.07%) ⬇️
appsec-graphql 48.90% <ø> (-0.06%) ⬇️
appsec-integration-active 36.90% <7.51%> (-0.02%) ⬇️
appsec-integration-latest 36.87% <7.51%> (-0.02%) ⬇️
appsec-integration-maintenance 36.89% <7.51%> (-0.02%) ⬇️
appsec-integration-oldest 36.89% <7.51%> (-0.02%) ⬇️
appsec-kafka 41.43% <68.80%> (-0.04%) ⬇️
appsec-ldapjs 40.65% <ø> (-0.07%) ⬇️
appsec-lodash 40.76% <ø> (-0.07%) ⬇️
appsec-macos 55.75% <ø> (-0.06%) ⬇️
appsec-mongodb-core 45.09% <ø> (-0.07%) ⬇️
appsec-mongoose 45.95% <ø> (-0.07%) ⬇️
appsec-mysql 47.99% <ø> (-0.02%) ⬇️
appsec-next-latest-11.1.4 29.15% <ø> (-0.08%) ⬇️
appsec-next-latest-13.2.0 30.86% <ø> (-0.09%) ⬇️
appsec-next-latest-gte.10.2.0.and.lt.11 28.66% <ø> (ø)
appsec-next-latest-gte.11.0.0.and.lt.13 30.84% <ø> (-0.09%) ⬇️
appsec-next-latest-gte.13.0.0.and.lt.14 31.09% <ø> (-0.09%) ⬇️
appsec-next-latest-gte.14.0.0.and.lte.14.2.6 30.93% <ø> (-0.09%) ⬇️
appsec-next-latest-gte.14.2.7.and.lt.15 30.93% <ø> (-0.09%) ⬇️
appsec-next-latest-gte.15.0.0 30.93% <ø> (-0.09%) ⬇️
appsec-next-oldest-11.1.4 29.16% <ø> (-0.08%) ⬇️
appsec-next-oldest-13.2.0 31.14% <ø> (-0.09%) ⬇️
appsec-next-oldest-gte.10.2.0.and.lt.11 28.78% <ø> (ø)
appsec-next-oldest-gte.11.0.0.and.lt.13 30.86% <ø> (-0.09%) ⬇️
appsec-next-oldest-gte.13.0.0.and.lt.14 31.33% <ø> (-0.08%) ⬇️
appsec-next-oldest-gte.14.0.0.and.lte.14.2.6 31.17% <ø> (-0.09%) ⬇️
appsec-next-oldest-gte.14.2.7.and.lt.15 31.17% <ø> (-0.09%) ⬇️
appsec-next-oldest-gte.15.0.0 31.17% <ø> (-0.09%) ⬇️
appsec-node-serialize 39.97% <ø> (-0.07%) ⬇️
appsec-passport 43.63% <ø> (-0.08%) ⬇️
appsec-postgres 47.54% <ø> (-0.07%) ⬇️
appsec-sourcing 39.31% <ø> (-0.07%) ⬇️
appsec-stripe 41.41% <ø> (-0.08%) ⬇️
appsec-template 40.13% <ø> (-0.07%) ⬇️
appsec-ubuntu 55.83% <ø> (-0.07%) ⬇️
appsec-windows 55.69% <ø> (-0.06%) ⬇️
debugger-ubuntu-active 43.18% <ø> (ø)
debugger-ubuntu-latest 43.13% <ø> (ø)
debugger-ubuntu-maintenance 43.62% <ø> (ø)
debugger-ubuntu-oldest 43.63% <ø> (ø)
instrumentations-instrumentation-bluebird 29.25% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-body-parser 36.86% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-child_process 34.74% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-cookie-parser 31.09% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-express 31.31% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-express-mongo-sanitize 31.21% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-express-session 36.55% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-fs 28.94% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-generic-pool 29.86% <ø> (ø)
instrumentations-instrumentation-http 36.22% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-knex 29.22% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-light-my-request 36.48% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-mongoose 30.35% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-multer 36.65% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-mysql2 34.77% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-passport 40.27% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-passport-http 40.05% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-passport-local 40.53% <ø> (-0.08%) ⬇️
instrumentations-instrumentation-pg 34.32% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-promise 29.19% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-promise-js 29.19% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-q 29.22% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-url 29.19% <ø> (-0.09%) ⬇️
instrumentations-instrumentation-when 29.21% <ø> (-0.09%) ⬇️
instrumentations-integration-esbuild-active 19.31% <7.51%> (+<0.01%) ⬆️
instrumentations-integration-esbuild-latest 19.29% <7.51%> (+<0.01%) ⬆️
instrumentations-integration-esbuild-maintenance 19.31% <7.51%> (+<0.01%) ⬆️
instrumentations-integration-esbuild-oldest 19.30% <7.51%> (+<0.01%) ⬆️
llmobs-ai 37.69% <ø> (-0.08%) ⬇️
llmobs-anthropic 37.75% <ø> (-0.08%) ⬇️
llmobs-bedrock 36.45% <ø> (-0.08%) ⬇️
llmobs-google-genai 36.77% <ø> (-0.08%) ⬇️
llmobs-langchain 36.48% <ø> (-0.07%) ⬇️
llmobs-openai 40.52% <ø> (-0.08%) ⬇️
llmobs-sdk-active 45.33% <ø> (-0.09%) ⬇️
llmobs-sdk-latest 45.26% <ø> (-0.09%) ⬇️
llmobs-sdk-maintenance 45.33% <ø> (-0.09%) ⬇️
llmobs-sdk-oldest 45.31% <ø> (-0.09%) ⬇️
llmobs-vertex-ai 36.92% <ø> (-0.08%) ⬇️
openfeature-macos 37.92% <ø> (ø)
openfeature-ubuntu 38.00% <ø> (ø)
openfeature-unit-active 48.51% <ø> (ø)
openfeature-unit-latest 48.35% <ø> (ø)
openfeature-unit-maintenance 48.51% <ø> (ø)
openfeature-unit-oldest 48.51% <ø> (ø)
openfeature-windows 37.80% <ø> (+0.05%) ⬆️
platform-core 37.27% <ø> (ø)
platform-esbuild 40.61% <ø> (ø)
platform-instrumentations-misc 30.77% <100.00%> (+0.08%) ⬆️
platform-integration-active 46.93% <ø> (ø)
platform-integration-latest 46.89% <ø> (ø)
platform-integration-maintenance 46.94% <ø> (ø)
platform-integration-oldest 47.10% <ø> (-0.01%) ⬇️
platform-shimmer 42.29% <ø> (ø)
platform-unit-guardrails 36.12% <ø> (ø)
platform-webpack 20.44% <8.26%> (+<0.01%) ⬆️
plugins-azure-durable-functions 36.74% <ø> (-0.17%) ⬇️
plugins-azure-event-hubs 34.75% <ø> (ø)
plugins-azure-service-bus 35.22% <ø> (ø)
plugins-bullmq 40.01% <ø> (-0.09%) ⬇️
plugins-cassandra 34.58% <ø> (-0.08%) ⬇️
plugins-cookie 26.65% <ø> (?)
plugins-cookie-parser 26.46% <ø> (ø)
plugins-crypto 27.69% <ø> (ø)
plugins-dd-trace-api 34.67% <ø> (-0.09%) ⬇️
plugins-express-mongo-sanitize 26.59% <ø> (ø)
plugins-express-session 26.42% <ø> (ø)
plugins-fastify 38.63% <ø> (-0.08%) ⬇️
plugins-fetch 34.96% <ø> (-0.08%) ⬇️
plugins-fs 35.02% <ø> (-0.09%) ⬇️
plugins-generic-pool 25.58% <ø> (ø)
plugins-google-cloud-pubsub 42.40% <ø> (-0.06%) ⬇️
plugins-grpc 37.40% <ø> (-0.08%) ⬇️
plugins-handlebars 26.63% <ø> (ø)
plugins-hapi 36.52% <ø> (-0.09%) ⬇️
plugins-hono 36.81% <ø> (-0.08%) ⬇️
plugins-ioredis 35.10% <ø> (-0.09%) ⬇️
plugins-knex 26.31% <ø> (ø)
plugins-langgraph 34.38% <ø> (-0.09%) ⬇️
plugins-ldapjs 24.20% <ø> (ø)
plugins-light-my-request 26.06% <ø> (ø)
plugins-limitd-client 29.49% <ø> (-0.09%) ⬇️
plugins-lodash 25.65% <ø> (ø)
plugins-mariadb 36.06% <ø> (-0.09%) ⬇️
plugins-memcached 34.76% <ø> (-0.09%) ⬇️
plugins-microgateway-core 35.65% <ø> (-0.09%) ⬇️
plugins-modelcontextprotocol-sdk 33.66% <ø> (-0.09%) ⬇️
plugins-moleculer 37.48% <ø> (-0.09%) ⬇️
plugins-mongodb 35.95% <ø> (-0.08%) ⬇️
plugins-mongodb-core 35.67% <ø> (-0.09%) ⬇️
plugins-mongoose 35.46% <ø> (-0.07%) ⬇️
plugins-multer 26.42% <ø> (ø)
plugins-mysql 35.60% <ø> (-0.22%) ⬇️
plugins-mysql2 35.82% <ø> (-0.09%) ⬇️
plugins-node-serialize 26.69% <ø> (ø)
plugins-opensearch 34.46% <ø> (-0.09%) ⬇️
plugins-passport-http 26.47% <ø> (ø)
plugins-pino 31.19% <ø> (-0.09%) ⬇️
plugins-postgres 33.84% <ø> (-0.08%) ⬇️
plugins-process 27.69% <ø> (ø)
plugins-pug 26.65% <ø> (ø)
plugins-redis 35.16% <ø> (-0.03%) ⬇️
plugins-router 39.10% <ø> (-0.08%) ⬇️
plugins-sequelize 25.36% <ø> (ø)
plugins-test-and-upstream-amqp10 35.00% <ø> (-0.09%) ⬇️
plugins-test-and-upstream-amqplib 40.23% <ø> (-0.09%) ⬇️
plugins-test-and-upstream-apollo 35.87% <ø> (-0.08%) ⬇️
plugins-test-and-upstream-avsc 34.34% <ø> (-0.09%) ⬇️
plugins-test-and-upstream-bunyan 30.57% <ø> (-0.09%) ⬇️
plugins-test-and-upstream-connect 37.19% <ø> (-0.09%) ⬇️
plugins-test-and-upstream-graphql 36.76% <ø> (-0.11%) ⬇️
plugins-test-and-upstream-koa 36.78% <ø> (-0.08%) ⬇️
plugins-test-and-upstream-protobufjs 34.55% <ø> (-0.09%) ⬇️
plugins-test-and-upstream-rhea 40.24% <ø> (-0.09%) ⬇️
plugins-undici 35.69% <ø> (-0.08%) ⬇️
plugins-url 27.69% <ø> (ø)
plugins-valkey 34.61% <ø> (-0.24%) ⬇️
plugins-vm 27.69% <ø> (ø)
plugins-winston 31.08% <ø> (-0.09%) ⬇️
plugins-ws 38.30% <ø> (+0.05%) ⬆️
profiling-macos 42.65% <ø> (-0.12%) ⬇️
profiling-ubuntu 43.06% <ø> (-0.07%) ⬇️
profiling-windows 39.96% <ø> (-0.07%) ⬇️
serverless-aws-sdk-latest-aws-sdk 34.66% <ø> (-0.07%) ⬇️
serverless-aws-sdk-latest-bedrockruntime 33.00% <ø> (-0.08%) ⬇️
serverless-aws-sdk-latest-client 22.38% <ø> (ø)
serverless-aws-sdk-latest-dynamodb 35.62% <ø> (-0.08%) ⬇️
serverless-aws-sdk-latest-eventbridge 28.76% <ø> (-0.08%) ⬇️
serverless-aws-sdk-latest-kinesis 38.56% <ø> (-0.08%) ⬇️
serverless-aws-sdk-latest-lambda 35.81% <ø> (-0.08%) ⬇️
serverless-aws-sdk-latest-s3 ?
serverless-aws-sdk-latest-serverless-peer-service 39.93% <ø> (-0.09%) ⬇️
serverless-aws-sdk-latest-sns 39.66% <ø> (-0.08%) ⬇️
serverless-aws-sdk-latest-sqs 38.94% <ø> (-0.08%) ⬇️
serverless-aws-sdk-latest-stepfunctions 34.38% <ø> (-0.08%) ⬇️
serverless-aws-sdk-latest-util 47.71% <ø> (ø)
serverless-aws-sdk-oldest-aws-sdk 34.75% <ø> (-0.07%) ⬇️
serverless-aws-sdk-oldest-bedrockruntime 33.28% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-client 22.75% <ø> (ø)
serverless-aws-sdk-oldest-dynamodb 35.69% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-eventbridge 28.81% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-kinesis 38.69% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-lambda 35.89% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-s3 33.83% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-serverless-peer-service 39.99% <ø> (-0.09%) ⬇️
serverless-aws-sdk-oldest-sns 39.86% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-sqs 38.83% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-stepfunctions 34.46% <ø> (-0.08%) ⬇️
serverless-aws-sdk-oldest-util 48.03% <ø> (ø)
serverless-azure-functions-eventhubs 38.40% <ø> (ø)
serverless-azure-functions-servicebus 38.46% <ø> (ø)
serverless-lambda 32.76% <ø> (-0.08%) ⬇️
test-optimization-cucumber-latest-7.0.0 ?
test-optimization-cucumber-latest-latest 53.05% <ø> (+0.12%) ⬆️
test-optimization-cucumber-oldest-7.0.0 50.35% <ø> (+0.12%) ⬆️
test-optimization-cypress-latest-12.0.0-commonJS 48.25% <ø> (+0.07%) ⬆️
test-optimization-cypress-latest-12.0.0-esm 48.31% <ø> (+0.10%) ⬆️
test-optimization-cypress-latest-14.5.4-commonJS 48.10% <ø> (+0.07%) ⬆️
test-optimization-cypress-latest-14.5.4-esm 48.13% <ø> (+0.07%) ⬆️
test-optimization-cypress-latest-latest-commonJS 48.60% <ø> (+0.07%) ⬆️
test-optimization-cypress-latest-latest-esm 48.63% <ø> (+0.07%) ⬆️
test-optimization-cypress-oldest-12.0.0-commonJS 48.29% <ø> (+0.07%) ⬆️
test-optimization-cypress-oldest-12.0.0-esm 48.32% <ø> (+0.07%) ⬆️
test-optimization-cypress-oldest-14.5.4-commonJS 48.14% <ø> (+0.07%) ⬆️
test-optimization-cypress-oldest-14.5.4-esm 48.17% <ø> (+0.07%) ⬆️
test-optimization-jest-latest-latest 54.82% <ø> (+0.08%) ⬆️
test-optimization-jest-latest-oldest 53.60% <ø> (+0.05%) ⬆️
test-optimization-jest-oldest-latest 54.82% <ø> (+0.08%) ⬆️
test-optimization-jest-oldest-oldest 53.56% <ø> (+0.08%) ⬆️
test-optimization-mocha-latest-latest 53.33% <ø> (+0.08%) ⬆️
test-optimization-mocha-latest-oldest 50.91% <ø> (+0.06%) ⬆️
test-optimization-mocha-oldest-latest 53.39% <ø> (+0.08%) ⬆️
test-optimization-mocha-oldest-oldest 50.88% <ø> (+0.08%) ⬆️
test-optimization-playwright-latest-latest-playwright-active-test-span 44.24% <ø> (+0.28%) ⬆️
test-optimization-playwright-latest-latest-playwright-atr 43.00% <ø> (+0.11%) ⬆️
test-optimization-playwright-latest-latest-playwright-efd 43.38% <ø> (+0.09%) ⬆️
test-optimization-playwright-latest-latest-playwright-final-status 43.45% <ø> (+0.11%) ⬆️
test-optimization-playwright-latest-latest-playwright-impacted-tests 42.93% <ø> (ø)
test-optimization-playwright-latest-latest-playwright-reporting 42.42% <ø> (+0.09%) ⬆️
test-optimization-playwright-latest-latest-playwright-test-management 44.61% <ø> (+0.10%) ⬆️
test-optimization-playwright-latest-oldest-playwright-active-test-span 44.30% <ø> (+0.28%) ⬆️
test-optimization-playwright-latest-oldest-playwright-atr 43.21% <ø> (+0.11%) ⬆️
test-optimization-playwright-latest-oldest-playwright-efd 43.43% <ø> (+0.09%) ⬆️
test-optimization-playwright-latest-oldest-playwright-final-status 43.50% <ø> (+0.11%) ⬆️
test-optimization-playwright-latest-oldest-playwright-impacted-tests 42.98% <ø> (ø)
test-optimization-playwright-latest-oldest-playwright-reporting 42.49% <ø> (+0.09%) ⬆️
test-optimization-playwright-latest-oldest-playwright-test-management 44.68% <ø> (+0.10%) ⬆️
test-optimization-playwright-oldest-latest-playwright-active-test-span 44.25% <ø> (+0.26%) ⬆️
test-optimization-playwright-oldest-latest-playwright-atr 43.04% <ø> (+0.11%) ⬆️
test-optimization-playwright-oldest-latest-playwright-efd 43.39% <ø> (+0.09%) ⬆️
test-optimization-playwright-oldest-latest-playwright-final-status 43.48% <ø> (+0.11%) ⬆️
test-optimization-playwright-oldest-latest-playwright-impacted-tests 42.97% <ø> (ø)
test-optimization-playwright-oldest-latest-playwright-reporting 42.44% <ø> (+0.09%) ⬆️
test-optimization-playwright-oldest-latest-playwright-test-management 44.62% <ø> (+0.10%) ⬆️
test-optimization-playwright-oldest-oldest-playwright-active-test-span 44.34% <ø> (+0.28%) ⬆️
test-optimization-playwright-oldest-oldest-playwright-atr 43.25% <ø> (+0.11%) ⬆️
test-optimization-playwright-oldest-oldest-playwright-efd 43.44% <ø> (+0.09%) ⬆️
test-optimization-playwright-oldest-oldest-playwright-final-status 43.54% <ø> (+0.11%) ⬆️
test-optimization-playwright-oldest-oldest-playwright-impacted-tests 43.02% <ø> (ø)
test-optimization-playwright-oldest-oldest-playwright-reporting 42.50% <ø> (+0.09%) ⬆️
test-optimization-playwright-oldest-oldest-playwright-test-management 44.70% <ø> (+0.10%) ⬆️
test-optimization-selenium-latest 45.39% <ø> (-0.16%) ⬇️
test-optimization-selenium-oldest 45.35% <ø> (+0.31%) ⬆️
test-optimization-testopt-active 46.98% <ø> (+0.13%) ⬆️
test-optimization-testopt-latest 46.94% <ø> (+0.13%) ⬆️
test-optimization-testopt-maintenance 46.98% <ø> (+0.13%) ⬆️
test-optimization-testopt-oldest 47.85% <ø> (+0.14%) ⬆️
test-optimization-vitest-latest 50.95% <ø> (+0.11%) ⬆️
test-optimization-vitest-oldest 47.68% <ø> (+0.33%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented May 4, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 5.81 MB
Deduped: 6.65 MB
No deduping: 6.65 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.0.1 | 82.56 kB | 817.39 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented May 4, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 81.97%
Overall Coverage: 86.73% (-0.02%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d06fc2e | Docs | Datadog PR Page | Give us feedback!

@BridgeAR
BridgeAR force-pushed the BridgeAR/2026-05-04-fix-kafka-clusterId branch from bf29a34 to 67fa7ed Compare May 4, 2026 16:32
@pr-commenter

pr-commenter Bot commented May 4, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-05-09 15:16:05

Comparing candidate commit d06fc2e in PR branch BridgeAR/2026-05-04-fix-kafka-clusterId with baseline commit 807fceb in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 1735 metrics, 109 unstable metrics.

@BridgeAR
BridgeAR force-pushed the BridgeAR/2026-05-04-fix-kafka-clusterId branch from 67fa7ed to 4fae935 Compare May 5, 2026 09:51
Comment thread packages/datadog-instrumentations/src/kafkajs.js Outdated
Comment thread packages/datadog-instrumentations/src/kafkajs.js
Comment thread packages/datadog-plugin-confluentinc-kafka-javascript/test/dsm.spec.js Outdated
Comment thread packages/datadog-plugin-kafkajs/test/index.spec.js
@BridgeAR
BridgeAR marked this pull request as ready for review May 5, 2026 13:41
@BridgeAR
BridgeAR requested review from a team as code owners May 5, 2026 13:41

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4fae935d57

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

BridgeAR added a commit that referenced this pull request May 5, 2026
…st disable

Two coordinated changes addressing review feedback on the producer wrap:

1. Replace the Symbol-keyed cluster reference (`producer[kCluster] = ...`)
   with a `WeakMap` exported from `helpers/kafka`. Symbols still mutate the
   kafkajs producer/consumer: they leak through `Reflect.ownKeys` /
   `Object.getOwnPropertySymbols`, force a hidden-class transition on the
   kafkajs-returned object, and trip on frozen producers. The side-table
   keeps the kafkajs object untouched. The integration test that downgrades
   the broker version reads the cluster through the same map instead of
   `Object.getOwnPropertySymbols`.

2. `refreshHeaderSupport` becomes a `let` that swaps to a module-level
   `noop` at both disable sites — proactive Produce <v3 detection and the
   reactive `KafkaJSProtocolError UNKNOWN` handler — so subsequent sends
   skip the helper call entirely.

Drive-by fix:

* Blank line between adjacent `it` blocks in `index.spec.js`.

Refs: #8253
BridgeAR added a commit that referenced this pull request May 5, 2026
Match the wrapper shape already used in the kafkajs index spec for the
same assertion.

Refs: #8253
BridgeAR added a commit that referenced this pull request May 5, 2026
Brokers that fail when any header field is present — kafkajs's
`KafkaJSProtocolError UNKNOWN`, `@confluentinc/kafka-javascript`'s
`KafkaJSError ERR_UNKNOWN`, kafkajs negotiating Produce <v3 — also reject
empty `headers: {}`. The disable path was meant to recover for those brokers,
but `cloneMessagesForInjection` still seeded `{}` on every cloned message so
the client kept seeing a header field and the broker kept rejecting it.

Refs: #8253
Comment thread packages/datadog-instrumentations/src/kafkajs.js
Comment thread packages/datadog-instrumentations/src/kafkajs.js
Comment thread packages/datadog-instrumentations/src/helpers/kafka.js Outdated
BridgeAR added a commit that referenced this pull request May 5, 2026
…nsureHeaders)

The two clone helpers in `helpers/kafka.js` differed only in what they did
with messages that arrived without a `headers` field: one preserved the
absence, the other seeded `headers: {}` for the producer plugin to inject
into. Pass that as a second argument to a single `cloneMessages` and let
the call site decide based on `disableHeaderInjection`. Same hot-path cost,
one fewer code path to keep in sync.

Refs: #8253 (comment)
BridgeAR added a commit that referenced this pull request May 8, 2026
…st disable

Two coordinated changes addressing review feedback on the producer wrap:

1. Replace the Symbol-keyed cluster reference (`producer[kCluster] = ...`)
   with a `WeakMap` exported from `helpers/kafka`. Symbols still mutate the
   kafkajs producer/consumer: they leak through `Reflect.ownKeys` /
   `Object.getOwnPropertySymbols`, force a hidden-class transition on the
   kafkajs-returned object, and trip on frozen producers. The side-table
   keeps the kafkajs object untouched. The integration test that downgrades
   the broker version reads the cluster through the same map instead of
   `Object.getOwnPropertySymbols`.

2. `refreshHeaderSupport` becomes a `let` that swaps to a module-level
   `noop` at both disable sites — proactive Produce <v3 detection and the
   reactive `KafkaJSProtocolError UNKNOWN` handler — so subsequent sends
   skip the helper call entirely.

Drive-by fix:

* Blank line between adjacent `it` blocks in `index.spec.js`.

Refs: #8253
BridgeAR added a commit that referenced this pull request May 8, 2026
Match the wrapper shape already used in the kafkajs index spec for the
same assertion.

Refs: #8253
BridgeAR added a commit that referenced this pull request May 8, 2026
Brokers that fail when any header field is present — kafkajs's
`KafkaJSProtocolError UNKNOWN`, `@confluentinc/kafka-javascript`'s
`KafkaJSError ERR_UNKNOWN`, kafkajs negotiating Produce <v3 — also reject
empty `headers: {}`. The disable path was meant to recover for those brokers,
but `cloneMessagesForInjection` still seeded `{}` on every cloned message so
the client kept seeing a header field and the broker kept rejecting it.

Refs: #8253
@BridgeAR
BridgeAR force-pushed the BridgeAR/2026-05-04-fix-kafka-clusterId branch from a43f28d to 6f2463e Compare May 8, 2026 11:17
BridgeAR added a commit that referenced this pull request May 8, 2026
…nsureHeaders)

The two clone helpers in `helpers/kafka.js` differed only in what they did
with messages that arrived without a `headers` field: one preserved the
absence, the other seeded `headers: {}` for the producer plugin to inject
into. Pass that as a second argument to a single `cloneMessages` and let
the call site decide based on `disableHeaderInjection`. Same hot-path cost,
one fewer code path to keep in sync.

Refs: #8253 (comment)
@tlhunter tlhunter changed the title fix(kafkajs): read clusterId from existing metadata, stop mutating us… fix(kafkajs): read clusterId from existing metadata, stop mutating user input May 8, 2026
BridgeAR added 6 commits May 9, 2026 16:52
…er input

Three coordinated changes to the kafka producer wrap:

1. The cluster id discovery used a separate admin connection whose
   `describeCluster()` rejection wasn't awaited, surfacing as
   `process.unhandledRejection`. Read
   `cluster.brokerPool.metadata.clusterId` instead, priming it on first
   send via `cluster.refreshMetadataIfNecessary()`; `sharedPromiseTo`
   collapses our call with kafkajs's internal one, so total latency is
   unchanged.

2. Header injection mutated caller-owned `message.headers` in place. The
   boundary now hands kafkajs and the plugin a shallow clone instead.

3. The Produce request didn't carry headers until v3 (Kafka 0.11), so
   injecting on older brokers either wastes work or trips
   `UNKNOWN_SERVER_ERROR` on a per-leader version mismatch. Read
   `cluster.brokerPool.versions[0].maxVersion` and skip injection when
   the broker negotiated <v3; the existing UNKNOWN handler stays as a
   safety net for mixed-version clusters.

Refs: #5837
Refs: #5270
…JS-compat path

The KafkaJS-compat producer's `send` wrapper handed the caller's
`payload.messages` array straight to the underlying client, so both
trace-header injection and the client's own `headers: null` post-publish
fixup mutated those objects in place. Shallow-clone the array on every
send so neither path touches caller-owned objects; the native producer
path already builds its own wrapper objects and is unaffected.

The disabled-header tracking moves from a module-level WeakSet to a
per-producer closure so the state's lifetime matches the producer it
describes.

Refs: #5837
…st disable

Two coordinated changes addressing review feedback on the producer wrap:

1. Replace the Symbol-keyed cluster reference (`producer[kCluster] = ...`)
   with a `WeakMap` exported from `helpers/kafka`. Symbols still mutate the
   kafkajs producer/consumer: they leak through `Reflect.ownKeys` /
   `Object.getOwnPropertySymbols`, force a hidden-class transition on the
   kafkajs-returned object, and trip on frozen producers. The side-table
   keeps the kafkajs object untouched. The integration test that downgrades
   the broker version reads the cluster through the same map instead of
   `Object.getOwnPropertySymbols`.

2. `refreshHeaderSupport` becomes a `let` that swaps to a module-level
   `noop` at both disable sites — proactive Produce <v3 detection and the
   reactive `KafkaJSProtocolError UNKNOWN` handler — so subsequent sends
   skip the helper call entirely.

Drive-by fix:

* Blank line between adjacent `it` blocks in `index.spec.js`.

Refs: #8253
Match the wrapper shape already used in the kafkajs index spec for the
same assertion.

Refs: #8253
Brokers that fail when any header field is present — kafkajs's
`KafkaJSProtocolError UNKNOWN`, `@confluentinc/kafka-javascript`'s
`KafkaJSError ERR_UNKNOWN`, kafkajs negotiating Produce <v3 — also reject
empty `headers: {}`. The disable path was meant to recover for those brokers,
but `cloneMessagesForInjection` still seeded `{}` on every cloned message so
the client kept seeing a header field and the broker kept rejecting it.

Refs: #8253
…nsureHeaders)

The two clone helpers in `helpers/kafka.js` differed only in what they did
with messages that arrived without a `headers` field: one preserved the
absence, the other seeded `headers: {}` for the producer plugin to inject
into. Pass that as a second argument to a single `cloneMessages` and let
the call site decide based on `disableHeaderInjection`. Same hot-path cost,
one fewer code path to keep in sync.

Refs: #8253 (comment)
@BridgeAR
BridgeAR force-pushed the BridgeAR/2026-05-04-fix-kafka-clusterId branch from 6f2463e to d06fc2e Compare May 9, 2026 14:59
BridgeAR added a commit that referenced this pull request May 11, 2026
Both kafkajs's producer wrap and the `@confluentinc/kafka-javascript`
KafkaJS-compat producer wrap mutated the caller's `message.headers` in
place. The kafkajs boundary seeded `headers: {}` on every message before
the plugin injected trace context; the confluent KafkaJS-compat path
handed `payload.messages` straight to the underlying client, which then
wrote `headers: null` on messages without headers. Either path leaves
the caller holding the side effects of someone else's instrumentation.

The boundary now hands kafkajs and the underlying confluent client a
shallow clone of the messages array via a single `cloneMessages`
helper. With injection enabled the helper seeds `headers: {}` on the
clone for the plugin to inject into; with injection disabled (after the
broker reported `KafkaJSProtocolError UNKNOWN` /
`KafkaJSError ERR_UNKNOWN`) the helper preserves the absence of
`headers` so brokers that reject any header field can recover.

Refs: #5270
Refs: #8253
BridgeAR added a commit that referenced this pull request May 11, 2026
The cluster id discovery used a separate `kafka.admin()` connection whose
`describeCluster()` rejection wasn't awaited, surfacing as
`process.unhandledRejection` whenever the admin handshake failed (auth,
broker restart, network blip). Read
`cluster.brokerPool.metadata.clusterId` instead, priming it on first send
via `cluster.refreshMetadataIfNecessary()`; kafkajs's `sharedPromiseTo`
collapses our call with its internal one, so total latency is unchanged.

The cluster reference for each producer/consumer is captured at creation
time via two new `addHook` entries on `src/producer/index.js` /
`src/consumer/index.js` and stored in a `WeakMap` exported from
`helpers/kafka`. The kafkajs object stays untouched — no Symbol-keyed
property to leak through `Reflect.ownKeys`, no string-keyed underscore
for user serializers to pick up, and the entry drops as soon as the
producer is GC'd.

Refs: #5270
Refs: #8253
@BridgeAR

Copy link
Copy Markdown
Member Author

Split into smaller PRs

@BridgeAR BridgeAR closed this May 11, 2026
BridgeAR added a commit that referenced this pull request May 12, 2026
The cluster id discovery used a separate `kafka.admin()` connection whose
`describeCluster()` rejection wasn't awaited, surfacing as
`process.unhandledRejection` whenever the admin handshake failed (auth,
broker restart, network blip). Read
`cluster.brokerPool.metadata.clusterId` instead, priming it on first send
via `cluster.refreshMetadataIfNecessary()`; kafkajs's `sharedPromiseTo`
collapses our call with its internal one, so total latency is unchanged.

The cluster reference for each producer/consumer is captured at creation
time via two new `addHook` entries on `src/producer/index.js` /
`src/consumer/index.js` and stored in a `WeakMap` exported from
`helpers/kafka`. The kafkajs object stays untouched — no Symbol-keyed
property to leak through `Reflect.ownKeys`, no string-keyed underscore
for user serializers to pick up, and the entry drops as soon as the
producer is GC'd.

Refs: #5270
Refs: #8253
BridgeAR added a commit that referenced this pull request May 12, 2026
The Produce request didn't carry headers until v3 (Kafka 0.11), so
injecting on older brokers either wastes work or trips
`UNKNOWN_SERVER_ERROR` on a per-leader version mismatch in mixed-version
clusters. Read `cluster.brokerPool.versions[0].maxVersion` once on first
send and disable injection proactively when the broker negotiated <v3;
the existing `KafkaJSProtocolError UNKNOWN` handler stays as the safety
net for mixed-version clusters where the seed broker advertised v3+ but
the leader couldn't parse the headers.

`refreshHeaderSupport` is a `let` that swaps to module-level `noop` at
both disable sites, so subsequent sends skip the helper call entirely.
The reactive `disableHeaderInjection` moves from the module-level
`disabledHeaderWeakSet` to a per-producer closure so the lifetime
matches the producer it describes.

Refs: #8253
dd-octo-sts Bot pushed a commit that referenced this pull request May 13, 2026
The cluster id discovery used a separate `kafka.admin()` connection whose
`describeCluster()` rejection wasn't awaited, surfacing as
`process.unhandledRejection` whenever the admin handshake failed (auth,
broker restart, network blip). Read
`cluster.brokerPool.metadata.clusterId` instead, priming it on first send
via `cluster.refreshMetadataIfNecessary()`; kafkajs's `sharedPromiseTo`
collapses our call with its internal one, so total latency is unchanged.

The cluster reference for each producer/consumer is captured at creation
time via two new `addHook` entries on `src/producer/index.js` /
`src/consumer/index.js` and stored in a `WeakMap` exported from
`helpers/kafka`. The kafkajs object stays untouched — no Symbol-keyed
property to leak through `Reflect.ownKeys`, no string-keyed underscore
for user serializers to pick up, and the entry drops as soon as the
producer is GC'd.

Refs: #5270
Refs: #8253
rochdev pushed a commit that referenced this pull request May 13, 2026
Both kafkajs's producer wrap and the `@confluentinc/kafka-javascript`
KafkaJS-compat producer wrap mutated the caller's `message.headers` in
place. The kafkajs boundary seeded `headers: {}` on every message before
the plugin injected trace context; the confluent KafkaJS-compat path
handed `payload.messages` straight to the underlying client, which then
wrote `headers: null` on messages without headers. Either path leaves
the caller holding the side effects of someone else's instrumentation.

The boundary now hands kafkajs and the underlying confluent client a
shallow clone of the messages array via a single `cloneMessages`
helper. With injection enabled the helper seeds `headers: {}` on the
clone for the plugin to inject into; with injection disabled (after the
broker reported `KafkaJSProtocolError UNKNOWN` /
`KafkaJSError ERR_UNKNOWN`) the helper preserves the absence of
`headers` so brokers that reject any header field can recover.

Refs: #5270
Refs: #8253
rochdev pushed a commit that referenced this pull request May 13, 2026
The cluster id discovery used a separate `kafka.admin()` connection whose
`describeCluster()` rejection wasn't awaited, surfacing as
`process.unhandledRejection` whenever the admin handshake failed (auth,
broker restart, network blip). Read
`cluster.brokerPool.metadata.clusterId` instead, priming it on first send
via `cluster.refreshMetadataIfNecessary()`; kafkajs's `sharedPromiseTo`
collapses our call with its internal one, so total latency is unchanged.

The cluster reference for each producer/consumer is captured at creation
time via two new `addHook` entries on `src/producer/index.js` /
`src/consumer/index.js` and stored in a `WeakMap` exported from
`helpers/kafka`. The kafkajs object stays untouched — no Symbol-keyed
property to leak through `Reflect.ownKeys`, no string-keyed underscore
for user serializers to pick up, and the entry drops as soon as the
producer is GC'd.

Refs: #5270
Refs: #8253
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants