Skip to content

Add Topic Id to Metadata Response#4300

Merged
Pranav Rathi (pranavrth) merged 7 commits intomasterfrom
dev_metadata-response-topic-id
Aug 7, 2023
Merged

Add Topic Id to Metadata Response#4300
Pranav Rathi (pranavrth) merged 7 commits intomasterfrom
dev_metadata-response-topic-id

Conversation

@pranavrth
Copy link
Copy Markdown
Member

As part of KIP-516, we need to add support for topic id in all the major APIs. As a prerequisite for KIP-848, we are adding topic id to Metadata Response in this PR. We are incrementing support for v12 for Metadata protocol though we still don't support topic id in Metadata Request for now. That will be added once we implement full support for topic id as part of KIP-516.

This PR includes:

  • Adding new data type UUID required to store topic id.
  • Base64 encoding of UUID
  • Extracting topic id from Metadata Response to internal Metadata struct
  • Some small refactoring

@pranavrth
Copy link
Copy Markdown
Member Author

There were 2 different places where base64 encoding was done in the code. Both of them are dependent on OpenSSL. I have moved 1 implementation to common place to be used as a generic base64 encoder.

Open points to discuss:

  • Though I have kept it dependent on OpenSSL, we should also make it independent of OpenSSL as topic id should not be dependent on OpenSSL.
  • 2nd place implementation of base64 encoding (in rdkafka_sasl_scram.c) is not changed to use generic implementation extracted in this PR.

@pranavrth Pranav Rathi (pranavrth) force-pushed the dev_metadata-response-topic-id branch from 6795ad5 to 1c9e5c2 Compare May 29, 2023 19:23
Comment thread src/rdkafka_sasl_scram.c
@emasab Emanuele Sabellico (emasab) marked this pull request as ready for review July 18, 2023 14:27
Comment thread src/rdbase64.c
Comment thread src/rdkafka_buf.h Outdated
Comment thread src/rdbase64.c Outdated
Comment thread src/rdkafka_proto.h Outdated
Comment thread src/rdkafka_proto.h Outdated
Comment thread src/rdkafka_buf.h Outdated
Comment thread src/rdkafka_buf.h Outdated
Comment thread src/rdkafka_proto.h Outdated
Comment thread src/rdkafka_buf.h Outdated
Comment thread src/rdkafka_buf.h Outdated
@pranavrth Pranav Rathi (pranavrth) force-pushed the dev_metadata-response-topic-id branch 2 times, most recently from b6a7b45 to b32bb78 Compare August 2, 2023 12:54
@pranavrth Pranav Rathi (pranavrth) force-pushed the dev_metadata-response-topic-id branch from 5073a8c to 895358a Compare August 2, 2023 16:17
@cla-assistant
Copy link
Copy Markdown

cla-assistant Bot commented Aug 3, 2023

CLA assistant check
All committers have signed the CLA.

Emanuele Sabellico (emasab) added a commit that referenced this pull request Aug 3, 2023
Needs to be updated after #4300
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sent some other improvements

Comment thread CHANGELOG.md Outdated
@@ -1,5 +1,11 @@
# librdkafka v2.2.0
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

version 2.2.1 for the moment as it doesn't change public API

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping 2.3.0 as we don't have plan for 2.2.1 right now.

Comment thread CHANGELOG.md
Comment thread INTRODUCTION.md Outdated
Comment thread src/rdbase64.c Outdated
Comment thread src/rdbase64.c Outdated
Comment thread src/rdkafka_proto.h Outdated
Comment thread src/rdkafka_proto.h Outdated
Comment thread src/rdkafka_proto.h Outdated
Comment thread src/rdkafka_proto.h
Comment thread src/rdkafka_sasl_oauthbearer_oidc.c
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR Pranav Rathi (@pranavrth) !

@pranavrth Pranav Rathi (pranavrth) merged commit 07262c4 into master Aug 7, 2023
@pranavrth Pranav Rathi (pranavrth) deleted the dev_metadata-response-topic-id branch August 7, 2023 14:01
Emanuele Sabellico (emasab) added a commit that referenced this pull request Aug 7, 2023
Needs to be updated after #4300
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants