Skip to content

Expose Kafka message's key and topic via virtual columns in tables with Kafka engines #5286

@abyss7

Description

@abyss7

The column for key should be named as _key - if the name is taken by user, then we add the number suffix until we get unused column name: e.g. _key1, _key2, etc.
The column for topic should be named as _topic and uses the same number-suffix logic.

If the Kafka message contains multiple rows, then virtual values are the same for all these rows.

Both virtual columns have type String.

Syntax example:

CREATE TABLE kafka (a UInt8, b String) ENGINE Kafka() SETTINGS …;
SELECT a, b, _topic, _key FROM kafka LIMIT 1;

Metadata

Metadata

Assignees

Labels

comp-message-queuesMessage queue integrations (Kafka, RabbitMQ, NATS table engines for stream ingestion/egress).feature

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions