Description

I was running list_consumer_group_offsets with a ConsumerGroupTopicPartitions that only includes the consumer group. It should have outputted all of the topic partitions for a consumer group, but instead resulted in an empty list for Kafka 2.3 and below.

Is this intended, and if so, is there a specific KIP/KAFKA card that is related? I was searching around and the closest thing was maybe this? Any help is appreciated in understanding this behavior difference between 2.3 and 2.4!

How to reproduce

1. Spin up a Kafka v2.3 Docker image. I'm working on the Kafka consumer integration for Datadog so I ran ddev env start kafka_consumer py3.9-2.3 to start a Docker container already configured with consumer groups, topics, and partitions. The test environment sets up a consumer group my_consumer that is consuming topics marvel and dc, both of which have 2 partitions. Our integration comes with confluent-kafka-python version 2.1.1.
2. Inside our Docker container, run the following small script:

from confluent_kafka import ConsumerGroupTopicPartitions
from confluent_kafka.admin import AdminClient


config = {
    "bootstrap.servers": "127.0.0.1:9092",
}
consumer_group = "my_consumer"

def main():
    kafka_client = AdminClient(config)

    offsets = kafka_client.list_consumer_group_offsets([ConsumerGroupTopicPartitions(consumer_group)])[consumer_group]
    print("Resulting topic-partitions for my_consumer: " + str(offsets.result().topic_partitions))


if __name__ == "__main__":
    main()

On Kafka 2.3, I get:

root@docker-desktop:/# python test.py 
Resulting topic-partitions: []

When I spin up a 2.4.1 container (by manually set the version of Kafka to 2.4.1 on the docker-compose.yaml), I get:

root@docker-desktop:/# python test.py
Resulting topic-partitions for my_consumer: [TopicPartition{topic=marvel,partition=0,offset=75,leader_epoch=0,error=None}, TopicPartition{topic=marvel,partition=1,offset=75,leader_epoch=0,error=None}, TopicPartition{topic=dc,partition=1,offset=75,leader_epoch=0,error=None}, TopicPartition{topic=dc,partition=0,offset=75,leader_epoch=0,error=None}]

I'm not completely sure if there is a bug, or if there is a just a minimum supported version of Kafka for the confluent-kafka-python library?

Checklist

Please provide the following information:

• confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
• Apache Kafka broker version:
• Client configuration: {...}
• Operating system: Linux
• Provide client logs (with 'debug': '..' as necessary)
• Provide broker log excerpts
• Critical issue