Magnus Edenhill (@edenhill) Are you still interested in this fix?

Emanuele Sabellico (@emasab) would this PR be of interest? As the sticky-cooperative partition assignment is essentially broken with manual commits, I was hoping I could get this reviewed. (Magnus Edenhill (@edenhill) reviewed the linked PR before so I don't foresee many issues)

Emanuele Sabellico (@emasab) have you had a chance to take a look at this PR?

William Morgan (@wmorgan6796) Roxane (@roxelo) thanks for this contribution! We're doing changes to the cooperative-sticky assignor and we can check this too while doing that.

Emanuele Sabellico (@emasab) is there a PR/branch that can be followed?

I had added this earlier to the changes we're making to the assignors, #4252, but stripped the changes recently as we think there are some further complexities we need to address.

The change seems to make it impossible to use this pattern inside a user-defined rebalance callback, which is a valid use case:

...
case RD_KAFKA_RESP_ERR__REVOKE_PARTITIONS:
                  rd_kafka_commit(rk, partitions, 0); // sync commit
 ...

(it's a valid usecase in Java as well, to call commitSync from the onPartitionsRevoked method of the ConsumerRebalanceListener).

Additionally, we were also thinking it's best to include two more tests, first, a test added to 0118 which checks the fact that commit should work inside a rebalance cb (if it's called before calling assign()), and a second test which can replicate the situation defined in #4059 and then work towards fixing that.

Thanks for your patience, we're discussing it further internally, to see how we can best help with it.

Milind L (@milindl) it seems that the fix for the cooperative sticky partition assignment strategy was figured out? If so I'll close this PR.

Hi William Morgan (@wmorgan6796) , no, it still isn't figured out. We can't simply disallow committing while rebalance is happening (it's a legitimate case to commit in a sync manner inside the rebalance cb), but at the same time, the issue with illegal generation persists. Please leave this PR open as we figure out the fix, it contains the entire context, and I'll add commits to the same branch too, thanks.

All committers have signed the CLA.

Hi. Any progress on this? We have also hit this issue.

Closing this as its been fixed with #4908