Two questions:

• So we think it makes sense to reapply this event even across runs?
• Will the callback logic get triggered on a run that get terminated due to version conflict?

So we think it makes sense to reapply this event even across runs?

What do you mean by "across runs"?

Will the callback logic get triggered on a run that get terminated due to version conflict?

Currently, callbacks are called when the workflow is completed, except if it's continued as new or reset.

So we think it makes sense to reapply this event even across runs?

I'm assuming you mean a workflow reset? If so then yes, it has similar semantics to a signal, but I don't see a reason to allow opting out of it ATM.

Will the callback logic get triggered on a run that get terminated due to version conflict?

We didn't explicitly test that, so I can't say for sure. I'm not 100% what this scenario is, would another run take over as current and the losing run made a zombie? If that's the case, can we copy over the callbacks to the conflict winner? I'm not sure it completely makes sense to do that, it's kind of undefined behavior, probably best to trigger the callback and report termination of the loser and not apply to the winner.

you mean a workflow reset?

No I mean reapply from zombie run to current run. I think it makes sense, so just confirm here.

would another run take over as current and the losing run made a zombie?

So all zombie runs will eventually be terminated (when the cluster creates the zombie run discovers there's a newer new run, it just terminate the run it has instead of putting it into zombie.)
The situation I am thinking is one workflow W starts a workflow W1 in cluster A, after a (forced) failover, it creates re-runs the operation and creates W2. W1 will eventually be terminated and W2 may still be running. Once terminated, callback in W1 will (or maybe not) run. So now W will see a failed nexus operation despite that W2 is still running.

can we copy over the callbacks to the conflict winner?

I think it's already the case since we are reapplying WorkflowExecutionOptionsUpdatedEvent from zombie to current run? If the zombie (eventually terminated) run will also try to report back then it's a race condition. and may not be the behavior we want.

How/Where in the code the zombie workflow is terminated? Is it here?
How do we know it's a zombie there?

Updated the PR to not process the callbacks when it's a zombie workflow.

The situation I am thinking is one workflow W starts a workflow W1 in cluster A, after a (forced) failover, it creates re-runs the operation and creates W2. W1 will eventually be terminated and W2 may still be running. Once terminated, callback in W1 will (or maybe not) run. So now W will see a failed nexus operation despite that W2 is still running.

Discussed offline: it should be a rare case, and we'll handle this case another time.

Not processing callback for zombie workflows already fixed ^ issue since W1 will no longer report back to W. What we discussed offline is a different case where there are two levels of zombie workflows.

Should be safe from what I can see. but cc @xwduan @hai719 @yux0 FYI.

Yeah. I think it is safe. This also may helped on a case where the terminated events comes from suppress should not be reapplied.

Not processing callback for zombie workflows already fixed ^ issue since W1 will no longer report back to W. What we discussed offline is a different case where there are two levels of zombie workflows.

Is this good enough?
I mean this will prevent the current cluster from generating callback tasks. But the termination event will eventually be replicated (in event-based replication) to another cluster, then in that cluster callback tasks will still be generated (and processed if failover happens)?