/area runtime
Describe the proposal
From an operational perspective, it makes sense that that we introduce a firehose of Workflow life-cycle events.
These are incredibly valuable for users who want an evented hook / trigger when a life-cycle event occurs.
This concept is not anything new, and is fully embraced in Azure Durable Function
I propose that the Workflow runtime uses Daprs Internal PubSub concept (same as how Dapr Outbox is implemented), to enable end-users to build their own solutions on top of the firehose of events.
List of use cases as to why this feature is needed
Use case 1 - Operations observability
A perfect and common example of this is subscribing to a firehose of failed events. This allows developers and operations people to keep track of critically important workflows and react accordingly, usually by raising alerts.
Use case 2 - Performance validation of the Workflows Runtime
I'm trying to test running many workflows at scale, but there is no easy way for me to see the state of my workflows. For example, if I schedule 10,000 workflows, I want to see 10,000 workflows complete eventually. There's no easy way for me to do this.
Having the lifecycle events would allow me to create a subscriber to persist the complete/failed events and easily aggregate them to check that I've achieved my goal.
Once again, lifecycle events are super important for driving monitoring/alerting in a production scenario.
Use case 3 - Driving internal implementations of Dapr Workflows capabilities & features
Another use-case of this feature would be to support the purging of instance data for dapr workflows.
Some users may want to only purge completed instances
Others may want to only purge failed instances etc.
The lifecycle events would be subscribed by the Workflow app itself (via PubSub), and if a certain status is reached, the user-code would proceed to call Purge on that instance
This allows a flexible model to achieve each of those goals, with no additional complexity for dapr maintainers.
Use case 4 - Secondary indexes / Deprecated Workflow Type detection
Another use-case is for building a secondary index of instance IDs which can be used to drive a polling process which checks for versions of workflows which are no longer running I.e when looking to deprecate a particular version of a workflow type from the code base.
/area runtime
Describe the proposal
From an operational perspective, it makes sense that that we introduce a firehose of Workflow life-cycle events.
These are incredibly valuable for users who want an evented hook / trigger when a life-cycle event occurs.
This concept is not anything new, and is fully embraced in Azure Durable Function
I propose that the Workflow runtime uses Daprs Internal PubSub concept (same as how Dapr Outbox is implemented), to enable end-users to build their own solutions on top of the firehose of events.
List of use cases as to why this feature is needed
Use case 1 - Operations observability
A perfect and common example of this is subscribing to a firehose of
failedevents. This allows developers and operations people to keep track of critically important workflows and react accordingly, usually by raising alerts.Use case 2 - Performance validation of the Workflows Runtime
I'm trying to test running many workflows at scale, but there is no easy way for me to see the state of my workflows. For example, if I schedule 10,000 workflows, I want to see 10,000 workflows complete eventually. There's no easy way for me to do this.
Having the lifecycle events would allow me to create a subscriber to persist the complete/failed events and easily aggregate them to check that I've achieved my goal.
Once again, lifecycle events are super important for driving monitoring/alerting in a production scenario.
Use case 3 - Driving internal implementations of Dapr Workflows capabilities & features
Another use-case of this feature would be to support the purging of instance data for dapr workflows.
Some users may want to only purge completed instances
Others may want to only purge failed instances etc.
The lifecycle events would be subscribed by the Workflow app itself (via PubSub), and if a certain status is reached, the user-code would proceed to call Purge on that instance
This allows a flexible model to achieve each of those goals, with no additional complexity for dapr maintainers.
Use case 4 - Secondary indexes / Deprecated Workflow Type detection
Another use-case is for building a secondary index of instance IDs which can be used to drive a polling process which checks for versions of workflows which are no longer running I.e when looking to deprecate a particular version of a workflow type from the code base.