fix(pubsub/pulsar): read processMode from component metadata and add async backpressure#4309
Conversation
f03a386 to
267031e
Compare
|
Super thanks |
There was a problem hiding this comment.
Pull request overview
This PR fixes two production-impacting issues in the Pulsar pubsub component: processMode from component metadata being ignored, and lack of backpressure in async processing leading to unbounded concurrent handler goroutines/unacked messages.
Changes:
- Add
ProcessModetopulsarMetadata, normalize/validate it during metadata parsing, and resolve the effective mode using component metadata with subscription override. - Add an async-mode semaphore in
listenMessageto bound concurrent handler goroutines toMaxConcurrentHandlers(real backpressure), and fix an async data race on a sharederrvariable. - Add new test suites covering
processModeresolution and async backpressure/concurrency limiting behavior.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| pubsub/pulsar/metadata.go | Adds ProcessMode to component metadata struct. |
| pubsub/pulsar/pulsar.go | Validates/normalizes processMode, defaults zero concurrency, adds async semaphore backpressure, and adjusts cancellation logging. |
| pubsub/pulsar/process_mode_test.go | Adds unit + integration-style tests for processMode parsing and effective mode resolution. |
| pubsub/pulsar/pulsar_semaphore_test.go | Adds tests for async semaphore concurrency limits, backpressure behavior, and shutdown/error scenarios. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…async backpressure Two fixes for the Pulsar pubsub component that caused unbounded unacked messages: 1. processMode was only read from subscription request metadata (req.Metadata), silently ignoring the value when set in component YAML metadata. Users configuring processMode: sync were unknowingly running in async mode. Fix: add ProcessMode to pulsarMetadata struct, normalize to lowercase at parse time, validate accepted values, and resolve effective mode in listenMessage using component metadata as default with subscription metadata override. 2. In async mode, every message spawned a goroutine with no upper bound. MaxConcurrentHandlers only controlled the channel buffer size, not concurrent goroutines. Fix: add a semaphore channel that limits concurrent async handlers to MaxConcurrentHandlers, creating real backpressure — when all slots are full the loop stops reading from the channel, the channel fills up, and the SDK stops requesting more messages from the broker. Additional fixes: - Pre-existing data race where shared err variable was written by concurrent goroutines in async mode. - MaxConcurrentHandlers=0 now falls back to default (100) instead of deadlocking the semaphore. - Invalid processMode values now return an error at parse time instead of silently falling through to async. - Normal context cancellation logged at Debug level instead of Error. Signed-off-by: Nelson Parente <[email protected]>
110f0bc to
3bf7db3
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
3bf7db3 to
dc999ec
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
dc999ec to
65a31bd
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
65a31bd to
53c6fdb
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…test flakiness - Refactor async mode from per-message goroutine + semaphore to a pre-spawned worker pool of MaxConcurrentHandlers goroutines reading from a shared work channel. Backpressure is achieved naturally when all workers are busy and the work channel fills up. - Split listenMessage into listenMessageSync and listenMessageAsync for clarity. - Validate processMode override in subscription metadata: invalid values now log a warning and fall back to component default instead of silently using async. - Replace all time.Sleep-based test assertions with require.Eventually and require.Never for deterministic, non-flaky CI behavior. - Add processMode field to metadata.yaml documentation. - Update stale semaphore references in comments to reflect worker pool. - Fix golangci-lint warnings (gocritic, intrange). Signed-off-by: Javier Aliaga <[email protected]>
53c6fdb to
3786517
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…async backpressure (dapr#4309) Signed-off-by: Nelson Parente <[email protected]> Signed-off-by: Javier Aliaga <[email protected]> Co-authored-by: Javier Aliaga <[email protected]>
…nc backpressure processMode can now be set at the component level (not just subscription request metadata) and overridden per subscription. Updated description for maxConcurrentHandlers to document the fixed worker pool with backpressure and the fallback behavior when set to 0. Ref: dapr/components-contrib#4309 Signed-off-by: Javier Aliaga <[email protected]>
…nc backpressure processMode can now be set at the component level (not just subscription request metadata) and overridden per subscription. Updated description for maxConcurrentHandlers to document the fixed worker pool with backpressure and the fallback behavior when set to 0. Ref: dapr/components-contrib#4309 Signed-off-by: Javier Aliaga <[email protected]>
…nc backpressure processMode can now be set at the component level (not just subscription request metadata) and overridden per subscription. Updated description for maxConcurrentHandlers to document the fixed worker pool with backpressure and the fallback behavior when set to 0. Ref: dapr/components-contrib#4309 Signed-off-by: Javier Aliaga <[email protected]>
Includes fix for Pulsar pub/sub processMode ignored from component metadata and missing async backpressure (dapr/components-contrib#4309). Signed-off-by: Javier Aliaga <[email protected]>
* chore: bump components-contrib to v1.17.3 Includes fix for Pulsar pub/sub processMode ignored from component metadata and missing async backpressure (dapr/components-contrib#4309). Signed-off-by: Javier Aliaga <[email protected]> * docs: add release notes for v1.16.12 Signed-off-by: Javier Aliaga <[email protected]> * ci: improve standalone validation baseline ref resolution Support push events by computing baseline ref from github.event.before or HEAD~1, instead of only working for pull requests. Signed-off-by: Javier Aliaga <[email protected]> --------- Signed-off-by: Javier Aliaga <[email protected]>
…processMode improvements (#5084) * docs(pulsar): add CloudEvents Avro schema wrapping and rawPayload option Document the new behavior where Dapr wraps user-provided Avro schemas inside a CloudEvents envelope before registering with the Pulsar Schema Registry. Add the new `<topic-name>.rawPayload` metadata field for topics that need to skip CE wrapping and register the inner schema directly. Ref: dapr/components-contrib#4302 Signed-off-by: Javier Aliaga <[email protected]> * docs(pulsar): clarify processMode as component-level metadata and async backpressure processMode can now be set at the component level (not just subscription request metadata) and overridden per subscription. Updated description for maxConcurrentHandlers to document the fixed worker pool with backpressure and the fallback behavior when set to 0. Ref: dapr/components-contrib#4309 Signed-off-by: Javier Aliaga <[email protected]> * docs(pulsar): Apply suggestions Co-authored-by: Alice Gibbons <[email protected]> Signed-off-by: Javier Aliaga <[email protected]> --------- Signed-off-by: Javier Aliaga <[email protected]> Co-authored-by: Alice Gibbons <[email protected]>
Includes fix for Pulsar pub/sub processMode ignored from component metadata and missing async backpressure (dapr/components-contrib#4309). Signed-off-by: Javier Aliaga <[email protected]>
* chore: bump components-contrib to v1.17.3 Includes fix for Pulsar pub/sub processMode ignored from component metadata and missing async backpressure (dapr/components-contrib#4309). Signed-off-by: Javier Aliaga <[email protected]> * docs: add release notes for v1.16.12 Signed-off-by: Javier Aliaga <[email protected]> * ci: improve standalone validation baseline ref resolution Support push events by computing baseline ref from github.event.before or HEAD~1, instead of only working for pull requests. Signed-off-by: Javier Aliaga <[email protected]> --------- Signed-off-by: Javier Aliaga <[email protected]>
* chore: bump components-contrib to v1.17.3 Includes fix for Pulsar pub/sub processMode ignored from component metadata and missing async backpressure (dapr/components-contrib#4309). Signed-off-by: Javier Aliaga <[email protected]> * docs: add release notes for v1.16.12 Signed-off-by: Javier Aliaga <[email protected]> * ci: improve standalone validation baseline ref resolution Support push events by computing baseline ref from github.event.before or HEAD~1, instead of only working for pull requests. Signed-off-by: Javier Aliaga <[email protected]> --------- Signed-off-by: Javier Aliaga <[email protected]>
…async backpressure (dapr#4309) Signed-off-by: Nelson Parente <[email protected]> Signed-off-by: Javier Aliaga <[email protected]> Co-authored-by: Javier Aliaga <[email protected]>
* Promote 1.17 to latest Signed-off-by: Marc Duiker <[email protected]> * Adding support to configure gRPC message limits in workflows in .NET Signed-off-by: Whit Waldo <[email protected]> * Fix CLI example for workflow versioning. Signed-off-by: joshvanl <[email protected]> * Apply suggestions from code review Co-authored-by: Copilot <[email protected]> Signed-off-by: Mark Fussell <[email protected]> * Updates resiliency of streaming HTTP Service Invocation Based on this PR dapr/dapr#9627 Signed-off-by: joshvanl <[email protected]> * workflow: retention policy is not retroactive Clarify that adding or changing a retention policy does not retroactively purge workflows already in a terminal state. Include CLI examples for retroactive cleanup using `dapr workflow purge --all-older-than` and `dapr workflow list --filter-status` combined with purge. Signed-off-by: joshvanl <[email protected]> * Workflow: `wf purge --all-older-than --all-filter-status` Update docs based on new CLI flag dapr/cli#1609 Signed-off-by: joshvanl <[email protected]> * Update docs for v1.17.2 (#5081) Signed-off-by: joshvanl <[email protected]> * docs(agents): update with v1.0 rcs out (#5071) * docs(agents): update with v1.0 rcs out Signed-off-by: Samantha Coyle <[email protected]> * fix: address copilot feedback Signed-off-by: Samantha Coyle <[email protected]> * Update daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-core-concepts.md Co-authored-by: Alice Gibbons <[email protected]> Signed-off-by: Sam <[email protected]> * Update daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-core-concepts.md Co-authored-by: Alice Gibbons <[email protected]> Signed-off-by: Sam <[email protected]> * Update daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-core-concepts.md Co-authored-by: Alice Gibbons <[email protected]> Signed-off-by: Sam <[email protected]> * Update daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-integrations.md Co-authored-by: Alice Gibbons <[email protected]> Signed-off-by: Sam <[email protected]> * Update daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-core-concepts.md Co-authored-by: Alice Gibbons <[email protected]> Signed-off-by: Sam <[email protected]> * Update daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-core-concepts.md Co-authored-by: Alice Gibbons <[email protected]> Signed-off-by: Sam <[email protected]> --------- Signed-off-by: Samantha Coyle <[email protected]> Signed-off-by: Sam <[email protected]> Co-authored-by: Alice Gibbons <[email protected]> Co-authored-by: Yaron Schneider <[email protected]> * docs(pulsar): add CloudEvents schema wrapping, rawschema option, and processMode improvements (#5084) * docs(pulsar): add CloudEvents Avro schema wrapping and rawPayload option Document the new behavior where Dapr wraps user-provided Avro schemas inside a CloudEvents envelope before registering with the Pulsar Schema Registry. Add the new `<topic-name>.rawPayload` metadata field for topics that need to skip CE wrapping and register the inner schema directly. Ref: dapr/components-contrib#4302 Signed-off-by: Javier Aliaga <[email protected]> * docs(pulsar): clarify processMode as component-level metadata and async backpressure processMode can now be set at the component level (not just subscription request metadata) and overridden per subscription. Updated description for maxConcurrentHandlers to document the fixed worker pool with backpressure and the fallback behavior when set to 0. Ref: dapr/components-contrib#4309 Signed-off-by: Javier Aliaga <[email protected]> * docs(pulsar): Apply suggestions Co-authored-by: Alice Gibbons <[email protected]> Signed-off-by: Javier Aliaga <[email protected]> --------- Signed-off-by: Javier Aliaga <[email protected]> Co-authored-by: Alice Gibbons <[email protected]> * chore: update Azure Static Web App base hostname to new account (#5095) Signed-off-by: Jorge André Gonçalves <[email protected]> * chore: fix Azure Static Web App staging URL region suffix (#5096) Signed-off-by: Jorge André Gonçalves <[email protected]> * chore: update Azure Static Web App to new instance (dapr-docs-site) (#5097) Signed-off-by: Jorge André Gonçalves <[email protected]> * revert: restore original Azure Static Web App configuration (#5098) Signed-off-by: Jorge André Gonçalves <[email protected]> * dapr v1.17.3 (#5089) Signed-off-by: Cassandra Coyle <[email protected]> Co-authored-by: Josh van Leeuwen <[email protected]> * Review comments Signed-off-by: joshvanl <[email protected]> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <[email protected]> Signed-off-by: Josh van Leeuwen <[email protected]> * Updates CLI version to 1.17.1 (#5107) Signed-off-by: joshvanl <[email protected]> * Update dapr to version v1.17.4 (#5110) * Update dapr to version v1.17.4 Signed-off-by: joshvanl <[email protected]> * Update daprdocs/content/en/operations/support/support-release-policy.md Co-authored-by: Copilot <[email protected]> Signed-off-by: Josh van Leeuwen <[email protected]> --------- Signed-off-by: joshvanl <[email protected]> Signed-off-by: Josh van Leeuwen <[email protected]> Co-authored-by: Copilot <[email protected]> * update to latest versions Signed-off-by: Cassandra Coyle <[email protected]> * [1.17] docs(agents): announce v1.0 release for dapr agents in docs (#5121) * docs(agents): announce v1.0 release for dapr agents in docs Signed-off-by: Samantha Coyle <[email protected]> * style: updates for copilot feedback Signed-off-by: Samantha Coyle <[email protected]> * style: address feedback from mark Signed-off-by: Samantha Coyle <[email protected]> --------- Signed-off-by: Samantha Coyle <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * Update filename numbering to match quickstarts/ on dapr-agents (#5143) Signed-off-by: Sergio Herrera <[email protected]> * Revert "Documentation for native-sidecar (#5041)" (#5146) This reverts commit 1534361. Signed-off-by: Marc Duiker <[email protected]> * Add Reo.dev and cookie banner (#5147) * Revert "Documentation for native-sidecar (#5041)" This reverts commit 1534361. Signed-off-by: Marc Duiker <[email protected]> * Add reo.dev and cookie banner Signed-off-by: Marc Duiker <[email protected]> --------- Signed-off-by: Marc Duiker <[email protected]> * Adds latest v1.17.7 (#5163) Signed-off-by: joshvanl <[email protected]> * Fix typo: occured -> occurred (#5158) Signed-off-by: SAY-5 <[email protected]> Co-authored-by: SAY-5 <[email protected]> * docs(outbox): document outboxInternalTopic metadata field (#5155) Add documentation for the outboxInternalTopic metadata field which allows overriding the auto-generated internal outbox topic name. Changes: - Add outboxInternalTopic to metadata fields table - Add outboxInternalTopic to YAML example - Update internal outbox topic section with override explanation Related: dapr/dapr#8933, dapr/dapr#9855 Signed-off-by: Szymon Polom <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * Fix asp.net core docs (#5164) Signed-off-by: Erwin <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * fix: correct broken ref shortcode link in (#5126) workflow-history-retention-policy Signed-off-by: Bilgin Ibryam <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * Update tracing-overview.md (#5119) Add Middleware.io because it supports OTEL and its becoming highly popular in the observability space. Signed-off-by: Shekhar Luhar <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * Add new configuration options for Pulsar component: redeliveryDelay, compressionType, and compressionLevel (#5116) Signed-off-by: MyMirelHub <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * docs(agents): document replay-aware logging for DurableAgent (#5149) * docs(agents): document context-aware logger for DurableAgent Signed-off-by: Rishabh Dewangan <[email protected]> * docs: address review nits (imports and formatting) Signed-off-by: Rishabh Dewangan <[email protected]> --------- Signed-off-by: Rishabh Dewangan <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * docs(agents): add MistralChatClient to supported LLMs (#5108) * docs(agents): add MistralChatClient to supported LLMs Signed-off-by: Rishabh Dewangan <[email protected]> * docs: explicitly document Mistral fallback behavior Signed-off-by: Rishabh Dewangan <[email protected]> * docs(agents): add Prompty configuration template documentation Signed-off-by: Rishabh Dewangan <[email protected]> * docs(agents): address review nits for Prompty and Mistral models Signed-off-by: Rishabh Dewangan <[email protected]> --------- Signed-off-by: Rishabh Dewangan <[email protected]> Co-authored-by: Sam <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * mirror update from components-contrib (#5061) Signed-off-by: Eric Jacobson <[email protected]> Co-authored-by: Marc Duiker <[email protected]> Co-authored-by: Mark Fussell <[email protected]> --------- Signed-off-by: Marc Duiker <[email protected]> Signed-off-by: Whit Waldo <[email protected]> Signed-off-by: joshvanl <[email protected]> Signed-off-by: Mark Fussell <[email protected]> Signed-off-by: Samantha Coyle <[email protected]> Signed-off-by: Sam <[email protected]> Signed-off-by: Javier Aliaga <[email protected]> Signed-off-by: Jorge André Gonçalves <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Josh van Leeuwen <[email protected]> Signed-off-by: Sergio Herrera <[email protected]> Signed-off-by: SAY-5 <[email protected]> Signed-off-by: Szymon Polom <[email protected]> Signed-off-by: Erwin <[email protected]> Signed-off-by: Bilgin Ibryam <[email protected]> Signed-off-by: Shekhar Luhar <[email protected]> Signed-off-by: MyMirelHub <[email protected]> Signed-off-by: Rishabh Dewangan <[email protected]> Signed-off-by: Eric Jacobson <[email protected]> Co-authored-by: Whit Waldo <[email protected]> Co-authored-by: joshvanl <[email protected]> Co-authored-by: Mark Fussell <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Sam <[email protected]> Co-authored-by: Alice Gibbons <[email protected]> Co-authored-by: Yaron Schneider <[email protected]> Co-authored-by: Javier Aliaga <[email protected]> Co-authored-by: Jorge André Gonçalves <[email protected]> Co-authored-by: Cassie Coyle <[email protected]> Co-authored-by: seherv <[email protected]> Co-authored-by: Sai Asish Y <[email protected]> Co-authored-by: SAY-5 <[email protected]> Co-authored-by: Szymon Połom <[email protected]> Co-authored-by: Erwin <[email protected]> Co-authored-by: Bilgin Ibryam <[email protected]> Co-authored-by: Shekhar Luhar <[email protected]> Co-authored-by: Mirel <[email protected]> Co-authored-by: Rishabh Dewangan <[email protected]> Co-authored-by: Eric Jacobson <[email protected]>
* Promote 1.17 to latest Signed-off-by: Marc Duiker <[email protected]> * Adding support to configure gRPC message limits in workflows in .NET Signed-off-by: Whit Waldo <[email protected]> * Fix CLI example for workflow versioning. Signed-off-by: joshvanl <[email protected]> * Apply suggestions from code review Co-authored-by: Copilot <[email protected]> Signed-off-by: Mark Fussell <[email protected]> * Updates resiliency of streaming HTTP Service Invocation Based on this PR dapr/dapr#9627 Signed-off-by: joshvanl <[email protected]> * workflow: retention policy is not retroactive Clarify that adding or changing a retention policy does not retroactively purge workflows already in a terminal state. Include CLI examples for retroactive cleanup using `dapr workflow purge --all-older-than` and `dapr workflow list --filter-status` combined with purge. Signed-off-by: joshvanl <[email protected]> * Workflow: `wf purge --all-older-than --all-filter-status` Update docs based on new CLI flag dapr/cli#1609 Signed-off-by: joshvanl <[email protected]> * Update docs for v1.17.2 (#5081) Signed-off-by: joshvanl <[email protected]> * docs(agents): update with v1.0 rcs out (#5071) * docs(agents): update with v1.0 rcs out Signed-off-by: Samantha Coyle <[email protected]> * fix: address copilot feedback Signed-off-by: Samantha Coyle <[email protected]> * Update daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-core-concepts.md Co-authored-by: Alice Gibbons <[email protected]> Signed-off-by: Sam <[email protected]> * Update daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-core-concepts.md Co-authored-by: Alice Gibbons <[email protected]> Signed-off-by: Sam <[email protected]> * Update daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-core-concepts.md Co-authored-by: Alice Gibbons <[email protected]> Signed-off-by: Sam <[email protected]> * Update daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-integrations.md Co-authored-by: Alice Gibbons <[email protected]> Signed-off-by: Sam <[email protected]> * Update daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-core-concepts.md Co-authored-by: Alice Gibbons <[email protected]> Signed-off-by: Sam <[email protected]> * Update daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-core-concepts.md Co-authored-by: Alice Gibbons <[email protected]> Signed-off-by: Sam <[email protected]> --------- Signed-off-by: Samantha Coyle <[email protected]> Signed-off-by: Sam <[email protected]> Co-authored-by: Alice Gibbons <[email protected]> Co-authored-by: Yaron Schneider <[email protected]> * docs(pulsar): add CloudEvents schema wrapping, rawschema option, and processMode improvements (#5084) * docs(pulsar): add CloudEvents Avro schema wrapping and rawPayload option Document the new behavior where Dapr wraps user-provided Avro schemas inside a CloudEvents envelope before registering with the Pulsar Schema Registry. Add the new `<topic-name>.rawPayload` metadata field for topics that need to skip CE wrapping and register the inner schema directly. Ref: dapr/components-contrib#4302 Signed-off-by: Javier Aliaga <[email protected]> * docs(pulsar): clarify processMode as component-level metadata and async backpressure processMode can now be set at the component level (not just subscription request metadata) and overridden per subscription. Updated description for maxConcurrentHandlers to document the fixed worker pool with backpressure and the fallback behavior when set to 0. Ref: dapr/components-contrib#4309 Signed-off-by: Javier Aliaga <[email protected]> * docs(pulsar): Apply suggestions Co-authored-by: Alice Gibbons <[email protected]> Signed-off-by: Javier Aliaga <[email protected]> --------- Signed-off-by: Javier Aliaga <[email protected]> Co-authored-by: Alice Gibbons <[email protected]> * chore: update Azure Static Web App base hostname to new account (#5095) Signed-off-by: Jorge André Gonçalves <[email protected]> * chore: fix Azure Static Web App staging URL region suffix (#5096) Signed-off-by: Jorge André Gonçalves <[email protected]> * chore: update Azure Static Web App to new instance (dapr-docs-site) (#5097) Signed-off-by: Jorge André Gonçalves <[email protected]> * revert: restore original Azure Static Web App configuration (#5098) Signed-off-by: Jorge André Gonçalves <[email protected]> * dapr v1.17.3 (#5089) Signed-off-by: Cassandra Coyle <[email protected]> Co-authored-by: Josh van Leeuwen <[email protected]> * Review comments Signed-off-by: joshvanl <[email protected]> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <[email protected]> Signed-off-by: Josh van Leeuwen <[email protected]> * Updates CLI version to 1.17.1 (#5107) Signed-off-by: joshvanl <[email protected]> * Update dapr to version v1.17.4 (#5110) * Update dapr to version v1.17.4 Signed-off-by: joshvanl <[email protected]> * Update daprdocs/content/en/operations/support/support-release-policy.md Co-authored-by: Copilot <[email protected]> Signed-off-by: Josh van Leeuwen <[email protected]> --------- Signed-off-by: joshvanl <[email protected]> Signed-off-by: Josh van Leeuwen <[email protected]> Co-authored-by: Copilot <[email protected]> * update to latest versions Signed-off-by: Cassandra Coyle <[email protected]> * [1.17] docs(agents): announce v1.0 release for dapr agents in docs (#5121) * docs(agents): announce v1.0 release for dapr agents in docs Signed-off-by: Samantha Coyle <[email protected]> * style: updates for copilot feedback Signed-off-by: Samantha Coyle <[email protected]> * style: address feedback from mark Signed-off-by: Samantha Coyle <[email protected]> --------- Signed-off-by: Samantha Coyle <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * Update filename numbering to match quickstarts/ on dapr-agents (#5143) Signed-off-by: Sergio Herrera <[email protected]> * Revert "Documentation for native-sidecar (#5041)" (#5146) This reverts commit 1534361. Signed-off-by: Marc Duiker <[email protected]> * Add Reo.dev and cookie banner (#5147) * Revert "Documentation for native-sidecar (#5041)" This reverts commit 1534361. Signed-off-by: Marc Duiker <[email protected]> * Add reo.dev and cookie banner Signed-off-by: Marc Duiker <[email protected]> --------- Signed-off-by: Marc Duiker <[email protected]> * Adds latest v1.17.7 (#5163) Signed-off-by: joshvanl <[email protected]> * Fix typo: occured -> occurred (#5158) Signed-off-by: SAY-5 <[email protected]> Co-authored-by: SAY-5 <[email protected]> * docs(outbox): document outboxInternalTopic metadata field (#5155) Add documentation for the outboxInternalTopic metadata field which allows overriding the auto-generated internal outbox topic name. Changes: - Add outboxInternalTopic to metadata fields table - Add outboxInternalTopic to YAML example - Update internal outbox topic section with override explanation Related: dapr/dapr#8933, dapr/dapr#9855 Signed-off-by: Szymon Polom <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * Fix asp.net core docs (#5164) Signed-off-by: Erwin <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * fix: correct broken ref shortcode link in (#5126) workflow-history-retention-policy Signed-off-by: Bilgin Ibryam <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * Update tracing-overview.md (#5119) Add Middleware.io because it supports OTEL and its becoming highly popular in the observability space. Signed-off-by: Shekhar Luhar <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * Add new configuration options for Pulsar component: redeliveryDelay, compressionType, and compressionLevel (#5116) Signed-off-by: MyMirelHub <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * docs(agents): document replay-aware logging for DurableAgent (#5149) * docs(agents): document context-aware logger for DurableAgent Signed-off-by: Rishabh Dewangan <[email protected]> * docs: address review nits (imports and formatting) Signed-off-by: Rishabh Dewangan <[email protected]> --------- Signed-off-by: Rishabh Dewangan <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * docs(agents): add MistralChatClient to supported LLMs (#5108) * docs(agents): add MistralChatClient to supported LLMs Signed-off-by: Rishabh Dewangan <[email protected]> * docs: explicitly document Mistral fallback behavior Signed-off-by: Rishabh Dewangan <[email protected]> * docs(agents): add Prompty configuration template documentation Signed-off-by: Rishabh Dewangan <[email protected]> * docs(agents): address review nits for Prompty and Mistral models Signed-off-by: Rishabh Dewangan <[email protected]> --------- Signed-off-by: Rishabh Dewangan <[email protected]> Co-authored-by: Sam <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * mirror update from components-contrib (#5061) Signed-off-by: Eric Jacobson <[email protected]> Co-authored-by: Marc Duiker <[email protected]> Co-authored-by: Mark Fussell <[email protected]> * Add Tuning Engines Dapr AI integration docs Signed-off-by: VC <[email protected]> * Update Maintainer Guide with latest CLI commands and release steps Signed-off-by: pittu sharma <[email protected]> * Fix duplicate archived_version mapping key in hugo.yaml (#5100) Signed-off-by: pittu sharma <[email protected]> * chore: Update docs-zh for v1.17 (#5091) Also updates the `docs-zh` submodule to its latest commit. And remove duplicated attribute in hugo.yml Signed-off-by: newbe36524 <[email protected]> Co-authored-by: Hagicode <[email protected]> * Limit the deployment of the version specific website to only preview and archive state. Signed-off-by: Marc Duiker <[email protected]> * Bump postcss from 8.5.6 to 8.5.15 Bumps [postcss](https://github.com/postcss/postcss) from 8.5.6 to 8.5.15. - [Release notes](https://github.com/postcss/postcss/releases) - [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md) - [Commits](postcss/postcss@8.5.6...8.5.15) --- updated-dependencies: - dependency-name: postcss dependency-version: 8.5.15 dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> * Bump picomatch Bumps and [picomatch](https://github.com/micromatch/picomatch). These dependencies needed to be updated together. Updates `picomatch` from 2.2.2 to 2.3.2 - [Release notes](https://github.com/micromatch/picomatch/releases) - [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md) - [Commits](micromatch/picomatch@2.2.2...2.3.2) Updates `picomatch` from 4.0.2 to 4.0.4 - [Release notes](https://github.com/micromatch/picomatch/releases) - [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md) - [Commits](micromatch/picomatch@2.2.2...2.3.2) --- updated-dependencies: - dependency-name: picomatch dependency-version: 2.3.2 dependency-type: indirect - dependency-name: picomatch dependency-version: 4.0.4 dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> * fix: allow workflow_dispatch + trim non-content assets for Free tier Signed-off-by: paulyuk <[email protected]> * update sdkdocs codeowners Signed-off-by: Mike Nguyen <[email protected]> * Adding guard to prevent PRs from merging that would otherwise be acceptable when they're marked with the "do-not-merge" label. Automatically runs whenever a label is removed in case the do-not-merge is removed (prompting immediate merge if conditions are acceptable) Signed-off-by: Whit Waldo <[email protected]> * Adding additional triggers and guards: If an "automerge: " label is applied and the changes in the PR don't correspond to the directory of the applied label, the label will be removed and the automerge will not happen. If an SDK maintainer comes along and applies an "automerge: " label for an SDK they're not a maintainer of, the label will be removed and the automerge will not happen If anyone at all comes along and applies an "automerge: " label and they're not part of the maintainer group corresponding with the label, the label will be removed and the automerge will not happen. Signed-off-by: Whit Waldo <[email protected]> * docs: clarify continue-as-new behavior for child workflows #5021 Signed-off-by: pittu sharma <[email protected]> * Apply suggestion from @marcduiker Signed-off-by: Marc Duiker <[email protected]> * Apply suggestion from @marcduiker Signed-off-by: Marc Duiker <[email protected]> --------- Signed-off-by: Marc Duiker <[email protected]> Signed-off-by: Whit Waldo <[email protected]> Signed-off-by: joshvanl <[email protected]> Signed-off-by: Mark Fussell <[email protected]> Signed-off-by: Samantha Coyle <[email protected]> Signed-off-by: Sam <[email protected]> Signed-off-by: Javier Aliaga <[email protected]> Signed-off-by: Jorge André Gonçalves <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Josh van Leeuwen <[email protected]> Signed-off-by: Sergio Herrera <[email protected]> Signed-off-by: SAY-5 <[email protected]> Signed-off-by: Szymon Polom <[email protected]> Signed-off-by: Erwin <[email protected]> Signed-off-by: Bilgin Ibryam <[email protected]> Signed-off-by: Shekhar Luhar <[email protected]> Signed-off-by: MyMirelHub <[email protected]> Signed-off-by: Rishabh Dewangan <[email protected]> Signed-off-by: Eric Jacobson <[email protected]> Signed-off-by: Marc Duiker <[email protected]> Signed-off-by: VC <[email protected]> Signed-off-by: pittu sharma <[email protected]> Signed-off-by: newbe36524 <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: paulyuk <[email protected]> Signed-off-by: Mike Nguyen <[email protected]> Co-authored-by: Whit Waldo <[email protected]> Co-authored-by: joshvanl <[email protected]> Co-authored-by: Mark Fussell <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Sam <[email protected]> Co-authored-by: Alice Gibbons <[email protected]> Co-authored-by: Yaron Schneider <[email protected]> Co-authored-by: Javier Aliaga <[email protected]> Co-authored-by: Jorge André Gonçalves <[email protected]> Co-authored-by: Cassie Coyle <[email protected]> Co-authored-by: seherv <[email protected]> Co-authored-by: Sai Asish Y <[email protected]> Co-authored-by: SAY-5 <[email protected]> Co-authored-by: Szymon Połom <[email protected]> Co-authored-by: Erwin <[email protected]> Co-authored-by: Bilgin Ibryam <[email protected]> Co-authored-by: Shekhar Luhar <[email protected]> Co-authored-by: Mirel <[email protected]> Co-authored-by: Rishabh Dewangan <[email protected]> Co-authored-by: Eric Jacobson <[email protected]> Co-authored-by: Marc Duiker <[email protected]> Co-authored-by: VC <[email protected]> Co-authored-by: pittu sharma <[email protected]> Co-authored-by: Newbe36524 <[email protected]> Co-authored-by: Hagicode <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: paulyuk <[email protected]> Co-authored-by: Mike Nguyen <[email protected]>
Summary
Two fixes for the Pulsar pubsub component that caused unbounded unacked messages (~30k observed in production):
processMode silently ignored from component metadata:
processModewas only read from subscription request metadata (req.Metadata), so settingprocessMode: syncin the component YAML spec had no effect. Users believed they were running sync but were actually running async. Fix: addProcessModetopulsarMetadatastruct, normalize to lowercase at parse time, validate accepted values, and resolve effective mode inlistenMessageusing component metadata as default with subscription metadata override.Async mode has no concurrency limit (no backpressure): Every message spawned a goroutine with no upper bound.
MaxConcurrentHandlersonly controlled the channel buffer size, not the number of concurrent goroutines. Fix: add a semaphore channel that limits concurrent async handlers toMaxConcurrentHandlers, creating real backpressure — when all slots are full the loop stops reading from the channel, the channel fills up, and the SDK stops requesting more messages from the broker.Additional fixes:
errvariable was written by concurrent goroutines in async modeMaxConcurrentHandlers=0now falls back to default (100) instead of deadlocking the semaphoreprocessModevalues now return an error at parse time instead of silently falling through to asyncFiles changed
pubsub/pulsar/metadata.go— addedProcessModefieldpubsub/pulsar/pulsar.go— updatedlistenMessagewith mode resolution + semaphore, validation inparsePulsarMetadatapubsub/pulsar/process_mode_test.go— 22 test cases for processMode resolution (unit + integration)pubsub/pulsar/pulsar_backpressure_test.go— 8 test cases for async backpressure (concurrency limits, backpressure propagation, graceful shutdown, error handling)Test plan
go test -race)