Skip to content

[release/13.6] Keep DCP watches draining and close abandoned connections - #21002

Merged
Maddy Montaquila (maddymontaquila) merged 2 commits into
release/13.6from
danegsta-dcp-watcher-13-6-backport
Oct 20, 2026
Merged

Maddy Montaquila (maddymontaquila) merged 2 commits into
release/13.6from
danegsta-dcp-watcher-13-6-backport

Conversation

@danegsta

@danegsta David Negstad (danegsta) commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Description

Backport of #20972 to release/13.6.

Aspire can leave a DCP watch open without an active reader when watch setup is canceled or resource handlers pause event consumption. An unread HTTP/2 watch can block DCP watcher publication and resource mutations, leaving resource operations stalled. This backport explicitly owns and closes watch responses and independently drains events without changing resource event ordering or lifecycle callback dispatch. The existing five-minute reconnection safeguard remains in place.

Related to #20609 and microsoft/dcp#286. These are client-side fixes, not a replacement for API-server slow-consumer handling. No dependency or public API changes.

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No

Customer Impact

Customers whose DCP watch setup is canceled or whose resource handlers stall event consumption can experience blocked resource updates and mutations. This fix keeps the watch reader draining and closes abandoned connections while preserving event ordering and lifecycle dispatch.

Testing

On the resolved release/13.6 implementation, .\restore.cmd passed. The following Windows run passed: 364 tests, 0 failures, 1 Linux-only platform skip. Coverage includes watch cancellation/disposal, paused consumers, ordering across reconnections, periodic restart, framing/malformed records, Status handling, retryable HTTP statuses, existing release timeout behavior, and DCP executor lifecycle behavior.

$env:MSBUILDTERMINALLOGGER='false'
dotnet test --project .\tests\Aspire.Hosting.Tests\Aspire.Hosting.Tests.csproj --no-launch-profile -- --filter-class '*.KubernetesServiceTests' --filter-class '*.DcpExecutorTests' --filter-class '*.PeriodicRestartAsyncEnumerableTests' --filter-not-trait 'quarantined=true' --filter-not-trait 'outerloop=true'

Source-PR-only evidence: the original head passed 367 focused tests with one Windows platform skip, all PR checks, and Linux dogfood with concurrent restart rounds, exit-7 recovery, and mutations after more than five minutes idle. That dogfood was not repeated on this release branch; no local E2E tests were run.

Risk

Medium. No public API changes, but the fix changes core watch concurrency and HTTP response lifetimes. Lossless buffering preserves events; a persistently blocked handler can accumulate queued events instead of backpressuring DCP. Release-specific API timeout and create behavior are unchanged.

Regression?

Unknown — no specific regression introduction or affected release was established.

(cherry picked from commit 785cf4c)

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: d4e69fd0-651a-47cb-aeed-cff843f6aecc
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21002

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21002"

@aspire-repo-bot
aspire-repo-bot Bot requested a balanced review from Copilot October 14, 2026 16:42
@github-actions github-actions Bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Oct 14, 2026
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Core concurrency and response-lifetime changes require human sign-off, with transport-failure recovery still unresolved.

1 open finding
What changed in this PR

Backports DCP watch reliability improvements to release/13.6 without public API or dependency changes.

Changes:

  • Explicitly owns watch responses and closes abandoned connections.
  • Drains events independently while preserving ordering and five-minute reconnection.
  • Adds parsing, retry, cancellation, and HTTP/2 regression coverage.
File Description
tests/​Aspire.Hosting.Tests/​Dcp/​KubernetesServiceTests.cs Adds HTTP/2 watch reliability tests.
tests/​Aspire.Hosting.Tests/​Dcp/​DcpApiServerCertificateFixture.cs Supplies a shared test certificate.
src/​Aspire.Hosting/​Dcp/​KubernetesService.cs Buffers events independently of consumers.
src/​Aspire.Hosting/​Dcp/​DcpPipelineBuilder.cs Retries recoverable watch statuses.
src/​Aspire.Hosting/​Dcp/​DcpKubernetesClient.cs Owns responses and parses watch records.

🧠 Review effort: Balanced

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: d4e69fd0-651a-47cb-aeed-cff843f6aecc

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Core concurrency and response-lifetime changes warrant human sign-off and confirmation of the separate 13.6.2 version-bump prerequisite.

0 open findings

1 resolved since last review

🧠 Review effort: Balanced

@github-actions

Copy link
Copy Markdown
Contributor

Tests selector

50 / 99 PR test projects · 4 PR jobs, from 5 changed files.

Selected PR test projects (50 / 99)

Aspire.Cli.EndToEnd.Tests, Aspire.Hosting.Analyzers.Tests, Aspire.Hosting.Azure.Kubernetes.Tests, Aspire.Hosting.Azure.Kusto.Tests, Aspire.Hosting.Azure.Provisioning.Tests, Aspire.Hosting.Azure.Tests, Aspire.Hosting.Blazor.Tests, Aspire.Hosting.Browsers.Tests, Aspire.Hosting.CodeGeneration.Go.Tests, Aspire.Hosting.CodeGeneration.Java.Tests, Aspire.Hosting.CodeGeneration.Python.Tests, Aspire.Hosting.CodeGeneration.Rust.Tests, Aspire.Hosting.CodeGeneration.TypeScript.Tests, Aspire.Hosting.Containers.Tests, Aspire.Hosting.DevTunnels.Tests, Aspire.Hosting.Docker.Tests, Aspire.Hosting.Dotnet.Tests, Aspire.Hosting.DotnetTool.Tests, Aspire.Hosting.EntityFrameworkCore.Tests, Aspire.Hosting.Foundry.Tests, Aspire.Hosting.Garnet.Tests, Aspire.Hosting.Go.Tests, Aspire.Hosting.Java.Tests, Aspire.Hosting.JavaScript.Tests, Aspire.Hosting.Kafka.Tests, Aspire.Hosting.Keycloak.Tests, Aspire.Hosting.Kubernetes.Tests, Aspire.Hosting.Maui.Tests, Aspire.Hosting.Milvus.Tests, Aspire.Hosting.MongoDB.Tests, Aspire.Hosting.MySql.Tests, Aspire.Hosting.Nats.Tests, Aspire.Hosting.OpenAI.Tests, Aspire.Hosting.Oracle.Tests, Aspire.Hosting.Orleans.Tests, Aspire.Hosting.PostgreSQL.Tests, Aspire.Hosting.Python.Tests, Aspire.Hosting.Qdrant.Tests, Aspire.Hosting.RabbitMQ.Tests, Aspire.Hosting.Radius.Tests, Aspire.Hosting.Redis.Tests, Aspire.Hosting.RemoteHost.Tests, Aspire.Hosting.Rust.Tests, Aspire.Hosting.Seq.Tests, Aspire.Hosting.SqlServer.Tests, Aspire.Hosting.Testing.Tests, Aspire.Hosting.Tests, Aspire.Hosting.Valkey.Tests, Aspire.Hosting.Yarp.Tests, Aspire.Playground.Tests

Selected PR jobs (4)

cli-starter-validation, extension-e2e, polyglot, typescript-api-compat


How these were chosen — grouped by what changed

⚠️ 45 of the 50 selected test projects come from a single change — src/Aspire.Hosting/Dcp/DcpKubernetesClient.cs.

🔧 src/Aspire.Hosting/Dcp/DcpKubernetesClient.cs (changed source)
→ 45 via the project graph

show 45

Aspire.Hosting.Analyzers.Tests (2 hops), Aspire.Hosting.Azure.Kubernetes.Tests (2 hops), Aspire.Hosting.Azure.Kusto.Tests (2 hops), Aspire.Hosting.Azure.Provisioning.Tests (3 hops), Aspire.Hosting.Azure.Tests, Aspire.Hosting.Browsers.Tests (2 hops), Aspire.Hosting.CodeGeneration.Go.Tests, Aspire.Hosting.CodeGeneration.Java.Tests, Aspire.Hosting.CodeGeneration.Python.Tests, Aspire.Hosting.CodeGeneration.Rust.Tests, Aspire.Hosting.CodeGeneration.TypeScript.Tests, Aspire.Hosting.Containers.Tests (2 hops), Aspire.Hosting.DevTunnels.Tests (2 hops), Aspire.Hosting.Docker.Tests (2 hops), Aspire.Hosting.DotnetTool.Tests (2 hops), Aspire.Hosting.EntityFrameworkCore.Tests (2 hops), Aspire.Hosting.Foundry.Tests (2 hops), Aspire.Hosting.Garnet.Tests (2 hops), Aspire.Hosting.Go.Tests (2 hops), Aspire.Hosting.Java.Tests (2 hops), Aspire.Hosting.JavaScript.Tests (2 hops), Aspire.Hosting.Kafka.Tests (2 hops), Aspire.Hosting.Keycloak.Tests (2 hops), Aspire.Hosting.Kubernetes.Tests (2 hops), Aspire.Hosting.Maui.Tests, Aspire.Hosting.Milvus.Tests (2 hops), Aspire.Hosting.MongoDB.Tests (2 hops), Aspire.Hosting.MySql.Tests (2 hops), Aspire.Hosting.Nats.Tests (2 hops), Aspire.Hosting.OpenAI.Tests (2 hops), Aspire.Hosting.Oracle.Tests (2 hops), Aspire.Hosting.Orleans.Tests (2 hops), Aspire.Hosting.PostgreSQL.Tests (2 hops), Aspire.Hosting.Python.Tests (2 hops), Aspire.Hosting.Qdrant.Tests (2 hops), Aspire.Hosting.RabbitMQ.Tests (2 hops), Aspire.Hosting.Redis.Tests (2 hops), Aspire.Hosting.RemoteHost.Tests, Aspire.Hosting.Rust.Tests (2 hops), Aspire.Hosting.Seq.Tests (2 hops), Aspire.Hosting.SqlServer.Tests (2 hops), Aspire.Hosting.Testing.Tests (2 hops), Aspire.Hosting.Valkey.Tests (2 hops), Aspire.Hosting.Yarp.Tests (2 hops), Aspire.Playground.Tests

🧪 tests/Aspire.Hosting.Tests/Dcp/DcpApiServerCertificateFixture.cs (changed test)
→ 1 directly: Aspire.Hosting.Tests
→ 3 via the project graph: Aspire.Hosting.Blazor.Tests, Aspire.Hosting.Dotnet.Tests, Aspire.Hosting.Radius.Tests

📦 affected project Aspire.Hosting
→ 1 test: Aspire.Cli.EndToEnd.Tests

🧪 tests/Aspire.Hosting.Tests/Dcp/KubernetesServiceTests.cs (changed test)
→ 1 directly: Aspire.Hosting.Tests

Job reasons

Job Triggered by
cli-starter-validation affected project Aspire.Hosting.JavaScript
extension-e2e • src/Aspire.Hosting/Dcp/DcpKubernetesClient.cs, src/Aspire.Hosting/Dcp/DcpPipelineBuilder.cs, src/Aspire.Hosting/Dcp/KubernetesService.cs
• affected project Aspire.Hosting
polyglot • affected project Aspire.Hosting
• affected project Aspire.Hosting.Azure.Provisioning.KeyVault
typescript-api-compat affected project Aspire.Hosting

Selection computed for commit 5204ab8.

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@maddymontaquila
Maddy Montaquila (maddymontaquila) merged commit acea43f into release/13.6 Oct 20, 2026
864 of 872 checks passed
@aspire-repo-bot

Copy link
Copy Markdown
Contributor

✅ No documentation update needed.

Step 5 branch taken: excluded → backport

Triggered signals: pr_body_has_cli_flag_mention (1 total) — but irrelevant since excluded == true overrides the recommendation.

Exclusion reasons (from .pr-docs-check/signals.json):

This PR is a backport of #20972 onto release/13.6. Per the exclusion rules, backport PRs are out of scope for docs generation — the user-facing documentation (if any) belongs with the original forward PR (#20972) on the default branch, not duplicated here. No docs PR was drafted.

This was referenced Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants