Skip to content

fix(local-runtime): recover stopped plugin instances - #766

Merged
fatelei merged 2 commits into
langgenius:mainfrom
Kevin9703:fix/local-runtime-stopped-instances
Jul 6, 2026
Merged

fix(local-runtime): recover stopped plugin instances#766
fatelei merged 2 commits into
langgenius:mainfrom
Kevin9703:fix/local-runtime-stopped-instances

Conversation

@Kevin9703

Copy link
Copy Markdown
Contributor

Description

Fixes #765

This pull request fixes a local runtime recovery issue where plugin-daemon can continue routing requests to a local plugin instance after that instance's stdio pipes have already been closed.

When this happens, dispatch fails before the request reaches the plugin process, for example:

PluginDaemonInternalServerError: write |1: file already closed
failed to write request

The root cause is that stopped local plugin instances are removed from the runtime instance pool asynchronously. During that gap, the load balancer can still select the stale instance, and the write path can still attempt to write to its closed stdin pipe.

This PR makes the local runtime recover from that state by:

  • marking PluginInstance as stopped synchronously when stdio is closed or stdout exits
  • skipping stopped instances during local runtime load balancing
  • evicting dead instances when stdin writes fail with closed-pipe style errors
  • waking the scheduler after eviction so replacement instances can be started promptly
  • retrying local write-before-dispatch failures within a small bounded window
  • serializing writes to a plugin instance's stdin to avoid interleaved payloads from concurrent sessions

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Performance improvement
  • Other

Essential Checklist

Testing

  • I have tested the changes locally and confirmed they work as expected
  • I have added unit tests where necessary and they pass successfully

Bug Fix (if applicable)

  • I have used GitHub syntax to close the related issue (e.g., Fixes #123 or Closes #123)

Additional Information

Local validation:

  • go test ./internal/core/local_runtime/... ./internal/core/io_tunnel/...
  • go build ./cmd/server

The change is limited to local plugin runtime instance lifecycle, load balancing, stdin write handling, and write-before-dispatch retry behavior. It does not change serverless runtime behavior.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces robust error recovery and instance eviction mechanisms for local plugin runtimes. It implements write retry logic with a budget in GenericInvokePlugin, adds state tracking (stopped flag) and thread-safe writes to PluginInstance, and updates the scheduling loop to prune stopped instances and immediately trigger rescheduling via a new scheduleKick channel. Additionally, it updates load balancing to skip stopped instances and adds unit tests for these recovery behaviors. The reviewer suggested adding a defensive nil check for s.inWriter in Write to prevent potential nil pointer dereferences during testing or partial initialization.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread internal/core/local_runtime/instance.go
@Kevin9703
Kevin9703 marked this pull request as ready for review July 2, 2026 03:53
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. bug Something isn't working go Pull requests that update go code plugin-daemon labels Jul 2, 2026
Comment thread internal/core/local_runtime/instance.go Outdated
@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Jul 6, 2026
@fatelei
fatelei merged commit 3b51c4d into langgenius:main Jul 6, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working go Pull requests that update go code lgtm This PR has been approved by a maintainer plugin-daemon size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Local runtime can keep routing requests to stopped plugin instances

2 participants