Skip to content

fix(adkrest): render missing toolset tools and sub-agent connections in agent graph - #1730

Open
karthik-0306 wants to merge 5 commits into
google:mainfrom
karthik-0306:fix-agentgraph-toolsets
Open

karthik-0306 wants to merge 5 commits into
google:mainfrom
karthik-0306:fix-agentgraph-toolsets

Conversation

@karthik-0306

@karthik-0306 karthik-0306 commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Linked issue

Closes: #1726

2. Describe the change:

Problem: The agent graph generator leaves out tools from Toolsets (like MCP, skills) because buildGraph only reads the explicit agent.Tools field. It also draws sub-agents of an llmagent without edges to their parent, so they appear as disconnected nodes.

Solution:

  • Toolset tools: each toolset's tools are listed and drawn. The graph runs outside any invocation, so a toolset that can't list its tools there is skipped and logged at WARN with its name and Go types. This covers a toolset that errors or panics, uses per-user auth, or whose server is down. adk-python also skips it, with a log line.
  • Bounded wait, without canceling: every toolset in the tree starts listing when the graph starts, each on its own goroutine under a context that keeps the request's values but not its cancellation. The graph waits at most 5s, then draws without the toolsets that aren't ready and logs them with kind=timeout. A listing is never canceled: canceling a stdio MCP toolset while it connects kills the server's process, and mcptoolset can't start it again, so every later run of the agent would fail. An unfinished listing completes in the background, and later runs reuse the session it opened. A listing still running is shared by later graphs, so a server that never answers leaves one goroutine behind, not one per request.
  • Listing context: toolsets are listed with a context that carries the agent's name and reads as empty for the user, session, state and invocation, instead of panicking (as exp/appinfo does). So toolsets like tool/toolsearch that read state still list their tools. It reports no invocation identity, so nothing lists tools as whoever requested the graph.
  • Sub-agent edges: an llmagent connects to each sub-agent. A workflow sub-agent is drawn as a cluster, so the edge goes into it with lhead, as in adk-python's agent_graph.py. Edges of an llmagent nested in a workflow cluster are drawn on the graph that gets serialized.
  • DOT IDs: node IDs, edge endpoints, cluster names and ltail/lhead values that aren't plain identifiers are quoted and escaped. MCP tool names like resolve-library-id previously made Graphviz reject the whole graph.

Behavior change

  • Graphs now show toolset tools and llmagent → sub-agent edges.
  • Agents or tools whose names aren't plain identifiers now render. That includes hyphens, dots, quotes and DOT keywords. On main these failed to parse or misrendered. Graphs with only plain names produce byte-identical DOT.
  • Building a graph now calls Toolset.Tools, so an MCP toolset connects to its server. The graph waits at most 5s. A toolset that fails or isn't ready by then is left out and logged at WARN; the client gets no error. Listings are never canceled, so later agent runs are unaffected, and a server that was still starting appears on the next render.

Testing Plan

Unit Tests: On a clean checkout with Go 1.26.10, go test -race -mod=readonly -count=1 -shuffle=on work passes. go mod tidy -diff, go build, go vet and golangci-lint run (v2.3.1) are clean in both modules. The two touched packages also pass 10 repeated shuffled -race runs.

With your source change reverted and your tests kept, which test fails?
I removed each piece of the change separately and confirmed the matching test fails:

  • toolset listing / sub-agent edge → TestAgentGraphWithToolsetAndSubAgent, TestAgentGraphLLMAgentWithWorkflowSubAgent
  • recover / skip of a failing toolset → TestAgentGraphSkipsToolsetsThatCannotListTools (panic / error)
  • edges drawn on the serialized graph → TestAgentGraphLLMAgentInsideWorkflow
  • DOT quoting → TestDotID, TestAgentGraphQuotesNamesThatAreNotDOTIdentifiers, TestBuildGraphImageHandlerQuotesToolsetToolNames
  • agent name in the toolset context → TestAgentGraphToolsetSeesItsAgent
  • logging, including a toolset whose Name() panics → TestAgentGraphLogsSkippedToolsets
  • listing not canceled by the budget or by the request ending → TestAgentGraphLeavesAStartingToolsetUsable, TestAgentRunAfterGraphWithSlowStdioServer (a real stdio MCP server through CommandTransport)
  • wait bounded by the budget → TestAgentGraphDoesNotWaitPastItsBudget
  • all toolsets start together → TestAgentGraphListsToolsetsTogether
  • running listings shared → TestAgentGraphSharesAListingStillRunning
  • session-less context with no identity → TestAgentGraphListsToolsetsWithoutASession (uses tool/toolsearch)

Manual End-to-End (E2E) Tests:
I ran the reproduction from #1726 with web -port 8097 api -include_debug_api and requested curl /api/dev/apps/root/build_graph_image.

  • before (as reported in Agent graph omits tools from Toolsets (MCP, skills) and draws llmagent sub-agents without edges #1726): toolset_tool was missing and helper was a floating node.
  • after: root connects to direct_tool, toolset_tool and helper.
  • more variants:
    • toolset tools named resolve-library-id and say"hi render.
    • a failing toolset, a hung MCP server and a nested workflow render with the rest drawn and WARN lines.
    • stdio MCP server that takes 7s to start (CommandTransport):
      • the graph answers 200 in 5.0s, POST /run afterwards returns 200, and the next graph shows the server's tool.
      • On c0ab228 the same run returned 500 exec: Stdout already set.
    • stdio server that never answers and ignores SIGTERM:
      • the graph answers 200 in 5.0s.
      • On c0ab228 the client got no response after 15s.

The graph outputs render with Graphviz 16.1.0.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

@karthik-0306
karthik-0306 force-pushed the fix-agentgraph-toolsets branch 2 times, most recently from 4cff038 to d0be297 Compare October 8, 2026 12:40

@jjsasha63 jjsasha63 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The root-level case from #1726 works: I removed each half of teh fix in turn and TestAgentGraphWithToolsetAndSubAgent went red both times. Outside that case,the new sub-agent edge breaks the graph for any coordinator with a workflow sub-agent, and resolving toolsets lets a single toolset fail the whole graph.

I ran probe tests against GetAgentGraph and the build_graph_image handler and compared them with main at 49904c0:

  • An llmagent root with a sequential, loop or parallel sub-agent gets the DOT body error: pipeline is not a node or a subgraph, served with HTTP 200. Main renders it.
  • A toolset whose Tools reads ctx.UserID(), or a tool.FilterToolset predicate that reads ctx.ReadonlyState(), panics with a nil pointer dereference and the client gets EOF. main renders it.
  • An mcptoolset with an unreachable server, or with per-user auth/gcp auth, turns the whole graph into a 500. Main renders it.
  • An llmagent inside a sequential agent gets no edges to its toolset tools or sub-agents.

Parity with adk-python 2.11.0 (src/google/adk/cli/agent_graph.py, src/google/adk/agents/llm_agent.py): the root-level behavior matches agent_graph.py:289-301. Python skips the edge when the sub-agent is a workflow agent and conects the parent into the cluster instead (agent_graph.py:159-160, 177-178, 196-199, 292-297). It also logs and skips a toolset that fails to load, so the rest of the graph still renders (llm_agent.py:224-245).

@karthik-0306
karthik-0306 force-pushed the fix-agentgraph-toolsets branch from d0be297 to 290c5dc Compare October 9, 2026 18:01
@karthik-0306

Copy link
Copy Markdown
Contributor Author

The root-level case from #1726 works: I removed each half of teh fix in turn and TestAgentGraphWithToolsetAndSubAgent went red both times. Outside that case,the new sub-agent edge breaks the graph for any coordinator with a workflow sub-agent, and resolving toolsets lets a single toolset fail the whole graph.

I ran probe tests against GetAgentGraph and the build_graph_image handler and compared them with main at 49904c0:

  • An llmagent root with a sequential, loop or parallel sub-agent gets the DOT body error: pipeline is not a node or a subgraph, served with HTTP 200. Main renders it.
  • A toolset whose Tools reads ctx.UserID(), or a tool.FilterToolset predicate that reads ctx.ReadonlyState(), panics with a nil pointer dereference and the client gets EOF. main renders it.
  • An mcptoolset with an unreachable server, or with per-user auth/gcp auth, turns the whole graph into a 500. Main renders it.
  • An llmagent inside a sequential agent gets no edges to its toolset tools or sub-agents.

Parity with adk-python 2.11.0 (src/google/adk/cli/agent_graph.py, src/google/adk/agents/llm_agent.py): the root-level behavior matches agent_graph.py:289-301. Python skips the edge when the sub-agent is a workflow agent and conects the parent into the cluster instead (agent_graph.py:159-160, 177-178, 196-199, 292-297). It also logs and skips a toolset that fails to load, so the rest of the graph still renders (llm_agent.py:224-245).

Thanks for the thorough review and edge-case testing. I've updated the PR to address all of these issues to bring it into full parity with the Python SDK:

  • Workflow sub-agent edges: Switched to using drawClusterEdge for sub-agents, which correctly connects the parent into the cluster instead of causing the not a node DOT error.
  • Panics and 500 errors on Toolsets: Wrapped the toolset.Tools() call in a defer recover() block and added an error check. If a toolset panics (e.g., from missing session context) or fails to load (e.g., unreachable MCP server), we catch it and simply skip that toolset so the rest of the graph continues to render cleanly.
  • Missing edges in nested workflows: Switched to routing these edges to parentGraph instead of graph so they serialize properly when an agent is nested inside a workflow cluster. I also added a small deduplication check in drawEdge to prevent drawing duplicate parallel lines.

@karolpiotrowicz karolpiotrowicz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things need to change before this merges. Toolset tool names go into the DOT source unquoted, so an MCP toolset with a hyphenated tool name now makes the whole graph unrenderable. The new failure paths also have no tests.

The four cases from the earlier review render correctly on this head. I drove each one through BuildGraphImageHandler and GetAgentGraph. An llmagent with a sequential, loop or parallel sub-agent gets an edge into the cluster with lhead. Toolsets that panic on ctx.UserID(), filter on ReadonlyState() or return an error are skipped and the rest of the graph renders. An llmagent inside a sequential agent gets edges to its tools, its toolset tools and its sub-agent. TestAgentGraphWithToolsetAndSubAgent fails on the PR's base (49904c0) and passes on this head (290c5dc).

Must change before merge:

  • Toolset tool names reach the DOT output as bare IDs. The new loop at agentgraphgenerator.go#L394-L402 hands each toolset tool to drawNode and drawEdge, which use the name as the node ID without quoting (L321, L343). MCP servers pick these names, and hyphens are common (Context7's resolve-library-id, for example). For an llmagent with one direct tool and a toolset returning resolve-library-id, the generated DOT contains root->resolve-library-id[...], and Graphviz rejects it with syntax error in line 5 near '-'. The web UI then has nothing it can draw. Before this change the same agent rendered, just without the toolset tool. A name containing " fails the same way. A directly configured tool with a hyphen already fails like this on main, so the gap is older than this PR, but the PR now sends every MCP tool name through it. Whatever the fix, every node ID the generator emits has to be valid DOT for any tool name, on edges as well as nodes.

  • The new failure paths have no tests. Under the package's tests, the recover block at L381-L383 and the skip branch at L388-L392 never run, so deleting either one leaves the suite green. No test builds an llmagent with a workflow sub-agent either, which is the drawClusterEdge call at L411-L416, and TestAgentGraphWithToolsetAndSubAgent is the only test that uses Toolsets at all, so a toolset on an llmagent nested inside a workflow agent is not covered. The cases worth pinning are an erroring toolset, a panicking toolset, an llmagent with a sequential sub-agent, and a toolset on an llmagent inside a sequential agent. Parsing the output with the existing parseGraph helper instead of strings.Contains would also catch the ID problem above: gographviz.ParseString rejects the hyphenated DOT.

Reproduction for the hyphenated name (drop into server/adkrest/internal/services, run go test -run TestHyphenatedToolsetName -v ./server/adkrest/internal/services/, then feed the logged DOT to dot -Tsvg)
package services

import (
	"context"
	"testing"

	"github.com/awalterschulze/gographviz"
	"google.golang.org/adk/v2/agent"
	"google.golang.org/adk/v2/agent/llmagent"
	"google.golang.org/adk/v2/tool"
)

type namedTool struct{ n string }

func (t namedTool) Name() string        { return t.n }
func (t namedTool) Description() string { return "" }
func (t namedTool) IsLongRunning() bool { return false }

type mcpLikeToolset struct{}

func (mcpLikeToolset) Name() string { return "mcp" }
func (mcpLikeToolset) Tools(agent.ReadonlyContext) ([]tool.Tool, error) {
	return []tool.Tool{namedTool{"resolve-library-id"}}, nil
}

func TestHyphenatedToolsetName(t *testing.T) {
	root, err := llmagent.New(llmagent.Config{Name: "root", Model: &dummyLLM{}, Toolsets: []tool.Toolset{mcpLikeToolset{}}})
	if err != nil {
		t.Fatal(err)
	}
	dot, err := GetAgentGraph(context.Background(), root, nil)
	if err != nil {
		t.Fatal(err)
	}
	t.Log(dot)
	if _, err := gographviz.ParseString(dot); err != nil {
		t.Errorf("generated DOT does not parse: %v", err)
	}
}

…n agent graph

Address review feedback on the agent graph changes for google#1726.

Quote DOT IDs that are not plain identifiers. Toolset tools now reach
the graph, and MCP servers pick their own tool names, so a name like
Context7's resolve-library-id made Graphviz reject the whole source. A
hyphenated direct tool or agent already failed this way on main, and a
bare `node`, `ns:tool` or a name with a double quote misrendered
silently. Every node, edge endpoint, cluster and ltail/lhead value now
goes through dotID: plain ASCII identifiers that are not DOT keywords
stay bare, so graphs that rendered before produce the same source, and
anything else is quoted with backslash and double quote escaped.
Labels get the same escaping; highlighting still matches raw names.

Give toolset resolution the agent being drawn, so a toolset or a
tool.FilterToolset predicate calling ctx.AgentName() works instead of
panicking and dropping the toolset. There is still no session or user,
so a toolset that needs them is skipped, as adk-python does.

Log a skipped toolset at WARN with the agent, toolset name, its Go
type, and the type of the error or panic value. The values are not
logged, since they can carry a server URL or credentials. Skip nil
toolsets before logging.

Bound toolset resolution with one 5-second budget for the whole walk.
An MCP server that accepts the connection and never answers held the
graph for over 90 seconds in a local probe. 10 seconds, as exp/appinfo
uses, plus the MCP SDK's ~5 seconds of session cleanup reached the web
launcher's default 15-second write timeout and the client got no
response; with 5 seconds it answers 200 in about 10 seconds.

Add tests that assert on the parsed graph for an erroring and a
panicking toolset, an llmagent with a sequential, loop or parallel
sub-agent, an llmagent with tools, a toolset and a sub-agent inside a
sequential agent, names that are not DOT identifiers (also through
BuildGraphImageHandler), the agent in the toolset context, the log
records, and the resolution budget.

Related: google#1726
@karthik-0306
karthik-0306 force-pushed the fix-agentgraph-toolsets branch from 47489ed to c0ab228 Compare October 16, 2026 09:21
@karthik-0306

Copy link
Copy Markdown
Contributor Author

@karolpiotrowicz @jjsasha63 thanks for the reviews. Pushed 47489ed, which addresses everything:

  • A, DOT IDs: every node, edge endpoint, cluster and lhead/ltail goes through dotID. Plain identifiers stay bare (byte-identical output); everything else is quoted with \ and " escaped. This also fixes hyphenated direct tools/agents on main. Your resolve-library-id case and names with ", \, node, ns:tool now parse with parseGraph and render in real Graphviz.

  • B, tests: an erroring toolset, a panicking toolset, an llmagent with a sequential/loop/parallel sub-agent, and @jjsasha63's pipeline{worker{...}, last} case. All assert on parseGraph. I removed each piece of production code in turn and confirmed the matching test fails.

  • C: the toolset context now carries the agent; tested with a FilterToolset on ctx.AgentName().

  • D: a skipped toolset logs at WARN with agent, toolset name, %T and the error/panic type. Two choices to confirm:

    • It logs types rather than the error text, since AGENTS.md forbids error text that may carry URLs or credentials.
    • It's WARN rather than ERROR because the graph still renders. Happy to switch to ERROR.

    It uses the default slog logger with a //nolint:forbidigo exemption, like exp/appinfo. Please confirm that's OK per AGENTS.md.

  • Also fixed: an MCP server that never answers held the graph for over 90s. Toolset resolution now has a 5s budget. I chose 5s, not appinfo's 10s, because 10s plus the MCP SDK's ~5s session cleanup hit the launcher's 15s write timeout and the client got no response.

Full suite with -race, go vet and golangci-lint v2.3.1 pass on Go 1.26.6.

@karolpiotrowicz
karolpiotrowicz dismissed their stale review November 1, 2026 21:30

Every item this review raised is fixed and verified on the current head.

@karolpiotrowicz karolpiotrowicz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest commit (c0ab228) adds two problems that need fixing before this PR merges. Building the graph can now leave a stdio MCP toolset broken for every later run of the agent. The comment that justifies the new 5s budget for building the graph also does not hold for stdio servers.

Both points from my earlier review are fixed. I rendered 40 names that are not plain identifiers as toolset tools, direct tools, sub-agents and workflow agents. They cover hyphens, quotes, trailing backslashes, DOT keywords, colons as in ns:tool, newlines, the empty string and invalid UTF-8. An agent cannot have an empty name, so the 40 names in four positions make 158 graphs rather than 160. On this head Graphviz accepts all of them. The 154 that are valid UTF-8 have the expected nodes and edges. On the previous head (290c5dc), 129 of the 158 failed to parse or had the wrong nodes and edges. The failure paths have tests now, and a tree that uses only plain names gives byte-identical DOT to the previous head. The checks I listed last time all pass again.

Must change before merge:

  • A graph request that runs out of budget while a stdio MCP server is starting breaks its toolset until the ADK server restarts. The 5s budget from agentgraphgenerator.go#L547-L549 at c0ab228 sets the deadline on the context mcptoolset connects with (client.go#L174-L194). When the MCP server needs longer than 5s to answer initialize, the MCP SDK closes the session and kills the server's process (go-sdk v1.8.0 client.go#L390-L393). mcptoolset keeps the same CommandTransport, whose exec.Cmd cannot be started a second time (go-sdk v1.8.0 cmd.go#L29-L46). Every later run of any agent using that toolset then fails with failed to extract tools from the tool set "mcp_tool_set": ... exec: Stdout already set (tools_processor.go#L40-L44).

    The toolset breaks before anyone runs the agent, because the dev UI requests the graph as soon as an app is selected. CommandTransport is the setup the mcptoolset docs show (set.go#L63-L65). A server launched through npx -y or uvx downloads its package before it starts, so its first initialize waits on the network. The graph build connects each toolset in turn under one 5s budget. A server that starts after another one gets what is left.

    On the previous head the reproduction below passes, because the graph build waits for initialize, and later agent runs reuse the MCP session that the graph build opened. The underlying problem is older than this PR. mcptoolset reuses a CommandTransport whose process was already killed, and on main a first agent run that is cancelled while the server starts breaks the toolset the same way. The new budget breaks the toolset this way whenever a server needs more than 5s to start, without anyone cancelling anything. Any fix has to leave later runs unaffected by building the graph.

  • The budget comment's timing is wrong for stdio servers. agentgraphgenerator.go#L518-L522 says an MCP toolset spends about 5s more after its context ends, which keeps the response under the launcher's 15s write timeout. For a stdio server the SDK waits 5s after closing stdin and another 5s after SIGTERM before it kills the process (go-sdk v1.8.0 cmd.go#L69-L107). The 5s budget plus up to 10s of shutdown reaches the 15s write timeout. In my test the server never answers and ignores SIGTERM. A handler calling GetAgentGraphWithTheme ran behind an http.Server with the launcher's 15s WriteTimeout, and after 15s the client got EOF with no body.

Reproduction for the stdio toolset (save as server/adkrest/internal/services/stdio_graph_repro_test.go and run go test -run TestAgentRunAfterGraphWithSlowStdioServer ./server/adkrest/internal/services/. It fails on this head and passes on the previous head.)
package services

import (
	"context"
	"iter"
	"os"
	"os/exec"
	"testing"
	"time"

	"github.com/modelcontextprotocol/go-sdk/mcp"
	"google.golang.org/genai"

	"google.golang.org/adk/v2/agent"
	"google.golang.org/adk/v2/agent/llmagent"
	"google.golang.org/adk/v2/model"
	"google.golang.org/adk/v2/runner"
	"google.golang.org/adk/v2/session"
	"google.golang.org/adk/v2/tool"
	"google.golang.org/adk/v2/tool/mcptoolset"
)

// Re-executed as a stdio MCP server that needs 7s before it answers.
func TestSlowStdioServerHelper(t *testing.T) {
	if os.Getenv("SLOW_MCP_SERVER") != "1" {
		t.Skip("helper process")
	}
	time.Sleep(7 * time.Second)
	srv := mcp.NewServer(&mcp.Implementation{Name: "slow", Version: "v0"}, nil)
	mcp.AddTool(srv, &mcp.Tool{Name: "lookup"}, func(context.Context, *mcp.CallToolRequest, struct{}) (*mcp.CallToolResult, struct{}, error) {
		return &mcp.CallToolResult{}, struct{}{}, nil
	})
	_ = srv.Run(context.Background(), &mcp.StdioTransport{})
	os.Exit(0)
}

type okModel struct{}

func (okModel) Name() string { return "ok" }
func (okModel) GenerateContent(context.Context, *model.LLMRequest, bool) iter.Seq2[*model.LLMResponse, error] {
	return func(yield func(*model.LLMResponse, error) bool) {
		yield(&model.LLMResponse{Content: genai.NewContentFromText("ok", genai.RoleModel)}, nil)
	}
}

func TestAgentRunAfterGraphWithSlowStdioServer(t *testing.T) {
	cmd := exec.Command(os.Args[0], "-test.run=^TestSlowStdioServerHelper$")
	cmd.Env = append(os.Environ(), "SLOW_MCP_SERVER=1")
	ts, err := mcptoolset.New(mcptoolset.Config{Transport: &mcp.CommandTransport{Command: cmd}})
	if err != nil {
		t.Fatal(err)
	}
	root, err := llmagent.New(llmagent.Config{Name: "root", Model: okModel{}, Toolsets: []tool.Toolset{ts}})
	if err != nil {
		t.Fatal(err)
	}

	// The dev UI requests the graph when the app is opened, before any run.
	if _, err := GetAgentGraph(context.Background(), root, nil); err != nil {
		t.Fatal(err)
	}

	r, err := runner.New(runner.Config{AppName: "app", Agent: root, SessionService: session.InMemoryService(), AutoCreateSession: true})
	if err != nil {
		t.Fatal(err)
	}
	for _, err := range r.Run(context.Background(), "user", "s1", genai.NewContentFromText("hi", genai.RoleUser), agent.RunConfig{}) {
		if err != nil {
			t.Fatalf("agent run after the graph: %v", err)
		}
	}
}

On your logging questions

WARN fits the level table in AGENTS.md (AGENTS.md#L163-L169), since the graph still renders. Logging the error's type rather than its text follows the AGENTS.md rule against URLs in log lines (AGENTS.md#L134-L143), because an MCP error can carry the server's URL. The //nolint:forbidigo follows the same pattern as exp/appinfo, which also skips a toolset that cannot list its tools and logs only the error's type (appinfo.go#L309).

Building the graph listed each toolset under a 5-second deadline. When a
stdio MCP server took longer than that to answer initialize, as one
launched through npx -y or uvx does while it downloads, the deadline
made the MCP SDK close the session and kill the server's process.
mcptoolset keeps the same CommandTransport, which cannot start the
process again, so every later run of the agent failed with "exec:
Stdout already set". The dev UI requests the graph as soon as an app is
opened, so this happened before anyone ran the agent. The deadline
also did not bound the response: the SDK waits up to 10 seconds more
for a stdio server to exit, which reached the web launcher's 15-second
write timeout.

List each toolset on its own goroutine under a context that keeps the
request's values but not its cancellation, and only stop waiting once
the budget is spent. A listing that has not returned is left out of
that graph, logged with kind "timeout", and finishes in the background,
so later runs reuse the session it opened. Start every toolset in the
tree when the graph starts, so each gets the whole budget, and let a
graph built while a listing is still running wait for that one, so a
server that never answers leaves one goroutine behind instead of one
per request.

List toolsets with a context that reads as empty instead of panicking
for the user, session, state and invocation a run would supply, as
exp/appinfo does, so toolsets such as tool/toolsearch that read state
list their tools. It reports no invocation identity, so nothing lists
tools as whoever requested the graph.

Read a skipped toolset's name for the log under a recover: a wrapper
around a nil toolset, such as tool.WithConfirmation(nil, ...), passes
the nil check and panics in Name as well as Tools.

Related: google#1726
@karthik-0306

Copy link
Copy Markdown
Contributor Author

@karolpiotrowicz thanks, your stdio repro was spot on. Both must-fix points are fixed in 07f79b4. I also merged main first, since mcptoolset changed in #1755 and #1844.

1. The graph broke stdio toolsets for later runs

The graph no longer cancels a toolset's listing; it only stops waiting for it.

  • Each Toolset.Tools call runs on its own goroutine under context.WithoutCancel of the request context.
  • After the 5s budget, a listing that hasn't returned is left out of that graph, logged with kind=timeout, and finishes in the background. Later runs reuse the session it opened.
  • Every toolset in the tree starts listing when the graph starts, so a server that starts after another one still gets the full budget.
  • A listing that is still running is shared by later graphs (keyed by agent and toolset), so a server that never answers leaves one goroutine behind, not one per request.

Your TestAgentRunAfterGraphWithSlowStdioServer is in the PR as agentgraph_stdio_test.go, shortened to a 1s server against a 100ms budget. Putting the deadline back on the listing context makes it fail again.

End to end with the web launcher and a CommandTransport server that sleeps 7s before serving:

c0ab228 07f79b4
first graph 200 in 7.07s 200 in 5.02s
POST /run afterwards 500: exec: Stdout already set 200
next graph lookup never appears lookup drawn, in 0.02s

2. The budget comment's timing

Nothing is canceled now, so no MCP shutdown time is added to the response, and the comment is rewritten to say so. With a stdio server that never answers and ignores SIGTERM, behind the launcher's 15s write timeout:

  • c0ab228: no response after 15.03s
  • now: 200 in 5.02s

TestAgentGraphDoesNotWaitPastItsBudget pins this with a toolset that ignores its context.

Non-blocking

  • Name() outside the recover: the name for the log is now read under a recover. tool.WithConfirmation(nil, true, nil) is in TestAgentGraphLogsSkippedToolsets; it's logged with toolset "", *tool.confirmationToolset, and panic type runtime.errorString.
  • Session-less context: toolsets are now listed with a context modeled on appinfo's. User, session, state and invocation read as empty instead of panicking, so toolsearch lists its tools; TestAgentGraphListsToolsetsWithoutASession uses a real toolsearch.New. Two differences from appinfo:
    • It reports no invocation identity even when the request context carries one, so nothing lists tools as the person who requested the graph.
    • AppName() is "", because the graph functions don't receive the app name.

Tests

I broke each new piece of code in turn and saw the matching test fail:

  • listing that inherits the request's cancellation
  • listing under the budget deadline (c0ab228's behavior)
  • unbounded wait
  • no up-front start
  • no sharing of running listings
  • unrecovered panic on the listing goroutine
  • unguarded Name()
  • nil state
  • identity passed through
  • empty AgentName()

Earlier mutations, such as quoting, the skip branch and the sub-agent and nested edges, still fail their tests.

On a clean checkout with Go 1.26.10:

  • go mod tidy -diff, build, vet and golangci-lint v2.3.1 are clean in both modules.
  • go test -race -shuffle=on work passes.
  • The two touched packages pass 10 repeated shuffled -race runs.

Older problems, out of scope

I can open issues for these:

  1. mcptoolset reuses a CommandTransport whose process was killed, so any canceled first connect breaks the toolset, on main too.
  2. exp/appinfo lists toolsets under a 10s WithTimeout, so the same stdio breakage applies there.

@karolpiotrowicz karolpiotrowicz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My change request from my earlier review stands. A graph request can still leave an MCP toolset unusable for every later run and block its Close, which is what I asked the fix to rule out.

What the new commit fixes:

  • With a stdio server that needs 7s before it answers, the graph now returns in 5s without that server's tools. Two agent runs started after the graph request now succeed, and the next graph request shows the server's tool. On the previous head (c0ab228) both runs failed with exec: Stdout already set.
  • The graph's response now stays within its 5s budget. With a stdio server that never answers and ignores SIGTERM, a graph endpoint served under the launcher's 15s write timeout returns 200 in 5s.
  • The toolset name for the log is now read under a recover.
  • toolsearch now lists its own tools in the graph.

The build, lint and test checks from my earlier reviews pass again on the current head, 07f79b4, and every line link below is pinned to it.

On the older problems you listed, #2111 already covers mcptoolset reusing a CommandTransport whose process was killed, and the exp/appinfo timeout is worth an issue of its own.

Must change before merge

A graph request against an MCP server that never answers leaves that toolset locked for the life of the process. The listing runs under context.WithoutCancel (agentgraphtoolsets.go#L111), so nothing can end it. While mcptoolset connects, getSession holds the connection's lock for the whole connect (client.go#L211-L231, client.go#L264-L272). The MCP SDK puts no timeout of its own on connecting. Against a server that accepts the connection and never answers, holding the lock through a connect that nothing can cancel has two effects:

  • Later runs never reconnect. Every later agent run waits for the lock until its own deadline and fails. For an HTTP server, later runs keep failing after the server recovers, because no new connect is ever tried. In the reproduction below the server holds its first request and serves every later one normally. After a graph request, two runs with a 6s deadline both fail, and the server sees one request in total. Without the graph request the second run succeeds. On the previous head the graph's deadline released the lock and both runs succeeded. With Config.Auth set to auth.StaticToken(...), which uses the per-credential session pool, both runs fail the same way on this head.
  • Close never returns. It takes the same lock and has no context (client.go#L91-L95), so it blocks for good. A program that defers Close then cannot shut down, and deferring it is what the mcptoolset docs show (set.go#L51-L69). A stdio server that never answers is never stopped either. Close blocks and the server stays alive whether or not the server ignores SIGTERM. With a server that keeps default signal handling, a goroutine dump shows Close waiting while the listing sits in Client.Connect. Without the graph request, Close returns at once. It also returns at once on the merge base with main (be8b100) and on the previous head, with or without a graph request.

The comment at agentgraphtoolsets.go#L40-L47 says that a listing left running finishes in the background and that later agent runs reuse its session. That does not hold when the server never answers. Whatever the fix, building the graph must not leave a toolset in a state that breaks or blocks a later run, a later graph or Close, including when its server never answers.

On the previous head, ending the listing at a deadline killed a stdio server's process. The toolset then stayed broken, because mcptoolset keeps reusing a transport whose process is gone (#2111). So a deadline that frees the lock within seconds brings back the failure from my earlier review for any server slower than that deadline.

Reproduction (save as server/adkrest/internal/services/stalled_server_repro_test.go and run go test -run TestAgentRunAfterGraphWithStalledServer ./server/adkrest/internal/services/. It fails on this head and passes on the previous head.)
package services

import (
	"context"
	"io"
	"iter"
	"net/http"
	"net/http/httptest"
	"sync/atomic"
	"testing"
	"time"

	"github.com/modelcontextprotocol/go-sdk/mcp"
	"google.golang.org/genai"

	"google.golang.org/adk/v2/agent"
	"google.golang.org/adk/v2/agent/llmagent"
	"google.golang.org/adk/v2/model"
	"google.golang.org/adk/v2/runner"
	"google.golang.org/adk/v2/session"
	"google.golang.org/adk/v2/tool"
	"google.golang.org/adk/v2/tool/mcptoolset"
)

type replyModel struct{}

func (replyModel) Name() string { return "reply" }
func (replyModel) GenerateContent(context.Context, *model.LLMRequest, bool) iter.Seq2[*model.LLMResponse, error] {
	return func(yield func(*model.LLMResponse, error) bool) {
		yield(&model.LLMResponse{Content: genai.NewContentFromText("ok", genai.RoleModel)}, nil)
	}
}

// The MCP server holds its first request until the test ends and serves every
// later request normally, like a server that stalls once and recovers.
func TestAgentRunAfterGraphWithStalledServer(t *testing.T) {
	srv := mcp.NewServer(&mcp.Implementation{Name: "stall", Version: "v0"}, nil)
	mcp.AddTool(srv, &mcp.Tool{Name: "lookup"}, func(context.Context, *mcp.CallToolRequest, struct{}) (*mcp.CallToolResult, struct{}, error) {
		return &mcp.CallToolResult{}, struct{}{}, nil
	})
	h := mcp.NewStreamableHTTPHandler(func(*http.Request) *mcp.Server { return srv }, nil)
	stop := make(chan struct{})
	var requests atomic.Int32
	hs := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if requests.Add(1) == 1 {
			select {
			case <-stop:
			case <-r.Context().Done():
			}
			return
		}
		h.ServeHTTP(w, r)
	}))
	t.Cleanup(func() { close(stop); hs.CloseClientConnections(); hs.Close() })

	ts, err := mcptoolset.New(mcptoolset.Config{Endpoint: hs.URL})
	if err != nil {
		t.Fatal(err)
	}
	root, err := llmagent.New(llmagent.Config{Name: "root", Model: replyModel{}, Toolsets: []tool.Toolset{ts}})
	if err != nil {
		t.Fatal(err)
	}

	// Remove this call and the second run succeeds.
	if _, err := GetAgentGraph(context.Background(), root, nil); err != nil {
		t.Fatal(err)
	}

	r, err := runner.New(runner.Config{AppName: "app", Agent: root, SessionService: session.InMemoryService(), AutoCreateSession: true})
	if err != nil {
		t.Fatal(err)
	}
	var runErr error
	for _, sid := range []string{"s1", "s2"} {
		ctx, cancel := context.WithTimeout(context.Background(), 6*time.Second)
		runErr = nil
		for _, err := range r.Run(ctx, "user", sid, genai.NewContentFromText("hi", genai.RoleUser), agent.RunConfig{}) {
			if err != nil {
				runErr = err
				break
			}
		}
		cancel()
	}
	if runErr != nil {
		t.Errorf("second run, after the server recovered: %v (server saw %d requests)", runErr, requests.Load())
	}

	closed := make(chan struct{})
	go func() { _ = ts.(io.Closer).Close(); close(closed) }()
	select {
	case <-closed:
	case <-time.After(10 * time.Second):
		t.Error("Close still blocked after 10s")
	}
}

if llmAgent, ok := root.(llmagentinternal.Agent); ok {
for _, toolset := range llmagentinternal.Reveal(llmAgent).Toolsets {
use := toolsetUse{agent: root, toolset: toolset}
if toolset == nil || !use.keyable() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

toolsetLister.start skips any toolset whose type cannot be a map key. A value type with a slice field is one example, like catalog in the toolsearch example (main.go#L107-L112 at the current head, 07f79b4). tools then calls listTools for such a toolset only when the walk reaches it. That listing is not recorded in running, so each later graph request starts its own.

Skipping such a toolset has two effects on the current head, 07f79b4. First, when an earlier toolset is still listing as the budget runs out, a toolset that cannot be a map key gets none of the budget. That toolset is then left out of each graph built while the earlier one is still listing, even when that toolset's listing is instant. Second, if a toolset that cannot be a map key hangs in Tools, each graph request leaves one more blocked call behind, whereas a pointer toolset leaves one in total.

The toolsetWaitBudget comment says every toolset starts listing at once and gets the whole budget, which does not hold for these toolsets. ADK's own toolset constructors all return pointers, so the skipped start need not block the merge.

Test showing both (save as server/adkrest/internal/services/unkeyable_toolset_repro_test.go and run go test -run TestToolsetThatCannotBeAMapKey ./server/adkrest/internal/services/. It fails on this head and passes when sliceToolset is passed as a pointer.)
package services

import (
	"context"
	"strings"
	"sync/atomic"
	"testing"

	"google.golang.org/adk/v2/agent"
	"google.golang.org/adk/v2/agent/llmagent"
	"google.golang.org/adk/v2/tool"
)

// blockedToolset can be a map key. Its Tools blocks until release is closed.
type blockedToolset struct{ release chan struct{} }

func (blockedToolset) Name() string { return "blocked" }
func (b blockedToolset) Tools(agent.ReadonlyContext) ([]tool.Tool, error) {
	<-b.release
	return nil, nil
}

// sliceToolset cannot be a map key. Its Tools blocks while release is open.
type sliceToolset struct {
	names   []string
	release chan struct{}
	calls   *atomic.Int32
}

func (sliceToolset) Name() string { return "slice" }
func (s sliceToolset) Tools(agent.ReadonlyContext) ([]tool.Tool, error) {
	if s.calls != nil {
		s.calls.Add(1)
	}
	if s.release != nil {
		<-s.release
	}
	return []tool.Tool{newTestTool("slice_tool")}, nil
}

func TestToolsetThatCannotBeAMapKey(t *testing.T) {
	release := make(chan struct{})
	t.Cleanup(func() { close(release) })

	// Listed instantly, but behind a toolset that is still listing.
	root, err := llmagent.New(llmagent.Config{Name: "root", Model: &dummyLLM{},
		Toolsets: []tool.Toolset{blockedToolset{release}, sliceToolset{names: []string{"a"}}}})
	if err != nil {
		t.Fatal(err)
	}
	if dot, _ := GetAgentGraph(context.Background(), root, nil); !strings.Contains(dot, "slice_tool") {
		t.Error("slice_tool is left out of the graph")
	}

	// Hangs: three graph requests. Each call to Tools runs on a goroutine of
	// its own and stays blocked.
	var calls atomic.Int32
	hung, err := llmagent.New(llmagent.Config{Name: "hung", Model: &dummyLLM{},
		Toolsets: []tool.Toolset{sliceToolset{names: []string{"a"}, release: release, calls: &calls}}})
	if err != nil {
		t.Fatal(err)
	}
	for range 3 {
		_, _ = GetAgentGraph(context.Background(), hung, nil)
	}
	if n := calls.Load(); n > 1 {
		t.Errorf("3 graph requests left %d blocked calls to Tools, want 1 shared", n)
	}
}

@karthik-0306
karthik-0306 force-pushed the fix-agentgraph-toolsets branch from 963c48e to 07f79b4 Compare November 5, 2026 16:48
@karthik-0306

Copy link
Copy Markdown
Contributor Author

@karolpiotrowicz thanks, you're right: on 07f79b4 a server that never answers keeps the toolset locked, so later runs fail and Close blocks. Before pushing a fix, I'd like your preference between two directions:

  1. Draw each toolset as a single node (<Toolset.Name()>) and never call Toolset.Tools from the graph. The graph then can't affect the toolset, later runs or Close. I have this working locally: your stalled-server and stdio tests pass. The downside is that the graph shows the toolset name instead of its tools.
  2. Go back to listing tools with a deadline (as in c0ab228), and rely on fix(mcptoolset): let a CommandTransport reconnect #2115 to let a killed stdio server restart. This keeps the tools in the graph, but depends on fix(mcptoolset): let a CommandTransport reconnect #2115 landing first.

Which would you prefer? Happy to go either way.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agent graph omits tools from Toolsets (MCP, skills) and draws llmagent sub-agents without edges

4 participants