Skip to content

fix: honor checkpoint object in thread state updates - #7115

Merged
WillemJiang merged 5 commits into
bytedance:mainfrom
gmll-star:fix/7077-checkpoint-object-state
Oct 23, 2026
Merged

WillemJiang merged 5 commits into
bytedance:mainfrom
gmll-star:fix/7077-checkpoint-object-state

Conversation

@gmll-star

@gmll-star gmll-star commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Prefer the structured checkpoint object when updating thread state.
  • Validate the checkpoint thread ID and preserve its namespace.
  • Add regression coverage for checkpoint-object forks and cross-thread rejection.

Validation

  • tests/test_gateway_services.py tests/test_threads_router.py: 456 passed.
  • tests/blocking_io: 255 passed, 8 skipped.
  • Ruff check and format check passed for app/gateway/services.py.
  • On Windows, the unchanged skillscan orchestrator was loaded directly from its tracked Git object during test collection because local file access had caused it to disappear; the PR code itself was tested from this checkout.
  • The previous 422-case services claim was incorrect and has been replaced with the verified results above.

Fixes #7077

Non-root namespace review follow-up

  • Bind the checkpointer directly for state-only mutation graph operations in non-empty namespaces, including the readback after an update.
  • Reproduced the reported HTTP 500 with real persisted subgraph checkpoints before the fix. Added full- and delta-mode endpoint regressions verifying the selected parent, persisted updated title, and unchanged root checkpoint.
  • tests/test_checkpoint_state.py tests/test_gateway_services.py tests/test_threads_router.py: 470 passed.
  • tests/blocking_io/test_agent_factory_construction.py tests/blocking_io/test_tool_assembly_offloop.py: 9 passed.
  • Ruff check and format check passed on the three changed files.
  • For this follow-up run, a test-only skillscan import shim was used because of the same Windows file-access issue; skill scanning behavior was not validated in these tests, and the shim is not part of this PR.

Namespace-only selector follow-up

  • Reject nonexistent non-root namespaces with HTTP 404 when no checkpoint ID is supplied, while allowing updates of an existing namespace's latest checkpoint.
  • Document the private LangGraph 1.2.9 constant dependency. Added a regression that reproduced the phantom namespace write before the fix, and covered existing namespace selectors with and without IDs in full and delta modes.
  • tests/test_checkpoint_state.py tests/test_gateway_services.py tests/test_threads_router.py: 473 passed; Ruff check and format check passed for the three modified files.
  • These tests used the same test-only skillscan import shim described above; the shim was not committed and skill scanning was not validated.

Prefer the checkpoint object over the deprecated checkpoint_id, validate
its thread_id, and preserve checkpoint namespace when updating state.

Add regression coverage for checkpoint-object forks and thread mismatch.
@CLAassistant

CLAassistant commented Oct 19, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions Bot added area:backend Gateway / runtime / core backend under backend/ needs-validation Touches front/back contract surface; needs real-path validation risk:high High risk: backend API, agents, sandbox, auth, deps, CI size/M PR changes 100-300 lines labels Oct 19, 2026
@WillemJiang

Copy link
Copy Markdown
Collaborator

@gmll-star, thanks for your contribution. Please click the CLA button to sign the CLA first, otherwise we cannot merge your code.

@willem-bd willem-bd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head 155b439 (verified against merge-base 92e9afe).

The checkpoint-object fix itself checks out — I verified each claim independently against this head:

  • The sync builders (build_checkpoint_state_accessor, build_checkpoint_state_mutation_accessor) accept checkpoint_ns: str = "" as keyword-only params and carry it into the checkpointer config.
  • checkpoint_ns is forwarded only when non-empty at every hop (router → mutation boundary → read/mutation builders → abuild_checkpoint_state_accessor), so existing test doubles that only accept checkpoint_id keep working — 447 tests across test_gateway_services.py + test_threads_router.py pass apart from the issue below.
  • The ns-aware aget_tuple existence check runs before any write, so an unknown (or trimmed, per #7041) selector still 404s explicitly instead of writing to head; thread-id mismatch → 400; the deprecated string form and object-wins precedence behave as #7077 specifies.
  • Both new tests pass at head and fail at base (I transplanted the head test file onto 92e9afe: both fail there — the fork silently parents to head).

So relative to #7100, this is the complete, working fix for #7077 — with one blocker that needs to be dropped before merge.

else:
existing_runtime_context["client"] = existing_client_context

if not context:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR silently reverts the Gateway-side client-context handling from #7051, which has nothing to do with the checkpoint fix and is not mentioned in the description.

At this head, services.py deletes _sanitize_request_client_context, the configurable["client"] strip, the config['context']['client'] sanitization, and the body.context["client"] forwarding (the sanitize_client_context import is gone too — its only remaining callers are inside deerflow.runtime.client_context itself). Consequences at head:

  • body.config.configurable.client is no longer stripped, so arbitrary caller-supplied client data persists verbatim into configurable — which is written to the checkpoint store and echoed by the run API. This is exactly what the removed code guarded against.
  • body.config.context.client reaches runtime.context unsanitized (unbounded/nested caller data; the prompt block is still safe because render_client_context_for_prompt re-sanitizes at render time, but the raw dict persists).
  • Top-level body.context.client is now silently dropped instead of forwarded sanitized, so the feat(context): support client context capabilities #7051 feature stops working for SDK clients (dynamic_context_middleware reads runtime.context["client"] and now finds nothing).

Hard evidence this is a regression, not a refactor: 9 tests in backend/tests/test_gateway_services.py — a file this PR does not touch — pass at merge-base 92e9afe and fail at this head (..._propagates_sanitized_client_context_to_runtime_only, ..._sanitizes_config_context_client_without_body_context, ..._drops_configurable_client, ..._logs_dropped_client_fields_without_values, ..._does_not_log_a_clean_client, and test_start_run_forwards_sanitized_client_context_to_runtime_context_only x4 params). I ran both sides to confirm.

The description's validation note ("Related gateway services tests passed (422 cases)") does not hold at 155b439. Please drop every services.py hunk unrelated to checkpoint_ns — including the unrelated run_after_metadata comment reword further down — so the PR contains only the checkpoint fix, which as noted above is correct and complete.

@gmll-star
gmll-star force-pushed the fix/7077-checkpoint-object-state branch from 440a528 to 7f3563c Compare October 20, 2026 00:57
"configurable": {
"thread_id": thread_id,
"checkpoint_ns": "",
"checkpoint_ns": checkpoint_ns,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Bind the mutation graph before using a nonempty checkpoint namespace

A valid checkpoint with a nonempty namespace passes the new existence check, but the subsequent accessor.aupdate() fails. LangGraph interprets checkpoint_ns as a subgraph path, while build_state_mutation_graph() contains only the mutation node and CheckpointStateAccessor.bind() does not provide the configurable checkpointer override needed to operate directly within that namespace.

I reproduced this with the locked LangGraph 1.2.9: persist a checkpoint by invoking a parent graph containing a child subgraph, then POST /api/threads/{thread_id}/state with that checkpoint's actual {thread_id, checkpoint_ns, checkpoint_id} selector and a title update. The lookup succeeds, but the request returns HTTP 500 (Failed to update thread state), with ValueError: Subgraph child not found from abulk_update_state. The same endpoint's root-namespace control succeeds.

Please make the mutation accessor support writes and readback in the selected namespace, and add a regression using a real persisted nonroot checkpoint that verifies the updated values and parent checkpoint ID. Both new tests currently use checkpoint_ns: "", so they miss this path.

@willem-bd willem-bd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review at head 72e51b8 (one new commit, "fix: support state updates in checkpoint namespaces"), covering only what changed since 7f3563c. Both prior asks are resolved, verified at this head:

  • The undeclared #7051 revert is gone from the diff — services.py keeps _sanitize_request_client_context and all client-context handling, and all 9 previously failing tests in test_gateway_services.py pass again (I ran them at this head).
  • The namespace-write 500 from the earlier review is fixed: direct_namespace=True binds the checkpointer via CONFIG_KEY_CHECKPOINTER for nonempty namespaces, and the new test_update_thread_state_in_persisted_subgraph_namespace regression (real persisted subgraph checkpoint, both full and delta modes, parent-checkpoint-id + root-untouched assertions) passes at this head and matches what was asked.

Two small new items below: one behavior gap in namespace-only selectors, one nit on the private LangGraph import.

checkpoint_id = str(raw_checkpoint_id) if raw_checkpoint_id else raw_checkpoint_id
raw_checkpoint_ns = body.checkpoint.get("checkpoint_ns")
if raw_checkpoint_ns is not None:
checkpoint_ns = str(raw_checkpoint_ns)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] A checkpoint object with a nonempty checkpoint_ns but no checkpoint_id skips the if checkpoint_id is not None: existence check below, and with direct_namespace=True the write now succeeds into an arbitrary namespace. I probed this head: POST /api/threads/{id}/state with checkpoint: {"thread_id": ..., "checkpoint_ns": "child:ghost"} (no checkpoints exist in that namespace) returns 200 and materializes a brand-new checkpoint in child:ghost, confirmed via checkpointer.aget_tuple — root state stays intact, but the endpoint's 404 discipline for unknown selectors (the branch just below) does not apply to namespace-only selectors, and before this PR the state API could never create a nonroot namespace at all. Consider running the same ns-aware aget_tuple existence check whenever checkpoint_ns is nonempty, even without checkpoint_id, returning 404 when the namespace has no checkpoints.

# A state-only mutation graph has no subgraphs to delegate to. Bind
# the saver explicitly so LangGraph reads/writes this namespace on
# the graph itself, including when reading back an updated config.
from langgraph._internal._constants import CONFIG_KEY_CHECKPOINTER

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nit] This reaches into LangGraph's private internals (langgraph._internal._constants); the repo so far imports private LangGraph modules only from tests. It works at the locked langgraph >=1.2.9,<1.3 — I verified CONFIG_KEY_CHECKPOINTER resolves at 1.2.9 — and the deferred import confines any breakage to the nonempty-namespace mutation path, but a minor upgrade that moves the constant would turn subgraph-namespace state updates into runtime 500s. Worth a brief comment naming that version coupling so the next langgraph bump knows to grep for it.

@willem-bd willem-bd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 3b9d2cc. The earlier checkpoint selection and namespace failures are addressed. I found one remaining P2 side effect in nonroot title updates, detailed inline.

Validation: 669 related checkpoint, Gateway service/router, checkpoint-mode, compaction, rollback and factory-offload tests passed. All seven new regressions fail against the original router as expected. Additional HTTP probes through the real accessor builders reproduce the title-isolation failure in both full and delta modes; an in-memory root-only title-sync guard makes those probes pass. Ruff lint/format, guidance checks and current CI passed. No production source files were changed during this review.

checkpoint_id = str(raw_checkpoint_id) if raw_checkpoint_id else raw_checkpoint_id
raw_checkpoint_ns = body.checkpoint.get("checkpoint_ns")
if raw_checkpoint_ns is not None:
checkpoint_ns = str(raw_checkpoint_ns)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep child titles out of root conversation metadata

Now that a nonempty checkpoint_ns targets a child checkpoint, the later title synchronization at lines 1718–1725 also needs to respect that namespace. It currently calls thread_store.update_display_name(thread_id, new_title, ...) for every title update, so changing a child's title renames the root conversation in thread search results even though its root checkpoint was correctly left unchanged.

I reproduced this through the real thread/assistant/accessor builders with a persisted child subgraph in both full and delta modes: first name the root Root conversation, then POST a title update of forked with the child's {thread_id, checkpoint_ns, checkpoint_id} selector. The response is 200 and the child checkpoint has the correct selected parent; GET /api/threads/{id}/state still returns the root title Root conversation, but POST /api/threads/search now lists that conversation as forked. The root checkpoint itself is byte-for-byte unchanged.

Please restrict the update_display_name synchronization to the root namespace (not checkpoint_ns) and extend the namespace regression to assert that thread metadata/search titles remain unchanged after a child title update. A temporary in-memory version of that guard makes the failing full/delta probes pass without affecting the root controls.

Only synchronize state titles into conversation metadata for the root checkpoint namespace. Cover persisted child updates with and without checkpoint IDs in full and delta modes, and retain root title synchronization and branch sequence cleanup.
@github-actions github-actions Bot added area:docs Documentation and Markdown only size/L PR changes 300-700 lines and removed size/M PR changes 100-300 lines labels Oct 22, 2026
@WillemJiang WillemJiang added this to the 2.2.0 milestone Oct 23, 2026
sun04222 added a commit to sun04222/deer-flow that referenced this pull request Oct 23, 2026
…rs/AGENTS.md skipped - file absent on fork base)
@WillemJiang
WillemJiang merged commit d318a89 into bytedance:main Oct 23, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:backend Gateway / runtime / core backend under backend/ area:docs Documentation and Markdown only needs-validation Touches front/back contract surface; needs real-path validation risk:high High risk: backend API, agents, sandbox, auth, deps, CI size/L PR changes 300-700 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] POST /threads/{id}/state silently ignores the checkpoint object and mutates the thread head

4 participants