Skip to content

fix(storage): keep redis upserts and their index writes atomic - #3848

Open
konlue wants to merge 3 commits into
agentscope-ai:mainfrom
konlue:fix/redis-upsert-index-atomicity
Open

konlue wants to merge 3 commits into
agentscope-ai:mainfrom
konlue:fix/redis-upsert-index-atomicity

Conversation

@konlue

@konlue konlue commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

AgentScope Version

2.0.12dev (main @ d8ebf89, rebased over #3806)

Description

RedisStorage persisted every record together with its index entries in separate awaited round trips. A connection failure or process interruption between the writes left the durable record outside every index: an interrupted upsert_schedule never fires (the scheduler discovers work only through list_all_schedules(), which reads the global index, and there is no reconciliation) and is invisible to the user; an interrupted upsert_channel is silently not restored at startup (list_all_channels()); interrupted upsert_mcp / upsert_skill calls desynchronize the name indexes used by get_*_by_name and the duplicate-name checks (#3840). #3816 fixed the same invariant for upsert_knowledge_document; this extends it to all remaining upserts.

Changes — every upsert's record and index writes now commit in one MULTI/EXEC pipeline (client.pipeline(transaction=True), the idiom already used by upsert_message), covering:

  • upsert_schedule (record + per-user index + global scheduler index),
  • upsert_channel (record + per-user index + all-channels index + bot-id binding),
  • upsert_mcp / upsert_skill (record + id set + name hash, including the rename HDEL),
  • upsert_sop / upsert_sop_run (record + per-user index + per-procedure index),
  • upsert_team, upsert_voice_profile, upsert_agent, upsert_credential, upsert_session (creation path; record + indexes),
  • upsert_knowledge_base.

The record value is written with the single-command SET ... EX form introduced by #3806 (pipe.set(key, value, ex=self.key_ttl)), so the pipelines speak the same idiom as the rest of the file — voice profiles now carry the configured TTL like other records, unchanged from #3806's semantics.

Pre-transaction reads (the name-conflict check, the previous-record read for renames and created_at preservation) stay outside the pipeline with unchanged semantics; a failure inside the async with block discards the queued writes, so no record survives outside its indexes. (upsert_knowledge_document itself is left to #3816.)

How to test

New TestUpsertIndexAtomicity in the existing tests/storage_redis_test.py, using a pipeline wrapper whose execute always raises (fault injection, mirroring #3805's reproduction):

  • test_interrupted_schedule_upsert_is_invisible / test_interrupted_channel_upsert_is_invisible / test_interrupted_skill_upsert_is_invisible / test_interrupted_team_upsert_is_invisible — fail on main (the record key survives while list_schedules / list_all_schedules / list_all_channels / get_*_by_name / list_teams all come back empty), pass with this fix (nothing applied);
  • test_successful_upserts_reach_every_index — schedule, channel (including the bot-id binding), skill and team all land in every index;
  • test_skill_rename_updates_the_name_index_atomically — renaming drops the previous name hash entry in the same commit.
python -m pytest tests/storage_redis_test.py
# 105 passed, 25 warnings (includes #3806's TestRecordKeyTTL).

Fixes #3840

Checklist

  • An issue has been created for this PR
  • I have read the CONTRIBUTING.md
  • Docstrings are in Google style
  • Related documentation has been updated (e.g. links, examples, etc.) in documentation repository
  • Code is ready for review

@github-actions github-actions Bot added the pr/awaiting-review Waiting for maintainers to review this PR label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Thanks for the pull request. One thing worth sorting out before it is reviewed:

If you think it belongs here anyway, say so and a maintainer will take a look.

Every RedisStorage upsert wrote the record and its index entries as
separate awaited round trips. A connection failure or process
interruption between the writes left the durable record outside every
index: an interrupted upsert_schedule never fires (the scheduler only
discovers work through the global index) and is invisible to the user,
an interrupted upsert_channel is not restored at startup, and the
mcp/skill name indexes desynchronize from their records. agentscope-ai#3840 showed
the failure with fault injection on fakeredis; agentscope-ai#3816 fixed the same
invariant for knowledge documents.

Wrap each upsert's record and index writes in a MULTI/EXEC pipeline
(client.pipeline(transaction=True)), covering credential, agent,
session creation, schedule, channel, mcp and skill upserts including
the mcp/skill rename paths. A failure inside the transaction discards
the queued writes, so no record survives outside its indexes.
Self-review caught four more upserts with the same separate-write shape
that the first pass missed: upsert_sop, upsert_sop_run (record + two
index entries), upsert_team and upsert_voice_profile. Convert them to
the same atomic pipeline and extend the fault-injection tests with an
interrupted/successful team pair.
…diom

Rebase onto the merge of agentscope-ai#3806, which collapsed _set_with_ttl into a
single SET ... EX command: replace the SET + conditional EXPIRE pairs
inside the upsert pipelines with the same single-command form (and
drop the redundant voice_profile gap — voice profiles now carry the
configured TTL like other records), so every upsert keeps writing its
record and index entries in one MULTI/EXEC transaction while speaking
the same idiom as the rest of the file.
@konlue
konlue force-pushed the fix/redis-upsert-index-atomicity branch from 7e42809 to ba623b8 Compare November 9, 2026 15:47

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr/awaiting-review Waiting for maintainers to review this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: RedisStorage upserts write the record and its index entries in separate round trips — an interrupted upsert orphans schedules and channels

1 participant