Skip to content

Harden long-running servers and roll back secrets on failed saves - #763

Merged
sak0a merged 3 commits into
mainfrom
fix/long-running-server-hygiene
Oct 23, 2026
Merged

sak0a merged 3 commits into
mainfrom
fix/long-running-server-hygiene

Conversation

@sak0a

@sak0a sak0a commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

What Changed

  • SQLite: a statement prepare that fails is no longer cached. The Node SQLite client keeps a prepared statement for 10 minutes, but now caches a failure for zero time, so the next call prepares it again. GitManager's status cache already uses this pattern.
  • Git: the polled status fetch now runs with -c maintenance.auto=false and --no-auto-gc, so it never starts git maintenance run --auto. The config setting also covers the submodule fetches it starts, which do not get the flag. Fetches and pulls the user starts are unchanged.
  • Settings secrets: provider environment secrets and CLIProxyAPI hub secrets now share one set of secret store changes per settings update:
    • A changed key is stored before settings.json is written, and its previous value is recorded first.
    • If anything fails before the write completes (provider or hub persistence, normalization, or the write itself), the old values are put back and new keys are deleted.
    • Removed keys are deleted only after a successful write, and only when the written file no longer refers to them.
  • Behaviour change: if deleting an unused provider key fails after a successful write, the update no longer fails. The failure is logged and the unused key stays, as hub keys already work. The two warning messages now say "settings secret" and log the key name, never the value.

Why

A server that runs for weeks hit three problems:

The WAL size cap and bounded event log records already match the reference. Replay-marker pruning does not apply, because Ryco's server stores no DPoP replay markers on disk.

Testing

  • bun run check: formatting ok, 0 lint errors, typecheck:fast 12/12 ok
  • RYCO_TEST_MAX_WORKERS=4 bun run --cwd apps/server test -- src/persistence/NodeSqliteClient.test.ts src/vcs/GitVcsDriverCore.test.ts src/serverSettings.test.ts: 3 files, 117 passed
  • RYCO_TEST_MAX_WORKERS=4 bun run check:changed: typecheck ok, related ryco-cli tests ok, read-from-disk ryco-cli tests ok
  • Each new test fails on the old code:
    • A query whose table is missing is prepared again once the table exists, on both the row path and the .values path.
    • The exact arguments of the status fetch are checked.
    • A real-git repository where gc --auto is due keeps its pack count after refreshStatusUpstream.
    • Two failed writes to an unwritable settings.json leave the secret store, the settings.json bytes and the loaded settings unchanged. The same edit succeeds once the directory is writable again.
    • A provider key change in the same update as a refused hub entry is rolled back.

Follow-ups

  • Other fetches (branch picker, worktree prefetch) still allow gc --auto on purpose. They run when the user acts, not on a timer.
  • The SQLite client used under Bun (@effect/sql-sqlite-bun) is upstream code. It does not cache failed statements, so it was not changed.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@sak0a sak0a changed the title Keep long-running servers healthy: SQLite retries, git fetches, settings secrets Harden long-running servers and roll back secrets on failed saves Oct 22, 2026
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:L labels Oct 22, 2026
sak0a and others added 3 commits October 23, 2026 17:24
The Node SQLite client caches prepared statements for 10 minutes, but the
cache kept a failed prepare for the same 10 minutes. A prepare that failed
for a passing reason (a schema lock, or "no such table" while the separate
CLI process is still migrating the shared database) then failed every
later call of that statement until the entry expired.

The cache's time to live now depends on the lookup result: a prepared
statement keeps the configured TTL, a failure is not cached, so the next
call prepares the statement again. GitManager's status cache already uses
the same pattern. Port of pingdotgg/t3code#10584 (d06f0ff104).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The source control status poll fetches each repository's upstream
remote in the background, and every fetch ends with `git maintenance run
--auto`. When that gc fails, for example on a repository with missing
objects, git retries it on the next fetch and leaves a full-size
tmp_pack_* behind each time, so a node that polls for weeks can fill the
disk.

The status fetch now passes --no-auto-gc (git 2.23+, a synonym of
--no-auto-maintenance) and -c maintenance.auto=false. The flag covers the
fetch itself; the config reaches the submodule fetches it starts, which
do not inherit the flag. User-initiated fetches and pulls keep git's
normal maintenance. Port of pingdotgg/t3code#13812 (95030dc674), plus
the submodule case.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Sensitive provider-instance environment variables live in the secret
store, and an update set and removed their keys before settings.json was
written. Only CLIProxyAPI hub keys were put back when that write (or a
later step of the update) failed, so a failed save could lose a removed
API key for good, keep a replaced one that settings.json no longer
matched, or leave a new key behind. A hub refused later in the same
update left the provider keys changed as well.

The hub's staging is now shared by both: one set of secret changes per
update stores keys at once and remembers what each replaced, `restore`
puts those values back (or removes new keys) if any step before the
completed write fails, and keys the written file no longer refers to are
only removed after the write, where a failure is logged and leaves a
harmless unused key. A key is recorded before it is written, since the
store can fail after replacing it (chmod after rename). Port of
pingdotgg/t3code#12487 (6d68677ff9).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@sak0a
sak0a force-pushed the fix/long-running-server-hygiene branch from 27d825c to 16a380b Compare October 23, 2026 15:24
@sak0a
sak0a merged commit 73da9ab into main Oct 23, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant