Skip to content

fix: serialize AgentInstance lifecycle execution - #2857

Merged
EItanya merged 5 commits into
mainfrom
fix/instance-lifecycle-identity
Sep 20, 2026
Merged

EItanya merged 5 commits into
mainfrom
fix/instance-lifecycle-identity

Conversation

@EItanya

@EItanya EItanya commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Concurrent lifecycle retries can issue duplicate Actor mutations, and delayed callers can act after a newer operation. Keep the current lifecycle generation and non-expiring executor claim on the AgentInstance row. Claim and completion check that identity; superseded observers return an explicit conflict without touching the runtime.

A successful joined caller observes current instance state while its generation remains current. Remove the separate operation-history table, serialized input/result copies, and historical lifecycle result retention. Create/Fork retries preserve instance identity and return current state without provisioning again.

Deletion leaves a DELETED tombstone that reserves the owner/request ID, revokes shares, and releases runtime revision and checkpoint pins. Fork source identity survives checkpoint deletion; a generated foreign-key column pins that checkpoint only while the instance is live. Normal instance/task/share access excludes tombstones, and reused creation request IDs cannot recreate deleted compute.

Read-only preparation may retry before execution is claimed. Ambiguous runtime failures and failed database completion retain the claim and resource pins, blocking conflicting lifecycle operations including Delete. Preserve pinned fork creation and per-Actor egress policy setup before publishing READY.

Part 2 follows #2792. Shared A2A/checkpoint execution and multi-replica observation remain later work; live Substrate settlement has not been verified. This requires a clean database and no overlapping older lifecycle binaries.

Validation:

  • Core internal/package suites, including SQL preparation and migration Up/Down.
  • PostgreSQL-backed lifecycle and store race tests covering exclusive claims, stale/ABA completions, delayed observers, ambiguous mutations, failed database completion, and egress failures.
  • Tombstone request reuse, task/share access, concurrent share creation/deletion, and checkpoint/revision pin cleanup.
  • Authenticated lifecycle service integration with real PostgreSQL and controlled Actor responses.
  • Go lint and diff checks.

The unreleased schema is consolidated in migration 1. The existing CI migration-immutability check rejects that edit; no forward migration or policy bypass is included.

Design and implementation tracker are updated locally and remain uncommitted.

@EItanya
EItanya requested a review from a team as a code owner September 17, 2026 14:44
@github-actions github-actions Bot added the bug Something isn't working label Sep 17, 2026
@EItanya
EItanya force-pushed the fix/instance-lifecycle-identity branch from 063e895 to 6999689 Compare September 18, 2026 14:25
// pending or uncertain operations must never expire.
type InstanceOperation struct {
ID uuid.UUID
SourceCheckpointID *uuid.UUID // Pinned fork intent; nil for ordinary creation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think we should have a Kind field here so we don't need to read Instance.Kind every time, so we won't mix the instance input with operation identity?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added Kind from the existing persisted column and switched admission, completion, and dispatch to use it. Instance remains the admitted input snapshot.

// conflicting operations indefinitely until its executor records completion.
// Callers authorize access. Missing instances return ErrNotFound, except that a
// completed Delete can return its retained result to an already authorized caller.
func (c *Client) BeginAgentInstanceOperation(ctx context.Context, instanceID string, kind apiv1alpha1.AgentInstanceOperation) (*InstanceOperation, error) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could some of the lifecycle comparisons be extracted into named helpers, which would make it easier to read and follow? Perhaps adding some inline comments would suffice.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Named the compound conditions and documented completed retries, Delete supersession, and preparation failure versus claimed completion inline. Straightforward comparisons stay local; no additional helpers.

@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Sep 18, 2026
@EItanya
EItanya merged commit ed28928 into main Sep 20, 2026
29 of 30 checks passed
@EItanya
EItanya deleted the fix/instance-lifecycle-identity branch September 20, 2026 13:09
teemow added a commit to giantswarm/kagent-upstream that referenced this pull request Sep 25, 2026
Kagent 1.0 recorded core version 1 with an earlier 000001 baseline, and
000001 has since changed in place (upstream kagent-dev#2847, kagent-dev#2857, kagent-dev#2868). Goose
sees version 1 applied on a database created by 1.0 and runs nothing, so
the controller finds no runtime_revision.credentials, no agent instance
operation_id and executor_id, no generated pinned_checkpoint_id and no
DELETED state, and fails to store a runtime revision with
column "credentials" of relation "runtime_revision" does not exist.

000002 tells the two baselines apart by the four columns the current one
added: none of them present is the 1.0 baseline, which it upgrades with
the current baseline's columns, constraints (under the names a fresh
install gives them) and partial index; all of them present is a database
the current 000001 created, which it leaves unchanged; anything else is
neither baseline and fails the migration, in its transaction.

agent_instance_checkpoint.source_name stays: the 1.0 controller reads and
writes it in every checkpoint query and the current controller never
names it, so a migrated database still runs 1.0 and nothing is lost. The
Down section has nothing to undo for the same reason.

TestCoreUpgradesThe1_0Baseline replays the 1.0 baseline (a copy of it in
testdata/core-1.0) with rows, runs the builtin migrations over it and
compares the schema with a fresh install's, which 000002 must leave as
000001 made it; TestCoreRefusesAMixedBaseline proves the refusal.
TestBuiltinMigrationsRoundTrip expects every migration a source ships.
teemow added a commit to giantswarm/kagent-upstream that referenced this pull request Sep 25, 2026
Kagent 1.0 recorded core version 1 with an earlier 000001 baseline, and
000001 has since changed in place (upstream kagent-dev#2847, kagent-dev#2857, kagent-dev#2868). Goose
sees version 1 applied on a database created by 1.0 and runs nothing, so
the controller finds no runtime_revision.credentials, no agent instance
operation_id and executor_id, no generated pinned_checkpoint_id and no
DELETED state, and fails to store a runtime revision with
column "credentials" of relation "runtime_revision" does not exist.

000002 tells the two baselines apart by the four columns the current one
added: none of them present is the 1.0 baseline, which it upgrades with
the current baseline's columns, constraints (under the names a fresh
install gives them) and partial index; all of them present is a database
the current 000001 created, which it leaves unchanged; anything else is
neither baseline and fails the migration, in its transaction.

agent_instance_checkpoint.source_name stays: the 1.0 controller reads and
writes it in every checkpoint query and the current controller never
names it, so a migrated database still runs 1.0 and nothing is lost. The
Down section has nothing to undo for the same reason.

TestCoreUpgradesThe1_0Baseline replays the 1.0 baseline (a copy of it in
testdata/core-1.0) with rows, runs the builtin migrations over it and
compares the schema with a fresh install's, which 000002 must leave as
000001 made it; TestCoreRefusesAMixedBaseline proves the refusal.
TestBuiltinMigrationsRoundTrip expects every migration a source ships.
alexdarbyshire pushed a commit to alexdarbyshire/kagent that referenced this pull request Oct 5, 2026
Concurrent lifecycle retries can issue duplicate Actor mutations, and
delayed callers can act after a newer operation. Keep the current
lifecycle generation and non-expiring executor claim on the
AgentInstance row. Claim and completion check that identity; superseded
observers return an explicit conflict without touching the runtime.

A successful joined caller observes current instance state while its
generation remains current. Remove the separate operation-history table,
serialized input/result copies, and historical lifecycle result
retention. Create/Fork retries preserve instance identity and return
current state without provisioning again.

Deletion leaves a DELETED tombstone that reserves the owner/request ID,
revokes shares, and releases runtime revision and checkpoint pins. Fork
source identity survives checkpoint deletion; a generated foreign-key
column pins that checkpoint only while the instance is live. Normal
instance/task/share access excludes tombstones, and reused creation
request IDs cannot recreate deleted compute.

Read-only preparation may retry before execution is claimed. Ambiguous
runtime failures and failed database completion retain the claim and
resource pins, blocking conflicting lifecycle operations including
Delete. Preserve pinned fork creation and per-Actor egress policy setup
before publishing READY.

Part 2 follows kagent-dev#2792. Shared A2A/checkpoint execution and multi-replica
observation remain later work; live Substrate settlement has not been
verified. This requires a clean database and no overlapping older
lifecycle binaries.

Validation:
- Core internal/package suites, including SQL preparation and migration
Up/Down.
- PostgreSQL-backed lifecycle and store race tests covering exclusive
claims, stale/ABA completions, delayed observers, ambiguous mutations,
failed database completion, and egress failures.
- Tombstone request reuse, task/share access, concurrent share
creation/deletion, and checkpoint/revision pin cleanup.
- Authenticated lifecycle service integration with real PostgreSQL and
controlled Actor responses.
- Go lint and diff checks.

The unreleased schema is consolidated in migration 1. The existing CI
migration-immutability check rejects that edit; no forward migration or
policy bypass is included.

Design and implementation tracker are updated locally and remain
uncommitted.

---------

Signed-off-by: Eitan Yarmush <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants