Repository navigation
fix: serialize AgentInstance lifecycle execution - #2857
Conversation
Signed-off-by: Eitan Yarmush <[email protected]>
Signed-off-by: Eitan Yarmush <[email protected]>
Signed-off-by: Eitan Yarmush <[email protected]>
063e895 to
6999689
Compare
| // pending or uncertain operations must never expire. | ||
| type InstanceOperation struct { | ||
| ID uuid.UUID | ||
| SourceCheckpointID *uuid.UUID // Pinned fork intent; nil for ordinary creation. |
There was a problem hiding this comment.
Do you think we should have a Kind field here so we don't need to read Instance.Kind every time, so we won't mix the instance input with operation identity?
There was a problem hiding this comment.
Added Kind from the existing persisted column and switched admission, completion, and dispatch to use it. Instance remains the admitted input snapshot.
| // conflicting operations indefinitely until its executor records completion. | ||
| // Callers authorize access. Missing instances return ErrNotFound, except that a | ||
| // completed Delete can return its retained result to an already authorized caller. | ||
| func (c *Client) BeginAgentInstanceOperation(ctx context.Context, instanceID string, kind apiv1alpha1.AgentInstanceOperation) (*InstanceOperation, error) { |
There was a problem hiding this comment.
Could some of the lifecycle comparisons be extracted into named helpers, which would make it easier to read and follow? Perhaps adding some inline comments would suffice.
There was a problem hiding this comment.
Named the compound conditions and documented completed retries, Delete supersession, and preparation failure versus claimed completion inline. Straightforward comparisons stay local; no additional helpers.
Signed-off-by: Eitan Yarmush <[email protected]>
Signed-off-by: Eitan Yarmush <[email protected]>
Kagent 1.0 recorded core version 1 with an earlier 000001 baseline, and 000001 has since changed in place (upstream kagent-dev#2847, kagent-dev#2857, kagent-dev#2868). Goose sees version 1 applied on a database created by 1.0 and runs nothing, so the controller finds no runtime_revision.credentials, no agent instance operation_id and executor_id, no generated pinned_checkpoint_id and no DELETED state, and fails to store a runtime revision with column "credentials" of relation "runtime_revision" does not exist. 000002 tells the two baselines apart by the four columns the current one added: none of them present is the 1.0 baseline, which it upgrades with the current baseline's columns, constraints (under the names a fresh install gives them) and partial index; all of them present is a database the current 000001 created, which it leaves unchanged; anything else is neither baseline and fails the migration, in its transaction. agent_instance_checkpoint.source_name stays: the 1.0 controller reads and writes it in every checkpoint query and the current controller never names it, so a migrated database still runs 1.0 and nothing is lost. The Down section has nothing to undo for the same reason. TestCoreUpgradesThe1_0Baseline replays the 1.0 baseline (a copy of it in testdata/core-1.0) with rows, runs the builtin migrations over it and compares the schema with a fresh install's, which 000002 must leave as 000001 made it; TestCoreRefusesAMixedBaseline proves the refusal. TestBuiltinMigrationsRoundTrip expects every migration a source ships.
Kagent 1.0 recorded core version 1 with an earlier 000001 baseline, and 000001 has since changed in place (upstream kagent-dev#2847, kagent-dev#2857, kagent-dev#2868). Goose sees version 1 applied on a database created by 1.0 and runs nothing, so the controller finds no runtime_revision.credentials, no agent instance operation_id and executor_id, no generated pinned_checkpoint_id and no DELETED state, and fails to store a runtime revision with column "credentials" of relation "runtime_revision" does not exist. 000002 tells the two baselines apart by the four columns the current one added: none of them present is the 1.0 baseline, which it upgrades with the current baseline's columns, constraints (under the names a fresh install gives them) and partial index; all of them present is a database the current 000001 created, which it leaves unchanged; anything else is neither baseline and fails the migration, in its transaction. agent_instance_checkpoint.source_name stays: the 1.0 controller reads and writes it in every checkpoint query and the current controller never names it, so a migrated database still runs 1.0 and nothing is lost. The Down section has nothing to undo for the same reason. TestCoreUpgradesThe1_0Baseline replays the 1.0 baseline (a copy of it in testdata/core-1.0) with rows, runs the builtin migrations over it and compares the schema with a fresh install's, which 000002 must leave as 000001 made it; TestCoreRefusesAMixedBaseline proves the refusal. TestBuiltinMigrationsRoundTrip expects every migration a source ships.
Concurrent lifecycle retries can issue duplicate Actor mutations, and delayed callers can act after a newer operation. Keep the current lifecycle generation and non-expiring executor claim on the AgentInstance row. Claim and completion check that identity; superseded observers return an explicit conflict without touching the runtime. A successful joined caller observes current instance state while its generation remains current. Remove the separate operation-history table, serialized input/result copies, and historical lifecycle result retention. Create/Fork retries preserve instance identity and return current state without provisioning again. Deletion leaves a DELETED tombstone that reserves the owner/request ID, revokes shares, and releases runtime revision and checkpoint pins. Fork source identity survives checkpoint deletion; a generated foreign-key column pins that checkpoint only while the instance is live. Normal instance/task/share access excludes tombstones, and reused creation request IDs cannot recreate deleted compute. Read-only preparation may retry before execution is claimed. Ambiguous runtime failures and failed database completion retain the claim and resource pins, blocking conflicting lifecycle operations including Delete. Preserve pinned fork creation and per-Actor egress policy setup before publishing READY. Part 2 follows kagent-dev#2792. Shared A2A/checkpoint execution and multi-replica observation remain later work; live Substrate settlement has not been verified. This requires a clean database and no overlapping older lifecycle binaries. Validation: - Core internal/package suites, including SQL preparation and migration Up/Down. - PostgreSQL-backed lifecycle and store race tests covering exclusive claims, stale/ABA completions, delayed observers, ambiguous mutations, failed database completion, and egress failures. - Tombstone request reuse, task/share access, concurrent share creation/deletion, and checkpoint/revision pin cleanup. - Authenticated lifecycle service integration with real PostgreSQL and controlled Actor responses. - Go lint and diff checks. The unreleased schema is consolidated in migration 1. The existing CI migration-immutability check rejects that edit; no forward migration or policy bypass is included. Design and implementation tracker are updated locally and remain uncommitted. --------- Signed-off-by: Eitan Yarmush <[email protected]>
Concurrent lifecycle retries can issue duplicate Actor mutations, and delayed callers can act after a newer operation. Keep the current lifecycle generation and non-expiring executor claim on the AgentInstance row. Claim and completion check that identity; superseded observers return an explicit conflict without touching the runtime.
A successful joined caller observes current instance state while its generation remains current. Remove the separate operation-history table, serialized input/result copies, and historical lifecycle result retention. Create/Fork retries preserve instance identity and return current state without provisioning again.
Deletion leaves a DELETED tombstone that reserves the owner/request ID, revokes shares, and releases runtime revision and checkpoint pins. Fork source identity survives checkpoint deletion; a generated foreign-key column pins that checkpoint only while the instance is live. Normal instance/task/share access excludes tombstones, and reused creation request IDs cannot recreate deleted compute.
Read-only preparation may retry before execution is claimed. Ambiguous runtime failures and failed database completion retain the claim and resource pins, blocking conflicting lifecycle operations including Delete. Preserve pinned fork creation and per-Actor egress policy setup before publishing READY.
Part 2 follows #2792. Shared A2A/checkpoint execution and multi-replica observation remain later work; live Substrate settlement has not been verified. This requires a clean database and no overlapping older lifecycle binaries.
Validation:
The unreleased schema is consolidated in migration 1. The existing CI migration-immutability check rejects that edit; no forward migration or policy bypass is included.
Design and implementation tracker are updated locally and remain uncommitted.