Skip to content

Commit 943798f

Browse files
authored
fix: recover stale runtime adapter claims (#448)
1 parent 35f791a commit 943798f

7 files changed

Lines changed: 292 additions & 16 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88

99
### Fixed
1010

11+
- Made new runtime-adapter ticket claims provisional until agent connection or lease registration, allowing authenticated recovery of expired inactive first claims while preserving all existing and confirmed adapter IDs.
1112
- Separated shared automation tokens from signed user-token keys, preserving shared-token-only automation while requiring distinct session signing material for GitHub login.
1213
- Required retained coordinator ownership records before orphan sweeps delete AWS or Azure machines or release EC2 Mac hosts, while keeping tag-only and legacy candidates visible in reports.
1314
- Verified the pinned GitHub CLI release artifacts before installing them in the default Cloudflare sandbox image and preserved true AMD64/ARM64 target selection during cross-platform builds.

docs/commands/adapter.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,14 @@ atomically rotated token is never pinned in the relay process. Coordinator
155155
authentication may be a static login token or the normal shell-free
156156
token-command configuration.
157157

158+
The first ticket for a previously unused adapter ID creates a ten-minute
159+
provisional owner/org claim. A successful agent connection or registered lease
160+
binding makes that claim durable. Another normally authenticated owner may
161+
recover an expired provisional claim only when it has no connected agent,
162+
pending relay request, unexpired ticket, live registered lease, or pending
163+
workspace deletion. Existing adapter claims created before this contract are
164+
treated as durable, so upgrades do not change current adapter ownership.
165+
158166
`--local-socket` is required. It must be an absolute, clean Unix-socket path in
159167
an existing directory owned by the current user and not writable by group or
160168
others. The socket must also be owned by the current user with mode `0600`.

docs/features/coordinator.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -141,9 +141,14 @@ WebVNC/share operation still requires the record to exist.
141141

142142
Runtime adapters connect outbound to `/agent`, so their local lifecycle service
143143
does not need an inbound public route. The coordinator issues a short-lived,
144-
single-use ticket after normal owner authentication. The first ticket safely
145-
claims the adapter ID for that owner and organization; lease registration
146-
rejects unclaimed IDs and claims owned by another identity. The relay permits
144+
single-use ticket after normal owner authentication. The first ticket for a new
145+
adapter ID creates a ten-minute provisional owner/org claim. Agent connection or
146+
successful registered lease binding confirms it as durable. An expired
147+
provisional claim can be recovered by another normally authenticated owner only
148+
when no connected agent, pending relay request, unexpired ticket, live registered
149+
lease, or pending workspace deletion still uses it. Existing and confirmed
150+
claims remain durable. Lease registration rejects unclaimed IDs and claims owned
151+
by another identity. The relay permits
147152
only the versioned workspace create, inspect, delete, and desktop-connection
148153
methods. Public proxy `DELETE` requests are rejected; destructive dispatch must
149154
go through a registered lease release so the coordinator can fence its immutable

docs/features/runtime-adapter-stack.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,10 @@ files, or provider credentials. Keep the bearer token between the fleet UI and
8686
### Outbound coordinator control
8787

8888
1. `adapter connect` authenticates to the coordinator using normal Crabbox
89-
configuration and obtains a short-lived relay ticket.
89+
configuration and obtains a short-lived relay ticket. A new adapter ID starts
90+
with a ten-minute provisional owner/org claim; agent connection or successful
91+
lease registration makes it durable. Only expired inactive provisional claims
92+
are recoverable, and existing adapter claims remain durable across upgrades.
9093
2. It opens an outbound WebSocket and accepts only the documented typed
9194
workspace operations.
9295
3. Each operation is forwarded through the verified, current-user-owned Unix

worker/src/fleet.ts

Lines changed: 101 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,7 @@ const codeViewerTicketTTLSeconds = 120;
143143
const codeViewerSessionTTLSeconds = 8 * 60 * 60;
144144
const egressTicketTTLSeconds = 120;
145145
const runtimeAdapterTicketTTLSeconds = 120;
146+
const runtimeAdapterProvisionalClaimTTLSeconds = 10 * 60;
146147
const runtimeAdapterDeleteRetryBaseMs = 5_000;
147148
const runtimeAdapterDeleteRetryMaxMs = 60_000;
148149
const runtimeAdapterDeleteDispatchGraceMs = 1_000;
@@ -257,6 +258,10 @@ interface RuntimeAdapterIdentityRecord {
257258
owner: string;
258259
org: string;
259260
createdAt: string;
261+
claimVersion?: 1;
262+
claimState?: "provisional" | "confirmed";
263+
claimExpiresAt?: string;
264+
confirmedAt?: string;
260265
}
261266

262267
interface RuntimeAdapterPendingRequest {
@@ -3675,6 +3680,7 @@ export class FleetCoordinator {
36753680
{ status: 409 },
36763681
);
36773682
}
3683+
let runtimeAdapterIdentity: RuntimeAdapterIdentityRecord | undefined;
36783684
if (effectiveRuntimeAdapterID && effectiveRuntimeAdapterWorkspaceID) {
36793685
const identity = await this.state.storage.get<RuntimeAdapterIdentityRecord>(
36803686
runtimeAdapterIdentityKey(effectiveRuntimeAdapterID),
@@ -3701,6 +3707,7 @@ export class FleetCoordinator {
37013707
{ status: 409 },
37023708
);
37033709
}
3710+
runtimeAdapterIdentity = identity;
37043711
}
37053712

37063713
const leases = await this.leaseRecords();
@@ -3817,6 +3824,9 @@ export class FleetCoordinator {
38173824
clearRuntimeAdapterDeleteMetadata(record);
38183825
}
38193826
await this.putLease(record);
3827+
if (runtimeAdapterIdentity) {
3828+
await this.confirmRuntimeAdapterIdentity(runtimeAdapterIdentity, nowISO);
3829+
}
38203830
await this.scheduleAlarm();
38213831
return json({ lease: record }, { status: existing ? 200 : 201 });
38223832
}
@@ -5802,28 +5812,49 @@ export class FleetCoordinator {
58025812
const owner = requestOwner(request);
58035813
const org = requestOrg(request, this.env);
58045814
const identityKey = runtimeAdapterIdentityKey(adapterID);
5805-
const identity = await this.state.storage.get<RuntimeAdapterIdentityRecord>(identityKey);
5815+
await this.cleanupExpiredRuntimeAdapterTickets();
5816+
const now = new Date();
5817+
let identity = await this.state.storage.get<RuntimeAdapterIdentityRecord>(identityKey);
58065818
if (
58075819
identity &&
58085820
(identity.adapterID !== adapterID || identity.owner !== owner || identity.org !== org)
58095821
) {
5810-
return json(
5811-
{
5812-
error: "adapter_id_conflict",
5813-
message: "runtime adapter id belongs to another owner or organization",
5814-
},
5815-
{ status: 409 },
5816-
);
5822+
if (!(await this.runtimeAdapterIdentityReclaimable(identity, adapterID, now.getTime()))) {
5823+
return json(
5824+
{
5825+
error: "adapter_id_conflict",
5826+
message: "runtime adapter id belongs to another owner or organization",
5827+
},
5828+
{ status: 409 },
5829+
);
5830+
}
5831+
identity = undefined;
58175832
}
5818-
await this.cleanupExpiredRuntimeAdapterTickets();
5819-
const now = new Date();
58205833
if (!identity) {
5821-
await this.state.storage.put<RuntimeAdapterIdentityRecord>(identityKey, {
5834+
identity = {
58225835
adapterID,
58235836
owner,
58245837
org,
58255838
createdAt: now.toISOString(),
5826-
});
5839+
claimVersion: 1,
5840+
claimState: "provisional",
5841+
claimExpiresAt: new Date(
5842+
now.getTime() + runtimeAdapterProvisionalClaimTTLSeconds * 1000,
5843+
).toISOString(),
5844+
};
5845+
await this.state.storage.put<RuntimeAdapterIdentityRecord>(identityKey, identity);
5846+
} else if (
5847+
identity.claimVersion === 1 &&
5848+
identity.claimState === "provisional" &&
5849+
!identity.confirmedAt
5850+
) {
5851+
identity = {
5852+
...identity,
5853+
claimExpiresAt: new Date(
5854+
now.getTime() + runtimeAdapterProvisionalClaimTTLSeconds * 1000,
5855+
).toISOString(),
5856+
};
5857+
await this.state.storage.put<RuntimeAdapterIdentityRecord>(identityKey, identity);
58275858
}
58285859
const ticket: RuntimeAdapterTicketRecord = {
58295860
ticket: newRuntimeAdapterTicket(),
@@ -5877,6 +5908,7 @@ export class FleetCoordinator {
58775908
{ status: 401 },
58785909
);
58795910
}
5911+
await this.confirmRuntimeAdapterIdentity(identity, new Date().toISOString());
58805912
const upgrade = this.state.createWebSocketUpgrade({
58815913
maxPayload: runtimeAdapterRelayFrameLimit,
58825914
});
@@ -7421,6 +7453,63 @@ export class FleetCoordinator {
74217453
);
74227454
}
74237455

7456+
private async runtimeAdapterIdentityReclaimable(
7457+
identity: RuntimeAdapterIdentityRecord,
7458+
adapterID: string,
7459+
now: number,
7460+
): Promise<boolean> {
7461+
if (
7462+
identity.adapterID !== adapterID ||
7463+
identity.claimVersion !== 1 ||
7464+
identity.claimState !== "provisional" ||
7465+
identity.confirmedAt !== undefined ||
7466+
!Number.isFinite(Date.parse(identity.claimExpiresAt ?? "")) ||
7467+
Date.parse(identity.claimExpiresAt ?? "") > now ||
7468+
this.runtimeAdapterAgents.has(adapterID) ||
7469+
[...this.runtimeAdapterPending.values()].some((pending) => pending.adapterID === adapterID)
7470+
) {
7471+
return false;
7472+
}
7473+
const [tickets, leases] = await Promise.all([
7474+
this.state.storage.list<RuntimeAdapterTicketRecord>({
7475+
prefix: runtimeAdapterTicketPrefix(),
7476+
}),
7477+
this.leaseRecords(),
7478+
]);
7479+
if (
7480+
[...tickets.values()].some(
7481+
(ticket) => ticket.adapterID === adapterID && Date.parse(ticket.expiresAt) > now,
7482+
)
7483+
) {
7484+
return false;
7485+
}
7486+
return !leases.some(
7487+
(lease) =>
7488+
lease.runtimeAdapterID === adapterID &&
7489+
(leaseIsLive(lease) || Boolean(lease.runtimeAdapterDeleteRequestedAt)),
7490+
);
7491+
}
7492+
7493+
private async confirmRuntimeAdapterIdentity(
7494+
identity: RuntimeAdapterIdentityRecord,
7495+
confirmedAt: string,
7496+
): Promise<void> {
7497+
if (
7498+
identity.claimVersion !== 1 ||
7499+
identity.claimState !== "provisional" ||
7500+
identity.confirmedAt !== undefined
7501+
) {
7502+
return;
7503+
}
7504+
const confirmed: RuntimeAdapterIdentityRecord = {
7505+
...identity,
7506+
claimState: "confirmed",
7507+
confirmedAt,
7508+
};
7509+
delete confirmed.claimExpiresAt;
7510+
await this.state.storage.put(runtimeAdapterIdentityKey(identity.adapterID), confirmed);
7511+
}
7512+
74247513
private async consumeWebVNCTicket(
74257514
request: Request,
74267515
identifier: string,

worker/test/coordinator-runtime.test.ts

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,10 @@ class MemoryStorage implements CoordinatorStorage {
3939
.map(([key, value]) => [key, value as T]),
4040
);
4141
}
42+
43+
value<T>(key: string): T | undefined {
44+
return this.values.get(key) as T | undefined;
45+
}
4246
}
4347

4448
class MemoryRuntime implements CoordinatorRuntime {
@@ -148,6 +152,11 @@ describe("coordinator runtimes", () => {
148152
expect(runtime.upgradeOptions).toEqual({ maxPayload: runtimeAdapterRelayFrameLimit });
149153
expect(runtime.acceptedTags).toEqual(["adapter:example-adapter", "runtime-adapter-agent"]);
150154
expect(runtime.acceptedAttachment).toMatchObject({ desktopTimeoutMs: 180_000 });
155+
expect(runtime.storage.value("runtime-adapter-identity:example-adapter")).toMatchObject({
156+
claimVersion: 1,
157+
claimState: "confirmed",
158+
confirmedAt: expect.any(String),
159+
});
151160
});
152161

153162
it("keeps provider-backed portal requests outside the lifecycle queue", () => {

0 commit comments

Comments
 (0)