Skip to content

Commit ccbd215

Browse files
authored
TypeScript SDK: take the task state store retention in seconds (#75445)
* TypeScript SDK: take the task state store retention in seconds retentionMs was the one duration in the SDK that was not seconds, next to executionTimeout, retryDelay and dagrunTimeout, which all are. Rename it to retentionSeconds so every duration carries the same unit. The HITL example modules now export their Dags, so another bundle can serve them. * TypeScript SDK: keep the HITL example Dags unexported Importing them would run the module-level serve() call, so the importing bundle's own serve() would then fail. * TypeScript SDK: name the task state store option retention Every other public duration is a bare name in seconds, and the Python argument is retention too. * TypeScript SDK: report an overflowing retention in seconds The caller passes seconds, so the error now uses the same unit. * TypeScript SDK docs: state the seconds rule once in a Durations section The task state store section links to it instead of carrying the rule. * TypeScript SDK: name the operator timeout option timeout Every duration in the SDK is now a bare name in seconds, as defer's timeout is in Python. ---------
1 parent 5223758 commit ccbd215

11 files changed

Lines changed: 75 additions & 52 deletions

File tree

‎airflow-core/docs/authoring-and-scheduling/language-sdks/typescript.rst‎

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -565,7 +565,7 @@ The ``TaskClient``
565565
* - ``setXCom({ key, value, ... })``
566566
- Publishes an XCom value.
567567
* - ``taskStateStore``
568-
- This task instance's key/value store, with ``get<T>(key)``, ``set(key, value, { retentionMs? })``,
568+
- This task instance's key/value store, with ``get<T>(key)``, ``set(key, value, { retention? })``,
569569
``delete(key)`` and ``clear()``. See :ref:`typescript-sdk/task-state-store`.
570570

571571
.. note::
@@ -595,23 +595,25 @@ for the full concept, including the equivalent Python API.
595595
let jobId = await store.get<string>("job_id");
596596
if (jobId == null) {
597597
jobId = await submitSparkJob();
598-
await store.set("job_id", jobId, { retentionMs: NEVER_EXPIRE });
598+
await store.set("job_id", jobId, { retention: NEVER_EXPIRE });
599599
}
600600
601601
const result = await waitForSparkJob(jobId);
602602
await store.delete("job_id");
603603
return result;
604604
}
605605
606-
**Retention.** ``retentionMs`` is milliseconds to retain the key, counted from the time of the write:
606+
**Retention.** ``retention`` is how long to keep the key, in seconds (see :ref:`typescript-sdk/durations`),
607+
counted from the time of the write:
607608

608-
* A number retains the key for that many milliseconds from now; ``retentionMs: 0`` expires the key
609+
* A number retains the key for that many seconds from now; ``retention: 0`` expires the key
609610
immediately.
610611
* ``NEVER_EXPIRE`` (imported from ``apache-airflow-ts-sdk``) stores the key with no expiry, regardless of
611612
the deployment default.
612-
* When ``retentionMs`` is omitted the runtime reads ``AIRFLOW__STATE_STORE__DEFAULT_RETENTION_DAYS``, which
613-
the coordinator sets from ``[state_store] default_retention_days``. A value of ``0`` there means the key
614-
never expires. This is not the same as ``retentionMs: 0`` above, which expires the key immediately.
613+
* When ``retention`` is omitted the runtime reads ``AIRFLOW__STATE_STORE__DEFAULT_RETENTION_DAYS``,
614+
which the coordinator sets from ``[state_store] default_retention_days``. A value of ``0`` there means the
615+
key never expires. This is not the same as ``retention: 0`` above, which expires the key
616+
immediately.
615617

616618
**Keys** must be non-empty strings of at most 512 characters; they may contain slashes. **Values** must be
617619
JSON-compatible (see :ref:`typescript-sdk/types`), and can never be ``null`` — ``set`` rejects a ``null``
@@ -632,6 +634,15 @@ Anything a task writes to stdout or stderr, such as ``console.log`` or ``console
632634
Airflow task log: stdout at ``INFO`` level, stderr at ``ERROR``. There is no dedicated structured-logging API
633635
yet.
634636

637+
.. _typescript-sdk/durations:
638+
639+
Durations
640+
---------
641+
642+
Every duration in the SDK is a number of seconds, with no unit in its name: the Dag and task options
643+
``retryDelay``, ``executionTimeout`` and ``dagrunTimeout``, ``responseTimeout`` on a human-in-the-loop task,
644+
``pokeInterval`` on ``triggerDagRun``, and ``retention`` in the task state store.
645+
635646
.. _typescript-sdk/types:
636647

637648
XCom type mapping

‎ts-sdk/example/src/main.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ export async function writeAndReadTaskState() {
6666
const store = getClient().taskStateStore;
6767
const { runId, tryNumber } = getContext();
6868

69-
await store.set(JOB_ID_KEY, runId, { retentionMs: NEVER_EXPIRE });
69+
await store.set(JOB_ID_KEY, runId, { retention: NEVER_EXPIRE });
7070
await store.set(STATE_SCRATCH_KEY, { attempt: tryNumber });
7171

7272
const jobId = await store.get<string>(JOB_ID_KEY);

‎ts-sdk/src/coordinator/client.ts‎

Lines changed: 17 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -142,7 +142,8 @@ const TASK_STATE_STORE_ABSENT_POLICY: AbsentRowPolicy = {
142142
// `[state_store] default_retention_days` at launch
143143
// (task-sdk/src/airflow/sdk/coordinators/_subprocess.py).
144144
const RETENTION_DAYS_ENV_VAR = "AIRFLOW__STATE_STORE__DEFAULT_RETENTION_DAYS";
145-
const MS_PER_DAY = 24 * 60 * 60 * 1000;
145+
const MS_PER_SECOND = 1000;
146+
const SECONDS_PER_DAY = 24 * 60 * 60;
146147

147148
function assertKey(key: string): void {
148149
if (typeof key !== "string") {
@@ -157,28 +158,29 @@ function assertKey(key: string): void {
157158
// past 9999, and the supervisor silently drops a frame it cannot decode
158159
// rather than replying with an error, so the task would hang instead of
159160
// failing fast. Route both retention paths through this check.
160-
function toExpiresAt(ms: number): string {
161-
const at = new Date(Date.now() + ms);
161+
function toExpiresAt(seconds: number): string {
162+
const at = new Date(Date.now() + seconds * MS_PER_SECOND);
162163
if (Number.isNaN(at.getTime()) || at.getUTCFullYear() > 9999) {
163164
throw new RangeError(
164-
`retention of ${ms}ms overflows the wire timestamp; use NEVER_EXPIRE instead`,
165+
`a retention that ends ${seconds}s from now overflows the wire timestamp; ` +
166+
"use NEVER_EXPIRE instead",
165167
);
166168
}
167169
return at.toISOString();
168170
}
169171

170-
/** Resolve `retentionMs` (or the deployment default) to a wire `expires_at`. */
171-
function resolveExpiresAt(retentionMs: number | typeof NEVER_EXPIRE | undefined): string | null {
172-
if (retentionMs === undefined) {
172+
/** Resolve `retention` (or the deployment default) to a wire `expires_at`. */
173+
function resolveExpiresAt(seconds: number | typeof NEVER_EXPIRE | undefined): string | null {
174+
if (seconds === undefined) {
173175
return resolveDefaultExpiresAt();
174176
}
175-
if (retentionMs === NEVER_EXPIRE) {
177+
if (seconds === NEVER_EXPIRE) {
176178
return null;
177179
}
178-
if (!Number.isFinite(retentionMs) || retentionMs < 0) {
179-
throw new RangeError(`retentionMs must be >= 0 or NEVER_EXPIRE, got ${String(retentionMs)}`);
180+
if (!Number.isFinite(seconds) || seconds < 0) {
181+
throw new RangeError(`retention must be >= 0 or NEVER_EXPIRE, got ${String(seconds)}`);
180182
}
181-
return toExpiresAt(retentionMs);
183+
return toExpiresAt(seconds);
182184
}
183185

184186
// The coordinator always passes the setting, so an absent or malformed value is
@@ -189,8 +191,8 @@ function resolveDefaultExpiresAt(): string | null {
189191
if (raw === undefined) {
190192
throw new Error(
191193
`${RETENTION_DAYS_ENV_VAR} is not set, so the default retention is unknown. It carries ` +
192-
"[state_store] default_retention_days to the runtime; pass retentionMs or NEVER_EXPIRE to " +
193-
"set the expiry explicitly.",
194+
"[state_store] default_retention_days to the runtime; pass retention or " +
195+
"NEVER_EXPIRE to set the expiry explicitly.",
194196
);
195197
}
196198
const days = Number(raw);
@@ -202,7 +204,7 @@ function resolveDefaultExpiresAt(): string | null {
202204
`[state_store] default_retention_days must be >= 0, got ${raw}. Set to 0 to disable expiry.`,
203205
);
204206
}
205-
return days === 0 ? null : toExpiresAt(days * MS_PER_DAY);
207+
return days === 0 ? null : toExpiresAt(days * SECONDS_PER_DAY);
206208
}
207209

208210
export function createCoordinatorClient(
@@ -277,7 +279,7 @@ export function createCoordinatorClient(
277279
ti_id: tiId,
278280
key,
279281
value,
280-
expires_at: resolveExpiresAt(opts.retentionMs),
282+
expires_at: resolveExpiresAt(opts.retention),
281283
};
282284
await rpc("SetTaskStateStore", "OKResponse", msg, () => undefined, "throw");
283285
},

‎ts-sdk/src/coordinator/operator-runner.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,7 @@ export function buildOperatorContext(deps: OperatorContextDeps): OperatorContext
108108
return {
109109
type: "AwaitInputTask",
110110
state: "awaiting_input",
111-
timeout: toIsoDuration(opts.timeoutSeconds),
111+
timeout: toIsoDuration(opts.timeout),
112112
next_method: EXECUTE_COMPLETE,
113113
next_kwargs: opts.kwargs ?? {},
114114
};
@@ -119,7 +119,7 @@ export function buildOperatorContext(deps: OperatorContextDeps): OperatorContext
119119
state: "deferred",
120120
classpath: opts.classpath,
121121
trigger_kwargs: opts.kwargs,
122-
trigger_timeout: toIsoDuration(opts.timeoutSeconds),
122+
trigger_timeout: toIsoDuration(opts.timeout),
123123
// `_defer_task` hands the trigger the task's queue only when triggerer queues are enabled.
124124
queue:
125125
opts.queue !== undefined

‎ts-sdk/src/hitl/operator.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,7 @@ async function executeHITL(task: HITLTask<never>, op: OperatorContext): Promise<
9393
});
9494

9595
logs.info("Waiting for response", { task_id: ctx.taskId });
96-
return op.awaitInput({ timeoutSeconds: task.responseTimeout });
96+
return op.awaitInput({ timeout: task.responseTimeout });
9797
}
9898

9999
/**

‎ts-sdk/src/sdk/client-types.ts‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -51,8 +51,8 @@ export interface SetXComOpts {
5151
}
5252

5353
/**
54-
* Pass as {@link TaskStateStoreSetOpts.retentionMs} to store a task state store
55-
* key with no expiry, regardless of the deployment default.
54+
* Pass as {@link TaskStateStoreSetOpts.retention} to store a task state
55+
* store key with no expiry, regardless of the deployment default.
5656
*
5757
* A symbol rather than a number, so a retention that reaches `Infinity` by
5858
* accident (a division by zero, for example) is rejected instead of silently
@@ -63,14 +63,14 @@ export const NEVER_EXPIRE: unique symbol = Symbol("apache-airflow-ts-sdk/NEVER_E
6363
/** Options for a task state store write. */
6464
export interface TaskStateStoreSetOpts {
6565
/**
66-
* Milliseconds to retain the key, counted from the write; `0` expires it
67-
* immediately.
66+
* How long to keep the key, in seconds, counted from the write; `0` expires
67+
* it immediately.
6868
*
6969
* Omitting it follows the deployment's `[state_store] default_retention_days`,
7070
* which the coordinator passes to the runtime. Pass {@link NEVER_EXPIRE} to
7171
* store the key with no expiry regardless of that default.
7272
*/
73-
retentionMs?: number | typeof NEVER_EXPIRE;
73+
retention?: number | typeof NEVER_EXPIRE;
7474
}
7575

7676
/** Airflow Connection details returned by a task client. */

‎ts-sdk/src/sdk/client.ts‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -133,7 +133,7 @@ export interface TaskClient {
133133
* let jobId = await store.get<string>("job_id");
134134
* if (jobId == null) {
135135
* jobId = await submit();
136-
* await store.set("job_id", jobId, { retentionMs: NEVER_EXPIRE });
136+
* await store.set("job_id", jobId, { retention: NEVER_EXPIRE });
137137
* }
138138
* ```
139139
*
@@ -157,9 +157,11 @@ export interface TaskStateStore {
157157
/**
158158
* Store a value, replacing any existing value for the key.
159159
*
160+
* `opts.retention` is how long to keep the key, in seconds.
161+
*
160162
* @throws `TypeError` when `key` is not a string, or `value` is null,
161163
* undefined, NaN, or Infinity.
162-
* @throws `RangeError` when `key` is empty; when `retentionMs` is
164+
* @throws `RangeError` when `key` is empty; when `retention` is
163165
* negative, NaN, or Infinity; when the deployment's
164166
* `[state_store] default_retention_days` is invalid; or when the resolved
165167
* expiry would overflow the wire timestamp (year 9999).

‎ts-sdk/src/sdk/operator.ts‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -61,19 +61,19 @@ export interface OperatorContext {
6161
fail(message: string): RuntimeRetryTask | RuntimeTaskState;
6262
/** Skip `taskIds`, as `SkipMixin.skip` does. */
6363
skip(taskIds: readonly string[]): Promise<void>;
64-
/** Park in `awaiting_input`. The next run calls `executeComplete`. */
64+
/** Park in `awaiting_input`. The next run calls `executeComplete`. `timeout` is in seconds. */
6565
awaitInput(opts?: {
66-
timeoutSeconds?: number;
66+
timeout?: number;
6767
kwargs?: Record<string, JsonValue>;
6868
}): RuntimeAwaitInputTask;
6969
/**
7070
* Defer to a Python trigger. The next run calls `executeComplete`. Without a `queue`, the
71-
* trigger gets the task's queue when triggerer queues are enabled.
71+
* trigger gets the task's queue when triggerer queues are enabled. `timeout` is in seconds.
7272
*/
7373
defer(opts: {
7474
classpath: string;
7575
kwargs: Record<string, JsonValue>;
76-
timeoutSeconds?: number;
76+
timeout?: number;
7777
queue?: string | null;
7878
}): RuntimeDeferTask;
7979
}

‎ts-sdk/tests/coordinator/client.test.ts‎

Lines changed: 18 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -165,7 +165,7 @@ describe("writes do not read a supervisor 404 as absence", () => {
165165
[
166166
"taskStateStore.set",
167167
"SetTaskStateStore",
168-
(c: TaskClient) => c.taskStateStore.set("k", 1, { retentionMs: 60_000 }),
168+
(c: TaskClient) => c.taskStateStore.set("k", 1, { retention: 60 }),
169169
],
170170
[
171171
"taskStateStore.delete",
@@ -446,7 +446,7 @@ describe("taskStateStore.set retention", () => {
446446
vi.setSystemTime(new Date("2026-01-01T00:00:00.000Z"));
447447
const { client: c, sent } = recordingClient({ type: "OKResponse", ok: true });
448448

449-
await c.taskStateStore.set("k", 1, { retentionMs: NEVER_EXPIRE });
449+
await c.taskStateStore.set("k", 1, { retention: NEVER_EXPIRE });
450450

451451
expect(sent[0]).toEqual({
452452
type: "SetTaskStateStore",
@@ -457,22 +457,22 @@ describe("taskStateStore.set retention", () => {
457457
});
458458
});
459459

460-
it("sends now + retentionMs for a finite retentionMs", async () => {
460+
it("sends now + retention for a finite retention", async () => {
461461
vi.useFakeTimers();
462462
vi.setSystemTime(new Date("2026-01-01T00:00:00.000Z"));
463463
const { client: c, sent } = recordingClient({ type: "OKResponse", ok: true });
464464

465-
await c.taskStateStore.set("k", 1, { retentionMs: 60_000 });
465+
await c.taskStateStore.set("k", 1, { retention: 60 });
466466

467467
expect(sent[0]).toMatchObject({ expires_at: "2026-01-01T00:01:00.000Z" });
468468
});
469469

470-
it("sends now (expire immediately) for retentionMs 0", async () => {
470+
it("sends now (expire immediately) for retention 0", async () => {
471471
vi.useFakeTimers();
472472
vi.setSystemTime(new Date("2026-01-01T00:00:00.000Z"));
473473
const { client: c, sent } = recordingClient({ type: "OKResponse", ok: true });
474474

475-
await c.taskStateStore.set("k", 1, { retentionMs: 0 });
475+
await c.taskStateStore.set("k", 1, { retention: 0 });
476476

477477
expect(sent[0]).toMatchObject({ expires_at: "2026-01-01T00:00:00.000Z" });
478478
});
@@ -535,15 +535,23 @@ describe("taskStateStore.set guards", () => {
535535
it.each([
536536
["negative", -1],
537537
["NaN", Number.NaN],
538-
])("rejects retentionMs=%s without sending a request", async (_label, retentionMs) => {
538+
])("rejects retention=%s without sending a request", async (_label, retention) => {
539539
const { client: c, sent } = recordingClient();
540-
await expect(c.taskStateStore.set("k", 1, { retentionMs })).rejects.toThrow(RangeError);
540+
await expect(c.taskStateStore.set("k", 1, { retention })).rejects.toThrow(RangeError);
541+
expect(sent).toHaveLength(0);
542+
});
543+
544+
it("reports an overflowing retention in seconds", async () => {
545+
const { client: c, sent } = recordingClient();
546+
await expect(c.taskStateStore.set("k", 1, { retention: 1e12 })).rejects.toThrow(
547+
"a retention that ends 1000000000000s from now",
548+
);
541549
expect(sent).toHaveLength(0);
542550
});
543551

544552
it("rejects an empty key without sending a request", async () => {
545553
const { client: c, sent } = recordingClient();
546-
await expect(c.taskStateStore.set("", 1, { retentionMs: 60_000 })).rejects.toThrow(RangeError);
554+
await expect(c.taskStateStore.set("", 1, { retention: 60 })).rejects.toThrow(RangeError);
547555
expect(sent).toHaveLength(0);
548556
});
549557
});
@@ -577,7 +585,7 @@ describe("task state store key type guard", () => {
577585
"set",
578586
(c: TaskClient) => {
579587
// @ts-expect-error exercising the runtime guard against an untyped caller.
580-
return c.taskStateStore.set(undefined, 1, { retentionMs: 60_000 });
588+
return c.taskStateStore.set(undefined, 1, { retention: 60 });
581589
},
582590
],
583591
[

‎ts-sdk/tests/coordinator/operator-runner.test.ts‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -157,13 +157,13 @@ describe("buildOperatorContext", () => {
157157
it.each([
158158
[undefined, null],
159159
[30, "PT30S"],
160-
])("parks with a timeout of %s as %s", (timeoutSeconds, timeout) => {
160+
])("parks with a timeout of %s as %s", (timeout, expected) => {
161161
const { op } = makeContext();
162162

163-
expect(op.awaitInput({ timeoutSeconds })).toEqual({
163+
expect(op.awaitInput({ timeout })).toEqual({
164164
type: "AwaitInputTask",
165165
state: "awaiting_input",
166-
timeout,
166+
timeout: expected,
167167
next_method: "execute_complete",
168168
next_kwargs: {},
169169
});
@@ -178,7 +178,7 @@ describe("buildOperatorContext", () => {
178178
it("defers to the trigger, resuming through execute_complete", () => {
179179
const { op } = makeContext();
180180

181-
expect(op.defer({ classpath: "pkg.Trigger", kwargs: { a: 1 }, timeoutSeconds: 5 })).toEqual({
181+
expect(op.defer({ classpath: "pkg.Trigger", kwargs: { a: 1 }, timeout: 5 })).toEqual({
182182
type: "DeferTask",
183183
state: "deferred",
184184
classpath: "pkg.Trigger",

0 commit comments

Comments
 (0)