Skip to content

Commit b6538c3

Browse files
authored
TS SDK: declare task dependencies by calling tasks (#73435)
1 parent 90972c4 commit b6538c3

16 files changed

Lines changed: 1039 additions & 366 deletions

File tree

‎airflow-core/docs/authoring-and-scheduling/language-sdks/typescript.rst‎

Lines changed: 43 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -121,16 +121,12 @@ That top-level ``await`` makes the module a runnable bundle entry point.
121121
The ``dagId`` a handler binds must match the ``dag_id`` of the Python Dag, and the ``taskId`` a
122122
``@task.stub`` function in that Dag, including any TaskGroup prefix.
123123

124-
``register`` takes any number of task handlers and ``bundle.serve()`` serves exactly what is registered,
125-
so a task left out is not part of the packed bundle and is marked removed at runtime.
124+
``register`` takes any number of task handlers and Dags, and ``bundle.serve()`` serves exactly what is
125+
registered, so a task left out is not part of the packed bundle and is marked removed at runtime.
126126
A second ``bundle.serve()`` call is rejected.
127127
Registering holds no sockets and starts nothing, so a unit test can build a bundle and dispatch a handler
128128
through ``bundle.getTaskHandler(dagId, taskId)`` without a coordinator runtime.
129129

130-
``Dag`` is another interface, for a Dag declared in TypeScript rather than in Python, and is still a work
131-
in progress. ``new Dag`` and ``dag.task`` take a trailing options object (``spec`` on both, plus
132-
``inputs`` on a task) that is not used yet; do not set them.
133-
134130
TaskFlow arguments
135131
~~~~~~~~~~~~~~~~~~
136132

@@ -242,6 +238,47 @@ task instance.
242238
inside the scheduler process, so they must be present where the scheduler can read them. The API server
243239
and Dag processor do not need them.
244240

241+
.. _typescript-sdk/native-dag:
242+
243+
Declaring a Dag in TypeScript
244+
-----------------------------
245+
246+
A ``Dag`` is declared on this side rather than in Python: its tasks and the edges between them are
247+
written in TypeScript. The surface is still growing, so a Dag declared this way is not served to
248+
Airflow yet.
249+
250+
``dag.task(taskId, handler)`` returns a *factory*. Calling it places the task in the Dag and supplies the
251+
handler's arguments, so the call graph is the task graph:
252+
253+
.. code-block:: typescript
254+
255+
import { Dag } from "apache-airflow-ts-sdk";
256+
257+
const dag = new Dag("ts_etl");
258+
259+
const extract = dag.task("extract", async (): Promise<number> => 42);
260+
const transform = dag.task("transform", async (rows: number, region: string) => rows * 2);
261+
const load = dag.task("load", async (total: number) => {});
262+
263+
load(transform(extract(), "us"));
264+
265+
Arguments are passed in the order the handler declares them. A handler that declares a single object of
266+
named arguments can also be called with that object, which names each input instead of ordering it:
267+
268+
.. code-block:: typescript
269+
270+
const store = dag.task("store", async ({ total }: { total: number }) => {});
271+
272+
store({ total: extract() });
273+
274+
Each argument takes either an upstream reference or a literal JSON value. A reference has to be the
275+
argument itself: one buried inside an array or an object is a literal, and draws no edge.
276+
277+
Every task has to be called exactly once. An uncalled task fails when the Dag is read, so none can be
278+
left out of the graph by accident.
279+
280+
``new Dag`` and ``dag.task`` also take a trailing ``spec`` object that is not used yet; do not set it.
281+
245282
Writing tasks
246283
-------------
247284

‎ts-sdk/adr/0002-native-dag-interface.md‎

Lines changed: 19 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -28,11 +28,15 @@ Proposed. Revised after the review on #72047.
2828
1. **`dag.task(handler)` returns a factory, and the task id is optional.** With no id the task takes
2929
the handler's function name (`dag.task(extract)` → task `"extract"`); `dag.task(taskId, handler)`
3030
sets it explicitly, which an anonymous handler must do. Calling the factory both places the task in
31-
the Dag and supplies its arguments by name — the shape Python TaskFlow uses for
31+
the Dag and supplies its arguments, in the order the handler declares them
32+
(`load(transform(extract(), "us"))`). A handler that declares a single object of named arguments can
33+
also be called with that object — the shape Python TaskFlow uses for
3234
`load(transformed=transform(...))`.
3335
2. **The call graph is the task graph.** `tsc` checks every wired key against the handler's own
3436
parameter type, and a `TaskRef` exists only once its producing call has returned, so a cycle
35-
through arguments is unrepresentable rather than rejected by a validator.
37+
through arguments is unrepresentable rather than rejected by a validator. A reference passed by
38+
position is checked against the argument's own type, which is what tells the two call shapes apart
39+
when a handler declares a single argument.
3640
3. **Every task is called exactly once.** An uncalled task fails when the Dag is read, so none can be
3741
silently left out of the graph.
3842
4. **`before` and `after` draw order-only edges** — the TypeScript pair for `>>` and `<<`, both
@@ -145,21 +149,20 @@ convention.
145149
by design. Native declaration is what fills them, generated from the serialized-Dag JSON schema the
146150
way `src/generated/supervisor.ts` is. This ADR does not choose those fields; it fixes where an
147151
author writes them.
148-
- `TaskOptions` collapses into `TaskSpec`. The shipped third argument to `dag.task` is
149-
`{ inputs, spec }`; with wiring moved to the factory call, `inputs` is no longer an option and the
150-
third argument is the spec itself.
152+
- `TaskOptions` carries the spec and the handler's positional argument names, which the packer fills in
153+
from the parameter list so the Dag names each argument as its handler does. With wiring moved to the
154+
factory call, `inputs` is no longer an option.
151155
- `TaskHandlerArgs` is removed from the public API, `DagRegistry` becomes `Bundle`, and
152156
`serveDags(registry)` becomes `bundle.serve()`, which breaks
153157
0.1.0-beta1 authors; see [ADR-0001](0001-mixed-lang-dag-interface.md) for the shipped call sites
154158
that change.
155159

156160
## Alternatives
157161

158-
- **Positional wiring** (`load(transform(extract()))`), which becomes expressible once data no longer
159-
shares an object with `ctx`/`client`, since a handler can then take its arguments positionally and
160-
`Parameters<typeof handler>` is a real tuple. Rejected: it removes the key names from every call
161-
site, and those names are what keeps flat, one-statement-per-task wiring readable at twenty tasks.
162-
A handler may still take several positional arguments; only the *wiring* stays named.
162+
- **Named-only wiring**, rejected in the review on #73435: naming every input reads well at twenty
163+
tasks but forces an object around a single argument, and positional calls are what TypeScript
164+
authors write. Both are offered, and the handler's own parameter list decides which one a task can
165+
use.
163166
- **Injected `ctx`/`client` arguments**, mimicking the Python signature. Rejected, per the above and
164167
because feeling native to TypeScript matters more than matching Python's parameter list.
165168

@@ -175,9 +178,12 @@ convention.
175178
the same object, which forced every typed handler to declare `TArgs & TaskHandlerArgs` and left the
176179
top-level argument namespace open to collisions with an author's own parameter names. Getters close
177180
both.
178-
- **The spec argument already has its slot.** `dag.task(taskId, handler, options)` reads
179-
`{ inputs = {}, spec = {} }` and runs `validateEmptySpec` on the spec today
180-
(`ts-sdk/src/sdk/dag.ts`), so task fields land on a path that exists rather than a new one.
181+
- **The spec argument already has its slot.** `dag.task(taskId, handler, options)` reads `{ spec = {} }`
182+
and runs `validateEmptySpec` on it (`ts-sdk/src/sdk/dag.ts`), so task fields land on a path that
183+
exists rather than a new one.
184+
- **A positional argument binds by order, and its name is a label.** The serialized Dag names each
185+
argument, so the packer reads the names from the handler's parameter list; `arg0`, `arg1` and so on
186+
stand in for a name it cannot see, without changing which value reaches which argument.
181187
- **A `TaskRef` is inert** — a handle for wiring, not a promise. Nothing in a Dag file executes a task
182188
body.
183189
- **A defaulted task id is resolved at pack time, not read at runtime.** esbuild renames function

‎ts-sdk/src/coordinator/manifest.ts‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
*/
1919

2020
import { SUPERVISOR_API_VERSION } from "./protocol.js";
21-
import { listBundleDags, type Bundle } from "../sdk/bundle.js";
21+
import { bundleDagTaskIds, finalizeBundleDags, type Bundle } from "../sdk/bundle.js";
2222

2323
export const AIRFLOW_METADATA_FLAG = "--airflow-metadata";
2424

@@ -36,7 +36,10 @@ export interface BundleManifest {
3636

3737
export function buildBundleManifest(bundle: Bundle): BundleManifest {
3838
const taskHandlers: BundleManifest["task_handlers"] = {};
39-
for (const { dagId, tasks } of listBundleDags(bundle)) {
39+
// The manifest is the bundle reporting what it provides, so this is where its
40+
// Dags are finalized: a Dag missing an edge is reported here rather than packed.
41+
finalizeBundleDags(bundle);
42+
for (const [dagId, tasks] of bundleDagTaskIds(bundle)) {
4043
if (typeof dagId !== "string") {
4144
throw new Error("Dag ID must be a string");
4245
}

‎ts-sdk/src/coordinator/runtime.ts‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ import {
5555
type StartupDetails,
5656
} from "./protocol.js";
5757
import { getArgNames } from "../sdk/arg-names.js";
58-
import { listBundleTasks, type Bundle } from "../sdk/bundle.js";
58+
import { bundleDagTaskIds, type Bundle } from "../sdk/bundle.js";
5959
import { runInTaskScope, type TaskContext } from "../sdk/task.js";
6060
import type { JsonValue } from "../sdk/client-types.js";
6161

@@ -148,10 +148,10 @@ export async function startCoordinator(
148148
const runtimeLogs = logs.child("runtime");
149149
runtimeLogs.debug("Connecting log socket", { logs_addr: parsed.logsAddr });
150150
await logs.connect(parsed.logsAddr);
151-
const tasks = listBundleTasks(bundle);
151+
const byDag = bundleDagTaskIds(bundle);
152152
runtimeLogs.info("Coordinator runtime started", {
153-
registered_tasks: tasks,
154-
count: tasks.length,
153+
registered_tasks: Object.fromEntries(byDag),
154+
count: [...byDag.values()].reduce((total, tasks) => total + tasks.length, 0),
155155
// Cadwyn schema version this SDK was generated against. Logged
156156
// for operator visibility; not sent on the wire.
157157
supervisor_api_version: SUPERVISOR_API_VERSION,
@@ -270,7 +270,7 @@ function handleParse(
270270
// TypeScript-native Dag parsing is not yet supported.
271271
// Respond with an empty result so the Python-stub-Dag workflow works.
272272
logs.info("Parse-mode response (TS Dag parsing not yet supported)", {
273-
registered_tasks: listBundleTasks(bundle),
273+
registered_tasks: Object.fromEntries(bundleDagTaskIds(bundle)),
274274
});
275275
const response: RuntimeDagFileParsingResult = {
276276
type: "DagFileParsingResult",
@@ -295,7 +295,7 @@ async function handleTask(
295295
logs.warning("No handler registered for task", {
296296
dag_id: ti.dag_id,
297297
task_id: ti.task_id,
298-
available: listBundleTasks(bundle),
298+
available: Object.fromEntries(bundleDagTaskIds(bundle)),
299299
});
300300
// A missing handler means this bundle cannot run the task, so retrying the
301301
// same bundle/configuration mismatch would not help.

‎ts-sdk/src/index.ts‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,16 @@ export { ConnectionNotFoundError, VariableNotFoundError } from "./sdk/client.js"
2626
export { SUPERVISOR_API_VERSION } from "./coordinator/index.js";
2727
export type { ArgNameMap } from "./sdk/arg-names.js";
2828
export type { Registerable } from "./sdk/bundle.js";
29-
export type { DagSpec, TaskInputs, TaskOptions, TaskRef, TaskSpec } from "./sdk/dag.js";
29+
export type {
30+
DagSpec,
31+
PositionalInputs,
32+
TaskFactory,
33+
TaskInput,
34+
TaskInputs,
35+
TaskOptions,
36+
TaskRef,
37+
TaskSpec,
38+
} from "./sdk/dag.js";
3039
export type { TaskClient } from "./sdk/client.js";
3140
export type { ConnectionResult, GetXComOpts, JsonValue, SetXComOpts } from "./sdk/client-types.js";
3241
export type { TaskContext, TaskFunction } from "./sdk/task.js";

0 commit comments

Comments
 (0)