You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a834126
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/reference/openlineage.md
+260Lines changed: 260 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -186,6 +186,12 @@ Captures materialization run metadata:
186
186
187
187
## Lineage Visualization
188
188
189
+
### Option 1: Feast UI (Built-in)
190
+
191
+
Feast includes a built-in OpenLineage consumer that can receive, store, and visualize lineage from **all** OpenLineage producers (Airflow, Spark, dbt, Feast itself, etc.) directly in the Feast UI. See the [OpenLineage Consumer](#openlineage-consumer) section below.
192
+
193
+
### Option 2: Marquez
194
+
189
195
Use [Marquez](https://marquezproject.ai/) to visualize your Feast lineage:
190
196
191
197
```bash
@@ -216,3 +222,257 @@ Then access the Marquez UI at http://localhost:3000 to see your feature lineage.
Feast can act as an **OpenLineage consumer**, receiving lineage events from any OpenLineage-compatible producer and displaying them in the Feast UI. This eliminates the need for a separate Marquez deployment when you want to visualize cross-system data lineage alongside your feature store.
231
+
232
+
### Consumer Architecture
233
+
234
+
```
235
+
Producers (Airflow, Spark, dbt, Feast, Flink, …)
236
+
│
237
+
▼
238
+
POST /api/v1/lineage ──→ Event Processor ──→ Lineage Store (SQL)
239
+
│
240
+
▼
241
+
Feast UI
242
+
┌──────────────────────────┐
243
+
│ Lineage tab │
244
+
│ ├─ OpenLineage Graph │
245
+
│ │ (all producers) │
246
+
│ └─ ☐ Feast Only Lineage │
247
+
│ (registry view) │
248
+
│ │
249
+
│ Events tab │
250
+
│ └─ Event browser │
251
+
└──────────────────────────┘
252
+
```
253
+
254
+
When the consumer is **not** enabled, the Feast UI shows only the original registry-based lineage view — no tabs are added.
255
+
256
+
### Enabling the Consumer
257
+
258
+
Add the `consumer` section under `openlineage` in your `feature_store.yaml`:
|`consumer.enabled`|`false`| Enable the OpenLineage consumer |
296
+
|`consumer.store_type`|`sql`| Storage backend type. Currently only `sql` is supported |
297
+
|`consumer.connection_string`| - | Optional separate database connection string. If omitted, reuses the SQL registry database |
298
+
|`consumer.api_key`| - | API key that producers must provide when sending events |
299
+
|`consumer.namespace_mapping`|`{}`| Maps OpenLineage namespaces to Feast projects for RBAC scoping |
300
+
301
+
### Consumer API Endpoints
302
+
303
+
When the consumer is enabled, the following endpoints are available on the Feast REST registry server:
304
+
305
+
#### Event Receiver (Producer-facing)
306
+
307
+
| Endpoint | Method | Description |
308
+
|----------|--------|-------------|
309
+
|`/api/v1/lineage`|`POST`| Receive a single OpenLineage event (or array of events) |
310
+
|`/api/v1/lineage/batch`|`POST`| Receive a batch of OpenLineage events |
311
+
312
+
Both endpoints require the `X-API-Key` header (or `Authorization: Bearer <key>`) if `consumer.api_key` is configured.
313
+
314
+
#### Admin Endpoints
315
+
316
+
| Endpoint | Method | Description |
317
+
|----------|--------|-------------|
318
+
|`/lineage/openlineage/reset`|`DELETE`| Purge all OpenLineage data. Accepts optional `?namespace=X` to delete only a specific namespace. Requires API key. |
319
+
320
+
#### OpenLineage Query Endpoints (UI-facing)
321
+
322
+
| Endpoint | Method | Description |
323
+
|----------|--------|-------------|
324
+
|`/lineage/openlineage/graph`|`GET`| Full lineage graph with all nodes, edges, and symlinks |
325
+
|`/lineage/openlineage/graph/{node_type}/{namespace}/{name}`|`GET`| Lineage graph centered on a specific node |
326
+
|`/lineage/openlineage/events`|`GET`| Browse stored events with filtering |
327
+
|`/lineage/openlineage/jobs`|`GET`| List all known OpenLineage jobs |
328
+
|`/lineage/openlineage/datasets`|`GET`| List all known OpenLineage datasets |
329
+
|`/lineage/openlineage/runs`|`GET`| List runs with optional `?job_namespace=X&job_name=Y` filtering |
330
+
|`/lineage/openlineage/runs/{run_id}`|`GET`| Single run detail with input/output datasets |
When both the OpenLineage producer and consumer are enabled, Feast's own events (from `feast apply`, materialization, etc.) are automatically ingested into the local consumer store — no HTTP transport is needed.
375
+
376
+
```yaml
377
+
# In feature_store.yaml
378
+
openlineage:
379
+
enabled: true
380
+
namespace: my_project
381
+
consumer:
382
+
enabled: true
383
+
api_key: change-me # pragma: allowlist secret
384
+
```
385
+
386
+
### Feast UI Lineage Views
387
+
388
+
When the consumer is enabled, the lineage page in the Feast UI shows two tabs:
389
+
390
+
**Lineage tab**
391
+
392
+
- **OpenLineage Graph** (default) — shows lineage from all OpenLineage producers with cross-producer connectivity. Nodes are color-coded by producer (colors generated dynamically). The graph supports filtering by type, producer, and object name. Clicking a node opens a **detail panel** showing description, schema, tags, features, entities, data quality metrics, data source info, other facets, and **run history** (for job nodes — see [Per-Run Lineage](#per-run-lineage-run-history)).
393
+
- **Feast Only Lineage** (checkbox) — switches to the original Feast registry view (DataSource → FeatureView → FeatureService) powered entirely by the Feast registry.
394
+
395
+
**Events tab**
396
+
397
+
- Browse individual OpenLineage events with filtering by event type, job name, and run ID. Expand any event to inspect the full JSON payload.
398
+
399
+
### Cross-Producer Lineage Connectivity
400
+
401
+
The consumer automatically links datasets across different producers when they refer to the same physical data. Linking mechanisms:
402
+
403
+
1. **Shared namespace + name** — If Airflow writes to `s3://bucket/path` and Spark reads from the same `s3://bucket/path`, the graph connects them automatically.
404
+
2. **SymlinksDatasetFacet** — Producers can declare aliases. For example, Feast can declare that its internal `driver_hourly_stats` is a symlink to the Spark output at `s3://bucket/features/driver_hourly_stats/`.
405
+
3. **dataSource URI matching** — Datasets with matching `dataSource.uri` facets are linked even if their namespace or name differ.
406
+
407
+
Compatible producers include Airflow, Spark, dbt, Flink, Feast, Dagster, and Great Expectations.
408
+
409
+
### RBAC for Lineage
410
+
411
+
The OpenLineage consumer integrates with Feast's existing RBAC:
412
+
413
+
- **Write access** (producers sending events): Authenticated via API key in the `X-API-Key` header
414
+
- **Read access** (UI viewing lineage): Namespace-based filtering maps OpenLineage namespaces to Feast projects. Users see only lineage data for namespaces they have access to via the `namespace_mapping` configuration
415
+
416
+
### Lineage Cleanup / Reset
417
+
418
+
Over time the OpenLineage store accumulates historical data. Two mechanisms are provided for cleanup:
419
+
420
+
#### Admin Reset Endpoint
421
+
422
+
Use the `DELETE /lineage/openlineage/reset` endpoint to purge lineage data. The endpoint requires the same API key used for event ingestion.
A full purge deletes data from all seven `openlineage_*` tables. A namespace-scoped purge deletes jobs, datasets, runs, events, edges, and symlinks associated with that namespace, leaving other namespaces intact.
435
+
436
+
#### Feast Teardown Hook
437
+
438
+
When you run `feast teardown`, Feast automatically cleans up OpenLineage data for the project's namespace (if the consumer is configured). This ensures that tearing down a Feast project doesn't leave orphaned lineage data behind.
439
+
440
+
```bash
441
+
# Tears down the Feast project AND its OpenLineage lineage
442
+
feast teardown
443
+
```
444
+
445
+
### Per-Run Lineage (Run History)
446
+
447
+
The consumer tracks individual pipeline runs in the `openlineage_runs` table. When you click on a **job node** in the OpenLineage Graph, the detail panel shows a **Run History** section with:
448
+
449
+
- A table of past runs: truncated run ID, status badge (COMPLETE, FAIL, RUNNING, ABORT), start time, and duration
450
+
- Click any run to expand its **inputs and outputs** — the specific datasets that run consumed and produced
The run detail response includes `inputs` and `outputs` arrays, each containing the dataset namespace, name, and any I/O facets recorded by the producer.
463
+
464
+
### Database Schema
465
+
466
+
The consumer creates the following tables (automatically on first startup):
467
+
468
+
| Table | Purpose |
469
+
|-------|---------|
470
+
| `openlineage_events` | Raw event storage with JSON payloads |
471
+
| `openlineage_jobs` | Deduplicated job records with producer, description, and facets |
472
+
| `openlineage_datasets` | Deduplicated dataset records with schema, facets, and Feast mapping |
473
+
| `openlineage_runs` | Run lifecycle tracking (START/COMPLETE/FAIL) |
474
+
| `openlineage_run_io` | Input/output relationships between runs and datasets |
| `openlineage_dataset_symlinks` | Cross-producer dataset linking via `SymlinksDatasetFacet` and `dataSource` URI matching |
477
+
478
+
By default these tables are created in the **same database** as the SQL registry (hybrid storage). Set `consumer.connection_string` to store them in a separate database instead.
0 commit comments