Skip to content

Commit a834126

Browse files
authored
feat: Add OpenLineage Consumer to Feast - receive, store, and visualize cross-producer lineage (#6549)
* feat: Added support for OpenLineage Consumer Signed-off-by: ntkathole <[email protected]> * feat: Added run history Signed-off-by: ntkathole <[email protected]> * test: Added test coverage for openlineage Signed-off-by: ntkathole <[email protected]> --------- Signed-off-by: ntkathole <[email protected]>
1 parent 58016bc commit a834126

36 files changed

Lines changed: 6228 additions & 89 deletions

‎.secrets.baseline‎

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/reference/openlineage.md‎

Lines changed: 260 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,12 @@ Captures materialization run metadata:
186186

187187
## Lineage Visualization
188188

189+
### Option 1: Feast UI (Built-in)
190+
191+
Feast includes a built-in OpenLineage consumer that can receive, store, and visualize lineage from **all** OpenLineage producers (Airflow, Spark, dbt, Feast itself, etc.) directly in the Feast UI. See the [OpenLineage Consumer](#openlineage-consumer) section below.
192+
193+
### Option 2: Marquez
194+
189195
Use [Marquez](https://marquezproject.ai/) to visualize your Feast lineage:
190196

191197
```bash
@@ -216,3 +222,257 @@ Then access the Marquez UI at http://localhost:3000 to see your feature lineage.
216222
| Entity | InputDataset |
217223
| FeatureService | OutputDataset |
218224
| Materialization | RunEvent (START/COMPLETE/FAIL) |
225+
226+
---
227+
228+
## OpenLineage Consumer
229+
230+
Feast can act as an **OpenLineage consumer**, receiving lineage events from any OpenLineage-compatible producer and displaying them in the Feast UI. This eliminates the need for a separate Marquez deployment when you want to visualize cross-system data lineage alongside your feature store.
231+
232+
### Consumer Architecture
233+
234+
```
235+
Producers (Airflow, Spark, dbt, Feast, Flink, …)
236+
│
237+
▼
238+
POST /api/v1/lineage ──→ Event Processor ──→ Lineage Store (SQL)
239+
│
240+
▼
241+
Feast UI
242+
┌──────────────────────────┐
243+
│ Lineage tab │
244+
│ ├─ OpenLineage Graph │
245+
│ │ (all producers) │
246+
│ └─ ☐ Feast Only Lineage │
247+
│ (registry view) │
248+
│ │
249+
│ Events tab │
250+
│ └─ Event browser │
251+
└──────────────────────────┘
252+
```
253+
254+
When the consumer is **not** enabled, the Feast UI shows only the original registry-based lineage view — no tabs are added.
255+
256+
### Enabling the Consumer
257+
258+
Add the `consumer` section under `openlineage` in your `feature_store.yaml`:
259+
260+
```yaml
261+
project: my_project
262+
registry:
263+
registry_type: sql
264+
path: postgresql://user:****@host:5432/feast # pragma: allowlist secret
265+
266+
openlineage:
267+
enabled: true
268+
namespace: my_project
269+
consumer:
270+
enabled: true
271+
store_type: sql
272+
# Optional: separate database for lineage storage.
273+
# If omitted, the SQL registry database is reused.
274+
# connection_string: postgresql://user:****@host:5432/feast_lineage
275+
api_key: "change-me" # pragma: allowlist secret
276+
namespace_mapping:
277+
airflow_ns: my_project
278+
spark_ns: my_project
279+
```
280+
281+
Or via environment variables:
282+
283+
```bash
284+
export FEAST_OPENLINEAGE_CONSUMER_ENABLED=true
285+
export FEAST_OPENLINEAGE_CONSUMER_STORE_TYPE=sql
286+
export FEAST_OPENLINEAGE_CONSUMER_API_KEY=change-me # pragma: allowlist secret
287+
# Optional separate DB:
288+
# export FEAST_OPENLINEAGE_CONSUMER_CONNECTION_STRING=postgresql://...
289+
```
290+
291+
### Consumer Configuration Options
292+
293+
| Option | Default | Description |
294+
|--------|---------|-------------|
295+
| `consumer.enabled` | `false` | Enable the OpenLineage consumer |
296+
| `consumer.store_type` | `sql` | Storage backend type. Currently only `sql` is supported |
297+
| `consumer.connection_string` | - | Optional separate database connection string. If omitted, reuses the SQL registry database |
298+
| `consumer.api_key` | - | API key that producers must provide when sending events |
299+
| `consumer.namespace_mapping` | `{}` | Maps OpenLineage namespaces to Feast projects for RBAC scoping |
300+
301+
### Consumer API Endpoints
302+
303+
When the consumer is enabled, the following endpoints are available on the Feast REST registry server:
304+
305+
#### Event Receiver (Producer-facing)
306+
307+
| Endpoint | Method | Description |
308+
|----------|--------|-------------|
309+
| `/api/v1/lineage` | `POST` | Receive a single OpenLineage event (or array of events) |
310+
| `/api/v1/lineage/batch` | `POST` | Receive a batch of OpenLineage events |
311+
312+
Both endpoints require the `X-API-Key` header (or `Authorization: Bearer <key>`) if `consumer.api_key` is configured.
313+
314+
#### Admin Endpoints
315+
316+
| Endpoint | Method | Description |
317+
|----------|--------|-------------|
318+
| `/lineage/openlineage/reset` | `DELETE` | Purge all OpenLineage data. Accepts optional `?namespace=X` to delete only a specific namespace. Requires API key. |
319+
320+
#### OpenLineage Query Endpoints (UI-facing)
321+
322+
| Endpoint | Method | Description |
323+
|----------|--------|-------------|
324+
| `/lineage/openlineage/graph` | `GET` | Full lineage graph with all nodes, edges, and symlinks |
325+
| `/lineage/openlineage/graph/{node_type}/{namespace}/{name}` | `GET` | Lineage graph centered on a specific node |
326+
| `/lineage/openlineage/events` | `GET` | Browse stored events with filtering |
327+
| `/lineage/openlineage/jobs` | `GET` | List all known OpenLineage jobs |
328+
| `/lineage/openlineage/datasets` | `GET` | List all known OpenLineage datasets |
329+
| `/lineage/openlineage/runs` | `GET` | List runs with optional `?job_namespace=X&job_name=Y` filtering |
330+
| `/lineage/openlineage/runs/{run_id}` | `GET` | Single run detail with input/output datasets |
331+
332+
#### Registry Query Endpoints
333+
334+
| Endpoint | Method | Description |
335+
|----------|--------|-------------|
336+
| `/lineage/registry` | `GET` | Feast registry lineage (entities, feature views, services) |
337+
| `/lineage/registry/all` | `GET` | All registry objects with full metadata |
338+
| `/lineage/objects/{object_type}/{object_name}` | `GET` | Detail for a specific registry object |
339+
| `/lineage/complete` | `GET` | Complete registry lineage with relationships |
340+
| `/lineage/complete/all` | `GET` | Complete registry lineage for all objects |
341+
342+
### Configuring Producers to Send Events to Feast
343+
344+
Configure any OpenLineage producer to send events to your Feast instance:
345+
346+
#### Airflow
347+
348+
```python
349+
# In airflow.cfg or environment
350+
OPENLINEAGE_URL = "http://feast-registry:8080/api"
351+
OPENLINEAGE_API_KEY = "change-me" # pragma: allowlist secret
352+
```
353+
354+
#### Spark
355+
356+
```properties
357+
spark.openlineage.transport.type=http
358+
spark.openlineage.transport.url=http://feast-registry:8080/api
359+
spark.openlineage.transport.endpoint=/v1/lineage
360+
spark.openlineage.transport.auth.type=api_key
361+
spark.openlineage.transport.auth.apiKey=change-me
362+
```
363+
364+
#### dbt
365+
366+
```yaml
367+
# In profiles.yml or environment
368+
OPENLINEAGE_URL: "http://feast-registry:8080/api"
369+
OPENLINEAGE_API_KEY: "change-me" # pragma: allowlist secret
370+
```
371+
372+
#### Feast (Self-reporting)
373+
374+
When both the OpenLineage producer and consumer are enabled, Feast's own events (from `feast apply`, materialization, etc.) are automatically ingested into the local consumer store — no HTTP transport is needed.
375+
376+
```yaml
377+
# In feature_store.yaml
378+
openlineage:
379+
enabled: true
380+
namespace: my_project
381+
consumer:
382+
enabled: true
383+
api_key: change-me # pragma: allowlist secret
384+
```
385+
386+
### Feast UI Lineage Views
387+
388+
When the consumer is enabled, the lineage page in the Feast UI shows two tabs:
389+
390+
**Lineage tab**
391+
392+
- **OpenLineage Graph** (default) — shows lineage from all OpenLineage producers with cross-producer connectivity. Nodes are color-coded by producer (colors generated dynamically). The graph supports filtering by type, producer, and object name. Clicking a node opens a **detail panel** showing description, schema, tags, features, entities, data quality metrics, data source info, other facets, and **run history** (for job nodes — see [Per-Run Lineage](#per-run-lineage-run-history)).
393+
- **Feast Only Lineage** (checkbox) — switches to the original Feast registry view (DataSource → FeatureView → FeatureService) powered entirely by the Feast registry.
394+
395+
**Events tab**
396+
397+
- Browse individual OpenLineage events with filtering by event type, job name, and run ID. Expand any event to inspect the full JSON payload.
398+
399+
### Cross-Producer Lineage Connectivity
400+
401+
The consumer automatically links datasets across different producers when they refer to the same physical data. Linking mechanisms:
402+
403+
1. **Shared namespace + name** — If Airflow writes to `s3://bucket/path` and Spark reads from the same `s3://bucket/path`, the graph connects them automatically.
404+
2. **SymlinksDatasetFacet** — Producers can declare aliases. For example, Feast can declare that its internal `driver_hourly_stats` is a symlink to the Spark output at `s3://bucket/features/driver_hourly_stats/`.
405+
3. **dataSource URI matching** — Datasets with matching `dataSource.uri` facets are linked even if their namespace or name differ.
406+
407+
Compatible producers include Airflow, Spark, dbt, Flink, Feast, Dagster, and Great Expectations.
408+
409+
### RBAC for Lineage
410+
411+
The OpenLineage consumer integrates with Feast's existing RBAC:
412+
413+
- **Write access** (producers sending events): Authenticated via API key in the `X-API-Key` header
414+
- **Read access** (UI viewing lineage): Namespace-based filtering maps OpenLineage namespaces to Feast projects. Users see only lineage data for namespaces they have access to via the `namespace_mapping` configuration
415+
416+
### Lineage Cleanup / Reset
417+
418+
Over time the OpenLineage store accumulates historical data. Two mechanisms are provided for cleanup:
419+
420+
#### Admin Reset Endpoint
421+
422+
Use the `DELETE /lineage/openlineage/reset` endpoint to purge lineage data. The endpoint requires the same API key used for event ingestion.
423+
424+
```bash
425+
# Purge ALL OpenLineage data
426+
curl -X DELETE -H "X-API-Key: your-key" \
427+
http://localhost:8080/api/v1/lineage/openlineage/reset
428+
429+
# Purge only a specific namespace
430+
curl -X DELETE -H "X-API-Key: your-key" \
431+
"http://localhost:8080/api/v1/lineage/openlineage/reset?namespace=airflow://prod-cluster"
432+
```
433+
434+
A full purge deletes data from all seven `openlineage_*` tables. A namespace-scoped purge deletes jobs, datasets, runs, events, edges, and symlinks associated with that namespace, leaving other namespaces intact.
435+
436+
#### Feast Teardown Hook
437+
438+
When you run `feast teardown`, Feast automatically cleans up OpenLineage data for the project's namespace (if the consumer is configured). This ensures that tearing down a Feast project doesn't leave orphaned lineage data behind.
439+
440+
```bash
441+
# Tears down the Feast project AND its OpenLineage lineage
442+
feast teardown
443+
```
444+
445+
### Per-Run Lineage (Run History)
446+
447+
The consumer tracks individual pipeline runs in the `openlineage_runs` table. When you click on a **job node** in the OpenLineage Graph, the detail panel shows a **Run History** section with:
448+
449+
- A table of past runs: truncated run ID, status badge (COMPLETE, FAIL, RUNNING, ABORT), start time, and duration
450+
- Click any run to expand its **inputs and outputs** — the specific datasets that run consumed and produced
451+
452+
#### Run History API
453+
454+
```bash
455+
# List runs for a specific job
456+
curl "http://localhost:8080/api/v1/lineage/openlineage/runs?job_namespace=spark://emr-cluster&job_name=feature_engineering"
457+
458+
# Get a single run with its I/O datasets
459+
curl "http://localhost:8080/api/v1/lineage/openlineage/runs/{run_id}"
460+
```
461+
462+
The run detail response includes `inputs` and `outputs` arrays, each containing the dataset namespace, name, and any I/O facets recorded by the producer.
463+
464+
### Database Schema
465+
466+
The consumer creates the following tables (automatically on first startup):
467+
468+
| Table | Purpose |
469+
|-------|---------|
470+
| `openlineage_events` | Raw event storage with JSON payloads |
471+
| `openlineage_jobs` | Deduplicated job records with producer, description, and facets |
472+
| `openlineage_datasets` | Deduplicated dataset records with schema, facets, and Feast mapping |
473+
| `openlineage_runs` | Run lifecycle tracking (START/COMPLETE/FAIL) |
474+
| `openlineage_run_io` | Input/output relationships between runs and datasets |
475+
| `openlineage_lineage_edges` | Materialized lineage graph edges for efficient traversal |
476+
| `openlineage_dataset_symlinks` | Cross-producer dataset linking via `SymlinksDatasetFacet` and `dataSource` URI matching |
477+
478+
By default these tables are created in the **same database** as the SQL registry (hybrid storage). Set `consumer.connection_string` to store them in a separate database instead.

‎infra/feast-operator/api/v1/featurestore_types.go‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,35 @@ type OpenLineageConfig struct {
111111
// Keys must be valid Feast OpenLineageConfig YAML field names.
112112
// +optional
113113
ExtraConfig map[string]string `json:"extraConfig,omitempty"`
114+
// Consumer configures the OpenLineage consumer (event receiver) that enables
115+
// Feast to receive and display lineage from external producers (Airflow, Spark, dbt, etc.).
116+
// +optional
117+
Consumer *OpenLineageConsumerConfig `json:"consumer,omitempty"`
118+
}
119+
120+
// OpenLineageConsumerConfig configures the OpenLineage consumer (event receiver).
121+
// When enabled, the Feast REST server exposes POST /api/v1/lineage to receive
122+
// OpenLineage events from any producer, storing them for visualization in the Feast UI.
123+
type OpenLineageConsumerConfig struct {
124+
// Enable the OpenLineage consumer.
125+
Enabled bool `json:"enabled"`
126+
// StoreType is the storage backend for lineage events. Currently only "sql" is supported.
127+
// +kubebuilder:default="sql"
128+
// +kubebuilder:validation:Enum=sql
129+
// +optional
130+
StoreType *string `json:"storeType,omitempty"`
131+
// Reference to a Secret containing the key "connection_string" for a separate
132+
// lineage database. If omitted, the SQL registry database is reused.
133+
// +optional
134+
ConnectionStringSecretRef *corev1.LocalObjectReference `json:"connectionStringSecretRef,omitempty"`
135+
// Reference to a Secret containing the key "api_key" that producers must
136+
// provide in the X-API-Key header when sending events.
137+
// +optional
138+
ApiKeySecretRef *corev1.LocalObjectReference `json:"apiKeySecretRef,omitempty"`
139+
// NamespaceMapping maps OpenLineage namespaces to Feast projects for
140+
// RBAC-based filtering of lineage data in the UI.
141+
// +optional
142+
NamespaceMapping map[string]string `json:"namespaceMapping,omitempty"`
114143
}
115144

116145
// FeatureStoreSpec defines the desired state of FeatureStore

‎infra/feast-operator/api/v1/zz_generated.deepcopy.go‎

Lines changed: 42 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)