Skip to content

Commit c0e7e5d

Browse files
ntkatholentkathole
authored
feat: Feast First-Class LabelView Implementation (#6292)
* feat: Feast First-Class LabelView Implementation Signed-off-by: ntkathole <[email protected]> * feat: Added UI for Labels and LabelView Signed-off-by: ntkathole <[email protected]> * feat: Added improved labelView UI version 2 Signed-off-by: ntkathole <[email protected]> --------- Signed-off-by: ntkathole <[email protected]>
1 parent 07b4782 commit c0e7e5d

120 files changed

Lines changed: 12694 additions & 3420 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424
* [Data ingestion](getting-started/concepts/data-ingestion.md)
2525
* [Entity](getting-started/concepts/entity.md)
2626
* [Feature view](getting-started/concepts/feature-view.md)
27+
* [Label view](getting-started/concepts/label-view.md)
2728
* [Feature retrieval](getting-started/concepts/feature-retrieval.md)
2829
* [Point-in-time joins](getting-started/concepts/point-in-time-joins.md)
2930
* [\[Alpha\] Saved dataset](getting-started/concepts/dataset.md)

‎docs/adr/ADR-0012-label-view.md‎

Lines changed: 519 additions & 0 deletions
Large diffs are not rendered by default.

docs/adr/feature-view-versioning.md renamed to docs/adr/rfc-feature-view-versioning.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# RFC: Feature View Versioning
22

3-
**Status:** In Review
3+
**Status:** Accepted
44
**Authors:** @farceo
55
**Branch:** `featureview-versioning`
66
**Date:** 2026-03-17

‎docs/getting-started/concepts/label-view.md‎

Lines changed: 356 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,356 @@
1+
# Label View
2+
3+
## What is a Label View?
4+
5+
A **label view** is a Feast primitive for storing **judgments about entities** — reward signals, safety scores, human reviews, ground-truth answers — separately from the **immutable observations** stored in [feature views](feature-view.md).
6+
7+
| | Feature View | Label View |
8+
|---|---|---|
9+
| **Stores** | What was observed | What was judged |
10+
| **Example** | Agent prompt and response | `response_quality: "poor"`, `is_safe: 0` |
11+
| **Writers** | Usually one source | Multiple labelers (human, LLM judge, code) |
12+
| **Changes over time** | Append-only | Updated by new label writes |
13+
14+
Use a label view when you need **governed, training-ready labels** that multiple sources can write, disagree on, and resolve — not when you need append-only feature data.
15+
16+
## Prerequisites
17+
18+
Before using label views, you need:
19+
20+
* Feast with the LabelView primitive
21+
* A feature repo with at least one `Entity`
22+
* A `PushSource` (or batch `DataSource`) for label ingestion
23+
* `feast apply` run after defining label views
24+
25+
## Why use Label Views?
26+
27+
**Separate features from judgments.** Mixing reward labels into feature views blurs two different lifecycles — observations are append-only; labels are overwritten, corrected, and debated.
28+
29+
**Support multiple labelers.** Human reviewers, safety scanners, and LLM judges can all label the same entity. Feast tracks who wrote what via `labeler_field` and resolves conflicts via `ConflictPolicy`.
30+
31+
**Generate training datasets.** Compose label views with feature views in a `FeatureService` and retrieve features + labels together with point-in-time correctness.
32+
33+
**Annotate in the UI.** Configure annotation profiles so data scientists can label data directly in the Feast UI — entity forms, document spans, bulk review, or active learning.
34+
35+
## Feedback vs Expectations
36+
37+
Not all labels serve the same purpose. Feast distinguishes two common types — both stored in a label view, modeled with field names and tags:
38+
39+
| | **Feedback** | **Expectation** |
40+
|---|---|---|
41+
| **Question** | How good was the actual output? | What is the correct answer? |
42+
| **Example** | `response_quality: "poor"`, `relevance: "irrelevant"` | `ground_truth: "relevant"`, `is_default: 1` |
43+
| **Typical writers** | Human, LLM judge, automated code | Human experts (gold standard) |
44+
| **Training use** | Reward signal, quality filter, active-learning queue | Supervised target column |
45+
| **Conflict handling** | Common — use `ConflictPolicy` | Rare — usually one authoritative source |
46+
47+
Feast does not require separate primitives for feedback and expectations. Model them with **field names** and **tags**:
48+
49+
```python
50+
tags={
51+
"feast.io/field-role:response_quality": "feedback", # judgment about output
52+
"feast.io/field-role:ground_truth": "expectation", # correct answer
53+
}
54+
```
55+
56+
**Practical pattern:**
57+
58+
* **Feedback label view** — multi-labeler, `ConflictPolicy`, fields like `response_quality`, `safety_score`, `relevance`
59+
* **Expectation fields or view** — stable ground truth, fields like `ground_truth`, `expected_answer`, `is_default`
60+
* **Mixed view** — one label view with both (e.g. RAG: `relevance` = feedback, `ground_truth` = expectation)
61+
62+
## Sources of Labels
63+
64+
Label views accept labels from any source. Track the writer in `labeler_field`:
65+
66+
| Source | `labeler` example | Typical role |
67+
|---|---|---|
68+
| **Human reviewer** | `[email protected]` | Feedback or expectation |
69+
| **LLM judge** | `gpt-4-evaluator` | Feedback (quality scores) |
70+
| **Automated scanner** | `nemo-guardrails` | Feedback (safety signals) |
71+
| **Batch import** | `risk-ops-team` | Expectation (historical outcomes) |
72+
73+
```python
74+
labels_df = pd.DataFrame({
75+
"user_id": ["user-001"],
76+
"response_quality": ["poor"],
77+
"is_safe": [0],
78+
"labeler": ["[email protected]"],
79+
"event_timestamp": [pd.Timestamp.now()],
80+
})
81+
store.push("agent_feedback_labels_push_source", labels_df)
82+
```
83+
84+
## When to use Label Views
85+
86+
| Use a **FeatureView** when… | Use a **LabelView** when… |
87+
|---|---|
88+
| Data is observational and append-only | Data is a judgment or annotation |
89+
| One source writes the data | Multiple labelers may disagree |
90+
| No conflict resolution needed | You need governed conflict resolution |
91+
| No labeling UI needed | You want structured annotation workflows |
92+
93+
## How Label Views Work
94+
95+
### Step 1: Define a label view
96+
97+
```python
98+
from datetime import timedelta
99+
100+
from feast import Entity, FeatureService, Field, FileSource, PushSource
101+
from feast.labeling import ConflictPolicy, LabelView
102+
from feast.types import Float32, Int64, String
103+
104+
user = Entity(name="user_id", join_keys=["user_id"])
105+
106+
agent_feedback_labels = LabelView(
107+
name="agent_feedback_labels",
108+
entities=[user],
109+
schema=[
110+
Field(name="response_quality", dtype=String),
111+
Field(name="is_safe", dtype=Int64),
112+
Field(name="reviewer_notes", dtype=String),
113+
Field(name="labeler", dtype=String),
114+
],
115+
source=PushSource(
116+
name="agent_feedback_labels_push_source",
117+
batch_source=FileSource(
118+
name="agent_feedback_labels_batch",
119+
path="data/agent_feedback.parquet",
120+
timestamp_field="event_timestamp",
121+
),
122+
),
123+
labeler_field="labeler",
124+
conflict_policy=ConflictPolicy.LAST_WRITE_WINS,
125+
reference_feature_view="user_profile",
126+
description="Human and automated feedback on agent responses.",
127+
tags={
128+
"feast.io/labeling-method": "entity-form",
129+
"feast.io/field-role:response_quality": "feedback",
130+
"feast.io/field-role:is_safe": "feedback",
131+
"feast.io/field-role:reviewer_notes": "metadata",
132+
"feast.io/label-values:response_quality": "excellent,good,acceptable,poor,harmful",
133+
"feast.io/label-values:is_safe": "1,0",
134+
"feast.io/label-widget:response_quality": "enum",
135+
"feast.io/label-widget:is_safe": "binary",
136+
"feast.io/label-widget:reviewer_notes": "text",
137+
},
138+
)
139+
```
140+
141+
Run `feast apply` to register the label view.
142+
143+
### Step 2: Push labels
144+
145+
Labels are written with `FeatureStore.push()`:
146+
147+
```python
148+
import pandas as pd
149+
from feast import FeatureStore
150+
151+
store = FeatureStore(repo_path="feature_repo/")
152+
153+
labels_df = pd.DataFrame({
154+
"user_id": ["user-001", "user-002"],
155+
"response_quality": ["good", "harmful"],
156+
"is_safe": [1, 0],
157+
"reviewer_notes": ["Accurate summary", "Unsafe medical advice"],
158+
"labeler": ["human-reviewer", "nemo-guardrails"],
159+
"event_timestamp": pd.to_datetime(["2025-01-15", "2025-01-15"]),
160+
})
161+
162+
store.push("agent_feedback_labels_push_source", labels_df)
163+
```
164+
165+
Each push appends to the offline store (full history retained) and updates the online store (latest value per key).
166+
167+
### Step 3: Annotate in the Feast UI
168+
169+
Open the label view in the Feast UI **Annotate** tab. The UI reads annotation tags and shows the right workflow:
170+
171+
1. Open **Label Views** in the sidebar
172+
2. Select a label view (check the **Annotation** badge on the list page)
173+
3. Go to the **Annotate** tab
174+
4. Choose a method (Entity Form, Document Span, Review & Edit, or Active Learning)
175+
5. Submit labels — they are pushed to the label view's `PushSource`
176+
177+
### Step 4: Join labels with features for training
178+
179+
```python
180+
training_service = FeatureService(
181+
name="agent_training_service",
182+
features=[
183+
user_profile_fv, # immutable features
184+
agent_feedback_labels, # mutable labels
185+
],
186+
)
187+
188+
training_df = store.get_historical_features(
189+
entity_df=entities_df,
190+
features=training_service,
191+
).to_df()
192+
```
193+
194+
Training pipelines get features and resolved labels in one retrieval call.
195+
196+
## Annotation Profiles
197+
198+
Annotation profiles configure **how** labels are created in the UI. Set them via `tags` — no schema changes required.
199+
200+
### Supported profiles
201+
202+
| Profile | Best for | UI experience |
203+
|---|---|---|
204+
| `entity-form` | RLHF, safety review, per-entity feedback | Form — one entity at a time |
205+
| `document-span` | RAG chunk labeling, span annotation | Load document, label chunks |
206+
| `table` | Bulk review, correcting existing labels | Editable table with dropdowns |
207+
| `active-learning` | Label high-value unlabeled entities | Queue from a reference feature view |
208+
209+
### Choosing a profile
210+
211+
Answer one question:
212+
213+
1. **"I need to review agent responses one at a time"** → `entity-form`
214+
2. **"I need to label document chunks for RAG"** → `document-span`
215+
3. **"I need to correct labels in bulk"** → `table`
216+
4. **"I want to label only the most valuable unlabeled items"** → `active-learning` (requires `reference_feature_view`)
217+
218+
The **Annotate** tab shows only relevant methods per profile:
219+
220+
| Profile | Methods shown |
221+
|---|---|
222+
| `document-span` | Document Span, Review & Edit |
223+
| `entity-form` | Entity Form, Review & Edit, Active Learning |
224+
| `table` | Review & Edit, Active Learning, Entity Form |
225+
| `active-learning` | Active Learning, Entity Form, Review & Edit |
226+
227+
### Tag reference
228+
229+
| Tag | Purpose | Example values |
230+
|---|---|---|
231+
| `feast.io/labeling-method` | Primary UI workflow | `entity-form`, `document-span`, `table` |
232+
| `feast.io/field-role:<field>` | Semantic role of a field | `feedback`, `expectation`, `label`, `metadata`, `content`, `span_start`, `span_end` |
233+
| `feast.io/label-values:<field>` | Allowed label values | `relevant,irrelevant` |
234+
| `feast.io/label-widget:<field>` | Input widget type | `enum`, `binary`, `text`, `number` |
235+
236+
## Examples by use case
237+
238+
### Agent feedback (RLHF / safety)
239+
240+
Feedback from human reviewers and automated safety layers on agent responses.
241+
242+
```python
243+
agent_feedback_labels = LabelView(
244+
name="agent_feedback_labels",
245+
entities=[user],
246+
schema=[
247+
Field(name="response_quality", dtype=String),
248+
Field(name="is_safe", dtype=Int64),
249+
Field(name="reviewer_notes", dtype=String),
250+
Field(name="labeler", dtype=String),
251+
],
252+
source=feedback_push_source,
253+
labeler_field="labeler",
254+
conflict_policy=ConflictPolicy.LAST_WRITE_WINS,
255+
reference_feature_view="user_profile",
256+
tags={"feast.io/labeling-method": "entity-form", ...},
257+
)
258+
```
259+
260+
### RAG chunk labeling (feedback + expectation)
261+
262+
One view can hold both a retrieval judgment and ground truth:
263+
264+
```python
265+
rag_chunk_labels = LabelView(
266+
name="rag_chunk_labels",
267+
entities=[chunk],
268+
schema=[
269+
Field(name="source_document", dtype=String),
270+
Field(name="chunk_text", dtype=String),
271+
Field(name="relevance", dtype=String), # feedback
272+
Field(name="ground_truth", dtype=String), # expectation
273+
Field(name="chunk_start", dtype=Int64),
274+
Field(name="chunk_end", dtype=Int64),
275+
Field(name="labeler", dtype=String),
276+
],
277+
source=rag_push_source,
278+
labeler_field="labeler",
279+
conflict_policy=ConflictPolicy.MAJORITY_VOTE,
280+
tags={
281+
"feast.io/labeling-method": "document-span",
282+
"feast.io/field-role:relevance": "feedback",
283+
"feast.io/field-role:ground_truth": "expectation",
284+
"feast.io/field-role:chunk_text": "content",
285+
"feast.io/label-values:relevance": "relevant,irrelevant",
286+
},
287+
)
288+
```
289+
290+
### Historical ground truth (batch labels)
291+
292+
Pre-existing label tables loaded via `feast materialize`:
293+
294+
```python
295+
loan_default_labels = LabelView(
296+
name="loan_default_labels",
297+
entities=[loan],
298+
schema=[
299+
Field(name="is_default", dtype=Int64),
300+
Field(name="delinquency_days", dtype=Int64),
301+
Field(name="loss_severity", dtype=Float32),
302+
Field(name="labeler", dtype=String),
303+
],
304+
source=PushSource(
305+
name="loan_default_labels_push_source",
306+
batch_source=FileSource(
307+
path="data/loan_defaults.parquet",
308+
timestamp_field="event_timestamp",
309+
),
310+
),
311+
labeler_field="labeler",
312+
conflict_policy=ConflictPolicy.LABELER_PRIORITY,
313+
tags={
314+
"feast.io/labeling-method": "table",
315+
"feast.io/field-role:is_default": "expectation",
316+
},
317+
)
318+
```
319+
320+
## Conflict policies
321+
322+
When multiple labelers write different values for the same entity, `ConflictPolicy` picks one value for **offline store reads** (training, UI browse):
323+
324+
| Policy | When to use |
325+
|---|---|
326+
| `LAST_WRITE_WINS` | Default. Most recent write wins. |
327+
| `LABELER_PRIORITY` | Trusted labelers override others (e.g. human over LLM judge). |
328+
| `MAJORITY_VOTE` | Consensus labeling (e.g. multiple annotators on RAG chunks). |
329+
330+
{% hint style="info" %}
331+
Conflict policies apply to the **offline store** (training). The **online store** always uses last-write-wins. Full label history is always retained in the offline store.
332+
{% endhint %}
333+
334+
## Best practices
335+
336+
**Name fields by intent.** Use `response_quality` (feedback) and `ground_truth` (expectation) — not generic `score` or `label`.
337+
338+
**Tag field roles.** Set `feast.io/field-role:<field>` to `feedback` or `expectation` so your team and UI know what each field means.
339+
340+
**Match conflict policy to label type.** Use `LABELER_PRIORITY` when humans correct automated judges. Use `MAJORITY_VOTE` for multi-annotator consensus. Use `LAST_WRITE_WINS` for simple feedback streams.
341+
342+
**Link to features.** Set `reference_feature_view` so the UI and documentation show which feature view the labels annotate.
343+
344+
**Separate noisy feedback from stable ground truth.** When possible, put expectations in dedicated fields or views with stricter writer conventions (human-only).
345+
346+
## Limitations
347+
348+
* Conflict policies are enforced on offline reads only; online store is always last-write-wins.
349+
* `LABELER_PRIORITY` requires explicit labeler ordering configuration.
350+
* Annotation profiles are UI configuration via tags — not enforced at the SDK write path.
351+
352+
## Next steps
353+
354+
* [Feature view](feature-view.md) — immutable features that label views annotate
355+
* [Feature retrieval](feature-retrieval.md) — point-in-time joins for training
356+
* [ADR-0012: LabelView](../../adr/ADR-0012-label-view.md) — full design rationale

0 commit comments

Comments
 (0)