Native Kafka Streaming Ingestion and Job Lifecycle Management #7108

Is your feature request related to a problem? Please describe.

Feast provides support for defining streaming data sources through KafkaSource and processing streaming data using StreamFeatureView. However, users are currently responsible for managing the Kafka ingestion pipeline themselves, including launching and maintaining the streaming processor.

This introduces additional operational complexity, as users need to build and maintain custom applications to consume Kafka events, process them through Feast's streaming infrastructure, and ensure reliable ingestion.

It would be useful if Feast could provide a native mechanism to manage this ingestion workflow, reducing the amount of custom infrastructure users need to maintain.

Describe the solution you'd like

Introduce native Kafka streaming ingestion support that allows users to configure and manage streaming ingestion jobs directly through Feast.

Ideally, this would include:

• Native ingestion job management: Allow users to launch and manage streaming ingestion jobs using registered KafkaSource and StreamFeatureView definitions.
• Consumer lifecycle management: Handle Kafka consumer initialization, topic subscriptions, consumer groups, and offset management.
• Fault tolerance: Support checkpointing, retries, and recovery from ingestion failures.
• Streaming processor integration: Reuse Feast's existing StreamProcessor abstraction and supported streaming engines rather than introducing a separate processing framework.
• Feature persistence: Support writing processed features to the configured online and offline stores.

The goal is to make Kafka-based streaming ingestion easier to configure, deploy, and operate without requiring users to implement and maintain their own Kafka-to-Feast ingestion applications.

Describe alternatives you've considered

1. Custom Kafka consumer: Users can implement their own Kafka consumer application, consume events, and push data into Feast. However, this requires maintaining additional infrastructure and handling ingestion reliability themselves.

2. External streaming orchestration: Users can deploy and manage Spark/Flink streaming jobs independently and integrate them with Feast. While this provides flexibility, it introduces additional operational overhead.

3. Existing StreamProcessor integration: Extend the existing streaming processor implementations to support more of the ingestion lifecycle, rather than building an entirely new streaming execution framework.

Additional context

Feast already provides several building blocks for streaming ingestion:

• KafkaSource for defining Kafka-backed streaming data sources.
• StreamFeatureView for defining streaming feature transformations.
• StreamProcessor for executing streaming transformations.
• Online and offline store abstractions for feature persistence.

This feature request aims to bridge the gap between these existing components by providing a more integrated ingestion and job management experience.

An initial implementation could focus on Kafka with an existing supported streaming engine, with the architecture designed to accommodate additional streaming sources and engines in the future.