Repository navigation
Modularize ingestion distributed compute engine support #444
Description
This is a companion to #402 and the larger topic of storage engine modularization which was realized in #529 and subsequent PRs that implemented the new interfaces.
Just as adding support for new storage engines tends to cause a dependency explosion for Feast ingestion & serving, the same is true for Beam Runner / job management adapter glue in core (this all could move to serving with future plans, but that won't change the fundamental problem this issue is about).
So for both storage and compute engines, I feel that some modularity strategy is needed for loose binding at build time, configurable for runtime. The goals would be to:
- Minimize dependency pains that developers and contributors to Feast need to deal with if they are not actively working on a particular stack. The dependency trees are often large and fragile, especially in the Hadoop ecosystem, such as Hive and Spark.
- Reduce deployment bloat if operators wish to package Feast internally with only the module JARs they need to support their organization's stack. (IIRC last I checked,
hadoop-commonorhadoop-clientleave you with close to 200MB of jars, andbeam-runners-sparkandbeam-sdks-java-io-hcatalogamong others have these deps [as provided scope, but the point stands I believe]).
Possibilities might be OSGi or java.util.ServiceLoader (and Spring integration or alternatives thereof). Open to other ideas!
Relates to #362
Activity
woop commentedon Jan 29, 2020
Agreed with this problem. I think we should be defining the exact extension points here very clearly though.
It can be hard to talk about in the abstract, so the questions I see are
Then more than that. I can see the introduction of this layer introducing a lot of overhead and complexity in the short term, even though it will pay dividends if teams are starting to fork the code base now (which might be the case already with 0.3). I would want to make sure we get alignment on the future direction of Feast so that we can stabilize the architecture before solidifying these modularization points.
stale commentedon Mar 29, 2020
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
dr3s commentedon Jun 5, 2020
I'm extremely wary of this type of complexity within the service. I'm pretty biased but I would prefer something more along the lines of these options:
I don't know what the state of the art is these days with OSGi or other SP frameworks but I don't think the complexity they bring is worth it.
woop commentedon Feb 8, 2021
Closing this issue since it is now stale. The Job Service manages jobs, and we have different launcher implementations available. Currently we are purely using Spark.