Authentication and authorization

Creating this issue to track the discussion around user authentication & authorization.

1. Prior Art: Auth as proposed in the Feast Projects RFC

In the Feast: Projects RFC we proposed a basic form of authentication as a "minmal" implementation for Feast 0.5.

Overview
At a high level, this proposal adds user authentication based on a new user table that Feast maintains. No API is made available to manage the creation/removal of users, but the actions taken by users is on the Feast Core API is limited to the resources they have access to. Resource access is managed only based on membership of users to projects.

Note on Roles: "Roles" were originally proposed but will be delayed until a future release can implement them properly. Right now there is only a full "owner" role which is implicit. All users either have ownership of a project, or not.

Assumptions:

User authentication only applies to Feast Core, not ingestion or "serving".

Out of scope:

Full user management (create or remove users, or manage user profiles).
JWT or functionality like sessions on user auth.
RBAC

Data model

API

Feast Core API
- New RPCs
  - AddMember: Add a user to a specific project
  - RemoveMember: Remove user from a specific project
  - ListMembers: List members in a project
  - ~~AddRole~~
  - ~~RemoveRole~~
Feast Core Data Model
- New table
  - Users: Users table
  - UserProjects: Relation between users & projects
  - ~~Roles: Roles table~~
  - ~~UserProjectRole: Relation between roles, users, and projects~~
- New classes
  - Users
  - ~~Roles~~
Python SDK
- Addition of user auth configuration
- New methods:
  - add_member()
  - remove_member()
  - list_members()
  - ~~add_role()~~
  - ~~remove_role()~~

2. Objectives

The main objective of this issue is to finalize two proposals. The first is a "minimal" set of functionality that we want to support, and the second is a "full" set of functionality. Since Feast 0.5 is our next release, the priority is to finalize the implementation of the minimal release.

Minimal: To implement a basic form of authentication in Feast 0.5.
Full: Create an extensible/pluggable authentication & authorization system that supports multiple providers, can be used by both services and users, and covers ingestion, Core, and serving. No PR exists for this functionality yet.

3. Requirements (minimal release)

The following user stories should be supported as part of the first minimal release (Feast 0.5).

3.1 User stories

As a Feast user, so that I can prevent other users from modifying my feature sets, I would like to have a way to prevent modification of feature sets by unauthorized users.
~~As a Feast user, so that I can prevent other users from seeing my feature sets, I would like to have a way to limit access to certain users to view my feature sets~~ [Edit: Removing this story. It would be simpler to allow full view access]
As a Feast user, so that I can allow my team to modify/view my feature sets, I would like to have a way to selectively grant or revoke access to feature sets for specific users.
As a Feast user, so that I can grant access to multiple feature sets in a project, I would like to manage access control to feature sets at the project level
As a Feast user, so that Feast can uniquely identify me, I would like to be able to authenticate myself to Feast.

3.2 In Scope

Authentication of users
Management of membership of users to projects for the purposes of granting access to feature sets within those projects
Project level access control (read vs write) based on membership of projects

3.3 Out of Scope

Full authorization or RBAC.
Stream, ingestion, and serving are also out of scope for the purposes of authentication and authorization.
Full user management (create or remove users, or manage user profiles).
JWT or functionality like sessions on user auth.

4. Open questions

Which authentication frameworks or providers should be supported for Feast 0.5?
Are there any frameworks or open source services that would make implementing authorization safer, especially offload the responsibility of managing user credentials?
Do our long term and short term goals for both user authentication and authorization align?
What should be in scope for authentication for Feast 0.5?

@woop

Thanks for starting this issue!

It's pretty definite what's deemed out of scope for the short term:

Short term: … Authorization and user management is out of scope. Stream, ingestion, and serving are also out of scope.

User authentication only applies to Feast Core, not ingestion or "serving".

Could we expand on what is in scope then, in more practical terms? I'm thinking user stories or use cases for this:

To implement a basic form of authentication in Feast 0.5 that prevents users/systems from modifying resources they shouldn't have access to.

I hope @woop won't mind me quoting from a comment thread on the auth section of the Projects RFC, this answer to my similar question there is helpful:

We are assuming that Core will be centrally hosted and accessible by most users in an organization, but we assume that Serving will be deployed locally to users (where retrieval is safe) or remotely (where network level access control is in effect).

This might be a false assumption, but for now it punts development of this functionality a little bit.

In summary, it amounts to creation or modification of feature sets only for now.

@ches

Thanks @ches. I have updated the comment above. Please have a look at the user stories to see if it makes sense now. I feel like the whole Projects RFC pollutes this discussion a little bit, but I guess its better to have that context.

Feel free to propose the removal, modification (feature sets vs projects?), or addition of new stories for 0.5, as long as the amount of work still fits our timeline.

Since auth has such a high risk in causing divergence, I think a safe approach here would be to see if we can come up with a super minimal implementation for 0.5 that leaves doors open for an RFC for the full auth implementation. We can then take a bit more time on that.

Based on the user stories above, we essentially want the following in a first cut:

Authenticate a user.
Find members (users) of project(s).
Add/remove user to project.

I did a bit of research on the options we have available to us.

Proposal 1: Basic auth built into Feast

As proposed above in introduction for Feast Projects RFC. We'd store users, a password hash, and a mapping to projects.

User is authenticated based on metadata (user, password hash) sent as metadata through gRPC call
User=>project mapping is found through a table in Feast Core database.
Adding or removing users is done through Feast Core database.

Pros:

Easiest to implement
Small surface area and complexity

Cons:

User/pass authenticated is inherintly less secure than modern methods.
Probably will be replaced entirely

Proposal 2: Keycloak/Ory + gRPC metadata

In this flow we would use Keycloak (or the Ory suite) as an IAM solution.

Feast administrators would need to deploy and manage their own Keycloak deployment, but this would provide identity and access management functionality.
It would be up to the caller of the Feast Core API to first authenticate themselves with Keycloak in order to have the necessary access tokens.
The Python SDK can be extended to produce a URL to allow users to log in through Keycloak
Requests are then sent as a JWT through gRPC metadata.
Feast Core implements an AuthenticationProvider (there is a reference implementation here that seems pretty good)
Authorization: It's still unclear to me whether sufficient information could be sent through JWT or if another call to Keycloak would be necessary for authorization purposes.

Pros:

Don't need to add data models or manage user identities or roles. Keycloak provides a user interface to manage IAM
Keycloak is very powerful and supports multiple authentication mechanisms and identity providers.
Auth is optional, and can be implemented by administrators only if needed

Cons:

Need to deploy and manage Keycloak.
A lot more development work to integrate this into Python SDK and Feast Core.
Surface area of changes is higher. If we use Spring Security then this implementation would be interwoven with the Feast Core API.
Role retrieval might require more development.

Proposal 3: Assume auth is outside of Feast (upstream)

In this proposal we assume that all requests that reach Feast are already authenticated and authorized. The assumption here is that a proxy or some upstream system is both auditing and controlling incoming requests.

Pros

Buys us time to produce a proper authentication RFC and get consensus
Doesn't pollute the Feast code base with a half-solution

Cons

Each team needs to implement their own proxy to provide auth

Envoy with external auth could be a solution here.

Proposal 4: Authentication Interceptor with OSGi/ServiceLoader

In this approach we would agree to an auth interface only. This interface is executed through a gRPC server interceptor on every request. It's up to each team to implement their own authentication module that they can load at startup through an OSGi or ServiceLoader module.

Pros

Provides high flexibility to teams to roll their own solutions
Wouldn't be too difficult to implement
Very clean solution since the only surface area for auth in Feast is the interceptor and interface
Doesn't pollute the Feast codebase with a half-solution
No changes to Feast data model

Cons

Agreeing to the contract would be a challenge in its own right.
Not sure if it would be possible to leverage spring security modules easily

Eliminated ideas

TLS mutual auth: Even if we could authenticate users with TLS, we still need a mapping between users and projects.
Oathkeeper/ Dex / Pomerium / Oauth Proxy: We can authenticate users but still need a way to map users to projects. Also not clear if it supports gRPC.

I think for the main objective of preventing modification of FeatureSpec by users who do not own/mantain the features, proposal 1 is a good start.

From the user stories, it seems like we need the authentication to be applied at FeatureSet level but I think it's simpler if we start at project level permission?

What about service accounts? Are they treated the same as users?

What about service accounts? Are they treated the same as users?

Good question, this would depend on the actual implementation that we go with. If we delegate this to a identity provider then most of them already support a distinct account type for service accounts. If we roll our own then the easiest solution is just to treat them the same.

I do think service accounts should be a component of a full auth RFC though, as opposed to this first release.

I haven't fully digested updates to the main issue and proposals, but one thing I can add:

Authorization: It's still unclear to me whether sufficient information could be sent through JWT or if another call to Keycloak would be necessary for authorization purposes.

We've used Keycloak in the past (moved away from it, for corporate more than technical reasons FWIW). Roughly speaking, Keycloak provides ways to map data available from integrated auth providers into JWT identity tokens (such as LDAP group memberships). So it's possible to implement authorization in the application without deep integration with Keycloak for authorization purposes. There are plenty of nuances to consider as with all things auth, but thought it'd be at least somewhat helpful to share this. Probably data in the tokens would become the contract Feast specifies, rather than a strict requirement of Keycloak (although it may be one of the most accessible relatively vendor-neutral solutions to recommend / use as reference).

And now, with skimming / shallow thought warning:

Broadly I lean toward 3 or 4 (external auth and interceptor implementation, respectively) in favor of deferring (spec, because it's such a large topic and a drag on momentum) and delegating (because it's virtually impossible to please everyone and be future proof when it comes to auth).

I acknowledge the goal for the most pressing practical use cases, and that Proposal 1 might be the fastest and most straightforward route to serving it. I think I'm okay with it if the community prioritizes the goal, given that the gRPC metadata mechanisms for auth are used—this was one of my first concerns, not polluting the surface area in the form of request schemas, etc. when the protocol provides means to avoid that and flexibility to change the implementation less intrusively.

I would agree with "Probably will be replaced entirely" though ☺️

Disclaimer: I am biased in that we have no immediate plans to use Feast Projects.

Oathkeeper/ Dex / Pomerium / Oauth Proxy: We can authenticate users but still need a way to map users to projects.

Should this be an eliminating factor though? Externalizing authorization is still something very hard to do without coupling to opinionated infrastructure dependencies, but authentication has viable options IMO, you've covered many great ones.

Also not clear if it supports gRPC.

This might be a de facto eliminator though. Unless we have time and people to add it, or to wait in hopes that the community will do it…

@ches

Oathkeeper/ Dex / Pomerium / Oauth Proxy: We can authenticate users but still need a way to map users to projects.

Should this be an eliminating factor though? Externalizing authorization is still something very hard to do without coupling to opinionated infrastructure dependencies, but authentication has viable options IMO, you've covered many great ones.

Also not clear if it supports gRPC.

This might be a de facto eliminator though. Unless we have time and people to add it, or to wait in hopes that the community will do it…

I look at all of them. I don't think a single one supports gRPC layer authentication. Not even Envoy. You would need to write your own Lua plugin to support it (which teams in Gojek have done), but its a non-trivial amount of work.

Broadly I lean toward 3 or 4 (external auth and interceptor implementation, respectively)

@ches I am busy drafting my proposal for auth right now. Give me another 30 minutes. I dont think external authentication is viable yet (I hope I am wrong though), but I think I have a road to interceptor level auth that is relatively modular.

After spiking out various options over the past week, I have settled on the following

Feast Auth (Minimal): Authentication through OpenID Connect and an Authorization interface

Overview

In this proposal we would only add auth to the Feast Core gRPC layer. This would be done through a gRPC interceptor. Authentication and authorization are both implemented as interfaces, of which we will implement a single example that suits our internal authentication (Google OAuth) and authorization layers (Keto)

Authorization and authentication are both optional and can be selectively enabled server side.

High level sequence

Clients (Users)

Feast clients must provide an Id token to Feast Core if Feast Core has enabled authentication.

Feast will try to discover this token in two places. First, it will look for FEAST_ID_TOKEN environmental variable. If it does not find the Id token there, it will try to discover it from the environment. If users have the Google Cloud SDK installed then Feast will try to load the Id token from there.

The Google Cloud SDK allows them to generate an id token for a period of 10 hours. Feast can then simply retrieve or regenerate tokens based on the user’s active gcloud configuration. In most cases this is not an action that the user has to take. It would also be trivial to allow them to generate these tokens through a "browser based flow", but for now we are leaving this out of scope.

Optionally a service account can be configured so that Id tokens can be generated using that service account. We will only allow this through the configuration of an environmental variable called GOOGLE_APPLICATION_CREDENTIALS.

Id tokens are sent as JWT through gRPC metadata and then validated server side to identify a user.

Clients that want direct access to Feast APIs without using the Feast Python SDK must generate and sign their own gRPC calls.

Server (Feast Core)

Authentication
If authentication is enabled, then all incoming requests to Feast must have a Google OIDC identity token. This token is only used to identify the user uniquely, not for authorization purposes.
This can be done within Feast Core without an external API call, using JWT signed by public keys.

Authentication will be implemented at the interceptor level. We will create an authentication interface that must be extended. If the Feast administrator enables authentication then they should also configure a specific implementation to use for their authentication interceptor. We will start with a Google OIDC Authentication interceptor.

If authentication is disabled then no interceptor will be initialized. No authentication is done on the HTTP level for now. We will defer that until the full RFC for auth.

Authorization
Authorization will also be implemented at the interceptor level. Authorization happens directly after authentication inside the same interceptor.

We will create an authorization interface that must be extended. If the administrator enables Authorization they should also configure a specific implementation to use for their interceptor.

The first implementation of this interceptor will be a KetoAuthorizationInterceptor.

Out of scope

HTTP level auth
Member management through AddMember() / RemoveMember(): This will be deferred and can be handled at the IAM system for now.
Plugin interface like OSGi (but we will start with interfaces)

Pros

Supports existing authentication methods that most users have available (Google OIDC).
Interface based approach is a lot cleaner and doesn't add more complexity to the code base.
Interface based approach means it should be easy to extend to different use cases.
No changes to the Feast data model
Easy to extend to a pluggable interface in the future

Cons

Implementations might be use case specific. Does not include OSGi/ServiceLoader.
Does not leverage spring security
Does add some complexity to the code base instead of externalizing it (proxy level auth).

References

@woop

@woop @ches we're polishing this off to try and make 0.6. Many of the changes are already done but there is one more that I would like to propose. Unfortunately we don't use Keto or any off the shelf Authorization provider. We have a custom homegrown REST auth provider. In order to avoid forking to implement our own AuthorizationProvider, I'd like to suggest a widening of the interface to make it easier to provide an escape hatch for people that want to write an http bridge to a custom provider:

  /**
   * Validates whether a user has any of the specified role on the project
   * 
   *
   * @param project Name of the Feast project
   * @param authentication Spring Security Authentication object
   * @param roles ProjectRole that the user needs
   * @return AuthorizationResult result of authorization query
   */
  AuthorizationResult checkAccess(String project, Authentication authentication, ProjectRole... roles);

Return a result object instead of throwing an exception - I generally avoid using exceptions for control flow.
Rename method to broad scope to checkAccess
Add ProjectRole enum with ADMIN and MEMBER roles
Provide an HttpAuthorizationProvider that will have a OpenApi definition anyone can implement

Reasons to hate this:

It's not future proof even if it's more extensible
I didn't look if there is standard we could implement for RESTful authorization. It's been awhile but generally haven't found much in the past.
Keto is great, we could just implement a Keto -> custom provider bridge. This seemed like a huge undertaking and not sure how our concepts map to Keto. This keeps the surface area small.

@dr3s

Hi @dr3s,

The proposed API looks mostly better.

I like the idea of an HTTP bridge as an escape hatching. Might actually be cleaner than having this Keto implementation as well.

Add ProjectRole enum with ADMIN and MEMBER roles

How does this differ from having everyone inside a project have write access and all guests having read access? Put differently, can we avoid creating roles for now?

It's not future proof even if it's more extensible

Which aspects wouldn't be future proof?

I didn't look if there is standard we could implement for RESTful authorization. It's been awhile but generally haven't found much in the past.

I think you will land on something close to Keto or Keycloak. There would have to be wrapping logic ontop of those APIs. Either inside Feast or in a separate bridge connector.

I don't want to rush you for 0.6, but I was hoping to cut by the end of next week. 0.6 was supposed to follow hot on the heels of 0.5.

Do you think that timeline is still realistic?

I think I'll have to give up on the bridge in this first PR in order to make next week. I can also drop the roles. I actually think it's better to pass in actions instead of roles and let the implementor decide how to map from write or read actions to roles. I'm definitely going to need no access, guest or read only access, write access, and probably some type of admin too. Probably better to add it later in another PR. I'll open another PR as soon as I can.

…

On Wed, Jun 10, 2020, 10:52 PM Willem Pienaar ***@***.***> wrote: Hi @dr3s <https://github.com/dr3s>, I like the idea of an HTTP bridge as an escape hatching. Might actually be cleaner than having this Keto implementation as well. Add ProjectRole enum with ADMIN and MEMBER roles How does this differ from having everyone inside a project have write access and all guests having read access? Put differently, can we avoid creating roles for now? It's not future proof even if it's more extensible Which aspects wouldn't be future proof? I didn't look if there is standard we could implement for RESTful authorization. It's been awhile but generally haven't found much in the past. I think you will land on something close to Keto or Keycloak. There would have to be wrapping logic ontop of those APIs. Either inside Feast or in a separate bridge connector. I don't want to rush you for 0.6, but I was hoping to cut by the end of next week. 0.6 was supposed to follow hot on the heals of 0.5. Do you think that is realistic? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#504 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABDNW7WQI4CQJHTGXX6K2DRWBBFDANCNFSM4K6OVBPQ> .

I'm definitely going to need no access, guest or read only access, write access, and probably some type of admin too. Probably better to add it later in another PR.

Yea I think these are all reasonable requests. The original RFC that auth was presented in was the Projects RFC, which kind of shoe-horned the functionality in. Auth deserves to be fleshed out fully, so maybe we should try and limit the functionality to the absolute MVP, and try and spec this out properly for 0.7?

Sounds good to me.

…

On Thu, Jun 11, 2020, 12:37 AM Willem Pienaar ***@***.***> wrote: I'm definitely going to need no access, guest or read only access, write access, and probably some type of admin too. Probably better to add it later in another PR. Yea I think these are all reasonable requests. The original RFC that auth was presented in was the Projects RFC, which kind of shoe-horned the functionality in. Auth deserves to be fleshed out fully, so maybe we should try and limit the functionality to the absolute MVP, and try and spec this out properly for 0.7? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#504 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABDNW6QOY5TEYTIKBFVVO3RWBNRRANCNFSM4K6OVBPQ> .

We can track the issue here but I think there is a gap in the design for auth where serving connects to core. It seems core is not really even authenticated until authorization is enabled, which then breaks serving when it tries to get feature set info.

I'm not sure why serving needs to pre-fetch serving info outside a user request but I think we need to enable auth properly in core even when authorization is disabled and then pass the auth token from serving to core. If pre-fetch is needed, we probably need support for fetching a JWT via oauth2 using preconfigured client credentials.

@jmelinav

We can track the issue here but I think there is a gap in the design for auth where serving connects to core. It seems core is not really even authenticated until authorization is enabled, which then breaks serving when it tries to get feature set info.

Yea I realized that as well. It doesn't require much, I believe we just need to change the conditional here https://github.com/feast-dev/feast/blob/master/auth/src/main/java/feast/auth/config/SecurityConfig.java

I'm not sure why serving needs to pre-fetch serving info outside a user request

For latency reasons this fetch happens async to incoming requests for features. Another way we could do this is to use the online store for persisting the information, but at the moment there is still communication to Feast Core to register the store in the first place (this allows Core to start jobs), and to retrieve feature sets.

but I think we need to enable auth properly in core even when authorization is disabled and then pass the auth token from serving to core. If pre-fetch is needed, we probably need support for fetching a JWT via oauth2 using preconfigured client credentials.

You are probably aware of this, but @jmelinav has submitted a PR to add that support.

I am currently refactoring auth a bit (not changing any functionality), and trying to add a few integration tests. I don't really feel confident with the tests we have right now. I dont mind reviewing and even getting your PRs merged in as they become ready, but I want to get proper tests out as part of #860 before we can "officially" support auth.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Navigation Menu

Repository navigation

Authentication and authorization #504

1. Prior Art: Auth as proposed in the Feast Projects RFC

2. Objectives

3. Requirements (minimal release)

3.1 User stories

3.2 In Scope

3.3 Out of Scope

4. Open questions

Proposal 1: Basic auth built into Feast

Proposal 2: Keycloak/Ory + gRPC metadata

Proposal 3: Assume auth is outside of Feast (upstream)

Proposal 4: Authentication Interceptor with OSGi/ServiceLoader

Eliminated ideas

Feast Auth (Minimal): Authentication through OpenID Connect and an Authorization interface

Overview

High level sequence

Clients (Users)

Server (Feast Core)

Out of scope

Pros

Cons

References

2 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Authentication and authorization #504

Description

1. Prior Art: Auth as proposed in the Feast Projects RFC

2. Objectives

3. Requirements (minimal release)

3.1 User stories

3.2 In Scope

3.3 Out of Scope

4. Open questions

Activity

ches commented on Feb 29, 2020

woop commented on Feb 29, 2020

woop commented on Feb 29, 2020

woop commented on Mar 1, 2020

Proposal 1: Basic auth built into Feast

Proposal 2: Keycloak/Ory + gRPC metadata

Proposal 3: Assume auth is outside of Feast (upstream)

Proposal 4: Authentication Interceptor with OSGi/ServiceLoader

Eliminated ideas

davidheryanto commented on Mar 2, 2020

zhilingc commented on Mar 2, 2020

woop commented on Mar 2, 2020

ches commented on Mar 5, 2020

ches commented on Mar 5, 2020

woop commented on Mar 5, 2020

woop commented on Mar 5, 2020

Feast Auth (Minimal): Authentication through OpenID Connect and an Authorization interface

Overview

High level sequence

Clients (Users)

Server (Feast Core)

Out of scope

Pros

Cons

References

2 remaining items

dr3s commented on Jun 11, 2020

woop commented on Jun 11, 2020

dr3s commented on Jun 11, 2020

woop commented on Jun 11, 2020

dr3s commented on Jun 11, 2020

dr3s commented on Jul 10, 2020

woop commented on Jul 11, 2020

stale commented on Sep 9, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions