Needed (eventually, soon?): Feature gates #1690

As we drive towards and then past GA, we will need a way to segregate unfinished / in-progress features and to get feedback on them -- we will need feature gates or something like it.

On API versioning:

We probably do not want to make our RPC service explicitly “v1”, though we could later add a “v2” if we need to. We would almost certainly need to guarantee round-trip across versions, so “v2” would only be if we wanted to totally re-organize some resource schemas. This is a tomorrow problem

How little can we get away with?

A lesson from k8s: “Alpha” / “Beta” / “GA” is a lie -- we should just do “preview” and “GA”. Preview things need to be off by default but enableable.

Can we get away with a single “enable preview” option which enables everything? Maybe just for a while?

How to implement this?

For gate-managed API fields:

Option1) All API ops go through the current RPC service.

Treat gate-disabled fields as unknown and reject any such operation. This could be open-coded or driven through metadata like DV. It would have to be considered in the defaulting logic, too (so we don’t assign defaults to disabled fields and then fail the op).

When using gate-enabled preview fields we may want a --allow-preview flag on the CLI and a similar header to the API server to explicitly opt-in to using preview features. Without opt-in, using preview fields would fail and doing a RMW cycle on an object which has preview fields would allow them to remain, as long as they did not change.

Option 2) Add a new RPC service which is explicitly the “preview” API. It would be the GA schema + preview fields.

Treat gate-disabled fields as unknown and reject any such operation. This could be open-coded or driven through metadata like DV. It would have to be considered in the defaulting logic, too (so we don’t assign defaults to disabled fields and then fail the op).

To use gate-enabled preview features you have to use the preview RPC service. This gives us an explicit opt-in. We would probably couple this with a --allow-preview flag on the CLI. Reading a resource through the “regular” RPCs would show preview fields but not allow changes to them. Doing a RMW cycle on an object which has preview fields would allow them to remain, as long as they did not change.

For gate-managed features without APIs:

We could either pass flags to all components like k8s or we could store it behind an API. We need a GetServerInfo() sort of RPC anyway.

Disabling a feature should neutralize any impact it has, as much as possible (e.g. bypass potentially buggy code).

To be decided:

• Do we need fine-grained gates?
• Do we require all features to start in preview, just API changes, or decide-as-we-go?
• Do we need something like API compat-versions?
• Can we steal more ideas and/or code from k8s? It's not a perfect fit code-wise but let's not make the same mistakes.

Julian Gutierrez Oschmann (@juli4n) Eitan Yarmush (@EItanya) Joe Betz (@jpbetz) Anna Pendleton (@annapendleton)