objectstore plugin: what an out-of-tree provider needs, with and without per-actor credentials

Cloud providers other than GCS should be able to ship their own object-store plugin from their own repository. This issue lists what works today and what is missing.

Without per-actor credentials: works today

A provider can build a plugin out of tree now, as long as it transfers with its own credentials.

The plugin API, objectstore.v1, is public in pkg/proto/objectstorepb/v1. A plugin serves NodeProvider and AssetProvider beside atelet, and ControlProvider beside ate-api-server, over a Unix socket.
atelet sends actor_jwt with every snapshot transfer. A plugin that does not authorize per actor ignores it. Our S3 backend does the same today.

The gap is installation. The manifests name our image, ko://go.ate.dev/substrate/cmd/objectstore-plugin, and its flags in manifests/ate-install/atelet.yaml and ate-api-server.yaml, and ate-setup has no setting for another plugin image. A provider has to patch those manifests.

With per-actor credentials: needs packages moved to `pkg/`

GCS scopes each transfer to the actor's own snapshots. A provider that wants the same writes its own exchange and issuer for its cloud, but it should reuse the checks that decide what an actor may access. Those are under internal/, so a binary outside this repository cannot import them.

These would move to pkg/:

Package	What it does
`internal/oidcjwt`	verifies the actor JWT against the `jwks` file
`internal/objectstoreplugin/actorlookup`	fetches the actor from ate-api-server, refuses a recreated one
`internal/objectstoreplugin/actorscope`	turns the actor's status into grants, read-write on its own prefix, read-only on its tag
checks in `internal/objectstoreplugin/exchange`	refuse a token that is not the worker-for-actor kind, refuse empty identity claims, and refuse a token whose node is not the actor's `assignedNode` or an actor outside `resources.TransferStates`. Every provider must apply these, so they belong beside `actorscope`
`ActorJWTAudience`, in `internal/objectstoreplugin/exchange`	the audience atelet mints every worker-for-actor JWT with, which every provider's verifier must accept

A package under pkg/ cannot import internal/, so these come along:

internal/actoridjwt, the claim types. oidcjwt and actorlookup import it.
internal/ateapiauth, the mTLS dial options actorlookup uses, with its imports internal/credbundle and internal/k8sresolver.
The part of internal/resources that actorscope and the checks use: the snapshot URI parsing, StoragePrefix, the actor prefix helper, and TransferStates.

None of these imports a cloud SDK or anything GCS-specific.

The manifest wiring for our exchange also applies to a provider's: the network policy that lets atelet reach port 8445 on ate-api-server, and the serving certificate, client certificate and jwks mounts on the control sidecar. A provider's sidecar needs the same, through the same manifest patches as above.

Stays ours

Our exchange proto (internal/proto/objectstoreexchangepb), the GCS issuer, the access boundary code and the node-side token cache are specific to GCS. A provider writes its own for its cloud.

ILLUM1N0X

One correction first: a package under pkg/ can import internal/. Go's internal rule checks the importer's path, so anything under the module root, pkg/... included, may import go.ate.dev/substrate/internal/..., and an external module can import that pkg/ package. I built a probe from an external module to confirm it. So nothing in the table has to move, and oidcjwt, actorlookup, ateapiauth, credbundle, k8sresolver and part of resources don't become public API.

Instead I'd add one small public package, pkg/objectstore/actorauthz, that runs the same checks the GCS exchange runs today. A provider then writes only its issuer and wire format. Sketch:

package actorauthz

// Audience is what atelet mints every worker-for-actor JWT with.
// exchange.ActorJWTAudience becomes an alias; atelet imports this.
const Audience = "objectstore-plugin.ate.dev"

type Config struct {
	Issuer    string
	JWKSFile  string
	APIServer APIServerConfig
}

// APIServerConfig says how to reach ate-api-server for GetActor.
type APIServerConfig struct {
	Address          string
	CAFile           string
	ServerName       string
	ClientCredBundle string
}

// New reads the key set once, so a bad path fails at start.
func New(ctx context.Context, cfg Config) (*Authorizer, error)
func (a *Authorizer) Close() error

// Authorize errors are gRPC status codes, mapped as the GCS exchange maps them.
func (a *Authorizer) Authorize(ctx context.Context, jwt string) (*Decision, error)

type Decision struct {
	Atespace, ActorName, ActorUID, Node string
	Grants                              []Grant
	Expiry                              time.Time // the JWT's expiry
}

type Access int

const (
	ReadOnly Access = iota + 1
	ReadWrite
)

// Prefix is the storage prefix URI, so no internal type leaks out.
type Grant struct {
	Prefix string
	Access Access
}

// Covers matches whole path segments after percent-decoding.
// ReadWrite grants cover ReadOnly requests.
func (d *Decision) Covers(uri string, access Access) bool

I'd land it in this order:

Extract the checks from exchange.Server into an internal actorauthz package, no behavior change, and have the GCS exchange call it. That's objectstore-plugin: move the credential exchange's actor checks into actorauthz #3138.
The pkg/ wrapper above.
The audience constant.
Document the provider exchange contract: the control sidecar runs in the ate-api-server pod for GetActor access, the JWKS mount, port 8445 and the NetworkPolicy from atelet.

The install side comes after: an ate-setup setting for another plugin's component, and moving the in-tree-only sidecar env and args out of the base manifests. The naming there should line up with #2378's storage settings.

ILLUM1N0X two questions. Is one wrapper package OK instead of moving the packages? And is a URI string the right public form for a grant's prefix, or would you rather see bucket and path exposed separately?

Answers to the above questions:

Yes, one wrapper works for me and keeps the public API small before GA. No pkg/ package imports internal/ today, so please call that out in the PR so maintainers can agree to it.
A URI string works for me, since it's the form snapshot locations take everywhere else (e.g. --snapshot-location, snapshot_uri). One note: The prefix has no trailing slash so a plain string prefix match also matches a different actor whose UID starts the same way. The GCS boundary adds the / for this reason.

Other notes:

For the audience constant, please move it to actorauthz.Audience and update atelet and the other callers in the same PR instead of keeping exchange.ActorJWTAudience as an alias. AGENTS.md asks us not to keep old names.
Regarding Covers: is this meant to replace the cache check in gcsissuer, or to check a single object URI? Ours compares whole grant sets by exact prefix, which works because both sides come from the same helper. If Azure needs the URI form, I'd like to understand what calls it before it goes into pkg/.

ILLUM1N0X

ILLUM1N0X I rebased #3138 on main and it now moves the audience to actorauthz.Audience, updates atelet, the objectstore plugin and the snapshotcredentials e2e suite, and deletes exchange.ActorJWTAudience. I applied the same rule to the exchange.Verifier and exchange.Lookup aliases it had added, so exchange.New takes the actorauthz types and the aliases are gone.

The rest is for the pkg/ PR. Grant.Prefix will end in /, built the same way the GCS boundary gets it (boundary.go#L38-L39, through objectstore.BucketPrefix), so a plain prefix match can't reach another actor whose UID starts the same way. That PR's description will also call out that it's the first pkg/ package importing internal/.

On Covers: it doesn't replace the cache check in gcsissuer, and the GCS exchange won't call it. That check asks whether a token minted for one grant set can serve another, and exact prefixes work there because both sets come from actorscope.FromActor. The GCS exchange never sees an object URI: the request carries only actor_jwt and force, and the access boundary rules do the scoping inside GCS.

It was meant for the Azure exchange, which isn't written yet. That exchange may need to check that one blob falls under one of the actor's grants before it signs a SAS for that blob. Nothing in the repo would call Covers today, so I'll leave it out of the pkg/ PR. If the Azure exchange needs it, I'll propose it together with that exchange.

Sounds good, thanks.
One more question for the pkg/ PR: what reads Decision.Expiry? The GCS exchange doesn't use the JWT's expiry

ILLUM1N0X

ILLUM1N0X nothing reads it, so the pkg/ PR leaves Expiry out. Following your #3138 review, its Decision will also drop Atespace, ActorName and Node and carry only ActorUID and Grants, and New builds the verifier from Audience itself.

objectstore-plugin: move the credential exchange's actor checks into …

Scoping the installation half of this issue. #3138 has merged. #3358 adds pkg/objectstore/actorauthz and is back in draft until the exchange moves to OIDC discovery in its own PR, which removes the jwks mount from the control sidecar. This covers the rest: how a provider gets its own plugin image into an install.

What a provider has to replace today

The image ko://go.ate.dev/substrate/cmd/objectstore-plugin, in two places: the objectstore-plugin init container in atelet.yaml (node mode) and in ate-api-server.yaml (control mode).
The args and env of both containers. ATE_STORAGE_BACKEND and the GCS/S3 settings are ours.
The startup probes, which exec /ko-app/objectstore-plugin healthcheck --socket=.... That path only exists in a ko-built image.

What a provider can keep as it is

The socket paths, /run/objectstore-plugin/{node,control}.sock, which atelet and ate-api-server dial.
The exchange port 8445, the ate-api-server Service port, and the ate-system network policy that admits atelet on it.
The servicedns and podidentity certificate mounts on the control sidecar, and the servicedns-ca trust bundle on the node sidecar.
None of these change if the provider's container keeps the name objectstore-plugin and the same mounts.

Proposal

Let ate-setup take a provider's kustomize component, e.g. a repeatable --component DIR, composed into every control plane apply after ours. ate-setup already composes components this way for --e2e-tuning, --fake-data-plane and others (Env.components in cmd/ate-setup/internal/steps/overlay.go). A provider ships one component that patches the image, args, env and probes of both objectstore-plugin containers. This is generic rather than an image-only flag, because a provider has to change args, env and probes too.
Document the sidecar contract providers can rely on: the container name, both sockets and the gRPC services on them, the healthcheck expectation, port 8445 and what calls it, and the mounts above. The rest of the manifests stay ours to change.
Add an example component under docs/ (or a test fixture) that swaps in a placeholder image, with a render test so the contract breaks loudly in our CI rather than in a provider's install.

Not proposing

A Helm chart or values file. The install is kustomize plus ate-setup today.
Moving the network policy or certificates into anything provider-specific. A provider reuses ours unchanged.

Open questions:

Does --image-repo pass a provider's non-ko:// image through untouched? Prebuilt.rewrite only matches ko:// references, so I expect so, but I haven't run it.
Should the probe be part of the contract (a healthcheck subcommand at a fixed path), or should the component replace it?

Happy to send 1–3 as one PR after #3358 if this shape works.

Verify actor JWTs in the credential exchange through OIDC discovery (a…

Verify actor JWTs in the credential exchange through OIDC discovery (#…

This is done?

Add pkg/objectstore/actorauthz so out-of-tree object-store providers …

Navigation Menu

Repository navigation

objectstore plugin: what an out-of-tree provider needs, with and without per-actor credentials #3095

Without per-actor credentials: works today

With per-actor credentials: needs packages moved to `pkg/`

Stays ours

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

objectstore plugin: what an out-of-tree provider needs, with and without per-actor credentials #3095

Description

Without per-actor credentials: works today

With per-actor credentials: needs packages moved to pkg/

Stays ours

Activity

aramase commented on Oct 22, 2026

ILLUM1N0X commented on Oct 23, 2026

aramase commented on Oct 23, 2026

ILLUM1N0X commented on Oct 23, 2026

aramase commented on Oct 23, 2026

aramase commented on Oct 29, 2026

BenTheElder commented on Nov 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

With per-actor credentials: needs packages moved to `pkg/`