This document describes the policy boundary introduced by #3401 and the intended end state of #3388.
The central change is simple:
Before: client parses authored YAML into a handwritten DTO,
then converts that DTO into the internal runtime protobuf.
After: client parses authored YAML into the public authored protobuf,
sends it unchanged through the public API,
and the gateway converts it into the internal runtime protobuf.
The public and internal messages are deliberately different. The public message contains author intent. The internal message contains normalized and derived state used for storage, composition, supervisor delivery, and policy enforcement.
flowchart LR
YAML[Policy YAML] --> DTO[Handwritten Rust policy DTO]
DTO --> CLIENT_CONVERT[Client-side conversion]
CLIENT_CONVERT --> INTERNAL[openshell.sandbox.v1 SandboxPolicy]
INTERNAL --> API[Gateway API]
API --> STORE[(Database)]
API --> COMPOSE[Provider and global composition]
COMPOSE --> SUPERVISOR[Supervisor]
classDef trust fill:#ffe6e6,stroke:#aa3333,color:#111;
class INTERNAL trust;
The client must understand both the human-authored format and the gateway's
runtime model. The same internal SandboxPolicy crosses every boundary:
- client to gateway
- gateway storage
- policy composition
- compute-driver and supervisor delivery
- enforcement
That coupling causes several problems:
- The Rust client owns a handwritten YAML model and conversion logic.
- Other SDKs must recreate that model if they want to parse policy files.
- Runtime-only fields appear in generated client types even though users must never set them.
- A runtime representation change can become a public API change.
- Every new authored field requires coordinated edits across the YAML DTO, client conversion, API message, storage, and runtime code.
flowchart LR
subgraph Client[CLI or SDK]
YAML[Policy YAML]
YAML_BRIDGE[Bounded YAML to JSON-compatible values]
PROTOJSON[Strict ProtoJSON decoding]
PUBLIC[openshell.policy.v1 SandboxPolicy]
SDK_OBJECT[Programmatically constructed generated message]
YAML --> YAML_BRIDGE --> PROTOJSON --> PUBLIC
SDK_OBJECT --> PUBLIC
end
PUBLIC -->|gRPC API| INGRESS[Gateway authored-policy boundary]
subgraph Gateway[Gateway]
INGRESS --> VALIDATE[Intrinsic and request-context validation]
VALIDATE --> LOWER[Lower and normalize]
LOWER --> INTERNAL[openshell.sandbox.v1 SandboxPolicy]
INTERNAL --> STORE[(Internal durable storage)]
INTERNAL --> COMPOSE[Provider and global composition]
COMPOSE --> EFFECTIVE[Internal effective policy]
end
EFFECTIVE -->|internal protocol| SUPERVISOR[Supervisor and enforcement]
INTERNAL --> BASE_PROJECT[Project base policy]
EFFECTIVE --> EFFECTIVE_PROJECT[Project read-only effective view]
BASE_PROJECT --> PUBLIC_RESPONSE[Public policy response]
EFFECTIVE_PROJECT --> PUBLIC_RESPONSE
classDef public fill:#e7f3ff,stroke:#2563a8,color:#111;
classDef internal fill:#fff1cc,stroke:#9a6b00,color:#111;
class PUBLIC,SDK_OBJECT,PUBLIC_RESPONSE public;
class INTERNAL,EFFECTIVE,STORE internal;
There are two client entry paths:
- A file-oriented client parses YAML into the generated public message.
- An SDK user constructs that same generated message directly.
Both send openshell.policy.v1.SandboxPolicy through the API. Neither client
constructs the internal runtime policy.
At API ingress, the gateway validates the public message and lowers it into
openshell.sandbox.v1.SandboxPolicy. All existing policy composition,
persistence, supervisor delivery, and enforcement code can continue using the
internal representation.
| Concern | Public authored policy | Internal runtime policy |
|---|---|---|
| Package | openshell.policy.v1 |
openshell.sandbox.v1 |
| Used by | CLI, Go, Python, TypeScript, external API | Gateway, storage, drivers, supervisor, enforcement |
| Represents | User-controlled intent | Normalized base or effective policy |
| YAML shape | Yes, through strict ProtoJSON-shaped YAML | No public file contract |
| Runtime-derived fields | Excluded | Allowed |
| Provider-composed rules | Only in a read-only effective projection | Stored and enforced |
| Validation | Portable intrinsic rules plus gateway validation | Domain, composition, and runtime checks |
| Compatibility promise | Public API and configuration contract | Internal wire and persistence contract |
Runtime-derived authority stays internal. Examples include:
advisor_proposed
provider_credentialed
The public network rule keeps repeated NetworkBinary binaries, where each
entry contains path. Omitted and empty lists canonicalize to the same state
and match no binaries. The authored schema has no any-binary marker or wildcard
binary state.
Those names and their internal field numbers are reserved in the public proto. Generated clients cannot set them, and the gateway does not need to remember to clear user-supplied values.
policy YAML
-> bounded YAML parser
-> JSON-compatible generic value
-> strict ProtoJSON decoder
-> openshell.policy.v1.SandboxPolicy
-> public gRPC request
-> gateway intrinsic validation
-> checked lowering and normalization
-> openshell.sandbox.v1.SandboxPolicy base policy
-> provider and global policy composition
-> openshell.sandbox.v1.SandboxPolicy effective policy
-> supervisor and enforcement
The gateway is authoritative. Client-side validation improves error messages, but it cannot replace validation after API ingress.
Lowering owns transformations such as:
- public
portsinto the runtime port representation - public binary messages into the runtime binary representation
- public nested MCP settings into runtime fields
- policy-name fallback from a surrounding map key
- default materialization
- removal or initialization of internal-only state
Handlers should call one shared lowering function. They should not implement field-by-field conversion independently.
Responses travel in the opposite direction:
internal base policy
-> base projection
-> canonical public authored policy
internal effective policy
-> effective read-only projection
-> public policy view without internal authority fields
The distinction matters. A base-policy response represents stored user intent and can support a semantic get-edit-set round trip. An effective-policy view may contain provider or global rules. It is useful for inspection but must not be presented as an authorable base policy unless the API deliberately supports that operation.
The intended base-policy invariant is:
project_base(lower(public_policy)) == canonicalize(public_policy)
The reverse invariant applies to valid internal base policies after normalization:
lower(project_base(internal_base_policy)) == canonicalize(internal_base_policy)
These are semantic invariants. Comments, YAML formatting, key order, quoting, anchors, aliases, and shorthand spelling are not preserved.
The public proto gives every SDK the same message graph, field names, presence rules, and generated API types:
Rust openshell.policy.v1.SandboxPolicy
Go openshell.policy.v1.SandboxPolicy
Python openshell.policy.v1.SandboxPolicy
TypeScript openshell.policy.v1.SandboxPolicy
Protobuf itself does not parse YAML. An SDK that advertises policy-file support still needs a small language-specific front end:
YAML library
-> bounded JSON-compatible value
-> strict ProtoJSON decoding using the shared descriptor
-> generated public policy message
That front end handles YAML safety rather than policy semantics. It should
reject duplicate keys, merge keys, unsupported tags, non-string map keys,
multiple documents, unsafe aliases, typed-field nulls, and resource-limit
violations. Middleware config remains the deliberate JSON-shaped exception.
SDKs that only accept generated objects or canonical JSON do not need a YAML dependency.
Cross-language fixtures should compare normalized public messages and stable validation rule identifiers. Exact parser prose does not need to match across languages.
Basic portable constraints live in buf.validate annotations and run through
Protovalidate in each SDK that offers local validation. This includes field
ranges, collection limits, required matcher choices, and fixed vocabularies.
Generated messages do not validate themselves, and the gateway always validates
again after API ingress.
Protovalidate officially supports Go, Python, and JavaScript/TypeScript, but not
Rust. The Rust gateway therefore needs a vetted Prost-compatible runtime or
equivalent checks while the annotations remain the cross-language source of
truth. Domain and context-dependent validation stays in openshell-policy and
openshell-server.
The gateway continues storing the internal normalized protobuf. Existing database rows therefore need no policy payload migration.
Internal protocols also keep the runtime message:
- gateway to compute driver
- gateway to supervisor
- supervisor policy loading
- effective-policy enforcement
The public schema can now evolve around author usability without forcing the runtime representation to match it. The runtime schema can add derived state without exposing new user-controlled authority.
Policy identity remains based on the normalized internal policy unless the API introduces a separate authored-policy identity.
| Component | Responsibility |
|---|---|
proto/policy.proto |
Public, author-controlled policy schema and Protovalidate annotations |
openshell-policy-schema |
Generated Rust public messages, bounded YAML and ProtoJSON codec, canonical output, and portable validation integration |
openshell-policy |
Public-to-internal lowering, internal-to-public projection, normalization, merge logic, domain validation |
openshell-core and proto/sandbox.proto |
Internal runtime policy messages |
openshell-server |
Authoritative API validation, context checks, composition, persistence, and response projection |
| Rust, Go, Python, and TypeScript SDKs | Generated public policy types; optional YAML helpers |
A temporary Rust DTO may remain behind openshell-policy-schema while
validation moves to the public proto. It is an implementation detail, not a
client or API contract. The target is to keep portable structural validation
in the protobuf schema and reserve Rust domain code for checks that depend on
policy composition or runtime state.
Provider profiles keep their own outer resource schema. They may reuse public authored endpoint and binary messages where the authoring semantics are the same. The gateway lowers those fragments before composing them into the internal effective policy.
Policy merge requests, advisor proposals, revision records returned by the API, and draft edits should also use public authored messages or fragments. Their handlers lower those fragments through the same shared boundary before calling internal merge code.
Internal storage records for provider profiles, revisions, and advisor state may continue embedding internal policy messages. Storage conversion belongs at the persistence boundary, not in SDKs.
- Internal authority fields never appear in author-facing generated types.
- Rust, Go, Python, and TypeScript share one policy schema.
- SDK users can construct typed policies without learning the runtime model.
- Policy YAML and public API objects describe the same authored message.
- The gateway owns one conversion boundary instead of trusting every client to reproduce internal normalization.
- Runtime storage and supervisor protocols can evolve independently from the authoring contract.
- Existing internal stored policies remain readable.
The cost is one explicit public-to-internal adapter and its reverse projection. That adapter is valuable code: it is the trust boundary between user intent and runtime authority.