Skip to content

Instantly share code, notes, and snippets.

@pimlock
Last active September 19, 2026 00:24
Show Gist options
  • Select an option

  • Save pimlock/0632dcc2dcce84835b89bfcc9fabb7f5 to your computer and use it in GitHub Desktop.

Select an option

Save pimlock/0632dcc2dcce84835b89bfcc9fabb7f5 to your computer and use it in GitHub Desktop.
OpenShell policy architecture: move public-to-internal translation to the gateway

OpenShell policy architecture: move translation to the gateway

This document describes the policy boundary introduced by #3401 and the intended end state of #3388.

The central change is simple:

Before: client parses authored YAML into a handwritten DTO,
        then converts that DTO into the internal runtime protobuf.

After:  client parses authored YAML into the public authored protobuf,
        sends it unchanged through the public API,
        and the gateway converts it into the internal runtime protobuf.

The public and internal messages are deliberately different. The public message contains author intent. The internal message contains normalized and derived state used for storage, composition, supervisor delivery, and policy enforcement.

Current architecture

flowchart LR
    YAML[Policy YAML] --> DTO[Handwritten Rust policy DTO]
    DTO --> CLIENT_CONVERT[Client-side conversion]
    CLIENT_CONVERT --> INTERNAL[openshell.sandbox.v1 SandboxPolicy]
    INTERNAL --> API[Gateway API]
    API --> STORE[(Database)]
    API --> COMPOSE[Provider and global composition]
    COMPOSE --> SUPERVISOR[Supervisor]

    classDef trust fill:#ffe6e6,stroke:#aa3333,color:#111;
    class INTERNAL trust;
Loading

The client must understand both the human-authored format and the gateway's runtime model. The same internal SandboxPolicy crosses every boundary:

  • client to gateway
  • gateway storage
  • policy composition
  • compute-driver and supervisor delivery
  • enforcement

That coupling causes several problems:

  • The Rust client owns a handwritten YAML model and conversion logic.
  • Other SDKs must recreate that model if they want to parse policy files.
  • Runtime-only fields appear in generated client types even though users must never set them.
  • A runtime representation change can become a public API change.
  • Every new authored field requires coordinated edits across the YAML DTO, client conversion, API message, storage, and runtime code.

Proposed architecture

flowchart LR
    subgraph Client[CLI or SDK]
        YAML[Policy YAML]
        YAML_BRIDGE[Bounded YAML to JSON-compatible values]
        PROTOJSON[Strict ProtoJSON decoding]
        PUBLIC[openshell.policy.v1 SandboxPolicy]
        SDK_OBJECT[Programmatically constructed generated message]

        YAML --> YAML_BRIDGE --> PROTOJSON --> PUBLIC
        SDK_OBJECT --> PUBLIC
    end

    PUBLIC -->|gRPC API| INGRESS[Gateway authored-policy boundary]

    subgraph Gateway[Gateway]
        INGRESS --> VALIDATE[Intrinsic and request-context validation]
        VALIDATE --> LOWER[Lower and normalize]
        LOWER --> INTERNAL[openshell.sandbox.v1 SandboxPolicy]
        INTERNAL --> STORE[(Internal durable storage)]
        INTERNAL --> COMPOSE[Provider and global composition]
        COMPOSE --> EFFECTIVE[Internal effective policy]
    end

    EFFECTIVE -->|internal protocol| SUPERVISOR[Supervisor and enforcement]
    INTERNAL --> BASE_PROJECT[Project base policy]
    EFFECTIVE --> EFFECTIVE_PROJECT[Project read-only effective view]
    BASE_PROJECT --> PUBLIC_RESPONSE[Public policy response]
    EFFECTIVE_PROJECT --> PUBLIC_RESPONSE

    classDef public fill:#e7f3ff,stroke:#2563a8,color:#111;
    classDef internal fill:#fff1cc,stroke:#9a6b00,color:#111;
    class PUBLIC,SDK_OBJECT,PUBLIC_RESPONSE public;
    class INTERNAL,EFFECTIVE,STORE internal;
Loading

There are two client entry paths:

  1. A file-oriented client parses YAML into the generated public message.
  2. An SDK user constructs that same generated message directly.

Both send openshell.policy.v1.SandboxPolicy through the API. Neither client constructs the internal runtime policy.

At API ingress, the gateway validates the public message and lowers it into openshell.sandbox.v1.SandboxPolicy. All existing policy composition, persistence, supervisor delivery, and enforcement code can continue using the internal representation.

Public and internal responsibilities

Concern Public authored policy Internal runtime policy
Package openshell.policy.v1 openshell.sandbox.v1
Used by CLI, Go, Python, TypeScript, external API Gateway, storage, drivers, supervisor, enforcement
Represents User-controlled intent Normalized base or effective policy
YAML shape Yes, through strict ProtoJSON-shaped YAML No public file contract
Runtime-derived fields Excluded Allowed
Provider-composed rules Only in a read-only effective projection Stored and enforced
Validation Portable intrinsic rules plus gateway validation Domain, composition, and runtime checks
Compatibility promise Public API and configuration contract Internal wire and persistence contract

Runtime-derived authority stays internal. Examples include:

advisor_proposed
provider_credentialed

The public network rule keeps repeated NetworkBinary binaries, where each entry contains path. Omitted and empty lists canonicalize to the same state and match no binaries. The authored schema has no any-binary marker or wildcard binary state.

Those names and their internal field numbers are reserved in the public proto. Generated clients cannot set them, and the gateway does not need to remember to clear user-supplied values.

Request flow

policy YAML
  -> bounded YAML parser
  -> JSON-compatible generic value
  -> strict ProtoJSON decoder
  -> openshell.policy.v1.SandboxPolicy
  -> public gRPC request
  -> gateway intrinsic validation
  -> checked lowering and normalization
  -> openshell.sandbox.v1.SandboxPolicy base policy
  -> provider and global policy composition
  -> openshell.sandbox.v1.SandboxPolicy effective policy
  -> supervisor and enforcement

The gateway is authoritative. Client-side validation improves error messages, but it cannot replace validation after API ingress.

Lowering owns transformations such as:

  • public ports into the runtime port representation
  • public binary messages into the runtime binary representation
  • public nested MCP settings into runtime fields
  • policy-name fallback from a surrounding map key
  • default materialization
  • removal or initialization of internal-only state

Handlers should call one shared lowering function. They should not implement field-by-field conversion independently.

Response flow

Responses travel in the opposite direction:

internal base policy
  -> base projection
  -> canonical public authored policy
internal effective policy
  -> effective read-only projection
  -> public policy view without internal authority fields

The distinction matters. A base-policy response represents stored user intent and can support a semantic get-edit-set round trip. An effective-policy view may contain provider or global rules. It is useful for inspection but must not be presented as an authorable base policy unless the API deliberately supports that operation.

The intended base-policy invariant is:

project_base(lower(public_policy)) == canonicalize(public_policy)

The reverse invariant applies to valid internal base policies after normalization:

lower(project_base(internal_base_policy)) == canonicalize(internal_base_policy)

These are semantic invariants. Comments, YAML formatting, key order, quoting, anchors, aliases, and shorthand spelling are not preserved.

SDK behavior

The public proto gives every SDK the same message graph, field names, presence rules, and generated API types:

Rust       openshell.policy.v1.SandboxPolicy
Go         openshell.policy.v1.SandboxPolicy
Python     openshell.policy.v1.SandboxPolicy
TypeScript openshell.policy.v1.SandboxPolicy

Protobuf itself does not parse YAML. An SDK that advertises policy-file support still needs a small language-specific front end:

YAML library
  -> bounded JSON-compatible value
  -> strict ProtoJSON decoding using the shared descriptor
  -> generated public policy message

That front end handles YAML safety rather than policy semantics. It should reject duplicate keys, merge keys, unsupported tags, non-string map keys, multiple documents, unsafe aliases, typed-field nulls, and resource-limit violations. Middleware config remains the deliberate JSON-shaped exception.

SDKs that only accept generated objects or canonical JSON do not need a YAML dependency.

Cross-language fixtures should compare normalized public messages and stable validation rule identifiers. Exact parser prose does not need to match across languages.

Basic portable constraints live in buf.validate annotations and run through Protovalidate in each SDK that offers local validation. This includes field ranges, collection limits, required matcher choices, and fixed vocabularies. Generated messages do not validate themselves, and the gateway always validates again after API ingress.

Protovalidate officially supports Go, Python, and JavaScript/TypeScript, but not Rust. The Rust gateway therefore needs a vetted Prost-compatible runtime or equivalent checks while the annotations remain the cross-language source of truth. Domain and context-dependent validation stays in openshell-policy and openshell-server.

Storage and internal protocols

The gateway continues storing the internal normalized protobuf. Existing database rows therefore need no policy payload migration.

Internal protocols also keep the runtime message:

  • gateway to compute driver
  • gateway to supervisor
  • supervisor policy loading
  • effective-policy enforcement

The public schema can now evolve around author usability without forcing the runtime representation to match it. The runtime schema can add derived state without exposing new user-controlled authority.

Policy identity remains based on the normalized internal policy unless the API introduces a separate authored-policy identity.

Ownership

Component Responsibility
proto/policy.proto Public, author-controlled policy schema and Protovalidate annotations
openshell-policy-schema Generated Rust public messages, bounded YAML and ProtoJSON codec, canonical output, and portable validation integration
openshell-policy Public-to-internal lowering, internal-to-public projection, normalization, merge logic, domain validation
openshell-core and proto/sandbox.proto Internal runtime policy messages
openshell-server Authoritative API validation, context checks, composition, persistence, and response projection
Rust, Go, Python, and TypeScript SDKs Generated public policy types; optional YAML helpers

A temporary Rust DTO may remain behind openshell-policy-schema while validation moves to the public proto. It is an implementation detail, not a client or API contract. The target is to keep portable structural validation in the protobuf schema and reserve Rust domain code for checks that depend on policy composition or runtime state.

Provider profiles and partial policy operations

Provider profiles keep their own outer resource schema. They may reuse public authored endpoint and binary messages where the authoring semantics are the same. The gateway lowers those fragments before composing them into the internal effective policy.

Policy merge requests, advisor proposals, revision records returned by the API, and draft edits should also use public authored messages or fragments. Their handlers lower those fragments through the same shared boundary before calling internal merge code.

Internal storage records for provider profiles, revisions, and advisor state may continue embedding internal policy messages. Storage conversion belongs at the persistence boundary, not in SDKs.

What this buys us

  • Internal authority fields never appear in author-facing generated types.
  • Rust, Go, Python, and TypeScript share one policy schema.
  • SDK users can construct typed policies without learning the runtime model.
  • Policy YAML and public API objects describe the same authored message.
  • The gateway owns one conversion boundary instead of trusting every client to reproduce internal normalization.
  • Runtime storage and supervisor protocols can evolve independently from the authoring contract.
  • Existing internal stored policies remain readable.

The cost is one explicit public-to-internal adapter and its reverse projection. That adapter is valuable code: it is the trust boundary between user intent and runtime authority.

Related material

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment