Skip to content

Commit 3f6aabf

Browse files
committed
feat(gateway/sandbox): add global and sandbox runtime settings flow
Closes #405 Refactor the sandbox policy polling channel into an effective settings response with config revision tracking, global policy source metadata, and merged global/sandbox key resolution. Add gateway-global and sandbox-scoped setting mutations with per-key mutual exclusion, global delete unlock semantics, and global policy override behavior. Extend the CLI with settings get/set/delete and --global policy flows, then document the new control-plane behavior in architecture and user docs. Signed-off-by: John Myers <[email protected]>
1 parent de9dcaa commit 3f6aabf

11 files changed

Lines changed: 1194 additions & 117 deletions

File tree

‎architecture/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -234,7 +234,7 @@ The CLI is the primary way users interact with the platform. It provides command
234234

235235
- **Gateway management** (`openshell gateway`): Deploy, stop, destroy, and inspect clusters. Supports both local and remote (SSH) targets.
236236
- **Sandbox management** (`openshell sandbox`): Create sandboxes (with optional file upload and provider auto-discovery), connect to sandboxes via SSH, and delete sandboxes.
237-
- **Top-level commands**: `openshell status` (cluster health), `openshell logs` (sandbox logs), `openshell forward` (port forwarding), `openshell policy` (sandbox policy management).
237+
- **Top-level commands**: `openshell status` (cluster health), `openshell logs` (sandbox logs), `openshell forward` (port forwarding), `openshell policy` (sandbox policy management), `openshell settings` (effective sandbox settings and global/sandbox key updates).
238238
- **Provider management** (`openshell provider`): Create, update, list, and delete external service credentials.
239239
- **Inference management** (`openshell cluster inference`): Configure cluster-level inference by specifying a provider and model. The gateway resolves endpoint and credential details from the named provider record.
240240

‎architecture/gateway.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -231,7 +231,7 @@ These RPCs are called by sandbox pods at startup to bootstrap themselves.
231231

232232
| RPC | Description |
233233
|-----|-------------|
234-
| `GetSandboxPolicy` | Returns the `SandboxPolicy` from a sandbox's spec, looked up by sandbox ID. |
234+
| `GetSandboxPolicy` | Returns effective sandbox config looked up by sandbox ID: policy payload, policy metadata, and effective settings. Global settings override sandbox-level values per key. |
235235
| `GetSandboxProviderEnvironment` | Resolves provider credentials into environment variables for a sandbox. Iterates the sandbox's `spec.providers` list, fetches each `Provider`, and collects credential key-value pairs. First provider wins on duplicate keys. Skips credential keys that do not match `^[A-Za-z_][A-Za-z0-9_]*$`. |
236236

237237
#### Policy Recommendation (Network Rules)

‎architecture/sandbox.md‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -347,11 +347,11 @@ sequenceDiagram
347347
The `run_policy_poll_loop()` function in `crates/openshell-sandbox/src/lib.rs` implements this loop:
348348

349349
1. **Connect once**: Create a `CachedOpenShellClient` that holds a persistent mTLS channel to the gateway. This avoids TLS renegotiation on every poll.
350-
2. **Fetch initial version**: Call `poll_policy(sandbox_id)` to establish the baseline `current_version`. On failure, log a warning and retry on the next interval.
350+
2. **Fetch initial config revision**: Call `poll_policy(sandbox_id)` to establish baseline `current_config_revision`. On failure, log a warning and retry on the next interval.
351351
3. **Poll loop**: Sleep for the configured interval, then call `poll_policy()` again.
352-
4. **Version comparison**: If `result.version <= current_version`, skip. The version is a monotonically increasing `u32` per sandbox.
353-
5. **Reload attempt**: Call `opa_engine.reload_from_proto(&result.policy)`. This runs the full `from_proto()` pipeline on the new policy, then atomically swaps the inner engine.
354-
6. **Status reporting**: On success, report `PolicyStatus::Loaded` to the gateway via `ReportPolicyStatus` RPC. On failure, report `PolicyStatus::Failed` with the error message. Status report failures are logged but do not affect the poll loop.
352+
4. **Config comparison**: If `result.config_revision == current_config_revision`, skip.
353+
5. **Reload attempt**: Call `opa_engine.reload_from_proto(policy)` when a policy payload is present. This runs the full `from_proto()` pipeline on the new policy, then atomically swaps the inner engine.
354+
6. **Status reporting**: On success/failure, report status only for sandbox-scoped policy revisions (`policy_source = SANDBOX`, `version > 0`). Global policy overrides still reload, but they do not write per-sandbox policy status history.
355355

356356
### `CachedOpenShellClient`
357357

@@ -365,24 +365,26 @@ pub struct CachedOpenShellClient {
365365
}
366366

367367
pub struct PolicyPollResult {
368-
pub policy: ProtoSandboxPolicy,
368+
pub policy: Option<ProtoSandboxPolicy>,
369369
pub version: u32,
370370
pub policy_hash: String,
371+
pub config_revision: u64,
372+
pub policy_source: PolicySource,
371373
}
372374
```
373375

374376
Methods:
375377
- **`connect(endpoint)`**: Establish an mTLS channel and return a new client.
376-
- **`poll_policy(sandbox_id)`**: Call `GetSandboxPolicy` RPC and return a `PolicyPollResult` containing the policy, version, and hash.
378+
- **`poll_policy(sandbox_id)`**: Call `GetSandboxPolicy` RPC and return a `PolicyPollResult` containing policy payload (optional), policy metadata, effective config revision, and policy source.
377379
- **`report_policy_status(sandbox_id, version, loaded, error_msg)`**: Call `ReportPolicyStatus` RPC with the appropriate `PolicyStatus` enum value (`Loaded` or `Failed`).
378380
- **`raw_client()`**: Return a clone of the underlying `OpenShellClient<Channel>` for direct RPC calls (used by the log push task).
379381

380382
### Server-side policy versioning
381383

382-
The gateway assigns a monotonically increasing version number to each policy revision per sandbox. The `GetSandboxPolicyResponse` includes `version` and `policy_hash` fields. The `ReportPolicyStatus` RPC records which version the sandbox successfully loaded (or failed to load), enabling operators to query `GetSandboxPolicyStatus` for the current active version and load history.
384+
The gateway assigns a monotonically increasing version number to each sandbox policy revision. `GetSandboxPolicyResponse` now also carries effective settings and a `config_revision` fingerprint that changes when effective policy/settings change (including global overrides).
383385

384386
Proto messages involved:
385-
- `GetSandboxPolicyResponse` (`proto/sandbox.proto`): `policy`, `version`, `policy_hash`
387+
- `GetSandboxPolicyResponse` (`proto/sandbox.proto`): `policy`, `version`, `policy_hash`, `settings`, `config_revision`, `policy_source`
386388
- `ReportPolicyStatusRequest` (`proto/openshell.proto`): `sandbox_id`, `version`, `status` (enum), `load_error`
387389
- `PolicyStatus` enum: `PENDING`, `LOADED`, `FAILED`, `SUPERSEDED`
388390
- `SandboxPolicyRevision` (`proto/openshell.proto`): Full revision metadata including `created_at_ms`, `loaded_at_ms`

‎crates/openshell-cli/src/main.rs‎

Lines changed: 200 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -164,6 +164,7 @@ const HELP_TEMPLATE: &str = "\
164164
forward: Manage port forwarding to a sandbox
165165
logs: View sandbox logs
166166
policy: Manage sandbox policy
167+
settings: Manage sandbox and global settings
167168
provider: Manage provider configuration
168169
169170
\x1b[1mGATEWAY COMMANDS\x1b[0m
@@ -249,9 +250,18 @@ const POLICY_EXAMPLES: &str = "\x1b[1mALIAS\x1b[0m
249250
\x1b[1mEXAMPLES\x1b[0m
250251
$ openshell policy get my-sandbox
251252
$ openshell policy set my-sandbox --policy policy.yaml
253+
$ openshell policy set --global --policy policy.yaml
254+
$ openshell policy delete --global
252255
$ openshell policy list my-sandbox
253256
";
254257

258+
const SETTINGS_EXAMPLES: &str = "\x1b[1mEXAMPLES\x1b[0m
259+
$ openshell settings get my-sandbox
260+
$ openshell settings set my-sandbox --key log_level --value debug
261+
$ openshell settings set --global --key log_level --value warn
262+
$ openshell settings delete --global --key log_level
263+
";
264+
255265
const PROVIDER_EXAMPLES: &str = "\x1b[1mEXAMPLES\x1b[0m
256266
$ openshell provider create --name openai --type openai --credential OPENAI_API_KEY
257267
$ openshell provider create --name anthropic --type anthropic --from-existing
@@ -397,6 +407,13 @@ enum Commands {
397407
command: Option<PolicyCommands>,
398408
},
399409

410+
/// Manage sandbox and gateway settings.
411+
#[command(after_help = SETTINGS_EXAMPLES, help_template = SUBCOMMAND_HELP_TEMPLATE)]
412+
Settings {
413+
#[command(subcommand)]
414+
command: Option<SettingsCommands>,
415+
},
416+
400417
/// Manage network rules for a sandbox.
401418
#[command(visible_alias = "rl", hide = true, help_template = SUBCOMMAND_HELP_TEMPLATE)]
402419
Rule {
@@ -1324,14 +1341,22 @@ enum PolicyCommands {
13241341
/// Update policy on a live sandbox.
13251342
#[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")]
13261343
Set {
1327-
/// Sandbox name (defaults to last-used sandbox).
1344+
/// Sandbox name (defaults to last-used sandbox when not using --global).
13281345
#[arg(add = ArgValueCompleter::new(completers::complete_sandbox_names))]
13291346
name: Option<String>,
13301347

13311348
/// Path to the policy YAML file.
13321349
#[arg(long, value_hint = ValueHint::FilePath)]
13331350
policy: String,
13341351

1352+
/// Apply as a gateway-global policy for all sandboxes.
1353+
#[arg(long)]
1354+
global: bool,
1355+
1356+
/// Skip the confirmation prompt for global policy updates.
1357+
#[arg(long)]
1358+
yes: bool,
1359+
13351360
/// Wait for the sandbox to load the policy.
13361361
#[arg(long)]
13371362
wait: bool,
@@ -1368,6 +1393,61 @@ enum PolicyCommands {
13681393
#[arg(long, default_value_t = 20)]
13691394
limit: u32,
13701395
},
1396+
1397+
/// Delete the gateway-global policy lock, restoring sandbox-level policy control.
1398+
#[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")]
1399+
Delete {
1400+
/// Delete the global policy setting.
1401+
#[arg(long)]
1402+
global: bool,
1403+
},
1404+
}
1405+
1406+
#[derive(Subcommand, Debug)]
1407+
enum SettingsCommands {
1408+
/// Show effective settings for a sandbox.
1409+
#[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")]
1410+
Get {
1411+
/// Sandbox name (defaults to last-used sandbox).
1412+
#[arg(add = ArgValueCompleter::new(completers::complete_sandbox_names))]
1413+
name: Option<String>,
1414+
},
1415+
1416+
/// Set a single setting key.
1417+
#[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")]
1418+
Set {
1419+
/// Sandbox name (defaults to last-used sandbox when not using --global).
1420+
#[arg(add = ArgValueCompleter::new(completers::complete_sandbox_names))]
1421+
name: Option<String>,
1422+
1423+
/// Setting key.
1424+
#[arg(long)]
1425+
key: String,
1426+
1427+
/// Setting value (stored as string).
1428+
#[arg(long)]
1429+
value: String,
1430+
1431+
/// Apply at gateway-global scope.
1432+
#[arg(long)]
1433+
global: bool,
1434+
1435+
/// Skip the confirmation prompt for global setting updates.
1436+
#[arg(long)]
1437+
yes: bool,
1438+
},
1439+
1440+
/// Delete a single gateway-global setting key.
1441+
#[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")]
1442+
Delete {
1443+
/// Setting key.
1444+
#[arg(long)]
1445+
key: String,
1446+
1447+
/// Delete at gateway-global scope.
1448+
#[arg(long)]
1449+
global: bool,
1450+
},
13711451
}
13721452

13731453
#[derive(Subcommand, Debug)]
@@ -1730,12 +1810,26 @@ async fn main() -> Result<()> {
17301810
PolicyCommands::Set {
17311811
name,
17321812
policy,
1813+
global,
1814+
yes,
17331815
wait,
17341816
timeout,
17351817
} => {
1736-
let name = resolve_sandbox_name(name, &ctx.name)?;
1737-
run::sandbox_policy_set(&ctx.endpoint, &name, &policy, wait, timeout, &tls)
1818+
if global {
1819+
run::sandbox_policy_set_global(
1820+
&ctx.endpoint,
1821+
&policy,
1822+
yes,
1823+
wait,
1824+
timeout,
1825+
&tls,
1826+
)
17381827
.await?;
1828+
} else {
1829+
let name = resolve_sandbox_name(name, &ctx.name)?;
1830+
run::sandbox_policy_set(&ctx.endpoint, &name, &policy, wait, timeout, &tls)
1831+
.await?;
1832+
}
17391833
}
17401834
PolicyCommands::Get { name, rev, full } => {
17411835
let name = resolve_sandbox_name(name, &ctx.name)?;
@@ -1745,6 +1839,54 @@ async fn main() -> Result<()> {
17451839
let name = resolve_sandbox_name(name, &ctx.name)?;
17461840
run::sandbox_policy_list(&ctx.endpoint, &name, limit, &tls).await?;
17471841
}
1842+
PolicyCommands::Delete { global } => {
1843+
if !global {
1844+
return Err(miette::miette!(
1845+
"sandbox policy delete is not supported; use --global to remove global policy lock"
1846+
));
1847+
}
1848+
run::gateway_setting_delete(&ctx.endpoint, "policy", &tls).await?;
1849+
}
1850+
}
1851+
}
1852+
1853+
// -----------------------------------------------------------
1854+
// Settings commands
1855+
// -----------------------------------------------------------
1856+
Some(Commands::Settings {
1857+
command: Some(settings_cmd),
1858+
}) => {
1859+
let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?;
1860+
let mut tls = tls.with_gateway_name(&ctx.name);
1861+
apply_edge_auth(&mut tls, &ctx.name);
1862+
1863+
match settings_cmd {
1864+
SettingsCommands::Get { name } => {
1865+
let name = resolve_sandbox_name(name, &ctx.name)?;
1866+
run::sandbox_settings_get(&ctx.endpoint, &name, &tls).await?;
1867+
}
1868+
SettingsCommands::Set {
1869+
name,
1870+
key,
1871+
value,
1872+
global,
1873+
yes,
1874+
} => {
1875+
if global {
1876+
run::gateway_setting_set(&ctx.endpoint, &key, &value, yes, &tls).await?;
1877+
} else {
1878+
let name = resolve_sandbox_name(name, &ctx.name)?;
1879+
run::sandbox_setting_set(&ctx.endpoint, &name, &key, &value, &tls).await?;
1880+
}
1881+
}
1882+
SettingsCommands::Delete { key, global } => {
1883+
if !global {
1884+
return Err(miette::miette!(
1885+
"sandbox settings cannot be deleted; use --global"
1886+
));
1887+
}
1888+
run::gateway_setting_delete(&ctx.endpoint, &key, &tls).await?;
1889+
}
17481890
}
17491891
}
17501892

@@ -2229,6 +2371,13 @@ async fn main() -> Result<()> {
22292371
.print_help()
22302372
.expect("Failed to print help");
22312373
}
2374+
Some(Commands::Settings { command: None }) => {
2375+
Cli::command()
2376+
.find_subcommand_mut("settings")
2377+
.expect("settings subcommand exists")
2378+
.print_help()
2379+
.expect("Failed to print help");
2380+
}
22322381
Some(Commands::Provider { command: None }) => {
22332382
Cli::command()
22342383
.find_subcommand_mut("provider")
@@ -2803,4 +2952,52 @@ mod tests {
28032952
other => panic!("expected SshProxy, got: {other:?}"),
28042953
}
28052954
}
2955+
2956+
#[test]
2957+
fn settings_set_global_parses_yes_flag() {
2958+
let cli = Cli::try_parse_from([
2959+
"openshell",
2960+
"settings",
2961+
"set",
2962+
"--global",
2963+
"--key",
2964+
"log_level",
2965+
"--value",
2966+
"warn",
2967+
"--yes",
2968+
])
2969+
.expect("settings set --global should parse");
2970+
2971+
match cli.command {
2972+
Some(Commands::Settings {
2973+
command:
2974+
Some(SettingsCommands::Set {
2975+
global,
2976+
yes,
2977+
key,
2978+
value,
2979+
..
2980+
}),
2981+
}) => {
2982+
assert!(global);
2983+
assert!(yes);
2984+
assert_eq!(key, "log_level");
2985+
assert_eq!(value, "warn");
2986+
}
2987+
other => panic!("expected settings set command, got: {other:?}"),
2988+
}
2989+
}
2990+
2991+
#[test]
2992+
fn policy_delete_global_parses() {
2993+
let cli = Cli::try_parse_from(["openshell", "policy", "delete", "--global"])
2994+
.expect("policy delete --global should parse");
2995+
2996+
match cli.command {
2997+
Some(Commands::Policy {
2998+
command: Some(PolicyCommands::Delete { global }),
2999+
}) => assert!(global),
3000+
other => panic!("expected policy delete command, got: {other:?}"),
3001+
}
3002+
}
28063003
}

0 commit comments

Comments
 (0)