Skip to content

Commit 80a2f8a

Browse files
committed
feat(supervisor): gate the agent trace relay behind [otlp.agent]
The relay started whenever a sandbox supervisor had a trace endpoint, so enabling gateway trace export also changed every workload's environment. Make it opt-in: `[openshell.gateway.otlp.agent]` with `enabled = true` turns it on, and its optional `endpoint` falls back to the shared `[openshell.gateway.otlp]` endpoint, like the OCSF table. The gateway validates the endpoint and drivers pass it to supervisors as OPENSHELL_OTLP_AGENT_ENDPOINT, separate from OPENSHELL_OTLP_ENDPOINT. Docker, Podman (including the Podman Machine host alias), Kubernetes and VM drivers carry it, in-process and as standalone binaries. The shared Kubernetes supervisor still does not relay. The relay now acknowledges OTLP metrics and logs exports with an empty success and discards them, so agent SDKs that export every signal to the generic endpoint stop retrying. Discards are counted and logged once. Local dev gateway tasks, the Skaffold values and the e2e harness enable the table; OPENSHELL_E2E_OTLP_AGENT=0 runs the inert case with trace export still on. Signed-off-by: Roland Huß <[email protected]>
1 parent 7537822 commit 80a2f8a

38 files changed

Lines changed: 556 additions & 57 deletions

File tree

‎.agents/skills/helm-dev-environment/SKILL.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -170,9 +170,10 @@ trace-context annotation. The same command exposes OTLP/gRPC on
170170
`http://127.0.0.1:4317` and, when deployed, the Kubernetes gateway on
171171
`http://127.0.0.1:8090`. The local `gateway:docker`, `gateway:podman`, and
172172
`gateway:vm` tasks detect the collector listener at startup and enable trace
173-
export only while it is reachable. Spans an agent exports from inside a sandbox
174-
arrive through the supervisor's relay with `openshell.telemetry.source=agent`
175-
and share `openshell.sandbox.id` with that sandbox's supervisor spans.
173+
export, OCSF log export, and the agent trace relay only while it is reachable.
174+
Spans an agent exports from inside a sandbox arrive through the supervisor's
175+
relay with `openshell.telemetry.source=agent` and share `openshell.sandbox.id`
176+
with that sandbox's supervisor spans.
176177

177178
The Skaffold profile for HA reverse-proxy development is available from
178179
`deploy/helm/openshell/`:

‎crates/openshell-core/src/sandbox_env.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,12 @@ pub const OTLP_ENDPOINT: &str = "OPENSHELL_OTLP_ENDPOINT";
3232
/// OTLP/gRPC collector endpoint for supervisor OCSF event export.
3333
pub const OTLP_OCSF_ENDPOINT: &str = "OPENSHELL_OTLP_OCSF_ENDPOINT";
3434

35+
/// OTLP/gRPC collector endpoint that supervisors relay workload traces to.
36+
///
37+
/// Set only when the gateway enables `[openshell.gateway.otlp.agent]`; absent
38+
/// keeps the relay inactive and leaves the workload environment unchanged.
39+
pub const OTLP_AGENT_ENDPOINT: &str = "OPENSHELL_OTLP_AGENT_ENDPOINT";
40+
3541
/// OCSF schema version that supervisors downgrade exported events to.
3642
pub const OTLP_OCSF_SCHEMA_VERSION: &str = "OPENSHELL_OTLP_OCSF_SCHEMA_VERSION";
3743

‎crates/openshell-driver-docker/src/lib.rs‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -212,6 +212,11 @@ pub struct DockerComputeConfig {
212212
#[serde(skip)]
213213
pub supervisor_ocsf_export: Option<openshell_core::sandbox_env::OcsfOtlpExport>,
214214

215+
/// OTLP/gRPC collector that supervisors relay workload traces to. The
216+
/// gateway supplies it from `[otlp.agent]`; driver TOML cannot set it.
217+
#[serde(skip)]
218+
pub supervisor_otlp_agent_endpoint: Option<String>,
219+
215220
/// Gateway name passed to supervisors for their telemetry. The gateway
216221
/// supplies its own name; driver TOML cannot set it.
217222
#[serde(skip)]
@@ -319,6 +324,7 @@ impl Default for DockerComputeConfig {
319324
supervisor_network_mode: DockerSupervisorNetworkMode::Host,
320325
host_gateway_ip: None,
321326
supervisor_ocsf_export: None,
327+
supervisor_otlp_agent_endpoint: None,
322328
supervisor_gateway_name: None,
323329
sandbox_runtime_image: None,
324330
supervisor_bin: None,
@@ -360,6 +366,7 @@ struct DockerDriverRuntimeConfig {
360366
bridge_default_endpoint: Option<String>,
361367
host_gateway_ip: Option<IpAddr>,
362368
supervisor_ocsf_export: Option<openshell_core::sandbox_env::OcsfOtlpExport>,
369+
supervisor_otlp_agent_endpoint: Option<String>,
363370
supervisor_gateway_name: Option<String>,
364371
ssh_socket_path: String,
365372
guest_tls: Option<DockerGuestTlsPaths>,
@@ -1018,6 +1025,9 @@ impl DockerComputeDriver {
10181025
bridge_default_endpoint,
10191026
host_gateway_ip,
10201027
supervisor_ocsf_export: docker_config.supervisor_ocsf_export.clone(),
1028+
supervisor_otlp_agent_endpoint: docker_config
1029+
.supervisor_otlp_agent_endpoint
1030+
.clone(),
10211031
supervisor_gateway_name: docker_config.supervisor_gateway_name.clone(),
10221032
ssh_socket_path: docker_config.ssh_socket_path.clone(),
10231033
guest_tls,
@@ -5830,7 +5840,8 @@ fn docker_child_environment(sandbox: &DriverSandbox) -> HashMap<String, String>
58305840
}
58315841

58325842
/// Environment that lets the supervisor export spans, joining the current
5833-
/// trace, and OCSF events, identifying the gateway that launched it.
5843+
/// trace, OCSF events, and relayed workload traces, identifying the gateway
5844+
/// that launched it.
58345845
fn supervisor_otlp_environment(config: &DockerDriverRuntimeConfig) -> Vec<String> {
58355846
let traces = config.supervisor_otlp_endpoint.iter().flat_map(|endpoint| {
58365847
std::iter::once((openshell_core::sandbox_env::OTLP_ENDPOINT, endpoint.clone()))
@@ -5840,12 +5851,22 @@ fn supervisor_otlp_environment(config: &DockerDriverRuntimeConfig) -> Vec<String
58405851
.supervisor_ocsf_export
58415852
.iter()
58425853
.flat_map(openshell_core::sandbox_env::OcsfOtlpExport::environment);
5854+
let agent = config
5855+
.supervisor_otlp_agent_endpoint
5856+
.iter()
5857+
.map(|endpoint| {
5858+
(
5859+
openshell_core::sandbox_env::OTLP_AGENT_ENDPOINT,
5860+
endpoint.clone(),
5861+
)
5862+
});
58435863
let gateway_name = config
58445864
.supervisor_gateway_name
58455865
.iter()
58465866
.map(|name| (openshell_core::sandbox_env::GATEWAY_NAME, name.clone()));
58475867
traces
58485868
.chain(ocsf)
5869+
.chain(agent)
58495870
.chain(gateway_name)
58505871
.map(|(name, value)| format!("{name}={value}"))
58515872
.collect()

‎crates/openshell-driver-docker/src/main.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,11 @@ struct Args {
5959
)]
6060
otlp_ocsf_schema_version: Option<String>,
6161

62+
/// OTLP/gRPC collector endpoint that supervisors relay workload traces
63+
/// to. Absent keeps the agent trace relay off.
64+
#[arg(long, env = "OPENSHELL_OTLP_AGENT_ENDPOINT")]
65+
otlp_agent_endpoint: Option<String>,
66+
6267
#[arg(long, env = "OPENSHELL_GATEWAY_NAME")]
6368
gateway_name: Option<String>,
6469
}
@@ -90,6 +95,7 @@ async fn main() -> Result<()> {
9095
docker_config.supervisor_image = Some(image);
9196
}
9297
docker_config.supervisor_otlp_endpoint = args.otlp_endpoint.clone();
98+
docker_config.supervisor_otlp_agent_endpoint = args.otlp_agent_endpoint.clone();
9399
docker_config.supervisor_gateway_name = args.gateway_name.clone();
94100
docker_config.supervisor_ocsf_export = args.otlp_ocsf_endpoint.clone().map(|endpoint| {
95101
openshell_core::sandbox_env::OcsfOtlpExport {

‎crates/openshell-driver-docker/src/tests.rs‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -330,6 +330,7 @@ fn runtime_config() -> DockerDriverRuntimeConfig {
330330
bridge_default_endpoint: Some("https://host.docker.internal:8443".to_string()),
331331
host_gateway_ip: None,
332332
supervisor_ocsf_export: None,
333+
supervisor_otlp_agent_endpoint: None,
333334
supervisor_gateway_name: None,
334335
ssh_socket_path: openshell_core::container_paths::SSH_SOCKET_PATH.to_string(),
335336
guest_tls: Some(DockerGuestTlsPaths {
@@ -1513,7 +1514,8 @@ fn docker_child_environment_strips_supervisor_control_keys() {
15131514
#[test]
15141515
fn supervisor_otlp_environment_carries_only_configured_signals() {
15151516
use openshell_core::sandbox_env::{
1516-
GATEWAY_NAME, OTLP_ENDPOINT, OTLP_OCSF_ENDPOINT, OTLP_OCSF_SCHEMA_VERSION, OcsfOtlpExport,
1517+
GATEWAY_NAME, OTLP_AGENT_ENDPOINT, OTLP_ENDPOINT, OTLP_OCSF_ENDPOINT,
1518+
OTLP_OCSF_SCHEMA_VERSION, OcsfOtlpExport,
15171519
};
15181520

15191521
let mut config = runtime_config();
@@ -1538,6 +1540,13 @@ fn supervisor_otlp_environment_carries_only_configured_signals() {
15381540
]
15391541
);
15401542

1543+
config.supervisor_ocsf_export = None;
1544+
config.supervisor_otlp_agent_endpoint = Some("http://agents:4317".to_string());
1545+
assert_eq!(
1546+
supervisor_otlp_environment(&config),
1547+
[format!("{OTLP_AGENT_ENDPOINT}=http://agents:4317")]
1548+
);
1549+
15411550
config.supervisor_gateway_name = Some("prod-west".to_string());
15421551
assert_eq!(
15431552
supervisor_otlp_environment(&config).last(),

‎crates/openshell-driver-kubernetes/src/config.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -372,6 +372,10 @@ pub struct KubernetesComputeConfig {
372372
/// export; driver TOML cannot set it.
373373
#[serde(skip)]
374374
pub supervisor_ocsf_export: Option<openshell_core::sandbox_env::OcsfOtlpExport>,
375+
/// OTLP/gRPC collector that supervisors relay workload traces to. The
376+
/// gateway supplies it from `[otlp.agent]`; driver TOML cannot set it.
377+
#[serde(skip)]
378+
pub supervisor_otlp_agent_endpoint: Option<String>,
375379

376380
/// Gateway name passed to supervisors for their telemetry. The gateway
377381
/// supplies its own name; driver TOML cannot set it.
@@ -502,6 +506,7 @@ impl Default for KubernetesComputeConfig {
502506
grpc_endpoint: String::new(),
503507
supervisor_otlp_endpoint: None,
504508
supervisor_ocsf_export: None,
509+
supervisor_otlp_agent_endpoint: None,
505510
supervisor_gateway_name: None,
506511
ssh_socket_path: openshell_core::container_paths::SSH_SOCKET_PATH.to_string(),
507512
client_tls_secret_name: String::new(),

‎crates/openshell-driver-kubernetes/src/driver.rs‎

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -912,7 +912,8 @@ impl std::fmt::Debug for KubernetesComputeDriver {
912912

913913
impl KubernetesComputeDriver {
914914
/// Environment that lets a supervisor export spans, joining the current
915-
/// trace, and OCSF events, identifying the gateway that launched it.
915+
/// trace, OCSF events, and relayed workload traces, identifying the
916+
/// gateway that launched it.
916917
fn supervisor_otlp_environment(&self) -> Vec<(&'static str, String)> {
917918
let mut environment = Vec::new();
918919
if let Some(endpoint) = self.config.supervisor_otlp_endpoint.as_deref() {
@@ -925,6 +926,12 @@ impl KubernetesComputeDriver {
925926
if let Some(export) = &self.config.supervisor_ocsf_export {
926927
environment.extend(export.environment());
927928
}
929+
if let Some(endpoint) = &self.config.supervisor_otlp_agent_endpoint {
930+
environment.push((
931+
openshell_core::sandbox_env::OTLP_AGENT_ENDPOINT,
932+
endpoint.clone(),
933+
));
934+
}
928935
if let Some(name) = &self.config.supervisor_gateway_name {
929936
environment.push((openshell_core::sandbox_env::GATEWAY_NAME, name.clone()));
930937
}
@@ -1078,10 +1085,11 @@ impl KubernetesComputeDriver {
10781085
}
10791086
if shared_supervisor.is_some()
10801087
&& (config.supervisor_otlp_endpoint.is_some()
1081-
|| config.supervisor_ocsf_export.is_some())
1088+
|| config.supervisor_ocsf_export.is_some()
1089+
|| config.supervisor_otlp_agent_endpoint.is_some())
10821090
{
10831091
warn!(
1084-
"the gateway supervisor topology does not export supervisor OTLP traces or OCSF events; shared sandbox instances keep console logs and gateway log push"
1092+
"the gateway supervisor topology does not export supervisor OTLP traces or OCSF events or relay workload traces; shared sandbox instances keep console logs and gateway log push"
10851093
);
10861094
}
10871095
let driver = Self {
@@ -8757,6 +8765,18 @@ mod tests {
87578765
]
87588766
);
87598767

8768+
let driver = KubernetesComputeDriver::new_for_test(KubernetesComputeConfig {
8769+
supervisor_otlp_agent_endpoint: Some("http://agents:4317".into()),
8770+
..Default::default()
8771+
});
8772+
assert_eq!(
8773+
driver.supervisor_otlp_environment(),
8774+
[(
8775+
openshell_core::sandbox_env::OTLP_AGENT_ENDPOINT,
8776+
"http://agents:4317".to_string()
8777+
)]
8778+
);
8779+
87608780
let driver = KubernetesComputeDriver::new_for_test(KubernetesComputeConfig {
87618781
supervisor_gateway_name: Some("prod-west".into()),
87628782
..Default::default()

‎crates/openshell-driver-kubernetes/src/main.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,11 @@ struct Args {
5757
)]
5858
otlp_ocsf_schema_version: Option<String>,
5959

60+
/// OTLP/gRPC collector endpoint that supervisors relay workload traces
61+
/// to. Absent keeps the agent trace relay off.
62+
#[arg(long, env = "OPENSHELL_OTLP_AGENT_ENDPOINT")]
63+
otlp_agent_endpoint: Option<String>,
64+
6065
#[arg(long, env = "OPENSHELL_GATEWAY_NAME")]
6166
gateway_name: Option<String>,
6267

@@ -298,6 +303,7 @@ async fn main() -> Result<()> {
298303
proxy_ca_bundle: args.proxy_ca_bundle,
299304
grpc_endpoint: args.grpc_endpoint.unwrap_or_default(),
300305
supervisor_otlp_endpoint: args.otlp_endpoint.clone(),
306+
supervisor_otlp_agent_endpoint: args.otlp_agent_endpoint.clone(),
301307
supervisor_gateway_name: args.gateway_name.clone(),
302308
supervisor_ocsf_export: args.otlp_ocsf_endpoint.clone().map(|endpoint| {
303309
openshell_core::sandbox_env::OcsfOtlpExport {

‎crates/openshell-driver-podman/src/config.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,10 @@ pub struct PodmanComputeConfig {
5151
/// export; driver TOML cannot set it.
5252
#[serde(skip)]
5353
pub supervisor_ocsf_export: Option<openshell_core::sandbox_env::OcsfOtlpExport>,
54+
/// OTLP/gRPC collector that supervisors relay workload traces to. The
55+
/// gateway supplies it from `[otlp.agent]`; driver TOML cannot set it.
56+
#[serde(skip)]
57+
pub supervisor_otlp_agent_endpoint: Option<String>,
5458

5559
/// Gateway name passed to supervisors for their telemetry. The gateway
5660
/// supplies its own name; driver TOML cannot set it.
@@ -540,6 +544,7 @@ impl Default for PodmanComputeConfig {
540544
grpc_endpoint: String::new(),
541545
supervisor_otlp_endpoint: None,
542546
supervisor_ocsf_export: None,
547+
supervisor_otlp_agent_endpoint: None,
543548
supervisor_gateway_name: None,
544549
gateway_port: openshell_core::config::DEFAULT_SERVER_PORT,
545550
ssh_socket_path: openshell_core::container_paths::SSH_SOCKET_PATH.to_string(),

‎crates/openshell-driver-podman/src/container.rs‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -717,6 +717,12 @@ fn build_env(
717717
.map(|(name, value)| (name.to_string(), value)),
718718
);
719719
}
720+
if let Some(endpoint) = &config.supervisor_otlp_agent_endpoint {
721+
env.insert(
722+
openshell_core::sandbox_env::OTLP_AGENT_ENDPOINT.into(),
723+
endpoint.clone(),
724+
);
725+
}
720726
if let Some(name) = &config.supervisor_gateway_name {
721727
env.insert(
722728
openshell_core::sandbox_env::GATEWAY_NAME.into(),
@@ -2774,6 +2780,22 @@ mod tests {
27742780
spec["env"][openshell_core::sandbox_env::OTLP_OCSF_SCHEMA_VERSION],
27752781
"1.1"
27762782
);
2783+
assert!(
2784+
spec["env"]
2785+
.get(openshell_core::sandbox_env::OTLP_AGENT_ENDPOINT)
2786+
.is_none(),
2787+
"the agent trace relay stays off unless the gateway enables it"
2788+
);
2789+
2790+
let config = PodmanComputeConfig {
2791+
supervisor_otlp_agent_endpoint: Some("http://agents:4317".to_string()),
2792+
..test_config()
2793+
};
2794+
let spec = build_container_spec(&sandbox, &config);
2795+
assert_eq!(
2796+
spec["env"][openshell_core::sandbox_env::OTLP_AGENT_ENDPOINT],
2797+
"http://agents:4317"
2798+
);
27772799

27782800
let config = PodmanComputeConfig {
27792801
supervisor_gateway_name: Some("prod-west".to_string()),

0 commit comments

Comments
 (0)